DESCRIPTION
Title of Invention:
IMAGE PROCESSING APPARATUS, IMAGING APPARATUS, IMAGE
PROCESSING METHOD, AND PROGRAM
Technical Field
[0001]
The present invention relates to an image processing
apparatus, an imaging apparatus, an image processing method,
and a program, and, more particularly, to an image processing
apparatus, an imaging apparatus, an image processing method,
and a program that perform the process of generating an image
used for displaying a three-dimensional image (3D image) using
a plurality of images captured while moving a camera.
Background Art
[0002]
In order to generate a three-dimensional image (also
called a 3D image or a stereoscopic image), it is necessary
to capture images from mutually different viewpoints, in other
words, a left-eye image and a right-eye image. Methods of
capturing images from mutually different viewpoints are
largely divided into two methods.
A first technique is a technique of simultaneously
imaging a subject from different viewpoints using a plurality
1
of camera units, that is, a technique using a so-called
multi-lens camera.
A second technique is a technique of consecutively
capturing images from mutually different viewpoints by moving
an imaging apparatus using a single camera unit, that is, a
technique using a so-called single-lens camera.
[0003]
For example, a multi-lens camera system that is used for
the above-described first technique has a configuration in
which lenses are included at positions separated from each
other and a subject can be simultaneously photographed from
mutually different viewpoints . However, a plurality of camera
units are necessary for such a multi-lens camera system, and
accordingly, there is a problem in that the camera system is
high priced.
[0004]
In contrast to this, a single-lens camera system that
is used for the above-described second technique may have a
configuration including one camera unit, which is similar to
the configuration of a camera in related art. In such a
configuration, images from mutually different viewpoints are
consecutively captured while moving a camera that includes one
camera unit, and a three-dimensional image is generated by
using a plurality of captured images.
As above, in a case where a single-lens camera system
2
is used, a relatively low-cost system can be realized by using
one camera unit, which is similar to a camera in related art.
[0005]
In addition, as a technique in related art that discloses
a technique of acquiring distance information of a subject from
images captured while a single-lens camera is moved, there is
NPL 1 "Acquiring Omni-directional Range Information (The
Transactions of the Institute of Electronics, Information and
Communication Engineers, D-II, Vol, J74-D-II, No. 4, 1991)".
In addition, also in NPL 2 "Omni -Directional Stereo", IEEE
Transaction On Pattern Analysis And Machine Intelligence, VOL.
14, No. 2, February 1992", a report of the same content as that
of NPL 1 is disclosed.
[0006]
In NPL 1 and NPL 2, a technique is disclosed in which
a camera is fixedly installed on a circumference that is
separated from the rotation center of a rotation target by a
predetermined distance, and distance information of a subject
is acquired by using two images acquired through two vertical
slits by consecutively capturing images while rotating a
rotation base.
[0007]
In addition, in PTL 1 (Japanese Unexamined Patent
Application Publication No. 11-164325), similarly to the
configurations disclosed in NPL 1 and NPL 2, a configuration
, 3
is disclosed in which images are captured while a camera is
installed to be separated from the rotation center of a rotation
target by a predetermined distance and is rotated, and by using
two images acquired through two slits, a left-eye panoramic
image and a right-eye panoramic image that are used for
displaying a three-dimensional image are acquired.
[0008]
As above, in the techniques in related art, it is
disclosed that, by using images acquired through slits while
a camera is rotated, a left-eye image and a right-eye image
that are used for displaying a three-dimensional image can be
acquired.
[0009]
Meanwhile, a technique for generating a panoramic image,
that is, a horizontally-long two-dimensional image by
capturing images while a camera is moved and connecting a
plurality of captured images is known. For example, in PTL
2 (Japanese Patent No. 3928222), PTL 3 (Japanese Patent No.
4293053), and the like, techniques for generating a panoramic
image are disclosed.
As above, when a two-dimensional panoramic image is
generated, a plurality of captured images acquired while a
camera is moved is used.
[0010]
In NPL 1, NPL 2, and PTL 1 described above, a principle
4
of acquiring a left-eye image and a right-eye image as
three-dimensional images by cutting out and connecting images
of predetermined areas using a plurality of images captured
by a capturing process such as a panoramic image generating
process is described.
[0011]
However, in a case where a left-eye image and a right-eye
image as a three-dimensional image or a two-dimensional
panoramic image is generated by cutting out images of
predetermined areas from a plurality of captured images
captured with a camera moving and connecting the images, for
example, through a user's swinging operation of the camera on
his hands, there is a case where the left-eye image and the
right-eye image for displaying a three-dimensional image
cannot be generated depending on the form of the movement of
the camera that is performed by a user. In addition, there
is a case where a two-dimensional panoramic image cannot be
generated. As a result, meaningless image data is recorded
on a medium as recording data, and a situation may occur in
which an image not according to user' s intention is reproduced
at the time of reproduction, or an image cannot be reproduced.
Citation List
[0012]
Patent Literature
[PTL 1] JP-A-11-164326
5
[PTL 2] Japanese Patent No. 3928222
[PTL 3] Japanese Patent No. 4293053
[0013]
Non Patent Literature
[NPL 1] "Acquiring Omni-directional Range Information
(The Transactions of the Institute of Electronics, Information
and Communication Engineers, D-II, Vol. J74-D-II, No. 4, 1991)"
[NPL 2] "Omni -Directional Stereo", IEEE Transaction
On Pattern Analysis And Machine Intelligence, VOL. 14, No. 2,
February 1992"
Summary of Invention
Technical Problem
[0014]
The present invention, for example, is devised in
consideration of the above-described problems, and an object
thereof is to provide an image processing apparatus, an imaging
apparatus, an image processing method, and a program, in a
configuration in which a left-eye image and a right-eye image
used for displaying' a three-dimensional image or a
two-dimensional panoramic image is generated from a plurality
of images captured while a camera is moved, capable of
performing an optimal image generating process in accordance
with the rotation or the movement state of the camera and, in
a case where the 2D panoramic image or the 3D image cannot be
generated, warning a user of such a situation.
6
Solution to Problem
[0015]
According to a first aspect of the present invention,
there is provided an image processing apparatus including: an
image composing unit that receives a plurality of images
captured from mutually different positions as inputs and
generates a composition image by connecting stripped areas that
are cut out .from each of the images, wherein the image composing
unit determines one processing aspect based on movement
information of an imaging apparatus at the time of capturing
an image from among (a) a composition image generating process
of a left-eye composition image and a right-eye composition
image that are used for displaying a three-dimensional image,
(b) a composition image generating process of a two-dimensional
panoramic image, and (c) stopping composition image generating
and performs the determined process.
[0016]
In addition, in an embodiment of the image processing
apparatus of the present invention, the above-described image
processing apparatus further includes: a turning momentum
detecting unit that acquires or calculates turning momentum
(0) of the imaging apparatus at the time of capturing an image;
and a translational momentum detecting unit that acquires or
calculates translational momentum (t) of the imaging apparatus
at the time of capturing an image, wherein the image composing
7
unit determines the processing aspect based on the turning
momentum (9) detected by the turning momentum detecting unit
and the translational momentum (t) detected by the
translational momentum detecting unit.
[0017]
Furthermore, in an embodiment of the image processing
apparatus of the present invention, the above-described image
processing apparatus further includes an output unit that
presents a warning or a notification to a user in accordance
with information of the determination of the image composing
unit.
[0018]
In addition, in an embodiment of the image processing
apparatus of the present invention, the above-described image
composing unit stops the composition image generating process
of the three-dimensional image and the two-dimensional
panoramic image in a case where the turning momentum (6)
detected by the turning momentum detecting unit is zero.
[0019]
Furthermore, in an embodiment of the image processing
apparatus of the present invention, the above-described image
composing unit performs one of the composition image generating
process of a two-dimensional panoramic image and the stopping
composition image generating in a case where the turning
momentum (9) detected by the turning momentum detecting unit
8
is not zero, and the translational momentum (t) detected by
the translational momentum detecting- unit is zero.
[0020]
In addition, in an embodiment of the image processing
apparatus of the present invention, the above-described image
composing unit performs one of the composition image generating
process of a three-dimensional image and the composition image
generating process of a two-dimensional panoramic image in a
case where the turning momentum (8) detected by the turning
momentum detecting unit is not zero, and the translational
momentum (t) detected by the translational momentum detecting
unit is not zero.
[0021]
Furthermore, in an embodiment of the image processing
apparatus of the present invention, in a case where case where
the turning momentum (9)'detected by the turning momentum
detecting unit is not zero, and the translational momentum (t)
detected by the translational momentum detecting unit is not
zero, the image composing unit performs a process in which LR
images of the 3D image, which are to be generated, are reversely
set in a case where 9-t < 0 and in a case where 9-t > 0.
[0022]
In addition, in an embodiment of the image processing
apparatus of the present invention, the above-described
turning momentum detecting unit is a sensor that detects the
9
turning momentum of the image processing apparatus.
[0023]
Furthermore, in an embodiment of the image processing
apparatus of the present invention, the translational momentum
detecting unit is a sensor that detects the translational
momentum of the image processing apparatus.
[0024]
In addition, in an embodiment of the image processing
apparatus of the present invention,, the above-described
turning momentum detecting unit is an image analyzing unit that
detects the turning momentum at the time of capturing an image
by analyzing captured images.
[0025]
Furthermore, in an embodiment of the image processing
apparatus of the present invention, the above-described
translational momentum detecting unit is an image analyzing
unit that detects the translational momentum at the time of
capturing an image by analyzing captured images.
[0026]
In addition, according to a second aspect of the present
invention, there is provided an imaging apparatus, including:
an imaging unit; and an image processing unit that performs
the image processing according to any one of claims 1 to 11.
[0027]
Furthermore, according to a third aspect of the present
10
invention, there is provided an image processing method that
is performed in an image processing apparatus including:
receiving a plurality of images captured from mutually
different positions as inputs and generating a composition
image by connecting stripped areas that are cut out from each
of the images by using an image composing unit, wherein, in
the receiving of a plurality of images and generating of a
composition image, one processing aspect is determined based
on movement information of an imaging apparatus at the time
of capturing an image from among (a) a composition image
generating process of a left-eye composition image and a
right-eye composition image that are used for displaying a
three-dimensional image, (b) a composition image generating
process of a two-dimensional panoramic image, and (c) stopping
composition image generating, and the determined process is
performed.
[0028]
In addition, according to a fourth aspect of the present
invention, there is provided a program that causes an image
processing apparatus to perform image processing, the program
causing an image composing unit to perform receiving a
plurality of images captured from mutually different positions
as inputs and generating a composition image by connecting
stripped areas that are cut out from each of the images, wherein,
in the receiving of a plurality of images and generating of
11
a composition image, one processing aspect is determined based
on movement information of an imaging apparatus at the time
of capturing an image from among (a) a composition image
generating process of a left-eye composition image and a
right-eye composition image that are used for displaying a
three-dimensional image, (b) a composition image generating
process of a two-dimensional panoramic image, and (c) stopping
composition image generating, and the determined process is
performed.
[0029]
In addition, the program according to the present
invention, for example, is a program that can be provided as
a storage medium in a computer-readable form for an information
processing apparatus or a computer system that can execute
various program codes or a communication medium. By providing
such a program in a computer-readable form, a process according
to the program is realized on the information processing
apparatus or the computer system.
[0030]
Other features and advantages of the present invention
will become more apparent by describing in detail exemplary,
embodiments with reference to the attached drawings. In
addition, a system described in this specif ication is a logical
aggregated configuration of a plurality of apparatuses, and
the apparatuses of each configuration are not limited to be
12
disposed inside a same casing.
Advantageous Effects of Invention
[0031]
According to the configuration of an embodiment of the
present invention, in a configuration in which a
two-dimensional panoramic image or images' for displaying a
three-dimensional image are generated by connecting stripped
areas cut out from a plurality of images, a configuration is
realizes in which a composition image that can be generated
is generated based on the movement of a camera, and the
determined composition image is generated. In a configuration
in which a two-dimensional panoramic image or a left-eye
composition image and a right-eye composition image used for
displaying a three-dimensional image are generated by
connecting striped areas cut out from a plurality of images,
the information of the movement of the imaging apparatus at
the time of capturing an image is analyzed, it is determined
whether a two-dimensional panoramic image or a
three-dimensional image can be generated, and the process of
generating a composition image that can be generated is
performed. In accordance with the turning momentum (6) and
the translational momentum (t) of the camera at the time of
capturing an image, one processing aspect is determined from
among (a) a composition image generating process of a left-eye
composition image and a right-eye composition image that are
13
used for displaying a three-dimensional image, (b) a
composition image generating process of a two-dimensional
panoramic image, and (c) stopping composition image generating,
and the determined process is performed. In addition, a
notification of the content of the processing or a warning is
presented to a user.
Brief Description of Drawings
[0032]
Fig. 1 is a diagram that illustrates a panoramic image
generating process.
Fig. 2 is a diagram that illustrates the process of
generating a left-eye image (L image) and a right-eye image
(R image) that are used for displaying a three-dimensional (3D)
image.
Fig. 3 is a diagram that illustrates a principle of
generating a left-eye image (L image) and a right-eye image
(R image) used for displaying a three-dimensional (3D) image.
Fig. 4 is a diagram that illustrates a reverse model using
a virtual imaging surface.
Fig. 5 is a diagram that illustrates a model for a process
of capturing a panoramic image (3D panoramic image).
Fig. 6 is a diagram that illustrates an image captured
in a panoramic image (3D panoramic image) capturing process
and an example of the setting of strips of a left-eye image
14
and a right-eye image.
Fig. 7 is a diagram that illustrates examples of a
stripped area connecting process and the process of generating
a 3D left-eye composition image (3D panoramic L image) and a
3D right-eye composition image (3D panoramic R image).
Fig. 8 is a diagram that illustrates an example of an
ideal camera moving process in a case where a 3D image or a
2D panoramic image is generated by cutting out stripped areas
from a plurality of images that are consecutively imaged while
a camera is moved. .
Fig. 9 is a diagram that illustrates an example of a camera
moving process for which a 3D image or a 2D panoramic image
cannot be generated by cutting out stripped areas from a
plurality of images that are consecutively imaged while a
camera is moved.
Fig. 10 is a diagram that illustrates a configuration
example of an imaging apparatus that is an image processing
apparatus according to an embodiment of the present invention.
Fig. 11 is a diagram that shows a flowchart illustrating
the sequence of an image capturing and composing process that
is performed by an image processing apparatus according to the
present invention.
Fig. 12 is a diagram that shows a flowchart illustrating
the sequence of a process determining process that is performed
by an image processing apparatus according to the present
15
invention.
Fig. 13 is a diagram that illustrates detection
information detected by a turning momentum detecting unit 211
and a translational momentum detecting unit 212 and processes
determined in accordance with the detection information
together.
Description of Embodiments
[0033]
Hereinafter, an image processing apparatus, an imaging
apparatus, an image processing method, and a program according
to the present invention will be described with reference to
the drawings. The description will be presented in the
following order of items.
1. Basic Configuration for Process of Generating Panoramic
Image and Three-dimensional (3D) Image
2. Problem in Generating 3D Image Using Stripped Areas of
Plurality of Images Captured While Camera Is Moved
3. Configuration Example of Image Processing Apparatus
According to Present Invention
4. Sequence of Image Capturing and Image Processing
5. Specific Configuration Example of Turning Momentum
Detecting Unit and Translational Momentum Detecting Unit
6. Example of Switching Between Processes Based On Turning
Momentum and Translational Momentum
16
[0034]
1. Basic Configuration for Process of Generating Panoramic
Image and Three-dimensional (3D) Image
The present invention relates to a process of generating
a left-eye image (L image) and a right-eye image (R image) used
for displaying a three-dimensional (3D) image by connecting
areas (stripped areas) of images that are cut out in the shape
of a strip by using a plurality of the images consecutively
captured while an imaging apparatus (camera) is moved.
[0035]
Cameras capable of generating a two-dimensional
panoramic image (2D panoramic image) using a plurality of
images that are consecutively captured while moving the cameras
have already been realized and used. First, a process of
generating a panoramic image (20 panoramic image) that is
generated as a two-dimensional composition image will be
described with reference to Fig. 1. In Fig. 1, diagrams that
illustrate (1) Imaging Process, (2) Captured Image, and (3)
Two-dimensional Composition Image (2D panoramic image) are
represented.
[0036]
A user sets a camera 10 to a panorama photographing mode,
holds the camera 10 in his hand, and, as illustrated in Fig.
1(1), moves the camera from the left side (point A) to the right
side (point B) with the shutter being pressed. When the user' s
17
pressing of the shutter under the setting of the panorama
photographing mode is detected, the camera 10 performs
consecutive image capturing operations. For example, about
10 to 100 images are consecutively captured.
[0037]
Such images are images 20 that are illustrated in Fig.
1(2). The plurality of images 20 are images that are
consecutively captured while the camera 10 is moved and are
images from mutually different viewpoints. For example, 100
images 20 captured from mutually different viewpoints are
sequentially recorded in a memory. A data processing unit of
the camera 10 reads out a plurality of images 20 that are
illustrated in Fig. 1(2) from the memory, cuts out stripped
areas that are used for generating a panoramic image from the
images, and performs the process of connecting the cut-out
stripped areas, thereby generating a 2D panoramic image 30 that
is illustrated in Fig. 1(3).
[0038]
The 2D panoramic image 30 illustrated in Fig. 1(3) is
a two-dimensional (2D) image and is an image that is
horizontally long by cutting out parts of captured images and
connecting the parts. Dotted lines represented in Fig. 1(3)
illustrates a connection portions of the images. The cut-out
area of each image 20 will be referred to as a stripped area.
[0039]
18
The image processing apparatus or the imaging apparatus
according to the present invention performs an image capturing
process as illustrated in Fig. 1, in other words, as illustrated
in Fig. 1(1), generates a left-eye image (L image) and a
right-eye image (R image) used for displaying a
three-dimensional (3D) image using a plurality of images that
are consecutively captured while the camera is moved.
[0040]
A basic configuration for the process of generating the
left-eye image (L image) and the right-eye image (R image) will
be described with reference to Fig. 2.
Fig. 2(a) illustrates one image 20 that is captured in
a panorama photographing process illustrated in Fig. 1(2).
[0041]
The left-eye image (L image) and the right-eye image (R
image) that are used for displaying a three-dimensional (3D)
image, as in the process of generating a 2D panoramic image
described with reference to Fig. 1, are generated by cutting
out predetermined striped areas from the image 20 and
connecting the stripped areas.
However, the stripped areas that are set as cut-out areas
are located at different positions for the left-eye image (L
image) and the right-eye image (R image).
[0042]
As illustrated in Fig. 2(a), there is a difference in
19
the cut-out positions of a left-eye image strip (L image strip)
51 and a right-eye image strip (R image strip) 52. Although
only one image 20 is illustrated in Fig. 2, for each of a
plurality of images captured with the camera moving, which are
illustrated in Fig. 1(2), left-eye image strips (L image
strips) and right-eye strips (R image strips) that are located
at different cut-out positions are set.
[0043]
Thereafter, by collecting and connecting only the
left-eye image strips (L image strips) , a 3D left-eye panoramic
image (3D panoramic L image) illustrated in Fig. 2(bl) can be
generated.
In addition, by collecting and connecting only the
right-eye image strips (R image strips), a 3D right-eye
panoramic image (3D panoramic R image) illustrated in Fig.
2(b2) can be generated.
[0044]
As above, by connecting the strips, of which the cut-out
positions are differently set, that are acquired from a
plurality of images captured with the camera moving, it is
possible to generate a left-eye image (L image) and a right-eye
image (R image) that are used for displaying a
three-dimensional (3D) image. This principle will be
described with reference to Fig. 3.
[0045]
20
Fig. 3 illustrates a situation in which a subject 80 is
photographed at two capturing positions (a) and (b) by moving
the camera 10. At position (a), as the image of the subject
80, an image seen from the left side is recorded in the left-eye
image strip. (L image strip) 51 of an imaging device 70" of the
camera 10. Next, as the image of the subject 80 at position
(b) to which the camera 10 is moved, an image seen from the
right side is recorded in the right-eye image strip (R image
strip) 52 of the imaging device.70 of the camera 10.
[0046]
As above, images of the same subject seen from mutually
different viewpoints are recorded in predetermined areas
(strip areas) of the imaging device 70.
By individually extracting these, in other words, by
collecting and connecting only the left-eye image strips (L
image strips), a 3D left-eye panoramic image (3D panoramic L
image) illustrated in Fig. 2(bl) is generated, and, by
collecting and connecting only the right-eye image strips (R
image strips), a 3D right-eye panoramic image (3D panoramic
R image) illustrated in Fig. 2(b2) is generated.
[0047]
In Fig. 3, in order to easy understanding, a movement
setting is represented in which the camera 10 crosses the
subject from the left side of the subject 80 to the right side,-
the movement of the camera 10 crossing the subject 80 is not
21
essential. As long as images seen from mutually different
viewpoints can be recorded in predetermined areas of the
imaging device 70 of the camera 10, a left-eye image and a
right-eye image that are used for displaying a 3D image can
be generated.
[0048]
Next, a reverse model using a virtual imaging surface
used in the description presented below will be described with
reference to Fig. 4. In Fig. 4, drawings of (a) image capturing
configuration, (b) forward model, and (c) reverse model are
represented.
[0049]
The image capturing configuration illustrated in Fig.
4 (a) illustrates a process configuration at a time when a
panoramic image, which is similar to that described with
reference to Fig. 3, is captured.
Fig. 4(b) illustrates an example of an image that is
actually captured into the imaging device 70 disposed inside
the camera 10 in the capturing process illustrated in Fig. 4(a).
In the imaging device 70, as illustrated in Fig. 4(b),
a left-eye image 72 and a right-eye image 73 are recorded in
a vertically reversed manner. In a case where description is
made using such a reversed image, in the description presented
below, the description will be made using the reverse model
illustrated in Fig. 4(c).
22
This reverse model is a model that is frequently used
in an explanation of an image in an imaging apparatus or the
like.
[0050]
In the reverse model that is illustrated in Fig. 4(c),
it is assumed that a virtual imaging device 101 is set in front
of the optical center 102 corresponding to the focal point of
the camera, and a subject image is captured into the virtual
imaging device 101. As illustrated in Fig. 4(c), in the virtual
imaging device 101, a subject A91 located on the left side in
front of the camera is captured into the left side, a subject
B92 located on the right side in front of the camera is captured
into the right side, and the images are set not to be verticallyreversed,
whereby the actual positional relation of the
subjects is directly reflected. In other words, an image
formed on the virtual imaging device 101 represents the same
image data as that of an actually captured image.
[0051],
In the description presented below, the reverse model
using this virtual imaging device 101 will be used.
As illustrated in Fig. 4(c), on the virtual imaging
device 101, a left-eye image (L image) 111 is captured into
the right side on the virtual imaging device 101, and a
right-eye image (R image) 112 is captured into the left side
on the virtual imaging device 101.
23
[0052]
2. Problem in Generating 3D Image or 2D Panoramic Image Using
Stripped Areas of Plurality of Images Captured While Camera
Is Moved
Next, problems in generating a 3D image or a 20 panoramic
image using stripped areas of plurality of images captured
while a camera is moved will be described.
[0053]
As a model for the process of capturing a panoramic image
(2D/3D panoramic image) , a capturing model that is illustrated
in Fig. 5 will be assumed. As illustrated in Fig. 5, the cameras
100 are placed such that the optical centers 102 of the cameras
100 are set to positions separated away from the rotation axis
P, which is the rotation center, by a distance R (turning
radius).
A virtual imaging surface 101 is set to the outer side
of the rotation axis P from the optical center 102 by a focal
distance f.
In such settings, the cameras 100 are rotated around the
rotation axis P in a clockwise direction (the direction from
A to B), and a plurality of images are consecutively captured.
[0054]
In each capturing point, other than the strips used for
generating a 2D panoramic image, images of a left-eye image
strip 111 and a right-eye image strip 112 are recorded on the
24
virtual imaging device 101.
For example, the recorded image has a configuration as
illustrated in Fig. 6.
Fig. 6 illustrates an image 110 that is captured by the
camera 100. In addition, this image 110 is the same as the
image formed on the virtual imaging surface 101.
In the image 110, as illustrated in Fig. 6, an area
(stripped area) that is offset to the left side from the center
portion of the image and is cut out in a strip shape is set
as the right-eye image strip 112, and an area (stripped area)
that is offset to the right side from the center portion of
the image and is cut out in a strip shape is set as the left-eye
image strip 111.
[0055]
In addition, in Fig. 6, a 2D panoramic image strip 115
that is used when a two-dimensional (2D) panoramic image is
generated is illustrated.
As illustrated in Fig. 6, a distance between the 2D
panoramic image strip 115 used for. a two-dimensional
composition image and the left-eye image strip 111 and a
distance between the 2D panoramic image strip 115 and the
right-eye image strip 112 are defined as "offsets" or "strip
offsets" = dl and d2.
In addition, a distance between the left-eye image strip
111 and the right-eye image strip 112 is defined as "inter-strip
25
offset" = D.
Furthermore, the inter-strip offset = (strip offset) x
2, and D = dl+d2.
[0056]
A strip width w is a width w that is common to all the
2D panoramic image strip 115, the left-eye image strip 111,
and the right-eye image strip 112 . This strip width is changed
in accordance with the moving speed of the camera and the like.
In a case where the moving speed of the camera is high, the
strip width w is widened, and, in a case where the moving speed
of the camera is low, the strip width w is narrowed. This point
will be described further in a later stage.
[0057]
The strip offset or the inter-strip offset may be set
to various values. For example, in a case where the strip
offset is set to large, the disparity between the left-eye image
and the right-eye image is large, and, in a case where the strip
offset is set to be small, the disparity between the left-eye
image and the right-eye image is small.
[0058]
In a case where the strip offset = 0, left-eye image strip
111 = right-eye image strip 112 = 2D panoramic image strip 115.
In such a case, a left-eye composition image (left-eye
panoramic image) that is acquired by composing the left-eye
image strips 111 and a right-eye composition image (right-eye
26
panoramic image) that is acquired by composing the right-eye
image strips 112 are completely the same image, that is, an
image that is the same as the two-dimensional panoramic image
acquired by composing the 2D panoramic image strips 115 and
cannot be used for displaying a three-dimensional image.
In the description presented below, the lengths of the
strip width w, the strip offset, and the inter-strip offset
are described as values that are defined as the numbers of
pixels.
[0059]
The data processing unit disposed inside the camera 100
acquires motion vectors between images that are consecutively
captured while the camera 100 is moved, and while the strip
areas are aligned such that the patterns of the above-described
strip areas are connected together, the data processing unit
sequentially determines strip areas cut out from each image
and connects the strip areas cut out from each image.
[0060]
In other words, a left-eye composition image (left-eye
panoramic image) is generated by selecting only the left-eye
image strips 111 from the images and connecting and composing
the selected left-eye image strips, and a right-eye composition
image (right-eye panoramic image) is generated by selecting
only the right-eye image strips 112 from the images and
connecting and composing the selected right-eye image strips.
27
[0061]
Fig. 7(1) is a diagram that illustrates an example of
a strip area connecting process . It is assumed that a capturing
time interval between images is At, and n+1 images are captured
between a capturing time T = 0 to nAt. Strip areas extracted
from the n+1 images are connected together.
[0052]
However, in a case where a 3D left-eye composition image
(3D panoramic L image) is generated, only the left-eye image
strips (L image strips) 111 are extracted and connected. In
addition, in a case where a 3D right-eye composition image (3D
panoramic R image) is generated, only the right-eye image
strips (R image strips) 112 are extracted and connected.
[0063]
As above, by collecting and connecting only the left-eye
image strips (L image strips) 111, the 3D left-eye composition
image (3D panoramic L image) illustrated in Fig. 7(23) is
generated.
In addition, by collecting and connecting only the
right-eye image strips (R image strips) 112, the 3D right-eye
composition image (3D panoramic R image) illustrated in Fig.
7(2b) is generated.
[0064]
As described with reference to Figs. 6 and 7, by composing
the 2D panoramic image strips 115 set in the image 100, the
28
two-dimensional panoramic image is generated. In addition,
by joining the stripped areas offset to the right side from
the center of the image 100, the 3D left-eye composition image
(3D panoramic L image) illustrated in Fig. 7 (2a) is generated.
In addition, by joining the stripped areas offset to the
left side from the center of the image 100, the 3D right-eye
composition image (3D panoramic R image) illustrated in Fig.
7(2b) is generated.
[0065]
In these two images, as described above with reference
to Fig. 3, while basically the same subject is imaged, the same
subject is imaged from mutually different positions, whereby
disparity occurs. By displaying the two images having
disparity therebetween in a display apparatus that can display
a 3D (stereoscopic) image, the subject as the imaging target
can be displayed in a stereoscopic manner.
[0066]
In addition, as display types of a 3D image, there are
various types.
For example, there are a 3D image displaying type
corresponding to a passive glass type in which images observed
by the left and right eyes are separated from each other by
using polarizing filters or color filters, a 3D image
displaying type corresponding to an active glass type in which
observed images are separated in time alternately for the left
29
and right eyes by alternately opening/closing left and right
liquid crystal shutters, and the like.
The left-eye image and the right-eye image that are
generated by the above-described strip connecting process can
be applied to each one of such types.
[0067]
As described above, by generating the left-eye image and
the right-eye image by cutting out stripped areas from each
one of a plurality of images that are consecutively captured
while a camera is moved, the left-eye image and the right-eye
image can be generated that are observed from
mutually-different viewpoints, that is, from the left-eye
position and the right-eye position.
[0068]
However, although strip areas are cut out from each one
of a plurality of images that are consecutively captured while
a camera is moved, there is a case where such a 3D image or
a 2D panoramic image cannot be generated.
[0069]
More specifically, for example, as illustrated in Fig.
8 (A) , in a case where the camera is moved in an are shape such
that the optical axes do not intersect with each other, strips
used for generating a 3D image or a 2D panoramic image can be
cut out.
However, there is a case where strips used for generating
30
a 3D image or a 2D panoramic image cannot be cut out from images
that are captured in accordance with a movement other than such
a movement.
For example, such a case is a case (bl) illustrated in
Fig. 9 in which a camera makes a translational motion not
accompanying turning or a case (b2) in which a camera is moved
along an arc shape such that the optical axes intersect with
each other in accordance with the movement of the camera.
[0070]
In a case where a user moves the camera through a camera
swinging operation or the like, it is difficult to move the
camera so as to draw an ideal track as illustrated in Fig. 8,
and the movement as illustrated in Fig. 9(bl) or 9(b2) may be
made.
An object of the present invention is to provide an image
processing apparatus, an imaging apparatus, an image
processing method, and a program capable of performing an
optimal image processing process in accordance with a turning
motion or a translational motion of a camera in a case where
an image is captured through such various forms of movement
and warning a user of the situation in a case where a 2D panoramic
image or a 3D image cannot be generated.
Hereinafter, this process will be described in detail.
[0071]
3. Configuration Example of Image Processing Apparatus
31
According to Present Invention
First, a configuration example of an imaging apparatus
as an image processing apparatus according to an embodiment
of the present invention will be described with reference to
Fig. 10.
An imaging apparatus 200 illustrated in Fig. 10
corresponds to the camera 10 that has been described with
reference to Fig. 1 and, for example, has a configuration that
allows a user to consecutively capture a plurality of images
in a panorama photographing mode with the imaging apparatus
held in his hand.
[0072]
Light transmitted from a subject is incident to an
imaging device 202 through a lens system 201. The imaging
device 202, for example, is configured by a CCD (Charge Coupled
Device) or a CMOS (Complementary Metal Oxide Semiconductor)
sensor.
[0073]
The subject image that is incident to the imaging device
202 is converted into an electrical signal by the imaging device
202. In addition, although not illustrated in the figure, the
imaging device 202 includes a predetermined signal processing
circuit, further converts an electrical signal converted by
the signal processing circuit, and supplies the digital image
data to an image signal processing unit 203.
32
[0074]
The image signal processing unit 203 performs image
signal processing such as gamma correction or contour
enhancement correction and displays an image signal as a result
of the signal processing on a display unit 204.
The image signal as the result of the processing
performed by the image signal processing unit 203 is supplied
to units including an image memory (for a composing process)
205 that is an image memory used for a composing process, an
image memory (for detecting the amount of movement) 2 06 that
is used for detecting the amount of movement between images
that are consecutively captured, and a movement amount
calculating unit 207 that calculates the amount of movement
between the images.
[0075]
The movement amount detecting unit 207 acquires an image
of a frame that is one frame before, which is stored in the
image memory (for detecting the amount of movement) 205,
together with the image signal that is supplied from the image
signal processing unit 203 and detects the amount of movement
between the current image and the image of the frame that is
one frame before . The number of pixels moved between the images
is calculated, for example, by performing a process of matching
pixels configuring two images that are consecutively captured,
in other words, a matching process in which captured areas of
33
the same subject are determined. In addition, basically, the
process is performed by assuming that the subject is stopped.
In a case where there is a moving subject, although a motion
vector other than a motion vector of the whole image is detected,
the process is performed while the motion vector corresponding
to the moving subject is not set as a detection target. In
other words, a motion vector (GMV: global motion vector)
corresponding to the movement of the whole image that occurs
in accordance with the movement of the camera is detected.
[0076]
In addition, for example, the amount of movement is
calculated as the number of moved pixels. The amount of
movement of image n is calculated by comparing image n and image
n-1 that precedes image n, and the detected amount of movement
(number of pixels) is stored in the movement amount memory 208
as an amount of movement corresponding to image n.
[0077]
In addition, the image memory (for the composing process)
205 is a memory for the process of composing the images that
have been consecutively captured, in other words, a memory in
which images used for generating a panoramic image are stored.
Although this image memory (for the composing process) 205 may
be configured such that all the images, for example, n+1 images
that are captured in the panorama photographing mode are stored
therein, for example, the image memory 205 may be set such that
34
end portions of an image is clipped out, and only a center area
of the image from which strip areas that are necessary for
generating a panoramic image is selected so as to be stored.
Through such setting, a required memory capacity can be
reduced.
[0078]
Furthermore, in the image memory (for the composing
process) 205, not only captured image data but also capturing
parameters such as a focal distance [f] and the like are
recorded as attribute information of an image in association
with the image. The parameters are supplied to an image
composing unit 220 together with the image data.
[0079]
Each one of the turning momentum detecting unit 211 and
the translational momentum detecting unit 212, for example,
is configured as a sensor that is included in the imaging
apparatus 200 or an image analyzing unit that analyzes a
captured image.
[0080]
In a case where the turning momentum detecting unit 211
is configured as a sensor, it is a posture detecting sensor
that detects the posture of the camera called pitch/roll/yaw
of the camera. The translational momentum detecting unit 212
is a movement detecting sensor that detects a movement of the
camera with respect to a world coordinate system as the movement
35
information of the camera. The detection information detected
by the turning momentum detecting unit 211 and the detection
information detected by the translational momentum detecting
unit 212 are supplied to the image composing unit 220.
[0081]
In addition, the detection information detected by the
turning momentum detecting unit 211 and the detection
information detected by the translational momentum detecting
. unit 212 may be configured to be stored in the image memory
(for the composing process) 205 as the attribute information
of the captured image together with the captured image when
an image is captured, and the detection information may be
configured to be input together with an image as a composition
target to the image composing unit 220 from the image memory
(for the composing process) 205.
[0082]
Furthermore, the turning momentum detecting unit 211 and
the translational momentum detecting unit 212 may be configured
not by sensors but by the image analyzing unit that performs
an image analyzing process. The turning momentum detecting
unit 211 and the translational momentum detecting unit 212
acquires information that is similar to the sensor detection
information by analyzing a captured image and supplies the
acquired information to the image composing unit 220. In such
a case, the turning momentum detecting unit 211 and the
35
translational momentum detecting unit 212 receive image data
from the image memory (for detecting the amount of movement)
206 as an input and perform image analysis. A specific example
of such a process will be described in a later stage.
[0083]
After the capturing process ends, the image composing
unit 220 acquires an image from the image memory (for the
composing process) 205, further acquires the other necessary
information, and performs an image composing process in which
stripped areas are cut out from the image, which is acquired
from the image memory (for the composing process) 205, and
connecting the stripped areas. Through this process, a
left-eye composition image and a right-eye'composition image
are generated.
[0084]
After the end of the capturing process, the image
composing unit 220 receives the amount of movement
corresponding to each image stored in the movement amount
memory 2(38, and the detection information (the information that
is acquired through sensor detection or image analysis)
detected by the turning momentum detecting unit 211 and the
translational momentum detecting unit 212 as inputs together
with a plurality of images (or partial images) that are stored
during the capturing process from the image memory (for the
composing process) 205.
37
[0085]
The image composing unit 220 cuts out strips from a
plurality of images that has been consecutively captured using
the input information and a connection process thereof, thereby
generating a left-eye composition image (left-eye panoramic
image) and a right-eye composition image (right-eye panoramic
image) as a 2D panoramic image or a 3D image. In addition,
the image composing unit 220 performs a compression process
such as JPEG for each image and then stores the compressed image
in a recording unit (recording medium) 221.
[0086]
In addition, the image composing unit 220 receives the
detection information (the information acquired through
sensor detection or image analysis) detected by the turning
momentum detecting unit 211 and the translational momentum
detecting unit 212 as inputs and determines the processing"
aspect.
More specifically, the image composing unit 220 performs
one of the following processes including:
(a) generation of a 3D panoramic image;
(b) generation of a 2D panoramic image; and
(c) no generation of both 3D and 2D panoramic images.
In addition, in the case where (a) generation of a 3D
panoramic image is performed, reversal of LR images (the
left-eye image and the right-eye image) or the like may be
38
performed in accordance with the detection information.
Furthermore, in a case where (c) no generation of both
3D and 2D panoramic images is performed, a warning output
process for a user and the like are performed.
In addition, a specific processing example thereof will
be described in detail in a later stage.
[0087]
The recording unit (recording medium) 221 stores
composition images that are composed by the image composing
unit 220, that is, the left-eye composition image (left-eye
panoramic image) and a right-eye composition image (right-eye
panoramic image).
The recording unit (recording medium) 221 may be any type
of recording medium as long as it is a recording medium on which
a digital signal can be recorded, and, for example, a recording
medium such as a hard disk, a magneto-optical disk, a DVD
(Digital Versatile Disc) , an MD (Mini Disk) , or a semiconductor
memory can be used.
[0088]
In addition, although not illustrated in Fig. 10, other
than the configuration illustrated in Fig. 10, the imaging
apparatus 200 includes an input operation unit that is used
for performing various inputs for setting the shutter and-the
zoom that can be operated by a user, a mode setting process,
and the like, a control unit that controls the process performed
39
by the imaging apparatus 200, and a storage unit (memory) that
stores a processing program and parameters of any other
constituent unit, parameters, and the like.
[0089]
The process of each constituent unit of the imaging
apparatus 200 that is illustrated in Fig. 10 and the
input/output of data are performed under the control of the
control unit disposed inside the imaging apparatus 200. The
control unit reads out a program that is stored in a memory
disposed inside the imaging apparatus 200 in advance and
performs overall control of the processes such as acquisition
of a captured image, data processing, generation of a
composition image, a process of recording the generated
composition image, a display process, and the like that are
performed in the imaging apparatus 200 in accordance with the
program. •
[0090]
4. Sequence of Image Capturing and Image Processing
Next, an example of the sequence of image capturing and
composing process that is performed by the image processing
apparatus according to the present invention will be described
with reference to a flowchart illustrated in Fig. 11.
The process according to the flowchart illustrated in
Fig. 11, for example, is performed under the control of the
control unit disposed inside the imaging apparatus 200 that
40
is illustrated in Fig. 10.
The process of each step of the flowchart that is
illustrated in Fig. 11 will be described.
First, after a hardware diagnosis and the initialization
are performed in accordance with turning the power on, the image
processing apparatus (for example, the imaging apparatus 200)
proceeds to Step SlOl.
[0091]
In Step SlOl, various capturing parameters are
calculated. In this Step SlOl, for example, information
relating to the brightness identified by an exposure system
is acquired, and capturing parameters such as a diaphragm value
and a shutter speed are calculated.
[0092]
Next, the process proceeds to Step S102, and the control
unit determines whether or not a shutter operation is performed
by a user. Here, it is assumed that the 3D image panorama
photographing mode has been set in advance.
In the 3D image panorama photographing mode, a process
is performed in which a plurality of images are consecutively
captured in accordance with user's shutter operations,
left-eye image strips and right-eye image strips are cut out
from the captured images, and a 1-eft-eye composition image
(panoramic image) and a right-eye composition image (panoramic
image) that can be used for displaying a 3D image are generated
41 .
and recorded.
[0093]
In Step S102, in a case where a user's shutter operation
has not been detected by the control unit, the process is
returned to Step SlOl.
On the other hand, in Step S102, in a case where a user' s
shutter operation is detected by the control unit, the process
proceeds to Step S103.
In Step S103, the control unit starts a capturing process
by performing control that is based on the parameters
calculated in Step SlOl. More specifically, for example, the
adjustment of a diaphragm driving unit of the lens system 201
illustrated in Fig. 10 and the like are performed, and image
capturing is started.
[0094]
The image capturing process is performed as a process
in which a plurality of images are consecutively captured.
Electrical signals corresponding to the consecutively
captured images are sequentially read out from the imaging
device 202 illustrated in Fig. 10, the process of gamma
correction, a contour enhancing correction, or the like is
performed by the image signal processing unit 203, and the
results of the process are displayed on the display unit 204
and are sequentially supplied to the memories 205 and 206 and
the movement amount detecting unit 207.
42-
[0095]
Next, the process proceeds to Step S104, and the amount
of movement between images is calculated. This process is the
process of the movement amount detecting unit 207 illustrated
in Fig. 10.
The movement amount detecting unit 207 acquires an image
of a frame that is one frame before, which is stored in the
image memory (for detecting the amount of movement) 206,
together with the image signal that is supplied from the image
signal processing unit 203 and detects the amount of movement
between the current image and the image of the frame that is
one frame before.
[0096]
In addition, as the amount of movement that is calculated
here, as described above, the number of pixels moved between
the images is calculated, for example, by performing a process
of matching pixels configuring two images that are
consecutively captured, in other words, a matching process in
which captured areas of the same subject are determined; In
addition, basically, the process is performed while assuming
that the subject is stopped. In a case where there is a moving
subject, although a motion vector other than a motion vector
of the whole image is detected, the process is performed while
the motion vector corresponding to the moving subject is not
set as a detection target. In other words, a motion vector
43
(GMV: global motion vector) corresponding to the movement of
the whole image that occurs in accordance with the movement
of the camera is detected.
[0097]
In addition, for example, the amount of movement is
calculated as the number of moved pixels. The amount of
movement of image n is calculated by comparing image n and image
n-1 that precedes image n, and the detected amount of movement
(number of pixels) is stored in the movement amount memory 208
as an amount of movement corresponding to image n.
This movement use storing process corresponds to the
storage process of Step S105. In Step S105, the amount of
movement between images that is detected in Step S104 is stored
in the movement amount memory 208 illustrated in Fig. 10 in
association with the ID of each one of the consecutively
captured images.
[0098]
Next, the process proceeds to Step S106, and, the image
that is captured in Step S103 and is processed by the image
signal processing unit 203 is stored in the image memory (for
the composing process) 205 illustrated in Fig. 10. In addition,
as described above, although this image memory (for the
composing process) 205 may be configured such that all the
images, for example, n+1 images that are captured in the
panorama photographing mode (or the 3D image panorama
44
photographing mode) are stored therein, for example, the image
memory 205 may be set such that end portions of an image is
clipped out, and only a center area of the image from which
strip areas that are necessary for generating a panoramic image
(3D panoramic image) is selected so as to be stored. Through
such setting, a required memory capacity can be reduced.
Furthermore, in the image memory (for the composing process)
205, an image may be configured to be stored after a compression
process such as JPEG or the like is performed for the image.
[0099]
Next, the process proceeds to Step S107, and the control
unit determines whether or not the shutter is continued to be
pressed by the user. In other words, the timing of completion
of capturing is determined.
In a case where the shutter is continued to be pressed
by the user, the process is returned to Step S103 so as to
continue the capturing process, and the imaging of the subject
is repeated.
On the other hand, in Step S107, in a case where the
pressing of the shutter is determined to have ended, in order
to proceeds to a capturing ending operation, the process
proceeds to Step S108.
[0100]
When the consecutive image capturing ends in the panorama
photographing mode, in Step S108, the image composing unit 220
45
determines a process to be performed. In other words, the image
composing unit 220 receives the detection information
(information that is acquired through sensor detection or image
analysis) of the turning momentum detecting unit 211 and the
translational momentum detecting unit 212 as inputs and
determines the aspect of the process.
More specifically, the image composing unit 220 performs
one of the following processes including:
(al) generation of a 3D panoramic image;
(a2) generation of a 3D panoramic image (accompanying a
reversal process of LR images) ;
(b) generation of a 2D panoramic image; and
(c) no generation of both 3D and 2D panoramic images.
In addition, as illustrated in (al) and (a2), also in
a case where a 3D panoramic image is generated, there is a case
where the reversal of LR images (the left-eye image and the
right-eye image) in accordance with the detection information.
Furthermore, in a case where both the 3D and 2D panoramic
images are not generated, a case where the process proceeds
to the determined process, or the like, a notification or a
warning is output to the user in each scene.
A specific example of the process of determining the
process to be performed, which is illustrated in Step S108,
will be described with reference to a flowchart illustrated
in Fig. 12.
46
[0101]
In Step S201, the image composing unit 220 receives the
detection information of the turning momentum detecting unit
211 and the translational momentum detecting unit 212 (the
information acquired through sensor detection or image
analysis) as inputs.
[0102]
In addition, the turning momentum detecting unit 211
acquires or calculates the turning momentum 6 of the camera
at time point when an image as an image composing process target
of the image composing unit 220 is captured and outputs the
value to the image composing unit 220. Here, the detection
information of the turning momentum detecting unit 211 may be
set so as to be directly output from the turning momentum
detecting unit 211 to the image composing unit 220, or it may
be configured such that the detection information is recorded
in the memory as the attribute information of the image together
with the image, and the image composing unit 220 acquires the
value that is recorded in the memory.
[0103]
Furthermore, the translational momentum detecting unit
212 acquires or calculates the translational momentum t of the
camera at time point when an image as an image composing process
target of the image composing unit 220 is captured and outputs
the value to the image composing unit 220. Here, the detection
• 47
information of the translational momentum detecting unit 212
may be set so as to be directly output from the translational
momentum detecting unit 212 to the image composing unit 220,
or it may be configured such that the detection information
is recorded in the memory as the attribute information of the
image together with the image, and the image composing unit
220 acquires the value that is recorded in the memory.
[0104]
In addition, the turning momentum detecting unit 211 and
the translational momentum detecting unit 212, for example,
are configured by a sensor or an image analyzing unit. A
specific configuration example and a processing example will
be described in a later stage.
[0105]
The image composing unit 220, first, in Step S202,
determines whether or not the turning momentum 6 of the camera
at the time of capturing an image, which is acquired by the
turning momentum detecting unit 211, is equal to zero. In
addition, a process may be configured to be performed in which
zero is determined in consideration of a measurement error or
the like in a case where a detected value is not completely
equal to zero but has a difference from zero in an allowed range
set in advance.
[0106]
In a case where the turning momentum of the camera at
48
the time of capturing an image is determined to be zero 0 =
0 in Step S202, the process proceeds to Step S203, and, in a
case where it is determined that Q T* 0, the process proceeds
to Step S205.
[0107]
In a case where the turning momentum of the camera at
the time of capturing an image is determined to be zero 8 =
0 in Step S202, the process proceeds to Step S203, and a warning
for notifying a user that both the 2D panoramic image and the
3D panoramic image cannot be generated is output.
In addition, the determination information of the image
composing unit 220 is output to the control unit of the
apparatus, and a warning or a notification according to the
determination information is displayed, for example, on the
display unit 204 under the control of the control unit.
Alternatively, a configuration may be employed in which an
alarm is output.
[0108]
A case where the turning momentum of the camera is zero
6 = 0 corresponds to an example described in advance with
reference to Fig. 9(bl). In a case where image capturing
accompanying such a movement, both the 2D panoramic image and
the 3D panoramic image cannot be generated,' and a warning for
notifying a user of this is output.
After this warning is output, the process proceeds to
49
step S204, and the process ends without performing the image
composing process.
[0109]
On the other hand, in a case where the turning momentum
of the camera at the time of capturing an image is determined
not to be zero 9 ?!: 0 in Step S202, the process proceeds to Step
S205, and it is determined whether or not the translational
momentum t of the camera at the time of capturing an image,
which is acquired by the translational momentum detecting unit .
212, is equal to zero. In addition, a process may be configured
to be performed in which zero is determined in consideration
of a measurement error or the like in a case where a detected
value is not completely equal to zero but has a difference from
zero in an allowed range set in advance.
[0110]
In a case where the translational momentum of the camera
at the time of capturing an image is determined to be zero t
= 0 in Step S205, the process proceeds to Step S206, and, in
a case where it is determined that t ?i: 0, the process proceeds
to Step S209.
In a case where the turning momentum of the camera at
the time of capturing an image is determined to be zero t =
0 in Step S205, the process proceeds to Step S206, and a warning
for notifying a user that the generation of the 3D panoramic
image cannot be performed is output.

Claims
[Claim 1] (Amended)
An image processing apparatus comprising:
an image composing unit that generates a composition
image by connecting stripped areas that are *cut out from each
image of a plurality of images captured from mutually different
positions,
wherein the image composing unit determines one
processing aspect based on movement information of an imaging
apparatus at the time of capturing an image from among (a) a
composition image generating process of a left-eye composition
image and a right-eye composition image that are used for
displaying a three-dimensional image, (b) a composition image
generating process of a two-dimensional panoramic image, and
(c) stopping composition image generating and performs the
determined process.
[Claim 2]
The image processing apparatus according to claim 1,
further comprising:
a turning momentum detecting unit that acquires or
calculates turning momentum (9) of the imaging apparatus at
the time of capturing an image; and
a translational momentum detecting unit that acquires
or calculates translational momentum (t) of the imaging
18
apparatus at the time of capturing an image; and
wherein the image composing unit determines the
processing aspect based on the turning momentum (9) detected
by the turning momentum detecting unit and the translational
momentum (t) detected by the translational momentum detecting
unit.
[Claim 3]
The image processing apparatus according to claim 1,
further comprising an output unit that presents a warning or
a notification to a user in accordance with information of the
determination of the image composing unit.
[Claim 4]
The image processing apparatus according to claim 2,
wherein the image composing unit stops the composition image
generating process of the three-dimensional image and the
two-dimensional panoramic image in a case where the turning
momentum (9) detected by the turning momentum detecting unit
is zero.
[Claim 5]
The image processing apparatus according to claim 2.,
wherein the image composing unit performs one of the
composition image generating process of a two-dimensional
panoramic image and the stopping composition image generating
in a case where the turning momentum (9) detected by the turning
momentum detecting unit is not zero, and the translational
1^
momentum (t) detected by the translational momentum detecting
unit is zero.
[Claim 6]
The image processing apparatus according to claim 2,
wherein the image composing unit performs one of the
composition image generating process of a three-dimensional
image and the composition image generating process of a
two-dimensional panoramic image in a case where the turning
momentum (9) detected by the turning momentum detecting unit
is not zero,, and the translational momentum (t) detected by
the translational momentum detecting unit is not zero.
[Claim 7]
The image processing apparatus according to claim 6,
wherein, in a case where case where the turning momentum (0)
detected by the turning momentum detecting unit is not zero,
and the translational momentum (t) detected by the
translational momentum detecting unit is not zero, the image
composing unit performs a process in which LR images of the
3D image, which are to be generated, are reversely set in a
case where 6-t < 0 and in a case where 9-t > 0.
[Claim 8]
The image processing apparatus according to claim 2,
wherein the turning momentum detecting unit is a sensor that
detects the turning momentum of the image processing apparatus .
[Claim 9]
80
The image processing apparatus according to claim 2,
wherein the translational momentum detecting unit is a sensor
that detects the translational momentum of the image processing
apparatus.
[Claim 10] -
The image processing apparatus according to claim 2,
wherein the turning momentum detecting unit is an image
analyzing unit that detects the turning momentum at the time
of capturing an image by analyzing captured images.
[Claim 11]
The image processing apparatus according to claim 2,
wherein the translational momentum detecting unit is an image
analyzing unit that detects the translational momentum at the
time of capturing an image by analyzing captured images.
[Claim 12] •
An imaging apparatus comprising:
an imaging unit; and
an image processing unit that performs the image
processing according to any one of claims 1 to 11.
[Claim 13] (Amended)
An image processing method that is performed in an image
processing apparatus, the image processing method comprising:
generating a composition image by connecting stripped
areas that are cut out from each image of a plurality of images
captured from mutually different positions by using an image
%composing unit,
wherein, in the generating of a composition image, one
processing aspect is determined based on movement information
of an imaging apparatus at the time of capturing an image from
among (a) a composition image generating process of a left-eye
composition image and a right-eye composition image that are
used for displaying a three-dimensional image, (b) a
composition image generating process of a two-dimensional
panoramic image, and (c) stopping composition image generating,
and the determined process is performed.
[Claim 14] (Amended)
A program that causes an image processing apparatus to
perform image processing, the program causing an image
composing unit to perform generating a composition image by
connecting stripped areas that are cut out from each of a
plurality of images captured from mutually different
positions,
wherein, in the generating of a composition image, one
processing aspect is determined based on movement information
of an imaging apparatus at the time of capturing an image from
among (a) a composition image generating process of a left-eye
composition image and a right-eye composition image that are
used for displaying a three-dimensional image, (b) a
composition image generating process of a two-dimensional
panoramic image, and (c) stopping composition image generating,
82.
• and the determined process is performed'.