DESCRIPTION
Title of Invention: IMAGING APPARATUS AND IMAGING METHOD
Technical Field
[0001]
The present invention relates to an imaging apparatus
and an imaging method, and in particular, an imaging
apparatus and an imaging method where disparity information
can be obtained without deteriorating image quality of the
image.
Background Art
[0002]
Heretofore, a system to display stereoscopic image by
imaging a common subject by two video cameras from the right
and left at the same time, and presenting the obtained image
to each of right and left eyes, is known. However, with
such a system, the device is large and not only mobility is
lacking, but also shift of the optical axis of the right and
left cameras easily happens, and obtaining image with
appropriate disparity has been difficult.
[0003]
For example, there are some cases where an interval
between two cameras becomes great, or the right and left
optical axes are shifted during zooming due to individual
thereby to a signal processing unit 41. Here, the basic
image is an image based on which a disparity image will be
generated.
[0040]
The imaging device 40-1 and imaging device 40-2 perform
imaging of the image of the subject (hereinafter, referred
to as disparity detecting image) by converting the incident
light input from the condenser lens 38-1 and condenser lens
38-2 into electrical signals. The imaging device 40-1 and
imaging device 40-2 supply a disparity detection image
obtained by imaging, to the signal processing unit 42-1 and
signal processing unit 42-2.
[0041]
Here, the disparity detection image imaged by the
imaging device 40-1 is an image of the subject viewing the
subject from the left side (for example, left eye). This
disparity detection image (hereinafter, particularly
referred to as disparity detection image for left eye) is
used as disparity information when generating disparity
image for left eye. That is to say, image of the subject
looked from a point of view of the disparity detection image
for left eye is generated as disparity image for left eye.
[0042]
In the same way, the disparity detection image imaged
by the imaging device 40-2 is an image of the subject
viewing the subject from the right side (for example, right
eye), and this disparity detection image (hereinafter,
particularly referred to as disparity detection image for
right eye) is used as disparity information when generating
disparity image for right eye.
[0043]
The right and left disparity detection images thus
imaged in the imaging apparatus 11 and the basic image are
both images with disparity. These basic image and disparity
detection images may be a still image or a moving image.
[0044]
Note that, in the following, in the event that there is
no need to distinguish between the transmissive mirror 35-1
and transmissive mirror 35-2 in particular, these will be
referred to simply as transmissive mirror 35, and in the
event that there is no need to distinguish between the
mirror 36-1 and mirror 36-2 in particular, these will be
referred to simply as mirror 36. Also, in the following, in
the event that there is no need to distinguish between the
condenser lens 38-1 and condenser lens 38-2 in particular,
these will be referred to simply as condenser lens 38, and
in the event that there is no need to distinguish between
the imaging device 40-1 and imaging device 40-2 in
particular, these will be referred to simply as imaging
device 40.
The signal processing unit 41 subjects the basic image
supplied from the imaging device 39 to various signal
processing such as A/D (Analog/Digital) conversion
processing, demosaicing processing, and white balance
adjustment processing, and supplies this to the image memory
43-1 and image memory 43-2.
[0046]
The signal processing unit 42-1 and signal processing
unit 42-2 subjects the disparity detection image supplied
from the imaging device 40-1 and imaging device 40-2 to
various signal processing such as A/D (Analog/Digital)
conversion processing, demosaicing processing, and white
balance adjustment processing, and supplies this to the
image memory 43-1 and image memory 43-2. Note that, in the
following, in the event that there is no need to distinguish
between the signal processing unit 42-1 and signal
processing unit 42-2 in particular, these will be referred
to simply as signal processing unit 42.
100471
The image memory 43-1 and image memory 43-2 temporarily
store the basic image from the signal processing unit 41 and
the disparity detection images from the signal processing
unit 42-1 and signal processing unit 42-2. Also, the image
memory 43-1 and image memory 43-2 temporarily store
disparity image supplied from the disparity image generating
unit 44-1 and disparity image generating unit 44-2, and
supplies the stored disparity image to the encoding unit 45-
1 and encoding unit 45-2.
[0048]
Note that, in the following, in the event that there is
no need to distinguish between the image memory 43-1 and
image memory 43-2 in particular, these will be referred to
simply as image memory 43.
[0049]
The disparity image generating unit 44-1 acquires
disparity detection image and basic image from the image
memory 43-1, generates a disparity image for the left eye
based on the disparity detection image and basic image, and
supplies this to the image memory 43-1. The disparity image
generating unit 44-1 includes a corresponding point
searching unit 51-1, and the corresponding point searching
unit 51-1 searches for pixels in the basic image
corresponding to the pixels regarding each pixel of the
disparity detection image at the time of generating
disparity image. Note that the corresponding image is the
image where the same subject is displayed, and in the
following, the pixels in the basic image corresponding to
the pixels of the disparity detection image will be referred
to as corresponding pixels.
The disparity image generating unit 44-2 acquires the
disparity detection image and basic image from the image
memory 43-2, generates disparity image for right eye based
on the disparity detection image and basic image, and
supplies this to the image memory 43-2. The disparity image
generating unit 44-2 includes corresponding point searching
unit 51-2, and the corresponding point searching unit 51-2
searches for the corresponding pixels in the basic image
corresponding to the pixels regarding each pixel of the
disparity detection image at the time of generating
disparity image.
[0051]
Note that in the event that there is no need to
distinguish between the disparity image generating unit 44-1
and disparity image generating unit 44-2 in particular,
these will be referred to simply as disparity image
generating unit 44, and in the event that there is no need
to distinguish between the corresponding point searching
unit 51-1 and corresponding point searching unit 51-2 in
particular, these will be referred to simply as the
corresponding point searching unit 51.
[0052]
The encoding unit 45-1 and encoding unit 45-2 read out
and encode the disparity image from the image memory 43-1
and image memory 43-2 and outputs. Note that, in the
following, in the event that there is no need to distinguish
between the encoding unit 45-1 and encoding unit 45-2 in
particular, the will be referred to simply as encoding unit
45.
[0053]
We can consider that the imaging apparatus 11, which
generates disparity image as described above, to be made up
of two function blocks, with the imaging unit 81 to image
the disparity detection image and basic image, and the video
generating unit 82 to generate the disparity image from the
imaged disparity detection image and basic image.
LO0541
Here, the imaging unit 81 is made up of the
interchangeable lens 21, lens mount 31, and relay lens unit
33 through imaging device 40, and the video generating unit
82 is made up of the signal processing units 41 through
encoding unit 45.
[0055]
[Description of Disparity Image Generating Processing]
Next, description will be made regarding operation of
the imaging apparatus 11.
[0056]
For example, when a user operates the imaging apparatus
11, and instructs to generate right and left disparity image
so as to perform stereoscopic image of the subject, the
imaging apparatus 11 starts disparity image generating
processing and generates disparity image.
[0057]
In step S11, the transmissive mirror 35 separates a
light flux 10 input from the subject.
[0058]
That is to say, upon the user operating the imaging
apparatus 11 so as to start imaging, the incident light from
the subject is converged by the interchangeable lens 21 and
further made to be parallel light by the relay lens unit 33.
After part of the incident light input from the relay lens
unit 33 is shielded by the diaphragm 34 and adjustment of
amount of light is performed, this is input to the
transmissive mirror 35. The transmissive mirror 35
transmits part of the input incident light, inputs to the
condenser lens 37, and reflects the remaining incident light.
[0059]
The incident light input to the transmissive mirror 35
is reflected by the mirror 36 and further converged at the
condenser lens 38 and input to the imaging device 40. Also,
the incident light which has transmitted the transmissive
mirror 35 is converged at the condenser lens 37 and input to
the imaging device 39.
[0060]
For example, in the event that transmittance of the
transmissive mirror 35 is 50%, half of the incident light
from the subject is input to the imaging device 39 and onefourth
light of the incident light is input to the two
imaging devices 40. Note that the basic image imaged by the
imaging device 39 is the image based on which the final
disparity image will be generated, so the transmittance of
light of the transmissive mirror 35 is preferably as high as
possible.
[0061]
In step S12, the imaging device 39 images the basic
image by converting the incident light input from the
condenser lens 37 into electrical signals. The basic image
obtained in this way is, of which amount of light of the
incident light input in the imaging device 39 is generally
lowered by an amount equivalent to the reflectance of the
transmissive mirror 35, the same as with an image obtained
by imaging the subject with a general imaging apparatus.
Accordingly, if reflectance (transmittance) at the
transmissive mirror 35 is set as appropriate, deterioration
of the image quality does not occur, thereby obtaining a
high-definition basic image.
[0062]
The imaging device 39 supplies the basic image obtained
by imaging to the signal processing unit 41. The signal
processing unit 41 subjects the basic image from the imaging
device 39 to various signal processing such as A/D
conversion processing, and supplies this to the image memory
43 so as to store it.
LO0631
In step S13, the imaging device 40 images the disparity
detection image and supplies the obtained disparity
detection image to the signal processing unit 42, by
converting the incident light input from the condenser lens
38 into electrical signals. The signal processing unit 42
subjects the disparity detection image from the imaging
device 40 to various signal processing such as A/D
conversion processing, and supplies this in the image memory
43 so as to store it.
[0064]
There are cases where the right and left disparity
detection image obtained in this way do not have the same
thickness of the light fluxes input to each of the two
imaging devices 40, so brightness and resolution may not be
equal between the right and left. However, in these
disparity detection image, information relating to the
disparity of the subject from a different point of view is
included in these disparity detection images, so a highdefinition
disparity image where brightness and the like are
equal between right and left can be obtained by using
disparity information obtained from the disparity detection
image and high-definition basic image.
[0065]
In step S14, the corresponding point searching unit 51
searches for the corresponding image on the basic image
stored in the image memory 43, regarding each pixel of the
disparity detection image stored in the image memory 43.
For example, searching for the corresponding pixels is
performed between the disparity detection image and basic
image, by taking a predetermined region made up of one or
multiple pixels as an increment and calculating correlation
and sum of difference absolute value.
[0066]
Specifically, in the event that searching for the
corresponding pixels is performed using the sum of absolute
differences, the corresponding point searching unit 51 takes
a pixel on the disparity detection image as the pixel of
interest and also takes the predetermined region where the
pixel of interest on the disparity detection image is the
center, as a region of interest. Also, the corresponding
point searching unit 51 finds, by taking a predetermined
region on the basic image with the same magnitude as with
the region of interest as a corresponding region of interest,
the absolute difference between the pixel value of the same
positional pixel of the region of interest and corresponding
region of interest, and calculates the sum of the absolute
differences of each pixel (sum of absolute differences).
LO0671
The corresponding point searching unit 51 searches for
the corresponding region of interest where the sum of
absolute differences is smallest, by moving the position of
the corresponding region of interest on the basic image and
calculating the above-described sum of absolute differences.
The corresponding point searching unit 51 then takes the
pixel which is positioned in the center of the corresponding
region of interest where the sum of absolute differences is
smallest, i.e., the pixel which is the same position as with
the pixel of interest, as a corresponding pixel as to the
pixel of interest. In this way, the corresponding point
searching unit 51 searches for the corresponding pixel on
the basic image as to the pixel thereof, for each pixel on
the disparity detection image.
[0068]
The corresponding region of interest where the sum of
absolute differences is smallest is the most similar region
in the basic image as to the region of interest on the
disparity detection image, so we can say that the a
corresponding pixel is a region where the same subject as
with the pixel of interest is displayed.
[0069]
Also, for example, in the event that searching for the
corresponding pixels is performed using correlation, the
corresponding point searching unit 51 takes a pixel on the
disparity detection image as a pixel of interest and also
takes the predetermined region where the pixel of interest
on the disparity detection image is the center, as a region
of interest. Also, the corresponding point searching unit
51 calculates the correlation value of the region of
interest and corresponding region of interest with the
predetermined region on the basic image of the same
magnitude as with the region of interest as the
corresponding region of interest.
[0070]
For example, if we say that region of interest and
corresponding region of interest are regions made up of N
pixels, and the pixel value of the ilth pixel (where 1 < i 5
N) in the region of interest is xi and the pixel value of
the pixel of the corresponding region of interest which is
in the same position as with this ilth pixel is yi, the
correlation value C is obtained by the following Expression
(1).
[00711
[math. 11
i =l
Correlation value C =
[0072]
Note that, in Expression (I), X represents an average
value of the pixel values in the pixel within the region of
interest, and Y represents an average value of the pixel
values in the pixel within the corresponding region of
interest. The greater the correlation value C found in this
way is, the higher the correlation between the region of
interest and corresponding region of interest is.
[0073]
The corresponding point searching unit 51 searches for
the corresponding region of interest where the correlation
value C is maximum, by calculating the correlation value C
moving the position of the corresponding region of interest
on the basic image. The corresponding point searching unit
51 then takes the pixel which is positioned in the center of
the corresponding region of interest where the correlation
value C is maximum, i.e., the pixel which is the same
position as with the pixel of interest as a corresponding
pixel as to the pixel of interest. In this way, if the
correlation value is used, even in the event that there is
luminance difference between the disparity detection image
and basic image, corresponding pixels can be detected with
high precision.
[0074]
In step S15, the disparity image generating unit 44
generates a disparity image based on the search result of
the corresponding pixels, and supplies the obtained
disparity image to the image memory 43 to store. The
disparity image generating unit 44 generates the disparity
image by moving the corresponding pixels on the basic image
to the position of each pixel of the disparity detection
image, and performing interpolation processing using the
corresponding pixels.
[ 0 0 7 5 1
Specifically, for example, the disparity image
generating unit 44 generates the disparity image by taking a
pixel on the disparity image which is to be generated as a
pixel of interest, and taking a pixel value of the
corresponding pixel as to the pixel of the disparity
detection pixel which is in the same position as the pixel
of interest, as the pixel value of the pixel of interest.
In this case, an image where each pixel on the disparity
detection image has been replaced with the corresponding
pixel on the basic image corresponding to the pixels thereof
and obtained, is taken as the disparity image.
[0076]
Generating right and left disparity images using one
high-definition basic image, the basic image itself is
reflected in the disparity image with regard to image
quality such as brightness and resolution, so as a result, a
disparity image where the image quality such as brightness
and resolution is equal between the right and left can be
obtained.
[0077]
The disparity image obtained in this way is stored in
the image memory 43, encoded by the encoding unit 45 and
output downstream. The disparity image output downstream is
displayed on the display unit or stored in the storing unit.
When the disparity image is output from the imaging
apparatus 11, the disparity image generating processing is
ended.
LO0781
As described above, at the transmissive mirror 35,
disparity information can be obtained without deteriorating
image quality of the image of the subject, by transmitting
incident light with a predetermined transmittance and
imaging a basic image and at the same time, splitting the
remaining incident light into right and left and imaging
right and left disparity detection images.
[0079]
In particular, transmittance of the transmissive mirror
35 being set as appropriate allows appropriate allocation of
amount of light to the imaging sensor 39 and imaging sensor
40, i.e., appropriate allocation of the image quality to be
performed. Using the basic image and disparity detection
image obtained by imaging then enables obtaining of a
disparity image with an appropriate magnitude of disparity
and a disparity image in which the image quality is equal
between the right and left, while maintaining the image
quality of the basic image, thereby displaying a highdefinition
stereoscopic image.
[0080]
[Division of Entrance Pupil]
Next, description will be made regarding disparity of
the right and left disparity image. Fig. 3 is a conceptual
diagram of an entrance pupil PU in the imaging apparatus 11.
[0081]
A pupil (pupil) is the image of an aperture stop when
the lens has seen from the subject side or image formation
side. With the imaging apparatus 11, the entrance pupil PU
is the image of the diaphragm 34. Here, when the radius of
the circle equivalent to the entrance pupil PU is r, the
following Expression (2) holds.
[0082]
2r = f/F ...( 2)
[0083]
Note that in Expression ( 2 ) , f is the focal distance of
the relay lens unit 33, and F is an F-number of the relay
lens unit 33. Accordingly, in the event that the focal
distance is fixed, it can be seen that the diameter 2r of
the entrance pupil PU is inversely proportionate to the Fnumber.
[0084]
With the imaging apparatus 11, the converged incident
light is split to the right and left at the position of the
diaphragm 34, so we will consider the left half-circle and
right half-circle where the circle of the entrance pupil PU
has split to the right and left in Fig. 3. Threedimensional
effect is obtained based on the disparity
between the user's eyes (relative disparity), however, with
the entrance pupil PU, it can be thought that optical axes
determining disparity pass through each center of gravity of
the left half-circle and right half-circle.
COO851
The centers of gravity of the half-circles of the
radius r can be found geometrically and are placed at the
distance of 4r / 3n from the center of the circle.
Accordingly, the distance between the center of gravity GL
of the left half-circle and the center of gravity GR of the
right half-circle (distance D between centers of gravity) is
found by the following Expression (3).
[0086]
D = 8r/3n ...( 3)
[0087]
From this Expression (3), it can be seen that the
distance D between centers of gravity, i.e., the distance
between the centers of gravity of the light fluxes of the
incident light split to the right and left at the
transmissive mirror 35 is, when stopping the diaphragm 34
down, smaller in proportion thereto. In other words,
adjusting the obtained three-dimensional effect can be
realized by changing the aperture of the diaphragm 34. The
results of an experiment performed to confirm this
assumption will be described in the following.
[0088]
[Relationship between Distance Between Centers of Gravity
and Base-Line Length (Base Line)]
Fig. 4 is a diagram illustrating a relationship between
the distance D between centers of gravity and base-line
length (base line). Here, theoretical values of the
distance between centers of gravity and experimental values
of the base-line length are indicated regarding two types of
lenses, #A and #B, as the interchangeable lens 21.
[0089]
The lens #A is a zoom lens of which the smallest F-stop
is 1.8 and the focal distance is 10 to 100 mm. The zoom
ratio of this lens #A is 10 times and the focal distance of
the wide end (end of the wide-angle side) is 10 mm. The
lens #B is a zoom lens of which the smallest F-stop is 2.8
and the focal distance is 13.5 to 570 mm. The zoom ratio of
this lens #B is 42 times and the focal distance of the wide
end (end of the wide-angle side) is 13.5 mm. The imaging
distance (the distance from the subject to the imaging
sensor) for both is assumed to be 6.5 rn.
[0090]
With the above-described Expression (2) and Expression
(3), the distances D between centers of gravity of the
lenses #A and #B are calculated to be 23.1 mm and 14.9 mm,
respectively. On the other hand, in the measured value, the
base-line length found by the experiment was 20.0 mrn and
12.0 mm regarding lens #A and lens #B, respectively.
[0091]
From this experimental result, it can be seen that
while there is some decrease observed from the theoretical
value which is presumed to be due to the diffraction effect,
the distance D between centers of gravity of the half-circle
of the entrance pupil PU which is the image of the diaphragm
34 approximately matches the base-line length. Also, it can
be seen from Expression (3) that the distance D between
centers of gravity can be changed by the aperture of the
diaphragm 34, and accordingly, base-line length can be also
controlled by the aperture of the diaphragm 34.
[0092]
According to the imaging apparatus 11, the minimum
value of the distance D between centers of gravity is
assumed to be approximately 7 mm. It is thought that a
value of this order as the base-line length can cause
sensing of the three-dimensional effect. In particular, in
the event that shooting distance is long, it is thought that
unless the base-line length has a length of a certain level,
the three-dimensional effect cannot be obtained. When
increasing the base-line length, the three-dimensional
effect is clearer at approximately 32 mm, but on the other
hand the degree of blurriness of the background increases.
When the base-line length reaches a region beyond 65 rnm, it
is thought that a puppet-theater effect will occur,
resulting in unnatural image composition. Accordingly, it
can be thought that a base-line length range yielding a
natural looking stereoscopic image is approximately 7 to 65
rnm .
[0093]
[Relationship between Magnification of Zoom and Disparity]
Fig. 5 is a diagram illustrating a relationship between
magnification of the subject image by the zoom and disparity.
In Fig. 5A, the position for the left eye is L, the position
for the right eye is R, and dots on the subject are A and B.
With angle LAR viewing point A as angle of convergence €IA of
point A, and angle LBR viewing point B as angle of
convergence €IB of point B, disparity d between point A and
point B (relative disparity) is given by the following
Expression (4).
[0094]
d = 0, - €IA ... ( 4 )
[0095]
Here, when angle ALB is h and angle ARB is g, the angle
of convergence €IA is approximately equal to angle h and the
angle of convergence €IB is approximately equal to angle g.
Accordingly, the following Expression (5) is obtained by
Expression (4).
[0096]
d = g - h...(5)
[0097]
Also, assuming that distance between both eyes is D,
the distance from both eyes to a point A is DAI the distance
from the both eyes to a point B is DB, and distance between
point A and point B from both eyes is 6, then g = D / DB, h =
D / DA and DB = DA - 6 hold, so d = D6 / (DA~- 6DA) is
obtained by Expression (5) . Here, DA >> 6 holds, so d = D6 /
D~~ holds.
[0098]
Also, Fig. 5B is a diagram illustrating positional
relationship in the event that magnification of n times has
been performed as compared with Fig. 5A. In Fig. 5B, a dash
is affixed at the end of each symbol regarding the angle,
position, and distance which have changed after zooming.
[0099]
With the example in Fig. 5B, the subject image is
magnified n times, so g' = ng and h1 = nh hold. At this
time, disparity dl is represented as in the following
Expression (6) .
[ OlOO]
dl = OBI - €IA'
= g' - h'
= n(g - h)
= nd ... ( 6 )
[ OlOl]
As can be understood from Expression (6), n-times
disparity occurs by n-time magnification. This means that
the three-dimensional effect of the subject image to be
displayed is increased when being zoomed to the tele end
side (end on the telephoto side). In other words, at the
time of zoom shooting, appropriate disparity can be obtained
even with a short base-line length.
[0102]
In this way, according to the imaging apparatus 11,
disparity of the image presented in front of both eyes can
be suitably reduced, by splitting the light converged by the
interchangeable lens 21 to the right and left using the
transmissive mirror 35 and mirror 36.
[0103]
The disparity obtained with the imaging apparatus 11
can be controlled by the aperture of the diaphragm 34 and
zoom ratio (rate of magnification) at the time of zoom
shooting. That is to say, disparity image with appropriate
disparity can be obtained by controlling the aperture of the
diaphragm 34 and zoom scale by the interchangeable lens 21.
[0104]
Generally, sensitivity of eyes regarding disparity is
high, and while visual acuity is normally measured in order
of tenths of degrees for visual angle, it is said that the
resolving power thereof regarding disparity is one order
higher (see Howard I. P., Rogers B. J.: Stereo Acuity
(Chap.5), Binocular Vision and Stereopsis, P.162, Oxford
University Press, Oxford (1995) . ) . Accordingly, to keep
disparity suitably small is important to perceive the threedimensional
effect naturally and alleviate visual fatigue,
even if under the condition where the disparity is within
that of the above-described example.
[0105]
[Another Configuration Example of Imaging Apparatus]
Note that, with the above, description has made that a
disparity image is generated by the disparity detection
image and basic image inside the imaging apparatus 11, but
this may be also generated outside the imaging apparatus 11.
[0106]
In such a case, the imaging apparatus 11 is configured
as illustrated in Fig. 6, for example. Note that in Fig. 6,
the portions corresponding to the case in Fig. 1 are denoted
with the same reference numerals and the description thereof
will be omitted as appropriate.
[0107]
In Fig. 6, the imaging apparatus 11 is different from
the imaging apparatus 11 in Fig. 1 in that the signal
processing unit 41 through encoding unit 45 are not provided
and the other configurations are the same. That is to say,
the imaging apparatus 11 in Fig. 6 is configured only of the
imaging unit 81, and images the disparity detection image
and basic image by receiving incident light from the subject,
and outputs the obtained disparity detection image and basic
image.
[0108]
The disparity detection image and basic image output
from the imaging apparatus 11 is input to the downstream
video generating unit 82 and the video generating unit 82
generates and outputs the disparity image based on these
image.
[0109]

[Configuration Example of Video Storage/Playing System]
Next, description will be made regarding another
embodiment to which the present invention has been applied.
Fig. 7 is a diagram illustrating a configuration example of
a video storage/playing system to which the present
invention has been applied. This video storage/playing
system is configured of an of imaging unit 81, a video
generating unit 82, a video storage unit 83, a video playing
unit 84, and a display unit 85. Note that, in Fig. 7, the
portions corresponding to the case in Fig. 1 are denoted
with the same reference numerals and the description thereof
will be omitted as appropriate.
[01101
The imaging unit 81 images basic image and right and
left disparity detection images and supplies these to the
video generating unit 82. The video generating unit 82
generates disparity images using the basic image and
disparity detection images supplied from the imaging unit 81,
and stores these in the video storage unit 83.
[Olll]
The video storage unit 83 temporarily stores the
disparity image supplied from the video generating unit 82.
The video playing unit 84 reads out the disparity image from
video storage unit 83 and supplies this to the display unit
85 so as to display the stereoscopic image on the display
unit 85.
[0112]
The video playing unit 84 is configured of a decoding
unit 151-1, a decoding unit 151-2, a display control unit
152-1, and a display control unit 152-2. The decoding unit
151-1 and decoding unit 151-2 read out and decode each of
the disparity image for the left eye and disparity image for
the right eye from the video storage unit 83 and supplies
these to the display control unit 152-1 and display control
unit 152-2.
[0113]
The display control unit 152-1 and display control unit
152-2 supplies the disparity images supplied from the
decoding unit 151-1 and decoding unit 151-2 to the display
unit 85 so as to be displayed. Note that, in the following,
in the event that there is no need to distinguish between
the decoding unit 151-1 and decoding unit 151-2 in
particular, these will be referred to simply as decoding
unit 151, and in the event that here is no need to
distinguish between the display control unit 152-1 and
display control unit 152-2 in particular, these will be
referred to simply as display control unit 152.
[0114]
The display unit 85 performs stereoscopic display on
the disparity image supplied from the display control unit
152 with a predetermined display format. For example, an
arrangement may be conceived for the display unit 85 wherein
circularly polarized light or linearly polarized light
filters are assembled into two projectors, so as to present
each of right and left disparity images, and a user wears
circularly polarized light or linearly polarized light
glasses corresponding to the display, and views the
disparity image.
[0115]
Also, the display unit 85 may be configured of a flat
panel display with a filter, and use a display system with
no glasses for viewing, such as a lenticular system or a
parallax barrier system, whereby right and left disparity
images may be presented to the flat panel display at the
same time. In particular, with the display system in the
display unit 85, employing a display format where the right
and left disparity images are not displayed alternately
these disparity images are displayed at the same time,
enables visual fatigue of a user to be alleviated.
[0116]
[Description of Disparity Image Generating Processing]
Next, description will be made regarding operation of a
video storage/playing system.
[0117]
For example, when a user instructs the video
storage/playing system to generate right and left disparity
images so as to perform stereoscopic display of the subject,
the video storage/playing system starts disparity image
generating processing and generates disparity images. In
the following, description will be made regarding disparity
image generating processing performed by the video
storage/playing system, with reference to the flowchart in
Fig. 8.
[0118]
Note that processing in step S41 through step S45 is
the same as the processing in step S11 through step 515 in
Fig. 2, so the description thereof will be omitted as
appropriate. With these processes, when disparity images
are generated, the generated disparity images are supplied
to the image memory 43 and stored.
[0119]
In step S46, then, the encoding unit 45 in the video
generating unit 82 reads out and encodes disparity images
from the image memory 43, and supplies to the video storage
unit 83. In step S47, the video storage unit 83 stores the
disparity images supplied from the encoding unit 45 and the
disparity image generating processing is ended.
[0120]
As described above, at the transmissive mirror 35, the
incident light is transmitted by a predetermined
transmittance and the basic image is imaged, and at the same
time the other incident light is split to the right and left
and the right and left disparity detection images are imaged,
thereby obtaining disparity information without
deteriorating image quality of the subject image.
[0121]
[Description of Disparity Image Playing Processing]
Next, with reference to the flowchart in Fig. 9,
description will be made regarding disparity image playing
processing to play disparity image, based on the disparity
image stored in the video storage unit 83 by disparity image
generating processing. This disparity image playing
processing is started when playing of the disparity image is
instructed by the user.
[01221
In step S71, the decoding unit 151 reads out the
instructed disparity image to be played from the video
storage unit 83. In step S72, the decoding unit 151 decodes
the read out disparity image and supplies this to the
display control unit 152.
[01231
In step S73, the display control unit 152 supplies the
disparity image supplied from the decoding unit 151 to the
display unit 85 so as to perform stereoscopic display and
the disparity image playing processing is ended. Thus,
stereoscopic display of a subject in the disparity image is
performed at the display unit 85.
Incidentally, with the video storage/playing system in
Fig. 7, motion blurring (Blur) and jerkiness (Jerkiness) can
be reduced by speeding up the frame rate from generating
disparity image until displaying disparity image.
[0125]
Motion blurring often happens due to slipping of video
on the retina at the time of following and watching a moving
subject (eye pursuit) with a hold-type display in particular,
in addition to a decline of MTF (Modulation Transfer
Function) when imaging. Here, a hold-type display is a
display method where video is continuously displayed on a
film, a crystal projector, or the like, during a frame
period.
Also, jerkiness means that smoothness of video is lost
and motions are jerky. This jerkiness often happens at the
time of fixing line of sight to watch video imaged using a
high-speed shutter (fixation viewing). The frame rate of
imaging and displaying, the aperture ratio of imaging of the
camera (open time / frame time), visual perception
properties, and the like, are involved in such deterioration
of the moving image quality.
[0127]
Frame rates of 24 frames per second (24 Hz) for movies,
and 60 fields per second (60 Hz) for television broadcasting,
are normally used. With the video storage/playing system,
taking into consideration blurring due to motion and
jerkiness, imaging of the disparity detection images and
basic image is performed at a rate more than 60 frames per
second (60 Hz), preferably a rate of 230 to 250 frames per
second (240 Hz f 10 Hz). Thus, insufficient resolving power
in the temporal direction can be solved and accurate outline
information can be also obtained as to the moving subject.
[0128]
Further, when taking into consideration of broadcasting
systems, a rate of 290 to 310 frames per second (300 Hz f 10
Hz) is important as a common multiple of 50 frames (50 Hz)
per second often used in Europe and 60 frames per second (60
Hz) often used in Japan and the United States of America in
that a rate of per-second frames (300 Hz f 10 Hz)
facilitates image processing such as image compositing, rate
conversion processing, and so forth.
[0129]
Further, a rate of 590 to 610 frames per second (600 Hz
f 10 Hz) is also important as a common multiple which
further includes 24 frames per second (24 Hz) of movies in
this rate of 290 to 310 frames per second (300 Hz + 10 Hz)
in that image processing is facilitated. Accordingly,
imaging of the disparity detection images and basic image
may be performed with a rate of 290 to 310 frames per second
and a rate of 590 to 610 frames per second.
[0130]
In this way, according to the video storage/playing
system, setting disparity of the image presented to both
eyes to an appropriate magnitude, and further speeding up
the frame rate when imaging image, enables imaging of highquality
stereoscopic images similar as what people see the
natural world, readily viewed and with little aberration.
Also, the optical axis is not actually changed by the zoom
ratio of the interchangeable lens 21 and there is no need to
have image processing such as moving the position of the
image depending on the zoom ratio, so high-precision zoom
effect can be easily realized.
[0131]
[Another Configuration Example of Video Storage/Playing
S ys tem]
Further, in Fig. 7, as an example of the video
storage/playing system, description has made regarding a
case where disparity image is generated from the disparity
detection images and basic image, where various signal
processing has been performed, however, disparity image may
be generated from these images in the state where the
disparity detection images and basic image are RAW images.
[0132]
In such a case, the video storage/playing system is
configured as shown in Fig. 10, for example. Note that, in
Fig. 10, the portions corresponding to the case in Fig. 1 or
Fig. 7 are denoted with the same reference numerals and the
description thereof will be omitted as appropriate.
[0133]
The video storage/playing system in Fig. 10 is
different from the video storage/playing system in Fig. 7 in
that a video generating unit 181 is provided instead of the
video generating unit 82 of the video storage/playing system
in Fig. 7, and the other configurations are the same.
[0134]
With the video storage/playing system in Fig. 10,
disparity detection image and basic image are supplied from
the imaging unit 81 to the video generating unit 181. That
is to say, the basic image output from the imaging device 39
of the imaging unit 81 is supplied to the signal processing
unit 42-1 and signal processing unit 42-2 of the video
generating unit 181. Also, the disparity detection images
output from the imaging device 40-1 and imaging device 40-2
of the imaging unit 81 are supplied to the signal processing
unit 42-1 and signal processing unit 42-2 of the video
generating unit 181.
[0135]
The signal processing unit 42-1 subjects the disparity
detection image and basic image supplied from the imaging
unit 81 to A/D conversion processing and supplies to the
disparity image generating unit 44-1. The disparity image
generating unit 44-1 generates disparity image based on the
disparity detection image and basic image supplied from the
signal processing unit 42-1 and supplies these to the signal
processing unit 42-1. At this time, the corresponding point
searching unit 51-1 provided to the disparity image
generating unit 44-1 searches for the corresponding pixels
from the basic image, regarding each pixel of the disparity
detection image.
[0136]
Also, the signal processing unit 42-1 subjects the
disparity image supplied from the disparity image generating
unit 44-1 to various signal processing such as demosaicing
processing, and white balance adjustment processing and
supplies this to the image memory 43-1 so as to be stored.
The disparity image supplied to the image memory 43-1 is
reads out by the encoding 45-1 and encoded, and supplied to
the video storage unit 83.
In the same way, the signal processing unit 42-2
subjects the disparity detection image and basic image
supplied from the imaging unit 81 to A/D conversion
processing and supplies to the disparity image generating
unit 44-2. The disparity image generating unit 44-2
generates disparity image based on the disparity detection
image and basic image supplied from the signal processing
unit 42-2 and supplies these to the signal processing unit
42-2. At this time, the corresponding point searching unit
51-2 provided to the disparity image generating unit 44-2
searches for the corresponding pixels from the basic image,
regarding each pixel of the disparity detection image.
LO1381
Also, the signal processing unit 42-2 subjects the
disparity image supplied from the disparity image generating
unit 44-2 to various signal processing such as demosaicing
processing and white balance adjustment processing, and
supplies this to the image memory 43-2 so as to be stored.
The disparity image supplied to the image memory 43-2 is
reads out by the encoding 45-2 and encoded, and supplied to
the video storage unit 83.
[01391
In this way, generating a disparity image in a state
where the disparity detection image and basic image are RAW
images enables higher-quality disparity images to be
obtained.
[0140]
Note that the embodiment of the present invention is
not limited to the above-described embodiment, and that
various modifications may be made without departing from the
essence of the present invention.
Reference Signs List
[0141]
11 imaging apparatus
21 interchangeable lens
33 relay lens unit
34 diaphragm
35-1, 35-2, and 35 transmissive mirror
36-1, 36-2, and 36 mirror
37 condenser lens
38-1, 38-2, and 38 condenser lens
39 imaging device
40-1, 40-2, and 40 imaging device
44-1, 44-2, and 44 disparity image generating unit
51-1, 51-2, and 51 corresponding point searching unit

CLAIMS
[Claim 11
An imaging apparatus comprising:
first converging means to converge incident light from
a subject;
transmitting means to transmit the incident light
converged by the first converging means to be parallel
light; splitting means to transmit, at a predetermined
transmittance, the incident light made to be parallel light
by the transmitting means, and also split part of the
incident light made to be the parallel light to the right
and left;
second converging means to converge the incident light
which has transmitted the splitting means;
first imaging means to image a basic image of the
subject, by converting the incident light converged by the
second converging means into electric signals;
third converging means to converge each incident light
split by the splitting means; and
second imaging means to image disparity detection
images having disparity as to each other, by converting each
of the incident light converged by the third converging
means into electrical signals.
[Claim 21
The imaging apparatus according to Claim 1, further
comprising:
disparity image generating means to generate disparity
images so as to perform stereoscopic display of the subject,
by generating images with the same point of view as with the
disparity detection images, regarding each of the disparity
detection images, based on the basic image.
[Claim 31
The imaging apparatus according to Claim 2, wherein the
disparity image generating means generate the disparity
images, by taking a pixel of the disparity image to be
generated as the pixel of interest, and obtaining a pixel
value of the pixel of interest, based on a pixel value of
the pixel in the base image corresponding to the pixel of
the disparity detection image which is at the same position
as the pixel of interest.
[Claim 41
The imaging apparatus according to Claim 2, further
comprising:
amount of light adjusting means which, being arranged
between the transmitting means and splitting means, shield
part of the incident light from the transmitting means, so
as to change the distance between centers of gravity of the
incident light split to the right and left by the splitting
means.
[Claim 51
The imaging apparatus according to Claim 4, wherein the
amount of light adjusting means shield part of the incident
light so that the distance between centers of gravity is 7
to 65 mm.
[Claim 61
The imaging apparatus according to Claim 2, wherein the
first converging means include a zoom lens to magnify an
image of the subject, and wherein relative disparity between
the disparity detection image is controlled by magnification
of the image of the subject by the zoom lens.
[Claim 71
The imaging apparatus according to Claim 2, wherein the
first imaging means and the second imaging means image the
basic image and disparity detection images with a rate of 60
frames per second or more.
[Claim 81
The imaging apparatus according to Claim 7, wherein the
first imaging means and the second imaging means image the
basic image and disparity detection images with a rate of
230 to 250 frames per second.
[Claim 91
The imaging apparatus according to Claim 7, wherein the
first imaging means and the second imaging means image the
basic image and disparity detection images with a rate of
290 to 310 frames per second.
[Claim 101
The imaging apparatus according to Claim 7, wherein the
first imaging means and the second imaging means image the
basic image and disparity detection images with a rate of
590 to 610 frames per second.
[Claim 111
The imaging apparatus according to Claim 2, further
comprising:
storage means to store the disparity images; and
display control means to perform stereoscopic display
of the subject, based on the disparity images stored in the
storage means.
[Claim 121
An imaging method of an imaging apparatus including
first converging means to converge incident light
from a subject,
transmitting means to transmit the incident light
converged by the first converging means to be parallel light,
splitting means to transmit, at a predetermined
transmittance, the incident light made to be parallel light
by the transmitting means and also split part of the
incident light to be the parallel light to the right and
left,
second converging means to converge the incident
light which has transmitted the splitting means,
first imaging means to image a basic image of the
subject, by converting the incident light converged by the
second converging means into electric signals,
third converging means to converge each incident
light split by the splitting means, and
second imaging means to image disparity detection
image having disparity as to each other, by converting eac.h
of the incident light converged by the third converging
means into electrical signals;
the method comprising the steps of:
the first converging means converging the incident
light;
the transmitting means transmitting the incident light
from the first converging means;
the splitting means splitting part of the incident
light while transmitting the incident light from the
transmitting means;
the second converging means converging the incident
light which has transmitted the splitting means;
the first imaging means imaging the basic image;
the third converging means converging the split
incident light; and
the second imaging means imaging the disparity
detection images.