IMAGE PROCESSING APPARATUS AND IMAGE PROCESSING METHOD
BACKGROUND
[OOOl]
5 The present disclosure relates to an image processing apparatus and an
image processing method and more particularly, to an image processing apparatus, an
image processing method, and a program that enable mismatching of front and rear
frames to be inconspicuous, when an interpolation method changes.
[0002]
10 A glasses-free 3D display apparatus that enables a user to perceive a
stereoscopic image without wearing glasses, in three-dimensional (3D) image display
processing, has begun to be put to practical use. The glasses-free 3D display
apparatus includes a lenticular sheet or a parallax barrier provided on a display
surface and controls images input to left and right eyes by a viewing position. That
15 is, the glasses-free 3D display apparatus performs a control operation such that a left
visual point image corresponding to an image observed from the left eye is observed
by the left eye and a right visual point image corresponding to an image observed
from the right eye is observed by the right eye.
[0003]
20 However, according to the above method, a correct stereoscopic vision is
obtained only at limited viewing positions with respect to a display. Therefore,
when an observation position of a user is different from a prescribed position, a
reverse vision in which an image for the right eye (right visual point image) may be
input to the left eye and an image for the left eye (left visual point image) may be
25 input to the right eye or crosstalk in which the left visual point image and the right
visual point image are mixed occurs.
[0004]
Meanwhile, a configuration in which not only a standard left visual point
image and a standard right visual point image corresponding to one regular
30 observation position and images from new visual points set not to cause the crosstalk
when the display is observed at the other observation positions are generated and

detecting parallax from two original 3D images of an input left visual point image (L
image) and an input right visual point image (R image) and determining virtual
visual point positions different from visual point positions of the input LR images on
the basis of a crosstalk amount or a hsional parallax range has been disclosed.
5
SUMMARY
[OO lo]
As described above, the interpolatio'n methods of performing the
interpolation on the basis of the two visual point images of the left visual point image
10 (L image) and the right visual point image (R image) when the visual point images of
the virtual visual points other than the two visual points are generated have been
suggested.
[OO 1 11
However, when only the single interpolation method corresponds to all
15 output phases, distortion or blur may occur in the generated virtual visual point
images.
[OO 121
When the plurality of interpolation methods are used together, if the output
phase may temporally change, the virtual visual point image may temporally change.
20 [0013]
It is desirable to enable mismatching of fiont and rear frames to be
inconspicuous, when an interpolation method changes.
[00 141
According to an embodiment of the present disclosure, there is provided an
25 image processing apparatus including a parallax estimating unit that generates
parallax information from a left visual point image to be an image signal for a left
eye applied to multi-dimensional image display and a right visual point image to be
an image signal for a right eye applied to the multi-dimensional image display, an
interpolation direction control unit that controls changing of an interpolation
30 direction of a virtual visual point image including a visual point image other than the
left visual point image and the right visual point image, according to a parameter
showing a degree of a variation based on the parallax information generated by the
parallax estimating unit, and a virtual visual point image generating unit that
generates the virtual visual point image in the interpolation direction of which the
changing is controlled by the interpolation direction control unit.
5 [0015]
The interpolation direction control unit may prohibit the changing of the
interpolation direction of the virtual visual point image, when the variation shown by
the parameter is large.
[00 161
10 The interpolation direction control unit may perform the changing of the
interpolation direction of the virtual visual point image, when the variation shown by
the parameter is small.
[OO 171
The variation based on the parallax information that is generated by the
15 parallax estimating unit may be a time variation.
[00 181
The image processing apparatus may krther include a reliability calculating
unit that calculates reliability of the parallax information generated by the parallax
estimating unit. The parameter showing the degree of the variation based on the
20 parallax information generated by the parallax estimating unit may be the reliability
of the parallax information calculated by the reliability calculating unit, and the
interpolation direction control unit may control the changing of the interpolation
direction of the virtual visual point image, according to the reliability of the parallax
information calculated by the reliability calculating unit.
25 [0019]
The parameter showing the degree of the variation based on the parallax
information generated by the parallax estimating unit may be a scale value calculated
fiom the parallax information generated by the parallax estimating unit, and the
interpolation direction control unit may control the changing of the interpolation
30 direction of the virtual visual point image, according to the scale value calculated
from the parallax information generated by the parallax estimating unit.
[0020]
The interpolation direction control unit may select one direction as the
interpolation direction of the virtual visual point image, according to the parameter
showing the degree of the variation based on the parallax information generated by
5 the parallax estimating unit, when the selected one direction is selected as the
interpolation direction of the virtual visual point image continuously for a constant
time, the interpolation direction control unit may change the interpolation direction
of the virtual visual point image to the selected one direction, and when the selected
one direction is not selected as the interpolation direction of the virtual visual point
10 image continuously for the constant time, the interpolation direction control unit may
prohibit the changing of the interpolation direction of the virtual visual point image.
[002 11
The virtual visual point image generating unit may set a convergence
position of a visual point position to a left visual point or a right visual point and
15 calculates a virtual visual point position to generate the virtual visual point image,
using the parallax information generated by the parallax estimating unit, and
generates the virtual visual point image in the interpolation direction of which the
changing is controlled by the interpolation direction control unit, at the calculated
virtual visual point position.
20 COO221
The virtual visual point image generating unit may set a convergence
position of a visual point position to any position between a left visual point and a
right visual point and calculates a virtual visual point position to generate the virtual
visual point image, using the parallax information generated by the parallax
25 estimating unit, and generates the virtual visual point image in the interpolation
direction of which the changing is controlled by the interpolation direction control
unit, at the calculated virtual visual point position.
[0023]
The image processing apparatus may hrther include a face detecting unit
30 that detects a position of a face of a user who views the virtual visual point image
which is generated by the virtual visual point image generating unit and is displayed
on a display unit. The interpolation direction control unit may control the changing
of the interpolation direction of the virtual visual point image, according to the
position of the face of the user detected by the face detecting unit.
1 [0024]
I 5 A display unit that displays the virtual visual point image generated by the
virtual visual point image generating unit may be wearable on a head of a user, the
image processing apparatus may further include a face detecting unit that detects a
position and a direction of a face of the user who views the virtual visual point image
displayed on the display unit, and the interpolation direction control unit may control
10 the changing of the interpolation direction of the virtual visual point image,
according to the position and the direction of the face of the user detected by the face
detecting unit.
[0025]
The image processing apparatus may fbrther include a scene change
15 detecting unit that detects a scene change from the left visual point image or the right
visual point image. The interpolation direction control unit may perform the
changing of the interpolation direction of the virtual visual point image, when the
scene change is detected by the scene change detecting unit.
[0026]
20 According to an embodiment of the present disclosure, there is provided an
image processing method including causing an image processing apparatus to
generate parallax information from a left visual point image to be an image sigal for
a left eye applied to multi-dimensional image display and a right visual point image
to be an image signal for a right eye applied to the multi-dimensional image display,
25 causing the image processing apparatus to control changing of an interpolation
direction of a virtual visual point image including a visual point image other than the
left visual point image and the right visual point image, according to a parameter
showing a degree of a variation based on the generated parallax information, and
causing the image processing apparatus to generate the virtual visual point image in
30 the interpolation direction of which the changing is controlled.
[0028]
According to an embodiment of the present disclosure, the parallax
information is generated from the left visual point image to be the image signal for
the left eye applied to the multi-dimensional image display and the right visual point
image to be the image signal for the right eye applied to the multi-dimensional image
5 display and the changing of the interpolation direction of the virtual visual point
image including the visual point image other than the left visual point image and the
right visual point image is controlled according to the parameter showing the degree
of the variation based on the generated parallax information. In addition, the virtual
visual point image is generated in the interpolation direction of which the changing is
10 controlled.
[0029]
According to the embodiments of the present disclosure described above, a
virtual visual point image including a visual point image other than a left visual point
image and a right visual point image can be generated. In particular, when an
15 interpolation direction changes, mismatching of front and rear frames can be
prevented fiom being conspicuous.
BRIEF DESCRIPTION OF THE DRAWINGS
[0030]
20 FIG 1 is a diagram illustrating the related art;
FIG 2 is a bock diagram illustrating a configuration example of an image
processing. apparatus to which the present disclosure is applied;
FIG 3 is a diagram illustrating an example of processing of a parallax
estimating unit;
2 5 FIG 4 is a diagram illustrating an example of processing of a parallax
estimating unit;
FIG 5 is a diagram illustrating generation processing of a basic virtual
visual point image;
FIG 6 is a diagram illustrating generation processing of a basic virtual
30 visual point image;
FIG 7 is a diagram illustrating generation processing of a basic virtual
visual point image;
FIG 8 is a block diagram illustrating a configuration example of a virtual
visual point image generating unit;
FIG 9 is a diagram illustrating visual point position adjustment processing;
5 FIG 10 is a diagram illustrating a setting example of a virtual visual point
image position;
FIG 11 is a diagram illustrating selection processing of an interpolation
direction;
FIG 12 is a block diagram illustrating a configuration example of an image
. .-
10 synthesizing unit;
FIG 13 is a block diagram illustrating a configuration example of a one
visual point image synthesizing unit;
FIG 14 is a flowchart illustrating an example of image processing of an
image processing apparatus;
15 FIG 15 is a flowchart illustrating visual point position adjustment
processing:
FIG 16 is a flowchart illustrating selection processing of an interpolation
direction;
FIG 17 is a block diagram illustrating a configuration example of an image
20 processing apparatus to which the present disclosure is applied;
FIG 18 is a diagram illustrating processing of a reliability calculating unit;
FIG 19 is a diagram illustrating processing of a reliability calculating unit;
FIG 20 is a diagram illustrating processing of a reliability calculating unit;
FIG 21 is a diagram illustrating processing of a reliability calculating unit;
25 FIG 22 is a block diagram illustrating a configuration example of a virtual
visual point image generating unit;
FIG 23 is a diagram illustrating a setting example of a virtual visual point
image position;
FIG 24 is a diagram illustrating selection processing of an interpolation
30 direction;
FIG 25 is a flowchart illustrating an example of image processing of an
image processing apparatus;
FIG 26 is a flowchart illustrating visual point position adjustment
processing;
FIG 27 is a flowchart illustrating selection processing of an interpolation
5 direction;
FIG 28 is a diagram illustrating an image processing apparatus to which the
present disclosure is applied;
FIG 29 is a block diagram illustrating a configuration example of an image
processing apparatus;
10 FIG 30 is a block diagram illustrating a configuration example of a virtual
visual point image generating unit;
FIG 31 is a diagram illustrating a setting example of a virtual visual point
image position;
FIG 32 is a diagram illustrating an image processing apparatus to which the
15 present disclosure is applied;
FIG 33 is a block diagram illustrating a configuration example of an image
processing apparatus;
FIG 34 is a diagram illustrating an operation of a visual point position
measuring unit;
20 FIG 35 is a block diagram illustrating a configuration example of a virtual
visual point image generating unit;
FIG 36 is a flowchart illustrating an example of image processing of an
image processing apparatus;
FIG 37 is a flowchart illustrating visual point position adjustment
25 processing:
FIG 38 is a flowchart illustrating selection processing of an interpolation
direction;
FIG 39 is a diagram illustrating an image processing apparatus to which the
present disclosure is applied;
30 FIG 40 is a block diagram illustrating a configuration example of an image
processing apparatus;
FIG 41 is a block diagram illustrating a configuration example of an image
processing apparatus to which the present disclosure is applied;
FIG 42 is a diagram illustrating processing of a scene change detecting unit;
FIG 43 is a block diagram illustrating a configuration example of a virtual
5 visual point image generating unit that executes analysis processing of a scene;
FIG 44 is a block diagram illustrating a configuration example of a virtual
visual point image generating unit that executes selection processing of an
interpolation direction and image synthesis processing;
FIG 45 is a flowchart illustrating an example of image processing of an
10 image processing apparatus;
FIG 46 is a flowchart illustrating scene analysis processing;
FIG 47 is a flowchart illustrating selection processing of an interpolation
direction;
FIG 48 is a block diagram illustrating a configuration example of an image
15 processing apparatus to which the present disclosure is applied;
FIG 49 is a flowchart illustrating an example of image processing of an
image processing apparatus;
FIG 50 is a flowchart illustrating scene analysis processing;
FIG 51 is a flowchart illustrating selection processing of an interpolation
20 direction; and
FIG 52 is a block diagram illustrating a configuration example of a
computer.
DETAILED DESCRIPTION OF THE EMBODIMENT(S)
25 [OOOO]
Hereinafter, preferred embodiments of the present disclosure will be
described in detail with reference to the appended drawings. Note that, in this
specification and the appended drawings, structural elements that have substantially
the same function and structure are denoted with the same reference numerals, and
30 repeated explanation of these structural elements is omitted.
[003 11
The following description will be made in the order described below.
1. Description of Related Art
2. First Embodiment (Parallax Range)
3. Second Embodiment (Reliability)
5 4. Third Embodiment (Motion Parallax)
5. Fourth Embodiment (Motion Parallax + Face Detection)
6. Fifth Embodiment (Head-Mounted Display)
7. Sixth Embodiment (Off-Line Processing)
8. Seventh Embodiment (Computer)
10 [0032]
4. Description of Related Art>
[Description of Related Art>
First, the related art will be described with reference to FIG 1.
[0033]
15 In the related art, in an apparatus that generates an image displayed on a '
glasses-free 3D display 11, an output image is generated on the basis of a visual point
image A input to a left eye of a user who views the glasses-free 3D display 11 or a
visual point image B input to a right eye.
[0034]
20 Here, the case in which an interpolation direction changes from the visual
point image A input to the left eye to the visual point image B input to the right eye at
times t-1 and t is considered. In the images, thick lines of a vertical direction are
included.
[0035]
25 Specifically, at the time t-1, estimation parallax L-R is calculated from the
visual point image A input to the left eye and an output image of the time t-1 is
generated. At the time t, estimation parallax R-L is calculated fiom the visual point
image B input to the right eye and an output image of the time t is generated.
[0036]
30 As illustrated at an upper right side of FIG 1, if the estimation parallax L-R
and the estimation parallax R-L are matched, the thick line of the output image of the
time t-1 and the thick line of the output image of the time t are almost aligned.
COO3 71
However, as illustrated at a lower right side of FIG 1, at least one of the
estimation parallax L-R and the estimation parallax R-L is wrong, the thick line of
5 the output image of the time t-1 and the thick line of the output image of the time t
may not be aligned.
[003 81
Thereby, at the times tl to t-1, error of a change in which the thick line of
the output image is viewed as if the thick line has jumped in a horizontal direction or
10 a depth direction may be conspicuous.
[0039]
Therefore, in the present disclosure, mismatching of front and rear frames is
prevented from being conspicuous, when the interpolation direction changes.
Hereinafter, the present disclosure will be described in detail.
15 [0040]
<2. First Embodiment (Parallax Range)>
[Configuration Example of Image Processing Apparatus]
FIG 2 is a block diagram illustrating a configuration example of an image
processing apparatus to which the present disclosure is applied.
20 [0041]
In the example of FIG 2, an image processing apparatus 100 includes a left
visual point image (L image) input unit 101, a right visual point image (R image)
input unit 102, a parallax estimating unit 103, a virtual visual point image generating
unit 105, and a display control unit 106. An image that is generated in the image
25 processing apparatus 100 is output to a display unit 11 0.
[0042]
In the configuration illustrated in FIG 2, the display unit 110 is provided
outside the image processing apparatus 100. However, the display unit 1 10 may be
provided inside the image processing apparatus 100.
30 [0043]
FIG 2 illustrates a main configuration of the image processing apparatus.
Therefore, the image processing apparatus 100 includes a control unit having a
program execution function such as a CPU to execute data processing control, a
storage unit to store a program executed in the control unit or various parameters,
and an input unit to input parameters or image data, in addition to the configuration
5 illustrated in FIG 2. For example, the control unit executes processing to be
described below according to the program stored in the storage unit in advance.
100441
The left visual point image (L image) input unit 101 and the right visual
point image (R image) input unit 102 input a left visual point image (L image) and a
10 right visual point image (R image) for three-dimensional (3D) image display that are
generated in advance, respectively. The left visual point image (L image)
corresponds to an image observed fiom a left eye and the right visual point image (R
image) corresponds to an image observed from a right eye.
[0045]
15 However, the two images are two standard LR images. That is, the two
images are LR images that are observed as a correct 3D image when a display is
observed fiom a prescribed position, for example, a center position of the front, in a
glasses-free 3D display apparatus that includes a lenticular sheet or a parallax barrier
provided on a display surface. When an observation position of a user is different
20 from the prescribed position, a reverse vision in which an image for the right eye
(right visual point image) may be input to the left eye and an image for the left eye
(left visual point image) may be input to the right eye or crosstalk in which the left
visual point image and the right visual point image are mixed occurs.
[0046]
25 Meanwhile, the image processing apparatus 100 generates images from new
visual points (virtual visual points) not causing the crosstalk when the display is
observed at various observation positions, on the basis of input LR images
corresponding to one regular observation position, that is, the standard left visual
point image and the standard right visual point image.
30 [0047]
The parallax estimating unit 103 receives the left visual point image (L
image) and the right visual point image (R image) and generates parallax information
on the basis of these images. Hereinafter, the L image and the R image are
collectively called LR images. The parallax information becomes information that
corresponds to a deviation between images (pixel deviation of a horizontal direction)
5 of the same object included in the input LR images and corresponds to a distance of
the object. Specifically, the parallax estimating unit 103 generates data that has
parallax information (object distance information) of each pixel unit or each pixel
region unit.
[0048]
10 The L image from the left visual point image (L image) input unit 10 1, the R
image from the right visual point image (R image) input unit 102, and the parallax
information from the parallax estimating unit 103 are input to the virtual visual point
image generating unit 105.
[0049]
15 The virtual visual point image generating unit 105 receives each information
and generates a virtual visual point image. For example, the virtual visual point
image generating unit 105 adjusts a parallax amount on the basis of a parallax
distribution calculated from the parallax information from the parallax estimating
unit 103, executes determination processing of a virtual visual point position, and
20 generates a virtual visual point image corresponding to the determined virtual visual
point position.
[0050]
The virtual visual point image generating unit 105 executes generation
processing of the virtual visual point image based on the parallax distribution. That
25 is, a total of N visual point images that are obtained by adding the other visual point
images to the two visual point images of the input LR images 'are generated and
output. For example, the virtual visual point image generating unit 105 calculates
output phases corresponding to the N visual points, selects an interpolation direction
according to the parallax distribution, and generates a virtual visual point image of
30 the selected interpolation direction. This processing will be described in detail
below.
LO05 11
The virtual visual point image that is generated by the virtual visual point
image generating unit 105 is output to the display unit 110 through the display
control unit 106 and is displayed.
5 [0052]
The display image that is generated by the image processing apparatus
according to the present disclosure is a display image in the glasses-free 3D display
apparatus in which the user can view a stereoscopic image without wearing the
glasses.
10 [0053]
The display unit 11 0 is a display unit that performs glasses-fiee 3D display.
Specifically, the display unit 110 is a display unit that includes a lenticular sheet or a
parallax barrier provided on a display surface and can control images input to the left
eye and the right eye by the viewing position.
15 [0054]
The display control unit 106 outputs the N visual point images generated by
the virtual visual point image generating unit 105 to the display unit 110. The
display control unit 106 generates display information according to a display
configuration of the display unit 1 10.
20 [0055]
The image processing apparatus 100 can be configured as an imaging
apparatus such as a camera including an imaging unit or a display apparatus such as a
PC or a television. When the image processing apparatus 100 is configured as the
imaging apparatus or the display apparatus, the image processing apparatus 100 has a
25 function according to each apparatus.
[0056]
For example, the camera has an imaging unit that images LR images
corresponding to different visual point images and generates multiple visual point
images using the LR images input from the imaging unit.
30 [0057]
[Processing of Parallax Estimating Unit]
Next, processing of the parallax estimating unit 103 will be described. The
parallax estimating unit 103 receives a left visual point image (L image) and a right
visual point image (R image) and generates parallax information on the basis of these
images. The parallax information becomes information that corresponds to a
5 deviation between images (pixel deviation of a horizontal direction) of the same
object included in the standard LR images and corresponds to a distance of the object.
Specifically, the parallax estimating unit 103 generates data that has parallax
information (object distance information) of each pixel unit.
[0058]
10 The acquisition of the parallax information is executed by the following
existing methods.
(a) Parallax information acquisition processing of a block matching base
(b)' Parallax information acquisition processing of a dynamic programming (DP)
matching base
15 (c) Parallax information acquisition processing of a segmentation base
(d) Parallax information acquisition processing of a learning base
(e) Parallax information acquisition processing of a combination of the methods
described above
For example, the parallax information is acquired by any method of (a) to
20 (e) described above.
[0059]
The parallax information acquisition processing of the block matching base
will be simply described with reference to FIG 3. The parallax estimating unit 103
uses a left visual point image (L image) and a right visual point image (R image) to
25 be input original standard images, selects a pixel region (block) 121 of the L image,
and detects a pixel region (block) 122 similar to the selected block, from the R image.
[0060]
That is, the parallax estimating unit 103 selects blocks (matching blocks)
determined as imaging regions of the same object, from the LR images. The
30 parallax estimating unit 103 measures a position deviation (the number of pixels in a
horizontal direction) of the matching blocks between the LR images.
[0061]
In FIG 3, a pixel in the R image that corresponds to an attention pixel LP =
(5, 3) of the pixel region (block) 121 of the L image is an attention pixel RP = (7, 3)
of the pixel region (block) 122. In this case, parallax d (5, 3) between the LR
5 images at a pixel position (x, y) of the L image = (5, 3) is calculated as represented
by the following expression 1.
[0062]
[Expression 11
Parallax d (5, 3) = (7, 3) - (5,3) = (2, 0)
10 That is, the parallax d of the pixel position (x, y) of the L image = (5, 3)
becomes two pixels.
[0063]
The position deviation of the block changes according to the distance of the
object imaged in the block. That is, the position deviation of the block corresponds
15 to the distance of the object and information of the position deviation is acquired as
the parallax information.
[0064]
As an expression form of the parallax information, there is a depth map
(distance image or parallax map). The depth map (parallax map) is an image in
20 which parallax (object distance) of each pixel unit of the L image and the R image is
expressed by brightness of a pixel unit. For example, a high-brightness region
shows a close object (object close to the camera) and a low-brightness region shows
a remote object (object remote fiom the camera). That is, the depth map is an
image in which the object distance is shown by the brightness.
25 [0065]
As illustrated in FIG 4, the parallax estimating unit 103 acquires not only
LR parallax information to be information of parallax of the R image based on the L
image described above with reference to FIG 3 but also RL parallax information to
be information of parallax of the L image based on the R image. In an example of
30. FIG. 4, the LR parallax information is shown by a solid line and the RL parallax
information is shown by a dotted line or a one-dotted chain line.
[0066]
As shown by the solid line and the dotted line, signs of the LR parallax
information and the RL parallax information are different from each other, but the
LR parallax information and the RL parallax information are basically matched with
5 each other. However, the RL parallax may not be matched with the LR parallax due
to occlusion, as in the RL parallax shown by the one-dotted chain line.
[0067]
As a method of acquiring the RL parallax information, there are the
following two methods.
10 (f) Parallax information acquisition processing of a block matching base in a reverse
reference image
(g) Inversion processing of a sign of LR parallax + interpolation processing fiom
parallax of adjacent pixels
[0068]
15 In this way, in the parallax estimating unit 103, the LR parallax information
and the RL parallax information are acquired and generated.
[0069]
[Outline of Operation of Virtual Visual Point Image Generating Unit]
Next, basic virtual visual point image generation processing based on the
20 input LR images that is executed by the virtual visual point image generating unit
105 will be described.
[0070]
The L image from the left visual point image (L image) input unit 101, the R
image from the right visual point image (R image) input unit 102, and the parallax
25 information from the parallax estimating unit 103 are input to the virtual visual point
image generating unit 105. The virtual visual point image generating unit 105
receives each information and generates a virtual visual point image.
[007 11
For example, the virtual visual point image generating unit 105 determines
30 virtual visual points of a preset number (for example, 10) and generates a virtual
visual point image corresponding to each virtual visual point. The virtual visual
point image generating unit 105 generates the virtual visual point image using the
input standard LR images. That is, the virtual visual point image generating unit
105 generates the virtual visual point image using the left visual point image (L
image) and the right visual point image (R image) to be the input images. A
5 specific example of virtual visual point image generation processing will be
described with reference to FIG. 5.
[0072]
In the example of FIG 5, an original left visual point image 0, image) 13 1
and an original right visual point image (R image) 132 that are input to the image
10 processing apparatus and a virtual visual point image 133 that is generated on the
basis of the LR images are illustrated.
[0073]
The left visual point image & image) 13 1 is an image that is observed from
a left visual point position at the standard position and the right visual point image (R
15 image) 132 is an image that is observed from a right visual point position at the
standard position.
[0074]
FIG 5 illustrates a processing example of the case in which a visual point
position of the left visual point image (I, image) 131 is set to 0.0, a visual point
20 position of the right visual point image (R image) 132 is set to 1.0, and an image
observed from a visual point position between the visual point positions 0.0 to 1.0 =
0.3 is generated as the virtual visual point image 133.
[0075]
The same object (apple) is imaged at different positions in the left visual
25 point image (L image) 131 and the right visual point image (R image) 132. In the L
image and the R image, the positions of the same object become different from each
other, because the visual point positions become different from each other.
[0076]
When the virtual visual point image 133 of the visual point position = 0. 3
30 between the visual point position = 0.0 and the visual point position = 1.0 is
generated, the position of the object (apple) is set by linear interpolation. By
changing the object position along a straight line L1 illustrated in FIG 5 and
determining the object position of the virtual visual point image at each virtual visual
point, the virtual visual point image can be generated.
[0077]
5 As such, the virtual visual point image of each virtual visual point position
is generated by linear interpolation processing based on the input LR images.
[0078]
When the virtual visual point image is generated, the virtual visual point
image can be generated by processing for blending two images using both the input
10 LR images. Alternatively, the virtual visual point image can be generated by
processing for shifting the object position according to the virtual visual point
position using only the L image or the R image, that is, one image. Alternatively,
processing for generating the virtual visual point image using only the L image at the
virtual visual point position close to the side of the L image and generating the
15 virtual visual point image using only the R image at the position close to the R image
may be executed.
[0079]
An example of determination processing of a pixel value of the virtual
visual point image 131 based on the processing for blending the input LR images
20 will be described with reference to FIG 6.
[OOSO]
In the example of FIG 6, a pixel P(x, y) 141 of an input l& visual point
image 0, image) at a visual point position = 0 and a correspondence pixel 142 of the
pixel P of the L image in an input right visual point image (R image) at a visual point
25 position = 1 are illustrated. In addition, a correspondence pixel 143 of the pixel P of
the L image in a virtual visual point image at a visual point position = 0 is illustrated.
In this case, cf, is a value of 0 to 1.
[OOS 11
When the parallax of the pixel P(x, y) 141 of the left visual point image (L
30 image) is d(x, y) [pixel], a pixel position of the correspondence pixel 143 of the pixel
P(x, y) of the L image in the virtual visual point image is a pixel Q(x + iD d(x, y), y).
That is, a pixel value of the pixel Q(x + 0 - d(x, y), y) ini the virtual visual point
image is set to a pixel value of the pixel P(x, y) 141 of the left visual point image (L
image).
[0082]
5 As such, a pixel value of each pixel of the virtual visual point image is se on
the basis of the parallax information of the pixel of the left visual point image (L
image).
[0083]
A pixel value of a pixel not embedded in the virtual visual point image by
10 the above processing is determined by processing in which the right visual point
image (R image) is applied, interpolation processing based on pixel values of
adjacent pixels, or processing for performing interpolation by a pixel of the same
coordinates of the left visual point image.
[0084]
15 In an example of FIG 7, a horizontal line 15 1 of the left visual point image
(L image), a horizontal line 152 of the right visual point image (R image), and a
horizontal line 153 of the virtual visual point image are illustrated. An arrow
illustrated in FIG. 7 is a line that connects a pixel position of the left visual point
image (L image) and a pixel position of the right visual point image (R image)
20 applicable to determine a pixel value of the horizontal line 153 of the virtual visual
point image.
[0085]
In the horizontal line 153 of the virtual visual point image illustrated in FIG
7, 1 shows a region in which pixel values are set by constituent pixel values of the
25 horizontal line 151 of the left visual point image (L image), 2 shows a region in
which pixel values are set by constituent pixel values of the horizontal line 152 of the
right visual point image (R image), and 3 shows the other region.
[0086]
As such, setting of the pixel value of the virtual visual point image is
30 executed by the following three processing.
1. A corresponding pixel position at an output visual point position is calculated with
respect to each pixel of the left visual point image (L image) and a pixel value of the
left visual point image (L image) is interpolated to the pixel position.
2. A corresponding pixel position at an output visual point position is calculated with
respect to each pixel of the right visual point image (R image) and a pixel value of
5 the right visual point image (R image) is interpolated to the pixel position.
3. Interpolation processing based on adjacent pixels is performed with respect to a
pixel of the output visual point image that is not interpolated by the processing of 1
and 2.
[0087]
10 The processing that is described with reference to FIGS. 6 and 7 is basic
processing for generating an image from the virtual visual point different from the
LR images, on the basis of the input LR images.
[0088]
The virtual visual point image generating unit 105 of the image processing
15 apparatus according to the present disclosure applies a scale value (parallax range)
calculated from parallax information, on the basis of the basic processing. That is,
the virtual visual point image generating unit 105 determines a generated virtual
visual point position and an interpolation direction on the basis of the scale value and
generates a final virtual visual point image.
20 COO891
[Detail of Virtual Visual point Image Generating Unit]
Next, the detail of the virtual visual point image generating unit 105 will be
described.
[0090]
2 5 The L image from the left visual point image (L image) input unit 1 0 1, the R
image from the right visual point image (R image) input unit 102, and the parallax
information from the parallax estimating unit 103 are input to the virtual visual point
image generating unit 105.
[0091]
30 The virtual visual point image generating unit 105 adjusts a parallax amount,
that is, determines a generated virtual visual point position (phase), on the basis of a
parallax distribution (parallax range) calculated from the input information, and
selects an interpolation direction according to a scale value. The virtual visual point
image generating unit 105 generates a virtual visual point image corresponding to the
determined virtual visual point position (phase), on the basis of an image of the
5 selected interpolation direction. The virtual visual point image generating ugit 105
synthesizes the generated virtual visual point image, that is, an image of the adjusted
visual point position and outputs a synthesis image to a rear step.
[0092]
The virtual visual point image generating unit 105 generates virtual visual
10 point images corresponding to the determined virtual visual point positions (phases),
on the basis of the L image and the R image, and outputs the image of the selected
interpolation direction among the generated images to the rear step.
[0093]
FIG 8 is a diagram illustrating a configuration example of the virtual visual
15 point image generating unit.
100941
In the example of FIG 8, the virtual visual point image generating unit 105
includes a visual point position adjusting unit 161 and an image synthesizing unit
162.
20 [0095]
The parallax information is supplied from the parallax estimating unit 103 to
the visual point position adjusting unit 161. The visual point position adjusting unit
161 adjusts the parallax amount on the basis of the parallax information from the
parallax estimating unit 103 and determines the virtual visual point position (phase)
25 and the interpolation direction. The visual point position adjusting unit 161
supplies information of the determined virtual visual point position and information
of the determined interpolation direction to the image synthesizing unit 162.
[0096]
The L image from the left visual point image (L image) input unit 101, the R
30 image from the right visual point image (R image) input unit 102, the parallax
information from the parallax estimating unit 103, and the information of the virtual
visual point position and the interpolation direction from the visual point position
adjusting unit 16 1 are input to the image synthesizing unit 162.
[0097]
The image synthesizing unit 162 synthesizes the LR images with the image
5 of the adjusted visual point position, on the basis of the input information, and
outputs N visual point images after the synthesis to the display control unit 106 of a
rear step.
[0098]
[Operation of Visual Point Position Adjusting Unit]
10 Next, visual point position adjustment processing of the visual point
position adjusting unit 161 will be described with reference to FIG 9.
[0099]
The visual point position adjusting unit 161 generates a histogram of
parallax illustrated in FIG. 9 for each frame of the parallax information from the
15 parallax estimating unit 103 and executes the following processing. In the
histogram of the parallax, a horizontal axis shows the parallax and a vertical axis
shows the number of pixels (frequency).
[O loo]
First, the visual point position adjusting unit 161 calculates a maximum
20 value dmax and a minimum value dmin of the parallax, on the basis of the histogram
of the parallax. Next, the visual point position adjusting unit 161 sets a larger value
of Idmaxl and ldminl as a parallax range drange and calculates a scale value scale =
drangeldsafe.
[OlOl]
25 In this case, dsafe shows a target parallax value (prescribed value) and is
previously set from the following information: parallax in which crosstalk is settled
within an allowable range (display device dependency) or a comfortable parallax
range (3D consortium security guideline).
[O 1 021
30 The visual point position adjusting unit 161 calculates an output phase of
each visual point and calculates an interpolation direction. That is, the visual point
position adjusting unit 161 calculates a scale value as a parameter showing a degree
of a variation (in this case, a time variation) based on the parallax information and
executes visual point position adjustment processing according to the variation
shown by the scale value.
5 [0103]
[Calculation Processing of Output Phase]
First, calculation processing of an output phase will be described with
reference to FIG 10. The visual point position adjusting unit 161 determines
parallax of a virtual visual point image to be generated, that is, a position (phase) of
10 the virtual visual point image to be generated, according to the calculated scale value.
[0 1041
Specifically, the visual point position adjusting unit 16 1 executes
determination processing of the virtual visual point position illustrated in FIG 10,
according to the scale value of 0 to 1. If the scale value (parallax range) is small, a
15 time variation, that is, mismatching is likely to be small and if the scale value is large,
the time variation, that is, the mismatching is likely to be large.
[0 1051
FIG 10 is a diagram illustrating a setting example of the virtual visual point
image position when the scale value is 0 to 1. The visual point position = 0 is a
20 visual point position corresponding to the input L image and the visual point position
=I is a visual point position corresponding to the input R image.
[0 1 061
That is, an image b on a line of the scale value = 1 corresponds to the input
L image input from the left visual point image (L image) input unit 101 and an image
25 h corresponds to an input R image input from the right visual point image (R image)
input unit 102.
[0 1071
The other vertical lines on the line of the scale value = 1 show positions
(phases) of virtual visual point images generated in the virtual visual point image
30 generating unit 105, when the scale value is 1 (the mismatching is likely to be large).
In this example, a total of nine different visual point images of a to i including the
input LR images are generated and output.
[0108]
In the case of the scale value = 1, the visual point position adjusting unit 161
determines the images a to i at an upper stage of FIG 10 as the setting positions of
5 the virtual visual point images and outputs virtual visual point position information
to the image synthesizing unit 162. The generation processing of the virtual visual
point image is executed according to the processing described above with reference
to FIGS. 5 to 7.
[O 1091
10 In the case of the scale value = 0.5, that is, a middle value, the visual point
position adjusting unit 161 determines images a2 to i2 at an middle stage of FIG 10
as the setting positions of the virtual visual point images and outputs virtual visual
point position information to the image synthesizing unit 162.
[OllO]
15 In the case of the middle scale value = 0.5, as illustrated in FIG 10, a
parallax range of the virtual visual point images a2 to i2 becomes narrower than a
parallax range of the virtual visual point images a to i in the case of the scale value =
1.
[Olll]
20 In the case of the scale value = 0, that is, in the case where the mismatching
is rarely generated, the visual point position adjusting unit 161 determines images a3
to i3 at a lower stage of FIG 10 as the setting positions of the virtual visual point
images and outputs virtual visual point position information to the image
synthesizing unit 162.
25 [0112]
The image positions of the images a3 to i3 at the lower stage of FIG. 10
correspond to the image position of the input R image. That is, in this case, the
input R image is output as it is without generating a new virtual visual point image.
The virtual visual point image generating unit 105 outputs the input L image as it is
30 and only the input LR images are output to the display unit.
[0113]
The visual point position adjusting unit 161 executes calculation processing
of the setting position (phase) of the virtual visual point image according to the
following algorithm.
[0114]
5 The virtual visual point images that are generated in the case of the scale
value = 1 are determined in advance. For example, the virtual visual point images
are the virtual visual point images at the positions of a to i illustrated in FIG 10.
The calculated scale value is set as S (0 i S). When an original visual point
position is set as VO and the visual point position is converged into the right at the
10 scale value = 0, a virtual visual point image position (phase) V that is set according
to the scale value is represented by the following expression 2.
[Expression 21
V=(VO-1).S+1
[0115]
15 [Selection Processing of Interpolation Direction]
Next, selection processing of an interpolation direction of the visual point
position adjusting unit 161 will be described with reference to FIG 11. In an
example of FIG 11, a horizontal axis shows a phase and a vertical axis shows a scale
value S. The scale value is a value that is equal to or more than 0 and is not limited
20 to 0 to 1. In the example of FIG 11, N is illustrated as a value more than th.
However, N may have a value that is more than 1. In the example of FIG 11, the
case in which the position converged at the scale value S = 0 is the right (1) will be
described.
[0 1161
25 The visual point position adjusting unit 161 selects an interpolation
direction according to the scale value. At this time, as described above, when the
scale value (parallax range) is small, the mismatching of left and right images is
small. For this reason, the visual point position adjusting unit 161 sets the right as a
temporary interpolation direction, when the scale value is more than a predetermined
30 threshold value th. That is, in this case, the visual point position adjusting unit 161
prohibits changing of the interpolation direction to the left.
[0117]
Meanwhile, in the case in which the scale value is equal to or smaller than
the predetermined threshold value thy the visual point position adjusting unit 161 sets
the temporary interpolation direction, such that the interpolation is performed from
5 an image of the close side. That is, when a visual point phase is 0.5 or less, the
visual point position adjusting unit 161 sets the left as the temporary interpolation
direction and when the visual point phase is more than 0.5, the visual point position
adjusting unit 161 sets the right as the temporary interpolation direction. In this
case, the visual point position adjusting unit 161 performs changing of the
10 interpolation direction (permits the changing of the interpolation direction).
[0118]
Thereby, the changing of the interpolation direction when the mismatching
of the left and right images is large can be prevented.
[0119]
15 The visual point position adjusting unit 161 executes time stabilization
processing as follows. That is, when the temporary interpolation direction is the left
for a constant time, the visual point position adjusting unit 161 sets the interpolation
direction as the left and when the temporary interpolation direction is the right for the
constant time, the visual point position adjusting unit 161 sets the interpolation
20 direction as the right. In the other cases, the visual point position adjusting unit 161
sets the same direction as a previous frame to the interpolation direction.
[O 1201
In the case of a start frame, the temporary interpolation direction (close
image) is set to the interpolation direction.
25 [0121]
Thereby, the changing of the interpolation direction shown by an arrow A or
B can be suppressed from being frequently generated. That is, the time variation of
the high frequency of the interpolation direction and the variations at different
timings of both eyes of the left and right eyes can be suppressed.
30 [0122]
In the above description, the example of the case in which the position
converged at the scale value S = 0 is the right (1) has been described. However, the
position converged at the scale value S = 0 may be the left (0). When the position
converged at the scale value S = 0 is the left (0) and the scale value is more than the
predetermined threshold value th, the left is set as the temporary interpolation
5 direction.
[0 1231
[Configuration of Image Synthesizing Unit]
FIG 12 is a diagram illustrating a configuration example of the image
synthesizing unit 162.
10 [0124]
In the example of FIG 12, the image synthesizing unit 162 includes one
visual point image synthesizing units 171 -1 to 171-N corresponding to the generated
virtual visual point images including the input LR images.
[0 1251
15 The L image from the left visual point image (L image) input unit 10 1, the R
image from the right visual point image (R image) input unit 102, and the parallax
information (lewright) from the parallax estimating unit 103 are input to the one
visual point image synthesizing units 171 -1 to 171 -N.
[0 1261
20 An interpolation direction 1 and an output phase position 1 of a visual point
1 are input from the visual point position adjusting unit 161 to the one visual point
image synthesizing unit 171-1. The one visual point image synthesizing unit 171 -1
generates a virtual visual point image corresponding to the output phase position 1,
on the basis of the parallax information, using the input L image and the input R
25 image. The one visual point image synthesizing unit 171-1 selects the virtual visual
point image generated using the image of the direction (the left or the right)
corresponding to the interpolation direction 1 and outputs the virtual visual point
image as a synthesis image 1 to the display control unit 106 of the rear step.
[0 1271
30 An interpolation direction 2 and an output phase position 2 of a visual point
2 are input from the visual point position adjusting unit 161 to the one visual point
image synthesizing unit 171 -2. The one visual point image synthesizing unit 171-2
generates a virtual visual point image corresponding to the output phase position 2,
on the basis of the parallax information,. using the input L image and the input R
image. The one visual point image synthesizing unit 17 1-2 selects the virtual visual
5 point image generated using the image of the direction (the left or the right)
corresponding to the interpolation direction 2 and outputs the virtual visual point
image as a synthesis image 2 to the display control unit 106 of the rear step.
[0128]
An interpolation direction N and an output phase position N of a visual
10 point N are input from the visual point position adjusting unit 161 to the one visual
point image synthesizing unit 171 -N. The one visual point image synthesizing unit
171-N generates a virtual visual point image corresponding to the output phase
position N, on the basis of the parallax information, using the input L image and the
input R image. The one visual point image synthesizing unit 171-N selects the
15 virtual visual point image generated using the image of the direction (the left or the
right) corresponding to the interpolation direction N and outputs the virtual visual
point image as a synthesis image N to the display control unit 106 of the rear step.
[0129]
Hereinafter, the one visual point image synthesizing units 171 -1 to 171 -N
20 are collectively described as the one visua.1 point image synthesizing units 171, when
it is not necessary to distinguish the one visual point image synthesizing units 171-1
to 17 1-N in particular.
[0130]
[Configuration of One Visual point Image Synthesizing Unit]
25 FIG 13 is a diagram illustrating a configuration example of the one visual
point image synthesizing unit 17 1.
[0131]
The one visual point image synthesizing unit 171 includes a left image
synthesizing unit 18 1, a right image synthesizing unit 182, and a selecting unit 183.
30 [0132]
The L image from the left visual point image (L image) input unit 101, the
parallax information (left) from the parallax estimating unit 103, and the output
phase position from the visual point position adjusting unit 161 are input to the left
image synthesizing unit 18 1. The left image synthesizing unit 181 generates the
virtual visual point image corresponding to the output phase position, on the basis of
5 the parallax information (left), using the input L image, and outputs the virtual visual
point image to the selecting unit 183.
[0133]
The R image from the right visual point image (R image) input unit 102, the
parallax information (right) from the parallax estimating unit 103, and the output
10 phase position from the visual point position adjusting unit 161 are input to the right
image synthesizing unit 182. The right image synthesizing unit 182 generates the
virtual visual point image corresponding to the output phase position, on the basis of
the parallax information (right), using the input R image, and outputs the virtual
visual point image to the selecting unit 183.
15 [0134]
The interpolation direction from the visual point position adjusting unit 16 1,
the virtual visual point image generated using the L image from the left image
synthesizing unit 181, and the virtual visual point image generated using the R iinage
from the right image synthesizing unit 182 are input to the selecting unit 183.
20 [0135]
The selecting unit 183 selects the virtual visual point image generated using
the image of the direction corresponding to the interpolation direction from the visual
point position adjusting unit 161 and outputs the virtual visual point image as a
synthesis image to the display control unit 106 of the rear step.
25 [0136]
[Processing Example of Image Processing Apparatus]
Next, image processing of the image processing apparatus 100 of FIG. 2 will
be described with reference to a flowchart of FIG 14.
[0137]
30 In step S101, the left visual point image (L image) input unit 101 and the
right visual point image (R image) input unit 102 input the left visual point image (L
image) and the right visual point image (R image), respectively.
[0138]
The input left visual point image (L image) and right visual point image (R
image) are supplied to the parallax estimating unit 103 and the virtual visual point
5 image generating unit 105.
[0 1391
In step S102, the parallax estimating unit 103 estimates the parallax using
the supplied left visual point image (L image) and right visual point image (R image),
as described above with reference to FIGS. 3 and 4. The parallax information of the
10 estimation result by the parallax estimating unit 103 is supplied to the virtual visual
point image generating unit 105.
[0 1401
In steps S103 and S104, the virtual visual point image generating unit 105
executes the virtual visual point image generation processing.
15 [0141]
That is, in step S 103, the visual point position adjusting unit 161 adjusts the
visual point position. The visual point position adjustment processing is described
below with reference to FIG 15. The information of the output phase positions of
the N visual points and the information of the interpolation directions of the N visual
20 points are generated by step S103 and are supplied to the image synthesizing unit
162.
[0 1421
The L image from the left visual point image (L image) input unit 101, the R
image from the right visual point image (R image) input unit 102, and the parallax
25 information from the parallax estimating unit 103 are input to the image synthesizing
unit 162.
[O 1431
In step S104, the image synthesizing unit 162 synthesizes the LR images
with the image of the adjusted visual point position, on the basis of the input
30 information.
[0 1441
That is, as described above with reference to FIGS. 12 and 13, the image
synthesizing unit 162 generates the virtual visual point image corresponding to the
output phase position, on the basis of the parallax information, 'using the input L
image and the input R image. The one visual point image synthesizing unit 171
5 selects the virtual visual point image generated using the image of the direction (the
left or the right) corresponding to the interpolation direction and outputs the virtual
visual point image as the synthesis image to the display control unit 106 of the rear
step.
[0 1451
10 In step S105, the display control unit 106 displays the N visual point images
on the display unit 110.
[O 1461
[Example of Visual Point Position Adjustment Processing]
Next, an example of the visual point position adjustment processing in step
15 S103 of FIG 14 will be described with reference to a flowchart of FIG 15. In the
example of FIG 15, the visual point position is converged into the right, when the
scale value is 0.
[0 1471
In step S111, the visual point position adjusting unit 161 calculates the
20 maximum value dmax of the parallax and the minimum value dmin of the parallax,
on the basis of the histogram of the parallax.
[0148]
In step S112, the visual point position adjusting unit 161 sets a larger value
of ldmaxl and ldminl as a parallax range drange. In step S113, the visual point
25 position adjusting unit 161 calculates the scale value scale = drangeldsafe.
[0 1491
In step S114, the visual point position adjusting unit 161 calculates the
output phase on the basis of the scale value, as described above with reference to FIG.
10. The output phase position that is calculated by the processing of step S114 is
30 output to the image synthesizing unit 162.
[0150]
In step S115, the visual point position adjusting unit 161 executes the
selection processing of the interpolation direction described above with reference to
FIG. 11. The selection processing of the interpolation direction will be described
with reference to a flowchart of FIG 16.
5 [0151]
In this case, n shows a visual point number, N shows the total number of
visual points, St shows a scale value, S-th shows a threshold value (parameter), t
shows a time (frame), to shows a certain time (parameter), Vn,t shows a visual point
phase, Dn,t shows an interpolation direction, and DYn,t shows a temporary
10 interpolation direction.
[0 1 521
In step S12 1, the visual point position adjusting unit 16 1 substitutes -1 for t.
In step S122, the visual point position adjusting unit 161 determines whether all
scenes end. When it is determined that all of the scenes end, the visual point
15 position adjusting unit 161 ends the interpolation direction selection processing.
[0 1 531
When it is determined in step S122 that all of the scenes do not end, the
processing proceeds to step S123. In step S 123, the visual point position adjusting
unit 161 substitutes t+l for t. In step S124, the visual point position adjusting unit
20 161 substitutes 0 for n.
[0154]
In step S125, the visual point position adjusting unit 161 determines
whether n is equal to or more than N. When it is determined that n is equal to or
more than N, the processing returns to step S122 and the following processing is
25 repeated.
[0155]
When it is determined in step S125 that n is smaller than N, the processing
proceeds to step S126. In step S126, the visual point position adjusting unit 161
substitutes n+l for n. In step S127, the visual point position adjusting unit 161
30 determines whether St is more than S-th. When it is determined in step S127 that
St is equal to or smaller than S-thy the processing proceeds to step S128.
[0 1561
In step S128, the visual point position adjusting unit 161 determines
whether Vn,t is equal to or smaller than 0.5. When it is determined that Vn,t is
equal to smaller than 0.5, the processing proceeds to step S129 and the visual point
5 position adjusting unit 161 substitutes "left" for D'n,t. That is, in step S129, the left
is set to the temporary interpolation direction.
. .
[0157]
When it is determined in step S127 that St is more than S-thy the processing
proceeds to step S130. When it is determined in step S128 that Vn,t is more than
10 0.5, the processing proceeds to step S130.
[0158]
In step S130, the visual point position adjusting unit 161 substitutes "right"
for D'n,t. That is, in step S130, the right is set to the temporary interpolation
direction.
15 [0159]
In step S13 1, the visual point position adjusting unit 161 determines
whether t is 0. When it is determined that t is not 0, the processing proceeds to step
S132. In step S132, the visual point position adjusting unit 161 substitutes a
smaller value of TO and t for to.
20 [0160]
In step S133, the visual point position adjusting unit 161 determines
whether all D'n,s are "left" in s = t-to to t. When it is determined in step S133 that
all D'n,s are not "left" in s = t-to to t, the processing proceeds to step S134.
[0161]
25 In step S134, the visual point position adjusting unit 161 determines
whether all D'n,s are "right" in s = t-to to t. When it is determined in step S134 that
all DYn,s are "rightyy in s = t-to to t, the processing proceeds to step S135. In step
S135, the visual point position adjusting unit 161 substitutes "right" for Dn,t. That
is, in step S135, the right is set to the interpolation direction.
30 [0162]
When it is determined in step S133 that all DYn,s are "left", the processing
proceeds to step S136. In step S136, the visual point position adjusting unit 161
substitutes "left" for Dn,t. That is, in step S136, the left is set to the interpolation
direction.
[0 1 631
5 When it is determined in step S134 that all DYn,sa re not "right" in s = t-to to
t, the processing proceeds to step S137. In step S137, the visual point position
adjusting unit 161 substitutes.Dn,t-1 for Dn,t. That is, in step S137, an interpolation
direction of a previous frame is set to the interpolation direction.
[0 1 641
10 Meanwhile, when it is determined in step S13 1 that t is 0, the processing
proceeds to step S138. In step S138, the visual point position adjusting unit 161
substitutes DYn,t for Dn,t. That is, in step S138, the temporary interpolation
direction is set to the interpolation direction.
[0 1 651
15 In an example of FIG 16, the processing after step S131 is the time
stabilization processing.
[0 1661
As described above, because the interpolation direction is set according to
the scale value (parallax range), changing of the interpolation direction when
20 mismatching of the left and right images is large can be prevented.
[0 1671
Because the time stabilization processing is executed, the changing of the
interpolation direction can be suppressed from being frequently generated. That is,
the time variation of the high frequency of the interpolation direction and the
25 variations at different timings of both eyes of the left and right eyes can be
suppressed.
[0 1681
<3. Second Embodiment (Reliability)>
[Configuration Example of Image Processing Apparatus]
30 FIG 17 is a block diagram illustrating another configuration example of an
image processing apparatus to which the present disclosure is applied. In the
example of FIG 17, instead of the scale value described above with reference to FIG
9, reliability is calculated as a parameter showing a degree of a variation (in this case,
a time variation) based on the parallax information and visual point position
adjustment processing is executed according to the variation shown by the reliability.
5 [0169]
In the example of FIG 17, an image processing apparatus 200 includes a left
visual point image (L image) input unit 101, a right visual point image (R image)
input unit 102, a parallax estimating unit 103, a reliability calculating unit 201, a
virtual visual point image generating unit 202, and a display control unit 106. An
10 image that is generated in the image processing apparatus 200 is output to the display
unit 110.
[0 1 701
The image processing apparatus 200 of FIG 17 is the same as the image
processing apparatus 100 of FIG 2 in that the left visual point image (L image) input
15 unit 101, the right visual point image (R image) input unit 102, the parallax
estimating unit 103, and the display control unit 106 are provided. However, the
image processing apparatus 200 of FIG. 17 is different from the image processing
apparatus 100 of FIG 2 in that the reliability calculating unit 201 is additionally
provided and the virtual visual point image generating unit 105 is replaced by the
20 virtual visual point image generating unit 202.

What is claimed is:
1. An image processing apparatus comprising:
a parallax estimating unit that generates parallax information from a left
visual point image to be an image signal for a left eye applied to multi-dimensional
5 image display and a right visual point image to be an image signal for a right eye
applied to the multi-dimensional image display;
an interpolation direction control unit that controls changing of an
interpolation direction of a virtual visual point image including a visual point image
other than the left visual point image and the right visual point image, according to a
10 parameter showing a degree of a variation based on the parallax information
generated by the parallax estimating unit; and
a virtual visual point image generating unit that generates the virtual visual
point image in the interpolation direction of which the changing is controlled by the
interpolation direction control unit.
15
2. The image processing apparatus according to claim 1,
wherein the interpolation direction control unit prohibits the changing of the
interpolation direction of the virtual visual point image, when the variation shown by
the parameter is large.
20
3. The image processing apparatus according to claim 2,
wherein the interpolation direction control unit performs the changing of the
interpolation direction of the virtual visual point image, when the variation shown by
the parameter is small.
25
4. The image processing apparatus according to claim 2,
wherein the variation based on the parallax information that is generated by
the parallax estimating unit is a time variation.
30 5. The image processing apparatus according to claim 2, further comprising:
a reliability calculating unit that calculates reliability of the parallax
information generated by the parallax estimating unit,
wherein the parameter showing the degree of the variation based on the
parallax information generated by the parallax estimating unit is the reliability of the
parallax information calculated by the reliability calculating unit, and
5 the interpolation direction control unit controls the changing of the
interpolation direction of the virtual visual point image, according to the reliability of
the parallax information calculated by the reliability calculating unit.
6. The image processing apparatus according to claim 2,
10 wherein the parameter showing the degree of the variation based on the
parallax information generated by the parallax estimating unit is a scale value
calculated from the parallax information generated by the parallax estimating unit,
and
the interpolation direction control unit controls the changing of the
15 interpolation direction of the virtual visual point image, according to the scale value
calculated from the parallax information generated by the parallax estimating unit.
7. The image processing apparatus according to claim 2,
wherein the interpolation direction control unit selects one direction as the
20 interpolation direction of the virtual visual point image, according to the parameter
showing the degree of the variation based on the parallax information generated by
the parallax estimating unit,
when the selected one direction is selected as the interpolation direction of
the virtual visual point image continuously for a constant time, the interpolation
25 direction control unit changes the interpolation direction of the virtual visual point
image to the selected one direction, and
when the selected one direction is not selected as the interpolation direction
of the virtual visual point image continuously for the constant time, the interpolation
direction control unit prohibits the changing of the interpolation direction of the
30 virtual visual point image.
8. The image processing apparatus according to claim 2,
wherein the virtual visual point image generating unit sets a convergence
position of a visual point position to a left visual point or a right visual point and
calculates a virtual visual point position to generate the virtual visual point image,
5 using the parallax information generated by the parallax estimating unit, and
generates the virtual visual point image in the interpolation direction of which the
changing is controlled by the interpolation direction control unit, at the calculated
virtual visual point position.
10 9. The image processing apparatus according to claim 2,
wherein the virtual visual point image generating unit sets a convergence
position of a visual point position to any position between a left visual point and a
right visual point and calculates a virtual visual point position to generate the virtual
visual point image, using the parallax information generated by the parallax
15 estimating unit, and generates the virtual visual point image in the interpolation
direction of which the changing is controlled by the interpolation direction control
unit, at the calculated virtual visual point position.
10. The image processing apparatus according to claim 1, further comprising:
20 a face detecting unit that detects a position of a face of a user who views the
virtual visual point image which is generated by the virtual visual point image
generating unit and is displayed on a display unit,
wherein the interpolation direction control unit controls the changing of the
interpolation direction of the virtual visual point image, according to the position of
25 the face of the user detected by the face detecting unit.
11. The image processing apparatus according to claim 1,
wherein a display unit that displays the virtual visual point image generated
by the virtual visual point image generating unit is wearable on a head of a user,
30 the image processing apparatus further comprises a face detecting unit that
detects a position and a direction of a face of the user who views the virtual visual
point image displayed on the display unit, and
G..
the- interpolation direction control unit controls the changing of the
interpolation direction of the virtual visual point image, according to the position and
the direction of the face of the user detected by the face detecting unit.
5
12. The image processing apparatus according to claim 1, hrther comprising:
a scene change detecting unit that detects a scene change from the left visual
point image or the right visual point image,
wherein the interpolation direction control unit performs the changing of the
10 interpolation direction of the virtual visual point image, when the scene change is
detected by the scene change detecting unit.
13. An image processing method comprising:
causing an image processing apparatus to generate parallax information
15 from a left visual point image to be an image signal for a left eye applied to multi-
I
dimensional image display and a right visual point image to be an image signal for a
right eye applied to the multi-dimensional image display;
causing the image processing apparatus to control changing of an
interpolation direction of a virtual visud point image including a visual point image
20 other than the left visual point irnage.and the right visual point image, according to a
parameter showing a degree of a variation based on the generated parallax
information; and
causing the image processing apparatus to generate the virtual visual point
image in the interpolation direction of which the changing is controlled.