DESCRIPTION
IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, AND
PROGRAM
5 TECHNICAL FIELD
[0001]
The present invention relates to an image processing
apparatus, an image processing method, and a program, and more
specifically, to an image processing apparatus, an image
10 processing method, and a program for performing parallax
control of a parallax image that supports stereoscopic vision
(three-dimensional vision).
BACKGROUND ART
15 [0002]
In order to generate a so-called parallax image (also
referred to as a three-dimensional image or a 3D image) that
supports stereoscopic vision (three-dimensional vision), it
is necessary to prepare images f rom different viewpoints, i. e.,
20 a left eye image and a right eye image. More specifically,
images having parallax are generated as a left eye image and
a right eye image, and control is performed such that the left
eye image is observed with only the left eye of an observer
(user) and the right eye image is observed with only the right
25 eye of the observer (user), so that the observer can feel
stereoscopic feeling.
[0003]
Configurations for allowing a user to observe each image
with only one of the eyes include various methods. Examples
30 include a passive-glasses method for separating images
observed with the right and left eyes from each other using
1
SP262338WO00
polarization filters and color filters and an active-glasses
method for time-divisionally separating images to right and
left images using a liquid crystal shutter. In general,
stereoscopic images can be observed by observing images with
5 such special three-dimensional vision glasses.
As described above, various methods have been suggested
in the past as methods for achieving three-dimensional vision
by presenting images having parallax.
[0004]
10 However, when the parallax range is too narrow, i.e.,
when a width between a subject seen in proximity and a subject
seen at a distance is too narrow, there is a problem in that
a user is unable to sufficiently feel stereoscopic feeling.
On the other hand, it is reported that there are some observers
15 who feel tired or sickness when they continue to observe
thrne-dimensional images with large parallax ranges for a long
period of time. As a method for solving such problem, a method
for controlling the amount of parallax has been suggested in
order to give appropriate stereoscopic feeling.
20 [0005]
For example, Patent Document 1 (Japanese Patent
Application Laid-Open No. 07-167633) discloses a method for
detecting a retinal image difference (disparity)
corresponding to parallax from right and left images (a left
25 eye image and a right eye image) constituting a
three-dimensional image and adjusting the parallax by
horizontally shifting the right and left images on the basis
of the detected retinal image difference (disparity)
information. In other words, this method disclosed in Patent
30 Document 1 is an image conversion method for setting a parallax
range that can be comfortably seen by an observer, by
2
SP262338W000
horizontally shifting the entire image using an average value
of disparities.
[0006]
It should be noted that the disparity is a retinal image
5 difference or a value corresponding to a distance, on a display
unit, between a subject in the left eye image and the same
subject in the right eye image, and the larger the disparity
is, the larger the parallax is.
This method disclosed in Patent Document 1 is an image
10 conversion method for setting a parallax range that can be
comfortably seen by an observer, by horizontally shifting the
entire image using an average value of disparities.
[0007]
In this configuration disclosed in Patent Document 1,
15 the amount of horizontal shift is determined using an average
value of disparities, and there is an advantage in that
conversion can be performed with less amount of processing.
In this configuration disclosed in Patent Document 1, however,
the entire right and left images are horizontally shifted,
20 and there is a problem in that the dynamic range of parallax
(difference in depth between the closest subject and the
farthest subject) cannot be controlled.
[0008]
On the other hand, Non-patent Document 1 ("Stereoscopic
25 image generation based on depth image for 3D TV", L. Zhang
and W. J. Tam, IEEE Trans. On Broadcasting, Vol. 51, No. 2,
June 2005) discloses a method for generating a disparity map
describing disparity corresponding to each pixel of images
(displacement between a left image and a right image) from
30 the right and left images and recovering images for different
viewpoints, using the disparity map and the original images.
3
SP262338W000
[0009]
With this method disclosed in Non-patent Document 1,
an image from a new viewpoint can be generated, and images
can be generated in which the dynamic range of parallax of
5 the stereo image are also controlled. In this method described
in Non-patent Document 1, however, it is essential to generate
the disparity map describing disparity (image difference) for
each pixel of images, and this increases the processing cost,
and there is a problem in that it is difficult to reduce the
10 size of the circuit. Moreover, since the quality of the
generated images depends on the accuracy of the disparity map,
it is required to generate the disparity map with a high
resolution and a high degree of accuracy. It is difficult
to generate such disparity map with a high degree of accuracy,
15 which becomes a factor for increasing the size of the circuit.
CITATION LIST
PATENT DOCUMENT
[0010]
20 Patent Document 1: Japanese Patent Application Laid-Open No.
07-167633
NON-PATENT DOCUMENT
[0011]
Non-patent Document 1: "Stereoscopic image generation based
25 on depth image for 3D TV", L. Zhang and W. J. Tam, IEEE Trans.
On Broadcasting, Vol. 51, No. 2, June 2005
SUMMARY OF THE INVENTION
PROBLEMS TO BE SOLVED BY THE INVENTION
30 [0012]
The present invention is made in view of, for example,
4
SP262338WO00
the above problems, and it is an object of the present invention
to provide an image processing apparatus, an image processing
method, and a program capable of controlling parallax of stereo
images without generating any disparity map describing
5 disparity for each pixel of images or by only generating a
rough disparity map including less amount of data that can
be generated with less processing.
SOLUTIONS TO PROBLEMS
10 [0013]
A first aspect of the present invention lies in:
an image processing apparatus including:
a left image transforming unit for inputting a left image
which is to be presented to a left eye and which is applied
15 to display of a stereoscopic image, changing a phase of an
image signal of a left image in a right direction or a left
direction, and generating a left image-transformed image; and
a right image transforming unit for inputting a right
image which is to be presented to a right eye and which is
20 applied to display of a stereoscopic image, changing a phase
of an image signal of a right image in the left direction or
the right direction, and generating a right image-transformed
image,
wherein the left image transforming unit and the right
25 image transforming unit extract feature quantities of the image
signals of the input images, and generate the left
image-transformed image and the right image-transformed image
using image transformprocessing to which the extracted f ea Lure
quantities are applied.
30 [0014]
Further, in an embodiment of the image processing
SP262338WO00
apparatus according to the present invention, the left image
transforming unit includes a left image differentiating device
for generating a differential signal of the image signal of
the left image as the feature quantity, and a left image
5 combining unit for generating the left image-trans formed image
by executing combining processing in which the differential
signal of the left image or a transformed signal of the
differential signal is added to the left image signal, wherein
the right image transforming unit includes a right image
10 differentiating device for generating a differential signal
of the image signal of the right image as the feature quantity,
and a right image combining unit for generating the right
image-transformed image by executing combining processing in
which the differential signal of the right image or a
15 transformed signal of the differential signal is added to the
right image signal.
[0015]
Further, in an embodiment of the image processing
apparatus according to the present invention, the left image
20 transforming unit includes a left image non-linear
transforming unit for executing non-linear transform
processing of the differential signal of the left image, and
the left image combining unit generates the left
image-transformed image by executing combining processing in
25 which a transformed signal generated by the left image
non-linear transforming unit is added to the left image signal,
and the right image transforming unit includes a right image
non-linear transforming unit for executing non-linear
transform processing of the differential signal of the right
30 image, and the right image combining unit generates the right
image-transformed image by executing combining processing in
6
SP262338W000
which a transformed signal generated by the right image
non-linear transforming unit is added to the right image
signal.
[0016]
5 Further, in an embodiment of the image processing
apparatus according to the present invention, the left image
differentiating device and the right image differentiating
device execute differential processing to which first-order
differentiation filters having differential filter
10 coefficient series of opposite patterns are applied.
[0017]
Further, in an embodiment of the image processing
apparatus according to the present invention, the left image
differentiating device and the right image differentiating
15 device execute differential processing according to a same
differential mode, and one of the left image combining unit
and the right image combining unit adds the differential signal
of each image or the transformed signal of the differential
signal to the input image signal, and the other of them performs
20 processing of deducting the differential signal of each image
or the transformed signal of the differential signal from the
input image signal.
[0018]
Further, in an embodiment of the image processing
25 apparatus according to the present invention, the left image
differentiating device and the right image differentiating
device execute differential processing of a brightness signal
of the input image signal.
[0019]
30 Further, in an embodiment of the image processing
apparatus according to the present invention, the image
7
SP262338W000
processing apparatus further includes a parallax detecting
unit generating parallax information by analyzing arrangement
of a corresponding point which corresponds to a same subject
portion in the left image and the right image which are input
5 into the image processing apparatus, and the left image
differentiating device and the right image differentiating
device execute differential processing by changing a
differential processing mode in accordance with the parallax
information generated by the parallax detecting unit.
10 [0020]
Further, in an embodiment of the image processing
apparatus according to the present invention, the parallax
detecting unit generates disparity polarity information
indicating that the arrangement of the corresponding point
15 which corresponds to the same subject portion in the left image
and the right image which are input into the image processing
apparatus is any one of the following settings (a) and (b) :
(a) a corresponding point of the left image is located at left
of a corresponding point of the right image; and (b) the
20 corresponding point of the left image is located at right of
the corresponding point of the right image, and the left image
differentiating device and the right image differentiating
device execute differential processing to which first-order
differentiation filters having differential filter
25 coefficient series of opposite patterns are applied, in
accordance with the disparity polarity information generated
by the parallax detecting unit.
[0021]
Further, in an embodiment of the image processing
30 apparatus according to the present invention, the parallax
detecting unit generates parallax information by applying
8
SP262338W000
reduced images or decimated images of the left image and the
right image which are input to the image processing apparatus.
[0022]
Further, in an embodiment of the image processing
5 apparatus according to the present invention, the image
processing apparatus further includes a control signal input
unit for inputting a control signal for controlling change
to at least one of a differential processing mode for the left
image differentiating device and the right image
10 differentiating device and a transform processing mode for
the left image non-linear transforming unit and the right image
non-linear transforming unit.
[0023]
Further, a second aspect of the present invention lies
15 in:
an image processing method executed by an image
processing apparatus, including:
a left image transforming step for causing a left image
transforming unit to input a left image which is to be presented
20 to a left eye and which is applied to display of a stereoscopic
image, to change a phase of an image signal of a left image
in a right direction or a left direction, and to generate a
left..image-transformed image; and
a right image transforming step for causing a right image
25 transforming unit to input a right image which is to be presented
to a right eye and which is applied to display of a stereoscopic
image, to change a phase of an image signal of a right image
in the left direction or the right direction, and to generate
a right image-transformed image,
30 wherein the left image transforming step and the right
image transforming step are steps for extracting feature
9
SP262338WO00
quantities of the image signals of the input images, and
generating the left image-transformed image and the right
image-transformed image using image transform processing to
which the extracted feature quantities are applied.
5 [0024]
Further, a third aspect of the present invention lies
A program for causing an image processing apparatus to
execute image processing, wherein the program causes the image
10 processing apparatus to execute:
a left image transforming step for causing a left image
transforming unit to input a left image which is to be presented
to a left eye and which is applied to display of a stereoscopic
image, to change a phase of an image signal of a left image
15 in a right direction or a left direction, and to generate a
left image-transformed image; and
a right image transforming step for causing a right image
transforming unit to input a right image which is to be presented
to a right eye and which is applied to display of a stereoscopic
20 image, to change a phase of an image signal of a right image
in the left direction or the right direction, and to generate
a right image-transformed image,
wherein in the left image transforming step and the right
image transforming step, feature quantities of the image
25 signals of the input images are extracted, and the left
image-transformed image and the right image-transformed image
are generated using image transform processing to which the
extracted feature quantities are applied.
[0025]
30 It should be noted that the program of the present
invention is, for example, a program that can be provided,
10
SP262338WO00
as a storage medium or a communication medium provided in a
computer-readable format, to a general purpose system that
can execute various program codes. By providing such programs
in the computer-readable format, processing according to the
5 programs is achieved on a computer system.
[0026]
Other objects, features, and advantages of the present
invention will become apparent from more detailed description
based on attached drawings and embodiments of the present
10 invention explained below. In this specification, a system
is a logical configuration of a set of multiple apparatuses,
and an apparatus of each configuration is not necessarily
limited to be provided within the same housing.
15 EFFECTS OF THE INVENTION
[0027]
According to a configuration of an embodiment of the
present invention, an apparatus and a method for performing
a parallax control of a left image and a right image applied
20 to display of a stereoscopic image are provided. The image
processing apparatus according to the present invention
includes a left image transforming unit for generating a left
image-transformed image by changing a phase of an image signal
of a left image which is to be presented to a left eye in a
25 right direction or a left direction and a right image
transforming unit for generating a right image-transformed
image by changing a phase of an image signal of a right image
which is to be presented to a right eye in the left direction
or the right direction. For example, each image transforming
30 unit generates a differential signal by applying, to an input
image, differential filter coefficients of coefficient series
11
SP262338WO00
of opposite characteristics, and generates a
parallax-controlled transformed signal using combining
processing in which the differential signal or a non-linear
signal of this differential signal is added to an original
5 image signal. This processing achieves processing for, e.g.,
reduction or enlargement of a parallax range.
BRIEF DESCRIPTION OF DRAWINGS
[0028]
10 Fig. 1 is a figure explaining an example of configuration
of an image processing apparatus according to an embodiment
of the present invention.
Figs.2(a) and2(b) are figures explaining differential
processing performed by a differentiating device of the image
15 processing apparatus according to the embodiment of the present
invention.
Fig. 3 is a figure explaining non-linear transform
processing performed by a non-linear transforming unit of the
image processing apparatus according to the embodiment of the
20 present invention.
Figs. 4(a) to 4(c) are figures explaining an example
of processing performed by a left image transforming unit of
the image processing apparatus according to the embodiment
of the present invention.
25 Figs. 5(a) to 5(c) are figures explaining an example
of processing performed by a right image transforming unit
of the image processing apparatus according to the embodiment
of the present invention.
Fig. 6 is a figure explaining an example of image
30 transform processing executed by the image processing
apparatus according to the embodiment of the present invention.
12
SP262338W000
Figs. 7(a) and 7(b) are figures explaining illustrating
a specific example of control processing of image transform
processing executed by an image processing apparatus according
to an embodiment 1 of the present invention.
5 Fig. 8 is a figure explaining an example of
correspondence of signal patterns in image transform
processing executed by the image processing apparatus
according to the embodiment of the present invention.
Fig. 9 is a figure explaining an example of
10 correspondence of signal patterns in image transform
processing executed by the image processing apparatus
according to the embodiment of the present invention.
Figs. 10 (a) and 10 (b) are figures explaining an example
of correspondence of signal patterns in image transform
15 processing executed by the image processing apparatus
according to the embodiment of the present invention.
Figs. 11 (a) and 11 (b) are figures explaining an example
of correspondence of signal. patterns in image transform
processing executed by the image processing apparatus
20 according to the embodiment of the present invention.
Fig. 12 is a figure explaining an example of image
transform processing executed by the image processing
apparatus according to the embodiment of the present invention.
Figs. 13 (a) and 13 (b) are figures explaining a specific
25 example of control processing achieved with image transform
processing executed by an image processing apparatus according
to an embodiment 2 of the present invention.
Figs. 14 (a) and 14 (b) are figures explaining a specific
example of control processing achieved with image transform
30 processing executed by an image processing apparatus according
to an embodiment 3(a) of the present invention.
13
SP262338W000
Figs. 15 (a) and 15 (b) are figures explaining a specific
example of control processing achieved with image transform
processing executed by an image processing apparatus according
to an embodiment 3(b) of the present invention.
5 Figs. 16(a) and 16 (b) are figures explaining a specific
example of control processing achieved with image transform
processing executed by an image processing apparatus according
to an embodiment 4(a) of the present invention.
Figs. 17 (a) and 17 (b) are figures explaining a specific
10 example of control processing achieved with image transform
processing executed by an image processing apparatus according
to an embodiment 4(b) of the present invention.
Fig. 18 is a figure explaining an example of
configuration of the image processing apparatus according to
15 the embodiment of the present invention.
Figs. 19 (a) and 19 (b) are figures explaining a specific
example of control processing achieved with image transform
processing executed by an image processing apparatus according
to an embodiment 5(a) of the present invention.
20 Figs. 20(a) and 20(b) are figures explaining a specific
example of control processing achieved with image transform
processing executed by an image processing apparatus according
to an embodiment 5(b) of the present invention.
Fig. 21 is a figure explaining an example of
25 configuration of the image processing apparatus according to
the embodiment of the present invention.
Fig. 22 is a figure explaining an example of
configuration of a differentiating device of the image
processing apparatus according to the embodiment of the present
30 invention.
Fig. 23 is a figure explaining an example of
14
SP262338WO00
configuration of the image processing apparatus according to
the embodiment of the present invention.
MODE FOR CARRYING OUT THE INVENTION
5 [0029]
An image processing apparatus, an image processing
method, and a program according to the present invention will
be hereinafter explained in detail with reference to drawings.
The following explanation will be made in order in accordance
10 with corresponding items listed below.
A. Embodiment 1: Example of reducing processing of
parallax range set at a deeper side with respect to a display
surface of a display unit
B. Embodiment 2: Example of enlarging processing of
15 parallax range set at a deeper side with respect to a display
surface of a display unit
C. Embodiment 3: Example of control
(reducing/enlarging) processing of parallax range set at a
closer side with respect to a display surface of a display
20 unit
D. Embodiment 4: Example of control processing of
parallax range set at either side of a display surface of a
display unit
E. Embodiment 5: the embodiment for performing parallax
25 control using simplified disparity map
F. Embodiment 6: the embodiment having configuration
to enable input of signal for control of shift mode
[0030]
[A. Embodiment 1: Example of reducing processing of
30 parallax range set at a deeper side with respect to a display
surface of a display unit]
15
SP262338WO00
An embodiment 1 of the present invention will be
explained with reference to Fig. 1 and subsequent drawings.
The embodiment 1 is an embodiment in which parallax
control can be performed without using any disparity map in
5 which a value of disparity corresponding to image displacement
of right and left images (a left eye image and a right eye
image) constituting a stereoscopic image (three-dimensional
image) (a retinal image difference and the like, and a display
position distance between a left image and a right image on
10 a display unit) is associated with each pixel.
In the present embodiment, an image processing apparatus
is achieved, which executes parallax control such as reducing
processing and enlarging processing of a parallax range of
right and left images (left eye image and right eye image)
15 constituting a three-dimensional image without generating any
dis--Rarity map.
[0031]
An example of configuration of an image processing
apparatus according to the present embodiment will be explained
20 with reference to Fig. 1. As illustrated in Fig. 1, an image
processing apparatus 100 includes a left image transforming
unit 110 for receiving a left image (L1 image) 10, performing
image transform, and generating a parallax-adjusted left image
(L2 image) 50 in which parallax is adjusted, and also includes
25 a right image transforming unit 120 for receiving a right image
(RI image) 20, performing image transform, and generating a
parallax-adjusted right image (R2 image) 60 in which parallax
is adjusted.
[0032]
30 The left image (L1 image) 10 and the right image (R1
image) 20 serving as input images are a left eye image and
16
SP262338WO00
a right eye image, respectively, which are applied for display
of a stereoscopic image (three-dimensional image) . In other
words, the left image (L1 image) 10 and the right image (Rl
image) 20 are images in which parallaxes are set in accordance
5 with subject distances, such as images taken from different
viewpoints. It should be noted that the left image (L1 image)
10 and the right image (Rl image) 20 maybe either still pictures
or motion pictures such as video data. In either of the cases
of the motion picture or the still picture, a pair of the left
10 image (Ll image) 10 and the right image (Rl image) 20 which
are respectively applied to display three-dimensional images
is successively input to a left image transforming unit 110
and a right image transforming unit 120, and these units
generate the parallax-adjusted left image (L2 image) 50 and
15 the parallax-adjusted right image (R2 image) 60 in which
parallax is adjusted by applying transform processing of the
images.
[0033]
0
The left image transforming unit 110 includes a left
20 image input unit 111 for receiving the left image (L1) 10,
a left image differential processing unit 112 for performing
differential processing of the left image 10, a left non-linear
transforming unit 113 for non-linearly transforming a
differential signal of the left image 10, a left image combining
25 unit 114 for combining the left image 10 and the differential
signal non-linearly transformed, and a left image output unit
115 for outputting the converted parallax-adjusted left image
(L2) 50.
[0034]
30 The right image transforming unit 120 includes a right
image input unit 121 for receiving the right image (Rl) 20,
17
SP262338W000
a right image differential processing unit 122 for performing
differential processing of the right image 20, a right
non-linear transforming unit 123 for non-linearly
transforming a differential signal of the right image 20, a
5 right image combining unit 124 for combining the right image
20 and the differential signal non-linearly transformed, and
a right image output unit 125 for outputting the converted
parallax-adjusted right image (R2) 60.
[0035]
10 The left image transforming unit 110 and the right image
transforming unit 120 perform processing to generate the
parallax-adjusted left image 50 and the parallax-adjusted
right image 60 in which parallaxes are controlled, by
extracting spatial feature quantities of the input image
15 signals and applying different emphasis processing on the
extracted feature quantities, and to output the
parallax-adjusted left image 50 and the parallax-adjusted
right image 60 via the left image output unit 115 and the right
image output unit 125.
20 [0036]
For example, the left image output unit 115 and the right
image output unit 125 are connected to a display unit for
performing display processing of the three-dimensional images,
and performs display processing of three-dimensional images
25 in which the parallax-adjusted left image (L2) 50 and the
parallax-adjusted right image (R2) 60 in which parallaxes are
adjusted are applied.
Alternatively, the left image output unit 115 and the
right image output unit 125 are connected to a recording unit
30 for executing recording processing to a recording medium, so
that recording processing is performed to record the
18
SP262338W000
parallax-adjusted left image (L1) 50 and parallax-adjusted
right image (R1) 60 to the recording medium. Alternatively,
the left image output unit 115 and the right image output unit
125 may be configured to transmit the parallax-adjusted left
5 image (L1) 50 and parallax-adjusted right image (Rl) 60 to
an external apparatus via a communication unit.
[0037]
The details of processing executed by the left image
transforming unit 110 will be explained.
10 The left image input unit 111 receives the left image
(L1) 10 generated in advance to be applied to display of the
three-dimensional image.
The left image differentiating device 112 extracts image
feature quantities from the left image 10 which is input to
15 the left image transforming unit 110. More specifically, the
left image differentiating device 112 retrieves a brightness
signal from the left image 10, and generates a differential
signal (HL) of the brightness signal. For example, the left
image differentiating device 112 receives a brightness signal
20 of an image in a horizontal direction, and generates a signal
obtained by performing first-order differentiation on the
input brightness signal. In the first order differential
proc?ssing,for example, a linear first-order differentiation
filter and the like with three taps in the horizontal direction
25 is used.
[0038]
Fig. 2 illustrates an example of generating processing
of a differential signal of the left image differentiating
device 112.
30 Fig. 2 shows an example of signals listed below.
(a) input signal
19
SP262338WO00
(b) differential signal
[0039]
Fig. 2(a) is an example of input image signal.
Fig. 2 (b) is an image obtained by performing differential
5 processing on the input images igna I of Fig. 2 (a). For example,
Fig. 2(b) showsabrightnessdifferential signal (HL) generated
by the left image differentiating device 112.
In the embodiment, a case will be explained where the
brightness signal is adopted as processing data. However,
10 not only the brightness signal but also a color signal (such
as RGB) may be adopted as the processing target data.
[0040]
The left image non-linear transforming unit 113
non-linearly transforms the differential signal (HL) which
15 is output from the left image differentiating device 112, and
outputs the signal as a non-linear transformed signal (EL)
to the image combining unit 114.
Fig. 3 illustrates an example of non-linear transform
processing executed by the left image non-linear transforming
20 unit 113. The horizontal axis denotes a brightness value of
the differential signal (brightness differential signal)
which is output from the left image differentiating device
112, and the vertical axis denotes an output brightness value
provided as a result of the non-linear transform processing
25 performed by the left image non-linear transforming unit 113.
[0041]
The left image non-linear transforming unit 113
transforms the input differential signal (In (= HL) ) using
a function f (x) defined in advance, and outputs a
30 parallax-emphasized signal (Out (= EL)). More specifically,
Out = f (In) holds. At this occasion, various kinds of setting
20
SP262338W000
may be used in the function f (x). An example of function
f (x) includes the following function.
f (x) = xr
An exponential function as described in the above
5 expression is used. y is a coefficient set in advance, and
it can be set to various values.
The transform function used in the left image non-linear
transforming unit 113 is not limited to the exponential
function. Alternatively, linear transform maybe performed.
10 [0042]
As indicated with broken lines in Fig. 1, the processing
of the left image non-linear transforming unit 113 may be
omitted. In this case, the differential signal (HL) which
is output from the left image differentiating device 112 is
15 directly input to the left image combining unit 114.
[0013]
The left image combining unit 114 performs processing
of generating the parallax-adjusted left eye image 50 by
applying the left image (L1) 10 and spatial feature quantities
20 generated fromthis left image (Ll) 10, i.e., the differential
signal (HL) of the brightness signal or the non-linear
transformed signal (EL) generated by non-linearly converting
this,, differential signal.
[0044]
25 Subsequently, the processing of the left image combining
unit 114 will be explained.
The left image combining unit 114 performs processing
of generating the parallax-adjusted left eye image 50 by
applying the left image 10 which is the input image and the
30 spatial feature quantities generated from this left image 10,
i.e., the differential signal of the brightness signal or the
21
SP262338W000
non-linear transformed signal generated by non-linearly
converting this differential signal. The generated
parallax-adjusted left image 50 is output via the left image
output unit 115, and for example, it is displayed on the display
5 unit.
[0045]
Figs. 4(a) to 4(c) illustrate an example of image
combining processing performed by the left image combining
unit 114. The following signals are shown from the top of
10 Figs. 4(a) to 4(c).
(a) input signal (L1)
(b) differential signal (HL)
(c) combined image signal (= parallax-adjusted left
image signal (L2))
15 [0046]
(a) Input signal (Ll) denotes brightness change of the
left image (Ll) 10, which is input into the image processing
apparatus 100, for any given one horizontal line of any given
frame of video data, for example. This shows an example of
20 one line in which there is a high brightness region having
high brightness in a central portion.
Ina region P from a line position (xl) to a line position
(x2), (.a) input signal (L1) shows a change such that the
brightness gradually increases. Between line positions (x2)
25 to (x3) , there is a high brightness portion in which high level
brightness is maintained. Thereafter, in a region Q from a
line position (x3) to a line position (x4), (a) input signal
(Ll) shows a change such that the brightness gradually
decreases.
30 [0047]
(b) Differential signal (HL) denotes a, result of
22
SP262338W000
differential of (a) input signal (L1). This differential
signal is a signal generated by the left image differentiating
device 112 of the left image transforming unit 110 as
illustrated in Fig. 1.
5 It should be noted that differential filter coefficients
used by the left image differentiating device 112 make a
differential filter having differential filter coefficients
of the following sequence as illustrated in Fig. 4(b).
1, 0, -1
10 The differential processing performed with the filter
having the differential filter coefficients of the above
sequence means applying:
a differential signal (In') of a pixel (n) in a horizontal
line,
15 a pixel value of a pixel (n-1) before (at the left of)
the pixel (n) (in the present example, brightness) = (I (n-1) )
and
a pixel value of a pixel (n+1) after (at the right of)
the pixel (n) (in the present example, brightness) = (I (n-I-1)) ,
20 and calculating as follows.
In' = (I (n-1)) - (I (n+l))
[0048]
As illustrated in Figs. 4 (a) to 4 (c) , the differential
signal (LH) generated by the left image differentiating device
25 112 has a negative value in the region P in which the brightness
change of (a) input left image signal (L1) is positive, and
the differential signal (LH) generated by the left image
differentiating device 112 has a positive value in the region
Q in which the brightness change of (a) input left image signal
30 (L1) is negative.
[0049]
23
SP262338W000
(c) Combined signal (parallax-adjusted left image
signal (L2)) is a signal generated by the left image combining
unit 114 of the left image transforming unit 110 as illustrated
in Fig. 1. The left image combining unit 114 generates the
5 combined signal (parallax-adjusted left image signal (L2))
by executing combining processing for adding (a) input left
image signal (L1) and (b) left image differential signal (HL) .
As a result, the combined signal (parallax-adjusted left image
signal (L2)) as illustrated in Fig. 4(c) is generated.
10 [0050]
As can be understood from this figure, the brightness
change portions P, Q of (a) input signal (Li) are phase-changed
(shifted) in the right direction in (c) combined image signal
(= parallax-adjusted left image signal (L2)). This is the
15 effect caused by the combining (adding) processing of the
differential signal to the input signal (L1).
[0051]
In the explanation of Figs. 4(a),to 4(c), an example
is shown in which the processing of the left image non-linear
20 transforming unit 113 as illustrated in Fig. 1 is omitted.
However, when the processing of the left image nor-linear
transforming unit 113 is added, the amplitude of the
differential signal of Fig. 4 (b) is controlled (for example,
suppressed).
25 In this case, (c) combined image signal (=
parallax-adjusted left image signal (L2) ), which is the result
of the combining processing performed by the left image
combining unit 114, is also a phase-changed (shifted) signal,
i.e., in which the brightness change portion P, Q are moved
30 to the right side with respect to (a) input signal (L1) , just
like the one as illustrated in Figs. 4(a) to 4(c).
24
SP262338WO00
[0052]
As described above, the left image transforming unit
110 as illustrated in Fig. 1 generates and outputs the
parallax-adjusted left image (L2) 50 in which the brightness
5 change portion of the left image (Ll) 10, i.e., the input image,
is shifted in the right direction.
[0053]
Subsequently, the processing of the right image
transforming unit 120 as illustrated in Fig. 1 will be explained.
10 The right image transforming unit 120 has the same
configuration as the left image transforming unit 110.
More specifically, the right image transforming unit
120 includes a right image input unit 121 for receiving the
right image (Rl) 20, a right image differential processing
15 unit 122 for performing differential processing of the right
image 20, a right non-linear transforming unit 123 for
non-linearly transforming a differential signal of the right
image 20, a right image combining unit 124 for combining the
right image 20 and the differential signal non-linearly
20 transformed, and a right image output unit 125 for outputting
the converted parallax-adjusted right image (R2) 60.
[0054]
Basically, the processing of each constituent unit of
the right image transforming unit 120 performs the same
25 processing as the processing of each corresponding constituent
unit of the left image transforming unit 110.
However, the right image transforming unit 120 is
different in coefficients of a differential filter applied
by the right image differential processing unit 122 that
30 performs the differential processing of the right image 20.
The coefficients of the differential filter applied by
25
SP262338W000
the right image differential processing unit 122 have a
sequence obtained by inversing the coefficients of the
differential filter applied by the left image differential
processing unit 112 of the left image transforming unit 110.
5 In other words, the coefficients of the differential filter
applied by the right image differential processing unit 122
have a sequence of an inverse pattern.
[0055]
An example of the right image (R1) 20 which is input
10 to the right image transforming unit 120, the right image
differential signal (HR) generated by the right image
differential processing unit 122, and the combined signal
(parallax-adjusted right image signal (R2) generated by the
right image combining unit 124 will be explained with reference
15 to Figs. 5(a) to 5(c).
The example of the signals as illustrated in Figs. 5 (a)
to 5(c) corresponds to Figs. 4(a) to 4(c) described above as
the example of the signals of the left image transforming unit
110.
20 [0056]
More specifically, Figs. 5(a) to 5(c) illustrate an
example of image combining processing performed by the right
image combining unit 124. The following signals are shown
from the top of Figs. 5(a) to 5(c).
25 (a) input signal (R1)
(b) differential signal (HR)
(c) combined image signal (= parallax-adjusted right
image signal (R2))
[0057]
30 (a) Input signal (RI) denotes brightness change of the
right image (R1) 20, which is input into the image processing
26
SP262338W000
apparatus 100, for any given one horizontal line of any given
frame of video data, for example. This shows an example of
one line in which there is a high brightness region having
high brightness in a central portion.
5 Ina region P from a line position (xl) to a line position
(x2), (a) input signal (R1) shows a change such that the
brightness gradually increases. Between line positions (x2)
to (x3) , there is a high brightness portion in which high level
brightness is maintained. Thereafter, in a region Q from a
10 line position (x3) to a line position (x4), (a) input signal
(Rl) shows a change such that the brightness gradually
decreases.
[0058]
(b) Differential signal (HR) denotes a result of
15 differential of (a) input signal (R1). This differential
signal is a signal generated by the right image differentiating
device 122 of the right image transforming unit 120 as
illustrated in Fig. 1.
It shouldbe noted that, as described above, differential
20 filter coefficients used by the right image differentiating
device 122 is constituted by the coefficient sequence obtained
by inversing the differential filter coefficients used by the
left-image differentiating device 112. More specifically,
as illustrated in Fig. 5(b), the differential filter
25 coefficients used by the right image differentiating device
122 make a differential filter having differential filter
coefficients of the following sequence as illustrated in Fig.
5 (b) .
-1, 0, 1
30 The differential processing performed with the filter
having the differential filter coefficients of the above
27
SP262338WO00
sequence means applying:
a differential signal (In' ) of a pixel (n) in a horizontal
line,
a pixel value of a pixel (n-1) before (at the left of)
5 the pixel (n) (in the present example, brightness) _ (I (n-1)) ,
and
a pixel value of a pixel (n+1) after (at the right of)
the pixel (n) (in the present example, brightness) = (I (n+l) ,
and calculating as follows.
10 In' (I (n-1)) + (I (n+l))
[0059]
As illustrated in Figs. 5 (a) to 5 (c) , the differential
signal (HR) generated by the right image differentiating device
122 has a positive value in the region P in which the brightness
15 change of (a) input right image signal (Rl) is positive, and
the differential signal (HR) generated by the right image
differentiating device 122 has a negative value in the region
Q in which the brightness change of (a) input right image signal
(R1) is negative.
20 [0060]
(c) Combined signal (parallax-adjusted right image
signal (R2)) is a signal generated by the right image combining
unit 124 of the right image transforming unit 120 as illustrated
in Fig. 1. The right image combining unit 124 generates the
25 combined signal (parallax-adjusted right image signal (R2) )
by executing combining processing for adding (a) input right
image signal(R1)and(b) right image differential signal (HR).
As a resul t, the combined signal (parallax-adjusted right image
signal (R2)) as illustrated in Fig. 5(c) is generated.
30 [0061]
As can be understood from this figure, the brightness
28
SP262338W000
change portions P, Q of (a) input signal (R1) are phase-changed
(shifted) in the left direction in (c) combined image signal
(= parallax-adjusted right image signal (R2)). This is the
effect caused by the combining (adding) processing of the
5 differential signal to the input signal (R1).
[0062]
In the explanation of Figs. 5(a) to 5(c), an example
is shown in which the processing of the right image non-linear
transforming unit 123 as illustrated in Fig. 1 is omitted.
10 However, when the processing of the right image non-linear
transforming unit 123 is added, the amplitude of the
differential signal of Fig. 5(b) is controlled (for example,
suppressed).
In this case, (c) combined image signal
15 parallax-adjusted right image signal (R2) , which is the result
ofthe combining processing performed by the right image
combining unit 124, is also a phase-changed (shifted) signal,
i.e., in which the brightness change portion P, Q are moved
to the left side, i.e., in the left direction, with respect
20 to (a) input signal (Rl), just like the one as illustrated
in Figs. 5(a) to 5(c).
[0063]
As described above, the right image transforming unit
120 as illustrated in Fig. 1 generates and outputs the
25 parallax-adjusted left image (R2) 50 in which the brightness
change portion of the right image (R1) 20, i.e., the input
image, is shifted in the left direction.
[0064]
Subsequently, correspondence between the input image
30 and the output image of the image processing apparatus 100
will be explained with reference to Fig. 6.
29
SP262338W000
The following signals patterns are shown from the top
of Fig. 6.
(Ll) input left image signal
(L2) output left image signal (parallax-adjusted left
5 image signal)
(Ri) input right image signal
(R2) output right image signal (parallax-adjusted left
image signal)
These correspond to the signal patterns explained with
10 reference to Figs. 4(a) to 5(c).
[0065]
The input image signal of the image processing apparatus
100 includes the following pair of image signals.
(Ll) input left image signal
15 (Rl) input right image signal
These are signals for displaying a three-dimensional
image, in which parallax is, set in advance.
More specifically, for example, in Fig. 6, a displacement
is set between a displace position of (L1) input left image
20 signal and a display position of (Rl) input right image signal
in the central portion of the brightness change portion [P].
The parallax is generated by this displacement of the display
positions.
[0066]
25 In the explanation below, a distance on a display unit
between the display position of a subject in a left image and
the display position of the same subject in a right image is
defined as "disparity".
The disparity between the input images (L1), (R1) in
30 the central portion of the brightness change portion [P] is
set as follows:
30
SP262338W000
a point in (LI) input left image signal (p1),
a point in (Rl) input right image signal (p2), and
a distance [V1] between these points.
[0067]
5 In contrast, the disparity of the output images, i . e . ,
parallax-adjusted image signals is as follows:
a point in (L2) output left image signal
(parallax-adjusted left image signal) (p3),
a point in (R2) output right image signal
10 (parallax-adjusted left image signal) (p4.), and
a distance [V2] between these points is the disparity.
[0068
As described above, the disparities in the central
portion of the brightness change portion [P] are as follows:
15 the disparity Vl of the pair of the input images, and
the disparity V2 of the output images (parallax-adjusted
image signals),
which are different values.
In the present example, the following expression holds.
20 V2 < VI
The disparity in the output images is reduced as compared
with the disparity of the input images. With this reduction
of the disparity, the parallax is suppressed.
This effect is obtained as an effect of phase change
25 (shift) of each image.
[0069]
The effect of the parallax adjusting process of the image
processing apparatus according to the present embodiment 1
will be explained with reference to Figs. 7 (a) and 7 (b) . Figs.
30 7(a) and 7(b) show the following figures.
(a) parallax range Si of input image
31
SP262338WO00
(b) parallax range S2 of parallax-adjusted image
These two parallax ranges are shown. It should be noted
that the left image and the right image (left eye image and
right eye image) constituting the three-dimensional image
5 includes subjects having various subject distances (distances
from a camera) , and various parallaxes according to the subject
distances are set in the left image and the right image.
[0070]
Figs. 7 (a) and 7 (b) show a left eye 201 and a right eye
10 202 of the observer (user) and a display screen 210 of a display
unit which is executing the three-dimensional image display
process.
An image having various parallaxes according to the
subject distances are presented on the display screen 210.
15 [0071]
It should be noted that a retinal image difference
indicating a positional displacement of images on retinas of
the eyes, i.e., the left eye 201 and the, right eye 202 of the
observer (user) and a distance between the same subjects on
20 the display screen are used as an index value of parallax.
In this case, the distance of the same subjects on the display
screen is defined as disparity, and disparities are compared.
When_.the disparity is zero, the parallax is zero. This is
a case where the display positions, on the display screen 210,
25 of a subject in a left image observed with the left eye 201
of the observer (user) and a right image observed with the
right eye 202 of the observer (user) are located at the same
position.
[0072]
30 More specifically, a point A 221 in the center of the
display screen 210 of Fig. 7 (a) is a subject A at which parallax
32
SP262338W000
= 0 holds, i.e., a subject A at which disparity = 0 holds.
In this case, the user feels as if the subject A is located
on the display screen 210.
[0073]
5 It should be noted that [Aleft/right image] in the figure
means that the display position, on the display screen 210,
of the subject A in the left image for a three-dimensional
image observed with the left eye 201 of the user is at the
same position as the display position, on the display screen
10 210, of the subj ec t Ain the right image for a three--dimensional
image observed with the right eye 202 of the user.
[0074]
On the other hand, the display positions, on the display
screen 210, of the following images as illustrated in Fig.
15 7(a) are different:
a left image B and a right image B, and
a left image C and a right image C.
This means that the display position of the subjects
B, C in the left image for a three-dimensional image observed
20 with the left eye 201 of the user is at a position different
from the display position of the subjects B, C in the right
image for a three-dimensional image observed with the right
eye 202 of the user.
[0075]
25 In such case, the user feels as if the subjects B, C
are located at positions different from the display screen
210.
More specifically, the subject B is observed as if the
subject B is located at a point 222 in the space, which is
30 a crossing point between a line connecting the left eye 201
of the observer and the display position of the left image
33
SP262338WO00
B on the display screen 210 and a line connecting the right
eye 202 of the observe and the display position of the right
image B on the display screen 210.
On the other hand, the subject C is observed as if the
5 subject C is located at a point 223 in the space, which is
a crossing point between a line connecting the left eye 201
of the observer and the display position of the left image
C on the display screen 210 and a line connecting the right
eye 202 of the observe and the display position of the right
10 image C on the display screen 210.
[0076]
As described above, the position of the subject felt
by the observer is set in accordance with the distance
(disparity) of the same subject, on the display screen 210,
15 between the left image and the right image. This will be
explained using a disparity indicating a subject interval
between the left image and the right image on the display screen
210.
The disparity of the subject A is zero.
20 The subject A in the left image and the subject A in
the right image are displayed at the same position on the display
screen 210, and therefore, the disparity is zero.
The position of the subject A realized by the observer
is the position 221 on the display screen 210.
25 [0077]
The disparity of the subject B is DB.
The distance, on the display screen 210, between the
subject B in the left image and the subject B in the right
image is DB, and therefore, the disparity is DB.
30 The position of the subject B realized by the observer
is the position 222 which is farther from the observer than
34
SP262338WO00
display screen 210.
[0078]
The disparity of the subject C is DC.
It should be noted that DB < DC holds.
5 The distance, on the display screen 210, between the
subject C in the left image and the subject C in the right
image is DC, and therefore, the disparity is DC.
The position of the subject C realized by the observer
is the position 223 which is farther from the observer than
10 display screen 210 and is farther from the subject B.
[0079]
When the subjects A, B, C are observed in the images,
the parallax range observed by the observer is a width [Si]
from the point A at the closest display screen position to
15 the point of the subject C which is the farthest in the image.
[00`30]
Fig. 7 (b) illustrates an observation state of an image
displayed using images converted by the image processing
apparatus 100 as illustrated in Fig. 1, i.e., the
20 parallax-adjusted left image (L2) 50 and the parallax-adjusted
right image (R2) 60, for example.
The subjects A2, B2, C2 are the same subjects as A, B,
C of _Fig. 7(a), but as a result of the transform processing
described above, image shift occurs based on the phase change
25 of the image signals, whereby the subject positions observed
by the observer are changed.
[0081]
The disparity of the subject A2 is 0.
The subject A2 in the left image and the subject A2 in
30 the right image are displayed at the same position on the display
screen 210, and therefore, the disparity is zero.
35
SP262338W000
The position of the subject A2 realized by the observer
is a position 231 on the display screen 210.
[0082]
The disparity of the subject B2 is DB2.
5 The distance, on the display screen 210, between the
subject B2 in the left image and the subject B2 in the right
image is DB2, and therefore, the disparity is DB2.
The position of the subject B realized by the observer
is a position 232 which is farther from the observer than display
10 screen 210.
However, the position of the subject B2 realized by the
observer is set at a position closer to the observer (closer
to the display screen 210) than subject position B in the input
image before the conversion as illustrated in Fig. 7(a). This
15 is a result of the above shift processing of the images.
[0093]
The disparity of the subject C2 is DC2.
It should be noted that DB2 < DC2 holds.
The distance, on the display screen 210, between the
20 subject C2 in the left image and the subject C2 in the right
image is DC2, and therefore, the disparity is DC2-
The position of the subject C2 realized by the observer
is a position 233 which is farther from the observer than display
screen 210 and is farther from the subject B2.
25 However, the position of the subject C2 realized by the
observer is set at a position closer to the observer (closer
to the display screen 210) than subject position C in the input
image before the conversion as illustrated in Fig. 7(a). This
is a result of the above shift processing of the images.
30 [0084]
As a result of the image transform processing of the
36
S2262338W000
image processing apparatus 100, for example, the left image
position of the point B slightly moves in the right direction,
and the right image position of the point B slightly moves
in the left direction, so that the disparity is reduced. In
5 other words, the disparity decreases from DB to DB2,
[0085]
The left image position of the point C, which is located
at a further deeper position, greatly moves in the right
direction, and the right image position of the point C greatly
10 moves in the left direction, so that the disparity is further
reduced.
The disparity decreases from DC to DC2.
It should be rioted that the following inequality holds .
(DC-DC2) > (DB-DB2)
15 This means that the reduction width of the disparity
is greater for the deeper subject.
[0086]
As a result, as can be understood by comparing the
parallax ranges Si, S2 of Figs. 7(a) and 7(b),
20 The parallax range Sl (A to C) of the input image before
the conversion change to the parallax range S2 (A2 Lo C2) of
the output image (parallax-adjusted image) after the
conversion.
In other words, in this example, the parallax range S2
25 (A2 to C2) of the output image (parallax-adjusted image) is
reduced as compared with the parallax range S1 (A to C) of
the input image.
[0087]
The parallax suppressing effect explained with
30 reference to Figs- 7 (a) and 7 (b) is the effect that is provided
as a result of the image shift explained previously with
37
SP262338WO00
reference to Figs. 4(a) to 4(c) and 5(a) to 5(c).
In other words, the left image differential processing
unit 113 performs differential calculation using the
coefficients as illustrated in Fig. 4(b), and the differential
5 signal is added to the left image, or the differential signal
is added to the left image after the non-linear processing,
whereby an image of a subject away from the focal position
is displaced in the right direction with respect to the input
image.
10 On the other hand, the 1 right image differential
processing unit 123 performs differential calculation using
the coefficients as illustrated in Fig. 5(b), and the
differential signal is added to the right image, or the
differential signal is added to the right image after the
15 non-linear processing, whereby an image of a subject away from
the, focal position is displaced in the left direction with
respect to the input image..
As a result, the disparity between the left image and
the right image is less than that of the input image, and the
20 parallax range can be reduced, e.g., from the parallax range
S1 of Fig. 7(a) to the parallax range S2 as illustrated in
Fig. 7(b).
[0088]
In the image transform processing executed by the image
25 processing apparatus 100 as illustrated in Fig. 1, the change
of the disparity is the least at a pixel position in focus
(for example, a subject closest to the observer) in the image
serving as the conversion target image, and the farther a point
is away from the focal position, the greater the change of
30 the disparity becomes.
In the example as illustrated in Figs. 7(a) and 7(b),
38
S2262338W000
the point A which is the closest to the observer is in focus,
and there is hardly any change of the disparity at the point
A (which will be referred to as DA) . In the image, it is
considered that, as a subject moves to a deeper side (to A,
5 B, and then to C) , the subject moves away from the focal point,
and this increases blur.
In this case,
The disparities of the subjects A, B, and C before the
image conversion are denoted as DA, DB, and DC, respectively,
10 and
The disparities of the subjects A, B, and C after the
image conversion are denoted as DA2, DB2, and DC2,
respectively.
Then, the change widths of the disparities are set as
15 follows.
(DC2-DC) > (DB2-DB) > (DA2-DA) ^ 0
[0089]
The non-linear transforming units 113, 123 as
illustrated in Fig. 1 use, for example, the non-linear
20 transform processing described above with reference to Fig.
3 to perform processing of raising the low-frequency region
of the differential signal. With this processing, for example,
processing can be performed to raise the level of the
differential signal in a blur region (low-frequency region)
25 which is out of focus. With this processing, the processing
explained with reference to Figs. 7 (a) and 7(b) is enabled,
i.e., control can be performed such that, the change of the
disparity is reduced to the minimum at a pixel position in
focus (for example, a subject closest to the observer) , and
30 as a subject moves away from the focal position, the change
of the disparity is increased.
39
SP262338W000
[0090]
A specific example of correspondence between the input
signal to the image processing apparatus 100, the differential
signal, and the output signal will be explained with reference
5 to Figs. 8 to 11.
Figs. 8 to 11 illustrate lines of the input signal, the
differential signal, and the output signal. Each line
corresponds to a portion of a horizontal line of an image signal.
The vertical axis denotes brightness, and the horizontal axis
10 denotes a pixel position in the horizontal direction.
The input signal is a line representing the brightness
of a ,horizontal line constituting the left image (L1) 10 or
the right image (Rl) 20 which are input to the image processing
apparatus 100.
15 The differential signal is a signal generated by the
differential processing performed by the differentiating
device (the left image differentiating device 112 or the right
image differentiating device 122) of the image processing
apparatus 100.
20 The output signal is a signal generated by combining
processing performed by the combining unit (the left image
combining unit 114 or the right image combining unit 124) of
the image processing apparatus 100.
[0091]
25 The signals indicated in Figs. 8 to 11 correspond to
the following processing examples.
(1) Fig. 8: processing example where differential (in
the present embodiment, the left image differentiating device
112) processing is applied to a high-frequency region
30 constituting an image with differential filter coefficients
[1, 0, -1] (right direction shift)
40
SP262338W000
(2) Fig. 9: processing example where differential (in
the present embodiment, the right image differentiating device
122) processing is applied to a high-frequency region
constituting an image with differential filter coefficients
5 [-1, 0, 1] (left direction shift)
(3a) Fig. 10(a): processing example where differential
(in the present embodiment, the left image differentiating
device 112) processing is applied to a low-frequency region
constituting an image with differential filter coefficients
10 [1, 0, -1] (right direction shift)
(3b) Fig. 10(b): processing example where differential
(in the present embodiment, the right image differentiating
device 122) processing is applied to a low-frequency region
constituting an image with differential filter coefficients
15 [-1, 0, 1] (left direction shift)
(4a) Fig. 11(a): processing example where differential
(in the present embodiment, the left image differentiating
device 112) processing is applied to a low-frequency region
constituting an image with differential filter coefficients
20 [1, 0, -1] , and further, the differential signal is amplified
using non-linear transform (right direction shift)
(4b) Fig. 11(b): processing example where differential
(in the present embodiment, the right image differentiating
device 122) processing is applied to a low-frequency region
25 constituting an image with differential filter coefficients
[-1, 0, 1] , and further, the differential signal is amplified
using non-linear transform (left direction shift)
[0092]
Fig. 8 illustrates processing example where
30 differential (in the present embodiment, the left image
differentiating device 112) processing is applied to a
41
SP262338W000
high-frequency region constituting an image with differential
filter coefficients [1, 0, -1].
A differential signal (b) is generated by differential
processing performed on the input signal (a), and an output
5 signal (c) is generated by combining processing of the input
signal and the differential signal.
As can be understood from the figure, the output signal
(c) is shifted in the right direction with respect to the input
signal (a).
10 [0093]
Fig. 9 is processing example where differential (in the
present embodiment, the right image differentiating device
122) processing is applied to a high-frequency region
constituting an image with differential filter coefficients
15 [-1, 0, 1].
A differential signal (b) is generated by differential
processing performed on the input signal (a) , and an output
signal (c) is generated by combining processing of the input
signal and the differential signal.
20 As can be understood from the figure, the output signal
(c) is shifted in the left direction with respect to the input
signal (a).
[0094]
Fig. 10(a) is processing example where differential (in
25 the present embodiment, the left image differentiating device
112) processing is applied to a low-frequency region
constituting an image with differential filter coefficients
[1, 0, -1] .
As can be understood from the figure, the output signal
30 (c) is shifted in the right direction with respect to the input
signal (a).
42
SP262338WO00
Fig. 10(b) is processing example where differential (in
the present embodiment, the right image differentiating device
122) processing is applied to a low-frequency region
constituting an image with differential filter coefficients
5 [-l, 0, 1].
As can be understood from the figure, the output signal
(c) is shifted in the left direction with respect to the input
signal (a).
[0095]
10 Fig. 11(a) is processing example where differential (in
the present embodiment, the left image differentiating device
112) processing is applied to a low-frequency region
constituting an image with differential filter coefficients
[1, 0, -1] , and further, the differential signal is amplified
15 using non-linear transform.
As can be understood from the figure, the output signal
(c) is shifted in the right direction with respect to the input
signal (a).
Fig. 11(b) is processing example where differential (in
20 the present embodiment, the right image differentiating device
122) processing is applied to a low-frequency region
constituting an image with differential filter coefficients
[-1, 1], and further, the differential signal is amplified
using non-linear transform.
25 As can be understood from the figure, the output signal
(c) is shifted in the left direction with respect to the input
signal (a).
[0096]
The non-linear transforming units 113, 123 as
30 illustrated in Fig. 1 use, for example, the non-linear
transform processing described above with reference to Fig.
43
SP262338W000
3 to perform processing of raising the low-frequency region
of the differentialsignal. With this processing, forexample,
processing can be performed to raise the level of the
differential signal in an out-of-focus low-frequency region.
5 With this processing, forexample, processing can be performed
to increase the amount of shift (the amount of change of phase)
in the low-frequency region.
More specifically, in accordance with the setting for
the aspect of the transform processing performed by the
10 non-linear transforming units 113, 123, control can be achieved
to increase or decrease the different in the amount of shift
between the low-frequency region and the high-frequency
region.
[0097]
15 [B. Embodiment 2: Example of enlarging processing of
parallax range set at a deeper side with respect to a display
surface of a display unit]
Subsequently, an embodiment 2 of an image processing
apparatus according to the present invention will be explained.
20 The image processing apparatus according to the
embodiment 2 has the same configuration as the image processing
apparatus 100 as illustrated in Fig. 1 explained in the
embodiment 1.
However, the embodiment 2 has such configuration that
25 the differential filter applied by the left image
differentiating device 112 of the left image transforming unit
110 and the differential filter applied by the right image
differentiating device 122 of the right image transforming
unit 120 are interchanged as compared with the case of the
30 embodiment 1.
As a result of this change of the differential filters,
44
SP262338W000
the image shift directions of the left image and the right
image are opposite directions. As a result, although the
reducing processing of the parallax range is performed in the
embodiment 1, processing to enlarge the parallax range is
5 performed in the present embodiment.
[0098]
Correspondence between the input image and the output
image of the image processing apparatus 100 according to the
present embodiment will be explained with reference to Fig.
10 12. Fig. 12 is the same figure as Fig. 6 explained in the
embodiment 1. The following signal patterns are shown from
the top of Fig. 12.
(Ll) input left image signal
(L2) output left image signal (parallax-adjusted left
15 image signal)
(R1) input right image signal
(R2) output right image signal (parallax-adjusted left
image signal)
[0099]
20 The input image signal of the image processing apparatus
100 includes the following pair of image signals.
(L1) input left image signal
(R1) input right image signal
These are signals for displaying a three-dimensional
25 image, in which parallax is set in advance.
More specifically, for example, in Fig. 12, a
displacement (disparity) is set between a displace position
of (L1) input left image signal and a display position of (R1)
input right image signal in the central portion of a brightness
30 change portion [P]. The parallax is generated by this
displacement of the display positions.
45
SP262338W000
[0100]
The disparity between the input images (L1), (R1) in
the central portion of the brightness change portion [P] is
set as follows:
5 a point in (Ll) input left image signal (pl),
a point in (Rl) input right image signal (p2), and
a distance [V1] between these points.
[0101]
In contrast, the disparity of the output images, i.e.,
10 parallax-adjusted image signals is as follows:
a point in (L2) output left image signal
(parallax-adjusted left image signal) (p5),
a point in (R2) output right image signal
(parallax-adjusted left image signal) (p6), and
15 a distance [V3] between these points is the disparity.
[0102]
In the present embodiment, the differential filter
applied by the left image differentiating device 112 of the
left image transforming unit 110 make a differential filter
20 having differential filter coefficients of the following
sequence as illustrated in Fig. 12 (L2).
-1, 0, 1
More specifically, in the embodiment 1 described above,
it is the differential filter suitable for the right image
25 differentiating device 122 of the right image transforming
unit 120.
The differential processing performed with the filter
having the differential filter coefficients of the above
sequence means applying:
30 differential data (In') of a pixel (n) in a horizontal
line,
46
SP262338WO00
a pixel value of a pixel (n-1) before (at the left of)
the pixel (n) (in the present example, brightness) _ (I (n-1)) ,
and
a pixel value of a pixel (n+l) after (at the right of)
5 the pixel (n) (in the present example, brightness) _ (I (n+l)) ,
and calculating as follows.
In' = - (I (n-1)) + (I (n+l))
[0103]
This differential signal is the same signal as the
10 differential signal explained with reference to Fig. 5(b) in
the embodiment 1. As a result, this combining (adding)
processing result of the differential signal and the original
input signal is the output left image signal (parallax-adjusted
left image (L2)) as illustrated in Fig. 12 (L2) obtained by
15 shifting the brightness change portions P, Q of (L1) input
signal of Fig. 12 in the left direction.
[0104]
On the other hand, the differential filter applied by
the right image differentiating device 122 of the right image
20 transforming unit 120 make a differential filter having
differential filter coefficients of the following sequence
as illustrated in Fig. 12(b) (R2).
1, 0, -1
It is the differential filter suitable having the
25 differential filter coefficients of the above sequence.
More specifically, in the embodiment 1 described above,
it is the differential filter suitable for the left image
differentiating device 112 of the left image transforming unit
110.
30 The differential processing performed with the filter
having the differential filter coefficients of the above
47
SP262338W000
sequence means applying:
differential data (In') of a pixel (n) in a horizontal
line,
a pixel value of a pixel (n-1) before (at the left of)
5 the pixel (n) (in the present example, brightness) = (I (n-1)) ,
and
a pixel value of a pixel (n+1) after (at the right of)
the pixel (n) (in the present example, brightness) = (I (n+1)) ,
and calculating as follows.
10 In' _ (I (n-1)) - (I (n+1))
[0105]
This differential signal is the same signal as the
differential signal explained with reference to Fig. 4 (b) in
the embodiment 1. As a result, this combining (adding)
15 processing result of the differential signal and the original
input signal is the output right image signal
(parallax-adjusted left image (R2) ) as illustrated in Fig.
12 (R2) obtained by shifting the brightness change portions
P, Q of (R1) input signal of Fig. 12 in the right direction.
20 Although the non-linear transform processing is omitted
in the explanation, the shift direction is set in the same
manner even when the non-linear transform processing is
performed.
[0106]
25 In the embodiment 2, as described above, processing is
performed while the differential filters applied by the
differentiating devices for the left and right images according
to the embodiment 1 are interchanged, so that the embodiment
2 is configured such that the shift directions of the left
30 image and the right image are the opposite directions.
[0107]
48
SP262338W000
As a result, as illustrated in Fig. 12, the disparities
in the central portion of the brightness change portion [P]
are as follows.
The disparity V1 of the pair of input images, and
5 The disparity V3 of the output images (parallax-adjusted
image signals),
which are different values.
In the present example,
VI < V3
10 holds, and the disparity V3 in the output images is
increased as compared with the disparity Vl of the input images.
With this increase of the disparity, the parallax is amplified.
This effect is obtained as an effect of shift of each
image.
15 [0108]
The effect of the parallax adjusting process of the image
processing apparatus according to the present embodiment 2
will be explained with reference to Figs. 13(a) and 13(b).
Figs. 13(a) and 13(b) show the following figures.
20 (a) parallax range Si of input image
(b) parallax range S3 of parallax-adjusted image
Like Figs. 7(a) and 7(b) of the embodiment 1 described
above, Figs. 13(a) and 13(b) show a left eye 201 and a right
eye 202 of the observer (user) and a display screen 210 of
25 a display unit which is executing the three-dimensional image
display process.
An image having various parallaxes according to the
subject distances are presented on the display screen 210.
[0109]
30 In this explanation, disparity which is a distance of
the same subjects on the display screen is used as an index
49
SP262338WO00
value of parallax.
Fig. 13 (a) is a figure corresponding to an input image
before the parallax-adjusting process, and is the same figure
as Fig. 7(a). In Fig. 13(a),
5 The disparity of the subject A is zero,
The disparity of the subject B is DB, and
The disparity of the subject C is DC.
The following expression holds.
0 < DB < DC
10 The observation position of the subject A realized by
the user is on the display. The observation position of the
subject B realized by the user is deeper than display (farther
from the observer) . The observation position of the subject
C realized by the user is at a position still farther than
15 that of the subject B.
The parallax range observed by the observer is a width
[S1] from the point A at the closest display screen position
to the point C which is the farthest subject observation
position.
20 [0110]
Fig. 13 (b) illustrates an observation state of an image
displayed using images converted by the image processing
apparatus 100 as illustrated in Fig. 1 according to the present
embodiment 2, i.e., the parallax-adjusted left image (L2) 50
25 and the parallax-adjusted right image (R2) 60, for example.
The subjects A3, B3, C3 are the same subjects as A, B,
C of Fig. 13(a), but with the transform processing described
above, image shift is executed, whereby the subject positions
observed by the observer are changed.
30 [0111]
The disparity of the subject A3 is 0.
50
SP262338W000
The subject A3 in the left image and the subject A3 in
the right image are displayed at the same position on the display
screen 210, and therefore, the disparity is zero.
The position of the subject A3 realized by the observer
5 is a position 241 on the display screen 210.
[0112)
The disparity of the subject B3 is DB3.
The distance, on the display screen 210, between the
subject 33 in the left image and the subject B3 in the right
10 image is DB3, and therefore, the disparity is DB3..
The position of the subject B3 realized by the observer
is a position 242 which is farther from the observer than display
screen 210.
However, the position of the subject B3 realized by the
15 observer is set at a position farther from the observer (farther
from the display screen 210) than subject position B in the
input image before the conversion as illustrated in Fig. 13(a).
This is a result of the above shift processing of the images.
[0113]
20 The disparity of the subject C3 is DC3.
It should be noted that DB3 < DC3 holds.
The distance, on the display screen 210, between the
subject C3 in the left image and the subject C3 in the right
image is DC3, and therefore, the disparity is DC3.
25 The position of the subject C3 realized by the observer
is a position 243 which is farther fromthe observer than display
screen 210 and is farther from the subject B2.
However, the position of the subject C3 realized by the
observer is set at a position farther from the observer (farther
30 from the display screen 210) than subject position C in the
input image before the conversion as illustrated in Fig. 13(a).
51
SP262338W000
This is a result of the above shift processing of the images.
[0114]
In the present embodiment 2, the image transform
processing is performed by the image processing apparatus 100
5 so that the image shift is executed in the direction opposite
to that of the embodiment 1 described above.
More specifically, the left image position of the point
B slightly moves in the left direction, and the right image
position thereof slightly moves in the right direction, so
10 that the disparity is increased. In other words, the disparity
increases from DB to DB3.
[0115]
The left image position of the point C, which is located
still deeper, greatly moves in the left direction, and the
15 right image position thereof greatly moves in the right
direction, so that the disparity is further increased. In
other words, the disparity increases from DC to DC3.
It should be noted that the following expression holds.
(DC3-DC) > (DB3-DB)
20 This means that the enlargement width is larger for the
disparity of the deeper subject.
[0116]
As a result, as can be understood by comparing the
parallax ranges S1, S3 of Figs. 13(a) and 13(b),
25 The parallax range Sl (A to C) of the input image before
the conversion change to the parallax range S3 (A3 to C3) of
the output image (parallax-adjusted image) after the
conversion.
In other words, in this example, the parallax range S3
30 (A3 to C3) of the output image (parallax-adjusted image) is
enlarged as compared with the parallax range Sl (A to C) of
52
SP262338WO00
the input image.
[0117]
The parallax enlarging effect explained with reference
to Figs. 13(a) and 13(b) is the effect that is provided as
5 a result of the image shift explained previously with reference
to Fig. 12.
In other words, the left image differential processing
unit 113 performs differential calculation using the
coefficients as illustrated in Fig. 12 (L2), and the
10 differential signal is added to the left image, or the
differential signal is added to the left image after the
non-linear processing, whereby an image of a subject away from
the focal position is displaced in the left direction with
respect to the input image.
15 On the other hand, the right image differential
pro-essing unit 123 performs differential calculation using
the coefficients as illustrated in Fig. 12 (R2), and the
differential signal is added to the right image, or the
differential signal is added to the right image after the
20 non-linear processing, whereby an image of a subject away from
the focal position is displaced in the right direction with
respect to the input image.
As a result, the disparity between the left image and
the right image is more than that of the input image, and the
25 parallax range can be enlarged, e.g., from the parallax range
S1 of Fig. 13 (a) to the parallax range S3 as illustrated Fig.
13(b).
[0118]
In the image transform processing executed by the image
30 processing apparatus 100, as described above, the change of
the disparity is the least at a pixel position in focus (for
53
SP262338W000
example, a subject closest to the observer) in the image serving
as the conversion target image, and the farther a point is
away from the focal position, the greater the change of the
disparity becomes.
5 In the situation as illustrated in Fig. 13, the point
A at the closest position is in focus, and the disparity of
the point A hardly changes in view of the fact that the subject
at a deeper position has greater subject blur.
[0119]
10 When the point A at the closest position is in focus,
it is considered that, in the image, as a subject moves to
a deeper side (to A, B, and then to C) , the subject moves away
from the focal point, and this increases blur.
In this case,
15 The disparities of the subjects A, B, and C before the
image conversion are denoted as DA, DB, and DC, respectively,
and
The disparities of the. subjects A, B, and C after the
image conversion are denoted as DA3, DB3, and DC3,
20 respectively.
Then, the change widths of the disparities are set as
follows.
(DC3-DC) > (DB3-DB) > (DA3-DA) ^0
[0120]
25 [C. Embodiment 3: Example of control
(reducing/enlarging) processing of parallax range set at a
closer side with respect to a display surface of a display
unit]
In the embodiment 1 and the embodiment 2, the example
30 of processing where the image for three-dimensional display
which is input to the image processing apparatus is an image
54
SP262338W000
in which parallax setting of the left image (L1) 10 and the
right image (R1) 20 is set at a deeper side of the display
surface of the display unit, i.e., an image in which the parallax
setting of the left image (L1) 10 and the right image (R1)
5 20 is set at a farther position from the observer has been
explained. In other words,
In the embodiment 1, the example of processing for
reduction of the parallax range set at a deeper side of the
display surface of the display unit has been explained.
10 In the embodiment 2, the example of processing for
enlarging of the parallax range set at a deeper side of the
display surface of the display unit has been explained.
[0121]
However, images applied to display of three-dimensional
15 images are not limited to the images explained in the
embodiments 1, 2. There is an image in which parallax is set
so that a subject protrudes to a closer side of the display
surface of the display unit, i.e., to a position closer to
the observer.
20 Hereinafter, an example of processing where an input
image subjected to parallax control is a three-dimensional
image in which an observation position of a subject is set
at a_closer side (in an observer direction ) with respect to
the display surface will be explained as the embodiment 3.
25 In the embodiment 3(a), an example of processing for
reduction of a parallax range set at a side closer than display
surface of the display unit will be explained.
In the embodiment 3(b), an example of processing for
enlarging of a parallax range set at a side closer than display
30 surface of the display unit will be explained.
These examples of processing will be explained in order.
55
SP262338W000
It should be noted that the present embodiment 3 is
achieved using the same configuration as the image processing
apparatus 100 as illustrated in Fig. 1 explained in the
embodiment 1.
5 [0122]
(a) Example of processing for reduction of a parallax
range set at a side closer than display surface of the display
unit
First, the example of processing for reduction of a
10 parallax range set at a side closer than display surface of
the display unit will be explained with reference to Figs.
14(a) and 14(b).
Figs. 14(a) and 14(b) show the following figures.
(a) parallax range Sl of input image
15 (b) parallax range S4 of parallax-adjusted image
Figs. 14(a) and 14(b) further show a left eye 201 and
a right eye 202 of the observer (user) and a display screen
210 of a display unit which is executing the three-dimensional
image display process.
20 An image having various parallaxes according to the
subject distances are presented on the display screen 210.
[0123]
Like the embodiments 1, 2 described above, disparity
which is a distance of the same subjects on the display screen
25 is used as an index value of parallax in this explanation.
Fig. 14 (a) is a figure corresponding to an input image
before the parallax-adjusting process. In Fig. 14(a),
The disparity of the subject E is DE which is zero,
The disparity of the subject F is DF, and
30 The disparity of the subject G is DG.
The magnitudes of the disparities satisfy the following
56
SP262338W000
relationship.
DG > DF > DE ^ 0
[0124]
The observation position of the subject E realized by
5 the user is on the display. The observation position of the
subject F realized by the user is closer than display (closer
to the observer) . The observation position of the subject
G realized by the user is at a position still closer than that
of the subject F.
10 The parallax range observed by the observer is a width
[Si] from the point E at the farthest display screen position
to the point G which is the closest subject observation
position.
[0125]
15 As described above, the subject is observed as if the
subject is located at a point in the space, which is a crossing
point between a line connecting the left eye 201 of the observer
and the display position of the left image on the display screen
210 and a line connecting the right eye 202 of the observe
20 and the display position of the right image on the display
screen 210.
In the present embodiment, the images of the subjects,
i.e., the left image and the right image, (E to G right images
and E to G left images) are set so that the crossing point
25 is set at a position closer than display screen 210 (a direction
closer to the observer).
When the three-dimensional image with such setting is
shown, the observer realizes, as illustrated in Fig. 14(a),
the subjects located closer to the observer than display screen
30 210.
[0126]
57
SP262338W000
As described above, the parallax range of the input image
as illustrated in Fig. 14(a) is a width [Sl] from the point
E at the farthest display screen position to the point G which
is the closest subject observation positions
[0127]
Fig. 14 (b) illustrates an observation state of an image
displayed using images converted by the image processing
apparatus 100 as illustrated in Fig. 1 according to the present
embodiment 3(a), i.e., the parallax-adjusted left image (L2)
10 50 and the parallax-adjusted right image (R2) 60, for example.
The subjects E4, F4, G4 are the same subjects as E, F,
G of Fig. 14 (a) , but with the transform processing described
above, image shift is executed, whereby the subject positions
observed by the observer are changed.
15 [0128]
The disparity of the subject E4 is 0.
The subject E4 in the left image and the subject E4 in
the right image are displayed at the same position on the display
screen 210, and therefore, the disparity is zero.
20 The position of the subject E4 realized by the observer
is a position on the display screen 210.
[0129)
The disparity of the subject F4 is DF4.
The distance, on the display screen 210, between the
25 subject F4 in the left image and the subject F4 in the right
image is DF4, and therefore, the disparity is DF4.
The position of the subject F4 realized by the observer
is a position closer to the observer than display screen 210,
i.e., the position F4 at the side of the observer.
30 However, the position of the subject F4 realized by the
observer is set at a position farther from the observer (closer
58
SP262338WO00
to the display screen 210) than subject position F in the input
image before the conversion as illustrated in Fig. 14 (a) . This
is a result of the above shift processing of the images.
[0130]
5 The disparity of the subject G4 is DG4.
It should be noted that DF4 < DG4 holds.
The distance, on the display screen 210, between the
subject G4 in the left image and the subject G4 in the right
image is DG4, and therefore, the disparity is DG4.
10 The position of the subject G4 realized by the observer
is a position closer to the observer than display screen 210,
i.e., the position G4 at the side of the observer.
However, the position of the subject G4 realized by the
observer is set at a position farther from the observer (closer
15 to the display screen 210) than subject position Gin the input
image before the conversion as illustrated in Fig. 14 (a) . This
is a result of the above shift processing of the images.
[0131]
In the present embodiment 3(a), the image transform
20 processing is performed by the image processing apparatus 100
so that the image shift is executed in the same direction as
that of the embodiment 2 described above.
More specifically, the same image shift processing as
that explained with reference to Fig. 12 is executed. That
25 is, the following image shift processing is executed:
for the left image (Ll) 10, the shift processing is
performed in the left direction, and
for the right image (R1) 20, the shift processing is
performed in the right direction.
30 [0132]
As a result, the position F of the left image (Fig. 14(a) ,
59
SP262338W000
F, left image) slightly moves in the left direction to the
position F4 of the left image in Fig. 14 (b) , and the position
F of the right image (Fig. 14(a), F, right image) slightly
moves in the right direction to the position F4 of the right
5 image in Fig. 14(b). Accordingly, the disparity decreases.
In other words, the disparity decreases from DF to DF4.
[0133]
At the position G which is located still closer to the
observer (observer side), the position G of the left image
10 (Fig. 14 (a) , G, left image) greatly moves in the left direction
to the position G4 of the left image in Fig. 14(b), and the
position G of the right image (Fig. 14(a), G, right image)
greatly moves in the right direction to the position G4 of
the right image in Fig. 14(b). With this movement, the
15 disparity becomes still more smaller.
The disparity decreases from DG to DG4.
It should be noted that, the following expression holds.
(DG-DG4) > (DF-DF4)
This means that the reducing width is larger for the
20 disparity of the subject closer to the observer (farther from
the display screen 210).
[0134]
As a result, as can be understood by comparing the
parallax ranges S1, S4 of Figs. 14(a) and 14(b),
25 The parallax range S1 (E to G) of the input image before
the conversion change to the parallax range S4 (E4 to G4) of
the output image (parallax-adjusted image) after the
conversion.
In other words, in this example, the parallax range S4
30 (E4 to G4) of the output image (parallax-adjusted image) is
reduced as compared with the parallax range S1 (E to G) of
60
SP262338WO00
the input image.
[0135]
The parallax range reducing effect explained with
reference to Figs. 14 (a) and 14 (b) is thee ffect that is provided
5 as a result of the image shift performed by the image processing
apparatus 100.
In the present embodiment 3(a), the left image
differentiating device 112 executes differential processing
like the embodiment 2. More specifically, the left image
10 differentiating device 112 executes the differential
processing by applying the following differential filter
coefficients as illustrated in Fig. 12 (L2),
[-1, 0, 1]
With this differential processing, the left image is
15 shifted in the left direction as illustrated in Fig. 12 (L1),
(L2).
On the other hand, the right image differentiating device
122 executes the differential processing by applying the
following differential filter coefficients as illustrated in
20 Fig. 12 (R2),
[1, 0, -1]
With this differential processing, the right image is
shifted in the right direction as illustrated in Fig. 12 (R1) ,
(R2).
25 [0136]
As illustrated in Figs. 14 (a) and 14 (b) , the left image
differential processing unit 113 performs the differential
calculation using the coefficients as illustrated in Fig. 12
(L2), and using the combining processing in which the
30 differential signal or the non-linear processing result of
the differential signal is added to the left image, a converted
61
SP262338W000
image is generated in which the image is displaced in the left
direction with respect to the input image.
On the other hand, the right image differential
processing unit 123 performs the differential calculation
5 using the coefficients as illustrated in Fig. 12 (R2), and
using the combining processing in which the differential signal
is added to the right image or the differential signal is added
to the right image of ter the non-linear processing, a converted
image is generated in which the image is displaced in the right
10 direction with respect to the input image.
As a result, the disparity between the left image and
the right image is less than that of the input image, and the
parallax range can be reduced, e.g., from the parallax range
Si of Fig. 14(a) to the parallax range S4 as illustrated in
15 Fig. 14(b).
[0137]
The present embodiment is also configured such that the
change of the disparity is the least at a pixel position in
focus (subject E) in the image serving as the conversion target
20 image, and the farther a point is away from the focal position
(to E, F, and then to G) , the greater the change of the disparity
becomes.
[01381
(b) Example of processing for enlarging of a parallax
25 range set at a side closer than display surface of the display
unit
Subsequently, the example of processing for enlarging
of a parallax range set at a side closer than display surface
of the display unit will be explained with reference to Figs.
30 15(a) and 15(b).
Figs. 15(a) and 15(b) show the following figures.
62
SP262338W000
(a) parallax range Si of input image
(b) parallax range S4 of parallax-adjusted image
Figs. 15(a) and 15(b) further show a left eye 201 and
a right eye 202 of the observer (user) and a display screen
5 210 of a display unit which is executing the three-dimensional
image display process.
An image having various parallaxes according to the
subject distances are presented on the display screen 210.
[0139]
10 Like the embodiments described above, disparity which
is a distance of the same subjects on the display screen is
used as an index value of parallax in this explanation.
Fig. 15 (a) is a figure corresponding to an input image
before the parallax-adjusting process, and is the same figure
15 as Fig. 14(a). In Fig. 15(a),
The disparity of the subject E is DE which is zero,
The disparity of the subject F is DF, and
The disparity of the subject G is DG.
The magnitudes of the disparities satisfy the following
20 relationship.
DG > DF > DE ^ 0
[0140]
The observation position of the subject E realized by
the user is on the display. The observation position of the
25 subject F realized by the user is closer than display (closer
to the observer). The observation position of the subject
G realized by the user is at a position still closer than that
of the subject F.
[0141]
30 This setting as illustrated in Fig. 15(a) is the same
setting as that of Fig. 14 (a) described above, and the observer
63
SP262338W000
realizes each subject at a position closer than display screen
210.
The parallax range of the input image as illustrated
in Fig. 15 (a) is a width [Si] from the point E at the farthest
5 display screen position to the point G which is the closest
subject observation position.
[0142]
Fig. 15 (b) illustrates an observation state of an image
displayed using images converted by the image processing
10 apparatus 100 as illustrated in Fig. 1 according to the present
embodiment 3(b), i.e., the parallax-adjusted left image (L2)
50 and the parallax-adjusted right image (R2) 60, for example.
The subjects E5, F5, G5 are the same subjects as E, F,
G of Fig. 15(a), but with the transform processing described
15 above, image shift is executed, whereby the subject positions
observed by the observer are changed.
[0143]
The disparity of the subject ES:is 0.
The subject E5 in the left image and the subject E5 in
20 the right image are displayed at the same position on the display
screen 210, and therefore, the disparity is zero.
The position of the subject ES realized by the observer
is a..position on the display screen 210.
[0144]
25 The disparity of the subject F5 is DFS.
The distance, on the display screen 210, between the
subject F5 in the left image and the subject F5 in the right
image is DF5, and therefore, the disparity is DFS.
The position of the subject F5 realized by the observer
30 is a position closer to the observer than display screen 210,
i.e., the position F5 at the side of the observer.
64
SP262338W000
However, the position of the subject F5 realized by the
observer is set at a position closer to the observer (farther
from the display screen 210) than subject position F in the
input image before the conversion as illustrated in Fig. 15 (a) .
5 This is a result of the above shift processing of the images.
[0145]
The disparity of the subject G5 is DG5.
It should be noted that DF5 < DG5 holds.
The distance, on the display screen 210, between the
10 subject G5 in the left image and the subject G5 in the right
image is DG5, and therefore, the disparity is DG5.
The position of the subject G5 realized by the observer
is a position closer to the observer than display screen 210,
i.e., the position G5 at the side of the observer.
15 However, the position of the subject G5 realized by the
observer is set at a position closer to the observer (farther
from the display screen 210), than subject position G in the
input image before the conversion as illustrated in Fig. 15 (a) .
This is a result of the above shift processing of the images.
20 [0146]
In the present embodiment 3(b), the image transform
processing is performed by the image processing apparatus 100
so that the image shift is executed in the same direction as
that of the embodiment 1 described above.
25 More specifically, the same image shift processing as
that explained with reference to Fig. 6 is executed. That
is, the following image shift processing is executed°
for the left image (L1) 10, the shift processing is
performed in the right direction, and
30 for the right image (RI) 20, the shift processing is
performed in the left direction.
65
SP262338W000
[0147]
Asa result, the position F of the left image (Fig. 15 (a),
F, left image) slightly moves in the right direction to the
position F5 of the left image in Fig. 15 (b) , and the position
5 F of the right image (Fig. 15(a), F, right image) slightly
moves in the left direction to the position F5 of the right
image in Fig. 15(b). Accordingly, the disparity increases.
In other words, the disparity increases from DF to DF5.
[0148]
10 At the position G which is located still closer to the
observer (observer side), the position G of the left image
(Fig. 15 (a) , G, left image) greatlymoves in the right direction
to the position G5 of the left image in Fig. 15(b), and the
position G of the right image (Fig. 15(a), G, right image)
15 greatly moves in the left direction to the position G5 of the
right image in Fig. 15(b). Accordingly, the disparity
increases. In other words, the disparity increases from DG
to DGS.
It should be noted that the following expression holds.
20 (DG5-DG) > (DF5-DF)
This means that the enlarging width is larger for the
disparity of the subject closer to the observer (farther from
the display screen 210).
[0149]
25 As a result, as can be understood by comparing the
parallax ranges S1, S5 of Figs. 15(a) and 15(b),
The parallax range S1 (E to G) of the input image before
the conversion change to the parallax range S5 (ES to G5) of
the output image (parallax-adjusted image) after the
30 conversion.
In other words, in this example, the parallax range S5
66
SP262338WO00
(E5 to G5) of the output image (parallax-adjusted image) is
enlarged as compared with the parallax range Si (E to G) of
the input image.
[0150]
5 The parallax range reducing effect explained with
reference to Figs. 15 (a) and 15 (b) is the effect that is provided
as a result of the image shift performed by the image processing
apparatus 100.
In the present embodiment 3(b), the same differential
10 processing as that of the embodiment 1 explained previously
is performed, and the same image shift processing as that
explained with reference to Fig. 6 is executed. That is, the
following image shift processing is executed:
for the left image (L1) 10, the shift processing is
15 performed in the left direction, and
for the right image (Rl) 20, the shift processing is
performed in the right direction.
[0151]
In the present embodiment 3(b), the left image
20 differentiating device 112 executes the differential
processing by applying the following differential filter
coefficients as illustrated in Fig. 6 (L2),
_ [1, 0, -1]
With this differential processing, the left image is
25 shifted in the right direction as illustrated in Fig. 6 (Ll),
(L2).
On the other hand, the right image differentiating device
122 executes the differential processing by applying the
following differential filter coefficients as illustrated in
30 Fig. 6 (R2),
[-1, 0, 1]
67
SP262338WO00
With this differential processing, the right image is
shifted in the left direction as illustrated in Fig. 6 (RI),
(R2) .
[0152]
5 As illustrated in Fig. 6, the left image differential
processing unit 113 performs the differential calculation
using the coefficients as illustrated in Fig. 6 (L2) , and using
the combining processing in which the differential signal or
the non-linear processing result of the differential signal
10 is added to the left image, a converted image is generated
in which the image is displaced in the right direction with
respect to the input image.
On the other hand, the right image differential
processing unit 123 performs the differential calculation
15 using the coefficients as illustrated in Fig. 6 (R2) , and using
the combining processing in which the differential signal is
added to the right image or the differential signal is added
to the right image after the non-linear processing, a converted
image is generated in which the image is displaced in the left
20 direction with respect to the input image.
As a result, the disparity between the left image and
the right image is larger than that of the input image, and
the parallax range can be enlarged, e.g., from the parallax
range Sl of Fig. 15 (a) to the parallax range S5 as illustrated
25 in Fig. 15(b).
[0153]
The present embodiment is also configured such that the
change of the disparity is the least at a pixel position in
focus (subject E) in the image serving as the conversion target
30 image, and the farther a point is away from the focal position
(to E, F, and then to G) , the greater the change of the disparity
68
S2262338WO00
becomes.
[0154]
[D. Embodiment 4: Example of control processing of
parallax range set at either side of a display surface of a
5 display unit]
Subsequently, the example of control processing of
parallax range set at either side of the display surface of
the display unit will be explained.
In the embodiments 1 and 2, the example of processing
10 performed on the image in which the parallax range is set at
a deeper side of the display surface has been explained. In
the embodiment 3, the example of processing performed on the
image in which the parallax range is set at a closer side of
the display surface has been explained. As described above,
15 the setting of the parallax range can be set in various forms
by changing the subject positions in the left image and the
right image. Therefore, an image in which the parallax range
is set at either side of the display surface of the display
unit can also be generated. Hereinafter, an example of
20 configuration forperforming the parallax control by inputting
such image in which the parallax range is set at either side
of the display surface of the display unit will be explained.
It should be noted that the present embodiment 4 is
achieved using the same configuration as the image processing
25 apparatus 100 as illustrated in Fig. 1 explained in the
embodiment 1.
[0155]
In the embodiment 4, the following two examples of
processing will be explained like the embodiment 3.
30 (a) Example of processing for shifting a left image in
a right direction and shifting a right image in a left direction
69
SP262338WO00
(b) Example of processing for shifting a left image in
a left direction and shifting a right image in a right direction
These examples of processing will be explained in order.
[0156]
5 (a) Example of processing for shifting a left image in
a right direction and shifting a right image in a left direction
First, the example of processing for shifting the left
image (L1) 10, which is input into the image processing
apparatus 100, in the right direction and shifting the right
10 image (R1) 20 in the left direction will be explained with
reference to Fig. 16.
In this example of processing, processing corresponding
to the processing as illustrated in Fig. 6 explained in the
embodiment 1 described above is executed.
15 More specifically, the following image shift processing
is :executed:
for the left image (Ll) 10, the shift processing is
performed in the right direction, and,
for the right image (Ri) 20, the shift processing is
20 performed in the left direction.
[0157]
In the present embodiment 4(a), the left image
differentiating device 112 executes the differential
processing by applying the following differential filter
25 coefficients as illustrated in Fig. 6 (L2),
[1, 0, -1]
With this differential processing, the left image is
shifted in the right direction as illustrated in Fig. 6 (Ll) ,
(L2).
30 On the other hand, the right image differentiatingdevice
122 executes the differential processing by applying the
70
SP262338W000
following differential filter coefficients as illustrated in
Fig. 6 (R2),
[-1, 0, 1]
With this differential processing, the right image is
5 shifted in the left direction as illustrated in Fig. 6 (Rl),
(R2).
[0158]
An example of parallax control processing according to
the present embodiment 4 (a) will be explained with reference
10 to Fig. 16.
Figs. 16(a) and 16(b) show the following figures.
(a) parallax range T1 of input image
(b) parallax range T5 of parallax-adjusted image
Figs. 16(a) and 16(b) further show a left eye 201 and
15 a right eye 202 of the observer (user) and a display screen
210 of a display unit which is executing the three-dimensional
image display process.
An image having various parallaxes according to the
subject distances are presented on the display screen 210.
20 In the present embodiment, an image which is set such
that the observer (user) realizes that the position of the
subject is at either side of the display screen is adopted
as au input image.
[0159]
25 As described above, the subject is observed as if the
subject is located at a point in the space, which is a crossing
point between a line connecting the left eye 201 of the observer
and the display position of the left image on the display screen
210 and a line connecting the right eye 202 of the observe
30 and the display position of the right image on the display
screen 210.
71
SP262338WO00
In the present embodiment, the images of the subjects,
i. e . , the left image and the right image, (H to J right images
and H to J left images) are set so that the crossing point
is set at a position either side of the display screen 210.
5 [0160]
More specifically, it is an input image which is set
so that:
a subject H is observed on the display screen 210,
a subject I is observed at a position closer to the
10 observer than display screen 210 (observer side), and
a subject J is observed at a deeper side of the display
screen 210.
[0161]
Like the embodiments described above, disparity which
15 is a distance of the same subjects on the display screen is
used as an index value of parallax in this explanation.
Fig. 16(a) is a figure corresponding to an input image
before the parallax-adjusting process,
In Fig. 16(a),
20 The disparity of the subject H is DH which is zero,
The disparity of the subject I is DI, and
The disparity of the subject J is DJ.
In this cas ®, the disparity of the subject I is set such
that the image of the subject I is observed at a position closer
25 to the observer than display screen 210, and the disparity
of the subject J is set such that the image of the subject
J is observed at a deeper side of the display screen 210.
[0162]
More specifically, as can be understood from the
30 arrangement of the display screen 210 of Fig. 16(a),
The image (I left image) of the subject I in the left
72
SP262338WO00
image is set at the right side of the display screen 210, and
The image (I right image) of the subject I in the right
image is set at the left side of the display screen 210.
In contrast,
5 The image (J left image) of the subject J in the left
image is set at the left side of the display screen 210, and
The image (J right image) of the subject J in the right
image is set at the right side of the display screen 210.
According to this setting, the subject I is observed
10 at a position closer to the observer than display screen 210
(observer side), and the subject J is observed at a deeper
side of the display screen 210.
[0163]
The parallax range observed by the observer is a width
15 [Ti] from the point J which is the farthest from the observer
tohe point I which is the closest to the observer.
Fig. 16 (b) illustrates an observation state of an image
displayed using images converted by the image processing
apparatus 100 as illustrated in Fig. 1 according to the present
20 embodiment 4(a), i.e., the parallax-adjusted left image (L2)
50 and the parallax-adjusted right image (R2) 60, for example.
The subjects H6, 16, J6 are the same subjects as H, I,
J of Fig. 16(a), but with the transform processing described
above, image shift is executed, whereby the subject positions
25 observed by the observer are changed.
[0164]
The disparity of the subject H6 is O.
The subject H6 in the left image and the subject H6 in
the right image are displayed at the same position on the display
30 screen 210, and therefore, the disparity is zero.
The position of the subject H6 realized by the observer
73
SP262338W000
is a position on the display screen 210.
[0165]
The disparity of the subject 16 is D16.
The distance, on the display screen 210, between the
5 subject 16 in the left image and the subject 16 in the right
image is DI6, and therefore, the disparity is D16.
The position of the subject 16 realized by the observer
is a position closer to the observer than display screen 210,
i.e., the position 16 at the side of the observer.
10 However, the position of the subject 16 realized by the
observer is set at a position closer to the observer. (farther
from the display screen 210) than subject position I in the
input image before the conversion as illustratedin Fig. 16(a).
This is a result of the above shift processing of the images.
15 [0166]
The disparity of the subject J6 is DJ6.
The distance, on the display screen 210, between the
subject J6 in the left image and the subject J6 in the right
image is DJ6, and therefore, the disparity is DJ6.
20 The position of the subject J6 realized by the observer
is a position deeper than display screen 210, i.e., theposition
J6 which is the farthest from the observer.
However, the position of the subject J6 realized by the
observer is set at a position closer to the observer (closer
25 to the display screen 210) than subject position J in the input
image before the conversion as illustrated in Fig. 16 (a) . This
is a result of the above shift processing of the images.
[0167]
In the present embodiment 4(a), the image transform
30 processing is performed by the image processing apparatus 100
so that the image shift is executed in the same direction as
74
SP262338WO00
that of the embodiment 1 described above.
More specifically, the same image shift processing as
that explained with reference to Fig. 6 is executed. That
is, the following image shift processing is executed:
5 for the left image (Ll) 10, the shift processing is
performed in the right direction, and
for the right image (Rl) 20, the shift processing is
performed in the left direction.
[0168]
10 As a result, the following processing is performed for
the subject I observed at a position closer to the observer
than display screen 210.
The position I of the left image (Fig. 16(a), I, left
image) moves in the right direction to the position 16 of the
15 left image in Fig. 16 (b) , and the position I of the right image
(Fig. 16(a), I, right image) moves in the left direction to
the position 16 of the right image in Fig. 16 (b) . Accordingly,
the disparity increases. In other words, the disparity
increases from DI to DI6.
20 [0169]
Further, the following processing is performed [or the
subject J observed at a position deeper than display screen
210.
The position J of the left image (Fig. 16(a), J, left
25 image) moves in the right direction to the position J6 of the
left image in Fig. 16 (b) , and the position J of the right image
(Fig. 16(a), J, right image) moves in the left direction to
the position J6 of the right image in Fig. 16 (b) . Accordingly,
the disparity decreases. In other words, the disparity
30 decreases from DJ to DJ6.
[0170]
75
SP262338WO00
As described above, in the processing of the embodiment
4(a),
for the subject I observed at a position closer to the
observer than display screen 210, the disparity is greatly
5 changed, and accordingly, it is observed at a position still
more closer to the observer.
On the other hand, for the subject Jobserved at a position
deeper than display screen 210, the disparity is slightly
changed, and even in this case, it is observed at a position
10 still more closer to the observer than position before the
conversion.
[0171]
As a result, as can be understood by comparing the
parallax ranges Ti, T6 of Figs. 16(a) and 16(b),
15 The parallax range Tl (I to J) of the input image before
the conversion change to the parallax range T6 (16 to J6) of
the output image (parallax-adjusted image) after the
conversion.
In other words, in this example, the parallax range T6
20 (16 to J6) of the output image (parallax-adjusted image) is
controlled so that the parallax range Ti (I to J) of the input
image is moved closer to the observer side as a whole.
[0172]
The parallax range moving effect explained with
25 reference to Figs. 16 (a) and 16 (b) is the effect that is provided
as a result of the image shift performed by the image processing
apparatus 100.
In the present embodiment 4 (a) , as described above, the
same differential processing as that of the embodiment 1
30 explained previously is performed, and the same image shift
processing as that explained with reference to Fig. 6 is
76
S2262338WO00
executed. That is, the following image shift processing is
executed:
for the left image (Ll) 10, the shift processing is
performed in the right direction, and
5 for the right image (R1) 20, the shift processing is
performed in the left direction.
[0173]
As illustrated in Fig. 6, the left image differential
processing unit 113 performs the differential calculation
10 using the coefficients as illustrated in Fig. 6 (L2) , and using
the combining processing in which the differential signal or
the non-linear processing result of the differential signal
is added to the left image, a converted image is generated
in which the image is displaced in the right direction with
15 respect to the input image.
On the other hand, the 1 right image differential
processing unit 123 performs differential calculation using
the coefficients as illustrated in Fig. .6 (R2), and using the
combining processing in which the differential signal is added
20 to the right image or the differential signal is added to the
right image after the non-linear processing, a converted image
is generated in which the image is displaced in the left
direction with respect to the input image.
As a result, the disparity between the left image and
25 the right image is changed as illustrated in Figs. 16(a) and
16(b), and the entire parallax range can be moved closer to
the observer, e.g., from the parallax range Tl of Fig. 16(a)
to the parallax range T6 as illustrated in Fig. 16(b).
[0174]
30 The present embodiment is also configured such that the
change of the disparity is the least at a pixel position in
77
SP262338WO00
focus (subject E) in the image serving as the conversion target
image, and the farther a point is away from the focal position,
the greater the change of the disparity becomes.
[0175]
5 (b) Example of processing for shifting a left image in
a left direction and shifting a right image in a right direction
Subsequently, the example of processing for shifting
the left image (L1) 10, which is input into the image processing
apparatus 100, in the left direction and shifting the right
10 image (R1) 20 in the right direction will be explained with
reference to Fig. 17.
In this example of processing, processing corresponding
to the processing as illustrated in Fig. 12 explained in the
embodiment 2 described above is executed. More specifically,
15 the following image shift processing is executed:
for the left image (L1) 10, the shift processing is
performed in the left direction, and
for the right image (R1) 20, the. shift processing is
performed in the right direction.
20 [0176]
In the present embodiment 4(a), the left image
differentiating device 112 executes the differential
processing by applying the following differential filter
coefficients as illustrated in Fig, 12 (L2),
25 [-1, 0, 1]
With this differential processing, the left image is
shifted in the left direction as illustrated in Fig. 12 (L1),
(L2).
On theotherhand, the right image differentiatingdevice
30 122 executes the differential processing by applying the
following differential filter coefficients as illustrated in
78
SP262338W000
Fig. 12 (R2),
[1, 0, -1]
With this differential processing, the right image is
shifted in the right direction as illustrated in Fig. 6 (Rl) ,
5 (R2).
[0177]
An example of parallax control processing according to
the present embodiment 4 (b) will be explained with reference
to Fig. 17.
10 Figs. 17(a) and 17(b) show the following figures.
(a) parallax range Ti of input image
(b) parallax range T5 of parallax-adjusted image
Figs. 17(a) and 17(b) further show a left eye 201 and
a right eye 202 of the observer (user) and a display screen
15 210 of a display unit which is executing the three-dimensional
image display process.
An image having various parallaxes according to the
subject distances are presented on the. display screen 210.
In the present embodiment, an image which is set such
20 that the observer (user) realizes that the position of the
subject is at either side of the display screen is adopted
as an input image.
[017$]
Fig. 17(a) is a figure illustrating the same parallax
25 range T1 of the input image as that of Fig. 16(a) explained
above.
More specifically, it is an input image which is set
so that:
a subject H is observed on the display screen 210,
30 a subject I is observed at a position closer to the
observer than display screen 210 (observer side), and
79
SP262338WO00
a subject J is observed at a deeper side of the display
screen 210.
In Fig. 17(a),
The disparity of the subject H is DH which is zero,
5 The disparity of the subject I is DI, and
The disparity of the subject J is DJ.
In this case, the disparity of the subject I is set such
that the image of the subject I is observed at a position closer
to the observer than display screen 210, and the disparity
10 of the subject J is set such that the image of the subject
J is observed at a deeper side of the display screen 210.
[,017 91
The parallax range observed by the observer is a width
[Ti] from the point J which is the farthest from the observer
15 to the point I which is the closest to the observer.
Fig. 17 (b) illustrates an observation state of an image
displayed using images converted by the image processing
apparatus 100 as illustrated in Fig. 1 according to the present
embodiment 4(b), i.e., the parallax-adjusted left image (L2)
20 50 and the parallax-adjusted right image (R2) 60, for example.
The subjects H7, 17, J7 are the same subjects as H, I,
J of Fig. 17(a), but with the transform processing described
above, image shift is executed, whereby the subject positions
observed by the observer are changed.
25 [0180]
The disparity of the subject H7 is 0.
The subject H7 in the left image and the subject 117 in
the right image are displayed at the same position on the display
screen 210, and therefore, the disparity is zero,
30 The position of the subject H7 realized by the observer
is a position on the display screen 210.
80
SP262338W000
[0181]
The disparity of the subject 17 is D17.
The distance, on the display screen 210, between the
subject I7 in the left image and the subject 17 in the right
5 image is DI7, and therefore, the disparity is D17.
The position of the subject 17 realized by the observer
is a position closer to the observer than display screen 210,
i.e., the position 17 at the side of the observer.
However, the position of the subject 17 realized by the
10 observer is set at a position farther from the observer (closer
to the display screen 210) than subject position I in the input
image before the conversion as illustrated in Fig. 17 (a) . This
is a result of the above shift processing of the images.
[0182]
15 The disparity of the subject J7 is DJ7.
The distance, on the display screen 210, between the
subject J7 in the left image and the subject J7 in the right
image is DJ7, and therefore, the disparity is DJ7.
The position of the subject J7 realized by the observer
20 is a position deeper than display screen 210, i. e. , the position
J7 which is the farthest from the observer.
However, the position of the subject J7 realized by the
observer is set at a position farther from the observer (farther
from the display screen 210) than subject position J in the
25 input image before the conversion as illustrated in Fig. 17(a).
This is a result of the above shift processing of the images.
[0183]
In the present embodiment 4(b), the image transform
processing is performed by the image processing apparatus 100
30 so that the image shift is executed in the same direction as
that of the embodiment 2 described above.
81
SP262338WO00
More specifically, the same image shift processing as
that explained with reference to Fig. 12 is executed. That
is, the following image shift processing is executed;
for the left image (LI) 10, the shift processing is
5 performed in the left direction, and
for the right image (Rl) 20, the shift processing is
performed in the right direction.
[0184]
As a result, the following processing is performed for
10 the subject I observed at a position closer to the observer
than display screen 210.
The position I of the left image (Fig. 17(a), I, left
image) moves in the left direction to the position 17 of the
left image in Fig. 17 (b) , and the position I of the right image
15 (Fig. 17(a), I, right image) moves in the right direction to
the position 17 of the right image in Fig. 17 (b) . Accordingly,
the disparity decreases. In other words, the disparity
decreases from DI to D17.
[0185]
20 Further, the following processing is performed for the
subject J observed at a position deeper than display screen
210.
The position J of the left image (Fig. 17 (a) , J, left
image) moves in the left direction to the position J7 of the
25 left image in Fig. 17 (b) , and the position J of the right image
(Fig. 17(a), J, right image) moves in the right direction to
the position J7 of the right image in Fig. 17 (b) . Accordingly,
the disparity increases. In other words, the disparity
increases from DJ to DJ7.
30 [0186]
As described above, in the processing of the embodiment
82
SP262338WO00
4(b),
for the subject I observed at a position closer to the
observer than display screen 210, the disparity is lightly
changed, and accordingly, it is observed at a position farther
5 from the observer, i . e. , a position closer to the display screen
210.
On the other hand, for the subject J observed at a position
deeper than display screen 210, the disparity is greatly
changed, and even in this case, it is observed at a position
10 farther from the observer than position before the conversion.
[0187]
As a result, as can be understood by comparing the
parallax ranges Ti, T7 of Figs. 17(a) and 17(b),
The parallax range Ti (I to J) of the input image before
15 the conversion change to the parallax range T7 (17 to J7) of
the output image (parallax-adjusted image) after the
conversion.
In other words, in this example; the parallax range T7
(17 to J7) of the output image (parallax-adjusted image) is
20 controlled so that the parallax range Ti (I to J) of the input
image is moved farther from the observer as a whole.
[0188]
The parallax range moving effect explained with
reference to Figs. 17 (a) and 17 (b) is the effect that is provided
25 as a result of the image shift performed by the image processing
apparatus 100.
In the present embodiment 4 (b) , as described above, the
same differential processing as that of the embodiment 2
explained previously is performed, and the same image shift
30 processing as that explained with reference to Fig. 12 is
executed. That is, the following image shift processing is
83
SP262338W000
executed:
for the left image (Ll) 10, the shift processing is
performed in the left direction, and
for the right image (R1) 20, the shift processing is
5 performed in the right direction.
[0189]
As illustrated in Fig. 12, the left image differential
processing unit 113 performs the differential calculation
using the coefficients as illustrated in Fig. 12 (L2) , and
10 using the combining processing in which the differential signal
or the non-linear processing result of the differential signal
is added to the left image, a converted image is generated
in which the image is displaced in the left direction with
respect to the input image.
15 On the other hand, the right image differential
processing unit 123 performs differential calculation using
the coefficients as illustrated in Fig. 12 (R2), and using
the combining processing in which the differential signal is
added to the right image or the differential signal is added
20 to the right image after the non-linear processing,a converted
image is generated in which the image is displaced in the right
direction with respect to the input image.
As a result, the disparity between the left image and
the right image is changed as illustrated in Figs. 17(a) and
25 17(b), and the entire parallax range can be moved away from
the observer, e.g., from the parallax range Ti of Fig. 17(a)
to the parallax range T7 as illustrated in Fig. 17(b).
[0190]
The present embodiment is also configured such that the
30 change of the disparity is the least at a pixel position in
focus (subject E) in the image serving as the conversion target
84
SP262338W000
image, and the farther a point is away from the focal position,
the greater the change of the disparity becomes.
[0191]
The configuration of the embodiment 1 to the embodiment
5 4 explained hereinabove does not require the processing of
generating the disparity map describing disparity (image
difference) corresponding to each pixel of the left image and
the right image and the processing using the disparity map.
Therefore, the processing cost is smaller, and the size of
10 the circuit can be reduced easily.
Subsequently, the embodiment using a simplified
disparity map will be explained.
[0192]
[E. Embodiment 5: the embodiment for performing parallax
15 control using simplified disparity map]
Subsequently, a configuration of an image processing
apparatus executing the parallax control using the simplified
disparity map and an example of processing will be explained
as the embodiment 5.
20 In the embodiment explained below, an image converting
apparatus capable of appropriately controlling the parallax
range by generating the simplified disparity map is achieved.
[0193]
An example of configuration of an image processing
25 apparatus 300 according to the embodiment 5 is shown in Fig.
18. The image processing apparatus 300 as illustrated in Fig.
18 is configured such that a parallax detecting unit 301 is
added to the image processing apparatus 100 as illustrated
in Fig. 1 explained as the image processing apparatus according
30 to the embodiment 1 explained above. The other configuration
is the same as the configuration as illustrated in Fig. 1.
85
SP262338W000
[0194]
As illustrated in Fig. 18, the image processing apparatus
300 includes a left image transforming unit 110 for receiving
a left image (L1 image) 10, performing image transform, and
5 generating a parallax-adjusted left image (L2 image) 50 in
which parallax is adjusted, and also includes a right image
transforming unit 120 for receiving a right image (Rl image)
20, performing image transform, and generating a
parallax-adjusted right image (R2 image) 60 in which parallax
10 is adjusted. In addition, the image processing apparatus 300
includes the parallax detecting unit 301.
[0195]
The left image transforming unit 110 includes a left
image input unit 111 for receiving the left image (L1) 10,
15 a left image differential processing unit 112 for performing
differential processing of the left image 10, a left non-linear
transforming unit 113 for non-linearly transforming a
differential signal of the left image 10, a left image combining
unit 114 for combining the left image 10 and the differential
20 signal non-linearly transformed, and a left image output unit
115 for outputting the converted parallax-adjusted left image
(L2) 50.
The right image transforming unit 120 includes a right
image input unit 121 for receiving the right image (R1) 20,
25 a right image differential processing unit 122 for performing
differential processing of the right image 20, a right
non-linear transforming unit 123 for non-linearly
transforming a differential signal of the right image 20, a
right image combining unit 124 for combining the right image
30 20 and the differential signal non-linearly transformed, and
a right image output unit 125 for outputting the converted
86
SP262338WO00
w
parallax-adjusted right image (R2) 60,
This left image transforming unit 110 and this right
image transforming unit 120 execute the same processing as
the processing explained in the embodiments 1 to 4 explained
5 above.
[0196]
The parallax detecting unit 301 receives the left image
brightness signal which is output from the left image input
unit 111 of the left image transforming unit 110 and the right
10 image brightness signal which is output from the right image
input unit 121 of the right image transforming unit 120, and
detects corresponding pixel positions in a horizontal
direction that are determined to be the same subjects in the
left image and the right image.
15 [0197]
The parallax detecting unit 301 according to the present
embodiment obtains the following determination result for a
positional relationship of corresponding pixel positions of
the left image and the right image (hereinafter referred to
20 as corresponding points).
A determination is made as to which of the following
patterns is applicable.
(a) the corresponding point of the left image is located
at the left of the corresponding point of the right image,
25 and
(b) the corresponding point of the left image is located
at the right of the corresponding point of the right image
[0198]
(a) the corresponding point of the left image is located
30 at the left of the corresponding point of the right image.
When the case of (a) will be explained with reference
87
SP262338W000
to Fig. 17 (a) explained in the embodiment 4 explained above,
the case of (a) is the corresponding relationship of the subject
J in the left image and the subject J in the right image.
In other words, the corresponding point (J left image)
5 of the left image is located at the left of the corresponding
point (J right image) of the right image.
In this kind of setting, the position at which the subj ect
is observed is at a side deeper than display screen (position
farther from the observer).
10 [0199]
(b) the corresponding point of the left image is located
at the right of the corresponding point of the right image.
When the case of (b) will be explained with reference
to Fig. 17(a) explained in the embodiment 4 explained above,
15 the case of (b) is the corresponding relationship of the subject
I i:i the left image and the subject I in the right image.
In other words, the corresponding point (I left image)
of the left image is located at the right. of the corresponding
point (I right image) of the right image.
20 In this kind of setting, the position at which the subject
is observed is at a side closer to the observer than display
screen (position closer to the observer).
[0200]
In the explanation below,
25 when (a) the corresponding point of the left image is
located at the left of the corresponding point of the right
image.
In this case, the disparity is a positive value.
When (b) the corresponding point of the left image is
30 located at the right of the corresponding point of the right
image.
88
SP262338W000
In this case, the disparity is a negative value.
When the disparity is positive, the position at which
the subject is observed is at a side deeper than display screen
(position farther from the observer).
5 When the disparity is negative, the position at which
the subject is observed is at a side closer to the observer
than display screen (position closer to the observer).
[0201]
A generally available disparity map is configured to
10 hold an accurate amount of pixel displacement (the number of
pixels) corresponding to each pixel of the left image and the
right image, but in the present embodiment, the parallax
detecting unit 301 detects only positive/negative (polarity)
of the above disparity of each pixel or a block including
15 multiple pixels.
[0292]
The parallax detecting unit 301 generates this rough
disparity map including disparity polarity information
corresponding to pixels or blocks, and inputs the rough
20 disparity map into the left image differential processing unit
112 of the left image transforming unit 110 and the right image
differential processing unit 122 of the right image
transforming unit 120.
[0203]
25 It should be noted that complicated parallax detection
processing for obtaining a high numerical accuracy of parallax
is not required in the parallax detection processing executed
by the parallax detecting unit 301 of the image processing
apparatus 300 according to the present embodiment. For
30 example, the disparity map is used to control the filter
coefficients of the differential processing unit, but in
89
SP262338W000
principle, the horizontal movement of the image relies on a
spatial frequency of an input image, and therefore, the
performance of parallax control is not greatly affected by
the spatial resolution of the disparity map. Therefore, in
5 the parallax detection processing, a reduced image may be used.
Alternatively, parallax information may be configured to be
detected only at points obtained by spatially decimating the
input image. With such configuration, the cost of the
processing can be reduced.
10 [0204]
Each of the left image differential processing unit 112
and the right image differential processing unit 122 switches
the mode of the differential processing in accordance with
the polarity at the target pixel position of the input disparity
15 map. More specifically, for example, each of the left image
differential processing unit 112 and the right image
differential processing unit 122 performs processing for
switching, as necessary, the processing applying the
differential filter having the setting as illustrated in Fig.
20 6 and the processing applying the differential filter having
the setting as illustrated in Fig. 12 explained in the
embodiments 1 and 2 explained above.
More specifically, the following processing are
performed.
25 (1) processing for shifting the left image in the right
direction, and shitting the right image in the left direction
(Fig. 6),
(2) processing for shifting the left image in the left
direction, and shitting the right image in the right direction
30 (Fig. 12).
Any one of the above processing is switched and executed
90
SP262338W000
in accordance with the polarity at the target pixel position
of the disparity map.
[0205]
Hereinafter, the following processing examples will be
5 explained in order with reference to Figs. 19 (a) , 19 (b) , 20 (a)
and 20(b), which are processing examples in which the image
processing apparatus 300 as illustrated in Fig. 18 of the
present embodiment 5 is applied to the input image in which
a polarity of disparity is in two directions (subjects are
10 displayed at a position deeper than display screen and at a
position closer to the observer than display screen).
The embodiment 5(a): the embodiment for reducing
parallax range
The embodiment 5(b): the embodiment for enlarging
15 parallax range
[0206]
(Embodiment 5(a): the embodiment for reducing parallax
range)
First, the embodiment for performing processing for
20 reducing a parallax range by applying polarity determination
information of disparity will be explained with reference to
Figs. 19(a) and 19(b).
[0207.]
Like those explained in each of the above embodiments,
25 Figs. 19(a) and 19(b) show the following figures.
(a) parallax range U1 of input image
(b) parallax range U8 of parallax-adjusted image
Figs. 19(a) and 19(b) further show a left eye 201 and
a right eye 202 of the observer (user) and a display screen
30 210 of a display unit which is executing the three-dimensional
image display process.
91
SP262338WO00
An image having various parallaxes according to the
subject distances are presented on the display screen 210.
In the present embodiment, an image which is set such
that the observer (user) realizes that the position of the
5 subject is at either side of the display screen is adopted
as an input image.
[0208]
Like Figs. 16(a) and 17(a) explained in the embodiment
4 explained above, subjects are observed at either side of
10 the display screen 210 in Fig. 19(a). More specifically, it
is an input image which is set so that:
a subject K is observed on the display screen 210,
a subject L is observed at a position closer to the
observer than display screen 210 (observer side), and
15 a subject M is observed at a deeper side of the display
screen 210.
In Fig. 19(a),
The disparity of the subject K is DH which is zero,
The disparity of the subject L is DL (disparity polarity
20 = negative (-)), and
The disparity of the subject M is DM (disparity polarity
= positive (+)).
In this case, the polarity of the disparity of the subject
L is negative, and the image of the subject L is set so that
25 it is observed at a position closer to the observer than display
screen 210. The polarity of the disparity of the subject M
is positive, and the image of the subject M is set so that
it is observed at a position deeper than display screen 210.
The parallax range is a parallax range[U1]corresponding
30 to the observation positions of the subjects L to M.
[0209]
92
SP262338W000
Fig. 19 (b) illustrates an observation state of an image
displayed using images converted by the image processing
apparatus 300 as illustrated in Fig. 18 according to the present
embodiment 5(a), i.e., the parallax-adjusted left image (L2)
5 50 and the parallax-adjusted right image (R2) 60, for example.
The subjects K8, L8, M8 are the same subjects as K, L,
M of Fig. 19 (a) , but since the image transform processing with
the image shift is executed in accordance with the disparity
polarity described above, whereby the subject positions
10 observed by the observer are changed.
[0210]
The disparity of the subject KB is zero.
The subject K8 in the left image and the subject K8 in
the right image are displayed at the same position on the display
15 screen 210, and therefore, the disparity is zero.
The position of the subject K8 realized by the observer
is a position on the display screen 210.
[0211]
The disparity of the subject L8 is DLB.
20 The distance, on the display screen 210, between the
subject L8 in the left image and the subject L8 in the right
image is DLB, and therefore, the disparity is DLB.
The position of the subject L8 realized by the observer
is a position closer to the observer than display screen 210,
25 i.e., the position L8 at the side of the observer.
However, the position of the subject LB realized by the
observer is set at a position farther from the observer (closer
to the display screen 210) than subject position L in the input
image beforethe conversion as illustrated in Fig. 19(a). This
30 is a result of the above shift processing of the images.
[0212]
93
SP262338WO00
In this shift processing, the polarity of the disparity
DL of the subject L as illustrated in Fig. 19 (a) is considered.
As described above, the disparity of the subject L is
DL (disparity polarity = negative (-) ) . In other words, the
5 corresponding point (L left image) of the left image is at
the right of the corresponding point (L right image) of the
right image.
As described above, in the present embodiment 5(a), in
the transform processing of the pixel position (pixel or block)
10 of which disparity polarity is negative (-),
"processing of shifting the left image in, the left
direction and shifting the right image in the right direction
(see Fig. 12)" is executed.
[0213]
15 More specifically, as illustrated in Fig. 12, the left
image differential processing unit 113 performs the
differential calculation using the coefficients [-1, 0, 1]
as illustrated in Fig. 12 (L2), and using the combining
processing in which the differential signal or the non-linear
20 processing result of the differential signal is added to the
left image, a converted image is generated in which the image
is displaced in the left direction with respect to the input
left.. image.
On the other hand, the right image differential
25 processing unit 123 performs the differential calculation
using the coefficients [1, 0, -1] as illustrated in Fig. 12
(R2), and using the combining processing in which the
differential signal is added to the right image or the
differential signal is added to the right image after the
30 non-linear processing, a converted image is generated in which
the image is displaced in the right direction with respect
94
SP262338WO00
to the input right image.
[0214]
With this image transform processing, the disparity of
the subject L is changed from DL as illustrated in Fig. 19 (a)
5 to DL8 as illustrated in Fig. 19 (b) . Asa result, the distance
between the corresponding point (L8 left image) of the left
image and the corresponding point (L8 right image) of the right
image is reduced. As a result, the subject image L8 is set
such that it is moved in a direction away from the observer
10 (closer to the display screen 210) with respect to the position
of the subject image L as illustrated in Fig. 19(a) before
the conversion.
[0215]
On the other hand, before the image conversion, the
15 disparity of the subject M located at a position deeper than
display screen 210 is DM. After the image conversion, the
disparity of the subject M becomes DM8 as illustrated in Fig.
19(b).
The position of the subject M8 realized by the observer
20 is at the position M8 deeper than display screen 210.
However, the position of the subject M8 realized by the
observer is set at a position closer to the observer (closer
to the display screen 210) than subject position M in the input
image before the conversion as illustrated in Fig. 19 (a) . This
25 is a result of the above shift processing of the images.
[0216]
In this shift processing, the polarity of the disparity
DM of the subject M as illustrated in Fig. 19(a) is also
considered,
30 As described above, the disparity of the subject M is
DM (disparity polarity = positive (+)). In other words, the
95
SP262338WO00
corresponding point (M left image) of the left image is located
at the left of the corresponding point (M right image) of the
right image.
As described above, in the present embodiment 5 (a) , in
5 the transform processing of the pixel position (pixel or block)
of which disparity polarity is positive (+),
"processing of shifting the left image in the right
direction and shifting the right image in the left direction
(see Fig. 6)" is executed.
10 [0217]
More specifically, as illustrated in Fig. 6, the left
image differential processing unit 113 performs the
differential calculation using the coefficients [1, 0, -1]
as illustrated in Fig. 6 (L2), and using the combining
15 processing in which the differential signal or the non-linear
pro=. essing result of the differential signal is added to the
left image, a converted image is generated in which the image
is displaced in the right direction with. respect to the input
left image.
20 On the other hand, the right image differential
processing unit 123 performs the differential calculation
using the coefficients [-1, 0, 1] as illustrated in Fig. 6
(R2),. and using the combining processing in which the
differential signal is added to the right image or the
25 differential signal is added to the right image after the
non-linear processing, a converted image is generated in which
the image is displaced in the left direction with respect to
the input right image.
[0218]
30 With this image transform processing, the disparity of
the subject M is changed from DM as illustrated in Fig. 19(a)
96
SP262338WO00
to DM8 as illustrated in Fig. 19 (b) . As a result, the distance
between the corresponding point (M8 left image) of the left
image and the corresponding point (M8 right image) of the right
image is reduced. As a result, the subject image M8 is set
5 such that it is moved in a direction closer to the observer
(closer to the display screen 210) with respect to the position
of the subject image M as illustrated in Fig. 19(a) before
the conversion.
[0219]
10 As a result, the parallax range of the three-dimensional
image displayed with the converted images is the parallax range
[U8] as illustrated in Fig. 19(b).
The parallax range [U8] after the conversion becomes
closer to the display screen 210 at either side of the display
15 screen, as compared with the parallax range [U1] of the images
before the conversion, so that the parallax range [U8] after
the conversion becomes a reduced parallax range.
[0220]
As described above, according the present embodiment
20 5(a), with the processing using the differential filter
coefficients in the opposite patterns, the shift processing
in the opposite directions is executed on the subject observed
at a position closer to the observer than display screen and
the subject observed at a position deeper than display screen.
25 In other words, different filters are selected and applied
in accordance with the polarities of disparities, and the image
shift processing (Fig. 6 or Fig. 12) is executed in different
directions.
[0221]
30 With this processing, the observation positions of not
only the subject observed at a position closer to the observer
97
SP262338W000
than display screen but also the subject observed at a position
deeper than display screen can be moved in the display screen
direction, and as a result, the parallax range can be reduced
(from Ul to U8) more efficiently.
5 [0222]
(Embodiment 5 (b) : the embodiment for enlarging parallax
range)
Subsequently, the embodiment for performing processing
for enlarging a parallax range by applying polarity
10 determination information of disparity will be explained with
reference to Figs. 20(a) and 20(b).
[0223]
Like those explained in each of the above embodiments,
Figs. 20(a) and 20(b) show the following figures.
15 (a) parallax range U1 of input image
(b) parallax range U9 of parallax-adjusted image
Figs. 20(a) and 20(b)further show a left eye 201 and
a right eye 202 of the observer (user) and a display screen
210 of a display unit which is executing the three-dimensional
20 image display process.
An image having various parallaxes according to the
subject distances are presented on the display screen 210.
In the present embodiment, an image which is set such
that the observer (user) realizes that the position of the
25 subject is at either side of the display screen is adopted
as an input image.
[0224]
Like Fig. 19(a), subjects are observed at either side
of the display screen 210 in Fig. 20(a). More specifically,
30 it is an input image which is set so that:
a subject K is observed on the display screen 210,
98
SP262338WO00
a subject L is observed at a position closer to the
observer than display screen 210 (observer side), and
a subject M is observed at a deeper side of the display
screen 210.
5 In Fig. 20(a),
The disparity of the subject K is DH which is zero,
The disparity of the subject L is DL (disparity polarity
negative (-)), and
The disparity of the subject M is DM (disparity polarity
10 = positive (+))
In this case, the polarity of the disparity of the subject
L is.negative, and the image of the subject L is set so that
it is observed at a position closer to the observer than display
screen 210. The polarity of the disparity of the subject M
15 is positive, and the image of the subject M is set so that
it =_s observed at a position deeper than display screen 210.
The parallax range is a parallax range [U1] corresponding
to the observation positions of the subjects L to M.
[0225]
20 Fig. 20(b) illustrates an observation state of an image
displayed using images converted by the image processing
apparatus 300 as illustrated in Fig. 18 according to the present
embodiment 5(b), i.e., the parallax-adjusted left image (L2)
50 and the parallax-adjusted right image (R2) 60, for example.
25 The subjects K9, L9, M9 are the same subjects as K, L,
Met Fig. 20(a) , but since the image transform processing with
the image shift is executed in accordance with the disparity
polarity described above, whereby the subject positions
observed by the observer are changed.
30 [0226]
The disparity of the subject K9 is zero.
99
SP262338W000
The subject K9 in the left image and the subject K9 in
the right image are displayed at the same position on the display
screen 210, and therefore, the disparity is zero.
The position of the subject K9 realized by the observer
5 is a position on the display screen 210.
[0227]
The disparity of the subject L9 is DL9.
The distance, on the display screen 210, between the
subject L9 in the left image and the subject L9 in the right
10 image is DL9, and therefore, the disparity is DL9.
The position of the subject L9 realized by the observer
is a position closer to the observer than display screen 210,
i.e., the position L9 at the side of the observer.
However, the position of the subject L9 realized by the
15 observer is set at a position closer to the observer (farther
from the display screen 210) than subject position L in the
input image before the conversion as illustrated in Fig. 20 (a) .
This is a result of the above shift processing of the images.
[0228]
20 In this shift processing, the polarity of the disparity
DLof the subject L as illustrated in Fig. 20 (a) is considered.
As described above, the disparity of the subject L is
DL (disparity polarity = negative (-)). In other words, the
corresponding point (L left image) of the left image is at
25 the right of the corresponding point (L right image) of the
right image.
As described above, the trans formprocessingofthepixel
position (pixel or block) of which disparity polarity is
negative (-) in the present embodiment 5 (b) is different from
30 that of the embodiment 5(a) explained above.
In the present embodiment 5 (b), 'processing of shifting
100
SP262338WO00
the left image in the right direction and shifting the right
image in the left direction (see Fig. 6)" is executed.
[0229]
More specifically, as illustrated in Fig. 6, the left
5 image differential processing unit 113 performs the
differential calculation using the coefficients [1, 0, -1]
as illustrated in Fig. 6 (L2), and using the combining
processing in which the differential signal or the non-linear
processing result of the differential signal is added to the
10 left image, a converted image is generated in which the image
is displaced in the right direction with respect to the input
left image.
On the other hand, the right image differential
processing unit 123 performs the differential calculation
15 using the coefficients [-1, 0, 1] as illustrated in Fig. 6
(R2), and using the combining processing in which the
differential signal is added to the right image or the
differential signal is added to the right image after the
non-linear processing, a converted image is generated in which
20 the image is displaced in the left direction with respect to
the input right image.
[0230]
With this image transform processing, the disparity of
the subject L is changed from DL as illustrated in Fig. 20 (a)
25 to DL9 as illustrated in Fig. 20 (b) . As a result, the distance
between the corresponding point (L9 left image) of the left
image and the corresponding point (L9 right image) of the right
image is increased. As a result, the subject image L9 is set
such that it is moved in a direction closer to the observer
30 (farther from the display screen 210) with respect to the
position of the subject image L as illustrated in Fig. 20(a)
101
SP262338WO00
before the conversion.
[0231]
On the other hand, before the image conversion, the
disparity of the subject M located at a position deeper than
5 display screen 210 is DM. As a result of the image conversion,
the disparity of the subject M becomes DM9 as illustrated in
Fig. 20(b).
The position of the subject M9 realized by the observer
is at the position M9 deeper than display screen 210.
10 However, the position of the subject M9 realized by the
observer is set at a position farther fromthe observer (farther
from the display screen 210) than subject position M in the
input image before the conversion as illustrated in Fig. 20 (a).
This is a result of the above shift processing of the images,
15 [0232]
In this shift processing, the polarity of the disparity
DM of the subject M as illustrated in Fig. 20(a) is also
considered.
As described above, the disparity of the subject M is
20 DM (disparity polarity = positive (+) ) . In other words, the
corresponding point (M left image) of the left image is located
at the left of the corresponding point (M right image) of the
right image.
As described above, in the present embodiment 5 (b) , in
25 the transform processing of the pixel position (pixel or block)
of which disparity polarity is positive (+),
"processing of shifting the left image in the loft
direction and shifting the right image in the right direction
(see Fig, 12)" is executed.
30 [0233]
More specifically, as illustrated in Fig. 12, the left
102
SP262338WO00
image differential processing unit 113 performs the
differential calculation using the coefficients [-1, 0, 1]
as illustrated in Fig. 12 (L2), and using the combining
processing in which the differential signal or the non-linear
5 processing result of the differential signal is added to the
left image, a converted image is generated in which the image
is displaced in the left direction with respect to the input
left image.
On the other hand, the right image differential
10 processing unit 123 performs the differential calculation
using the coefficients [1, 0, -1] as illustrated in Fig. 12
(R2), and using the combining processing in which the
differential signal is added to the right image or the
differential signal is added to the right image after the
15 non-linear processing, a converted image is generated in which
the, image is displaced in the right direction with respect
to the input right image.
[0234]
With this image transform processing, the disparity of
20 the subject M is changed from DM as illustrated in Fig. 20 (a)
to DM9 as illustrated in Fig. 20 (b) . As a result, the distance
between the corresponding point (M9 left image) of the left
image and the corresponding point (M9 right image) of the right
image is increased. As a result, the subject image M9 is set
25 such that it is moved in a direction farther from the observer
(farther from the display screen 210) with respect to the
position of the subject image M as illustrated in Fig. 20 (a)
before the conversion.
[0235]
30 As a result, the parallax range of the three-dimensional
image displayed with the converted images is the parallax range
103
SP262338WO00
[U9] as illustrated in Fig. 20(b).
The parallax range [U9] after the conversion becomes
farther from the display screen 210 at either side of the display
screen, as compared with the parallax range [Ul] of the images
5 before the conversion, so that the parallax range [U9] after
the conversion becomes an enlarged parallax range.
[0236]
As described above, according the present embodiment
5(b), with the processing using the differential filter
10 coefficients in the opposite patterns, the shift processing
in the opposite directions is executed on the subject observed
at a position closer to the observer than display screen and
the subject observed at a position deeper than display screen.
In other words, different filters are selected and applied
15 in accordance with the polarities of disparities, and the image
shift processing (Fig. 6 or Fig. 12) is executed in different
directions.
[0237]
With this processing, the observation positions of not
20 only the subject observed at a position closer to the observer
than display screen but also the subject observed at a position
deeper than display screen can be moved in a direction away
from the display screen direction, and as a result, the parallax
range can be enlarged (from Ul to U9) more efficiently.
25 [0238]
The present embodiment is also configured such that the
change of the disparity is the least at a pixel position in
focus (subject E) in the image serving as the conversion target
image, and the farther a point is away from the focal position,
30 the greater the change of the disparity becomes.
[0239]
104
SP262338W000
[F. Embodiment 6: the embodiment having configuration
to enable input of signal for control of shift mode]
Subsequently, an embodiment having a configuration to
enable input of a control signal for control of a shift mode
5 will be explained with reference to Figs. 21 to 23.
In the above embodiments, in the configuration as
illustrated in Fig. 1, a combination of [-1, 0, 1] and [1,
0, -1] is configured to be used as the differential filter
coefficients applied by the left image differentiating device
10 112 and the right image differentiating device 122, as
explained with reference to Figs 6 and 12.
[.0240]
More specifically, the setting is as follows.
In the embodiment 1, the setting of the differential
15 filter coefficients (the left image differentiating device:
[1,_.,0, -1], the right image differentiating device [-1, 0,
1] ) according to Fig. 6 is setting for shifting the left image
in the right direction and shifting the right image in the
left direction.

CLAIMS
1. An image processing apparatus comprising:
a left image transforming unit for inputting a left image
5 which is to be presented to a left eye and which is applied
to display of a stereoscopic image, changing a phase of an
image signal of a left image in a right direction or a left
direction, and generating aleft image-transformed image; and
a right image transforming unit for inputting a right
10 image which is to be presented to a right eye and which is
applied to display of a stereoscopic image, changing a phase
of an image signal of a right image in the left direction or
the right direction, and generating a right image-transformed
image,
15 wherein the left image transforming unit and the right
image transforming unit extract feature quantities of the image
signals of the input images, and generate the left
image-transformed image and the right image-transformed image
using image trans formprocessingto which the extracted feature
20 quantities are applied.
2. The image processing apparatus according to claim 1,
wherein the left image transforming unit includes:
a left image differentiating device for generating a
25 differential signal of the image signal of the left image as
the feature quantity; and
a left image combining unit for generating the left
image-transformed image by executing combining processing in
which the differential signal of the left image or a transformed
30 signal of the differential signal is added to the left image
signal, and
116
SP262338WO00
wherein the right image transforming unit includes:
a right image differentiating device for generating a
differential signal of the image signal of the right image
as the feature quantity; and
5 a right image combining unit for generating the right
image-transformed image by executing combining processing in
which the differential signal of the right image or a
transformed signal of the differential signal is added to the
right image signal.
10
3. The image processing apparatus according to claim 2,
wherein the left image transforming unit includes a left image
non-linear transforming unit for executing non-linear
transform processing of the differential signal of the left
15 image, and the left image combining unit generates the left
image-transformed image by executing combining processing in
which a transformed signal generated by the left image
non-linear transforming unit is added to the left image signal,
and
20 the right image transforming unit includes a right image
non-linear transforming unit for executing non-linear
transform processing of the differential signal of the right
image, and the right image combining unit generates the right
image-transformed image by executing combining processing in
25 which a transformed signal generated by the right image
non-linear transforming unit is added to the right image
signal.
4. The image processing apparatus according to claim 2 or
30 3, wherein the left image differentiating device and the right
image differentiating device execute differential processing
117
SP262338W000
to which first-order differentiation filters having
differential filter coefficient series of opposite patterns
are applied.
5 5. The image processing apparatus according to claim 2 or
3, wherein the left image differentiating device and the right
image differentiating device execute differential processing
according to a same differential mode, and
one of the left image combining unit and the right image
10 combining unit adds the differential signal of each image or
the transformed signal of the differential signal to the input
image signal, and the other of them performs processing of
deducting the differential signal of each image or the
transformed signal of the differential signal from the input
15 image signal.
6. The image processing apparatus according to any one of
claims 2 to 5, wherein the left image differentiating device
and the right image differentiating device execute
20 differential processing of a brightness signal of the input
image signal.
7. The image processing apparatus according to any one of
claims 2 to 6, wherein the image processing apparatus further
25 includes a parallax detecting unit generating parallax
information by analyzing arrangement of a corresponding point
which corresponds to a same subject portion in the left image
and the right image which are input into the image processing
apparatus, and
30 the left image differentiatingdevice and the right image
differentiating device execute differential processing by
118
SP262338W000
changing a differential processing mode in accordance with
the parallax information generated by the parallax detecting
unit.
5 8. The image processing apparatus according to claim 7,
wherein the parallax detecting unit generates disparity
polarity information indicating that the arrangement of the
corresponding point which corresponds to the same subject
portion in the left image and the right image which are input
10 into the image processing apparatus is any one of the'fol.l.owing
settings (a) and (b)e
(a) a corresponding point of the left image is located
at the left of a corresponding point of the right image; and
(b) the corresponding point of the left image is located
15 at the right of the corresponding point of the right image,
and
the left image differentiating device and the right image
differentiating device execute differential processing to
which first-order differentiation filters having
20 differential filter coefficient series of opposite patterns
are applied, in accordance with the disparity polarity
information generated by the parallax detecting unit.
9. The image processing apparatus according to claim 7 or
25 8, wherein the parallax detecting unit generates parallax
information by applying reduced images or decimated images
of the left image and the right image which are input to the
image processing apparatus.
30 10e The image processing apparatus according to any one of
claims 3 to 9, wherein the image processing apparatus further
119
SP262338W000
includes a control signal input unit for inputting a control
signal for controlling change to at least one of a differential
processing mode for the left image differentiating device and
the right image differentiating device and a transform
5 processing mode for the left image non-linear transforming
unit and the right image non-linear transforming unit
10
11. An image processing method executed by an image
processing apparatus, comprising:
a left image transforming step for causing a left image
transforming unit to input a left image which is to be presented
to a left eye and which is applied to display of a stereoscopic
image, to change a phase of an image signal of a left image
in a right direction or a left direction, and to generate a
15 left image-transformed image; and
a right image transforming step for causing a right image
transforming unit to input a right image which is to be presented
to a right eye and which is applied to display of a stereoscopic
image, to change a phase of an image signal of a right image
20 in the left direction or the right direction, and to generate
a right image-transformed image,
wherein the left image transforming step and the right
image transforming step are steps for extracting feature
quantities of the image signals of the input images, and
25 generating the left image-transformed image and the right
image-transformed image using image transform processing to
which the extracted feature quantities are applied.
12. A program for causing an image processing apparatus to
30 execute image processing, the program causing the image
processing apparatus to execute:
120
SP262338W000
a left image transforming step for causing a left image
transforming unit to input a left image which is to be presented
to a left eye and which is applied to display of a stereoscopic
image, to change a phase of an image signal of a left image
5 in a right direction or a left direction, and to generate a
left image-transformed image; and
a right image transforming step for causing a right image
transforming unit to input a right image which is to be presented
to a right eye and which is applied to display of a stereoscopic
10 image, to change a phase of an image signal of a right image
in the left direction or the right direction, and to generate
a right image-transformed image,
wherein in the• left image transforming step and the right
image transforming step, feature quantities of the image
15 signals of the input images are extracted, and the left
image-trans formed image and the right image-transformed image
are generated using image transform processing to which the
extracted feature quantities are applied.