1
SP346516WO00
DESCRIPTION IMAGE PROCESSING DEVICE, IMAGE PROCESSING METHOD, AND PROGRAM
TECHNICAL FIELD 5 [0001]
The present technology relates to an image processing device which emphasizes a stereoscopic effect of an image. More specifically, this relates to an image processing device which integrates a plurality of stereoscopic effect emphasis 10 processing results, an image display device, a processing method thereof, and a program which allows a computer to execute the method.
BACKGROUND ART
15 [0002]
Popularization of digital still cameras makes end users familiar with imaging everyday scenes. Because of a short focal distance of the digital still camera, an imaged image is sometimes planar and lacks a stereoscopic effect.
20 [0003]
Therefore, technology of calculating a subject distance and shading or blurring based on this is suggested in order to emphasize a stereoscopic effect in a stereoscopic image in which two persons are made common subjects of
25 interest of right and left eyes (for example, refer to Patent Document 1).
CITATION LIST PATENT DOCUMENT 30 [0004]
Patent Document 1: JP 2009-53748 A

2
SP346516WO00
SUMMARY OF THE INVENTION
PROBLEMS TO BE SOLVED BY THE INVENTION [0005] 5 In the above-described conventional technology, the stereoscopic effect of a planar shallow image is enhanced by stereoscopic distance measurement by using the stereoscopic image and a process to make the image look like deep based on this . However, the stereoscopic image formed of the images
10 of the right and left eyes is supposed in this conventional technology, so that it takes time to calculate the subject distance. Furthermore, it is not yet studied how to combine a plurality of stereoscopic effect emphasis processes such as the shading and blurring.
15 [0006]
The present technology is achieved in view of such situations and an object thereof is to suppose a plurality of processes for emphasizing the stereoscopic effect of the image and integrating a plurality of stereoscopic effect
20 emphasis processes in a mode suitable for the image.
SOLUTIONS TO PROBLEMS [0007]
This technology is achieved for solving the
25 above-described problems and a first aspect thereof is an image processing device provided with a stereoscopic effect emphasis processing unit which executes a plurality of stereoscopic effect emphasis processes on an input image, and an integration processing unit which integrates results of
30 the plurality of stereoscopic effect emphasis processes according to an integration coefficient used when the

3
SP346516WO00
plurality of stereoscopic effect emphasis processes is integrated obtained by analyzing the input image, an image processing method thereof or a program. This has an effect of integrating a plurality of stereoscopic effect emphasis 5 processes in a mode suitable for the image. It is also possible that the first aspect is further provided with an image analyzing unit which analyzes the input image to generate the integration coefficient used when the plurality of stereoscopic effect emphasis processes is integrated.
10 [0008]
In the first aspect, the image analyzing unit may also generate the integration coefficient based on a scene of the input image. This has an effect of integrating a plurality of stereoscopic effect emphasis processes based on scene
15 recognition. [0009]
In the first aspect, the image analyzing unit may also generate the integration coefficient based on the recognized scene and a feature amount included in the input image. This
20 has an effect of integrating a plurality of stereoscopic effect emphasis processes based on the scene recognition and the feature amount. [0010]
In the first aspect, when the integration coefficients
25 of the stereoscopic effect emphasis process to control a relative size and the stereoscopic effect emphasis process to perform positional control based on perspective are higher than predetermined thresholds, the image analyzing unit may further increase the integration coefficient of the
30 stereoscopic effect emphasis process to perform the
positional control. This has an effect of emphasizing a

4
SP346516WO00
stereoscopic effect of a deep image. [0011]
In the first aspect, when the integration coefficients of the stereoscopic effect emphasis process to emphasize a 5 texture gradient and the stereoscopic effect emphasis process according to a focus are higher than predetermined thresholds, the image analyzing unit may further increase the integration coefficients of the stereoscopic effect emphasis process to emphasize the texture gradient and the stereoscopic effect
10 emphasis process according to the focus. This has an effect of emphasizing the stereoscopic effect of the image with a sharp texture gradient. [0012]
In the first aspect, the image analyzing unit may
15 inhibit a lower integration coefficient of the integration coefficient of the stereoscopic effect emphasis process based on aerial perspective and the integration coefficient of the stereoscopic effect emphasis process by shading. This has an effect of emphasizing the stereoscopic effect while giving
20 priority to any one of the stereoscopic effect emphasis
process based on the aerial perspective and the stereoscopic
effect emphasis process by shading.
[0013]
In the first aspect, the image analyzing unit may
25 inhibit a lower integration coefficient of the integration coefficient of the stereoscopic effect emphasis process by shading and the integration coefficient of the stereoscopic effect emphasis process to emphasize the texture gradient or the integration coefficient of the stereoscopic effect
30 emphasis process according to the focus. This has an effect of emphasizing the stereoscopic effect while considering an

5
SP346516WO00
arrangement of an object. [0014]
In the first aspect, the image analyzing unit may inhibit a lower integration coefficient of the integration 5 coefficient of the stereoscopic effect emphasis process based on the aerial perspective and the integration coefficient of the stereoscopic effect emphasis process to control the relative size or the integration coefficient of the stereoscopic effect emphasis process to perform the
10 positional control based on the perspective. This has an effect of emphasizing the stereoscopic effect while considering an angle of view. [0015]
In the first aspect, the integration processing unit
15 may separately perform integration of the stereoscopic effect emphasis processes regarding a position and integration of the stereoscopic effect emphasis processes regarding a pixel value of the plurality of stereoscopic effect emphasis processes in order. This has an effect of combining the
20 stereoscopic effect emphasis processes of different types. [0016]
A second aspect of the present technology is an image display device provided with a stereoscopic effect emphasis processing unit which executes a plurality of stereoscopic
25 effect emphasis processes on an input image, an integration processing unit which integrates results of the plurality of stereoscopic effect emphasis processes according to an integration coefficient used when the plurality of stereoscopic effect emphasis processes is integrated
30 obtained by analyzing the input image, and a display unit which displays an image obtained by integrating the results

6
SP346516WO00
of the plurality of stereoscopic effect emphasis processes in the input image. This has an effect of displaying the image while integrating a plurality of stereoscopic effect emphasis processes in a mode suitable for the image. 5
EFFECTS OF THE INVENTION [0017]
The present technology may have an excellent effect of supposing a plurality of processes for emphasizing the 10 stereoscopic effect of the image and integrating a plurality of stereoscopic effect emphasis processes in a mode suitable for the image.
BRIEF DESCRIPTION OF DRAWINGS 15 [0018]
-i'Fig. 1 is a view illustrating a configuration example
j of an image display device of an embodiment of the present
technology.
Fig. 2 is a view illustrating a configuration example
20 of a stereoscopic effect emphasis processing unit 150 of the
embodiment of the present technology.
Fig. 3 is a view illustrating an example of a
stereoscopic effect emphasis process to control relative
sizes of objects by a first emphasis processing unit 151 of
25 the embodiment of the present technology.
Fig. 4 is a view illustrating an example of the
stereoscopic effect emphasis process to control a position
of the object based on perspective by a second emphasis
processing unit 152 of the embodiment of the present
30 technology.
Fig. 5 is a view illustrating an example of the

7
SP346516WO00
stereoscopic effect emphasis process to control motion of the object by a third emphasis processing unit 153 of the embodiment of the present technology,
Fig. 6 is a view illustrating an example of the 5 stereoscopic effect emphasis process based on aerial
perspective by a fourth emphasis processing unit 154 of the embodiment of the present technology.
Fig. 7 is a view illustrating a relationship between a depth and saturation in the stereoscopic effect emphasis 10 process based on the aerial perspective by the fourth emphasis processing unit 154 of the embodiment of the present technology.
Fig. 8 is a view illustrating an example of the stereoscopic effect emphasis process to emphasize a texture 15 gradient by a fifth emphasis processing unit 155 of the embodiment of the present technology.
Fig. 9 is a view illustrating an example of the stereoscopic effect emphasis process by shading by a sixth emphasis processing unit 156 of the embodiment of the present 20 technology.
Fig. 10 is a view illustrating an example of the stereoscopic effect emphasis process according to a focus by a seventh emphasis processing unit 157 of the embodiment of .the present technology. 25 Fig. 11 is a view illustrating a first configuration example of an image analyzing unit 140 of the embodiment of the present technology.
Fig. 12 is a view illustrating an example of a landscape as a scene of an image. 30 Fig. 13 is a view illustrating an example of a structure, a street and the like as the scene of the image.

8
SP346516WO00
Fig. 14 is a view illustrating an example of a portrait as the scene of the image.
Fig. 15 is a view illustrating a configuration example of a binding coefficient table 142 of the embodiment of the 5 present technology.
Fig. 16 is a flow diagram illustrating a procedure by the first configuration example of the image analyzing unit 140 of the embodiment of the present technology.
Fig. 17 is a view illustrating a second configuration 10 example of the image analyzing unit 140 of the embodiment of the present technology.
Fig. 18 is a view illustrating a configuration example of a general coefficient table 144 of the embodiment of the present technology. 15 Fig. 19 is a flow diagram illustrating a procedure by the second configuration example of the image analyzing unit 140 of the embodiment of the present technology.
Fig. 20 is a view illustrating a third configuration example of the image analyzing unit 140 of the embodiment of 20 the present technology.
Fig. 21 is a flow diagram illustrating a procedure by the third configuration example of the image analyzing unit 140 of the embodiment of the present technology.
Fig. 22 is a view illustrating a fourth configuration 25 example of the image analyzing unit 140 of the embodiment of the present technology.
Fig. 23 is a flow diagram illustrating a procedure by the fourth configuration example of the image analyzing unit 140 of the embodiment of the present technology. 30 Fig. 24 is a view illustrating an example of a
relationship between the depth and a scaling factor in the

9
SP346516WO00
stereoscopic effect emphasis process to control the relative sizes of the objects by the first emphasis processing unit 151 of the embodiment of the present technology.
Fig. 25 is a view illustrating an example of a 5 relationship between the depth and a motion amount in the stereoscopic effect emphasis process to control the position of the object based on the perspective by the second emphasis processing unit 152 of the embodiment of the present technology.
10 Fig. 26 is a view illustrating an example of the
relationship between the depth and the motion amount in the stereoscopic effect emphasis process to control the motion of the object by the third emphasis processing unit 152 of the embodiment of the present technology.
15 Fig. 27 is a view illustrating a procedure example of
an integration process of control of an area or-5 a position
•i by an integration processing unit 160 of the embodiment of
the present technology.
Fig. 28 is a view illustrating a procedure example of
20 the integration process of the control of a pixel value by
the integration processing unit 160 of the embodiment of the
present technology.
Fig. 29 is a view illustrating a first entire procedure
example of the image display device of the embodiment of the
25 present technology.
Fig. 30 is a view illustrating a second entire procedure
example of the image display device of the embodiment of the
present technology.
Fig. 31 is a view illustrating a third entire procedure
30 example of the image display device of the embodiment of the
present technology.

10
SP346516WO00
Fig. 32 is a view illustrating a fourth entire procedure example of the image display device of the embodiment of the present technology.
5 MODE FOR CARRYING OUT THE INVENTION [0019]
A mode for carrying out the present technology (hereinafter, referred to as an embodiment) is hereinafter described. Note that the description is given in the 10 following order.
1. Entire Configuration (Application Example to Image Display Device)
2. Stereoscopic Effect Emphasis Process

3. Image Analysis (Generation of Integration 15 Coefficient)
4. Integration Process
5. Entire Operation
[0020]
<1. Entire Configuration>
20 [Configuration of Image Display Device]
Fig. 1 is a view illustrating a configuration example of an image display device of an embodiment of the present technology. The image display device is provided with an input image supplying unit 110, a depth information obtaining 25 unit 120, an additional information obtaining unit 130, an image analyzing unit 140, a stereoscopic effect emphasis processing unit 150, an integration processing unit 160, and a display unit 170. [0021] 30 The input image supplying unit 110 is configured to supply an input image. An imaging device which images a

11
SP346516WO00
subject to generate an imaged image, an image server which outputs an image selected from saved images and the like, for example, supposedly serve as the input image supplying unit 110. The input image supplying unit 110 may supply depth 5 information and additional information corresponding to the input image together with the input image. [0022]
The depth information obtaining unit 120 is configured to obtain the depth information corresponding to the input
10 image unit. The depth information is the information
indicating a depth of each pixel forming the input image . The depth information supplied from the input image supplying unit 110 may be used. The depth information generated by a unit other than the input image supplying unit 110 may also
15 be obtained. The depth information obtaining unit 120 may also generate the depth information from the input image. [0023]
The additional information obtaining unit 130 is configured to obtain the additional information
20 corresponding to the input image. The additional
information is an image feature amount and a control parameter used by the image analyzing unit 140. The additional information supplied from the input image supplying unit 110 may be used. The additional information generated by a unit
25 other than the input image supplying unit 110 may also be obtained. The additional information obtaining unit 130 may also generate the additional information from the input image. [0024]
30 The image analyzing unit 140 is configured to analyze the input image to generate an integration coefficient used

12
SP346516WO00
when a plurality of stereoscopic effect emphasis processes is integrated. The integration coefficient is supplied to the stereoscopic effect emphasis processing unit 150 and the integration processing unit 160. The integration 5 coefficient is described later in detail. [0025]
The stereoscopic effect emphasis processing unit 150 is configured to execute a plurality of stereoscopic effect emphasis processes on the input image. The stereoscopic
10 effect emphasis processing unit 150 receives the input image supplied from the input image supplying unit 110 and the depth information obtained by the depth information obtaining unit 120 from the image analyzing unit 140 and applies a plurality of stereoscopic effect emphasis processes according to the
15 integration coefficient supplied from the image analyzing unit 140. [0026]
The integration processing unit 160 is configured to integrate results of a plurality of stereoscopic effect
20 emphasis processes executed by the stereoscopic effect
emphasis processing unit 150 according to the integration coefficient supplied from the image analyzing unit 140. The integration processing unit 160 outputs an integrated result as an output image.
25 [0027]
The display unit 170 is configured to display the output image output from the integration processing unit 160. The display unit 170 may be realized by an LCD (liquid crystal display) and the like, for example.
30 [0028]
Meanwhile, although the configuration example of the

13
SP346516WO00
image display device including the display unit 170 is herein described, an image processing device without the display unit 170 may also be configured. [0029] 5 <2. Stereoscopic Effect Emphasis Process>
[Configuration of Stereoscopic Effect Emphasis Processing Unit]
Fig. 2 is a view illustrating a configuration example of the stereoscopic effect emphasis processing unit 150 of
10 the embodiment of the present technology. The stereoscopic effect emphasis processing unit 150 is provided with processing units which perform seven types of stereoscopic effect emphasis processes. [0030]
15 That is to say, a first emphasis processing unit 151 is configured to perform the stereoscopic effect emphasis process to control a relative size. A second emphasis processing unit 152 is configured to perform the stereoscopic effect emphasis process to perform positional control based
20 on perspective. A third emphasis processing unit 153 is configured to perform the stereoscopic effect emphasis process to control motion (depth from motion) of an object.' A fourth emphasis processing unit 154 is configured to perform the stereoscopic effect emphasis process based on aerial
25 perspective. A fifth emphasis processing unit 155 is configured to perform the stereoscopic effect emphasis process to emphasize a texture gradient. A sixth emphasis processing unit 156 is configured to perform the stereoscopic effect emphasis process by shading. A seventh emphasis
30 processing unit 157 is configured to perform the stereoscopic effect emphasis process according to a focus. Hereinafter,

14
SP346516WO00
each of the stereoscopic effect emphasis processes is
described.
[0031]
[First Emphasis Processing Unit (Relative Size)] 5 Fig. 3 is a view illustrating an example of the
stereoscopic effect emphasis process to control the relative sizes of the objects by the first emphasis processing unit 151 of the embodiment of the present technology. In the stereoscopic effect emphasis process, the image is segmented
10 into respective regions. At the time of the segmentation, a mean shift method, a graph cut method and the like are used, for example. The depth information for each region is referred to and a foreground region (foreground object) located on a front side is extracted as a region to be processed
15 When the region to be processed is extracted, it is possible to use an object recognition algorithm such as an SVM (support vector machine) method and a boosting method for improving accuracy. In the stereoscopic effect emphasis process, a size of the foreground region is made relatively large as
20 illustrated in the drawing, and according to this, a
stereoscopic effect is emphasized. A scaling factor at that
time may be obtained from the depth.
[0032]
When there is a plurality of foreground regions, two
25 regions to be processed such as the nearest region and the second nearest region out of the foreground regions, for example, are obtained and an enlarging process is applied to a near region and a reducing process is applied to a far region. According to this, it becomes possible to emphasize a sense
30 of distance in a depth direction between the two regions. Herein, at the time of the enlarging process and the reducing

15
SP346516WO00
process, a bicubic method, a Lanczos method and the like are used, for example. The enlarging process and the reducing process may be performed on each of the two regions or on any one of the regions. When there is one foreground region, the 5 similar process is performed on the foreground region and a background region. [0033]
In the stereoscopic effect emphasis process, when the size of the foreground region is changed in the image, it is
10 necessary to change a pixel position in the depth information together by performing the enlarging process and the reducing process on the depth information as on the image as illustrated in the drawing. [0034]
15 [Second Emphasis Processing Unit (Perspective)]
Fig. 4 is a view illustrating an example of the stereoscopic effect emphasis process to control a position of the object based on the perspective by the second emphasis processing unit 152 of the embodiment of the present
20 technology. In the stereoscopic effect emphasis process, a horizontal line and a vanishing point in the image are detected and an intersection thereof is obtained to set a reference of the depth. Herein, it is possible to use Hough transform, for example, for detecting a straight line. As
25 in the stereoscopic effect emphasis process by the first emphasis processing unit 151, the image is segmented into the respective regions. At the time of the segmentation, a mean shift method, a graph cut method and the like are used, for example. The depth information for each region is referred
30 to and a foreground region (foreground object) located on a front side is extracted as a region to be processed. When

16
SP346516WO00
the region to be processed is extracted, it is possible to use an object recognition algorithm such as an SVM (support vector machine) method and a boosting method for improving accuracy. 5 [0035]
In the stereoscopic effect emphasis process, as for the foreground region to be processed, the position of the foreground region is moved in a direction of the straight line passing through a barycentric coordinate of the foreground
10 region and a coordinate of the vanishing point based on the vanishing point, for example, as illustrated in the drawing. That is to say, the foreground region is moved such that this comes near when this moves outward from the vanishing point and this goes far when this moves toward the vanishing point.
15 A motion amount at that time may be obtained from the depth.
[0036] ■*
■i In this stereoscopic effect emphasis process, when the
position of the foreground region is moved in the image, it
is necessary to change the pixel position in the depth
20 information together by moving the position of the foreground region in the depth information as in the image as illustrated in the drawing. [0037]
[Third Emphasis Processing Unit (Motion of Object)]
25 Fig. 5 is a view illustrating an example of the
stereoscopic effect emphasis process to control the motion of the object by the third emphasis processing unit 153 of the embodiment of the present technology. In the stereoscopic effect emphasis process, a motion vector is
30 obtained from the image. The image is segmented into the respective regions . At the time of the segmentation, the mean

17
SP346516WO00
shift method, the graph cut method, and a method of collecting the motion vectors having the same direction and the same size are used, for example. The depth information for each region is referred to and a foreground region (foreground object) 5 located on a front side is extracted as a region to be processed. Thereafter, a representative vector is obtained by obtaining an average value and the like for each segmented region. [0038]
In the stereoscopic effect emphasis process, the sense
10 of distance is controlled by adjustment of difference in size between the representative vector of the foreground region and that of the background region. For example, the sense of distance between the foreground and the background is emphasized by adjustment of a motion amount of the foreground
15 such that a difference value becomes larger. On the other hand, the sense of distance between the foreground and the background is inhibited by the adjustment of the motion amount of the foreground such that the difference value becomes smaller. A change amount at that time may be obtained from
2 0 the depth. [0039]
[Fourth Emphasis Processing Unit (Aerial Perspective) ] Fig. 6 is a view illustrating an example of the stereoscopic effect emphasis process based on the aerial
25 perspective by the fourth emphasis processing unit 154 of the embodiment of the present technology. In the stereoscopic effect emphasis process, a distant view is blurred or saturation of the distant view is decreased to control a sense of depth.
30 [0040]
First, it is determined whether the image may be subject

18
SP346516WO00
to the stereoscopic effect emphasis process. A landscape image is a typical example thereof. Therefore, when an area of the color of the sky included in an upper third area of the image is not smaller than a threshold or when a green region 5 included in the image is not smaller than a threshold, for example, it is determined that the image may be subject to the stereoscopic effect emphasis process. A case in which brightness is higher than a threshold th_y and the color is lower than thresholds th_r and th_g of R and G, respectively,
10 are detected for YCbCr and RGB in order to determine the color of the sky. The green region is similarly determined. [0041]
When it is determined that the image may be subject to the stereoscopic effect emphasis process, a process of
15 decreasing the saturation (Cb and Cr) more in an upper side in a vertical direction of the image an'd a process of blurring the distant view by a Gaussian filter and the like are applied. [0042]
It is also possible to perform the process of decreasing
20 the saturation of the distant view and the process of blurring the distant view based on the depth information as illustrated in Fig. 7. In this case, when it is indicated that the depth is greater than a threshold, the depth may be determined to be that of the distant view.
25 [0043]
[Fifth Emphasis Processing Unit (Texture Gradient)] Fig. 8 is a view illustrating an example of the stereoscopic effect emphasis process to emphasize the texture gradient by the fifth emphasis processing unit 155 of the
30 embodiment of the present technology. In the stereoscopic effect emphasis process, a texture is first detected. The

19
SP346516WO00
texture may be obtained by a region expanding method by using a size and a direction of an edge in an edge detection result, density in horizontal/vertical N taps and the like as feature amounts, for example. In the region expanding method, a label 5 is assigned to each pixel by using the feature amount in each pixel as density of the texture. [0044]
Out of the obtained textures, the texture of a near view is sharpened by application of an unsharp mask or by addition
10 of a high frequency component obtained from a high-pass filter. The texture of the distant view is dulled by the Gaussian filter and the like. For example, if a large depth value and a small depth value are those of the near view and the distant view, respectively, when the depth value is not smaller than
15 a threshold thl, the process for the near view is performed
considering that the value is that of the near view; and when
•J
this is not larger than a threshold th2, the process for the
distant view is performed considering that the value is that of the distant view. When the depth value is between the
20 thresholds thl and th2, a coefficient to be multiplied by a near view processing result and a distant view processing result is obtained by linear compensation using thl, th2, and the depth value and the near view processing result and the distant view processing result are blended.
25 [0045]
[Sixth Emphasis Processing Unit (Shading)] Fig. 9 is a view illustrating an example of the stereoscopic effect emphasis process by shading by the sixth emphasis processing unit 156 of the embodiment of the present
30 technology. In the stereoscopic effect emphasis process, a portion with unevenness of the depth is shaded. In the

20
SP346516WO00
stereoscopic effect emphasis process, a light source is estimated and it is specified from where the light is applied (top, bottom, right, and left of the image) . A region in which change in depth between the foreground and the background is 5 large is detected by using the depth information and it is determined whether this is a region shaded according to an estimated direction of the light. At that time, a region in which a sum of absolute difference of depth is not smaller than a threshold in the horizontal/vertical N taps in a
10 spatial direction becomes a detection target. [0046]
In the stereoscopic effect emphasis process, the brightness of the region determined to be the shaded region by the above-described process is inhibited. For example,
15 this is more inhibited as brightness contrast at a boundary between the foreground and the background of the image is higher, and this is less inhibited as the brightness contrast is lower. Alternatively, this is less inhibited when the change in the depth between the foreground and the background
20 is small, and this is more inhibited considering that it is more deeply shaded when the change in depth is larger. At that time, the change in depth between the foreground and the background may be calculated as the sum of absolute di f f erence of depth in the horizontal/vertical N taps in the spatial
25 direction. [0047]
[Seventh Emphasis Processing Unit (Focus)] Fig. 10 is a view illustrating an example of the stereoscopic effect emphasis process according to the focus
30 by the seventh emphasis processing unit 157 of the embodiment of the present technology. In the stereoscopic effect

21
SP346516WO00
emphasis process, an enhancement process is applied to a
focused portion.
[0048]
A region with a high spatial frequency obtained by FFT 5 and the like is estimated to be in focus, that is to say, the near view. On the other hand, a region with a low spatial frequency is estimated to be out of focus, that is to say, the distant view. A blurred portion is excluded though this is in the foreground. Furthermore, the near view is separated
10 from the distant view based on the depth information. In this case, a histogram of the depth information is generated, for example, and a mountain located on a far side is made the distant view and a mountain located on a near side is made the near view. The region with the high spatial frequency
15 and the near view portion obtained from the depth information are made the region in focus. [0049]
It is possible to emphasize the stereoscopic effect of the near view by emphasizing the edge and the texture by
20 applying the unsharp mask and by adding the high-frequency component obtained by the high-pass filter to the near view. When the large depth value and the small depth value are those of the near view and the far view, respectively, a degree of emphasis is made larger as the depth value is larger.
25 [0050]
In this manner, existing technology may be used in each of a plurality of stereoscopic effect emphasis processes by the stereoscopic effect emphasis processing unit 150. [0051]
30 <3. Image Analysis (Generation of Integration Coefficient)>

22
SP346516WO00
[First Configuration Example of Image Analyzing Unit]
Fig. 11 is a view illustrating a first configuration
example of the image analyzing unit 140 of the embodiment of
the present technology. The first configuration example of
5 the image analyzing unit 140 is provided with a scene
recognition processing unit 141 and a binding coefficient
table 142.
[0052]
The scene recognition processing unit 141 is configured
10 to recognize a scene estimated from the input image . Although various scenes of the image may be considered, three representative scenes of a landscape; a structure, a street and the like; and a portrait are supposed in this embodiment as described hereinafter.
15 [0053]
Fig. 12 is a view illustrating an example of the landscape as the scene of the image. In a case of the landscape, an entire image is the distant view and the foreground is not basically included. Fig. 13 is a view
20 illustrating an example of the structure, the street and the like as the scene of the image. In this case, the vanishing point is often included in the street and the like and the structure often becomes the foreground. Fig. 14 is a view illustrating an example of the portrait as the scene of the
25 image. In a case of the portrait, a person is included as the foreground and the background is often blurred. [0054]
The binding coefficient table 142 is the table which holds the integration coefficients of a plurality of
30 stereoscopic effect emphasis processes such that they
correspond to the scenes recognized by the scene recognition

23
SP346516WO00
processing unit 141. [0055]
Fig. 15 is a view illustrating a configuration example of the binding coefficient table 142 of the embodiment of the 5 present technology. The binding coefficient table 142 holds integration coefficients wl to w7 for each of the above-described seven stereoscopic effect emphasis processes for each scene. The above-described three scenes of the landscape; the structure, the street and the like; and 10 the portrait are illustrated. [0056]
The binding coefficient table 142 outputs the integration coefficients wl to w7 of the stereoscopic effect emphasis processes according to the scene indicated by the 15 scene recognition processing unit 141. The coefficients created in advance by learning and the like are used, for example, as the integration coefficients wl to w7 held by the binding coefficient table 142. [0057] 20 Fig. 16 is a flow diagram illustrating a procedure by the first configuration example of the image analyzing unit 140 of the embodiment of the present technology. [0058]
First, the scene recognition processing unit 141 25 recognizes the scene estimated from the input image (step S941). The binding coefficient table 142 is searched according to the recognized scene and the integration coefficients wl to w7 of the stereoscopic effect emphasis processes are output (step S942). 30 [0059]
[Second Configuration Example of Image Analyzing Unit]

24
SP346516WO00
Fig. 17 is a view illustrating a second configuration example of the image analyzing unit 140 of the embodiment of the present technology. The first configuration example of the image analyzing unit 14 0 is provided with a scene 5 recognition processing unit 143, a general coefficient table 144, a feature amount coefficient generating unit 145, and a binding coefficient generating unit 146. [0060]
The scene recognition processing unit 143 is configured
10 to recognize the scene estimated from the input image. The three scenes of the image of the landscape; the structure, the street and the like; and the portrait are supposed as in the first configuration example. [0061]
15 The general coefficient table 144 outputs general
coefficients Gl to G7 of the stereoscopic effect emphasis processes according to the scene indicated by the scene recognition processing unit 143. The general coefficients Gl to G7 are comprehensive weight coefficients indicating a
20 relationship among the respective stereoscopic effect emphasis processes. [0062]
The feature amount coefficient generating unit 145 is configured to generate feature amount coefficients Fl to F7
25 of the image from the input image. The feature amount coefficients Fl to F7 are the coefficients based on the feature amount of the image and are specifically obtained from a pixel value of the image, the feature amount of the image, and the depth of the depth information.
30 [0063]
The binding coefficient generating unit 146 is

25
SP346516WO00
configured to generate the integration coefficients wl to w7 from the general coefficients Gl to G7 output from the general coefficient table 144 and the feature amount coefficients Fl to F7 generated by the feature amount coefficient generating 5 unit 145 according to a next equation. That is to say, the next equation is satisfied among the integration coefficients wl to w7, the general coefficients Gl to G7, and the feature amount coefficients Fl to F7.
wi = Gi * Fi (i = 1 to 7)
10 [0064]
Although only one set of binding coefficients may be prepared for one item of a scene recognition result in the above-described first configuration example, since the general coefficient by the scene recognition result is
15 multiplied by the feature amount coefficient according to the
-' second configuration example, a finer stereoscopic effect
emphasis process may be performed for each image. [0065]
Herein, the feature amount coefficients Fl to F7 of the
20 image generated by the binding coefficient generating unit 146 are described. As described above, the feature amount coefficients Fl to F7 are generated from the input image. A guideline when the feature amount coefficients Fl to F7 are generated for each of the above-described seven stereoscopic
25 effect emphasis processes is hereinafter described. [0066]
An area ratio between the foreground being the processing target and another foreground is supposed, for example, as the image feature amount extracted for the
30 stereoscopic effect emphasis process to control the relative sizes of the objects by the first emphasis processing unit

26
SP346516WO00
151. The feature amount coefficient Fl of the stereoscopic effect emphasis process is generated such that the sense of distance between the two objects is larger as the areas of the two regions are closer. 5 [0067]
Presence of the vanishing point and the number of the straight lines forming the vanishing point are supposed as the image feature amount extracted for the stereoscopic effect emphasis process to control the position of the object
10 based on the perspective by the second emphasis processing unit 152. When there is the vanishing point, the feature amount coefficient F2 is increased. When the number of the straight lines forming the vanishing point is larger than the predetermined number, the image feature amount F2 is
15 increased, and when this is smaller, the feature amount
coefficient F2 is decreased. That is to say, the number of
■i the straight lines forming the vanishing point is used as an
indication of reliability of the vanishing point.
[0068]
20 A ratio between a size of the vector of the foreground and that of the background is supposed as the image feature amount extracted for the stereoscopic effect emphasis process to control the motion of the object by the third emphasis processing unit 153. The feature amount coefficient F3 of
25 the stereoscopic effect emphasis process is generated such that the sense of distance between the foreground and the background is larger as the ratio between the vector of the foreground and that of the background is closer. [0069]
30 A value obtained by averaging average values of
respective vertical lines of obtained gradients between

27
SP346516WO00
adjacent pixels in the vertical direction of the depth information among the number of pixels in a horizontal direction is supposed as the image feature amount extracted for the stereoscopic effect emphasis process based on the 5 aerial perspective by the fourth emphasis processing unit 154. When a line based on the gradient is drawn with the vertical direction and the depth direction plotted on an ordinate axis and an abscissa axis, respectively, a distance between the distant view and the near view is larger and a dynamic range
10 in the depth direction becomes larger as the gradient is milder. Therefore, the feature amount coefficient F4 of the stereoscopic effect emphasis process is generated such that the distance between the positions of the distant view and the near view is emphasized as the gradient is smaller.
15 [0070]
A gradient from a ■■ffine texture toward a rough texture in a texture detection result is supposed as the image feature amount extracted for the stereoscopic effect emphasis process to emphasize the texture gradient by the fifth emphasis
20 processing unit 155. This gradient is the gradient of the feature amounts between a barycenter of the finest texture region and that of the roughest texture region of the two or more texture regions separated by a threshold. The feature amount coefficient F5 of the stereoscopic effect emphasis
25 process is generated such that the depth is greater and this is more emphasized as the gradient is larger. [0071]
The number of cases in which absolute difference in the spatial direction of the depth information is not smaller than
30 a threshold, that is to say, a target area is supposed as the image feature amount extracted for the stereoscopic effect

28
SP346516WO00
emphasis process by shading by the sixth emphasis processing unit 156. The feature amount coefficient F6 of the stereoscopic effect emphasis process is generated such that the distance between the positions- of the foreground and the 5 background is larger and this is more emphasized as the target area is wider. [0072]
An area ratio of a distribution between the foreground and the background in a frequency distribution of the depth
10 information is supposed as the image feature amount extracted for the stereoscopic effect emphasis process according to the focus by the seventh emphasis processing unit 157. For example, when the area of the foreground is wider than the area of the background, the presence of a main subject to be
15 emphasized is estimated, so that the feature amount
coefficient F7 of the stereoscopic effect emphasis process
■i , is generated such that this is emphasized.
[0073]
Fig. 18 is a view illustrating a configuration example
20 of the general coefficient table 144 of the embodiment of the present technology. The general coefficient table 144 holds the general coefficients Gl to G7 for each of the above-described seven stereoscopic effect emphasis processes for each scene. The above-described three scenes
25 of the landscape; the structure, the street and the like; and the portrait are illustrated. [0074]
Fig. 19 is a flow diagram illustrating a procedure by the second configuration example of the image analyzing unit
30 140 of the embodiment of the present technology. [0075]

29
SP346516WO00
The scene recognition processing unit 143 first recognizes the scene estimated from the input image (step S943) . Then, the general coefficient table 144 is searched according to the recognized scene and the general 5 coefficients Gl to G7 of the stereoscopic effect emphasis processes are output (step S944). [0076]
The feature amount coefficient generating unit 145 generates the feature amount coefficients Fl to F7 of the 10 image from the input image (step S945). [0077]
Then, the binding coefficient generating unit 14 6 generates the integration coefficients wl to w7 from the general coefficients Gl to G7 output from the general 15 coefficient table 144 and the feature amount coefficients Fl to F7 generated by the feature amount coefficient generating unit 145 (step S946). [0078]
[Third Configuration Example of Image Analyzing Unit] 20 Fig. 20 is a view illustrating a third configuration example of the image analyzing unit 140 of the embodiment of the present technology. The third configuration example of the image analyzing unit 140 is provided with a feature amount coefficient generating unit 147, a scene recognition 25 processing unit 148, and a binding coefficient generating unit 149. [0079]
The feature amount coefficient generating unit 147 is configured to generate the feature amount coefficients Fl to 30 F7 of the image from the input image. The feature amount coefficients Fl to F7 are the coefficients based on the

30
SP346516WO00
feature amount of the image and are specifically obtained from a pixel value of the image, the feature amount of the image, and the depth of the depth information. Contents of the process by the feature amount coefficient generating unit 147 5 are similar to those of the above-described feature amount coefficient generating unit 145. [0080]
The scene recognition processing unit 148 is configured to estimate a specific scene based on the feature amount
10 coefficients Fl to F7 generated by the feature amount
coefficient generating unit 147. For example, when a weight of the aerial perspective is not smaller than a threshold, the image is estimated to be that of the landscape. When a weight of the perspective is not smaller than a threshold,
15 the image is estimated to be that of the structure. When the weights of the aerial perspective and the perspective are not larger than the thresholds, the image is estimated to be that of the portrait. [0081]
20 The binding coefficient generating unit 149 is
configured to adjust the feature amount coefficients Fl to F7 to generate the integration coefficients wl to w7 according to the scene estimated by the scene recognition processing unit 148. For example, when the image is estimated to be that
25 of the landscape, the integration coefficients wl, w2, and w6 of the stereoscopic effect emphasis processes of the relative size, the perspective, and the shading are inhibited and other integration coefficients are enhanced. When the image is estimated to be that of the structure, the
30 integration coefficient w4 of the stereoscopic effect
emphasis process of the aerial perspective is inhibited and

31
SP346516WO00
other integration coefficients are enhanced. When the image is estimated to be that of the portrait, the integration coefficients wl and w3 of the stereoscopic effect emphasis processes of the relative size and the motion are inhibited 5 and other integration coefficients are enhanced. [0082]
Fig. 21 is a flow diagram illustrating a procedure by the third configuration example of the image analyzing unit 140 of the embodiment of the present technology. 10 [0083]
First, the feature amount coefficient generating unit 147 generates the feature amount coefficients Fl to F7 of the image from the input image (step S 9 4 7) . Then, the s cene recognition processing unit 148 estimates the scene of the 15 image based on the generated feature amount coefficients Fl to F7 (step S948). Thereafter, the binding coefficient generating unit 149 adjusts the feature amount coefficients Fl to F7 to generate the integration coefficients wl to w7 according to the estimated scene (step S949). 20 [0084]
[Fourth Configuration Example of Image Analyzing Unit] Fig. 22 is a view illustrating a fourth configuration example of the image analyzing unit 140 of the embodiment of the present technology. The fourth configuration example of 25 the image analyzing unit 140 is provided with a feature amount coefficient generating unit 240 and five scene determination processing units 241 to 245. [0085]
The feature amount coefficient generating unit 24 0 is
30 configured to generate the feature amount coefficients Fl to
F7 of the image from the input image. The feature amount

32
SP346516WO00
coefficients Fl to F7 are the coefficients based on the feature amount of the image and are specifically obtained from a pixel value of the image, the feature amount of the image, and the depth of the depth information. The contents of the 5 process by the feature amount coefficient generating unit 24 0 are similar to those of the above-described feature amount coefficient generating unit 145. [0086]
The scene determination processing units 241 to 245 are
10 configured to determine whether the scene corresponds to the supposed scene based on the feature amount coefficients Fl to F7 generated by the feature amount coefficient generating unit 240, and when this corresponds, adjust the feature amount coefficients to output as the integration coefficients wl to
15 w7. Although the feature amount coefficients Fl to F7 are referred to and are adjusted to be output as the integration coefficients wl to w7 in the following description, it may also be interpreted that the feature amount coefficients Fl to F7 generated by the feature amount coefficient generating
20 unit 240 are referred to as the integration coefficients to be adjusted. [0087]
The first scene determination processing unit 241 is configured to refer to the feature amount coefficients Fl and
25 F2 to determine whether the scene is the supposed scene. Herein, a scene in which a person is imaged with a deep background is supposed and it is determined whether a condition that the feature amount coefficient Fl is larger than a threshold Thl and the feature amount coefficient F2
30 is larger than a threshold Th2 is satisfied. When the
condition is satisfied, the feature amount coefficients Fl

33
SP346516WO00
and F2 are increased. According to this, it becomes possible to emphasize the stereoscopic effect of the deep image. [0088]
The second scene determination processing unit 242 is 5 configured to refer to the feature amount coefficients F5 and F7 to determine whether the scene is the supposed scene. Herein, a scene with a sharp texture gradient is supposed and it is determined whether a condition that the feature amount coefficient F5 is larger than a threshold Th5 and the feature
10 amount coefficient F7 is larger than a threshold Th7 is satisfied. When the condition is satisfied, the feature amount coefficients F5 and F7 are increased. According to this, it becomes possible to emphasize the stereoscopic effect of the image with the sharp texture gradient. For
15 example, an image in which a front side of lawns is in focus, an image of a person wearing patterned clothing, a brick house and the like might correspond to this. [0089]
The third scene determination processing unit 243 is
2 0 configured to refer to the feature amount coefficients F4 and F6 to determine whether the scene is the supposed scene. The stereoscopic effect emphasis process based on the aerial perspective and the stereoscopic effect emphasis process by shading often conflict with each other, so that the feature
25 amount coefficients F4 and F6 are compared with each other and an effect of the stereoscopic effect emphasis process with a smaller feature amount coefficient is inhibited. For example, when the feature amount coefficient F6 is smaller, the image is considered to be the landscape image, so that
30 the feature amount coefficient F6 is further decreased such that the shading is not emphasized. When the feature amount

34
SP346516WO00
coefficient F4 is smaller, the image is considered to be an indoor image or the portrait, so that the feature amount coefficient F4 is further decreased such that the stereoscopic effect emphasis process based on the aerial 5 perspective is not emphasized. As a result of the
determination by the third scene determination processing unit 243, when the feature amount coefficient F6 is smaller, a process by the fourth scene determination processing unit 244 is executed, and when the feature amount coefficient F4
10 is smaller, a process by the fifth scene determination processing unit 245 is executed. [0090]
The fourth scene determination processing unit 244 is configured to refer to the feature amount coefficients Fl or
15 F2 and F4 to determine whether the scene is the supposed scene. Herein, the landscape image is supposed and the feature amount coefficients Fl or F2 and F4 are adjusted. When the feature amount coefficients Fl and F2 are increased by the first scene determination processing unit 241, an average value of them
20 is used as a reference, and otherwise, a larger one of the feature amount coefficients Fl and F2 is used as the reference to be compared with the feature amount coefficient F4 . When the feature amount coefficient F4 is larger, the object is considered to be located on the background side, so that the
25 feature amount coefficients Fl and F2 are inhibited. On the other hand, when the feature amount coefficient F4 is smaller, the foreground object is considered to be located nearby, so that the feature amount coefficient F4 is inhibited. [0091]
30 The fifth scene determination processing unit 245 is configured to refer to the feature amount coefficients F5 or

35
SP346516WO00
F7 and F6 to determine whether the scene is the supposed scene. Herein, the indoor image is supposed and the feature amount coefficients F5 or F7 and F6 are adjusted. When the feature amount coefficients F5 and F7 are increased by the second 5 scene determination processing unit 242, an average value of them is used as the reference, and otherwise, a larger one of the feature amount coefficients F5 and F7 is used as the reference to be compared with the feature amount coefficient F6. When the feature amount coefficient F6 is larger, the
10 image is considered to be a wide-angle image, so that the feature amount coefficients F5 and F7 are inhibited such that the shading is made sharper. On the other hand, when the feature amount coefficient F6 is smaller, the image is considered to be a micro image, so that the feature amount
15 coefficient F6 is inhibited such that the edge is made sharper. [0092]
The feature amount coefficients Fl to F7 adjusted in this manner are supplied to the stereoscopic effect emphasis
20 processing unit 150 and the integration processing unit 160 as the integration coefficients wl to w7. Meanwhile, the feature amount coefficient F3 is directly supplied as the integration coefficient w3 without special adjustment in the fourth configuration example.
25 [0093]
Fig. 23 is a flow diagram illustrating a procedure by the fourth configuration example of the image analyzing unit 140 of the embodiment of the present technology. First, the feature amount coefficient generating unit 240 generates the
30 feature amount coefficients Fl to F7 of the image from the input image (step S801).

36
SP346516WO00
[0094]
Then, the first scene determination processing unit 241 determines whether both the feature amount coefficients Fl and F2 are larger than the predetermined thresholds (step 5 S811) . That is to say, when the feature amount coefficient Fl is larger than the threshold Thl and the feature amount coefficient F2 is larger than the threshold Th2 (Yes at step S811), the feature amount coefficients Fl and F2 are emphasized (step S812).
10 [0095]
The second scene determination processing unit 242 determines whether both the feature amount coefficients F5 and F7 are larger than the predetermined thresholds {step S821) . That is to say, when the feature amount coefficient
15 F5 is larger than a threshold Th5 and the feature amount coefficient F7 is larger than a threshold Th7 (Yes at step S821), the feature amount coefficients F5 and F7 are emphasized (step S822). [0096]
2 0 Next, the third scene determination processing unit 243 determines a larger one of the feature amount coefficients F4 and F6 (step S831) . At that time, when the feature amount coefficient F6 is determined to be larger (Yes at step S831) , the feature amount coefficient F4 is inhibited (step S833) .
25 On the other hand, when the feature amount coefficient F4 is determined to be larger (No at step S831), the feature amount coefficient F6 is inhibited (step S832) . [0097]
When the feature amount coefficient F6 is inhibited at
30 step S832, the fourth scene determination processing unit 244 refers to the feature amount coefficients Fl or F2 and F4 to

37
SP346516WO00
perform the following determination. That is to say, when the feature amount coefficients Fl and F2 are increased by the first scene determination processing unit 241, the average value of them is used as the reference, and otherwise, 5 the larger one of the feature amount coefficients Fl and F2 is used as the reference to be compared with the feature amount coefficient F4 (step S841). When the feature amount coefficient F4 is larger (Yes at step S841), the feature amount coefficients Fl and F2 are inhibited (step S842). On
10 the other hand, when the feature amount coefficient F4 is smaller (No at step S841) , the feature amount coefficient F4 is inhibited (step S843). [0098]
On the other hand, when the feature amount coefficient
15 F4 is inhibited at step S833, the fifth scene determination processing unit 245 refers to the feature amount coefficients F5 or F7 and F6 and performs the following determination. That is to say, when the feature amount coefficients F5 and F7 are increased by the second scene determination processing
20 unit 242, the average value of them is used as the reference, and otherwise, the larger one of the feature amount coefficients F5 and F7 is used as the reference to be compared with the feature amount coefficient F6 (step S851) . When the feature amount coefficient F6 is larger (Yes at step S851),
25 the feature amount coefficients F5 and F7 are inhibited such that the shading is made sharper (step S852) . On the other hand, when the feature amount coefficient F6 is smaller (No at step S851), the feature amount coefficient F6 is inhibited such that the edge is made sharper (step S853) .
30 [0099]
In this manner, in the fourth configuration example of

38
SP346516WO00
the image analyzing unit 140, a more specific scene is estimated, the feature amount coefficients Fl to F7 are adjusted such that the stereoscopic effect emphasis process suitable for this is performed, and the integration 5 coefficients wl to w7 are generated based on this. [0100]
<4. Integration Process>
The stereoscopic effect emphasis processing unit 150 and the integration processing unit 160 perform the following
10 processes according to the integration coefficients wl to w7 generated by the above-described image analyzing unit 140. [0101]
[Process of First Emphasis Processing Unit (Relative Size)]
15 As described above, the first emphasis processing unit 151 performs the stereoscopic effect emphasis process to control the relative sizes of the objects. The input image and the depth information are input as input information to the first emphasis processing unit 151. A coordinate of the
20 region to be processed is input as the additional information. The integration coefficient wl generated by'the image analyzing unit 140 is input as the control parameter. A weight calculated from the area ratio between the objects may be reflected in the integration coefficient wl.
25 [0102]
When the integration coefficient wl is multiplied by a limit scaling factor zl of enlargement or reduction obtained by the emphasis process, a final limit scaling factor zl * wl is obtained. As illustrated in Fig. 24, a scaling factor
30 Zl changes according to the depth information of the region to be processed. In this drawing, a range of the scaling

39
SP346516WO00
factor changes according to the integration coefficient wl.
Therefore/ it is possible to obtain the scaling factor Zl from
the integration coefficient wl and the depth information.
[0103] 5 The first emphasis processing unit 151 outputs the
coordinate of the region to be processed and the scaling
factor Zl of enlargement or reduction as output information.
This may also output the depth information and the integration
coefficient wl. 10 [0104]
[Process of Second Emphasis Processing Unit
(Perspective)]
As described above, the second emphasis processing unit
152 performs the stereoscopic effect emphasis process to 15 control the position of the object based on the perspective.
The input image and the depth information are injput as the
input information to the second emphasis processing unit 152 .
The coordinate of the vanishing point (presence thereof) , the
number of the straight lines forming the vanishing point/ and 20 the coordinate of the region to be processed are input as the
additional information. The integration coefficient w2
generated by the image analyzing unit 140 is input as the
control parameter. A weight calculated from the coordinate
of the vanishing point (presence thereof) and the number of 25 the straight lines forming the vanishing point may be
reflected in the integration coefficient w2.
[0105]
When the integration coefficient w2 is multiplied by
a motion amount limit v2 obtained by the emphasis process, 30 a final limit motion amount v2 * w2 is obtained. As
illustrated in Fig. 25/ a motion amount V2 changes according

40
SP346516WO00
to the depth information of the region to be processed. In this drawing, a range of the motion amount changes according to the integration coefficient w2 . Therefore, it is possible to obtain the motion amount V2 from the integration 5 coefficient w2 and the depth information. [0106]
The second emphasis processing unit 152 outputs the coordinate of the region to be processed and the motion amount V2 for perspective control as the output information. This
10 may also output the depth information and the integration coefficient w2. [0107]
[Process of Third Emphasis Processing Unit (Motion of Object)]
15 As described above, the third emphasis processing unit 153 performs the stereoscopic effect emphasis process to control the motion of the object. The input image and the depth information are input as the input information to the third emphasis processing unit 153 . The motion vector of each
20 region such as the foreground and the background and the coordinate of the region to be processed are input as the additional information. The integration coefficient w3 generated by the image analyzing unit 140 is input as the control parameter. A weight calculated from the ratio
25 between the vector of the foreground and that of the
background may be reflected in the integration coefficient
w3.
[0108]
When the integration coefficient w3 is multiplied by
30 a motion amount limit v3 obtained by the emphasis process, a final limit motion amount v3 * w3 is obtained. As

41
SP346516WO00
illustrated in Fig. 26, a motion amount V3 changes according to the depth information of the region to be processed. In this drawing, a range of the motion amount changes according to the integration coefficient w3. Therefore, it is possible 5 to obtain the motion amount V3 from the integration coefficient w3 and the depth information. [0109]
The third emphasis processing unit 153 outputs the coordinate of the region to be processed and the motion amount
10 V3 for the stereoscopic effect emphasis by the motion as the output information. This may also output the depth information and the integration coefficient w3. [0110]
[Process of Fourth Emphasis Processing Unit (Aerial
15 Perspective)]
As described above, the fourth emphasis processing unit 154 performs the stereoscopic effect emphasis process based on the aerial perspective. The input image and the depth information are input as the input information to the fourth
20 emphasis processing unit 154. The average value among the number of pixels in the horizontal direction of the average values of the respective vertical lines of the obtained gradients between the adjacent pixels in the vertical direction of the depth information is input as the additional
25 information. The integration coefficient w4 generated by the image analyzing unit 140 is input as the control parameter. A weight calculated from the average value among the number of pixels in the horizontal direction of the average values of the respective vertical lines of the obtained gradients
30 between the adjacent pixels in the vertical direction of the depth information may be reflected in the integration

42
SP346516WO00
coefficient W4. If the vertical direction and the depth direction are plotted along the ordinate axis and the abscissa axis, respectively/ the dynamic range in the depth direction becomes larger as the gradient is milder. In this case, since 5 it is more desired to emphasize the aerial perspective as the gradient is milder, so that the weight is made larger. [0111]
As for pixel value control in the emphasis process, the integration coefficient w4 is multiplied by a processing
10 result r4 and a final emphasis amount r4 x v/4 is obtained. [0112]
The fourth emphasis processing unit 154 outputs the emphasis processing result r4 of the aerial perspective as the output information. This may also output the depth
15 information and the integration coefficient w4. [0113]
[Process of Fifth Emphasis Processing Unit (Texture Gradient)]
As described above, the fifth emphasis processing unit
20 155 performs the stereoscopic effect emphasis process to emphasize the texture gradient. The input image and the depth information are input as the input information to the fifth emphasis processing unit 155. The gradient from the fine texture toward the rough texture in the texture detection
25 result is input as the additional information. The
integration coefficient w5 generated by the image analyzing unit 140 is input as the control parameter. The gradient from the fine texture toward the rough texture in the texture detection result may be reflected in the integration
30 coefficient w5. This gradient is obtained as the gradient of the feature amount between the barycenter of the finest

43

SP346516WO00

texture region and that of the roughest texture region of the two or more texture regions separated by the threshold. [0114]
As for the pixel value control in the emphasis process, 5 the integration coefficient w5 is multiplied by a processing result r5 and a final emphasis amount r5 * w5 is obtained. [0115]
The fifth emphasis processing unit 155 outputs the emphasis processing result r5 of the texture gradient as the
10 output information. This may also output the depth information and the integration coefficient w5. [0116]
[Process of Sixth Emphasis Processing Unit (Shading)] As described above, the sixth emphasis processing unit
15 156 performs the stereoscopic effect emphasis process by shading. The input image and the depth information are input as the input information to the sixth emphasis processing unit 156. The number of the cases in which the absolute difference in the spatial direction of the depth information is not
20 smaller than the threshold {or the target area) is input as the additional information. The integration coefficient w6 generated by the image analyzing unit 140 is input as the control parameter. A weight calculated from the number of the cases in which the absolute difference in the spatial
25 direction of the depth information is not smaller than the threshold (or the target area) may be reflected in the integration coefficient w6. [0117]
As for the pixel value control in the emphasis process,
30 the integration coefficient w6 is multiplied by an emphasis processing result r6 of the shading, and a final emphasis

44
SP346516WO00
amount r6 * w6 is obtained.
[0118]
The sixth emphasis processing unit 156 outputs the
emphasis processing result r6 of the shading as the output 5 information. This may also output the depth information and
the integration coefficient w6,
[0119]
[Process of Seventh Emphasis Processing Unit (Focus)]
As described above, the seventh emphasis processing
10 unit 157 performs the stereoscopic effect emphasis process
according to the focus. The input image and the depth
information are input as the input information to the seventh
emphasis processing unit 157. An area ratio of the mountain
between the foreground and the background in the frequency 15 distribution of the depth information is input as the
additional information. The integration coefficient w7
generated by the image analyzing unit 140 is input as the
control parameter. A weight calculated from the area ratio
of the mountain between the foreground and the background in 20 the frequency distribution of the depth information may be
reflected in the integration coefficient w7.
[0120]
As for the pixel value control in the emphasis process,
the integration coefficient w7 is multiplied by an emphasis 25 processing result r7 of the focus, and a final emphasis amount
r7 x w7 is obtained.
[0121]
The seventh emphasis processing unit 157 outputs the
emphasis processing result r7 of the focus as the output 30 information. This may also output the depth information and
the integration coefficient w7.

45
SP346516WO00
[0122]
[Process of Integration Processing Unit 160] The process by the integration processing unit 160 is largely separated into an integration process of the control 5 of the area or the position and the integration process of the control of the pixel value. The integration process of the control of the area or the position is to integrate the processing results of the first emphasis processing unit 151, the second emphasis processing unit 152, and the third
10 emphasis processing unit 153 . The integration process of the control of the pixel value is to integrate the processing results of the fourth emphasis processing unit 154, the fifth emphasis processing unit 155, the sixth emphasis processing unit 156, and the seventh emphasis processing unit 157.
15 [0123]
Fig. 27 is a view illustrating a procedure example of the integration process of the control of the area or the position by the integration processing unit 160 of the embodiment of the present technology.
20 [0124]
First, a sum of the three integration coefficients wl to w3 regarding the control of the area or the position is calculated as an integrated integration coefficient Wa (step S9611) . Then, a value obtained by dividing the scaling factor
25 Zl of enlargement or reduction of the stereoscopic effect emphasis process to control the relative size by the first emphasis processing unit 151 by the integrated integration coefficient Wa is calculated as a final scaling factor Z of enlargement or reduction (step S9612) . The motion amount V2
30 of the stereoscopic effect emphasis process to control the position of the object based on the perspective by the second

46

SP346516WO00

emphasis processing unit 152 and the motion amount V2 of the stereoscopic effect emphasis process to control the motion of the object by the third emphasis processing unit 153 are added up (step S9613). Then, a value obtained by dividing 5 an added result by the integrated integration coefficient Wa is calculated as a final motion amount V of the region (step S9613). [0125]
Meanwhile, when the control of the area or the position
10 is performed on the image according to the scaling factor Z and the motion amount V finally obtained in this manner, the depth information and the additional information are also adjusted so as to correspond to this such that inconsistency is avoided.
15 [0126]
Fig. 28 is a view illustrating a procedure example of the integration process of the control of the pixel value by the integration processing unit 160 of the embodiment of the present technology.
20 [0127]
First, a sum of the four integration coefficients w4 to w7 regarding the control of the pixel value is calculated as an integrated integration coefficient Wb (step S9621). Then, values obtained by multiplying the processing results
25 r4 to r7 by the fourth emphasis processing unit 154, the fifth emphasis processing unit 155, the sixth emphasis processing unit 156, and the seventh emphasis processing unit 157 by the integration coefficients w4 to w7 corresponding to them, respectively, are added up (step S9622). Then, a value
30 obtained by dividing an added result by the integrated
integration coefficient Wb is calculated as a final pixel

47
SP346516WO00
value I '(step S9622). [0128]
The integration process of the control of the area or the position and the integration process of the control of 5 the pixel value herein described are different processes and the order of execution thereof is indifferent. However, it is required to perform a cascade process such that the one process is executed after the other process is completed. [0129]
10 <5. Entire Operation>
Entire operation of the image display device of the embodiment of the present technology heretofore described is hereinafter described. [0130]
15 Fig- 29 is a view illustrating a first entire procedure example of the image display device of the embodiment of the present technology. In this example, the stereoscopic effect emphasis processes are sequentially executed in order. [0131]
20 The image analyzing unit 140 performs an image analysis process on the input image supplied from the input image supplying unit 110 (step S940). Then, a control variable i is set to "1" (step S901) and the stereoscopic effect emphasis processing unit 150 executes an i-th stereoscopic effect
25 emphasis process (step S950) while the control variable i is incremented by "1" (step S903) until this becomes larger than N (step S902) . Herein, N is an integer and is set to "7" in the above-described example. Thereafter, the integration processing unit 160 performs the integration process (step
30 S960) . [0132]

48
SP346516WO00
Fig. 30 is a view illustrating a second entire procedure example of the image display device of the embodiment of the present technology. In this example, the stereoscopic effect emphasis processes are executed in parallel regardless 5 of the order. [0133]
The image analyzing unit 140 performs an image analysis process on the input image supplied from the input image supplying unit 110 (step S940). Then, the N stereoscopic 10 effect emphasis processes are executed in parallel (step S951) . Thereafter, the integration processing unit 160 performs the integration process (step S960) . [0134]
Fig. 31 is a view illustrating a third entire procedure 15 example of the image display device of the embodiment of the present technology. This example clearly describes that the integration process of the control of the area or the position and the integration process of the control of the pixel value are separately performed. 20 [0135]
The image analyzing unit 140 performs an image analysis process on the input image supplied from the input image supplying unit 110 (step S940). Then, the control variable i is set to "1" (step S904) and an i-th stereoscopic effect 25 emphasis process regarding the control of the area or the position is executed (step S952) while the control variable i is incremented by "1" (step S906) until this becomes larger than M (step S905) . Herein, M is an integer and is set to "3" in the above-described example. Thereafter, the 30 integration processing unit 160 performs the integration process regarding the control of the area or the position

49
SP346516WO00
(step S961) . [0136]
Then/ the control variable i is set to "1" (step S907) , and an i-th stereoscopic effect emphasis process regarding 5 the control of the pixel value is executed (step S953) while the control variable i is incremented by "1" (step S909) until this becomes larger than N (step S908). Herein, N is an integer and is set to "4" in the above-described example. Thereafter, the integration processing unit 160 performs the 10 integration process regarding the control of the pixel value (step S962). [0137]
Fig. 32 is a view illustrating a fourth entire procedure example of the image display device of the embodiment of the 15 present technology. This example clearly describes that the integration process of the control of the area or the position and the integration process of the control of the pixel value are separately performed and that the stereoscopic effect emphasis processes are executed in parallel regardless of the 20 order. [0138]
The image analyzing unit 140 performs an image analysis process on the input image supplied from the input image supplying unit 110 (step S940). Then, the M stereoscopic 25 effect emphasis processes regarding the control of the area or the position are executed in parallel (step S954). Thereafter, the integration processing unit 160 performs the integration process regarding the control of the area or the position (step S961). 30 [0139]
Then, the N stereoscopic effect emphasis processes

50
SP346516WO00
regarding the control of the pixel value are executed in parallel (step S955). Thereafter, the integration processing unit 160 performs the integration process regarding the control of the pixel value (step S962). 5 [0140]
Although the example in which the integration process of the control of the area or the position and the integration process of the control of the pixel value are executed in series (Fig. 31) and the example in which they are executed
10 in parallel (Fig. 32) are herein described, it is also
possible to execute one of them in series and the other of them in parallel. The order of the integration process of the control of the area or the position and the integration process of the control of the pixel value may be changed. The
15. process flow may be selected according to a mode of implementation of hardware or software. [0141]
In this manner, according to the embodiment of the present technology, the stereoscopic effect emphasis
20 processes are integrated based on the integration coefficient generated by the image analyzing unit 140, so that it is possible to integrate a plurality of stereoscopic effect emphasis processes in a mode suitable for the image. [0142]
25 Meanwhile, the above-described embodiment describes an example of embodying the present technology, and there is a correspondence relationship between items of the embodiment and the matters specifying the invention in claims. Similarly, there is a correspondence relationship between the
30 matters specifying the invention in claims and the items of the embodiment of the present technology having the same name.

51
SP346516WO00
However, the present technology is not limited to the embodiment and may be embodied with various modifications of the embodiment without departing from the spirit thereof. [0143] 5 The procedures described in the above-described
embodiment may be considered as a method having the series of procedures and may be considered as a program for allowing a computer to execute the series of procedures and a recording medium which stores the program. A CD (Compact Disc) , an MD
10 (MiniDisc), a DVD (digital versatile disk), a memory card, a Blu-ray (registered trademark) Disc and the like may be used, for example, as the recording medium. [0144]
Meanwhile, this technology may also have a following
15 configuration.
(1) An image processing device including a stereoscopic effect emphasis processing unit which executes a plurality of stereoscopic effect emphasis processes on an input image, an image analyzing unit which analyzes the input image to
20 generate an integration coefficient used when the plurality of stereoscopic effect emphasis processes is integrated, and an integration processing unit which integrates results of the plurality of stereoscopic effect emphasis processes according to the integration coefficient.
25 (2) The image processing device according to the (1),
wherein the image analyzing unit recognizes a scene of the input image and generates the integration coefficient based on the recognized scene.
(3) The image processing device according to the (2),
30 wherein the image analyzing unit includes a feature amount generating unit which generates a feature amount included in

52

SP346516WO00

the input image and generates the integration coefficient based on the recognized scene and the feature amount.
(4) The image processing device according to any one
of the (1) to (3)/ wherein, when the integration coefficient
5 of a stereoscopic effect emphasis process to control a relative size and the integration coefficient of a stereoscopic effect emphasis process to perform positional control based on perspective are higher than predetermined thresholds/ the image analyzing unit further increases the 10 integration coefficient of the stereoscopic effect emphasis process to control the relative size and the integration coefficient of the stereoscopic effect emphasis process to perform the positional control based on the perspective.
(5) The image processing device according to any one
15 of the (1) to (4), wherein, when the integration coefficient
of a stereoscopic effect emphasis process to emphasize a texture gradient and the integration coefficient of a stereoscopic effect emphasis process according to a focus are higher than predetermined thresholds, the image analyzing 20 unit further increases the integration coefficient of the stereoscopic effect emphasis process to emphasize the texture gradient and the integration coefficient of the stereoscopic effect emphasis process according to the focus.
(6) The image processing device according to any one
25 of the (1) to (5), wherein the image analyzing unit inhibits
a lower integration coefficient of the integration
coefficient of a stereoscopic effect emphasis process based
on aerial perspective and the integration coefficient of a
stereoscopic effect emphasis process by shading.
30 (7) The image processing device according to any one
of the (1) to (6), wherein the image analyzing unit inhibits

53
SP346516WO00
a lower integration coefficient of the integration coefficient of the stereoscopic effect emphasis process by shading and the integration coefficient of the stereoscopic effect emphasis process to emphasize the texture gradient or 5 the integration coefficient of the stereoscopic effect emphasis process according to the focus.
(8) The image processing device according to any one
of the (1) to (7), wherein the image analyzing unit inhibits
a lower integration coefficient of the integration
10 coefficient of the stereoscopic effect emphasis process based on the aerial perspective and the integration coefficient of the stereoscopic effect emphasis process to control the relative size or the integration coefficient of the stereoscopic effect emphasis process to perform the
15 positional control based on the perspective.
(9) The image processing device according to any one
of the (1) to (8)/ wherein the integration processing unit
separately performs integration of a stereoscopic effect
emphasis processes regarding a position and integration of
20 a stereoscopic effect emphasis processes regarding a pixel value of the plurality of stereoscopic effect emphasis processes in order.
(10) An image display device including a stereoscopic
effect emphasis processing unit which executes a plurality
25 of stereoscopic effect emphasis processes on an input image, an image analyzing unit which analyzes the input image to generate an integration coefficient used when the plurality of stereoscopic effect emphasis processes is integrated, an integration processing unit which integrates results of the
30 plurality of stereoscopic effect emphasis processes
according to the integration coefficient, and a display unit

54
SP346516WO00
which displays an image obtained by integrating the results of the plurality of stereoscopic effect emphasis processes in the input image.
(11) An image processing method including a
5 stereoscopic effect emphasis process step of executing a plurality of stereoscopic effect emphasis processes on an input image, an image analyzing step of analyzing the input image to generate an integration coefficient used when the plurality of stereoscopic effect emphasis processes is 10 integrated, and an integration process step of integrating results of the plurality of stereoscopic effect emphasis processes according to the integration coefficient.
(12) A program which allows a computer to execute a
stereoscopic effect emphasis process step of executing a
15 plurality of stereoscopic effect emphasis processes on an input image, an image analyzing step of analyzing, the input image to generate an integration coefficient used when the plurality of stereoscopic effect emphasis processes is integrated, and an integration process step of integrating
20 results of the plurality of stereoscopic effect emphasis processes according to the integration coefficient.
REFERENCE SIGNS LIST
[0145] 25 110 Input image supplying unit
120 Depth information obtaining unit
130 Additional information obtaining unit
140 Image analyzing unit
141,143,148 Scene recognition processing unit 30 142 Binding coefficient table
144 General coefficient table

55
SP346516WO00
145/147 Feature amount coefficient generating unit 146/149 Binding coefficient generating unit
150 Stereoscopic effect emphasis processing unit
151 to 157 Emphasis processing unit
160 Integration processing unit
170 Display unit
240 Feature amount coefficient generating unit
241 to 245 Scene determination processing unit

56
SP346516WO00

CLAIMS
1. An image processing device comprising:
a stereoscopic effect emphasis processing unit which 5 executes a plurality of stereoscopic effect emphasis processes on an input image; and
an integration processing unit which integrates results of the plurality of stereoscopic effect emphasis processes according to an integration coefficient used when 10 the plurality of stereoscopic effect emphasis processes is integrated obtained by analyzing the input image.
2. The image processing device according to claim 1,
further comprising: an image analyzing unit which analyzes
15 the input image to generate the integration coefficient used when the plurality of stereoscopic effect emphasis processes is integrated.
3. The image processing device according to claim 2,
20 wherein the image analyzing unit generates the integration
coefficient based on a scene of the input image.
4. The image processing device according to claim 3,
wherein the image analyzing unit generates the integration
25 coefficient based on the recognized scene and a feature amount included in the input image.
5. The image processing device according to claim 1,
wherein, when the integration coefficient of a stereoscopic
30 effect emphasis process to control a relative size and the integration coefficient of a stereoscopic effect emphasis

57
SP346516WO00
process to perform positional control based on perspective are higher than predetermined threshold, the image analyzing unit further increases the integration coefficient of the stereoscopic effect emphasis process to perform the 5 positional control.
6. The image processing device according to claim 1,
wherein, when the integration coefficient of a stereoscopic
effect emphasis process to emphasize a texture gradient and
10 the integration coefficient of a stereoscopic effect emphasis process according to a focus are higher than predetermined thresholds, the image analyzing unit further increases the integration coefficient of the stereoscopic effect emphasis process to emphasize the texture gradient and the integration
15 coefficient of the stereoscopic effect emphasis process according to the focus.
7. The image processing device according to claim 1,
wherein the image analyzing unit inhibits a lower integration
20 coefficient of the integration coefficient of a stereoscopic effect emphasis process based on aerial perspective and the integration coefficient of a stereoscopic effect emphasis process by shading.
25 8. The image processing device according to claim 1,
wherein the image analyzing unit inhibits a lower integration coefficient of the integration coefficient of a stereoscopic effect emphasis process by shading and the integration coefficient of a stereoscopic effect emphasis process to
30 emphasize a texture gradient or the integration coefficient of a stereoscopic effect emphasis process according to a

58
SP346516WO00
focus.
9. The image processing device according to claim 1,
wherein the image analyzing unit inhibits a lower integration
5 coefficient of the integration coefficient of a stereoscopic effect emphasis process based on aerial perspective and the integration coefficient of a stereoscopic effect emphasis process to cpntrol a relative size or the integration coefficient of a stereoscopic effect emphasis process to 10 perform positional control based on perspective.
10. The image processing device according to claim 1/
wherein the integration processing unit separately performs
integration of stereoscopic effect emphasis processes
15 regarding a position and integration of stereoscopic effect ■emphasis processes regarding a pixel value of the plurality of stereoscopic effect emphasis processes in order.
11. The image processing device according to claim 1,
20 further comprising: a display unit which displays an image
obtained by integrating the results of the plurality of stereoscopic effect emphasis processes in the input image.
12. An image processing method comprising: a stereoscopic
25 effect emphasis process step of executing a plurality of
stereoscopic effect emphasis processes on an input image; and an integration process step of integrating results of the plurality of stereoscopic effect emphasis processes according to an integration coefficient used when the 30 plurality of stereoscopic effect emphasis processes is integrated obtained by analyzing the input image.
Dated this 05/03/2015
[RANJNA MEHTA-DUTT]
OF REMFRY & SAGAR
ATTORNEY FOR THE APPLICANTS]