Specification
[0001]
The present technology relates to an image processing apparatus, an image processing method, and a program, and more particularly, to an image processing apparatus, an image processing method, and a program capable of detecting a parallax of a stereoscopic image formed by right and left images more accurately.
BACKGROUND
[0002]
A parallax obtained from a stereoscopic image formed by right and left images corresponds to a depth from a user who views the stereoscopic image to a subject. Therefore, parallax detection of a stereoscopic image can be applied to various kinds of application software such as parallax adjustment or viewpoint synthesis of a stereoscopic image and a three-dimensional interface. The parallax detection of a stereoscopic image is associated with a problem of detecting corresponding points in right and left images (for example, see Japanese Unexamined Patent Application Publication No. 2009-008539).
SUMMARY
[0004]
However, when there are a plurality of candidates for a reliable corresponding point on one image of a stereoscopic image with respect to a point on the other image of a stereoscopic image, such as when there is a flat or repetitively patterned portion on the image, it is difficult to match the points accurately, and thus the parallax may not be detected with high accuracy.
[0005]
For example, when the side planes of an object on an image or a plurality of objects overlap each other in a depth direction, the region of a specific subject is present only in one image of the stereoscopic image and the parallax is rarely detected by a method of detecting the corresponding points in the region.
[0006]
It is desirable to provide an image processing apparatus, an image processing method, and a program capable of detecting a parallax of a stereoscopic image more accurately.
[0007]
According to an embodiment of the present technology, there is provided an image processing apparatus including: a parallax map generation unit that generates a parallax map indicating a parallax of each pixel of a stereoscopic image based on right and left images forming the stereoscopic image; a reliability calculation unit that generates a reliability map indicating reliability of the parallax map based on the right and left images; a depth information estimation unit that generates a depth information map indicating a depth of a subject on the stereoscopic image based on one of the right and left images; a depth parallax conversion unit that converts the depth information map into a pseudo-parallax map indicating a parallax corresponding to the depth; and a parallax synthesis unit that synthesizes the parallax map and the pseudo-parallax map to generate a corrected parallax map based on the reliability map.
[0008]
The parallax synthesis unit may generate the corrected parallax map by adding the parallax map and the pseudo-parallax map in a weighted manner using a weight determined by the reliability map.
[0009]
The parallax synthesis unit may include a component synthesis unit that synthesizes a high-pass component of the pseudo-parallax map and a low-pass component of the parallax map to generate a synthesized map; and an adding unit that synthesizes the parallax map and the synthesized map to generate the corrected parallax map based on the reliability map.
[0010]
The parallax synthesis unit may include a component synthesis unit that synthesizes a low-pass component of the pseudo-parallax map and a high-pass component of the parallax map to generate a synthesized map; and an adding unit that synthesizes the parallax map and the synthesized map to generate the corrected parallax map based on the reliability map.
[0011]
The image processing apparatus may further include a residual map generation unit that generates a residual map indicating a residual between a region near a predetermined pixel of one of the right and left images and a region neur a corresponding pixel of the other image which corresponds to the predetermined pixei. The reliability calculation unit may generate the reliability map based on the residual map.
[0012]
The parallax map generation unit may generate the parallax map by detecting, as a pixel corresponding to a predetermined pixel of one of the right and left images, a pixel of the other image for which a residual between a region near the predetermined pixel of the one image and a region near a pixel of the other image is minimum. The image processing apparatus may further include a peak sharpness map generation unit that generates a peak sharpness map that indicates sharpness of a peak of the residual of the pixels lined up in a predetermined direction on the other image and is used for generating the reliability map.
[0013]
The image processing apparatus may further include a matching degree calculation unit that generates a matching degree map indicating a matching degree between a predetermined pixel of one of the right and left images and a first corresponding pixel of the other image corresponding to the predetermined pixel based on a detection result of the first corresponding pixel of the other image and a detection result of a second corresponding pixel of the one image corresponding to the first corresponding pixel of the other image. The reliability calculation unit may generate the reliability map based on the matching degree map.
[0014]
According to another embodiment of the present technology, there is provided an image processing method or a program that includes generating a parallax map indicating a parallax of each pixel of a stereoscopic image based on right and left images forming the stereoscopic image; generating a reliability map indicating reliability of the parallax map based on the right and left images; generating a depth information map indicating a depth of a subject on the stereoscopic image based on one of the right and left images; and converting the depth information map into a pseudo-parallax map indicating a parallax corresponding to the depth and synthesizing the parallax map and the pseudo-parallax map to generate a corrected parallax map based on the reliability map.
[0015]
According to still another embodiment of the present technology, a parallax map indicating a parallax of each pixel of a stereoscopic image is generated based on right and left images forming the stereoscopic image; a reliability map indicating reliability of the parallax map is generated based on the right and left images; a depth information map indicating a depth of a subject on the stereoscopic image is generated based on one of the right and left images; and the depth information map is converted into a pseudo-parallax map indicating a parallax corresponding to the depth, and the parallax map and the pseudo-parallax map are synthesized to generate a corrected parallax map based on the reliability map.
[0016]
According to still another embodiment of the present technology, there is provided an image processing apparatus including: a depth information estimation unit that generates a depth information map indicating a depth of a subject on a stereoscopic image based on one of right and left images forming the stereoscopic image; a depth parallax conversion unit that converts the depth information map into a pseudo-parallax map indicating a parallax corresponding to the depth; and a parallax map generation unit that generates a parallax map indicating the parallax of each pixel of the stereoscopic image based on the right and left images and the pseudo-parallax map.
[0017]
According to still another embodiment of the present technology, there is provided an image processing method or a program that includes generating a depth information map indicating a depth of a subject on a stereoscopic image based on one of right and left images forming the stereoscopic image; and converting the depth information map into a pseudo-parallax map indicating a parallax corresponding to the depth and generating a parallax map indicating the parallax of each pixel of the stereoscopic image based on the right and left images and the pseudo-parallax map.
[0010]
According to still another embodiment or the present technology, a depth information map indicating a depth of a subject on a stereoscopic image is generated based on one of right and left images forming the stereoscopic image, the depth information map is converted into a pseudo-parallax map indicating a parallax corresponding to the depth, and a parallax map indicating the parallax of each pixel of the stereoscopic image is generated based on the right and left images and the pseudo-parallax map.
[0019]
According to the embodiments of the present technology described above, the parallax of a stereoscopic image can be detected more accurately.
BRIEF DESCRIPTION OF THE DRAWINGS
[0020]
FIG 1 is a diagram illustrating an example of the configuration of an image processing apparatus according to an embodiment;
FIG 2 is a diagram illustrating an example of the configuration of a parallax detection unit;
FIG 3 is a diagram illustrating an example of the configuration of a parallax synthesis unit;
FIG 4 is a flowchart illustrating a parallax detection process;
FIG 5 is a flowchart illustrating a parallax map generation process;
FIG 6 is a diagram illustrating detection of a corresponding pixel;
FIG 7 is a diagram illustrating a function of calculating a residual;
FIG 8 is a diagram illustrating the sharpness of a peak of the residual;
FIG 9 is a diagram illustrating a function of calculating an evaluation value of the depth of a peak;
FIG 10 is a diagram illustrating a function of calculating an evaluation value of the width of a peak;
FIG. 11 is a diagram illustrating calculation of a matching degree;
FIG 12 is a diagram illustrating a function of calculating the matching degree;
FIG. 13 is a flowchart illustrating a corrected parallax map generation process;
FIG 14 is a diagram illustrating another example of the configuration of the parallax synthesis unit;
FIG 15 is a flowchart illustrating a corrected parallax map generation process;
FIG 16 is a diagram illustrating still another example of the configuration of the parallax synthesis unit;
FIG 17 is a flowchart illustrating a corrected parallax map generation process;
FIG 18 is a diagram illustrating another example of the configuration of the image processing apparatus;
FIG 19 is a flowchart illustrating a parallax detection process;
FIG 20 is a diagram illustrating detection of a corresponding pixel; and
FIG 21 is a diagram illustrating an example of the configuration of a computer.
DETAILED DESCRIPTION OF THE EMBODIMENT
[0021]
Hereinafter, preferred embodiments of the present technology will be described in detail with reference to the appended drawings. Note that, in this specification and the appended drawings, structural elements that have substantially the same function and structure are denoted with the same reference numerals, and repeated explanation of these structural elements is omitted.
[0022]
T12", the function N1(x) is equal to 1. Furthermore, in the range of "T1lT22", the function N2(x) is equal to 1. Furthermore, in the range of "T21T32", the function Ns(x) is equal to 0. Furthermore, in the range of "T31T42", the function N4(x) is equal to 0. Furthermore, in the range of "T41
Example of Configuration of Parallax Synthesis Unit
The foregoing description pertains to an example in which each parallax map and each pseudo-parallax maps are added to each other in the weighted manner in accordance with the reliability and are synthesized. However, only the high-pass component of the pseudo-parallax map may be used for the synthesis.
[0181]
In this case, the parallax synthesis unit 32R in FIG 1 has a configuration shown in, for example, FIG 14. In FIG 14, the same reference numerals are given to units corresponding to the units shown in FIG 3 and the description thereof will appropriately not be repeated.
[0182]
A parallax synthesis unit 32R in FIG 14 includes a high-pass component extraction unit 131R, a low-pass component extraction unit 132R, a component synthesis unit 133R, a multiplying unit 101R, a multiplying unit 102R, and an adding unit 103R.
[0183]
The high-pass component extraction unit 131R extracts a high-pass component from the R pseudo-parallax map supplied from the depth parallax conversion unit 31R, and then supplies the high-pass component to the component synthesis unit 133R. Further, the low-pass component extraction unit 132R extracts a low-pass component from the RL parallax map supplied from the corresponding point detection unit 61R, and then supplies the low-pass component to the component synthesis unit 133R.
[0184]
The component synthesis unit 133R synthesizes the high-pass component from the high-pass component extraction unit 131R and the low-pass component from the low-pass component extraction unit 132R, and then supplies the synthesized map obtained as the synthesis result to the multiplying unit 101R. The multiplying unit 101R multiples the pixel value of each pixel of the synthesized map supplied from the component synthesis unit 133R by a supplied weight "1-RRL(I, j), and then supplies the calculation result to the adding unit 103R.
[0185]
When the parallax synthesis unit 32R in FIG 1 has the configuration shown in FIG. 14, the parallax synthesis unit 32L also has the same configuration as the parallax synthesis unit 32R shown in FfG 14. Therefore, the parallax synthesis unit 32L is not illustrated and will not be described. Hereinafter, the units of the parallax synthesis unit 32 L corresponding to the high-pass component extraction unit 131R, the low-pass component extraction unit 132R, and the component synthesis unit 133R are referred to as a high-pass component extraction unit 131L, a low-pass component extraction unit 132L, and a component synthesis unit 133L, respectively.
[0186]
When it is not necessary to distinguish between the high-pass component extraction units 131R and 131L, between the low-pass component extraction units 132R and 132L, and between the component synthesis units 133R and 133L, respectively, the high-pass component extraction units 131R and 131L, the low-pass component extraction units 132R and 132L, and the component synthesis units 133R and 133L are referred to as the high-pass component extraction units 131, the low-pass component extraction units 132, and the component synthesis units 133, respectively, below.
[0187] Description of Corrected Parallax Map Generation Process
When each parallax synthesis unit 32 includes the high-pass component extraction unit 131, the low-pass component extraction unit 132, and the component synthesis unit 133, a corrected parallax map generation process in FIG 15 is performed in step SI 5 of the parallax detection process described above in FIG 4.
[0188]
Hereinafter, the corrected parallax map generation process of the parallax synthesis unit 32 will be described with reference to the flowchart of FIG 15.
[0189]
In step S101, the high-pass component extraction units 131 extract the high-pass components from the pseudo-parallax maps by performing a filtering process or the like on the pseudo-parallax maps supplied from the depth parallax conversion units 31, and then supply the extracted high-pass components to the component synthesis units 133. In step SI01, each of the high-pass component extraction units 131R and 1311. extracts the high-pass component. '
[0190]
In step S102, the low-pass component extraction units 132 extract the iow-pass components from the parallax maps by performing a filtering process or the like on the parallax maps supplied from the corresponding point detection units 61, and then supply the extracted low-pass components to the component synthesis units 133. In step S102, each of the low-pass component extraction units 132R and 132L also extracts the low-pass component.
[0191]
In step S103, the component synthesis units 133 synthesize the high-pass components from the high-pass component extraction units 131 and the low-pass components from the low-pass component extraction units 132, and then supply the synthesized maps obtained as the synthesis result to the multiplying units 101.
[0192]
For example, the pixel values of the pixels located at the same positions as the high-pass components and the low-pass components are added and are considered as the pixel values of the pixels of the synthesized maps located at the pixels. In step S103, each of the component synthesis units 133R and 133L also generates the synthesized map.
[0193]
In step S104, the adding units 103 add the synthesized maps to the parallax maps to generate the corrected parallax maps.
[0194]
For example, the multiplying unit 101R multiples the pixel value of each pixel of the synthesized map supplied from the component synthesis unit 133R by the weight "1-RRL(I, j), and then supplies the calculation result to the adding unit 103R. Further, the multiplying unit 102R multiples the pixel value of each pixel of the RL parallax map from the corresponding point detection 61R by a weight RRL(I, j), and then supplies the calculation result to the adding unit 103R. Then, the adding unit 103R adds the synthesized map from the multiplying unit 101R to the RL parallax map from the multiplying unit 102R to generate the corrected RL. parallax map, and then the corrected RL parallax map.
[0195]
When each synthesized map and each parallax map are synthesized, the pixels with high reliability are used without change as the pixels ot each corrected parallax map. As for the pixels with low reliability, only the low-pass components of the parallax map are used and the pixels of the pseudo-parallax map are used as the high-pass components. Further, to convert the parallax map and the synthesized map more smoothly, a smoothing process may be performed on the reliability map before the parallax map and the synthesized map are synthesized.
[0196]
The multiplying unit 101L, the multiplying unit 102L, and the adding unit 103L perform the same processes as the multiplying unit 101R, the multiplying unit 102R, and the adding unit 103R, respectively, to generate the corrected LR parallax map. When the corrected parallax map is generated and output, the corrected parallax map generation process ends. Then, the process of step S15 in FIG 4 also ends and the parallax detection process thus ends.
[0197]
Example of Configuration of Parallax Synthesis Unit
In FIG 14, The foregoing description pertains to an example in which the high-pass component of the pseudo-parallax map and the low-pass component of the parallax map are synthesized to generate the synthesized map. However, the low-pass component of the pseudo-parallax map and the high-pass component of the parallax map may be synthesized to obtain the synthesized map.
[0198]
In this case, the parallax synthesis unit 32R in FIG 1 has a configuration shown in, for example, FIG 16. In FIG 16, the same reference numerals are given to units corresponding to the units shown in FIG 14 and the description thereof will appropriately not be repeated.
[0199]
A synthesis unit 32R in FIG 16 includes a high-pass component extraction unit 131R, a low-pass component extraction unit 132R, a component synthesis unit !33R; a multiplying unit 101R, a multiplying unit 102R, and an adding unit 103R. In the parallax synthesis unit 32R in FIG 16, the IlL parallax map is supplied to the high-pass component extraction unit 131R and the R pseudo-parallax map is supplied to the low-pass component extraction unit 132R.
[0200]
The high-pass component extraction unit 131R extracts a high-pass component from the RL parallax map supplied from the corresponding point detection unit 61R, and then supplies the high-pass component to the component synthesis unit 133R. Further, the low-pass component extraction unit 132R extracts a low-pass component from the R pseudo-parallax map supplied from the depth parallax conversion unit 31R, and then supplies the low-pass component to the component synthesis unit 133R.
[0201]
When the parallax synthesis unit 32R has the configuration shown in FIG 16, the parallax synthesis unit 32L also has the same configuration as the parallax synthesis unit 32R shown in FIG 16. Therefore, the parallax synthesis unit 32L is not illustrated and will not be described.
[0202] Description of Corrected Parallax Map Generation Process
When the parallax synthesis unit 32R has the configuration shown in FIG 16, a corrected parallax map generation process shown in FIG 17 is performed in step S1 5 of the parallax detection process in FIG 4.
[0203]
Hereinafter, the corrected parallax map generation process of the parallax synthesis unit 32 will be described with reference to the flowchart of FIG 17.
[0204]
In step S131, the low-pass component extraction units 132 extract the low-pass components from the pseudo-parallax maps by performing a filtering process or the like on the pseudo-parallax maps supplied from the depth parallax conversion units 3!, and then supply the extracted low-pass components to the component synthesis units 133. In step S131.. each of the low-pass component extraction units 132R and 132L extracts the low-pass component.
[0205]
In step S132, the high-pass component extraction units 131 extract the high-pass components from the parallax maps by performing a filtering process or the like on the parallax maps supplied from the corresponding point detection units 61, and then supply the extracted high-pass components to the component synthesis units 133. In step S132, each of the high-pass component extraction units 131R and 131L also extracts the high-pass component.
[0206]
After the process of step S132, the processes of step S133 and step S134 are performed and the corrected parallax map generation process ends. Since these processes are the same as the processes of step S103 and step S104 in FIG 15, the description thereof will not be repeated. However, in step S133, the low-pass component of the pseudo-parallax map and the high-pass component of the parallax map are synthesized to generate the synthesized map.
[0207]
When the corrected parallax map generation process ends, step S15 in FIG 4 also ends and the parallax detection process thus ends.
[0208]
In the corrected parallax map generation process in FIG 17, the pixels with high reliability are used without change as the pixels of each corrected parallax map. As for the pixels with low reliability, only the high-pass components of the parallax map are used and the pixels of the pseudo-parallax map are used as the low-pass components. Further, to convert the parallax map and the synthesized map more smoothly, a smoothing process may be performed on the reliability map before the parallax map and the synthesized map are synthesized.
[0209]
Documents
Application Documents
| # |
Name |
Date |
| 1 |
2193-CHE-2012 POWER OF ATTORNEY 31-05-2012.pdf |
2012-05-31 |
| 2 |
2193-CHE-2012 FORM-5 31-05-2012.pdf |
2012-05-31 |
| 3 |
2193-CHE-2012 FORM-3 31-05-2012.pdf |
2012-05-31 |
| 4 |
2193-CHE-2012 FORM-2 31-05-2012.pdf |
2012-05-31 |
| 5 |
2193-CHE-2012 FORM-1 31-05-2012.pdf |
2012-05-31 |
| 6 |
2193-CHE-2012 DRAWINGS 31-05-2012.pdf |
2012-05-31 |
| 7 |
2193-CHE-2012 CORRESPONDENCE OTHERS 31-05-2012.pdf |
2012-05-31 |
| 8 |
2193-CHE-2012 ABSTRACT 31-05-2012.pdf |
2012-05-31 |
| 9 |
2193-CHE-2012 ENGLISH TRANSLATION 31-05-2012.pdf |
2012-05-31 |
| 10 |
2193-CHE-2012 DESCRIPTION (COMPLETE) 31-05-2012.pdf |
2012-05-31 |
| 11 |
2193-CHE-2012 CLAIMS 31-05-2012..pdf |
2012-05-31 |
| 12 |
2193-CHE-2012 FORM-3 07-09-2012.pdf |
2012-09-07 |
| 13 |
2193-CHE-2012 CORRESPONDENCE OTHERS 07-09-2012.pdf |
2012-09-07 |
| 14 |
2193-CHE-2012 FORM-3 07-08-2013.pdf |
2013-08-07 |
| 15 |
2193-CHE-2012 CORRESPONDENCE OTHERS 07-08-2013.pdf |
2013-08-07 |
| 16 |
2193-CHE-2012-FER.pdf |
2019-11-20 |
Search Strategy
| 1 |
SearchStrategyMatrix_2193_che_2012_08-11-2019.pdf |