DESCRIPTION
IMAGE PROCESSING APPARATUS AND IMAGE PROCESSING METHOD
TECHNICAL FIELD
5 [OOOl]
The present technique relates to an image
processing apparatus and an image processing method, and
more particularly, to an image processing apparatus and
an image processing method capable of generating a color
10 image of a display viewpoint using a color image and a
depth image of a predetermined viewpoint.
BACKGROUND ART
[0002]
15 In recent years, 3D images attract attention. A
generally used method for viewing a 3D image is such that
a person wears glasses that open the left eye shutter
during displaying of one of images of two viewpoints and
that open the right eye shutter during displaying of the
20 other of the images, and the person views the images of
the two viewpoints which are displayed alternately
(hereinafter referred to as "method using glasses").
[0003]
However, with such method using glasses, it is
25 necessary for a viewer to purchase the glasses in
addition to a 3D image display device, which reduces the
appetite for purchase. Moreover, the viewer has to wear
glasses during viewing, which is troublesome. Therefore,
the demand for a viewing method for viewing 3D images
30 without wearing glasses (hereinafter referred to as
"method without glasses").
[0004]
In the method without glasses, images of three or
more viewpoints are displayed in such a manner that the
viewable angle is different for each viewpoint, and when
5 the viewer sees images of two viewpoints with the right
and left eyes, the viewer can see a 3D image without
wearing glasses.
[0005]
The following method has been made as a display
10 method of 3D image according to the method without
glasses. In this method, a color image and a depth image
of a predetermined viewpoint are obtained, and using the
color image and the depth image, color images of multiple
viewpoints which are display viewpoints including
15 viewpoints other than the predetermined viewpoint are
generated and displayed. It should be noted that
"multiple viewpoints" mean three or more viewpoints.
[0006]
As a method for encoding color images of multiple
20 viewpoints and a depth image, a method for separately
encoding the color images and the depth image has been
suggested (for example, see Patent Document 1).
CITATION LIST
25 NON-PATENT DOCUMENT
[0007]
Non-Patent Document 1: INTERNATIONAL ORGANISATION FOR
STANDARDISATION ORGANISATION INTERNATIONALE DE
NORMALISATION ISO/IEC JTCl/SC29/WGll CODING OF MOVING
30 PICTURES AND AUDIO, Guangzhou, China, October 2010
SUMMARY OF THE INVENTION
PROBLEMS TO BE SOLVED BY THE INVENTION
[0008]
In conventional encoding standard and the like,
5 transmission of information suitable for displaying of 3D
image according to the method without glasses is not
taken into consideration. Therefore, in the conventional
encoding standard, a large amount of information is
transmitted, which is not necessary for generation of
10 color images of multiple viewpoints which are display
viewpoints including viewpoints other than the
predetermined viewpoint using a color image of a
predetermined viewpoint and a parallax related image
which is an image (depth image) related to parallax such
15 as a depth image, and moreover, necessary information is
transmitted.
[0009]
As a result, a reception side cannot generate a
color image of a viewpoint other than the viewpoint using
20 the color image of the predetermined viewpoint and the
parallax related image.
[OOlO]
The present technique is made in view of such
circumstances, and is to allow generation of a color
25 image of a display viewpoint using a color image of a
predetermined viewpoint and a parallax related image.
SOLUTIONS TO PROBLEMS
[OOll]
30 An image processing apparatus according to a first
aspect of the present technique is an image processing
apparatus including an encoding unit for generating a bit
stream by encoding a color image of a viewpoint and a
depth image of the viewpoint; a generation unit for
generating viewpoint generation information used to
5 generate a color image of a display viewpoint, in
accordance with a generation method of the color image of
the display viewpoint obtained by performing warping
processing using the color image and the depth image; a
transmission unit for transmitting the bit stream
10 generated by the encoding unit and the viewpoint
generation information generated by the generation unit.
[0012]
An image processing method according to a first
aspect of the present technique corresponds to the image
15 processing apparatus according to the first aspect of the
present technique.
[0013]
In the first aspect of the present technique, a bit
stream is generated by encoding a color image of a
20 viewpoint and a depth image of the viewpoint, and
viewpoint generation information is generated that is
used to generate a color image of a display viewpoint, in
accordance with a generation method of the color image of
the display viewpoint obtained by performing warping
25 processing using the color image and the depth image, and
the bit stream and the viewpoint generation information
are transmitted.
[0014]
An image processing apparatus according to a second
30 aspect of the present technique is an image processing
apparatus including a receiving unit for receiving a bit
- . . . .
a, 0 0 a, a, C 0 a, 't!
tn d a C tn 4J a -rl u 0 a o a, a, 4~ C C , a ' t ! a k c a, rd -5 2 u tn . c ~ J C V C E O o - r i a a m a, a,
a t n a , e , c , t n a -4 -ri 0 . O C , a , a , a , t n a rd E c t n ~ F: a , o c u c, o a a c , c a ~
u e r i a c , h - r l m a a 7 a , a u $ u $ a -4 r i rd
O + ' a , - r ( a 4 J c 3 m C @ u a,
4J 2
- 5 a a - 2 a, u - 5 t n c
. ri
d d + ' a - r l a , a , t n
0 - r i m a o a c a u -r( c
E S P - r l
4J 2 a o a 3 a ) c o a 2 -rl L!
U O a , d 3 C r d
g 2 0 5 't!
t r o d 0
P l O C , O +J 3 0 c, U o a , D O a 3 c a u b a a , a , ' t ! a a - d a a, c o u t n a o - r i a 4
a, a, L! C a , t n a , t n C C a , c ) m C c , c , ~ J ~
' t ! - a , a , - r l a a u a , a ~ , u t n a 0 c -A a, -rl a C, -4
C a , C ! U
c C L!
0 3 0 a , 4 J E a , - 5 0
a,
a a , a u c c u - o a, -crai 'ot !aa, r 4 cJ E a , aa ,auaC,
t n a , c , t n 0 c w h t n c a a m a, 0 rn u o ~ a a , ~ , c C w ' t ! C , u
O ~ C , 2 rd a, k m m U 0 7 - r i ' t ! ' t ! C , k
- r i u a m o u c o C U C ~c o u a, a
a a, -4 a , a a , - r i u - r i c , ~ , u ~ C ~ ! a , r d c 3 @ a c
o w v , a a c o u c c -4
U 0 3 C O n J U U = 1 - r l L ! C h
u U C , E 0
&'t! 0.4-rl 4J 1 a c u 5 5
a 0 a a,
C U C h [O 0 0 7 Q U 0 a 0 4 J C C h a h .
a , o ~ - r i 4 ~ a a , a ,c a 4~ c - r i c , ~ a , c , a
tn 0 C t n z c : o O m a 0 4 J 4 J 5 2 0 a l m a m a ,
w a - 4 3 2 ; O U . g C ~ D d a u a C,
0 2 C, -Qrl -Cr,i -rl
- 5 - t n a > a , a - r l 4 -~ri -F4 c
' t ! = 1 a , O P E ; E a , 4 J m
o a ' t ! a , ~ c - r i m - r i
4J 0 -rl G O & C o a l 2 5; a, C -4 0 -ri Q tn
d C - - c i h - r l t @ h ~ u U 4 ~ ( d - 3 - r i ( d h 2 C
= ~ ~ J O O ~ J ~ J O a~, a,J Ct,nC~~o u 4~ C, o -ri a, -ri
m a w a a ~ a a m a h c @ U a a a c h a a d c 3
C Ua O -1C4 5 ka, O OuLal,Oc t n 2 - a , c - m 4J Ua m Ca O- rrdlL a,! O O uL !o4 J - ra, l
-4 c c, - 4 a - r l m c o m 4J a, a C, c, d t n U
a a c + a , c a ) a , a a g u - r l a, c 7
o - Co a C, u a, Z tr a a 4c Ju a 2 Gt nc-ar i, ~ou -cr la L!,
ma ac- 4r (~h a oa c , k a~ ,eacC~Ct an Eh r doL! m a, + ,a - rc ( $ 2 - 2 2 a a oa C, a 1a0 a, m a r l O o o a . C, -ri a k 2 C :-;9 %a C U C a u s e w m a o u
4J a, 2 5 -4 -4 h ~ , ' t ! m m ' t !
C $ C 4 J U 4 J
a, a a, m u @ ~ - l C, -4 -ri
c c - r l . r l 3 m a - r ( o a , c , c t n a , a c , ~ - r i ~ o - r l 3 m a
-4 -rl a, a 3 7 -4 c u a -4 c a W O ~ C
't!
2 4J h
= 1 e 3 i o
(d 0 tn h C C u 7 0 0 Q C, C -ri a, (d Q +J a r d a , r d t n = l t n t n ~ ! a , c -4 tn c a o t n a , r d m tn
@3' C, t u! raic - ~a t ~n aEa a,c C4o E 't0! -TcI av,, a r~d u r~c c ~a c . d a - 0 rd m U I C - - A - D D C , 2 (I] a, ~ 0 0 - r l r d U l m 4J Ul
5 3 0 a -rl m -ri 3 c a - 4 ~ ~ 1 ~ 44~ - -d a - r l m r d m
a a , u a a , a ~ - r ( u - r i a , u m u a, a z m a , 3 u a a , - r i ~ a ,
O ( d 3 t n O U 0 4 J O O C L D a l d Q) U tn 0 0 U S 0 U u a , r d ~ a , o u a u d ~ !c o a o a o ~ , a a u a , o u c o
4J 't! -4 u a , a , a , o a 2 a , o c n k C 0 -rl 0 ' a a a u t n -
L ! a , a , k
m o 3 - 4 a a u - a - 5 u z 5 a h t n a
the display viewpoint is generated by performing warping
processing using the color image and the depth image and
the viewpoint generation information.
[0017]
5 An image processing apparatus according to the
first and second aspects can be achieved by causing a
computer to execute a program.
[0018]
In order to achieve the image processing apparatus
10 according to the first and second aspects, a program
executed by the computer can be provided by transmitting
the program via a transmission medium or recording the
program to a recording medium.
15 EFFECTS OF THE INVENTION
[0019]
According to the first aspect of the present
technique, necessary information for generation of a
color image of a display viewpoint using a color image of
20 a predetermined viewpoint and a depth image can be
transmitted.
[0020]
According to the second aspect of the present
technique, a color image of a display viewpoint can be
25 generated using a color image of a predetermined
viewpoint and a depth image.
BRIEF DESCRIPTION OF DRAWINGS
[0021]
30 Fig. 1 is a figure illustrating syntax of Multiview
acquisition information SEI.
Fig. 2 is a block diagram illustrating an example
of configuration of a first embodiment of an encoding
device serving as an image processing apparatus to which
the present technique is applied.
5 Fig. 3 is a figure illustrating an example of
configuration of an access unit of an encoded bit stream.
Fig. 4 is a figure illustrating an example of
description of a portion of SEI.
Fig. 5 is a flowchart explaining encoding
10 processing of the encoding device of Fig. 2.
Fig. 6 is a flowchart explaining the details of
multi-viewpoint encoding processing of Fig. 5.
Fig. 7 is a block diagram illustrating an example
of configuration of the first embodiment of a decoding
15 device serving as an image processing apparatus to which
the present technique is applied.
Fig. 8 is a figure explaining warping processing of
a depth image.
Fig. 9 is a flowchart explaining decoding
20 processing of the decoding device of Fig. 7.
Fig. 10 is a flowchart explaining the details of
multi-viewpoint decoding processing of Fig. 9.
Fig. 11 is a block diagram illustrating an example
of configuration of a second embodiment of an encoding
25 device serving as an image processing apparatus to which
the present technique is applied.
Fig. 12 is a figure explaining a parallax related
maximum value and a parallax related minimum value of
viewpoint generation information.
30 Fig. 13 is a figure explaining parallax accuracy
parameter of the viewpoint generation information.
Fig. 14 is a figure explaining an inter-camera
distance of the viewpoint generation information.
Fig. 15 is a block diagram illustrating an example
of configuration of the multi-viewpoint image encoding
5 unit of Fig. 11.
Fig. 16 is a figure illustrating an example of
configuration of an encoded bit stream.
Fig. 17 is a figure illustrating an example of
syntax of PPS of Fig. 16.
10 Fig. 18 is a figure illustrating an example of
syntax of a slice header.
Fig. 19 is a figure illustrating an example of
syntax of a slice header.
Fig. 20 is a flowchart explaining encoding
15 processing of the encoding device of Fig. 11.
Fig. 21 is a flowchart explaining the multiviewpoint
encoding processing of Fig. 20.
Fig. 22 is a block diagram illustrating an example
of configuration of the second embodiment of a decoding
20 device serving as an image processing apparatus to which
the present technique is applied.
Fig. 23 is a block diagram illustrating an example
of configuration of the multi-viewpoint image decoding
unit of Fig. 22.
25 Fig. 24 is a flowchart explaining the multiviewpoint
decoding processing of the multi-viewpoint
image decoding unit of Fig. 22.
Fig. 25 is a figure explaining parallax and depth.
Fig. 26 is a block diagram illustrating an example
30 of configuration of an embodiment of a computer.
Fig. 27 is a figure illustrating an example of
schematic configuration of a television device to which
the present technique is applied.
Fig. 28 is a figure illustrating an example of
schematic configuration of a portable telephone to which
5 the present technique is applied.
Fig. 29 is a figure illustrating an example of
schematic configuration of a recording/reproducing device
to which the present technique is applied.
Fig. 30 is a figure illustrating an example of
10 schematic configuration of an image-capturing device to
which the present technique is applied.
MODE FOR CARRYING OUT THE INVENTION
[0022]
15
Fig. 25 is a figure explaining parallax and depth.
[0023]
As illustrated in Fig. 25, when a color image of a
20 subject M is captured by a camera cl provided at a
position C1 and a camera c2 provided at a position C2, a
depth Z of the subject M which is a distance in a depth
direction from the camera cl (camera c2) is defined by
the following expression (a).
25 [0024]
Z = (L/d) x f (a
[0025]
It should be noted that L denotes a distance
between the position C1 and the position C2 in a
30 horizontal direction (hereinafter referred to as intercamera
distance). Variable d denotes a value obtained by
subtracting a distance u2 of the position of the subject
M on the color image captured by the camera c2 in the
horizontal direction from the center of the color image
from a distance ul of the position of the subject M on
5 the color image captured by the camera cl in the
horizontal direction from the center of the color image.
More specifically, d denotes parallax. Further, f
denotes the focal distance of the camera cl. In the
expression (a), the focal distances of the camera cl and
10 the camera c2 are the same.
[0026]
As shown in the expression (a), the parallax d and
the depth Z can be converted uniquely. Therefore, in
this specification, the parallax image representing the
15 parallax d of the color images of two viewpoints taken by
the camera cl and the camera c2 and the depth image
representing the depth Z are collectively referred to as
a depth image (parallax related image).
[0027]
20 It should be noted that the depth image (parallax
related image) may be an image representing the parallax
d or the depth Z, and the pixel value of the depth image
(parallax related image) may not be the parallax d or
depth Z itself. For example, a value obtained by
25 normalizing the parallax d or a value obtained by
normalizing a reciprocal number 1/Z of the depth Z may be
employed.
[0028]
A value I obtained by normalizing the parallax d
30 with 8 bits (0 to 255) can be obtained from the following
expression (b). The number of normalization bits of the
parallax d is not limited to eight bits. It may be other
bit numbers such as 10 bits and 12 bits.
[0029]
[Mathematical Formula 41
[0030]
In the expression (b), Dma, is the maximum value of
the parallax d, bin is the minimum value of the parallax
d. The maximum value Dm,, and the minimum value bin may
10 be set in unit of screen, or may be set in units of
multiple screens.
[0031]
A value y obtained by normalizing the reciprocal
number 1/Z of the depth Z with 8 bits (0 to 255) can be
15 obtained from the following expression (c). The number
of normalization bits of the reciprocal number 1/Z of the
depth Z is not limited to 8 bits. It may be other bit
numbers such as 10 bits and 12 bits.
[0032]
20 [Mathematical Formula 51
---
Znear G a r
In the expression (c), Zfa, denotes the maximum
value of the depth Z, and Z,,,, denotes the minimum value
25 of the depth Z. The maximum value Zf,, and the minimum
value Z,,,, may be set in unit of screen, or may be set in
units of multiple screens.
[0034]
As described above, in this specification, since
the parallax d and the depth Z can be converted uniquely,
the parallax image in which the value I obtained by
5 normalizing the parallax d is the pixel value and the
depth image in which the value y obtained by normalizing
the reciprocal number 1/Z of the depth Z is the pixel
value are collectively referred to as the depth image
(parallax related image). In this case, the color format
10 of the depth image (parallax related image) is either
YUV420 or YUV400, but may be other color formats.
[0035]
When the value I or the value y is not used as the
pixel value of the depth image (parallax related image)
15 and attention is given to the information of the value I
or the value y itself, the value I or the value y is
adopted as depth information. Further, what is obtained
by mapping the value I or the value y will be referred to
as a depth map.
20 [0036]

[Example of configuration of first embodiment of encoding
device ]
Fig. 2 is a block diagram illustrating an example
25 of configuration of a first embodiment of an encoding
device serving as an image processing apparatus to which
the present technique is applied.
[0037]
An encoding device 10 of Fig. 2 includes a multi-
30 viewpoint color image image-capturing unit 11, a multiviewpoint
color image correction unit 12, a multiviewpoint
parallax related image generation unit 13, a
viewpoint generation information generation unit 14, and
a multi-viewpoint image encoding unit 15.
[0038]
The encoding device 10 encodes a color image of a
predetermined viewpoint and a parallax related image, and
uses the color image of the predetermined viewpoint and
the parallax related image to add and transmit viewpoint
generation information which is information for
generating a color image of a viewpoint other than the
predetermined viewpoint.
[0039]
The multi-viewpoint color image image-capturing
unit 11 of the encoding device 10 captures color images
15 of multiple viewpoints, and provides them as multiviewpoint
color images to the multi-viewpoint color image
correction unit 12. The multi-viewpoint color image
image-capturing unit 11 generates, as image-capturing
information, information about image-capturing such as
20 the number of viewpoints of the color images, external
parameters, and range information, and provides the
information to the viewpoint generation information
generation unit 14.
[0040]
25 It should be noted that the external parameter is a
parameter for defining the position of multi-viewpoint
color image image-capturing unit 11 in the horizontal
direction.
[0041]
30 When the parallax related image generated by the
multi-viewpoint parallax related image generation unit 13
is a depth image, the range information includes the
minimum value (minimum value Z,,,,) and the maximum value
(maximum value Zf,,) of world coordinate values at
positions in the depth direction that would occur in the
5 multi-viewpoint parallax related image (multi-viewpoint
depth image). The minimum value and the maximum value
will be hereinafter referred to as a depth minimum value
and a depth maximum value, respectively. On the other
hand, when the parallax related image generated by the
10 multi-viewpoint parallax related image generation unit 13
is a parallax image, the range information includes the
minimum value (minimum value Dmi,) and the maximum value
(maximum value &,,) of parallax on the world coordinate
that would occur in the multi-viewpoint parallax related
15 image, and also information for identifying a color image
of a viewpoint adopted as a base point when a parallax
value is obtained. The minimum value and the maximum
value will be hereinafter referred to as a parallax
minimum value and a parallax maximum value, respectively.
20 [0042]
The multi-viewpoint color image correction unit 12
performs color correction, brightness correction,
distortion correction, and the like on the multiviewpoint
color images provided from the multi-viewpoint
25 color image image-capturing unit 11. Accordingly, the
focal distance of the multi-viewpoint color image imagecapturing
unit 11 in the corrected multi-viewpoint color
image in the horizontal direction (X direction) is the
same at all the viewpoints. The multi-viewpoint color
30 image correction unit 12 provides the corrected multiviewpoint
color image to the multi-viewpoint parallax
related image generation unit 13 and the multi-viewpoint
image encoding unit 15 as multi-viewpoint corrected color
images. The multi-viewpoint color image correction unit
12 generates information about the multi-viewpoint
5 corrected color image such as internal parameters as
color image information, and provides the information to
the viewpoint generation information generation unit 14.
LO0431
The internal parameters include the focal distance,
10 in the horizontal direction, of the multi-viewpoint color
image image-capturing unit 11 which is the same for all
the viewpoints in the multi-viewpoint corrected color
images, and the principal point which is the center of
the image, i.e., the position of the optical center of
15 the lens in the horizontal direction. The position of
the principal point in the horizontal direction may be
different for each viewpoint.
[0044]
The multi-viewpoint parallax related image
20 generation unit 13 generates parallax related images of
multiple viewpoints from the multi-viewpoint corrected
color images provided from the multi-viewpoint color
image correction unit 12. Then, the multi-viewpoint
parallax related image generation unit 13 provides the
25 generated parallax related images of multiple viewpoints
to the multi-viewpoint image encoding unit 15 as the
multi-viewpoint parallax related image.
[DO451
The multi-viewpoint parallax related image
30 generation unit 13 generates, as parallax related image
information (depth image information), information about
the multi-viewpoint parallax related image such as the
number of viewpoints of the multi-viewpoint parallax
related images, image type information indicating whether
the multi-viewpoint parallax related image is a depth
5 image or a parallax image (depth image identification
information). Then, the multi-viewpoint parallax related
image generation unit 13 provides the parallax related
image information to the viewpoint generation information
generation unit 14.
10 [00463
The viewpoint generation information generation
unit 14 functions as a generation unit, and uses the
multi-viewpoint corrected color image and the multiviewpoint
parallax related image to generate viewpoint
15 generation information in accordance with a predetermined
method for generating a color image of another viewpoint.
More specifically, the viewpoint generation information
generation unit 14 generates the color image
identification information and the parallax related image
20 identification information (depth image identification
information) on the basis of the number of viewpoints of
the color images provided from the multi-viewpoint color
image image-capturing unit 11 and the number of
viewpoints of the parallax related images provided from
25 the multi-viewpoint parallax related image generation
unit 13. It should be noted that the color image
identification information is information for identifying
the color image, and the parallax related image
identification information is information for identifying
30 the parallax related image.
[0047]
The viewpoint generation information generation
unit 14 generates an external parameter flag indicating
presence/absence of an external parameter at each
viewpoint corresponding to the multi-viewpoint parallax
5 related image on the basis of the external parameter
included in the image-capturing information provided by
the multi-viewpoint color image image-capturing unit 11.
Further, the viewpoint generation information generation
unit 14 generates an internal parameter flag indicating
10 presence/absence of an internal parameter at each
viewpoint corresponding to the multi-viewpoint parallax
related image on the basis of the internal parameter
included in the color image information provided by the
multi-viewpoint color image correction unit 12. Then,
15 the viewpoint generation information generation unit 14
generates viewpoint generation information from the color
image identification information, the parallax related
image identification information, the external parameter
flag, the internal parameter flag, the image-capturing
20 information, the color image information, and the
parallax related image information. The viewpoint
generation information is constituted by the color image
identification information, the parallax related image
identification information, the external parameter
25 information, the internal parameter information, the
number of viewpoints of the color images, the range
information, the number of viewpoints of the parallax
related images, and the image type information. It
should be noted that the external parameter information
30 is constituted by the external parameter flag and the
external parameter of the viewpoint corresponding to the
multi-viewpoint parallax related image, and the internal
parameter information is constituted by the internal
parameter flag and the internal parameter of the
viewpoint corresponding to the multi-viewpoint parallax
5 related image. The viewpoint generation information
generation unit 14 provides the generated viewpoint
generation information to the multi-viewpoint image
encoding unit 15.
[0048]
10 The multi-viewpoint image encoding unit 15
functions as an encoding unit, and encodes the multiviewpoint
corrected color image provided by the multiviewpoint
color image correction unit 12 and the multiviewpoint
parallax related image provided by the multi-
15 viewpoint parallax related image generation unit 13 in
accordance with the same method as MVC (Multiview Video
Coding) method, and adds the viewpoint generation
information provided by the viewpoint generation
information generation unit 14. The multi-viewpoint
20 image encoding unit 15 functions as a transmission unit,
and transmits, as an encoded bit stream, the bit stream
obtained as a result.
[0049]
[Example of configuration of encoded bit stream]
25 Fig. 3 is a figure illustrating an example of
configuration of an access unit of an encoded bit stream
generated by the encoding device 10 of Fig. 2.
[0050]
As illustrated in Fig. 3, the access unit of the
30 encoded bit stream is constituted by SPS (Sequence
Parameter Set), Subset SPS, PPS (Picture Parameter Set),
SEI, and slice.
[0051]
In the example of Fig. 3, the number of viewpoints
of the multi-viewpoint corrected color images and the
5 multi-viewpoint parallax related images is two. The
corrected color image A which is one of viewpoints of the
multi-viewpoint corrected color images of the two
viewpoints is encoded as a base view. The corrected
color image B of the other of the viewpoints, the
10 parallax related image A corresponding to the color image
A, and the parallax related image B corresponding to the
color image B are encoded as non-base views.
[0052]
As a result, the slices are arranged, for example,
15 in the following order: a slice for the color image A
encoded as the base view, a slice of the parallax related
image A encoded as the non-base view, and a slice for the
color image B, and then a slice for the parallax related
image B. In the header portion of each slice,
20 information for identifying PPS is described.
[0053]
SPS is a header including information about
encoding of the base view. Subset SPS is an extension
header including information about encoding of the base
25 view and the non-base view. PPS is a header including
information indicating encoding mode of the entire
picture and information for identifying SPS and Subset
SPS. SEI is additional information which is not
requisite for decoding, and includes viewpoint generation
30 information and the like.
[0054]
During decoding of the color image A which is
encoded as the base view, PPS is looked up on the basis
of information for identifying PPS described in the
header portion of the color image A, and SPS is looked up
5 on the basis of information for identifying SPS described
in PPS.
[0055]
On the other hand, during decoding of the parallax
related image A encoded as the non-base view, PPS is
10 looked up on the basis of information for identifying PPS
described in the header of the parallax related image A.
Sub SPS is looked up on the basis of information for
identifying Sub SPS described in PPS. During decoding of
the color image B and the parallax related image B
15 encoded as the non-base view, PPS is also looked up and
Sub SPS is also looked up just like during decoding of
the parallax related image A.
[0056]
[Example of a portion of description of SEI]
20 Fig. 4 is a figure illustrating an example of
description of a portion of SEI.
[0057]
In the second line from the top at the left side of
SEI of Fig. 4, the number of viewpoints
25 (nun color - view - minus 1) of the color image is described.
In the third line, the number of viewpoints
(nun- d epth- v iew- m inus-1) of the parallax related image is
described.
[0058]
30 In the fifth line from the top at the left side of
Fig. 4, the view ID (color - view - id) of the color image is
described as the color image identification information
of the color image of each viewpoint. In the seventh
line, the view ID (depth-view-id) of the parallax related
image is described as the parallax related image
5 identification information of the parallax related image
of each viewpoint. In the eighth line from the top at
the left side of Fig. 4, the internal parameter flag in
the internal parameter information is described for each
viewpoint corresponding to the multi-viewpoint parallax
10 related image. In the ninth line, the external parameter
flag in the external parameter information is described
for each viewpoint corresponding to the multi-viewpoint
parallax related image.
[0059]
15 In the 11th and 13th to 15th lines from the top at
the left side of Fig. 4, the focal distance in the
horizontal direction in the internal parameter is
described.
[0060]
20 In SEI of Fig. 4, real number x is described as a
floating point number using prec value, sign value,
exponent value, and a mantissa value defined by the
following expression (1) .
[0061]
25 When e = 0 holds,
v = Max(0, prec-30)
x = (-1) s, 2-(30 + V) an
When 0 < e I 62 holds,
v = Max(0, e+prec-31)
x = (-1) .2e-31.(l+ n/ZV)
- (1)
[ 0 0 62 1
In the expression (I), prec denotes prec value, s
denotes sign value, e denotes exponent value, and n
denotes mantissa value. Accordingly, sign value,
5 exponent value, mantissa value represent sign, exponent,
mantissa of the real number x, respectively.
[0063]
As described above, the real number x is described
as a floating point number, and therefore, in the 11th
10 and 13th to 15th lines from the top at the left side of
Fig. 4, prec value (prec-focal-length), sign value
(sign- f ocal- l ength- x ), exponent value
(exponent- f ocal- l ength- x ), mantissa value
(mantissa - focal - length-x) of the focal distance in the
15 horizontal distance are described, respectively.
[0064]
In the 12th line from the top at the left side of
Fig. 4, a value common to each viewpoint corresponding to
the multi-viewpoint parallax related image is described
20 as prec value (prec - principalpoint) at the position of
the principal point in the horizontal direction. In the
17th to 19th lines from the top at the left side of Fig.
4, sign value (sign- p rincipal- p oint- x ), exponent value
(exponent- p rincipal- p oint- x ), mantissa value
25 (mantissa - principal - point - x) at the position of principal
point in the horizontal direction are described for each
viewpoint corresponding to the multi-viewpoint parallax
related image.
[0065]
30 In the 2nd line from the top at the right side of
Fig. 4, a value common to each viewpoint corresponding to
the multi-viewpoint parallax related image is described
as prec value (prec-translation-param) of the external
parameter. In the 4th to 6th lines from the top at the
right side of Fig. 4, external parameters of sign value
5 (sign- t ranslation- x ), exponent value
(exponent translation - x), mantissa value
(mantissa - translation - x) are described for each viewpoint
corresponding to the multi-viewpoint parallax related
image.
10 [0066]
In the 9th lines from the top at the right side of
Fig. 4, image type information (depth- m ap- f lag) is
described. In the 11th to 12th lines from the top at the
right side of Fig. 4, when the image type information
15 indicates the depth image, the depth minimum value
(depth - nearest) and the depth maximum value
(depth - farthest) which are common to each viewpoint
corresponding to the multi-viewpoint parallax related
image in the range information are described.
20 [0067]
Further, in the 16th to 18th lines from the top at
the right side of Fig. 4, range information where the
image type information indicates the parallax image is
described. More specifically, in 16th and 17th lines,
25 the parallax minimum value (disparity- m inimum) and the
parallax maximum value (disparity - maximum) are described,
respectively. The parallax minimum value and the
parallax maximum value are different according to the
viewpoint, and therefore, they are generated and
30 described for each viewpoint.
[0068]
In the 18th line, the view ID
(reference depth view) of the color image - - is described,
for each viewpoint corresponding to the multi-viewpoint
parallax related image, as information for identifying
5 the color image of the viewpoint adopted as the base
point when the parallax value is derived.
[0069]
[Explanation about processing of encoding device]
Fig. 5 is a flowchart explaining encoding
10 processing of the encoding device 10 of Fig. 2.
[0070]
In step S10 in Fig. 5, the multi-viewpoint color
image image-capturing unit 11 of the encoding device 10
captures color images of multiple viewpoints, and
15 provides them as multi-viewpoint color images to the
multi-viewpoint color image correction unit 12.
[00711
In step Sll, the multi-viewpoint color image imagecapturing
unit 11 generates image-capturing information,
20 and provides it to the viewpoint generation information
generation unit 14.
[00721
In step 512, the multi-viewpoint color image
correction unit 12 performs color correction, brightness
25 correction, distortion correction, and the like on the
multi-viewpoint color images provided from the multiviewpoint
color image image-capturing unit 11. The
multi-viewpoint color image correction unit 12 provides
the corrected multi-viewpoint color image to the multi-
30 viewpoint parallax related image generation unit 13 and
the multi-viewpoint image encoding unit 15 as multiviewpoint
corrected color images.
[0073]
In step S13, the multi-viewpoint color image
correction unit 12 generates color image information, and
5 provides it to the viewpoint generation information
generation unit 14.
[0074]
In step S14, the multi-viewpoint parallax related
image generation unit 13 generates parallax related
10 images of multiple viewpoints from the multi-viewpoint
corrected color images provided from the multi-viewpoint
color image correction unit 12. Then, the multiviewpoint
parallax related image generation unit 13
provides the generated parallax related images of
15 multiple viewpoints to the multi-viewpoint image encoding
unit 15 as the multi-viewpoint parallax related image.
[0075]
In step S15, the multi-viewpoint parallax related
image generation unit 13 generates parallax related image
20 information, and provides it to the viewpoint generation
information generation unit 14.
[0076]
In step S16, the viewpoint generation information
generation unit 14 generates color image identification
25 information, parallax related image identification
information, internal parameter flag, and external
parameter flag on the basis of the image-capturing
information, the color image information, and the
parallax related image information.
30 [0077]
In step S17, the viewpoint generation information
generation unit 14 generates viewpoint generation
information. More specifically, the viewpoint generation
information generation unit 14 generates, as the
viewpoint generation information, color image
5 identification information, parallax related image
identification information, the number of viewpoints of
color images, external parameter information, internal
parameter information, range information, the number of
viewpoints of parallax related images, and image type
10 information. The viewpoint generation information
generation unit 14 provides the generated viewpoint
generation information to the multi-viewpoint image
encoding unit 15.
[0078]
15 In step S18, the multi-viewpoint image encoding
unit 15 performs multi-viewpoint encoding processing for
encoding the multi-viewpoint corrected color image and
the multi-viewpoint parallax related image, and adding
the viewpoint generation information and the like. The
20 details of the multi-viewpoint encoding processing will
be explained with reference to Fig. 6 explained later.
[0079]
In step S19, the multi-viewpoint image encoding
unit 15 transmits the encoded bit stream generated as a
25 result of step S18, and terminates the processing.
[0080]
Fig. 6 is a flowchart explaining the details of
multi-viewpoint encoding processing in step S18 of Fig. 5.
This multi-viewpoint encoding processing is done, for
30 example, in units of slices. In the multi-viewpoint
encoding processing of Fig. 6, the images to be encoded
are the color image A, the color image B, the parallax
related image A, and the parallax related image B.
[0081]
In step S31 of Fig. 6, the multi-viewpoint image
5 encoding unit 15 generates SPS of a target slice which is
slice of processing target, and gives an ID unique to SPS.
In step S32, the multi-viewpoint image encoding unit 15
generates Subset SPS of the target slice, and gives an ID
unique to the Subset SPS.
10 [0082]
In step S33, the multi-viewpoint image encoding
unit 15 generates PPS of the target slice including the
ID given in steps S31 and S32, and gives an ID unique to
the PPS. In step S34, the multi-viewpoint image encoding
15 unit 15 generates SEI including the viewpoint generation
information of the target slice.
[0083]
In step S35, the multi-viewpoint image encoding
unit 15 encodes the target slice of the color image A as
20 a base view, and adds a header portion including the ID
given in step S33. In step S36, the multi-viewpoint
image encoding unit 15 encodes the target slice of the
parallax related image A as a non-base view, and adds a
header portion including the ID given in step S33.
25 [0084]
In step S37, the multi-viewpoint image encoding
unit 15 encodes the target slice of the color image B as
a non-base view, and adds a header portion including the
ID given in step S33. In step S38, the multi-viewpoint
30 image encoding unit 15 encodes the target slice of the
parallax related image B as a non-base view, and adds a
header portion including the ID given in step S33.
[00851
Then, the multi-viewpoint image encoding unit 15
generates an encoded bit stream by arranging, in order,
5 the SPS, the Subset SPS, the PPS, the SEI, the target
slice of the color image A, the target slice of the
parallax related image A, the target slice of the color
image B, and the target slice of the parallax related
image B thus generated. Then, processing in step S18 in
10 Fig. 5 is performed again, and subsequently step S19 is
performed.
[0086]
In the multi-viewpoint encoding processing of Fig.
6, for the sake of explanation, the SPS is generated in
15 unit of slice, but when the SPS of the current target
slice is the same as the SPS of the previous target slice,
such SPS is not generated. The same can be said to
Subset SPS, PPS, and SEI.
[0087]
20 As described above, the encoding device 10 uses the
multi-viewpoint corrected color image and the multiviewpoint
parallax related image to generate the
viewpoint generation information in accordance with a
predetermined method for generating a color image of
25 another viewpoint, and transmits it together with the
color image of the predetermined viewpoint and the
parallax related image. Therefore, the decoding device
explained later can use the color image of the
predetermined viewpoint, the parallax related image, and
30 the viewpoint generation information to generate the
color image of a viewpoint different from the
predetermined viewpoint.
[0088]
[Example of configuration of first embodiment of decoding
device]
5 Fig. 7 is a block diagram illustrating an example
of configuration of the first embodiment of a decoding
device, serving as an image processing apparatus to which
the present technique is applied, which decodes an
encoded bit stream transmitted from the encoding device
10 10 of Fig. 2.
[0089]
The decoding device 30 of Fig. 7 is constituted by
a multi-viewpoint image decoding unit 31, a viewpoint
compositing unit 32, and a multi-viewpoint image display
15 unit 33.
[0090]
The multi-viewpoint image decoding unit 31 of the
decoding device 30 functions as a receiving unit, and
receives the encoded bit stream transmitted from the
20 encoding device 10 of Fig. 2. The multi-viewpoint image
decoding unit 31 extracts the viewpoint generation
information from the SEI of the received encoded bit
stream, and provides the viewpoint generation information
to the viewpoint compositing unit 32. The multi-
25 viewpoint image decoding unit 31 functions as a decoding
unit, and decodes the encoded bit stream in accordance
with decoding method corresponding to the encoding method
of the multi-viewpoint image encoding unit 15 of Fig. 2,
and generates multi-viewpoint corrected color images and
30 multi-viewpoint parallax related image. The multiviewpoint
image decoding unit 31 provides the multiviewpoint
corrected color images and the multi-viewpoint
parallax related image to the viewpoint compositing unit
32.
[0091]
5 The viewpoint compositing unit 32 uses the
viewpoint generation information provided by the multiviewpoint
image decoding unit 31 to perform warping
processing (the details of which will be explained later)
on the multi-viewpoint parallax related image provided by
10 the multi-viewpoint image decoding unit 31 for as many
display viewpoints as the number of viewpoints
corresponding to the multi-viewpoint image display unit
33.
[0092]
15 More specifically, the viewpoint compositing unit
32 identifies the multi-viewpoint parallax related images
for as many viewpoints as the number of viewpoints of the
parallax related image from among the multi-viewpoint
parallax related images and the multi-viewpoint corrected
20 color images provided by the multi-viewpoint image
decoding unit 31 on the basis of the parallax related
image identification information included in the
viewpoint generation information. Then, when the
internal parameter flag included in the viewpoint
25 generation information indicates presence of internal
parameter, the viewpoint compositing unit 32 obtains the
internal parameter from the viewpoint generation
information. Then, when the external parameter flag
included in the viewpoint generation information
30 indicates presence of external parameter, the viewpoint
compositing unit 32 obtains the external parameter from
the viewpoint generation information. Then, the
viewpoint compositing unit 32 performs warping processing
on the multi-viewpoint parallax related image to warp the
multi-viewpoint parallax related image according to the
5 display viewpoints on the basis of the internal parameter
and the external parameter thus obtained, and the image
type information and the range information included in
the viewpoint generation information.
[0093]
10 The warping processing is processing for
geometrically converting an image of a certain viewpoint
into an image of another viewpoint. The display
viewpoints include viewpoints other than the viewpoints
corresponding to the multi-viewpoint color image.
15 [0094]
The viewpoint compositing unit 32 uses the
viewpoint generation information and the parallax related
image of display viewpoints obtained as a result of
warping processing to perform warping processing on the
20 multi-viewpoint corrected color images provided by the
multi-viewpoint image decoding unit 31 to warp them
according to the display viewpoints. More specifically,
viewpoint compositing unit 32 identifies the multiviewpoint
color images of as many viewpoints as the
25 number of viewpoints of the color images from among the
multi-viewpoint corrected color images and the multiviewpoint
parallax related image provided by the multiviewpoint
image decoding unit 31 on the basis of the
color image identification information included in the
30 viewpoint generation information. Then, the viewpoint
compositing unit 32 uses the parallax related image for
the display viewpoints obtained as a result of the
warping processing to perform the warping processing on
the multi-viewpoint corrected color images to warp them
according to the display viewpoints.
5 [0095]
The viewpoint compositing unit 32 provides the
color images for the display viewpoints obtained as a
result, to the multi-viewpoint image display unit 33 as
the multi-viewpoint composed color images.
10 [0096]
The multi-viewpoint image display unit 33 displays
the multi-viewpoint composed color images provided by the
viewpoint compositing unit 32 in such a manner that the
viewable angle is different for each viewpoint. The
15 viewer sees the images of two given viewpoints with their
right and left eyes, thus seeing a 3D image from multiple
viewpoints without wearing glasses.
[0097]
[Explanation about warping processing of depth image]
20 Fig. 8 is a figure explaining warping processing of
a depth image.
[0098]
In the warping processing of the depth image, first,
as illustrated in Fig. 8, a position m (x, y, z) of each
25 pixel of the depth image for the viewpoint at a
processing target position t (t,,t,,t,) is associated, via
three dimensional space, with a position m' (x', y', z' ) of
the depth image for the display viewpoint at a warpingprocessed
position t' (tfXt,V ytfV z).
30 [0099]
More specifically, the position M (X,Y, 2 ) on the
world coordinate corresponding to the position m (x,y.z)
on the screen for the pixel of which depth value (value
y) is a value corresponding to the position Z (depth Z)
in the world coordinate in the depth direction in the
5 depth image for the viewpoint at the position t (tx,ty,tz)
is obtained according to the following expression (2).
and the position m' (xl.y',z') on the screen of the depth
image for the display viewpoint at the position t1
t t t corresponding to the position M (X.Y. Z) is
10 obtained.
15 [OlOl]
In the expression (2), R is a rotation matrix for
transforming the coordinate on the multi-viewpoint color
image image-capturing unit 11 into the world coordinate.
and is a fixed matrix determined in advance. R' is a
20 rotation matrix for transforming the coordinate on the
virtual image-capturing unit for capturing the color
image corresponding to the warping-processed depth image
into the world coordinate.
[0102]
25 In the expression (2), A is a matrix for
transforming the coordinate on the multi-viewpoint color
image image-capturing unit 11 into the coordinate on the
screen, and is represented by the following expression
(3)
30 [0103]
[Mathematical Formula 11
focal-length-x radial-distortion principaI_point-x
A = 0.0 foca I- l ength-y pr i nc i pa l ~ onit- y 1
(3)
[0104]
In the expression (3), focal - length - x,
focal - length - y represent the focal distance of the multi-
5 viewpoint color image image-capturing unit 11 in the
horizontal direction and the vertical direction (y
direction), respectively. It should be noted that
principal - point - x , principal - point - y represent the
positions of the principal points of the multi-viewpoint
10 color image image-capturing unit I1 in the horizontal
direction and the vertical direction, respectively. The
radial - distortion represents a strain coefficient in the
radius direction.
[0105]
15 However, the viewpoint compositing unit 32 does not
use the focal distance in the vertical direction and the
position of the principal point in the vertical direction
in the expression (3). Correction is made by the multiviewpoint
color image correction unit 12 so that the
20 strain coefficient in the radius direction becomes zero.
[0106]
A' is a matrix, expressed in the same manner as A,
for transforming the coordinate of the virtual imagecapturing
unit for capturing the color image
25 corresponding to the warping-processed depth image into
the coordinate on the screen, and is determined in
advance. In the expression ( 2 ) , s is a scaling factor,
and is a fixed value determined in advance. In this case,
t, and t, are fixed values determined in advance.
[0107]
Therefore, information other than the depth image
5 required to associate the position m (x,y,z) and the
position m' (x1,y',z') is the focal distance in the
horizontal direction, the position of the principal point
in the horizontal direction, the position of the multiviewpoint
color image image-capturing unit 11 in the
10 horizontal direction, and the depth minimum value and the
depth maximum value required to transform the depth value
into the position Z. Such information is included in the
viewpoint generation information.
[0108]
15 After the association is made as described above,
the pixel of the depth image having been subjected to the
warping processing corresponding to each pixel of the
depth image of the processing target is determined on the
basis of the position m' (xl,y',z' ) corresponding to the
20 position m (x,y, z ) of each pixel. Then, the depth value
of each pixel of the depth image of the processing target
is adopted as the depth value of the pixel of the
warping-processed depth image corresponding to the pixel.
[0109]
25 It should be noted that the warping processing for
the parallax image is done in the same manner as that
performed on the depth image except that the depth value
(value y) is replaced with the parallax value (value I).
[OllO]
30 [Explanation about processing of decoding device]
Fig. 9 is a flowchart explaining decoding
processing of the decoding device 30 of Fig. 7. This
decoding processing is started, for example, when the
encoded bit stream is transmitted from the encoding
device 10 of Fig. 2.
5 [Olll]
In step S51 of Fig. 9, the multi-viewpoint image
decoding unit 31 of the decoding device 30 receives the
encoded bit stream transmitted from the encoding device
10 of Fig. 2.
10 [0112]
In step 552, the multi-viewpoint image decoding
unit 31 decodes the received encoded bit stream, and
performs the multi-viewpoint decoding processing for
extracting the viewpoint generation information. The
15 details of the multi-viewpoint decoding processing will
be explained with reference to Fig. 10 explained later.
[0113]
In step S53, the viewpoint compositing unit 32
functions as a generation unit, and generates the multi-
20 viewpoint composed color image using the viewpoint
generation information provided by the multi-viewpoint
image decoding unit 31, the multi-viewpoint corrected
color images, and the multi-viewpoint parallax related
image.
25 [0114]
In step S54, the multi-viewpoint image display unit
33 displays the multi-viewpoint composed color image
provided by the viewpoint compositing unit 32 in such a
manner that the viewable angle, and terminates the
30 processing.
[0115]
Fig. 10 is a flowchart explaining the details of
multi-viewpoint decoding processing in step 552 of Fig. 9.
This multi-viewpoint decoding processing is done, for
example, in units of slices. In the multi-viewpoint
5 decoding processing of Fig. 10, the images of the
decoding target are the color image A, the color image B,
the parallax related image A, and the parallax related
image B.
[0116]
10 In step S71 of Fig. 10, the multi-viewpoint image
decoding unit 31 extracts SPS from the received encoded
bit stream. In step S72, the multi-viewpoint image
decoding unit 31 extracts Subset SPS from the encoded bit
stream. In step 573, the multi-viewpoint image decoding
15 unit 31 extracts PPS from the encoded bit stream.
[0117]
In step 574, the multi-viewpoint image decoding
unit 31 extracts SEI from the encoded bit stream, and
provides the viewpoint generation information included in
20 the SEI to the viewpoint compositing unit 32.
[0118]
In step 575, on the basis of the ID of PPS included
in the header of the target slice of the color image A,
the multi-viewpoint image decoding unit 31 activates the
25 PPS to which the ID is given. In step 576, on the basis
of the ID of SPS included in PPS and activated in step
575, the multi-viewpoint image decoding unit 31 activates
the SPS to which the ID is given.
[0119]
30 In step S77, the multi-viewpoint image decoding
unit 31 looks up the PPS and the SPS activated, and
decodes the target slice of the color image A as the base
view, and provides it to the viewpoint compositing unit
32.
[0120]
5 In step 578, on the basis of the ID of Subset SPS
included in PPS and activated, the multi-viewpoint image
decoding unit 31 activates the Subset SPS to which the ID
is given. In step S79, the multi-viewpoint image
decoding unit 31 looks up the PPS and the Subset SPS
10 activated, and decodes the target slice of the parallax
related image A as the non-base view, and provides it to
the viewpoint compositing unit 32.
[0121]
In step S80, the multi-viewpoint image decoding
15 unit 31 looks up the PPS and the Subset SPS activated,
and decodes the target slice of the color image B as the
non-base view, and provides it to the viewpoint
compositing unit 32. In step S81, the multi-viewpoint
image decoding unit 31 looks up the PPS and the Subset
20 SPS activated, and decodes the target slice of the
parallax related image B as the non-base view, and
provides it to the viewpoint compositing unit 32. Then,
processing in step S52 in Fig. 9 is performed again, and
subsequently step S53 is performed.
25 [0122]
In the multi-viewpoint decoding processing of Fig.
10, for the sake of explanation, the SPS, the Subset SPS,
the PPS, and the SEI are considered to be generated for
all the slices, and are configured to be extracted at all
30 times for each slice, but when there is a slice for which
the SPS, the Subset SPS, the PPS, and the SEI are not
generated, processing for extracting the SPS, the Subset
SPS, the PPS, and the SEI is skipped.
[0123]
As described above, the decoding device 30 receives,
5 from the encoding device 10, the color image of the
predetermined viewpoint and the parallax related image as
well as the viewpoint generation information. Therefore,
the decoding device 30 can use the color image of the
predetermined viewpoint, the parallax related image, and
10 the viewpoint generation information to generate the
color image of a viewpoint different from the
predetermined viewpoint.
[0124]
In contrast, the bit stream encoded according to
15 MVC method which is a conventional encoding method does
not include necessary information for generating a color
image of a viewpoint different from a predetermined
viewpoint using the color image of the predetermined
viewpoint and the parallax related image. More
20 specifically, it does not include information for
identifying the color image and parallax related image,
information for identifying the parallax image and the
depth image, and the range information.
[0125]
25 The multiview acquisition information SEI as
illustrated in Fig. 1 includes parameters for the inside
of the camera and the outside of the camera, but includes
much unnecessary information other than the viewpoint
generation information. Further, in the multiview
30 acquisition information SEI, the parameters for the
inside of the camera are described for all the viewpoints
or only for particular viewpoints, regardless of the
types of the parameters. Therefore, when the parameters
for the inside of the camera are described for all the
viewpoints, it includes much redundant information, and
5 when the parameters for only for particular viewpoints
are described, such information is insufficient. In the
multiview acquisition information SEI, the parameters
outside of the camera are described for all the
viewpoints, and therefore, much redundant information is
10 included.
[0126]
In the encoding device 10, the multi-viewpoint
parallax related image is generated from the multiviewpoint
corrected color image, but it may be generated
15 by sensors detecting the parallax value and the depth
value during image capturing of the multi-viewpoint color
image. This is applicable to the encoding device 50
explained later.
[0127]
20 The viewpoint generation information may include
any one of the color image identification information and
the parallax related image identification information.
In this case, the decoding device 30 identifies, as an
image not identified, one of images of the multi-
25 viewpoint color image and the multi-viewpoint parallax
related image that is not identified. For example, the
viewpoint generation information includes only the color
image identification information, the decoding device 30
identifies the multi-viewpoint color image on the basis
30 of the color image identification information, and
determines that the images other than that are the multiviewpoint
parallax related images.
[0128]
In the first embodiment, the viewpoint generation
information is transmitted in the SEI. Alternatively,
5 the viewpoint generation information may be included in
the VCL (Video Coding Layer), the SPS (Sequence Parameter
Set) and the PPS (Picture Parameter Set) of the NAL
(Network Abstraction Layer), and the like.
[0129]
[Example of configuration of second embodiment of
encoding device]
Fig. 11 is a block diagram illustrating an example
of configuration of a second embodiment of an encoding
15 device serving as an image processing apparatus to which
the present technique is applied.
[0130]
An encoding device 50 of Fig. 11 includes a multiviewpoint
color image image-capturing unit 51, a multi-
20 viewpoint color image correction unit 52, a multiviewpoint
parallax related image generation unit 53, a
viewpoint generation information generation unit 54, and
a multi-viewpoint image encoding unit 55. The encoding
device 50 transmits a portion of the viewpoint generation
25 information as information about encoding (encoding
parameter).
[0131]
More specifically, the multi-viewpoint color image
image-capturing unit'51 of the encoding device 10
30 captures color images of multiple viewpoints, and
provides them as multi-viewpoint color images to the
multi-viewpoint color image correction unit 52. The
multi-viewpoint color image image-capturing unit 51
generates external parameter, parallax related maximum
value (depth maximum value), and parallax related minimum
5 value (depth minimum value) (the details of which will be
explained later). The multi-viewpoint color image imagecapturing
unit 51 provides the external parameter, the
parallax related maximum value, and the parallax related
minimum value to the viewpoint generation information
10 generation unit 54, and provides the parallax related
maximum value and the parallax related minimum value to
the multi-viewpoint parallax related image generation
unit 53.
[0132]
15 When the parallax related image generated by the
multi-viewpoint parallax related image generation unit 53
is a depth image, the parallax related maximum value is a
depth maximum value, and when it is a parallax image, the
parallax related maximum value is a parallax maximum
20 value. When the parallax related image generated by the
multi-viewpoint parallax related image generation unit 53
is a depth image, the parallax related minimum value is a
depth minimum value, and when it is a parallax image, the
parallax related minimum value is a parallax minimum
25 value.
[0133]
The multi-viewpoint color image correction unit 52
performs color correction, brightness correction,
distortion correction, and the like on the multi-
30 viewpoint color images provided from the multi-viewpoint
color image image-capturing unit 51. Accordingly, the
focal distance of the multi-viewpoint color image imagecapturing
unit 51 in the corrected multi-viewpoint color
image in the horizontal direction (X direction) is the
same at all the viewpoints. The multi-viewpoint color
5 image correction unit 52 provides the corrected multiviewpoint
color image to the multi-viewpoint parallax
related image generation unit 53 and the multi-viewpoint
image encoding unit 55 as multi-viewpoint corrected color
images.
10 [0134]
On the basis of the parallax related maximum value
and the parallax related minimum value provided from the
multi-viewpoint color image image-capturing unit 51, the
multi-viewpoint parallax related image generation unit 53
15 generates parallax related images of multiple viewpoints
from the multi-viewpoint corrected color images provided
from the multi-viewpoint color image correction unit 52.
More specifically, the multi-viewpoint parallax related
image generation unit 53 obtains a parallax related value
20 which has not yet normalized at each pixel (the
reciprocal number 1/Z of the depth Z or the parallax d)
from the multi-viewpoint corrected color image for each
viewpoint of multiple viewpoints, and normalizes the
parallax related value which has not yet normalized on
25 the basis of the parallax related maximum value and the
parallax related minimum value. Then, the multiviewpoint
parallax related image generation unit 53
generates a parallax related image in which the parallax
related value (value y,value I) of each pixel normalized
30 is a pixel value of the pixel of the parallax related
image for each viewpoint of the multiple viewpoints.
focal distance of the multi-viewpoint color image imagecapturing
unit 51 in the corrected multi-viewpoint color
image in the horizontal direction (X direction) is the
same at all the viewpoints. The multi-viewpoint color
5 image correction unit 52 provides the corrected multiviewpoint
color image to the multi-viewpoint parallax
related image generation unit 53 and the multi-viewpoint
image encoding unit 55 as multi-viewpoint corrected color
images.
10 [0134]
On the basis of the parallax related maximum value
and the parallax related minimum value provided from the
multi-viewpoint color image image-capturing unit 51, the
multi-viewpoint parallax related image generation unit 53
15 generates parallax related images of multiple viewpoints
from the multi-viewpoint corrected color images provided
from the multi-viewpoint color image correction unit 52.
More specifically, the multi-viewpoint parallax related
image generation unit 53 obtains a parallax related value
20 which has not yet normalized at each pixel (the
reciprocal number 1/Z of the depth Z or the parallax d)
from the multi-viewpoint corrected color image for each
viewpoint of multiple viewpoints, and normalizes the
parallax related value which has not yet normalized on
25 the basis of the parallax related maximum value and the
parallax related minimum value. Then, the multiviewpoint
parallax related image generation unit 53
generates a parallax related image in which the parallax
related value (value y,value I) of each pixel normalized
30 is a pixel value of the pixel of the parallax related
image for each viewpoint of the multiple viewpoints.
[0135]
Then, the multi-viewpoint parallax related image
generation unit 53 provides the generated parallax
related images of multiple viewpoints to the multi-
5 viewpoint image encoding unit 55 as the multi-viewpoint
parallax related image. Further, the multi-viewpoint
parallax related image generation unit 53 generates
parallax accuracy parameter (depth accuracy parameter)
representing the accuracy of the pixel value of the
10 multi-viewpoint parallax related image, and provides the
parallax accuracy parameter to the viewpoint generation
information generation unit 54.
[0136]
The viewpoint generation information generation
15 unit 54 functions as a generation unit, and uses the
multi-viewpoint corrected color image and the multiviewpoint
parallax related image to generate viewpoint
generation information in accordance with a predetermined
method for generating a color image of another viewpoint.
20 More specifically, the viewpoint generation information
generation unit 54 obtains the inter-camera distance on
the basis of the external parameters provided by the
multi-viewpoint color image image-capturing unit 51. The
inter-camera distance is a distance between the position
25 of the multi-viewpoint color image image-capturing unit
51 in the horizontal direction when the multi-viewpoint
color image image-capturing unit 51 captures a color
image at each viewpoint of the multi-viewpoint parallax
related image and the position of the multi-viewpoint
30 color image image-capturing unit 51 in the horizontal
direction when the multi-viewpoint color image imagecapturing
unit 51 captures a color image having a
parallax corresponding to the parallax related image with
respect to the color image thus captured.
5 The viewpoint generation information generation
unit 54 adopts, as viewpoint generation information, the
parallax related maximum value and the parallax related
minimum value provided by the multi-viewpoint color image
image-capturing unit 51, the inter-camera distance, and
10 the parallax accuracy parameter provided by the multiviewpoint
parallax related image generation unit 53. The
viewpoint generation information generation unit 54
provides the generated viewpoint generation information
to the multi-viewpoint image encoding unit 55.
15 [0138]
The multi-viewpoint image encoding unit 55
functions as an encoding unit, and encodes the multiviewpoint
corrected color image provided by the multiviewpoint
color image correction unit 52 and the multi-
20 viewpoint parallax related image provided by the multiviewpoint
parallax related image generation unit 53 in
accordance with the method according to HEVC (High
Efficiency Video Coding) method. As of August, 2011,
Thomas Wiegand, Woo-jin Han, Benjamin Bross, Jens-Rainer
25 Ohm, Gary J. Sullivian, "WD3: Working Draft3 of High-
Efficiency Video Coding", JCTVC-E603 d5 (- version 5), May
20, 2011 was issued as a draft with regard to HEVC method.
[0139]
The multi-viewpoint image encoding unit 55 performs
30 differential encoding on the parallax related maximum
value, the parallax related minimum value, and the intercamera
distance in the viewpoint generation information
provided by the viewpoint generation information
generation unit 54, and causes such information to be
included information about encoding of the multi-
5 viewpoint parallax related image. Then, the multiviewpoint
image encoding unit 55 transmits as an encoded
bit stream, a bit stream including the multi-viewpoint
corrected color images and the multi-viewpoint parallax
related image which are encoded, the parallax related
10 maximum value and the parallax related minimum value
which are differential-encoded, the information about
encoding including the inter-camera distance, the
parallax accuracy parameter provided by the viewpoint
generation information generation unit 54, and the like.
15 [0140]
As described above, the multi-viewpoint image
encoding unit 55 differential-encodes and transmits the
parallax related maximum value, the parallax related
minimum value, and the inter-camera distance, and
20 therefore, can reduce the amount of codes of the
viewpoint generation information. In order to provide
comfortable 3D image, it is likely not to greatly change
the parallax related maximum value, the parallax related
minimum value, and the inter-camera distance between
25 pictures, and therefore, the differential encoding is
effective for reducing the amount of codes.
[0141]
[Explanation about viewpoint generation information]
Fig. 12 is a figure explaining a parallax related
30 maximum value and a parallax related minimum value of
viewpoint generation information.
[0142]
In Fig. 12, the horizontal axis denotes the
parallax related value that is not normalized, and the
vertical axis denotes the pixel value of the parallax
5 related image (depth information).
[0143]
As illustrated in Fig. 12, the multi-viewpoint
parallax related image generation unit 53 normalizes the
non-normalized parallax related value of each pixel (the
10 reciprocal number 1/Z of the depth Z or.parallax d) into,
for example, a value of 0 to 255, using the parallax
related minimum value Dmin and the parallax related
maximum value Dmax. Then, the multi-viewpoint parallax
related image generation unit 53 generates a parallax
15 related image in which the normalized parallax related
value of each pixel which is a value of any one of 0 and
255 (depth information) is a pixel value.
[0144]
More specifically, as described above, the pixel
20 value I of each pixel in the parallax related image is
represented by the following expression (4), using the
non-normalized parallax related value d (parallax d), the
parallax related minimum value Dminf and the parallax
related maximum value Dm,, of that pixel.
25 [0145]
[Mathematical Formula 21
Therefore, in the decoding device explained later,
30 it is necessary to restore the non-normalized parallax
related value d from the pixel value I of each pixel in
the parallax related image in accordance with the
following expression (5) using the parallax related
minimum value &in and the parallax related maximum value
5 Dmax
[0147]
[Mathematical Formula 31
[0148]
10 More specifically, as described above, the pixel
value y of each pixel in the parallax related image is
represented by the expression (c), using the nonnormalized
parallax related value Z (parallax Z), the
parallax related minimum value Z,,,,, and the parallax
15 related maximum value Zfar of that pixel. Therefore, in
the decoding device explained later, it is necessary to
restore the parallax related value Z from the pixel value
y of each pixel in the parallax related image using the
parallax related minimum value Z,,,, and the parallax
20 related maximum value Zfar. Accordingly, the parallax
related minimum value and the parallax related maximum
value are transmitted to the decoding device.
[0149]
Fig. 13 is a figure explaining parallax accuracy
25 parameter of the viewpoint generation information.
[0150]
As illustrated in the upper stage of Fig. 13, when
the non-normalized parallax related value (the reciprocal
number 1/Z of the depth Z or parallax d) per normalized
30 parallax related value (depth information) is 0.5, the
parallax accuracy parameter represents 0.5 which is the
accuracy of the normalized parallax related value (depth
information). As illustrated in the lower stage of Fig.
13, when the non-normalized parallax related value per
5 normalized parallax related value is one, the parallax
accuracy parameter represents 1.0 which is the accuracy
of the parallax related value.
[0151]
In the example of Fig. 13, the non-normalized
10 parallax related value at the viewpoint #1 at the first
viewpoint is 1.0, and the non-normalized parallax related
value at the viewpoint #2 at the second viewpoint is 0.5.
Therefore, the normalized parallax related value of the
viewpoint #1 is 1.0 even though the accuracy of the
15 parallax related value is either 0.5 or 1.0. On the
other hand, the parallax related value of the viewpoint
#2 is 0.5 even though the accuracy of the parallax
related value is 0.5, and the parallax related value of
the viewpoint #2 is 0 even though the accuracy of the
20 parallax related value is 1.0.
[0152]
Fig. 14 is a figure explaining an inter-camera
distance of the viewpoint generation information.
[0153]
25 As illustrated in Fig. 14, the inter-camera
distance of the parallax related image of the viewpoint
#1 with respect to the viewpoint #2 is a distance between
the position represented by the external parameter of the
viewpoint #1 and the position represented by the external
30 parameter of the viewpoint #2.
[0154]
[Example of configuration of multi-viewpoint image
encoding unit]
Fig. 15 is a block diagram illustrating an example
of configuration of the multi-viewpoint image encoding
5 unit 55 of Fig. 11.
[0155]
The multi-viewpoint image encoding unit 55 of Fig.
15 includes a slice encoding unit 61, a slice header
encoding unit 62, a PPS encoding unit 63, and an SPS
10 encoding unit 64.
[0156]
The slice encoding unit 61 of the multi-viewpoint
image encoding unit 55 encodes the multi-viewpoint
corrected color image provided by the multi-viewpoint
15 color image correction unit 52 and the multi-viewpoint
parallax related image provided by the multi-viewpoint
parallax related image generation unit 53 in accordance
with the method according to the HEVC method in units of
slices. The slice encoding unit 61 provides the slice
20 header encoding unit 62 with encoded data in units of
slices obtained as a result of encoding.
[0157]
The slice header encoding unit 62 determines that
the parallax related maximum value, the parallax related
25 minimum value, and the inter-camera distance in the
viewpoint generation information provided by the
viewpoint generation information generation unit 54 are
the parallax related maximum value, the parallax related
minimum value, and the inter-camera distance of the slice
30 of the current processing target, and holds them.
[0158]
The slice header encoding unit 62 also determines
whether the parallax related maximum value, the parallax
related minimum value, and the inter-camera distance of
the slice of the current processing target are the same
5 as the parallax related maximum value, the parallax
related minimum value, and the inter-camera distance of
the previous slice in the order of encoding with respect
to the slice, and this determination is made in unit to
which the same PPS is given (hereinafter referred to as
10 the same PPS unit).
[0159]
Then, when all the parallax related maximum value,
the parallax related minimum value, and the inter-camera
distance of the slice constituting the same PPS unit are
15 determined to be the same as the parallax related maximum
value, the parallax related minimum value, and the intercamera
distance of the previous slice in the order of
encoding, the slice header encoding unit 62 adds
information about encoding other than the parallax
20 related maximum value, the parallax related minimum value,
and the inter-camera distance of that slice as the slice
header of the encoded data of each slice constituting the
same PPS unit, and provides the information to the PPS
encoding unit 63. The slice header encoding unit 62
25 provides the PPS encoding unit 63 with a transmission
flag indicating absence of transmission of the
difference-encoded results of the parallax related
maximum value, the parallax related minimum value, and
the inter-camera distance.
30 [0160]
On the other hand, when all the parallax related
maximum value, the parallax related minimum value, and
the inter-camera distance of at least one slice
constituting the same PPS unit are determined not to be
the same as the parallax related maximum value, the
5 parallax related minimum value, and the inter-camera
distance of the previous slice in the order of encoding,
the slice header encoding unit 62 adds information about
encoding including the parallax related maximum value,
the parallax related minimum value, and the inter-camera
10 distance of that slice as the slice header to the encoded
data of the intra-type slice, and provides the
information to the PPS encoding unit 63.
[0161]
The slice header encoding unit 62 performs
15 difference encoding on the parallax related maximum value,
the parallax related minimum value, and the inter-camera
distance of the inter-type slice. More specifically, the
slice header encoding unit 62 subtracts the parallax
related maximum value, the parallax related minimum value,
20 and the inter-camera distance of the previous slice in
the order of encoding with respect to the slice from the
parallax related maximum value, the parallax related
minimum value, and the inter-camera distance of the
inter-type slice, respectively, and obtains a difference-
25 encoded result. Then, the slice header encoding unit 62
adds information about encoding including the differenceencoded
result of the parallax related maximum value, the
parallax related minimum value, and the inter-camera
distance as the slice header to the encoded data of the
30 inter-type slice, and provides the information to the PPS
encoding unit 63.
[ 0 162 ]
In this case, the slice header encoding unit 62
provides the PPS encoding unit 63 with a transmission
flag indicating presence of transmission of the
5 difference-encoded results of the parallax related
maximum value, the parallax related minimum value, and
the inter-camera distance.
[0163]
The PPS encoding unit 63 generates PPS including
10 the transmission flag provided from the slice header
encoding unit 62 and the parallax accuracy parameter in
the viewpoint generation information provided from the
viewpoint generation information generation unit 54 of
Fig. 11. The PPS encoding unit 63 adds, in the same PPS
15 unit, the PPS to the encoded data in units of slices to
which the slice header provided from the slice header
encoding unit 62 is added, and provides it to the SPS
encoding unit 64.
[0164]
20 The SPS encoding unit 64 generates SPS. Then, the
SPS encoding unit 64 adds, in units of sequences, the SPS
to the encoded data to which the PPS provided from the
PPS encoding unit 63 is added. The SPS encoding unit 64
transmits, as an encoded bit stream, the bit stream
25 obtained as a result.
[0165]
[Example of configuration of encoded bit stream]
Fig. 16 is a figure illustrating an example of
configuration of an encoded bit stream.
30 [0166]
In Fig. 16, for the sake of explanation, only
encoded data of slice of the multi-viewpoint parallax
related image are described. But in reality, the encoded
bit stream is also arranged with the encoded data of
slice of the multi-viewpoint color image.
5 [0167]
In the example of Fig. 16, the parallax related
maximum value, the parallax related minimum value, and
the inter-camera distance of the two inter-type slices
and one intra-type slice constituting the same PPS unit
10 of the PPS #O which is the 0th PPS are not the same as
the parallax related maximum value, the parallax related
minimum value, and the inter-camera distance,
respectively, of the previous slice in the order of
encoding. Therefore, the PPS #O includes a transmission
15 flag "1" indicating presence of transmission. In the
example of Fig. 16, the parallax accuracy of the slice
constituting the same PPS unit of the PPS #O is 0.5, and
the PPS #O includes "1" representing the parallax
accuracy 0.5 as the parallax accuracy parameter.
20 [0168]
Further, in the example of Fig. 16, the parallax
related minimum value of the intra-type slice
constituting the same PPS unit of the PPS #O is 10, and
the parallax related maximum value thereof is 50, and the
25 inter-camera distance thereof is 100. Therefore, the
slice header of the slice includes the parallax related
minimum value "10", the parallax related maximum value
"50", and the inter-camera distance "100".
[0169]
30 In the example of Fig. 16, the parallax related
minimum value of the first inter-type slice constituting
the same PPS unit of the PPS #O is 9, and the parallax
related maximum value thereof is 48, and the inter-camera
distance thereof is 105. Therefore, the slice header of
the slice includes, as the difference-encoded result of
5 the parallax related minimum value, difference "-1" which
is obtained by subtracting the parallax related minimum
value "10" of the intra-type slice which is arranged
before the slice in the order of encoding from the
parallax related minimum value "9" of the slice.
10 Likewise, the difference "-2" of the parallax related
maximum value is included as the difference-encoded
result of the parallax related maximum value, and the
difference "5" of the inter-camera distance is included
as the difference-encoded result of the inter-camera
15 distance.
[0170]
Further, in the example of Fig. 16, the parallax
related minimum value of the second inter-type slice
constituting the same PPS unit of the PPS #O is 7, and
20 the parallax related maximum value thereof is 47, and the
inter-camera distance thereof is 110. Therefore, the
slice header of the slice includes, as the differenceencoded
result of the parallax related minimum value,
difference "-2" which is obtained by subtracting the
25 parallax related minimum value "9" of the first intertype
slice which is arranged before the slice in the
order of encoding from the parallax related minimum value
"7" of the slice. Likewise, the difference "-1" of the
parallax related maximum value is included as the
30 difference-encoded result of the parallax related maximum
value, and the difference "5" of the inter-camera
distance is included as the difference-encoded result of
the inter-camera distance.
[0171]
In the example of Fig. 16, the parallax related
5 maximum value, the parallax related minimum value, and
the inter-camera distance of the two inter-type slices
and one intra-type slice constituting the same PPS unit
of the PPS #1 which is the 1st PPS are the same as the
parallax related maximum value, the parallax related
10 minimum value, and the inter-camera distance,
respectively, of the previous slice in the order of
encoding. More specifically, the parallax related
maximum value, the parallax related minimum value, and
the inter-camera distance of the two inter-type slices
15 and one intra-type slice constituting the same PPS unit
of the PPS #1 which is the 1st PPS are "7", "47", and
"110", respectively, which are the same as those of the
second inter-type slice constituting the same PPS unit of
the PPS #O. Therefore, the PPS #1 includes a
20 transmission flag "0" indicating absence of transmission.
In the example of Fig. 16, the parallax accuracy of the
slice constituting the same PPS unit of the PPS #1 is 0.5,
and the PPS #1 includes "1" representing the parallax
accuracy 0.5 as the parallax accuracy parameter.
25 [0172]
[Example of syntax of PPS]
Fig. 17 is a figure illustrating an example of
syntax of PPS of Fig. 16.
[0173]
30 As illustrated in Fig. 17, the PPS includes
parallax accuracy parameter (disparity- p recision) and
transmission flag (dsiparity- p ic- s ame- f lag). For example,
the parallax accuracy parameter is "0" when it indicates
the parallax accuracy (the accuracy of the depth
information) 1, and the parallax accuracy parameter is
5 "2" when it indicates the parallax accuracy (the accuracy
of the depth information) 0.25. As described above, the
parallax accuracy parameter is "1" when it indicates the
parallax accuracy (the accuracy of the depth information)
0.5. As described above, the transmission flag is "1"
10 when it indicates presence of transmission, and the
transmission flag is "0" when it indicates absence of
transmission.
101741
[Example of syntax of slice header]
15 Figs. 18 and 19 are figures illustrating an example
of syntax of slice header.
[01751
As illustrated in Fig. 19, when the transmission
flag is 1, and the type of the slice is intra-type, the
20 slice header includes the parallax related minimum value
(minimum - disparity), the parallax related maximum value
(maximum- d isparity), and the inter-camera distance
(translation- x ).
[0176]
25 On the other hand, when the transmission flag is 1,
and the type of the slice is inter-type, the slice header
includes the difference-encoded result of the parallax
related minimum value (delta- m inimum- d isparity), the
difference-encoded result of the parallax related maximum
30 value (delta- m aximum- d isparity), and the differenceencoded
result of the inter-camera distance
(delta translation -t ranslation- x ) .
[Explanation about processing of encoding device]
Fig. 20 is a flowchart explaining encoding
5 processing of the encoding device 50 of Fig. 11.
[0178]
In step Slll of Fig. 20, the multi-viewpoint color
image image-capturing unit 51 of the encoding device 50
captures color images of multiple viewpoints, and
10 provides them as multi-viewpoint color images to the
multi-viewpoint color image correction unit 52.
[0179]
In step ,5112, the multi-viewpoint color image
image-capturing unit 51 generates the parallax related
15 maximum value, the parallax related minimum value, and
the external parameter. The multi-viewpoint color image
image-capturing unit 51 provides the parallax related
maximum value, the parallax related minimum value, and
the external parameter to the viewpoint generation
20 information generation unit 54, and provides the parallax
related maximum value and the parallax related minimum
value to the multi-viewpoint parallax related image
generation unit 53.
[0180]
25 In step S113, the multi-viewpoint color image
correction unit 52 performs color correction, brightness
correction, distortion correction, and the like on the
multi-viewpoint color images provided from the multiviewpoint
color image image-capturing unit 51.
30 Accordingly, the focal distance of the multi-viewpoint
color image image-capturing unit 51 in the corrected
multi-viewpoint color image in the horizontal direction
(X direction) is the same at all the viewpoints. The
multi-viewpoint color image correction unit 52 provides
the corrected multi-viewpoint color image to the multi-
5 viewpoint parallax related image generation unit 53 and
the multi-viewpoint image encoding unit 55 as multiviewpoint
corrected color images.
[0181]
In step S114, on the basis of the parallax related
10 maximum value and the parallax related minimum value
provided from the multi-viewpoint color image imagecapturing
unit 51, the multi-viewpoint parallax related
image generation unit 53 generates parallax related
images of multiple viewpoints from the multi-viewpoint
15 corrected color images provided from the multi-viewpoint
color image correction unit 52. Then, the multiviewpoint
parallax related image generation unit 53
provides the generated parallax related images of
multiple viewpoints to the multi-viewpoint image encoding
20 unit 55 as the multi-viewpoint parallax related image.
101821
In step S115, the multi-viewpoint parallax related
image generation unit 53 generates parallax accuracy
parameters, and provides the parallax accuracy parameters
25 to the viewpoint generation information generation unit
54.
[0183]
In step S116, the viewpoint generation information
generation unit 54 obtains the inter-camera distance on
30 the basis of the external parameters provided by the
multi-viewpoint color image image-capturing unit 51.
[ 0 18 4 1
In step S117, the viewpoint generation information
generation unit 54 generates, as viewpoint generation
information, the parallax related maximum value and the
5 parallax related minimum value provided by the multiviewpoint
color image image-capturing unit 51, the intercamera
distance, and the parallax accuracy parameter
provided by the multi-viewpoint parallax related image
generation unit 53. The viewpoint generation information
10 generation unit 54 provides the generated viewpoint
generation information to the multi-viewpoint image
encoding unit 55.
[0185]
In step S118, the multi-viewpoint image encoding
15 unit 55 performs the multi-viewpoint encoding processing
for encoding the multi-viewpoint corrected color image
provided by the multi-viewpoint color image correction
unit 52 and the multi-viewpoint parallax related image
provided by the multi-viewpoint parallax related image
20 generation unit 53 in accordance with the method
according to HEVC method. The details of the multiviewpoint
encoding processing will be explained with
reference to Fig. 21 explained later.
[0186]
25 In step S119, the multi-viewpoint image encoding
unit 55 transmits the encoded bit stream obtained as a
result of the multi-viewpoint encoding processing, and
terminates the processing.
[0187]
30 Fig. 21 is a flowchart explaining the multiviewpoint
encoding processing in step S118 of Fig. 20.
[0188]
In step S131 of Fig. 21, the slice encoding unit 61
of the multi-viewpoint image encoding unit 55 (Fig. 15)
encodes the multi-viewpoint corrected color image
5 provided by the multi-viewpoint color image correction
unit 52 and the multi-viewpoint parallax related image
provided by the multi-viewpoint parallax related image
generation unit 53 in accordance with the method
according to the HEVC method in units of slices. The
10 slice encoding unit 61 provides the slice header encoding
unit 62 with encoded data in units of slices obtained as
a result of encoding.
[0189]
In step S132, the slice header encoding unit 62
15 determines that the parallax related maximum value, the
parallax related minimum value, and the inter-camera
distance in the viewpoint generation information provided
by the viewpoint generation information generation unit
54 are the parallax related maximum value, the parallax
20 related minimum value, and the inter-camera distance of
the slice of the current processing target, and holds
them.
[0190]
In step S133, the slice header encoding unit 62
25 determines whether the parallax related maximum value,
the parallax related minimum value, and the inter-camera
distance of all the slices constituting the same PPS unit
are the same as the parallax related maximum value, the
parallax related minimum value, and the inter-camera
30 distance of the previous slice in the order of encoding
with respect to the slice.
[0191]
When the inter-camera distance, the parallax
related maximum value, and the parallax related minimum
value are determined to be the same in step S133, the
5 slice header encoding unit 62 generates a transmission
flag indicating absence of transmission of the
difference-encoded results of the parallax related
maximum value, the parallax related minimum value, and
the inter-camera distance, and provides it to the PPS
10 encoding unit 63 in step S134.
[0192]
In step S135, the slice header encoding unit 62
adds, as slice header, information about encoding other
than the inter-camera distance, the parallax related
15 maximum value, and the parallax related minimum value of
that slice to the encoded data of each slice constituting
the same PPS unit which is the processing target of step
S133. Then, the slice header encoding unit 62 provides
the PPS encoding unit 63 with the encoded data of each
20 slice constituting the same PPS unit obtained as a result,
and proceeds to processing in step S140.
[0193]
On the other hand, when the inter-camera distance,
the parallax related maximum value, and the parallax
25 related minimum value are determined not to be the same
in step S133, the slice header encoding unit 62 generates
a transmission flag indicating presence of transmission
of the difference-encoded results of the parallax related
maximum value, the parallax related minimum value, and
30 the inter-camera distance, and provides it to the PPS
encoding unit 63 in step S136. The processing in steps
S137 to S139 explained later is performed for each slice
constituting the same PPS unit which is the processing
target of step S133.
[0194]
5 In step ,5137, the slice header encoding unit 62
determines whether the type of the slice constituting the
same PPS unit which is the processing target of step S133
is intra-type or not. When the type of the slice is
determined to be the intra-type in step S137, the slice
10 header encoding unit 62 adds information about encoding
including the parallax related maximum value, the
parallax related minimum value, and the inter-camera
distance of that slice as the slice header to the encoded
data of that slice in step S138. Then, the slice header
15 encoding unit 62 provides the PPS encoding unit 63 with
the encoded data in units of slices obtained as a result,
and proceeds to the processing in step S140.
[0195]
On the other hand, when the type of the slice is
20 determined not to be the intra-type in step S137, more
specifically, when the type of the slice is determined to
be the inter-type, the processing in step S139 is
performed. In step S139, the slice header encoding unit
62 performs difference encoding on the inter-camera
25 distance, the parallax related maximum value, and the
parallax related minimum value of that slice, and adds,
as slice header, information about encoding including the
difference-encoded result to that encoded data of that
slice. Then, the slice header encoding unit 62 provides
30 the PPS encoding unit 63 with the encoded data in units
of slices obtained as a result, and proceeds to the
processing in step S140.
[0196]
In step S140, the PPS encoding unit 63 generates
PPS including the transmission flag provided from the
5 slice header encoding unit 62 and the parallax accuracy
parameter in the viewpoint generation information
provided from the viewpoint generation information
generation unit 54 of Fig. 11.
[0197]
10 In step S141, the PPS encoding unit 63 adds, in the
same PPS unit, the PPS to the encoded data in units of
slices to which the slice header provided from the slice
header encoding unit 62 is added, and provides it to the
SPS encoding unit 64.
15 [0198]
In step S142, the SPS encoding unit 64 generates
SPS.
[0199]
In step S143, the SPS encoding unit 64 adds, in
20 units of sequences, the SPS to the encoded data to which
the PPS provided from the PPS encoding unit 63 is added,
and generates an encoded bit stream.
[0200]
As described above, the encoding device 50 arranges
25 the inter-camera distance, the parallax related maximum
value, and the parallax related minimum value in the
slice header as information about encoding. Accordingly,
for example, the inter-camera distance, the parallax
related maximum value, and the parallax related minimum
30 value can be used for encoding.
[0201]
[Example of configuration of second embodiment of
decoding device]
Fig. 22 is a block diagram illustrating an example
of configuration of the second embodiment of a decoding
5 device, serving as an image processing apparatus to which
the present technique is applied, which decodes an
encoded bit stream transmitted from the encoding device
50 of Fig. 11.
[0202]
10 In the configuration as illustrated in Fig. 22, the
same configurations as those of Fig. 7 are denoted with
the same reference numerals. Repeated explanations are
omitted as necessary.
[0203]
15 The configuration of the decoding device 80 of Fig.
22 is different from the configuration of Fig. 7 in that
the decoding device 80 includes a multi-viewpoint image
decoding unit 81 and a viewpoint compositing unit 82
instead of the multi-viewpoint image decoding unit 31 and
20 the viewpoint compositing unit 32. The decoding device
80 displays multi-viewpoint composed color images on the
basis of the viewpoint generation information transmitted
from the encoding device 50.
[0204]
25 More specifically, the multi-viewpoint image
decoding unit 81 of the decoding device 80 functions as a
receiving unit, and receives the encoded bit stream
transmitted from the encoding device 50 of Fig. 11. The
multi-viewpoint image decoding unit 81 extracts the
30 parallax accuracy parameter and the transmission flag
from the PPS included in the received encoded bit stream.
The multi-viewpoint image decoding unit 81 extracts the
inter-camera distance, the parallax related maximum value,
and the parallax related minimum value from the slice
header of the encoded bit stream in accordance with the
5 transmission flag. The multi-viewpoint image decoding
unit 81 generates viewpoint generation information
including the parallax accuracy parameter, the intercamera
distance, the parallax related maximum value, and
the parallax related minimum value, and provides it to
10 the viewpoint compositing unit 82.
102051
The multi-viewpoint image decoding unit 81
functions as a decoding unit, and decodes the encoded
data in units of slices included in the encoded bit
15 stream in accordance with the method corresponding to the
encoding method of the multi-viewpoint image encoding
unit 15 of Fig. 11, and generates the multi-viewpoint
corrected color image and the multi-viewpoint parallax
related image. The multi-viewpoint image decoding unit
20 81 provides the multi-viewpoint corrected color images
and the multi-viewpoint parallax related image to the
viewpoint compositing unit 82.
[0206]
The viewpoint compositing unit 82 uses the
25 viewpoint generation information provided by the multiviewpoint
image decoding unit 81 to perform warping
processing on the multi-viewpoint parallax related image
provided by the multi-viewpoint image decoding unit 81
to warp the image according to the display viewpoints.
30 More specifically, the viewpoint compositing unit 82
performs warping processing on the multi-viewpoint
parallax related image to warp the image according to the
display viewpoints with the accuracy corresponding to the
parallax accuracy parameter, on the basis of the intercamera
distance, the parallax related maximum value, the
5 parallax related minimum value, and the like included in
the viewpoint generation information. The viewpoint
compositing unit 82 uses the parallax related image of
display viewpoints obtained as a result of warping
processing to perform warping processing on the multi-
10 viewpoint corrected color images provided by the multiviewpoint
image decoding unit 81 to warp them according
to the display viewpoints. The viewpoint compositing
unit 82 provides the color images for the display
viewpoints obtained as a result, to the multi-viewpoint
15 image display unit 33 as the multi-viewpoint composed
color images.
[0207]
As described above, the viewpoint compositing unit
82 performs warping processing on the multi-viewpoint
20 parallax related images to warp them according to the
display viewpoints with the accuracy corresponding to the
viewpoint accuracy parameter on the basis of the parallax
accuracy parameter, and therefore, it is not necessary
for the viewpoint compositing unit 82 to wastefully
25 perform the warping processing with a high degree of
accuracy.
[0208]
In addition, the viewpoint compositing unit 82
performs warping processing on the multi-viewpoint
30 parallax related image to warp it according to the intercamera
distance, and therefore, when the parallax
corresponding to the parallax related value of the multiviewpoint
parallax related image having been subjected to
the warping processing is not in an appropriate range,
the parallax related value can be corrected to a value
5 corresponding to the parallax in the appropriate range on
the basis of the inter-camera distance.
[0209]
[Example of configuration of multi-viewpoint image
decoding unit]
10 Fig. 23 is a block diagram illustrating an example
of configuration of the multi-viewpoint image decoding
unit 81 of Fig. 22.
[0210]
The multi-viewpoint image decoding unit 81 of Fig.
15 23 includes an SPS decoding unit 101, a PPS decoding unit
102, a slice header decoding unit 103, and a slice
decoding unit 104.
[0211]
The SPS decoding unit 101 of the multi-viewpoint
20 image decoding unit 81 receives the encoded bit stream
transmitted from the encoding device 50 of Fig. 11, and
extracts the SPS in the encoded bit stream. The SPS
decoding unit 101 provides the PPS decoding unit 102 with
the encoded bit stream other than the SPS and the
25 extracted SPS.
[0212]
The PPS decoding unit 102 extracts PPS from the
encoded bit stream other than the SPS provided from the
SPS decoding unit 101. The PPS decoding unit 102
30 provides the slice header decoding unit 103 with the
encoded bit stream other than the SPS and the PPS, and
the PPS and the SPS extracted.
[0213]
The slice header decoding unit 103 extracts a slice
header from the encoded bit stream other than the SPS and
5 the PPS provided from the PPS decoding unit 102. When
the transmission flag included in the PPS provided by the
PPS decoding unit 102 is "1" indicating presence of
transmission, the slice header decoding unit 103 holds
the inter-camera distance, the parallax related maximum
10 value, and the parallax related minimum value included in
the slice header, or updates the inter-camera distance,
the parallax related maximum value, and the parallax
related minimum value held therein on the basis of the
difference-encoded result of the inter-camera distance,
15 the parallax related maximum value, and the parallax
related minimum value. The slice header decoding unit
103 generates viewpoint generation information from the
inter-camera distance, the parallax related maximum value,
and the parallax related minimum value, which are held,
20 and the parallax accuracy parameter included in the PPS,
and provides it to the viewpoint compositing unit 82.
Further, the slice header decoding unit 103 provides the
slice decoding unit 104 with the encoded data in units of
slices which is the encoded bit stream other than the SPS,
25 the PPS, and the slice header, and the SPS, the PPS, and
the slice header.
[0214]
The slice decoding unit 104 decodes the encoded
data in units of slices in accordance with the method
30 corresponding to the encoding method of the slice
encoding unit 61 (Fig. 15) on the basis of the SPS, the
PPS, and the slice header provided from the slice header
decoding unit 103. The slice header decoding unit 103
provides the viewpoint compositing unit 82 of Fig. 22
with the multi-viewpoint corrected color image and the
5 multi-viewpoint parallax related image obtained as a
result of decoding.
[0215]
[Explanation about processing of decoding device]
The decoding processing of the decoding device 80
10 of Fig. 22 is done in the same manner except the multiviewpoint
decoding processing in step S52 of Fig. 9, and
therefore, in the explanation below, only the multiviewpoint
decoding processing will be explained.
[0216]
15 Fig. 24 is a flowchart explaining the multiviewpoint
decoding processing of the multi-viewpoint
image decoding unit 81 of the decoding device 80 of Fig.
22.
[0217]
20 In step S161 of Fig. 24, the SPS decoding unit 101
of the multi-viewpoint image decoding unit 81 (Fig. 23)
extracts the SPS from the received encoded bit stream.
The SPS decoding unit 101 provides the PPS decoding unit
102 with the encoded bit stream other than the SPS and
25 the extracted SPS.
[0218]
In step S162, the PPS decoding unit 102 extracts
PPS from the encoded bit stream other than the SPS
provided from the SPS decoding unit 101. The PPS
30 decoding unit 102 provides the slice header decoding unit
103 with the encoded bit stream other than the SPS and
the PPS, and the PPS and the SPS extracted.
[0219]
In step S163, the slice header decoding unit 103
provides the parallax accuracy parameter included in the
5 PPS provided from the PPS decoding unit 102 to the
viewpoint compositing unit 82 as a portion of the
viewpoint generation information.
[0220]
In step S164, the slice header decoding unit 103
10 determines whether the transmission flag included in the
PPS provided by the PPS decoding unit 102 is "1"
indicating presence of transmission. It should be noted
that the subsequent processing in steps S165 to S174 is
done in units of slices.
15 [0221]
When the transmission flag is determined to be "1"
indicating presence of transmission in step S164, the
processing in step S165 is subsequently performed. In
step S165, the slice header decoding unit 103 extracts
20 the slice header including the parallax related maximum
value, the parallax related minimum value, and the intercamera
distance, or the difference-encoded result of the
parallax related maximum value, the parallax related
minimum value, and the inter-camera distance, from the
25 encoded bit stream other than the SPS and the PPS
provided from the PPS decoding unit 102.
[0222]
In step S166, the slice header decoding unit 103
determines whether the type of the slice is intra-type or
30 not. When the type of the slice is determined to be the
intra-type in step S166, the processing in step S167 is
subsequently performed.
[0223]
In step S167, the slice header decoding unit 103
holds the parallax related minimum value included in the
5 slice header extracted in step S165, and provides it to
the viewpoint compositing unit 82 as a portion of the
viewpoint generation information.
[0224]
In step S168, the slice header decoding unit 103
10 holds the parallax related maximum value included in the
slice header extracted in step S165, and provides it to
the viewpoint compositing unit 82 as a portion of the
viewpoint generation information.
[0225]
15 In step S169, the slice header decoding unit 103
holds the inter-camera distance included in the slice
header extracted in step S165, and provides it to the
viewpoint compositing unit 82 as a portion of the
viewpoint generation information. Then, the processing
20 in step S175 is performed.
[0226]
On the other hand, when the type of the slice is
determined not to be the intra-type in step S166, more
specifically, when the type of the slice is determined to
25 be the inter-type, the processing in step S170 is
performed.
[0227]
In step S170, the slice header decoding unit 103
adds the difference-encoded result of the parallax
30 related minimum value included in the slice header
extracted in step S165 to the held parallax related
minimum value. The slice header decoding unit 103
provides the parallax related minimum value restored by
adding operation to the viewpoint compositing unit 82 as
a part of the viewpoint generation information.
5 [0228]
In step S171, the slice header decoding unit 103
adds the difference-encoded result of the parallax
related maximum value included in the slice header
extracted in step S165 to the held parallax related
10 maximum value. The slice header decoding unit 103
provides the parallax related maximum value restored by
adding operation to the viewpoint compositing unit 82 as
a part of the viewpoint generation information.
[0229]
15 In step S172, the slice header decoding unit 103
adds the difference-encoded result of the inter-camera
distance included in the slice header extracted in step
S165 to the held inter-camera distance. The slice header
decoding unit 103 provides the inter-camera distance
20 restored by adding operation to the viewpoint compositing
unit 82 as a part of the viewpoint generation information.
Then, the processing in step S175 is performed.
[0230]
When the transmission flag is determined not to be
25 "1" indicating presence of transmission in step S164,
i.e., the transmission flag is determined to be "0"
indicating absence of transmission, the processing in
step S173 is subsequently performed.
[0231]
30 In step S173, the slice header decoding unit 103
extracts the slice header not including the parallax
related maximum value, the parallax related minimum value,
and the inter-camera distance, and the difference-encoded
result of the parallax related maximum value, the
parallax related minimum value, and the inter-camera
5 distance, from the encoded bit stream other than the SPS
and the PPS provided from the PPS decoding unit 102.
[0232]
In step S174, the slice header decoding unit 103
determines that the parallax related maximum value, the
10 parallax related minimum value, and the inter-camera
distance which are held, i.e., the parallax related
maximum value, the parallax related minimum value, and
the inter-camera distance of the previous slice in the
order of encoding, are the parallax related maximum value,
15 the parallax related minimum value, and the inter-camera
distance of the slice of the processing target, whereby
the parallax related maximum value, the parallax related
minimum value, and the inter-camera distance of the slice
of the processing target are restored. Then, the slice
20 header decoding unit 103 provides the parallax related
maximum value, the parallax related minimum value, and
the inter-camera distance thus restored to the viewpoint
compositing unit 82 as a portion of the viewpoint
generation information, and proceeds to the processing in
25 step S175.
[0233]
In step S175, the slice decoding unit 104 decodes
the encoded data in units of slices in accordance with
the method corresponding to the encoding method of the
30 slice encoding unit 61 (Fig. 15) on the basis of the SPS,
the PPS, and the slice header provided from the slice
header decoding unit 103. The slice header decoding unit
103 provides the viewpoint compositing unit 82 of Fig. 22
with the multi-viewpoint corrected color image and the
multi-viewpoint parallax related image obtained as a
5 result of decoding.
[0234]
As described above, the decoding device 80 can
decode the encoded bit stream in which the parallax
related maximum value, the parallax related minimum value,
10. and the inter-camera distance are arranged as the
information about encoding in the slice header.
Accordingly, for example, the encoded bit stream in which
the parallax related maximum value, the parallax related
minimum value, and the inter-camera distance are used for
15 encoding can be decoded.
[0235]
In the second embodiment, the parallax related
maximum value, the parallax related minimum value, and
the inter-camera distance in the viewpoint generation
20 information are included in the slice header, but may be
described in SPS, PPS, SEI, and the like. In the second
embodiment, the viewpoint generation information may also
include the color image identification information, the
parallax related image identification information, the
25 external parameter information, the internal parameter
information, the number of viewpoints of color images,
information for identifying color images of viewpoints
adopted as a base point for obtaining the parallax value,
the number of viewpoints of parallax related image, the
30 image type information, and the like.
[0236]

[Explanation about computer to which the present
technique is applied]
Subsequently, the above series of processing may be
5 performed by either hardware or software. When the
series of processing is performed by software, a program
constituting the software is installed in a generalpurpose
computer and the like.
[0237]
10 F I G . 26 is an exemplary configuration of an
embodiment of a computer to which the program for
executing the above series of processing is installed.
[0238]
The program may be previously stored in a storage
15 unit 808 and a ROM (Read Only Memory) 802 serving as
recording media incorporated into the computer.
[0239]
Alternatively, the program may be stored (recorded)
to a removable medium 811. This removable medium 811 may
20 be provided as so-called package software. In this case,
examples of the removable medium 811 include, for example,
a flexible disk, a CD-ROM (Compact Disc Read Only Memory),
a MO (Magneto Optical) disk, a DVD (Digital Versatile
Disc), a magnetic disk, a semiconductor memory, and the
25 like.
[0240]
The program can be installed to the computer from
the removable medium 811 as described above using a drive
810. Alternatively, the program may be downloaded to the
30 computer via a communication network or a broadcast
network, and the program can be installed to the storage
unit 808 provided therein. More specifically, the
program can be, for example, transferred from a download
site to the computer wirelessly via a man-made satellite
for digital satellite broadcasting, or may be transferred
5 to the computer via the wire, i.e., network such as a LAN
(Local Area Network) and the Internet.
[0241]
The computer has a CPU (Central Processing Unit)
801. The CPU 801 is connected to an input/output
10 interface 805 via a bus 804.
[0242]
When a user operates an input unit 806 to input a
command to the CPU 801 via the input/output interface 805,
the CPU 801 executes the program stored in the ROM 802 in
15 response to the command. Alternatively, the CPU 801
loads the program stored in the storage unit 808 to a RAM
(Random Access Memory) 803, and executes the program.
[0243]
Thus, the CPU 801 performs the processing according
20 to the above flowcharts or the processing performed with
the configurations in the block diagrams as described
above. Then, as necessary, the CPU 801 outputs the
processing result thereof from an output unit 807 via the
input/output interface 805, transmits the processing
25 result thereof from the communication unit 809, or
records the processing result thereof to the storage unit
808, for example.
[0244]
The input unit 806 is constituted by a keyboard, a
30 mouse, a microphone, and the like. The output unit 807
is constituted by an LCD (Liquid Crystal Display), a
speaker, and the like.
[0245]
In this case, in this specification, the processing
performed according to the program by the computer may
5 not be necessarily performed in time-series according to
the order described in the flowcharts. In other words,
the processing performed according to the program by the
computer include processing executed independently or in
parallel (for example, processing performed with objects
10 or parallel processing).
[0246]
The program may be processed by one computer
(processor), or may be processed by a plurality of
computers as distributed processing. Further, the
15 program may be transferred to a computer located far away
and may be executed.
[0247]
The present technique can be applied to an image
processing apparatus used for communication via a network
20 medium such as satellite broadcast, cable TV (television),
the Internet, and portable telephone, or for processing
on recording media such as optical, magnetic disks, and
flash memories.
[0248]
25 The above-explained image processing apparatus can
be applied to any given electronic device. Hereinafter,
an example thereof will be explained.
[0249]

30 [Example of configuration of television device]
Fig. 27 illustrates an example of schematic
configuration of television device to which the present
technique is applied. The television device 900 includes
an antenna 901, a tuner 902, a demultiplexer 903, a
decoder 904, a video signal processing unit 905, a
5 display unit 906, an audio signal processing unit 907, a
speaker 908, and an external interface unit 909. Further,
the television device 900 includes a control unit 910 and
a user interface unit 911.
[0250]
10 The tuner 902 tunes into and decodes a desired
channel from among broadcast signals received by the
antenna 901, and outputs the encoded bit stream thus
obtained to the demultiplexer 903.
[0251]
15 The demultiplexer 903 extracts packets of video and
audio of a program, which is a viewing target, from the
encoded bit stream, and outputs the data of the extracted
packets to the decoder 904. The demultiplexer 903 also
provides packets of data such as EPG (Electronic Program
20 Guide) to the control unit 910. When scrambled, the
demultiplexer and the like performs descrambling.
[0252]
The decoder 904 performs decoding processing of
packets, and outputs the video data generated by the
25 decoding processing to the video signal processing unit
905, and the audio data to the audio signal processing
unit 907.
[0253]
The video signal processing unit 905 performs noise
30 reduction and video processing according to user settings
on the video data. The video signal processing unit 905
generates, e.g., video data of a program displayed on the
display unit 906 and image data made by processing based
on an application provided via a network. The video
signal processing unit 905 generates video data for
5 displaying a menu screen and the like for allowing
selection of items, and overlays the video data on the
video data of the program. The video signal processing
unit 905 generates a driving signal on the basis of the
video data thus generated, and drives the display unit
10 906.
[0254]
The display unit 906 displays, e.g., video of
program, by driving a display device (for example, liquid
crystal display device and the like) on the basis of the
15 driving signal provided by the video signal processing
unit 905.
[0255]
The audio signal processing unit 907 performs
predetermined processing such as noise reduction on the
20 audio data, and D/A conversion processing and
amplification processing of the audio data having been
processed, and provides the data to the speaker 908, thus
outputting audio.
[0256]
25 The external interface unit 909 is an interface for
connecting an external device and a network, and exchange
data such as video data and audio data.
[0257]
The control unit 910 is connected to the user
30 interface unit 911. The user interface unit 911 is
constituted by, e.g., operation switches and a remote
control signal reception unit, and provides an operation
signal according to user operation to the control unit
910.
[0258]
5 The control unit 910 is constituted by a CPU
(Central Processing Unit), a memory, and the like. The
memory stores, e.g., programs executed by the CPU,
various kinds of data required when processing is
performed with the CPU, EPG data, and data obtained via a
10 network. The programs stored in the memory is read and
executed by the CPU with predetermined timing such as
during activation of the television device 900. The CPU
executes the program, thus controlling each unit so as to
cause the television device 900 to operate in accordance
15 with user operation.
[0259]
The television device 900 is provided with the bus
912 for connecting the tuner 902, the demultiplexer 903,
the decoder 904, the video signal processing unit 905,
20 the audio signal processing unit 907, the external
interface unit 909, and the control unit 910.
[0260]
In the television device configured as described
above, the decoder 904 is provided with a function of an
25 image processing apparatus (image processing method) of
the present application. Accordingly, using the color
image of the predetermined viewpoint and the parallax
related image, a color image of a viewpoint other than
the predetermined viewpoint can be generated.
30 [0261]

[Example of configuration of portable telephone]
Fig. 28 illustrates an example of schematic
configuration of portable telephone to which the present
technique is applied. A portable telephone 920 includes
5 a communication unit 922, an audio codec 923, a camera
unit 926, an image processing unit 927, a demultiplexer
928, a recording/reproducing unit 929, a display unit 930,
and a control unit 931. They are connected with each
other via a bus 933.
10 [0262]
The communication unit 922 is connected to the
antenna 921, and the audio codec 923 is connected to the
speaker 924 and the microphone 925. Further, the control
unit 931 is connected to the operation unit 932.
15 [0263]
The portable telephone 920 performs various kinds
of operation such as transmission and reception of audio
signal, transmission and reception of e-mail and image
data, image capturing, or recording of data, in various
20 kinds of modes such as audio phone call mode and data
communication mode.
[0264]
In the audio phone call mode, an analog audio
signal generated by the microphone 925 is, e.g.,
25 converted into audio data and compressed by the audio
codec 923, and is provided to the communication unit 922.
The communication unit 922 performs, e.g., modulation
processing and frequency conversion processing on the
audio data, and generates a transmission signal. The
30 communication unit 922 provides the transmission signal
to the antenna 921, and transmits the transmission signal
to a base station, not shown. The communication unit 922
performs, e.g., amplification, frequency conversion
processing and amplification processing on the reception
signal received by the antenna 921, and provides the
5 obtained audio data to the audio codec 923. The audio
codec 923 performs, e.g., data extension of the audio
data and conversion into analog audio signal, and outputs
the data to the speaker 924.
[0265]
10 When mail is transmitted in the data communication
mode, the control unit 931 receives character data which
are input by operation with the operation unit 932, and
displays the input characters on the display unit 930.
The control unit 931 also generates mail data on the
15 basis of, e.g., user instruction with the operation unit
932, and provides the mail data to the communication unit
922. The communication unit 922 performs, e.g.,
modulation processing and frequency conversion processing
on mail data, and transmits the obtained transmission
20 signal via the antenna 921. The communication unit 922
performs, e.g., amplification, frequency conversion
processing and amplification processing on the reception
signal received by the antenna 921, and restores the mail
data. The mail data are provided to the display unit 930,
25 and the contents of the mail are displayed.
[0266]
The portable telephone 920 can also store the
received mail data to a storage medium using the
recording/reproducing unit 929. The storage medium any
30 given rewritable storage medium. For example, the
storage medium may be a semiconductor memory such as RAM
and internal flash memory, a hard disk, or a removable
medium such as a magnetic disk, a magneto optical disk,
an optical disk, a USB memory, or a memory card.
[0267]
5 When image data are transmitted in the data
communication mode, the image data generated by the
camera unit 926 are provided to the image processing unit
927. The image processing unit 927 performs encoding
processing of the image data and generates encoded data.
10 [026'8]
The demultiplexer 928 multiplexes the encoded data
generated by the image processing unit 927 and the audio
data provided from the audio codec 923 in accordance with
predetermined method, and provides the data to the
15 communication unit 922. The communication unit 922
performs, e.g., modulation processing and frequency
conversion processing on the multiplexed data, and
transmits the obtained transmission signal via the
antenna 921. The communication unit 922 performs, e.g.,
20 amplification, frequency conversion processing and
amplification processing on the reception signal received
by the antenna 921, and restores the multiplexed data.
The multiplexed data are provided to the demultiplexer
928. The demultiplexer 928 separates the multiplexed
25 data, and provides the encoded data to the image
processing unit 927, and the audio data to the audio
codec 923. The image processing unit 927 performs
decoding processing of the encoded data and generates
image data. The image data are provided to the display
30 unit 930, and the received image is displayed. The audio
codec 923 converts the audio data into analog audio
signal, and provides the analog audio signal to the
speaker 924 so as to output the received audio.
[0269]
In the portable telephone configured as described
5 above, the image processing unit 927 is provided with a
function of an image processing apparatus (image
processing method) of the present application.
Accordingly, using the color image of the predetermined
viewpoint and the parallax related image, information
10 required to generate a color image of a viewpoint other
than the predetermined viewpoint can be transmitted.
Using the color image of the predetermined viewpoint and
the parallax related image, a color image of a viewpoint
other than the predetermined viewpoint can be generated.
15 LO2701

[Example of configuration of recording/reproducing
device]
Fig. 29 illustrates an example of schematic
20 configuration of recording/reproducing device to which
the present technique is applied. For example, a
recording/reproducing device 940 records audio data and
video data of received broadcasting program to a
recording medium, and provides a user with the recorded
25 data with timing according to user's instruction. For
example, the recording/reproducing device 940 can obtain
the audio data and the video data from another device,
and can record them to the recording medium. Further,
the recording/reproducing device 940 decodes and outputs
30 the audio data and the video data recorded in the
recording medium, thus allowing image display and audio
output with the monitor device and the like.
LO2711
The recording/reproducing device 940 includes a
tuner 941, an external interface unit 942, an encoder 943,
5 an HDD (Hard Disk Drive) unit 944, a disk drive 945, a
selector 946, a decoder 947, an OSD (On-Screen Display)
unit 948, a control unit 949, and a user interface unit
950.
[0272]
10 The tuner 941 tunes in to a desired channel from
broadcasting signals received by an antenna, not shown.
The tuner 941 outputs, to the selector 946, an encoded
bit stream obtained by amplifying the reception signal of
the desired channel.
15 [0273]
The external interface unit 942 may be constituted
by, for example, any one of an IEEE1394 interface, a
network interface unit, a USB interface, a flash memory
interface, and the like. The external interface unit 942
20 is an interface for connecting to an external device, a
network, a memory card, and the like, and receives data
such as the video data, the audio data, and the like thus
recorded.
[0274]
25 The encoder 943 performs encoding according to a
predetermined method when the video data and the audio
data provided from the external interface unit 942 are
not encoded, and outputs the encoded bit stream to the
selector 946.
30 [0275]
The HDD unit 944 records the content data such as
video and audio, other data of various kinds of programs,
and the l i k e t o an internal hard disk, and reads them
from the hard disk during play back and the l i k e .
[0276]
5 The d i s k d r i v e 945 records and reproduces the
signal to/from an optical disk loaded. For example, the
optical disk is a DVD disk (DVD-Video, DVD-RAM, DVD-R,
DVD-RW, DVD+R, DVD+RW and the l i k e ) , a Blu-ray disk, and
the l i k e .
10 [0277]
During recording of video and audio, t h e s e l e c t o r
946 s e l e c t s an encoded b i t stream from any one of the
tuner 941 and the encoder 943, and provides it to any one
of the HDD unit 944 and t h e d i s k drive 945. During
15 reproduction of video and audio, the selector 946
provides the decoder 947 with the encoded b i t stream
which is output from the HDD unit 944 o r t h e disk drive
945.
[0278]
20 The decoder 947 performs decoding processing of the
encoded b i t stream. The decoder 947 performs decoding
processing, thereby providing the generated video data t o
the OSD unit 948. The decoder 947 performs decoding
processing, thereby outputting the generated audio data.
25 [0279]
The OSD unit 948 generates video data for
displaying menu screen and the l i k e such as item
selection, and overlays and outputs them on the video
data which are output from the decoder 947.
30 [0280]
The control unit 949 is connected t o the user
interface unit 950. The user interface unit 950 is
constituted by, e.g., operation switches and a remote
control signal reception unit, and provides an operation
signal according to user operation to the control unit
5 949.
[0281]
The control unit 949 is constituted by a CPU, a
memory, and the like. The memory stores, e.g., programs
executed by the CPU, various kinds of data required when
10 processing is performed with the CPU. The programs
stored in the memory is read and executed by the CPU with
predetermined timing such as during activation of the
recording/reproducing device 940. The CPU executes the
program, thus controlling each unit so as to cause the
15 recording/reproducing device 940 to operate in accordance
with user operation.
[0282]
In the recording/reproducing device configured as
described above, the decoder 947 is provided with a
20 function of an image processing apparatus (image
processing method) of the present application. .
Accordingly, using the color image of the predetermined
viewpoint and the parallax related image, a color image
of a viewpoint other than the predetermined viewpoint can
25 be generated.
[0283]
[Example of configuration of image-capturing device]
Fig. 30 illustrates an example of schematic
30 configuration of image-capturing device to which the
present technique is applied. An image-capturing device
960 captures an image of a subject, displays the image of
the subject on the display unit, and records the image
data to a recording medium.
[0284]
5 The image-capturing device 960 includes an optical
block 961, an image-capturing unit 962, a camera signal
processing unit 963, an image data processing unit 964, a
display unit 965, an external interface unit 966, a
memory unit 967, a medium drive 968, an OSD unit 969, and
10 a control unit 970. The control unit 970 is connected to
a user interface unit 971. Further, the image data
processing unit 964, the external interface unit 966, the
memory unit 967, the medium drive 968, the OSD unit 969,
the control unit 970, and the like are connected via the
15 bus 972.
[0285]
The optical block 961 includes, e.g., a focus lens
and a diaphragm mechanism. The optical block 961 causes
an optical image of a subject to be formed on an image-
20 capturing surface of the image-capturing unit 962. The
image-capturing unit 962 includes a CCD or a CMOS image
sensor, and generates an electric signal according to an
optical image by photoelectric conversion, and provides
the electric signal to the camera signal processing unit
25 963.
[0286]
The signal processing unit 963 performs various
kinds of camera signal processing such as knee correction,
gamma correction, and color correction on the electric
30 signal provided by the image-capturing unit 962. The
camera signal processing unit 963 provides the image data
processing unit 964 with image data having been subjected
to the camera signal processing.
[0287]
The image data processing unit 964 performs
5 encoding processing on the image data provided from the
camera signal processing unit 963. The image data
processing unit 964 the external interface unit 966 and
the medium drive 968 with the encoded data generated by
performing the encoding processing. The image data
10 processing unit 964 performs decoding processing on the
encoded data provided by the external interface unit 966
and the medium drive 968. The image data processing unit
964 provides the display unit 965 with the image data
generated by performing the decoding processing. The
15 image data processing unit 964 performs processing for
providing the display unit 965 with the image data
provided from the camera signal processing unit 963, and
overlays the display data obtained from the OSD unit 969
on the image data and provides the data to the display
20 unit 965.
[0288]
The OSD unit 969 generates display data such as
menu screen and icons including symbols, characters, or
figures, and outputs the display data to the image data
25 processing unit 964.
[0289]
The external interface unit 966 is constituted by,
for example, a USB input/output terminal, and when image
is printed, it is connected to a printer. The external
30 interface unit 966 is also connected to a drive as
necessary, and removable medium such as a magnetic disk,
an optical disk, or the like is loaded as necessary, and
a computer program read therefrom is installed as
necessary. Further, the external interface unit 966
includes a network interface connected to a predetermined
5 network such as a LAN or the Internet. For example, the
control unit 970 reads the encoded data from the memory
unit 967 in accordance with a command from the user
interface unit 971, and can provide the data to another
device connected via the network from the external
10 interface unit 966. The control unit 970 obtains, via
external interface unit 966, the encoded data and the
image data provided from another device via the network,
and can provided the data to the image data processing
unit 964, for example.
15 [0290]
The recording medium driven the medium drive 968
may be any given removable medium which can be read and
written, such as a magnetic disk, an optical magnetic
disk, an optical disk, or a semiconductor memory. The
20 recording medium may be of any type as a removable medium,
and may be a tape device, or a disk, or a memory card.
It is to be understood that it may be a non-contact IC
card and the like.
[0291]
25 The medium drive 968 and the recording medium may
be integrated, and like, for example, an internal hard
disk drive and an SSD (Solid State Drive), the medium
drive 968 and the recording medium may be configured by a
non-movable storage medium.
30 [0292]
The control unit 970 is constituted by a CPU, a
memory, and the like. The memory stores, e.g., programs
executed by the CPU, various kinds of data required when
processing is performed with the CPU. The programs
stored in the memory is read and executed by the CPU with
5 predetermined timing such as during activation of the
image-capturing device 960. The CPU executes the program,
thus controlling each unit so as to cause the imagecapturing
device 960 to operate in accordance with user
operation.
10 [0293]
In the image-capturing device configured as
described above, the image data processing unit 964 is
provided with a function of an image processing apparatus
(image processing method) of the present application.
15 Accordingly, using the color image of the predetermined
viewpoint and the parallax related image, information
required to generate a color image of a viewpoint other
than the predetermined viewpoint can be transmitted.
Using the color image of the predetermined viewpoint and
20 the parallax related image, a color image of a viewpoint
other than the predetermined viewpoint can be generated.
[0294]
More specifically, the embodiments of the present
technique is not limited to the embodiments explained
25 above, and may be changed in various manners as long as
it is within the gist of the present technique.
[0295]
It should be noted that the present technique may
be configured as follows.
30 [0296]
(1)
An image processing apparatus including an encoding
unit for generating a bit stream by encoding a color
image of a viewpoint and a depth image of the viewpoint;
a generation unit for generating viewpoint generation
5 information used to generate a color image of a display
viewpoint, in accordance with a generation method of the
color image of the display viewpoint obtained by
performing warping processing using the color image and
the depth image; and a transmission unit for transmitting
10 the bit stream generated by the encoding unit and the
viewpoint generation information generated by the
generation unit.
(2)
The image processing apparatus as described in (I),
15 wherein the transmission unit transmits the viewpoint
generation information as an encoding parameter used for
encoding or decoding.
(3
The image processing apparatus as described in (1)
20 or (2), wherein the transmission unit transmits
difference between the viewpoint generation information
of the depth image and the viewpoint generation
information of a depth image located before the depth
image in the order of encoding.
25 (4)
The image processing apparatus as described in (3),
wherein when the slice of the bit stream is intra-slice,
the transmission unit transmits the viewpoint generation
information of the slice, and when the slice is inter-
30 slice, the transmission unit transmits the difference of
the slice.
(5
The image processing apparatus as described in (3)
or (4) further includes a setting unit for setting
difference identification information for identifying
5 presence/absence of the difference, wherein the
transmission unit transmits the difference identification
information which is set by the setting unit.
(6)
The image processing apparatus as described in (5),
10 wherein the transmission unit transmits the difference
identification information set by the setting unit in
such a manner that the difference identification
information is included in the PPS (Picture Parameter
Set) of the bit stream.
15 (7)
The image processing apparatus as described in any
one of (1) to (6), wherein the generation unit generates,
as the viewpoint generation information, information for
identifying the color image or information for
20 identifying the depth image.
( 8
The image processing apparatus as described in any
one of (1) to (7), wherein the depth image is a depthwise
image including a depth value representing a position of
25 a subject in a depth direction at each pixel of the color
image, or a parallax image including a parallax value
representing a distance between each pixel of the color
image and a pixel of a color image at a base point
corresponding to the pixel, and the generation unit
30 generates, as the viewpoint generation information, depth
image identification information indicating whether the
depth image is the depthwise image or the parallax image.
( 9 )
The image processing apparatus as described in ( 8 ) ,
wherein when the depth image is the depthwise image, the
5 generation unit generates, as the viewpoint generation
information, information indicating a minimum value and a
maximum value of world coordinate value at a position in
the depth direction that may occur in the depth image,
and when the depth image is the parallax image, the
10 generation unit generates, as the viewpoint generation
information, information indicating a minimum value and a
maximum value of parallax, in world coordinate, at a
position that may occur in the parallax image, and
information for identifying a color image of the base
15 point.
(10)
The image processing apparatus as described in any
one of (1) to (8), wherein the generation unit generates,
as the viewpoint generation information, a depth minimum
20 value representing the minimum value that would occur as
a pixel value of the depth image, a depth maximum value
representing the maximum value that would occur as a
pixel value of the depth image, and an image-capturing
position distance which is a distance between image-
25 capturing positions of multiple color images
corresponding to the depth image, and the transmission
unit transmits the viewpoint generation information as a
slice header of the bit stream.
(11)
30 An image processing method for an image processing
apparatus, including an encoding step for generating a
bit stream by encoding a color image of a viewpoint and a
depth image of the viewpoint, a generation step for
generating viewpoint generation information used to
generate a color image of a display viewpoint, in
5 accordance with a generation method of the color image of
the display viewpoint obtained by performing warping
processing using the color image and the depth image, and
a transmission step for transmitting the bit stream
generated in the processing of the encoding step and the
10 viewpoint generation information generated in the
processing of the generation step.
(12)
An image processing apparatus including a receiving
unit for receiving a bit stream obtained as a result of
15 encoding of a color image of a viewpoint and a depth
image of the viewpoint and viewpoint generation
information used to generate a color image of a display
viewpoint which is generated in accordance with a
generation method of the color image of the display
20 viewpoint obtained by performing warping processing using
the color image and the depth image, a decoding unit for
generating the color image and the depth image by
decoding the bit stream received by the receiving unit,
and a generation unit for generating the color image of
25 the display viewpoint by performing warping processing
using the color image and the depth image generated by
the decoding unit and the viewpoint generation
information received by the receiving unit.
(13
30 The image processing apparatus as described in (12),
wherein the receiving unit receives the viewpoint
generation information as an encoding parameter used for
encoding or decoding.
(14
The image processing apparatus as described in (12)
5 or (13), wherein the receiving unit receives difference
between the viewpoint generation information of the depth
image and the viewpoint generation information of a depth
image located before the depth image in the order of
encoding, the generation unit restores the viewpoint
10 generation information of the depth image corresponding
to the difference using the difference and the viewpoint
generation information of the depth image located, in the
order of encoding, before the depth image corresponding
to the difference, which are received by the receiving
15 unit, and performs warping processing using the restored
viewpoint generation information, the color image, and
the depth image, thus generating the color image of the
display viewpoint.
(15)
20 The image processing apparatus as described in (14),
wherein when the slice of the bit stream is intra-slice,
the receiving unit receives the viewpoint generation
information of the slice, and when the slice is interslice,
the receiving unit receives the difference of the
25 slice.
(16)
The image processing apparatus as described in (14)
or (15), wherein the receiving unit receives difference
identification information for identifying
30 presence/absence of the difference, and the generation
unit restores the viewpoint generation information on the
basis of the difference identification information
received by the receiving unit.
(17)
The image processing apparatus as described in (16),
5 wherein the receiving unit receives the difference
identification information included in the PPS (Picture
Parameter Set) of the bit stream.
(18)
The image processing apparatus as described in any
10 one of (12) to (17), wherein the receiving unit receives,
as the viewpoint generation information, information for
identifying the color image or information for
identifying the depth image, and the generation unit
identifies the color image and depth image received by
15 the receiving unit on the basis of the viewpoint
generation information, performs warping processing on
the depth image, and generates the color image of the
display viewpoint by performing warping processing on the
color image using the depth image of the display
20 viewpoint having been subjected to the warping processing.
(19)
The image processing apparatus as described in any
one of (12) to (18), wherein the depth image is a
depthwise image including a depth value representing a
25 position of a subject in a depth direction at each pixel
of the color image, or a parallax image including a
parallax value representing a distance between each pixel
of the color image and a pixel of a color image at a base
point corresponding to the pixel, and the receiving unit
30 receives, as the viewpoint generation information,
information indicating whether the depth image is a
depthwise image or a parallax image.
(20)
The image processing apparatus as described in (19),
wherein when the depth image is the depthwise image, the
5 receiving unit receives, as the viewpoint generation
information, information indicating a minimum value and a
maximum value of world coordinate value at a position in
the depth direction that may occur in the depth image,
and when the depth image is the parallax image, the
10 receiving unit receives information indicating a minimum
value and a maximum value of parallax, in world
coordinate, at a position that may occur in the parallax
image, and information for identifying a color image of
the base point.
15 (21)
The image processing apparatus as described in any
one of (12) to (19), wherein the receiving unit receives,
as a slice header of the bit stream, the viewpoint
generation information including a depth minimum value
20 representing the minimum value that would occur as a
pixel value of the depth image, a depth maximum value
representing the maximum value that would occur as a
pixel value of the depth image, and an image-capturing
position distance which is a distance between image-
25 capturing positions of multiple color images
corresponding to the depth image.
(22)
An image processing method for an image processing
apparatus, including a receiving step for receiving a bit
30 stream obtained as a result of encoding of a color image
of a viewpoint and a depth image of the viewpoint and
viewpoint generation information used to generate a color
image of a display viewpoint which is generated in
accordance with a generation method of the color image of
the display viewpoint obtained by performing warping
5 processing using the color image and the depth image, a
decoding step for generating the color image and the
depth image by decoding the bit stream received in the
processing of the receiving step, and a generation step
for generating the color image of the display viewpoint
10 by performing warping processing using the color image
and the depth image generated in the processing of the
decoding step and the viewpoint generation information
received in the processing of the receiving unit.
15 REFERENCE SIGNS LIST
[0297]
10 encoding device, 14 viewpoint generation
information generation unit, 15 multi-viewpoint image
encoding unit, 30 decoding device, 31 multi-viewpoint
20 image decoding unit, 32 viewpoint compositing unit

CLAIMS
1. An image processing apparatus comprising:
an encoding unit for generating a bit stream by
5 encoding a color image of a viewpoint and a depth image
of the viewpoint;
a generation unit for generating viewpoint
generation information used to generate a color image of
a display viewpoint, in accordance with a generation
10 method of the color image of the display viewpoint
obtained by performing warping processing using the color
image and the depth image; and
a transmission unit for transmitting the bit stream
generated by the encoding unit and the viewpoint
15 generation information generated by the generation unit.
2. (Amended) The image processing apparatus
according to claim 1, wherein the transmission unit
transmits the viewpoint generation information as
20 auxiliary information of the bit stream.
3. (Amended) The image processing apparatus
according to claim 1, wherein the transmission unit
transmits the viewpoint generation information as SEI
25 (Supplemental Enhancement Information) of the bit stream.
4. (Amended) The image processing apparatus
according to claim 1, wherein the generation unit
generates, as the viewpoint generation information,
30 information for identifying the color image or
information for identifying the depth image.
5. (Amended) The image processing apparatus
according to claim 1, wherein
the depth image is a parallax image including a
5 parallax value representing a distance between each pixel
of the color image and a pixel of a color image at a base
point corresponding to the pixel, and
the generation unit generates, as the viewpoint
generation information, parallax minimum value
10 information indicating a minimum value of the parallax
value of the parallax image and parallax maximum value
information indicating a maximum value of the parallax
value of the parallax image, and
the transmission unit transmits the viewpoint
15 generation information as SEI (Supplemental Enhancement
Information) .
6. (Amended) The image processing apparatus
according to claim 1, wherein
20 the depth image is a depthwise image including a
depth value representing a position of a subject in a
depth direction at each pixel of the color image,
the generation unit generates, as the viewpoint
generation information, depthwise minimum value
25 information indicating a minimum value of a pixel value
of the depth image and depthwise maximum value
information indicating a maximum value of the pixel value
of the depth image, and
the transmission unit transmits the viewpoint
30 generation information as SEI (Supplemental Enhancement
Information) .
7. (Amended) The image processing apparatus
according to claim 1, wherein
the depth image is a depthwise image including a
5 depth value representing a position of a subject in a
depth direction at each pixel of the color image, or a
parallax image including a parallax value representing a
distance between each pixel of the color image and a
pixel of a color image at a base point corresponding to
10 the pixel, and
the generation unit generates, as the viewpoint
generation information, depth image identification
information indicating whether the depth image is the
depthwise image or the parallax image.
15
8. (Amended) The image processing apparatus
according to claim 7, wherein
when the depth image is the depthwise image, the
generation unit generates, as the viewpoint generation
20 information, depthwise minimum value information
indicating a minimum value of world coordinate value at a
position in the depth direction in the depthwise image
and depthwise maximum value information indicating a
maximum value of world coordinate value at a position in
25 the depth direction in the depthwise image, and
when the depth image is the parallax image, the
generation unit generates, as the viewpoint generation
information, parallax minimum value information
indicating a minimum value of the distance of world
30 coordinate value in the parallax image and parallax
maximum value information indicating a maximum value of
the distance of world coordinate value in the parallax
image.
9. (Amended) The image processing apparatus
5 according to claim 1, wherein
the generation unit generates, as the viewpoint
generation information, depth minimum value information
indicating a minimum value of a pixel value in the.depth
image, depth maximum value information indicating a
10 maximum value of a pixel value in the depth image, and
image-capturing position distance information indicating
a distance between image-capturing positions of color
images corresponding to depth images,
the transmission unit transmits the viewpoint
15 generation information as SEI (Supplemental Enhancement
Information) .
10. (Amended) An image processing method for an image
processing apparatus, comprising:
20 an encoding step for generating a bit stream by
encoding a color image of a viewpoint and a depth image
of the viewpoint;
a generation step for generating viewpoint
generation information used to generate a color image of
25 a display viewpoint, in accordance with a generation
method of the color image of the display viewpoint
obtained by performing warping processing using the color
image and the depth image; and
a transmission step for transmitting the bit stream
30 generated in the processing of the encoding step and the
viewpoint generation information generated in the
processing of the generation step.
11. (Amended) An image processing apparatus
comprising:
a receiving unit for receiving a bit stream
obtained as a result of encoding of a color image of a
viewpoint and a depth image of the viewpoint and
viewpoint generation information used to generate a color
image of a display viewpoint which is generated in
accordance with a generation method of the color image of
the display viewpoint obtained by performing warping
processing using the color image and the depth image;
a decoding unit for generating the color image and
the depth image by decoding the bit stream received by
the receiving unit; and
a generation unit for generating the color image of
the display viewpoint by performing warping processing
using the color image and the depth image generated by
the decoding unit and the viewpoint generation
information received by the receiving unit.
12. (Amended) The image processing apparatus
according to claim 11, wherein the receiving unit
receives the viewpoint generation information as
25 auxiliary information of the bit stream.
13. (Amended) The image processing apparatus
according to claim 11, wherein the receiving unit
receives the viewpoint generation information as SEI
30 (Supplemental Enhancement Information).
14. (Amended) The image processing apparatus
according to claim 11, wherein
the receiving unit receives, as the viewpoint
generation information, information for identifying the
5 color image or information for identifying the depth
image, and
the generation unit identifies the color image and
depth image received by the receiving unit on the basis
of the viewpoint generation information, performs warping
10 processing on the depth image, and generates the color
image of the display viewpoint by performing warping
processing on the color image using the depth image of
the display viewpoint having been subjected to the
warping processing.
15
15. (Amended) The image processing apparatus
according to claim 11, wherein
the depth image is a parallax image including a
parallax value representing a distance between each pixel
20 of the color image and a pixel of a color image at a base
point corresponding to the pixel, and
the viewpoint generation information includes
parallax minimum value information indicating a minimum
value of the parallax value of the parallax image and
25 parallax maximum value information indicating a maximum
value of the parallax value of the parallax image, and
the receiving unit receives the viewpoint
generation information as SEI (Supplemental Enhancement
Information) .
30
16. (Amended) The image processing apparatus
according to claim 1, wherein
the depth image is a depthwise image including a
depth value representing a position of a subject in a
depth direction at each pixel of the color image,
5 the viewpoint generation information includes
depthwise minimum value information indicating a minimum
value of a pixel value of the depth image and depthwise
maximum value information indicating a maximum value of
the pixel value of the depth image, and
10 the receiving unit receives the viewpoint
generation information as SEI (Supplemental Enhancement
Information) .
17. (Amended) The image processing apparatus
15 according to claim 1, wherein
the depth image is a depthwise image including a
depth value representing a position of a subject in a
depth direction at each pixel of the color image, or a
parallax image including a parallax value representing a
20 distance between each pixel of the color image and a
pixel of a color image at a base point corresponding to
the pixel, and
the receiving unit receives, as the viewpoint
generation information, depth image identification
25 information indicating whether the depth image is the
depthwise image or the parallax image.
18. (Amended) The image processing apparatus
according to claim 17, wherein
30 when the depth image is the depthwise image, the
receiving unit receives, as the viewpoint generation
information, depthwise minimum value information
indicating a minimum value of world coordinate value at a
position in the depth direction in the depthwise image
and depthwise maximum value information indicating a
5 maximum value of world coordinate value at a position in
the depth direction in the depthwise image, and
when the depth image is the parallax image, the
receiving unit receives, as the viewpoint generation
information, parallax minimum value information
10 indicating a minimum value of the distance of world
coordinate value in the parallax image and parallax
maximum value information indicating a maximum value of
the distance of world coordinate value in the parallax
image.
15
19. (Amended) The image processing apparatus
according to claim 11, wherein
the receiving unit receives, as SEI (Supplemental
Enhancement Information), the viewpoint generation
20 information including depth minimum value information
indicating a minimum value of a pixel value in the depth
image, depth maximum value information indicating a
maximum value of a pixel value in the depth image, and
image-capturing position distance information indicating
25 a distance between image-capturing positions of color
images corresponding to depth images.
20 (Amended) An image processing method for an image
processing apparatus, comprising:
30 a receiving step for receiving a bit stream
obtained as a result of encoding of a color image of a
viewpoint and a depth image of the viewpoint and
I
viewpoint generation information used to generate a color
image of a display viewpoint which is generated in
accordance with a generation method of the color image of
4
5 the display viewpoint obtained by performing warping
processing using the color image and the depth image;
a decoding step for generating the color image and
the depth image by decoding the bit stream received in
the processing of the receiving step; and
10 a generation step for generating the col'or image of
the display viewpoint by performing warping processing
using the color image and the depth image generated in
the processing of the decoding step and the viewpoint
generation information received in the processing of the
15 receiving step.
Dated this September 1 1,20 13