DESCRIPTION
ENCODING DEVICE, ENCODING METHOD, DECODING DEVICE, AND
DECODING METHOD
5 TECHNICAL FIELD
[0001]
The present technology relates to an encoding
device, an encoding method, a decoding device, and a
decoding method, and more particularly, to an encoding
10 device, an encoding method, a decoding device, and a
decoding method capable of encoding and decoding multiviewpoint
images in accordance with a mode that is
compatible with an existing mode.
15 BACKGROUND ART
[0002]
Currently, as a 3D image viewing mode, generally,
there is a mode (hereinafter, referred to as a twoviewpoint
mode) in which two-viewpoint images alternately
20 displayed are seen by wearing glasses of which a left-eye
shutter is open at the time of displaying one image out
of two-viewpoint images, and a right-eye shutter is open
at the time of displaying the other image.
[0003]
25 However, in such a two-viewpoint mode, a viewer
needs to purchase glasses in addition to a 3D image
display device, and accordingly, the viewer's willingness
to buy reduces. In addition, since a viewer needs to
wear glasses for viewing, it annoys the viewer.
30 Accordingly, a demand for a viewing mode (hereinafter,
referred to as a multi-viewpoint mode) increases in which
1
a 3D image can be viewed without wearing glasses.
[0004]
In the multi-viewpoint mode, multi-viewpoint images
are displayed such that a viewable angle is different for
5 each viewpoint, and, a 3D image can be seen by a viewer
viewing images of arbitrary two viewpoints with left and
right eyes without wearing glasses.
[0005]
A display device that provides viewing of a multi-
10 viewpoint mode, for example, generates multi-viewpoint
images for a multi-viewpoint mode based on images of two
viewpoints for a two-viewpoint mode and displays the
generated multi-viewpoint images. More specifically, the
display device acquires disparity (depth) of two-
15 viewpoint images for a two-viewpoint mode using an image
disparity estimating technology (Depth Estimation). Then,
the display device generates a synthesized image of
multi-viewpoint images adjacent to a viewpoint
corresponding to the images of two viewpoints for a two-
20 viewpoint mode using a multi-viewpoint image generating
technology (View Generation) using the disparity between
images of two viewpoints and a synthesis technology (View
Synthesis) and displays the synthesized image.
[0006]
25 Existing encoding modes include an advanced video
coding (AVC) mode and a multi-view video coding (MVC)
mode.
[0007]
Fig. 1 is a diagram that illustrates an example of
30 an image processing system that encodes a 2D image in
accordance with the AVC mode and decodes the encoded
2
r
image.
[0008]
The image processing system 10 illustrated in Fig.
1 is configured by an imaging unit 11, an Ave encoder 12,
5 and an Ave decoder 13.
[0009]
The imaging unit 11 of the image processing system
10 captures an image A of a predetermined viewpoint and
supplies the captured image to the AVe.encoder 12. The
10 Ave encoder 12 encodes the image A that is supplied from
the imaging unit 11 in accordance with the Ave mode and
supplies the encoded image to the Ave decoder 13. The
Ave decoder 13 decodes the image A after encoding that is
supplied from the Ave encoder 12 in accordance with a
15 mode that corresponds to the Ave mode. The Ave decoder
13 outputs the image A that is acquired as a result of
the decoding process, thereby displaying the image. In
this way, the viewer can see a 2D image.
[0010]
20 Fig. 2 is a diagram that illustrates an example of
an image processing system that encodes an image of two
viewpoints in accordance with the Ave mode and decodes
the image.
[0011]
25 The image processing system 20 illustrated in Fig.
2 is configured by an imaging unit 21A, an imaging unit
21B, an Ave encoder 22, and an Ave decoder 23.
[0012]
The imaging unit 21A captures an image A of a
30 predetermined viewpoint, halves the resolution of the
image, and supplies a resultant image to the Ave encoder
3
22 as a left half image of the screen. In addition, the
imaging unit 21B captures an image B of a viewpoint that
is different from the viewpoint of the image A, halves
the resolution of the image, and supplies a resultant
5 image to the Ave encoder 22 as a right half image of the
screen. The AVe encoder 22 encodes an image of which the
left half is configured by the image A (hereinafter
referred to as a half-resolution image A) of which the
resolution has halved and of which the right h~lfis
10 configured by an image B (hereinafter, referred to as a
half-resolution image B) of which the resolution has
halved in accordance with the Ave mode and supplies the
encoded image to the Ave decoder 23.
[0013]
15 The Ave decoder 23 decodes the images after
encoding that are supplied from the Ave encoder 22 in
accordance with a mode that corresponds to the Ave mode.
The Ave decoder 23 outputs images acquired as a result of
the decoding process and alternately displays the half-
20 resolution images A and B. At this time, a viewer, for
example, wears glasses of which the left-eye shutter is
open at the time of displaying the half-resolution image
A and of which the right-eye shutter is open at the time
of displaying the half-resolution image B, views the
25 half-resolution image A only with the left eye, and views
the half-resolution image B only with the right eye. In
this way, the viewer can see a 3D image.
[0014]
Fig. 3 is a diagram that illustrates an image
30 processing system that encodes an image of two viewpoints
in accordance with the MVe mode and decodes the encoded
4
image.
[0015]
The image processing system 30 illustrated in Fig.
3 is configured by an imaging unit 31A, an imaging unit
5 31B, an MVC encoder 32, and an MVC decoder 33.
[0016]
The imaging unit 31A captures an image A of a
predetermined viewpoint and supplies the captured image
to the MVC encoder 32. In addition, the imaging unit 31B
10 captures an image B of a viewpoint that is different from
the viewpoint of the image A and supplies the captured
image to the MVC encoder 32. The MVC encoder 32 encodes
the image A that is supplied from the imaging unit 31A as
a base image in accordance with the AVC mode and decodes
15 the image B that is supplied from the imaging unit 31B as
a dependent image in accordance with the MVC mode. The
MVC encoder 32 supplies the images A and B after encoding
to the MVC decoder 33.
[0017]
20 The MVC decoder 33 decodes the images A and B after
encoding that are supplied from the MVC encoder 32 in
accordance with a mode that corresponds to the MVC mode.
The MVC decoder 33 outputs the images A and B that are
acquired as a result of the decoding process and
25 alternately displays the images. At this time, a viewer,
for example, wears glasses of which the left-eye shutter
is open at the time of displaying the image A and of
which the right-eye shutter is open at the time of
displaying the image B, views the image A only with the
30 left eye, and views the image B only with the right eye.
In this way, the viewer can see a 3D image.
5
,
[0018]
Fig. 4 is a diagram that illustrates an example of
an image processing system that generates an image of
multi-viewpoints by encoding an image of two viewpoints
5 in accordance with the MVC mode and decoding the image.
[0019]
The image processing system 40 illustrated in Fig.
4 is configured by imaging units 31A and 31B, an MVC
encoder 32, an MVC decoder 33, a disparity detecting unit
10 41, and an image generating unit 42. In Fig. 4, the same
reference numeral is assigned to a configuration that is
the same as that illustrated in Fig. 3, and description
thereof will not be appropriately presented.
[0020]
15 Images A and B that are acquired as a result of
decoding performed by the MVC decoder 33 are supplied to
the disparity detecting unit 41 of the image processing
system 40. The disparity detecting unit 41 detects the
disparity of each pixel of the images A and B using the
20 images A and B. The disparity detecting unit 41
generates a disparity image A' that represents the
disparity of each pixel of the image A and a disparity
image B' that represents the disparity of each pixel of
the image B and supplies the generated disparity images
25 to the image generating unit 42. The image generating
unit 42 generates an image of multi-viewpoints of which
the number corresponding to a display device not
illustrated in the figure by using the images A and B
acquired as a result of decoding performed by the MVC
30 decoder 33 and the disparity images A' and B' supplied
from the image generating unit 42. Then, the image
6
generating unit 42 converts the resolution of the
generated image of each viewpoint into "1/ the number of
viewpoints" of the resolution, synthesizes the images,
and outputs the synthesized image so as to be displayed.
5 [0021]
At this time, the multi-viewpoint images after the
synthesis are displayed such that the viewing angles are
different for each viewpoint, and a viewer can view a 3D
image without wearing glasses by seeing images of
10 arbitrary two viewpoints with his left and right eyes.
[0022]
Meanwhile, a mode for encoding a multi-viewpoint
image has also been devised (for example, see Patent
Document 1).
15
CITATION LIST
PATENT DOCUMENT
[0023]
Patent Document 1: Japanese Patent Application Laid-Open
20 No. 2008-182669
SUMMARY OF THE INVENTION
PROBLEMS TO BE SOLVED BY THE INVENTION
[0024]
25 However, there has been no mode for encoding an
image of multi-viewpoints which has compatibility with an
existing encoding mode for an image of two viewpoints or
less.
[0025]
30 The present technology has been contrived in view
of such situations and enables encoding and decoding of
7
an image of multi-viewpoints in a mode having
compatibility with an existing mode.
SOLUTIONS TO PROBLEMS
5 [0026]
According to a first aspect of the present
technology, there is provided an encoding device
including: a compatible image encoding unit that
generates a first encoded stream by designating a
10 compatible image out of multi-viewpoint images and
encoding the designated compatible image; an image
converting unit that converts resolution of an auxiliary
image used when multi-viewpoint images are generated from
the compatible image; an auxiliary image encoding unit
15 that generates a second encoded stream by encoding the
auxiliary image converted by the image converting unit; a
setting unit that sets compatible image designating
information designating the compatible image; and a
transmission unit that transmits the first encoded stream
20 generated by the compatible image encoding unit, the
second encoded stream generated by the auxiliary image
encoding unit, and the compatible image designating
information set by the setting unit.
[0027]
25 An encoding method according to the first aspect of
the present technology corresponds to the encoding device
according to the first aspect of the present technology.
[0028]
In the first aspect of the present technology, a
30 first encoded stream is generated by designating a
compatible image out of multi-viewpoint images and
8
encoding the designated compatible image, a second
encoded stream is generated by converting resolution of
an auxiliary image used when multi-viewpoint images are
generated from the compatible image and encoding the
5 auxiliary image of which the resolution is converted,
compatible image designating information designating the
compatible image is set, and the first encoded stream,
the second encoded stream, and the compatible image
designating information are transmitted.
10 [0029]
According to a second aspect of the present
technology, there is provided a decoding device
including: a reception unit that receives a first encoded
stream that is acquired as a result of encoding a
15 compatible image designated out of multi-viewpoint images,
a second encoded stream that is acquired as a result of
converting resolution of an auxiliary image used when a
multi-viewpoint image is generated from the compatible
image and encoding the auxiliary image, and compatible
20 image designating information that designates the
compatible image; a compatible image decoding unit that
identifies the first encoded stream based on the
compatible image designating information and decodes the
first encoded stream; an auxiliary image decoding unit
25 that decodes the second encoded stream; and an image
converting unit that converts the resolution of the
auxiliary image, of which the resolution is converted,
that is acquired as a result of decoding by the auxiliary
image decoding unit.
30 [0030]
A decoding method according to the second aspect of
9
the present technology corresponds to the decoding device
according to the second aspect of the present technology.
[0031]
In the second aspect of the present technology, a
5 first encoded stream that is acquired as a result of
encoding a compatible image designated out of multiviewpoint
images, a second encoded stream that is
acquired as a result of converting resolution of an
auxiliary image used when a multi-viewpoint image is
10 generated from the compatible image and encoding the
auxiliary image, and compatible image designating
information that designates the compatible image are
received, the first encoded stream is identified based on
the compatible image designating information, the first
15 encoded stream is decoded, the second encoded stream is
decoded, and the resolution of the auxiliary image, of
which the resolution is converted, that is acquired as a
result of decoding is converted.
[0032]
20 In addition, the encoding device according to the
first aspect and the decoding device according to the
second aspect may be realized by causing a computer to
execute a program.
[0033]
25 Furthermore, the program that is executed by a
computer so as to realize the encoding device according
to the first aspect and the decoding device according to
the second aspect may be provided while being transmitted
through a transmission medium or being recorded on a
30 recording medium.
10
•
EFFECTS OF THE INVENTION
[0034]
According to the first aspect of the present
technology, multi-viewpoint images can be encoded in a
5 mode having compatibility with an existing mode.
[0035]
In addition, according to the second aspect of the
present technology, multi-viewpoint images can be decoded
in accordance with a mode that has compatibility with an
10 existing mode.
BRIEF DESCRIPTION OF DRAWINGS
[0036]
Fig. 1 is a diagram that illustrates an example of
15 an image processing system that encodes a 2D image in
accordance with an AVC mode and decodes the encoded image.
Fig. 2 is a diagram that illustrates an example of
an image processing system that encodes an image of two
viewpoints in accordance with an AVC mode and decodes the
20 encoded image.
Fig. 3 is a diagram that illustrates an example of
an image processing system that encodes an image of two
viewpoints in accordance with an MVC mode and decodes the
encoded image.
25 Fig. 4 is a diagram that illustrates an example of
an image processing system that generates an image of
multi-viewpoints by encoding an image of two viewpoints
in accordance with the MVC mode and decoding the encoded
image.
30 Fig. 5 is a block diagram that illustrates a
configuration example of an encoding device according to
11
•
a first embodiment of the present technology.
Fig. 6 is a diagram that illustrates an example of
the description of compatibility information and
disparity image information.
5 Fig. 7 is a diagram that illustrates a detailed
example of the description of the compatibility
information.
Fig. 8 is a diagram that illustrates a detailed
example of the description of the disparity image
10 information.
Fig. 9 is a diagram that illustrates an example of
the description of inter-viewpoint distance information.
Fig. 10 is a flowchart that illustrates an encoding
process performed by the encoding device illustrated in
15 Fig. 5.
Fig. 11 is a flowchart that illustrates an encoding
process performed by the encoding device illustrated in
Fig. 5.
Fig. 12 is a diagram that illustrates a
20 configuration example of a decoding device that
corresponds to the encoding device illustrated in Fig. 5.
Fig. 13 is a flowchart that illustrates a decoding
process performed by the decoding device illustrated in
Fig. 12.
25 Fig. 14 is a block diagram that illustrates a
configuration example of an encoding device according to
a second embodiment of the present technology.
Fig. 15 is a flowchart that illustrates an encoding
process performed by the encoding device illustrated in
30 Fig. 14.
Fig. 16 is a flowchart that illustrates an encoding
12
•
process performed by the encoding device illustrated in
Fig. 14.
Fig. 17 is a diagram that illustrates a
configuration example of a decoding device that
5 corresponds to the encoding device illustrated in Fig. 14.
Fig. 18 is a flowchart that illustrates a decoding
process performed by the decoding device illustrated in
Fig. 17.
Fig. 19 is a block diagram that illustrates a
10 configuration example of an encoding device according to
a third embodiment of the present technology.
Fig. 20 is a flowchart that illustrates an encoding
process performed by the encoding device illustrated in
Fig. 19.
15 Fig. 21 is a flowchart that illustrates the
encoding process performed by the encoding device
illustrated in Fig. 19.
Fig. 22 is a diagram that illustrates a
configuration example of a decoding device that
20 corresponds to the encoding device illustrated in Fig. 19.
Fig. 23 is a flowchart that illustrates a decoding
process performed by the decoding device illustrated in
Fig. 22.
Fig. 24 is a diagram that illustrates another
25 example of the description of compatibility information
and disparity image information.
Fig. 25 is a diagram that illustrates another
detailed example of the description of the compatibility
information.
30 Fig. 26 is a diagram that illustrates another
detailed example of the description of the disparity
13
•
image information.
Fig. 27 is a diagram that illustrates another
example of the description of the inter-viewpoint
distance information.
5 Fig. 28 is a block diagram that illustrates a
configuration example of an encoding device according to
a fourth embodiment of the present technology.
Fig. 29 is a diagram that illustrates a further
another ~xample of the description of the disparity image
10 information.
Fig. 30 is a flowchart that illustrates an encoding
process performed by the encoding device illustrated in
Fig. 28.
Fig. 31 is a flowchart that illustrates the
15 encoding process performed by the encoding device
illustrated in Fig. 28.
Fig. 32 is a diagram that illustrates a
configuration example of a decoding device that
corresponds to the encoding device illustrated in Fig. 28.
20 Fig. 33 is a flowchart that illustrates a decoding
process performed by the decoding device illustrated in
Fig. 32.
Fig. 34 is a diagram that illustrates further
another example of the description of the disparity image
25 information.
Fig. 35 is a block diagram that illustrates a
configuration example of an encoding device according to
a fifth embodiment of the present technology.
Fig. 36 is a diagram that illustrates a detailed
30 example of the description of the disparity image
information.
14
•
Fig. 37 is a diagram that illustrates a detailed
example of the description of the disparity image
information.
Fig. 38 is a flowchart that illustrates an encoding
5 process performed by the encoding device illustrated in
Fig. 35.
Fig. 39 is a flowchart that illustrates the
encoding process performed by-the encoding device
illustrated in Fig. 35.
10 Fig. 40 is a diagram that illustrates a
configuration example of a decoding device that
corresponds to the encoding device illustrated in Fig. 35.
Fig. 41 is a flowchart that illustrates a decoding
process performed by the decoding device illustrated in
15 Fig. 40.
Fig. 42 is a block diagram that illustrates a
configuration example of an encoding device according to
a sixth embodiment of the present technology.
Fig. 43 is a flowchart that illustrates an encoding
20 process performed by the encoding device illustrated in
Fig. 42.
Fig. 44 is a flowchart that illustrates the
encoding process performed by the encoding device
illustrated in Fig. 42.
25 Fig. 45 is a diagram that illustrates a
configuration example of a decoding device that
corresponds to the encoding device illustrated in Fig. 42.
Fig. 46 is a flowchart that illustrates a decoding
process performed by the decoding device illustrated in
30 Fig. 45.
Fig. 47 is a diagram that illustrates examples of a
15
•
multiplexed pattern of an encoding target.
Fig. 48 is a diagram that illustrates features of
the effect according to multiplexing.
Fig. 49 is a diagram that illustrates other
5 examples of the multiplexed pattern of the encoding
target.
Fig. 50 is a diagram that illustrates further other
examples of the multiplexed pattern of the encoding
target.
10 Fig. 51 is a block diagram that illustrates a
configuration example of an encoding device according to
a seventh embodiment of the present technology.
Fig. 52 is a flowchart that illustrates an encoding
process performed by the encoding device illustrated in
15 Fig. 51.
Fig. 53 is a flowchart that illustrates the
encoding process performed by the encoding device
illustrated in Fig. 51.
Fig. 54 is a flowchart that illustrates an encoding
20 process performed by the encoding device illustrated in
Fig. 51.
Fig. 55 is a flowchart that illustrates an encoding
process performed by the encoding device illustrated in
Fig. 54.
25 Fig. 56 is a diagram that illustrates a
configuration example of a bit stream.
Fig. 57 is a diagram that illustrates an example of
the description of a Subset SPS according to the 3DV mode
for a disparity image illustrated in Fig. 56.
30 Fig. 58 is a diagram that illustrates an example of
the description of extension information for a disparity
16
•
image that is illustrated in Fig. 57.
Fig. 59 is a diagram that illustrates an example of
the description of VUI extension information for a
disparity image illustrated in Fig. 57.
5 Fig. 60 is a diagram that illustrates an example of
the description of an NAL header of encoded data
according to the 3DV mode.
Fig. 61 is a diagram that illustrates an example of
the description of disparity image header extension
10 information illustrated in Fig. 60.
Fig. 62 is a diagram that illustrates an example of
the description of SEI according to the 3DV mode that is
illustrated in Fig. 56.
Fig. 63 is a diagram that illustrates another
15 configuration example of the bit stream.
Fig. 64 is a diagram that illustrates an example of
the description of a Subset SPS according to the 3DV mode
for an image that is illustrated in Fig. 63.
Fig. 65 is a diagram that illustrates a
20 configuration example of a bit stream in a case where the
3DV mode is a mode that is compliant with an HEVC mode.
Fig. 66 is a diagram that illustrates an example of
the description of an SPS illustrated in Fig. 65.
Fig. 67 is a diagram that illustrates an example of
25 the description of the Subset SPS illustrated in Fig. 66.
Fig. 68 is a diagram that illustrates an example of
the description of VUI information of the Subset SPS
illustrated in Fig. 67.
Fig. 69 is a diagram that illustrates an example of
30 the description of SEI according to the 3DV mode that is
illustrated in Fig. 65.
17
•
Fig. 70 is a diagram that illustrates a
configuration example of a computer according to an
embodiment.
Fig. 71 is a diagram that illustrates a schematic
5 configuration of a television apparatus according to the
present technology.
Fig. 72 is a diagram that illustrates a schematic
configuration of a cellular phone according to the
present technology.
10 Fig. 73 is a diagram that illustrates a schematic
configuration of a recording and reproducing device
according to the present technology.
Fig. 74 is a diagram that illustrates a schematic
configuration of an imaging apparatus according to the
15 present technology.
MODE FOR CARRYING OUT THE INVENTION
[0037]

20 [Configuration Example of Encoding Device According to
First Embodiment]
Fig. 5 is a block diagram that illustrates a
configuration example of an encoding device according to
a first embodiment of the present technology.
25 [0038]
The encoding device 50 illustrated in Fig. 5 is
configured by imaging units 51A to 51C, an image
converting unit 52, a disparity image generating unit 53,
an image information generating unit 54, a compatibility
30 information generating unit 55, an inter-viewpoint
distance information generating unit 56, a disparity
18
•
image information generating unit 57, an encoder 58, and
a multiplexing unit 59.
[0039]
The encoding device 50 encodes an image of one
5 viewpoint that is included in a multi-viewpoint image as
a 2D image in accordance with an existing encoding mode,
thereby securing the compatibility with an existing
encoding device that encodes a 2D image.
[0040]
10 Hereinafter, out of multi-viewpoint images, an
image that is encoded in an existing encoding mode so as
to secure the compatibility with an existing encoding
device will be referred to as a compatible image, and an
image that is used for generating images of viewpoints
15 more than the number of viewpoints of a compatible image
using the compatible image will be referred to as an
auxiliary image.
[0041]
In the encoding device 50, the imaging unit 51A
20 captures an HD (high definition) image of a predetermined
viewpoint as an image A1 and supplies the captured image
to the image converting unit 52, the disparity image
generating unit 53, and the inter-viewpoint distance
information generating unit 56. The imaging unit 51B
25 captures an HD image of a viewpoint that is different
from the viewpoint of the image A1 as an image B1 at a
position that is separate from the imaging unit 51A by a
distance ~d1~ in a horizontal direction and supplies the
captured image to the image converting unit 52, the
30 disparity image generating unit 53, and the interviewpoint
distance information generating unit 56. The
19
•
imaging unit 51C captures an HD image of a viewpoint
differ.ent from the viewpoints of the images A1 and B1 as
an image C1 at a position separate from the imaging unit
51A by a distance dd1Ac in the horizontal direction that
5 is opposite to the imaging unit 51B and supplies the
captured image to the image converting unit 52, the
disparity image generating unit 53, and the interviewpoint
distance information generating unit 56.
[0042]
10 The viewpoints that correspond to the images B1 and
C1 are viewpoints that are located on the outer side out
of viewpoints of images that can be recognized as 3D
images. In this way, a decoding device that corresponds
to the encoding device 50 can generate multi-viewpoint
15 images by interpolating images of viewpoints located on
the further inner side than the viewpoints of the images
B1 and C1 by using the images A1 to C1. As a result, the
multi-viewpoint images can be generated with precision
that is higher than that of a case where images of
20 viewpoints located on the outer side are interpolated
using images of viewpoints located on the inner side.
The distances dd1~ and dd1Ac may be configured to be
either fixed or changed each time.
[0043]
25 The image converting unit 52 determines the image
A1, which is supplied from the imaging unit 51A of which
the position in the horizontal direction is located on
the inner side out of the imaging units 51A to 51C, as a
compatible image. The image converting unit 52 supplies
30 information that designates the image A1 as a compatible
image to the compatibility information generating unit 55.
20
•
Then, the image converting unit 52 directly supplies the
image A1 that is the compatible image to the encoder 58.
[0044]
In addition, the image converting unit 52 sets the
5 images B1 and C1 other than the image A1 as auxiliary
images and multiplexes the images B1 and C1 in accordance
with a predetermined multiplexing mode. More
specifically, for example, in a case where the
multiplexing mode is a side-by-side mode, the image
10 converting unit 52 (image converting unit) halves the
resolution of each one of the images B1 and C1. Then,
the image converting unit 52 (multiplexing unit)
multiplexes the image B1 (hereinafter, referred to as a
half-resolution image B1) of which the resolution has
15 halved and the image C1 (hereinafter, referred to as a
half-resolution image C1) of which the resolution has
halved such that the half-resolution image B1 becomes a
left-half image of the screen, and the half-resolution
image C1 becomes a right-half image of the screen. The
20 image converting unit 52 supplies a multiplexed image
that is acquired as a result of the multiplexing process
to the encoder 58 and supplies information that
represents a multiplexing mode of the auxiliary images to
the image information generating unit 54.
25 [0045]
The disparity image generating unit 53 detects the
disparity of each pixel of the images A1 to C1 using the
images A1 to C1 that are supplied from the imaging units
51A to 51C. The disparity image generating unit 53
30 generates a disparity image A1' that represents the
disparity of each pixel of the image A1 that is a
21
•
compatible image and directly supplies the generated
disparity image to the encoder 58. In addition, the
disparity image generating unit 53 (multiplexing unit)
generates a disparity image Bl' that represents the
5 disparity of each pixel of the image Bl that is an
auxiliary image and a disparity image Cl' that represents
the disparity of each pixel of the image Cl that is an
auxiliary image and multiplexes the generated disparity
images in accordance with a predetermined multiplexing
10 mode. The disparity image generating unit 53 supplies a
multiplexed image that is acquired as a result of the
multiplexing process to the encoder 58. In addition, the
disparity image generating unit 53 supplies information
that represents the multiplexing mode of the disparity
15 images of the auxiliary images to the disparity image
information generating unit 57.
[0046]
The image information generating unit 54 generates
information that represents the multiplexing mode of the
20 auxiliary images and the like as image information, which
is information relating to a compatible image and
auxiliary images, based on the information that is
supplied form the image converting unit 52 and supplies
the generated image information to the encoder 58.
25 [0047]
The compatibility information generating unit 55
(setting unit) generates information designating the
compatible image, a compatible mode, and the like as
compatibility information, which is information relating
30 to the compatibility, based on the information supplied
from the image converting unit 52 and supplies the
22
•
generated compatibility information to the encoder 58.
[0048]
Here, the compatible mode is a mode that represents
a method of encoding the compatible image. As examples
5 of the compatible mode, there are a mono mode that
represents an encoding method in which a compatible image
of one viewpoint is encoded in accordance with the AVC
mode, a frame packing mode th~t represents an encoding
·method in which compatible images of two viewpoints are
10 multiplexed and encoded in accordance with the AVC mode,
and a stereo mode that represents an encoding method in
which compatible images of two viewpoints are encoded in
accordance with the MVC mode.
[0049]
15 The inter-viewpoint distance information generating
unit 56 (setting unit) detects each inter-viewpoint
distance (hereinafter, referred to as an inter-viewpoint
distance) between two images out of the images A1 to C1
using the images A1 to C1 that are supplied from the
20 imaging units 51A to 51C. For example, the interviewpoint
distance information generating unit 56 detects
the distance ~d1~ between the imaging units 51A and 51B
in the horizontal direction and the distance ~d1Ac
between the imaging units 51A and 51C in the horizontal
25 direction as inter-viewpoint distances. The interviewpoint
distance information generating unit 56
generates information that represents the inter-viewpoint
distances and the like as inter-viewpoint distance
information, which is information relating to the inter-
30 viewpoint distance information, and supplies the
generated inter-viewpoint distance information to the
23
•
encoder 58.
[0050]
The disparity image information generating unit 57
generates information that represents the multiplexing
5 mode of the disparity images of the auxiliary images and
the like as disparity image information, which is
information relating to the disparity images, based on
the information that is supplied from the disparity image
generating unit 53 and supplies the generated disparity
10 image information to the encoder 58.
[0051]
The encoder 58 is configured by a compatible
encoder 61 and an auxiliary encoder 62. The compatible
encoder 61 (compatible image encoding unit) designates
15 the image A1 that is a compatible image from the
multiplexed image of the compatible image and the
auxiliary images supplied from the image converting unit
52, encodes the image A1 in accordance with the existing
Ave mode, adds various information to the encoded image,
20 and supplies an encoded stream that is acquired as a
result thereof to the multiplexing unit 59 as a
compatible stream (first encoded stream).
[0052]
The auxiliary encoder 62 (an auxiliary image
25 encoding unit and a disparity image encoding unit)
encodes the multiplexed image of the auxiliary images
that is supplied from the image converting unit 52 and
the multiplexed images of the disparity images A1' of the
compatible images and the disparity images of the
30 auxiliary images that are supplied from the disparity
image generating unit 53 in accordance with a
24
•
predetermined mode. As an encoding mode employed by the
auxiliary encoder 62, the AVC mode, the MVC mode, an
MPEG2 (Moving Picture Experts Group phase 2), or the like
can be used.
5 [0053]
In addition, the auxiliary encoder 62 adds the
image information supplied from the image information
generating unit 54, the compatibility information
supplied from compatibility information generating unit
10 55, the inter-viewpoint distance information supplied
from the inter-viewpoint distance information generating
unit 56, the disparity image information supplied from
the disparity image information generating unit 57, and
the like to an encoded image (the second stream, the
15 first disparity encoded stream, and the second disparity
encoded stream) that is acquired as a result of the
encoding process, thereby generating an encoded stream.
The auxiliary encoder 62 supplies the encoded stream to
the multiplexing unit 59 as an auxiliary stream.
20 [0054]
The multiplexing unit 59 respectively generates TSs
(transport streams) from the compatible stream supplied
from the compatible encoder 61 and the auxiliary stream
supplied from the auxiliary encoder 62, and multiplexes
25 the generated TSs. The multiplexing unit 59
(transmission unit) transmits a multiplexed stream that
is acquired as a result of the multiplexing process.
[0055]
[Example of Description of Information Included in
30 Auxiliary Stream]
Figs. 6 to 8 are diagrams that illustrate examples
25
•
of the description of the compatibility information and
the disparity image information that are included in the
auxiliary stream in a case where the AVC mode or the MVC
mode is used as an encoding mode.
5 [0056]
As illustrated in Fig. 6, the compatibility
information (3DV_view_structure) and the disparity image
information (depth~ap_structure) are arranged in the 8EI
(supplemental enhancement information) that is included
10 in the auxiliary stream.
[0057]
As the compatibility information
(3DV_view_structure), as illustrated in Fig. 7, the
number (num_of_views) of viewpoints (views) of an image
15 that is an encoding target, a compatible mode,
information (compatible_view) that designates a
compatible image, and the like are described.
[0058]
In addition, as the disparity image information
20 (depth_map_structure), as illustrated in Fig. 8, the
number (num_of_depth_map) of disparity images (depth map),
a flag (is_frame_packing) representing whether or not the
disparity images are multiplexed, a multiplexing system
(frame_packing_mode) of the disparity images, information
25 (comb_frame_packing_views) that designates the disparity
images that are multiplexed, and the like are described.
As the multiplexing modes, there are a side-by-side mode
(8B8), a top-and-bottom mode (TOB), and the like. Here,
the top-and-bottom mode is a mode in which one image out
30 of two images to be multiplexed is set as an upper image
of the screen, and the other image is set as a lower
26 I
•
image of the screen.
[0059]
The image information is similar to the disparity
image information except that the image information is
5 information of the compatible image and the auxiliary
image other than the disparity image and thus is not
illustrated in the figure.
[0060]
Fig. 9 is a diagram that illustrates an example of
10 the description of the inter-viewpoint distance
information that is included in the auxiliary stream in a
case where the AVC mode or the MVC mode is used as an
encoding mode.
[0061]
15 As illustrated in Fig. 9, the inter-viewpoint
distance information (3DV_view_info), similarly to the
compatibility information, the disparity image
information, and the image information, is arranged in
the SEI that is included in the auxiliary stream.
20 [0062]
As the inter-viewpoint distance information
(3DV_view_info), as illustrated in Fig. 9, the number
(num_of_views) of viewpoints of an image that is an
encoding target, the number (num_of_view_distance) of
25 detected inter-viewpoint distances, a combination
(view[O,I]) of two images corresponding to the interviewpoint
distance, an inter-viewpoint distance
(view_distance[view_distance_id]) that corresponds to the
combination, and the like are described.
30 [0063]
[Description of Process of Encoding Device]
27
•
Figs. 10 and 11 are flowcharts that illustrate an
encoding process that is performed by the encoding device
50 illustrated in Fig. 5. This encoding process, for
example, is started when the images A1 to C1 are output
5 from the imaging units 51A to 51C.
[0064]
In Step Sll illustrated in Fig. 10, the interviewpoint
distance information generating unit 56 detects
distances ~d1~ and ~d1Ac as inter-viewpoint distances
10 using the images A1 to C1 that are supplied from the
imaging units 51A to 51C.
[0065]
In Step S12, the inter-viewpoint distance
information generating unit 56 generates information that
15 represents the inter-viewpoint distances detected in Step
Sll and the like as inter-viewpoint distance information
and inputs the generated inter-viewpoint distance
information to the encoder 58.
[0066]
20 In Step S13, the image converting unit 52
determines the image A1 that is supplied from the imaging
unit 51A, of which the position in the horizontal
direction is located on the inner side out of the imaging
units 51A to 51C, as a compatible image and determines a
25 multiplexing mode of auxiliary images. The image
converting unit 52 supplies information that designates
the image A1 as a compatible image to the compatibility
information generating unit 55 and supplies the
multiplexing mode of the auxiliary images to the image
30 information generating unit 54.
[0067]
28
•
In Step 814, the compatibility information
generating unit 55 generates the information designating
the image A1 as a compatible image, a mono mode as a
compatible mode, and the like as compatible image based
5 on the information that is supplied from the image
converting unit 52 and inputs the generated compatibility
information to the encoder 58.
[0068]
In Step 815, the image information generating unit
10 54 generates information that represents the multiplexing
mode of auxiliary images and the like as image
information based on the information that is supplied
from the image converting unit 52 and inputs the
generated image information to the encoder 58.
15 [0069]
In Step S16, the image converting unit 52 sets
images B1 and C1 other than the image A1 as auxiliary
images and multiplexes the auxiliary images based on the
multiplexing mode of auxiliary images that is determined
20 in Step 813, thereby acquiring a multiplexed image of the
auxiliary images.
[0070]
In Step S17, the image converting unit 52 inputs
the multiplexed image of the image A1, which is a
25 compatible image, and the auxiliary image to the encoder
58.
[0071]
In Step S18 illustrated in Fig. 11, the disparity
image generating unit 53 detects the disparity of each
30 pixel of the images A1 to C1 using the images A1 to C1
that are supplied from the imaging units 51A to 51C and
29
•
generates disparity images A1' to e1' .
[0072]
In Step S19, the disparity image generating unit 53
determines a multiplexing mode of the disparity images of
5 auxiliary images and supplies information that represents
the multiplexing mode to the disparity image information
generating unit 57.
[0073]
In Step S20, the disparity image information
10 generating unit 57 generates information that represents
the multiplexing mode of the disparity images of
auxiliary images and the like as disparity image
information based on the information that is supplied
from the disparity image generating unit 53 and inputs
15. the generated disparity image information to the encoder
58.
[0074]
In Step S21, the disparity image generating unit 53
multiplexes the disparity images of the auxiliary images
20 based on the multiplexing mode of the disparity images of
the auxiliary images that is determined in Step S19,
thereby acquiring a multiplexed image of the disparity
images of the auxiliary images.
[0075]
25 In Step S22, the disparity image generating unit 53
inputs the disparity image A1' of the compatible image
and the multiplexed image of the disparity images of the
auxiliary images to the encoder 58.
[0076]
30 In Step S23, the compatible encoder 61 of the
encoder 58 encodes the image A1 that is a compatible
30
•
image supplied from the image converting unit 52 in
accordance with the existing Ave mode and supplies an
encoded stream acquired as a result thereof to the
multiplexing unit 59 as a compatible stream.
5 [0077]
In Step S24, the auxiliary encoder 62 encodes the
multiplexed image of the auxiliary images that is
supplied from the image converting unit 52, the disparity
image A1' of the compatible image that is supplied from
10 the disparity image generating unit 53, and the
multiplexed image of the disparity images of the
auxiliary images in accordance with a predetermined mode.
[0078]
In Step S25, the auxiliary encoder 62 adds the
15 image information supplied from the image information
generating unit 54, the compatibility information
supplied from the compatibility information generating
unit 55, the inter-viewpoint distance information
supplied from the inter-viewpoint distance information
20 generating unit 56, the disparity image information
supplied from the disparity image information generating
unit 57, and the like to the encoded image that is
acquired as a result of the encoding process by a process
of Step S24, thereby generating an encoded stream. The
25 auxiliary encoder 62 supplies the encoded stream to the
multiplexing unit 59 as an auxiliary stream.
[0079]
In Step S26, the multiplexing unit 59 generates TSs,
respectively, from the compatible stream that is supplied
30 from the compatible encoder 61 and the auxiliary stream
that is supplied from the auxiliary encoder 62,
31
•
multiplexes the generated TSs, and transmits a resultant
stream. Then, the process ends.
[0080]
As above, since the encoding device 50 performs
5 encoding in accordance with an existing encoding mode
with an image of one viewpoint that is included in the
multi-viewpoint image set as a compatible image, the
compatibility with an existing encoding device that
encodes a 2D image can be secured. In other words, the
10 encoding device 50 can perform encoding for multiviewpoint
images in accordance with a mode that has
compatibility with an existing encoding mode.
[0081]
In addition, since the encoding device 50 encodes
15 the images A1 to C1 of three viewpoints, a decoding
device that corresponds to the encoding device 50 can
generate multi-viewpoint images from the images A1 to C1
of three viewpoints. In this way, according to the
decoding device, compared to a case where multi-viewpoint
20 images are generated from images of two viewpoints, the
viewpoints of images that can be generated are not
limited, and multi-viewpoint images having relatively
high precision can be generated.
[0082]
25 Furthermore, since the encoding device 50 encodes
images with the resolution of the auxiliary images being
lowered, the processing cost of the encoding process and
the decoding process can be lower than that of a case
where encoding is performed without lowering the
30 resolution. For example, in a case where auxiliary
images of two viewpoints are encoded without lowering the
32
•
resolution thereof, while the processing cost of the
encoding process and the decoding process is the same as
that of an encoding process and a decoding process for
two HD images, the processing cost of the encoding
5 process for auxiliary images of two viewpoints, which is
performed by the encoding device 50, is the same as the
processing cost of the encoding process or the decoding
process for one HD image. As a result, it can be
prevented that the performance of the decoding process
10 performed by the decoding device greatly affects the
image quality of multi-viewpoint images.
[0083]
Furthermore, when multi-viewpoint images are
synthesized, as will be described later, a decoding
15 device that corresponds to the encoding device 50 lowers
the resolution at the rate of the reciprocal of the
number of viewpoints of the multi-viewpoint images,
whereby the lowering of the resolution of the auxiliary
images that is performed by the encoding device 50 does
20 not affect the image quality of the multi-viewpoint
images after the synthesis~
[0084]
In addition, since the encoding device 50 performs
encoding with the resolution of auxiliary images being
25 halved apd encodes the disparity images of the auxiliary
images while being halved, the amount of information of
an encoding target can be configured to be approximately
an amount that corresponds to 4 HD images (1080i x 4) in
the Ave mode.
30 [0085]
Here, in comprehensive view of the processing speed,
33
•
the processing performance with respect to power
consumption, a transmission data rate, a transmission
bandwidth, a bandwidth of a memory, a memory access speed,
and the like of a current display device that displays
5 multi-viewpoint images and a decoding device according
thereto, a value that is adequate as the amount of
information that can be processed by the decoding device
is considered to be about two times an amount that
corresponds to an HD image in the current state of the
10 MVC mode, that is, an amount that corresponds to 4 HD
images in the AVC mode. Accordingly, a decoding device
(decoding method) that corresponds to the encoding device
50 of which the amount of information of the encoding
target is about an amount that corresponds to 4 HD images
15 (1080i x 4) in the Ave mode can be realized at a
reasonable processing cost by a reasonable approach.
[0086]
In addition, according to the encoding device 50,
the amount of information of the encoding target is
20 reduced to an amount that corresponds to about 4 HD
images (1080i x 4) in the Ave mode, and accordingly, the
encoding device can be easily operated as a BD or a
broadcast application that has a limitation of a usable
bandwidth.
25 [0087]
Furthermore, since the encoding device 50 generates
disparity images and transmits the disparity images while
being included in an encoded stream, a decoding device
that corresponds to the encoding device 50 does not need
30 to generate a disparity image for generating multiviewpoint
images, and accordingly, the processing load of
34
•
the decoding device can be reduced. As a result, the
cost of the decoding device can be reduced. In addition,
it can be prevented that the disparity detecting
performance of the decoding device greatly affects the
5 image quality of multi-viewpoint images.
[0088]
In view of the situations of the display device and
the decoding device as described above, a main object of
the present technology is, by using various kinds of
10 information such as the compatibility information on the
premise of restrictions in consideration of a valid
processing cost and a reasonable realization technique,
to realize the generation of a multi-viewpoint image
having an improved image quality with high efficiency
15 while the compatibility with an existing format is
secured.
[0089]
[Configuration Example of Decoding Device]
Fig. 12 is a diagram that illustrates a
20 configuration example of a decoding device that decodes a
multiplexed stream that is transmitted from the encoding
device 50 illustrated in Fig. 5.
[0090]
The decoding device 120 illustrated in Fig. 12 is
25 configured by a separation unit 121, a decoder 122, an
image information acquiring unit 123, an inter-viewpoint
distance information acquiring unit 124, a disparity
image information acquiring unit 125, a compatibility
information acquiring unit 126, and an image generating
30 unit 127. The decoding device 120 decodes a multiplexed
stream that is transmitted from the encoding device 50,
35
•
generates an image of one viewpoint or images of
viewpoints, and displays the generated image on a display
device that is not illustrated in the figure.
[0091]
5 More specifically, the separation unit 121
(reception unit) of the decoding device 120 receives a
multiplexed stream that is transmitted from the encoding
device 50 and separates each TS. The separation unit 121
extracts a compatible stream and an auxiliary stream from
10 the separated TS and supplies the extracted streams to
the decoder 122.
[0092]
The decoder 122 is configured by a compatible
decoder 131 and an auxiliary decoder 132. The compatible
15 decoder 131 (compatible image decoding unit) of the
decoder 122 identifies the compatible stream out of the
compatible stream and the auxiliary stream supplied from
the separation unit 121 based on the information used for
specifying the compatible stream that is supplied from
20 the auxiliary decoder 132. The compatible decoder 131
decodes the encoded compatible image included in the
compatible stream in accordance with a mode that
corresponds to the Ave mode and supplies an image A1 that
is acquired as a result thereof to the image generating
25 unit 127.
[0093]
The auxiliary decoder 132 supplies the information
used for specifying the compatible stream to the
compatible decoder 131 based on the compatibility
30 information that is included in the auxiliary stream
supplied from the separation unit 121. The auxiliary
36
•
decoder 132 identifies the auxiliary stream out of the
compatible stream and the auxiliary stream that are
supplied from the separation unit 121 based on the
compatibility information. The auxiliary decoder 132
5 (auxiliary image decoding unit) decodes the multiplexed
image of auxiliary images, the disparity image AI' of the
compatible image, and the multiplexed image of disparity
images of the auxiliary images, which are encoded,
included in the auxiliary stream that is supplied from
10 the separation unit 121 in accordance with a mode that
corresponds to the auxiliary encoder 62 illustrated in
Fig. 5.
[0094]
The auxiliary decoder 132 supplies the multiplexed
15 image of the auxiliary images, the disparity image A1' of
the compatible image, and the multiplexed image of the
disparity images of the auxiliary images, which are
acquired as a result of the decoding process, to the
image generating unit 127. In addition, the auxiliary
20 decoder 132 supplies the image information included in
the auxiliary stream to the image information acquiring
unit 123 and supplies the inter-viewpoint distance
information to the inter-viewpoint distance information
acquiring unit 124. Furthermore, the auxiliary decoder
25 132 supplies the disparity image information included in
the auxiliary stream to the disparity image information
acquiring unit 125 and supplies the compatibility
information to the compatibility information acquiring
unit 126.
30 [0095]
The image information acquiring unit 123 acquires
37
•
the image information that is supplied from the auxiliary
decoder 132 and supplies the acquired image information
to the image generating unit 127. The inter-viewpoint
distance information acquiring unit 124 acquires the
5 inter-viewpoint distance information that is supplied
from the auxiliary decoder 132 and supplies the acquired
inter-viewpoint distance information to the image
generating unit 127.
[0096]
10 The disparity image information acquiring unit 125
acquires the disparity image information that is supplied
from the auxiliary decoder 132 and supplies the acquired
disparity image information to the image generating unit
127. The compatibility information acquiring unit 126
15 acquires the compatibility information that is supplied
from the auxiliary decoder 132 and supplies the acquired
compatibility information to the image generating unit
127.
[0097]
20 The image generating unit 127 outputs an image in
accordance with an instruction supplied from a viewer and
displays the image on a display device not illustrated in
the figure. More specifically, the image generating unit
127 (generation unit) generates images of three or more
25 viewpoints, which correspond to a display device not
illustrated in the figure, each having the resolution
that is a half of the resolution of the compatible image
or the auxiliary image by using the compatible image, the
multiplexed image of the auxiliary images, the disparity
30 image of the compatible image, and the multiplexed image
of the disparity images of the auxiliary images in
38
•
accordance with a viewer's instruction for displaying a
3D image of multi-viewpoints based on the image
information supplied from the image information acquiring
unit 123, the inter-viewpoint distance information
5 supplied from the inter-viewpoint distance information
acquiring unit 124, the disparity image information
supplied from the disparity image information acquiring
unit 125, the compatibility information supplied from the
compatibility information acquiring unit 126, and the
10 like.
[0098]
More specifically, the image generating unit 127
(separation unit) separates the disparity image of each
auxiliary image from the multiplexed image of the
15 disparity images of the auxiliary images based on the
information that represents a multiplexing mode of the
disparity images of the auxiliary images that is included
in the disparity image information supplied from the
disparity image information acquiring unit 125. In
20 addition, the image generating unit 127 separates each
auxiliary image from the multiplexed image of the
auxiliary images based on the information that represents
a multiplexing mode of auxiliary images that is included
in the image information supplied from the image
25 information acquiring unit 123.
[0099]
Furthermore, the image generating unit 127
determines the position of each viewpoint of the multiviewpoint
images to be generated based on the inter-
30 viewpoint distance information and the number of
viewpoints that corresponds to a display device not
39
•
illustrated in the figure. Then, the image generating
unit 127 generates the image of each viewpoint of which
the position is determined by using the compatible image,
each auxiliary image, the disparity image of the
5 compatible image, and the disparity image of each
auxiliary images. Then, the image generating unit 127
converts the resolution of the generated image of each
viewpoint into resolution that is ~l/the number of
viewpoints" of the resolution of the compatible image or
10 the auxiliary image, synthesizes the images, and displays
the synthesized image on a display device not illustrated
in the figure.
[0100]
At this time, the multi-viewpoint images after the
15 synthesis are displayed such that the viewing angles are
different for each viewpoint, and a viewer can view a 3D
image without wearing glasses by seeing images of
arbitrary two viewpoints with his left and right eyes.
[0101]
20 In addition, the image generating unit 127 outputs
the image A1 that is the compatible image supplied from
the compatible decoder 131 of the decoder 122 in
accordance with a viewer's instruction for displaying a
2D image, thereby displaying the image on a display
25 device not illustrated in the figure. Accordingly, the
viewer can view the 2D image.
[0102]
[Description of Process of Decoding Device]
Fig. 13 is a flowchart that illustrates a decoding
30 process performed by the decoding device 120 illustrated
in Fig. 12. This decoding process, for example, is
40
•
started when a multiplexed stream that is transmitted
from the encoding device 50 illustrated in Fig. 5 is
input to the decoding device 120.
[0103]
5 In Step S31 illustrated in Fig. 13, the separation
unit 121 of the decoding device 120 receives the
multiplexed stream that is transmitted from the encoding
device 50 and separates each TS. The separation unit 121
extracts the compatible stream and the auxiliary stream
10 from the separated TS and supplies the extracted streams
to the decoder 122. The auxiliary decoder 132 of the
decoder 122 supplies the information used for specifying
the compatible stream to the compatible decoder 131 based
on the compatibility information included in the
15 auxiliary stream that is supplied from the separation
unit 121.
[0104]
In Step S32, the compatible decoder 131 identifies
the compatible stream out of the compatible stream and
20 the auxiliary stream that are supplied from the
separation unit 121 based on the information used for
specifying the compatible stream that is supplied from
the auxiliary decoder 132.
[0105]
25 In Step S33, the compatible decoder 131 decodes the
compatible image that is included in the compatible
stream in accordance with a mode that corresponds to the
Ave mode and supplies an image A1 that is acquired as a
result of the decoding process to the image generating
30 unit 127.
[0106]
41
•
In Step S34, the image generating unit 127
determines whether or not an instruction for displaying a
2D image has been made by a viewer. In a case where it
is determined that the instruction for displaying a 2D
5 image has not been made by the viewer in Step S34, in
other words, in a case where an instruction for
displaying a 3D image of the multi-viewpoint mode has
been made by the viewer, the auxiliary decoder 132
identifies the auxiliary stream out of the compatible
10 stream and the auxiliary stream that are supplied from
the separation unit 121 based on the compatibility
information.
[0107]
In Step S35, the auxiliary decoder 132 decodes the
15 multiplexed image of auxiliary images, the disparity
image A1' of the compatible image, and the multiplexed
image of the disparity images of the auxiliary images,
which are encoded, included in the auxiliary stream in
accordance with a mode that corresponds to the auxiliary
20 encoder 62 illustrated in Fig. 5. The auxiliary decoder
132 supplies the multiplexed image of the auxiliary
images, the disparity image A1' of the compatible image,
and the multiplexed image of the disparity images of the
auxiliary images that are acquired as a result of the
25 decoding process to the image generating unit 127. In
addition, the auxiliary decoder 132 supplies the image
information that is included in the auxiliary stream to
the image information acquiring unit 123 and supplies the
inter-viewpoint distance information to the inter-
30 viewpoint distance information acquiring unit 124.
Furthermore, the auxiliary decoder 132 supplies the
42
•
disparity image information that is included in the
auxiliary stream to the disparity image information
acquiring unit 125 and supplies the compatibility
information to the compatibility information acquiring
5 unit 126.
[0108]
In Step S36, the image information acquiring unit
123 acquires the image information that is supplied from
the auxiliary decoder 132 and inputs the image
10 information to the image generating unit 127. In Step
S37, the inter-viewpoint distance information acquiring
unit 124 acquires the inter-viewpoint distance
information that is supplied from the auxiliary decoder
132 and inputs the inter-viewpoint distance information
15 to the image generating unit 127.
[0109]
In Step S38, the disparity image information
acquiring unit 125 acquires the disparity image
information that is supplied from the auxiliary decoder
20 132 and inputs the disparity image information to the
image generating unit 127. In Step S39, the
compatibility information acquiring unit 126 acquires the
compatibility information that is supplied from the
auxiliary decoder 132 and inputs the compatibility
25 information to the image generating unit 127.
[0110]
In Step S40, the image generating unit 127
determines the position of each viewpoint of a multiviewpoint
image to be generated based on the inter-
30 viewpoint distance information supplied from the interviewpoint
distance information acquiring unit 124 and the
43
•
number of viewpoints that corresponds to a display device
not illustrated in the figure. For example, in a case
where an inter-viewpoint distance that is included in the
inter-viewpoint distance information is short, the image
5 generating unit 127 also determines the position of a
viewpoint that is located on the outer side of the
viewpoints of images B1 and C1 as the position of a
viewpoint of the multi-viewpoint 3D image to be generated.
On the other hand, in a case where the inter-viewpoint
10 distance that is included in the inter-viewpoint distance
information is long, the image generating unit 127
determines only positions of viewpoints located on the
inner side of the images B1 and C1 as the positions of
viewpoints of the multi-viewpoint 3D image to be
15 generated.
[0111]
In Step S41, the image generating unit 127
generates images of the viewpoints each having the
resolution that is a half of the resolution of the
20 compatible image or the auxiliary image based on the
position of each viewpoint determined in Step S40, the
image information supplied from the image information
acquiring unit 123, the disparity image information
supplied from the disparity image information acquiring
25 unit 125, the compatibility information supplied from the
compatibility information acquiring unit 126, and the
like by using the compatible image, the multiplexed image
of the auxiliary images, the disparity image of the
compatible image, and the multiplexed image of the
30 disparity images of the auxiliary images.
[0112]
44
•
In Step S42, the image generating unit 127 converts
the resolution of the image of each viewpoint that is
generated in Step S41 into resolution that is ~11 the
number of viewpoints" of the resolution of the compatible
5 image or the auxiliary image and synthesizes the images
of each viewpoint after the conversion based on the
positions of the viewpoints.
[0113]
In Step S43, the image gene'rating unit 127 outputs
10 the multi-viewpoint image after the synthesis that is
acquired by the process of Step S42 to a display device
that is not illustrated in the figure, thereby displaying
the multi-viewpoint image such that viewing angles are
different for each viewpoint. Then, the process ends.
15 [0114]
On the other hand, in a case where it is determined
that an instruction for displaying a 2D image has been
made by a viewer in Step S34, in Step S44, the image
generating unit 127 outputs an image A1 that is a
20 compatible image supplied from the compatible decoder 131
to a display device not illustrated in the figure,
thereby displaying the image. Then, the process ends.
[0115]
In a decoding device that can decode only a
25 compatible stream that has compatibility with the
decoding device 120, the TS of the auxiliary stream is
ignored, and only the processes of Steps S33 and S44 are
performed.
[0116]
30 As above, the decoding device 120 can decode a
compatible stream and an auxiliary stream that are
45
•
encoded in accordance with a mode having compatibility
with an existing mode by the encoding device 50.
[0117]
In addition, since the decoding device 120
5 generates a multi-viewpoint image using a compatible
image of one viewpoint and an auxiliary image of two
viewpoints, the encoding device 50 may prepare only two
imaging units 51B and SlC in addition to the existing
imaging unit 51A that captures 'a compatible image of one
10 viewpoint as imaging units that generate multi-viewpoint
images. Accordingly, the installation of the imaging
units for generating multi-viewpoint images can be
performed in an easy manner at a low cost.
[0118]
15
[Configuration Example of Encoding Device According to
Second Embodiment]
Fig. 14 is a block diagram that illustrates a
configuration example of an encoding device according to
20 a second embodiment of the present technology.
[0119]
In the configuration illustrated in Fig. 14, the
same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 5.
25 Duplicate description will not be presented as is
appropriate.
[0120]
The configuration of the encoding device 140
illustrated in Fig. 14 is mainly different from the
30 configuration illustrated in Fig. 5 in that imaging units
141A to 1410, an image converting unit 142, a disparity
46
•
image generating unit 143, an inter-viewpoint distance
information generating unit 144, and an encoder 145 are
disposed instead of the imaging units 51A to 51C, the
image converting unit 52, the disparity image generating
5 unit 53, the inter-viewpoint distance information
generating unit 56, and the encoder 58.
[0121]
The encoding device 140 performs encoding in
accordance with the AVC.mode with images of two
10 viewpoints out of multi-viewpoint images set as
compatible images, thereby securing the compatibility
with an existing encoding device that encodes a 3D image
of the two viewpoints mode.
[0122]
15 More specifically, the imaging unit 141A of the
encoding device 140 captures an HD image of a
predetermined viewpoint as an image A2 and supplies the
captured image to the image converting unit 142, the
disparity image generating unit 143, and the inter-
20 viewpoint distance information generating unit 144. The
imaging unit 141B captures an HD image of a viewpoint
that is different from the viewpoint of the image A2 as
an image B2 at a position that is separate from the
imaging unit 141A by a distance dd2~ in a horizontal
25 direction and supplies the captured image to the image
converting unit 142, the disparity image generating unit
143, and the inter-viewpoint distance information
generating unit 144.
[0123]
30 The imaging unit 141C captures an HD image of a
viewpoint different from the viewpoints of the images A2
47
•
and B2 as an image C2 at a position separate from the
imaging unit 141B by a distance dd2Bc in the horizontal
direction that is opposite to the imaging unit 141A and
supplies the captured image to the image converting unit
5 142, the disparity image generating unit 143, and the
inter-viewpoint distance information generating unit 144.
The imaging unit 141D captures an HD image of a viewpoint
different from the viewpoints of the images A2 to C2 as
an image D2 at a position separate from the imaging unit
10 141A by a distance dd2AD in the horizontal direction that
is opposite to the imaging unit 141B and supplies the
captured image to the image converting unit 142, the
disparity image generating unit 143, and the interviewpoint
distance information generating unit 144.
15 [0124]
The viewpoints that correspond to the images C2 and
D2 are viewpoints that are located on the outer side out
of viewpoints of images that can be recognized as 3D
images. In this way, a decoding device that corresponds
20 to the encoding device 140 can generate multi-viewpoint
images by interpolating images of viewpoints located on
the further inner side than the viewpoints of the images
C2 and D2 by using the images A2 to D2. As a result, the
multi-viewpoint images can be generated with precision
25 that is higher than that of a case where images of
viewpoints located on the outer side are interpolated
using images of viewpoints located on the inner side.
The distances dd2~, dd2 Bc , and dd2 AD may be configured to
be either fixed or changed each time.
30 [0125]
The image converting unit 142 determines the image
48
•
A2 that is supplied from the imaging unit 141A of which
the position in the horizontal direction is located on
the inner side out of the imaging units 141A to 141D and
the image B2 that is supplied from the imaging unit 141B
5 as compatible images. Then, the image converting unit
142 multiplexes the images A2 and B2 that are the
compatible images in accordance with a predetermined
multiplexing mode and supplies the multiplexed image to
the encoder 145. In addition, the image converting unit
10 142 supplies information that designates the images A2
and B2 as compatible images to the compatibility
information generating unit 55.
[0126]
In addition, the image converting unit 142 sets the
15 images C2 and D2 other than the images A2 and B2 as
auxiliary images and multiplexes the images C2 and D2 in
accordance with a predetermined multiplexing mode. The
image converting unit 142 supplies a multiplexed image
that is acquired as a result of the multiplexing process
20 to the encoder 145. The image converting unit 142
supplies information that represents a multiplexing mode
of compatible images and auxiliary images to the image
information generating unit 54.
[0127]
25 The disparity image generating unit 143 detects the
disparity of each pixel of the images A2 to D2 using the
images A2 to D2 that are supplied from the imaging units
141A to 141D. The disparity image generating unit 143
generates a disparity image A2' that represents the
30 disparity of each pixel of the image A2 that is a
compatible image and a disparity image B2' that
49
•
represents the disparity of each pixel of the image B2
and multiplexes the disparity images A2' and B2' in
accordance with a predetermined multiplexing mode. The
disparity image generating unit 143 supplies a
5 multiplexed image that is acquired as a result thereof to
the encoder 145.
[0128]
In addition, the disparity image generating unit
143 generates a disparity image C2' that represents the
10 disparity of each pixel of the image C2 that is an
auxiliary image and a disparity image 02' that represents
the disparity of each pixel of the image 02 that is an
auxiliary image and multiplexes the disparity images C2'
and 02' in accordance with a predetermined multiplexing
15 mode. The disparity image generating unit 143 supplies a
multiplexed image that is acquired as a result thereof to
the encoder 145. The disparity image generating unit 143
supplies information that represents the multiplexing
mode of disparity images of compatible images and
20 auxiliary images to the disparity image information
generating unit 57.
[0129]
The inter-viewpoint distance information generating
unit 144 detects each inter-viewpoint distance between
25 the images A2 to 02 using the images A2 to 02 that are
supplied from the imaging units 141A to 1410. For
example, the inter-viewpoint distance information
generating unit 144 detects a distance ~d2~ between the
imaging units 141A and 141B in the horizontal direction,
30 a distance ~d2Bc between the imaging units 141B and 141C
in the horizontal direction, and a distance ~d2AD between
50
•
the imaging units 141A and 141D in the horizontal
direction as inter-viewpoint distances. The interviewpoint
distance information generating unit 144
generates information that represents the inter-viewpoint
5 distances and the like as inter-viewpoint distance
information and supplies the generated inter-viewpoint
distance information to the encoder 145.
[0130]
The encoder 145 is configured by a compatible
10 encoder 151 and an auxiliary encoder 152. The compatible
encoder 151 designates a multiplexed image of the
compatible images out of the multiplexed image of the
compatible images and the multiplexed image of the
auxiliary images that are supplied from the image
15 converting unit 142, encodes the multiplexed image of the
compatible images in accordance with the existing AVC
mode, adds various kinds of information thereto, and
supplies an acquired resultant encoded stream to the
multiplexing unit 59 as a compatible stream.
20 [0131]
The auxiliary encoder 152 encodes the multiplexed
image of the auxiliary images that is supplied from the
image converting unit 142, and the multiplexed image of
the disparity images of the compatible images and the
25 multiplexed image of the disparity images of the
auxiliary images that are supplied from the disparity
image generating unit 143 in accordance with a
predetermined mode. As the encoding mode employed by the
auxiliary encoder 152, for example, the AVC mode, the MVC
30 mode, or the like can be used.
[0132]
51
•
In addition, the auxiliary encoder 152 adds the
image information supplied from the image information
generating unit 54, the compatibility information
supplied from the compatibility information generating
5 unit 55, the inter-viewpoint distance information
supplied from the inter-viewpoint distance information
generating unit 144, the disparity image information
supplied from the disparity image information generating
unit 57, and the like, to an encoded image that is
10 acquired as a result of the encoding process, thereby
generating an encoded stream. The auxiliary encoder 152
supplies the encoded stream to the multiplexing unit 59
as an auxiliary stream.
[0133]
15 [Description of Process of Encoding Device]
Figs. 15 and 16 represent a flowchart that
illustrates an encoding process that is performed by the
encoding device 140 illustrated in Fig. 14. This
encoding process, for example, is started when the images
20 A2 to D2 are output from the imaging units l41A to 141D.
[0134]
In Step S51 illustrated in Fig. 15, the interviewpoint
distance information generating unit 144
detects distances ~d2~, ~d2Bc, and ~d2AD as inter-
25 viewpoint distances using the images A2 to D2 that are
supplied from the imaging units 141A to 141D.
[0135]
In Step S52, the inter-viewpoint distance
information generating unit 144 generates information
30 that represents the inter-viewpoint distances detected in
Step S51 and the like as inter-viewpoint distance
52
•
information and inputs the generated inter-viewpoint
distance information to the encoder 145.
[0136]
In Step S53, the image converting unit 142
5 determines the image A2 that is supplied from the imaging
unit 141A, of which the position in the horizontal
direction is located on the inner side out of the imaging
units 141A to 141D, and the image B2 that is supplied
from the imaging unit 141B as compatible images, and
10 determines multiplexing modes of the compatible images
and the auxiliary images. The image converting unit 142
supplies information that designates the images A2 and B2
as compatible images to the compatibility information
generating unit 55 and supplies the multiplexing modes of
15 the compatible images and the auxiliary images to the
image information generating unit 54.
[0137]
In Step S54, the compatibility information
generating unit 55 generates the information designating
20 the images A2 and B2 as compatible images, a frame
packing mode as a compatible mode, and the like as
compatibility information based on the information that
is supplied from the image converting unit 142 and inputs
the generated compatibility information to the encoder
25 145.
[0138]
In Step S55, the image converting unit 142
multiplexes the images A2 and B2 that are compatible
images based on the multiplexing mode of compatible
30 images that is determined in Step S53 and supplies the
multiplexed image to the encoder 145.
53
•
[0139]
In Step S56, the image information generating unit
54 generates information that represents the multiplexing
modes of compatible images and auxiliary images and the
5 like as image information based on the information that
is supplied from the image converting unit 142 and inputs
the generated image information to the encoder 145.
[0140]
In Step S57, the image convert~ng unit 142 sets
10 images C2 and D2 other than the images A2 and B2 as
auxiliary images and multiplexes the auxiliary images
based on the multiplexing mode of auxiliary images that
is determined in Step S53, thereby acquiring a
multiplexed image of the auxiliary images.
15 [0141]
In Step S58, the image converting unit 142 inputs
the multiplexed image of the compatible images and the
multiplexed image of the auxiliary images to the encoder
145.
20 [0142]
In Step S59 illustrated in Fig. 16, the disparity
image generating unit 143 detects the disparity of each
pixel of the images A2 to D2 using the images A2 to D2
that are supplied from the imaging units 141A to 141D and
25 generates disparity images A2' to D2'.
[0143]
In Step S60, the disparity image generating unit
143 determines multiplexing modes of the disparity images
of the compatible images and the disparity images of the
30 auxiliary images and supplies information that represents
the multiplexing modes to the disparity image information
54
•
generating unit 57.
[0144]
In Step S61, the disparity image information
generating unit 57 generates information that represents
5 the multiplexing modes of the disparity images of the
compatible images and the disparity images of the
auxiliary images and the like as disparity image
information based on the information that is supplied
from the disparity image generating unit 143 and inputs
10 the generated disparity image information to the encoder
145.
[0145]
In Step S62, the disparity image generating unit
143 multiplexes the disparity images A2' and B2' of the
15 compatible images based on the multiplexing mode of the
disparity images of compatible images that is determined
in Step S60 and multiplexes the disparity images e2' and
D2' of the auxiliary images based on the multiplexing
mode of the disparity images of auxiliary images.
20 [0146]
In Step S63, the disparity image generating unit
143 inputs the multiplexed image of the disparity images
of the compatible images and the multiplexed image of the
disparity images of the auxiliary images that are
25 acquired as a result of the multiplexing process
illustrated in Step S62 to the encoder 145.
[0147]
In Step S64, the compatible encoder 151 of the
encoder 145 encodes the multiplexed image of the
30 compatible images that is supplied from the image
converting unit 142 in accordance with the existing Ave
55
•
mode and supplies an encoded stream acquired as a result
thereof to the multiplexing unit 59 as a compatible
stream.
[0148]
5 In Step S65, the auxiliary encoder 152 encodes the
multiplexed image of the auxiliary images that is
supplied from the image converting unit 142 and the
multiplexed image of the disparity images of the
compatible il1lages and the multiplexed image of the
10 disparity images of the auxiliary images that are
supplied from the disparity image generating unit 53 in
accordance with a predetermined mode. Then, the process
proceeds to Step S66.
[0149]
15 The process of Steps S66 and S67 is the same as the
process of Steps S25 and S26 illustrated in Fig. 11, and
thus the description thereof will not be presented.
[0150]
As above, since the encoding device 140 encodes
20 images of two viewpoints out of multi-viewpoint images as
compatibles image in accordance with an existing encoding
mode, the compatibility with an existing encoding device
that encodes a 3D image of the two-viewpoint mode can be
secured.
25 [0151]
In addition, since the encoding device 140 encodes
the images A2 to D2 of four viewpoints, a decoding device
that corresponds to the encoding device 140 can generate
multi-viewpoint images from the images A2 to D2 of four
30 viewpoints. In this way, according to the decoding
device, compared to a case where multi-viewpoint images
56
•
are generated from images of two viewpoints, the
viewpoints of images that can be generated are not
limited, and multi-viewpoint images having relatively
high precision can be generated.
5 [0152]
[Configuration Example of Decoding Device]
Fig. 17 is a diagram that illustrates a
configuration example of a decoding device that decodes
the multiplexed stream transmitted from the encoding
10 device 140 illustrated in Fig. 14.
[0153]
In the configuration illustrated in Fig. 17, the
same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 12.
15 Duplicate description will not be presented as is
appropriate.
[0154]
The configuration of the decoding device 170
illustrated in Fig. 17 is mainly different from the
20 configuration illustrated in Fig. 12 in that an image
generating unit 171 is disposed instead of the image
generating unit 127. The decoding device 170 generates
images of two viewpoints or multi-viewpoint images by
decoding the multiplexed stream that is transmitted from
25 the encoding device 140 and displays the generated images
on a display device not illustrated in the figure.
[0155]
More specifically, the image generating unit 171 of
the decoding device 170 outputs images in accordance with
30 a display instruction supplied from a viewer, thereby
displaying the images on a display device (not
57
•
illustrated). When described in more detail, the image
generating unit 171 generates images of three or more
viewpoints, which correspond to a display device (not
illustrated), each having the resolution that is a half
5 of the resolution of the compatible image or the
auxiliary image by using the multiplexed image of the
compatible images, the multiplexed image of the auxiliary
images, the multiplexed imageo~ the disparity images of
the compatible images, and the multiplexed image of the
10 disparity images of the auxiliary images in accordance
with a viewer's instruction for displaying a 3D image of
multi-viewpoint mode based on the image information
supplied from the image information acquiring unit 123,
the inter-viewpoint distance information supplied from
15 the inter-viewpoint distance information acquiring unit
124, the disparity image information supplied from the
disparity image information acquiring unit 125, the
compatibility information supplied from the compatibility
information acquiring unit 126, and the like.
20 [0156]
More specifically, the image generating unit 171
separates the disparity image of each auxiliary image
from the multiplexed image of the disparity images of the
auxiliary images based on the information that represents
25 a multiplexing mode of the disparity images of the
auxiliary images that is included in the disparity image
information supplied from the disparity image information
acquiring unit 125. In addition, the image generating
unit 171 separates the disparity image of each compatible
30 image from the multiplexed image of the disparity images
of the compatible images based on the information, which
58
•
represents the multiplexing mode of the disparity images
of the compatible images, included in the disparity image
information.
[0157]
5 Furthermore, the image generating unit 171
separates each auxiliary image from the multiplexed image
of the auxiliary images based on the information that
represents a multiplexing mode of the auxiliary images
that is included in the image information supplied from
10 the image information acquiring unit 123. In addition,
the image generating unit 171 separates each compatible
image from the multiplexed image of the compatible images
based on the information that represents a multiplexing
mode of the compatible images that is included in the
15 image information.
[0158]
Furthermore, the image generating unit 171
determines the position of each viewpoint of the multiviewpoint
images to be generated based on the inter-
20 viewpoint distance information and the number of
viewpoints that corresponds to a display device not
illustrated in the figure. Then, the image generating
unit 171 generates the image of each viewpoint of which
the position is determined by using each compatible image,
25 each auxiliary image, the disparity image of each
compatible image, and the disparity image of each
auxiliary image. Then, the image generating unit 171
converts the resolution of the generated multi-viewpoint
images into resolution that is "lithe number of
30 viewpoints" of the resolution of the compatible image or
the auxiliary image, synthesizes the images, and displays
59
•
the synthesized image on a display device not illustrated
in the figure.
[0159]
At this time, the multi-viewpoint images after the
5 synthesis are displayed such that the viewing angles are
different for each viewpoint, and a viewer can view a 3D
image without wearing glasses by seeing images of
arbitrary two viewpoints with his left and right eyes.
[0160]
10 In addition, the image generating unit 171
separates the multiplexed image of the compatible images
that is supplied from the decoder 122 into images A2 and
B2 of which the resolution is a half of the resolution of
the compatible image based on the image information
15 supplied from the image information acquiring unit 123 in
accordance with an instruction for displaying a 3D image
of the two-viewpoint mode that is supplied from a viewer.
Then, the image generating unit 171 alternately outputs
the separated images A2 and B2 of which the resolution is
20 a half of the resolution of the compatible image, thereby
displaying the separated images on a display device not
illustrated in the figure. At this time, the viewer can
view a 3D image by wearing glasses in which one of a
left-eye shutter and a right-eye shutter is open at the
25 time of displaying the image A2, and the other is open at
the time of displaying the image B2 and viewing the
images A2 and B2 that are alternately displayed on the
display device.
[0161]
30 [Description of Process of Decoding Device]
Fig. 18 is a flowchart that illustrates a decoding
60
•
process performed by the decoding device 170 illustrated
in Fig. 17. This decoding process, for example, is
started when a multiplexed stream that is transmitted
from the encoding device 140 illustrated in Fig. 14 is
5 input to the decoding device 170.
[0162]
The process of Steps S81 to S83 illustrated in Fig.
18 is the same as the process of Steps S31 to S33
illustrated in Fig. 13, and thus the description thereof
10 will not be presented.
[0163]
After the process of Step S83, in Step S84, the
image generating unit 171 determines whether or not an
instruction for displaying a 3D image of the two-
15 viewpoint mode has been made by a viewer. In a case
where it is determined that the instruction for
displaying a 3D image of the two-viewpoint mode has not
been made by the viewer in Step S84, in other words, in a
case where an instruction for displaying a 3D image of
20 the multi-viewpoint mode has been made, the process
proceeds to Step S85. The process of Steps S85 to S93 is
the same as the process of Steps S35 to S43 illustrated
in Fig. 13, and thus the description thereof will not be
presented.
25 [0164]
On the other hand, in a case where it is determined
that the instruction for displaying a 3D image of the
two-viewpoint mode has been made by the viewer in Step
S84, the process proceeds to Step S94.
30 [0165]
In Step S94, the image generating unit 171
61
•
separates the multiplexed image of the compatible images
that is acquired as result of the decoding process
performed by the compatible decoder 131 based on the
information that represents a multiplexing mode of the
5 compatible image that is included in the image
information supplied from the image information acquiring
unit 123.
[0166]
In Step S95, the image generating unit 171
10 alternately outputs the images A2 and B2 that are the
compatible images, of which the resolution is a half of
the resolution of the separated compatible image to a
display device not illustrated in the figure, thereby
displaying the images. Then, the process ends.
15 [0167]
In addition, in a decoding device that can decode
only a compatible stream that has compatibility with the
decoding device 170, a TS of the auxiliary stream is
ignored, and only the process of Steps S83, S94, and S95
20 is performed. However, in such a case, in the process of
Step S94, a multiplexed image of compatible images is
separated based on not the multiplexing mode of the
compatible image that is included in the image
information but the multiplexing mode of the compatible
25 image that is determined in advance.
[0168]
As above, the decoding device 170 can decode a
compatible stream and an auxiliary stream that are
encoded by the encoding device 140 in accordance with a
30 mode that has compatibility with an existing mode.
[0169]
62
•
In addition, since the decoding device 170
generates multi-viewpoint images using compatible images
of two viewpoints and auxiliary images of two viewpoints,
the encoding device 140 may prepare only two imaging
5 units 141C and 141D other than the existing imaging units
141A and 141B that capture compatible images of two
viewpoints as imaging units used for generating multiviewpoint
images. Accordingly, the installation of the
imaging units for generating multi-viewpoint images can
10 be easily performed at a low cost.
[0170]

15
[Configuration Example of Encoding Device According to
Third Embodiment]
Fig. 19 is a block diagram that illustrates a
configuration example of an encoding device according to
a third embodiment of the present technology.
[0171]
25
In the configuration illustrated in Fig. 19, the
20 same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 5 or 14.
Duplicate description will not be presented as is
appropriate.
[0172]
The configuration of the encoding device 180
illustrated in Fig. 19 is mainly different from the
configuration illustrated in Fig. 14 in that an image
converting unit 181 and an encoder 182 are disposed
instead of the image converting unit 142 and the encoder
30 145. The encoding device 180 secures the compatibility
with an existing encoding device that encodes a 3D image
63
•
of the two-viewpoint mode by encoding images of two
viewpoints out of multi-viewpoint images in accordance
with the MVC mode as compatible images.
[0173]
5 More specifically, the image converting unit 181 of
the encoding device 180, similarly to the image
converting unit 142 illustrated in Fig. 14, determines an
image A2 that is supplied from the imaging unit 141A, of
which the position in the horizontal direction is located
10 on the inner side out of the imaging units 141A to 141D,
and an image B2 that is supplied from the imaging unit
141B as compatible images. Then, the image converting
unit 181 directly supplies the images A2 and B2 that are
compatible images to the encoder 182. In addition, the
15 image converting unit 181, similarly to the image
converting unit 142, supplies information that designates
the images A2 and B2 as compatible images to the
compatibility information generating unit 55.
[0174]
20 In addition, the image converting unit 181,
similarly to the image converting unit 142, sets images
C2 and D2 other that the images A2 and B2 as auxiliary
images and multiplexes the auxiliary images in accordance
with a predetermined multiplexing mode. The image
25 converting unit 181 supplies a multiplexed image that is
acquired as a result of the multiplexing process to the
encoder 182 and supplies information that represents the
multiplexing mode of the auxiliary images to the image
information generating unit 54.
30 [0175]
The encoder 182 is configured by a compatible
64
•
encoder 191 and an auxiliary encoder 152. The compatible
encoder 191 of the encoder 182 designates compatible
images from among the compatible images and the
multiplexed image of the auxiliary images that are
5 supplied from the image converting unit 181 and encodes
the image A2 out of the compatible images as a base image
in accordance with the existing AVC mode, and encodes the
image B2 as a dependent image in accordance with the
existing MVC mode. The compatible encoder 191 generates
10 an encoded stream by adding various kinds of information
to an encoded image that is acquired as a result of the
encoding process and supplies the encoded stream to the
multiplexing unit 59 as a compatible stream.
[0176]
15 [Description of Process of Encoding Device]
Figs. 20 and 21 are flowcharts that illustrate an
encoding process that is performed by the encoding device
180 illustrated in Fig. 19. This encoding process, for
example, is started when the images A2 to 02 are output
20 from the imaging units 141A to 1410.
[0177]
The process of Steps Slll and Sl12 illustrated in
Fig. 20 is the same as the process of Steps S51 and S52
illustrated in Fig. 15, and thus the description thereof
25 will not be presented.
[0178]
After the process of Step Sl12, in Step Sl13, the
image converting unit 181 determines the image A2 that is
supplied from the imaging unit 141A, of which the
30 position in the horizontal direction is located on the
inner side out of the imaging units 141A to 1410, and the
65
•
image B2 that is supplied from the imaging unit 141B as
compatible images and determines a multiplexing mode of
the auxiliary images. The image converting unit 181
supplies information that designates the images A2 and B2
5 as compatible images to the compatibility information
generating unit 55 and supplies the multiplexing mode of
the auxiliary images to the image information generating
unit 54. Then, the process proceeds to Step Sl14.
[0179]
10 The process of Steps Sl14 to Sl17 is the same as
the process of Steps S14 to S17 illustrated in Fig. 10,
and thus the description thereof will not be presented.
The process of Steps Sl18 to S122 is the same as the
process of Steps S59 to S63 illustrated in Fig. 16, and
15 thus the description thereof will not be presented.
[0180]
After the process of Step S122, in Step S123, the
compatible encoder 191 of the encoder 182 encodes the
image A2 out of the compatible images supplied from the
20 image converting unit 181 as a base image in accordance
with the existing Ave mode and encodes the image B2 as a
dependent image in accordance with the existing Mve mode.
The compatible encoder 191 generates an encoded stream by
adding various kinds of information to an encoded image
25 that is acquired as a result of the encoding process and
supplies the encoded stream to the multiplexing unit 59
as a compatible stream. Then, the process proceeds to
Step S124.
[0181]
30 The process of Steps S124 to S126 is the same as
the process of Steps S65 to S67 illustrated in Fig. 16,
66
•
and thus the description thereof will not be presented.
[0182]
As above, since the encoding device 180 encodes
images of two viewpoints out of multi-viewpoint images as
5 compatible images in accordance with an existing encoding
mode, the compatibility with an existing encoding device
that encodes a 3D-image of the two-viewpoint mode can be
secured.
[0183]
10 [Configuration Example of Decoding Device]
Fig. 22 is a diagram that illustrates a
configuration example of a decoding device that decodes
the multiplexed stream transmitted from the encoding
device 180 illustrated in Fig. 19.
15 [0184]
In the configuration illustrated in Fig. 22, the
same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 12.
Duplicate description will not be presented as is
20 appropriate.
[0185]
The configuration of the decoding device 200
illustrated in Fig. 22 is mainly different from the
configuration illustrated in Fig. 12 in that a decoder
25 201 and an image generating unit 202 are disposed instead
of the decoder 122 and the image generating unit 127.
The decoding device 200 generates images of two
viewpoints, or multi-viewpoint images by decoding the
multiplexed stream that is transmitted from the encoding
30 device 180 and displays the generated images on a display
device not illustrated in the figure.
67
•
[0186]
More specifically, the decoder 201 of the decoding
device 200 is configured by a compatible decoder 211 and
an auxiliary decoder 132. The compatible decoder 211 of
5 the decoder 201, similarly to the compatible decoder 131
illustrated in Fig. 12, identifies the compatible stream
out of the compatible stream and the auxiliary stream
that are supplied from the separation unit 121 based on
the information used for specifying a compatible stream
10 that is supplied from the auxiliary decoder 132. The
compatible decoder 211 decodes an encoded compatible
image that is included in the compatible stream in
accordance with a mode that corresponds to the MVC mode
and supplies images A2 and B2 that are acquired as a
15 result of the decoding process to the image generating
unit 202.
[0187]
The image generating unit 202 outputs the image in
accordance with a display instruction supplied from a
20 viewer, thereby displaying the image on a display device
not illustrated in the figure. Described in more detail,
the image generating unit 202 generates images of three
or more viewpoints, which correspond to a display device
not illustrated in the figure, each having the resolution
25 that is a half of the resolution of the compatible image
or the auxiliary image by using the compatible image, the
multiplexed image of the auxiliary images, the
multiplexed image of the disparity images of the
compatible images, and the multiplexed image of the
30 disparity images of the auxiliary images in accordance
with a viewer's instruction for displaying a 3D image of
68
•
the multi-viewpoint mode based on the image information
supplied from the image information acquiring unit 123,
the inter-viewpoint distance information supplied from
the inter-viewpoint distance information acquiring unit
5 124, the disparity image information supplied from the
disparity image information acquiring unit 125, the
compatibility information supplied from the compatibility
information acquiring unit 126, and the like.
[0188]
10 More specifically, the image generating unit 202
separates the disparity image of each auxiliary image
from the multiplexed image of the disparity images of the
auxiliary images based on the information that represents
a multiplexing mode of the disparity images of the
15 auxiliary images that is included in the disparity image
information supplied from the disparity image information
acquiring unit 125. In addition, the image generating
unit 202 separates the disparity image of each compatible
image from the multiplexed image of the disparity images
20 of the compatible images based on the information that
represents a multiplexing mode of the disparity images of
the compatible images that is included in the disparity
image information.
[0189]
25 Furthermore, the image generating unit 202
separates each auxiliary image from the multiplexed image
of the auxiliary images based on the information that
represents a multiplexing mode of the auxiliary images
that is included in the image information supplied from
30 the image information acquiring unit 123. In addition,
the image generating unit 202 determines the positions of
69
•
viewpoints of multi-viewpoint images to be generated
based on the inter-viewpoint distance information and the
number of viewpoints that corresponds to a display device
not illustrated in the figure. Then, the image
5 generating unit 202 generates an image of each viewpoint
of which the position is determined by using each
compatible image, each auxiliary image, the disparity
image of each compatible image, and the disparity image
of each auxiliary image. Then, the image generating unit
10 202, similarly to the image generating unit 127, converts
the resolution of the generated multi-viewpoint images
into the resolution that is "l/the number of viewpoints"
of the resolution of the compatible image or the
auxiliary image, synthesizes the converted images, and
15 displays the synthesized image on a display device not
illustrated in the figure.
[0190]
At this time, the multi-viewpoint images after the
synthesis are displayed such that the viewing angles are
20 different for each viewpoint, and a viewer can view a 3D
image without wearing glasses by seeing images of
arbitrary two viewpoints with his/her left and right eyes.
[0191]
In addition, the image generating unit 202
25 alternately outputs the images A2 and B2 as compatible
images supplied from the decoder 122 in accordance with
an instruction for displaying a 3D image of the twoviewpoint
mode that is supplied from a viewer, thereby
displaying the images on a display device not illustrated
30 in the figure. At this time, the viewer can view a 3D
image by wearing glasses in which one of a left-eye
70
•
shutter and a right-eye shutter is open at the time of
displaying the image A2, and the other is open at the
time of displaying the image B2 and viewing the images A2
and B2 that are alternately displayed on the display
5 device.
[0192]
[Description of Process of Decoding Device]
Fig. 23 is a flowchart that illustrates a decoding
process performed by the decoding device 200 illustrated
10 in Fig. 22. This decoding process, for example, is
started when a multiplexed stream that is transmitted
from the encoding device 180 illustrated in Fig. 19 is
input to the decoding device 200.
[0193]
15 The process of Steps S141 to S153 illustrated in
Fig. 23 is the same as the process of Steps S81 to S93
illustrated in Fig. 18, and thus the description thereof
will not be presented.
[0194]
20 In a case where it is determined that the
instruction for displaying a 3D image of the twoviewpoint
mode has been made by the viewer in Step S144,
the process proceeds to Step S154. In Step S154, the
image generating unit 202 alternately outputs images A2
25 and B2 that are compatible images acquired as a result of
the decoding process performed by the compatible decoder
211 to a display device not illustrated in the figure,
thereby displaying the images.
[0195]
30 In a decoding device that can decode only a
compatible stream that has compatibility with the
71
•
decoding device 200, the TS of the auxiliary stream is
ignored, and only the process of Steps S143 and S154 is
performed.
[0196]
5 As above, the decoding device 200 can decode a
compatible stream and an auxiliary stream that are
encoded in accordance with a mode having compatibility
with an existing mode by the encoding device 180.
[0197]
10
In the description presented above, although the
compatibility information, the disparity image
15 information, the image information, and the interviewpoint
distance information have been described to be
included in the auxiliary stream, the above-described
information may be included in the TS. In such a case,
for example, the compatibility information, the disparity
20 image information, the image information, and the interviewpoint
distance information are described in
descriptors of a PMT (program map table) or an SIT
(selection information table) within the packet of the TS
of the auxiliary stream.
25 [0198]
Figs. 24 to 26 are diagrams that illustrate'
examples of the description of the compatibility
information and the disparity image information in a case
where the compatibility information, the disparity image
30 information, the image information, and the interviewpoint
distance information are described in the
72
•
descriptor of the PMT.
[0199]
As illustrated in Fig. 24, in a case where the
compatibility information and the disparity image
5 information are described in the descriptor of the PMT, a
descriptor (3DV_view_structure_descriptor) in which
compatibility information is arranged and a descriptor
(depth_map_structure_descriptor) in which the disparity
image information is arranged are disposed as the
10 descriptor of the PMT.
[0200]
In the descriptor (3DV_view_structure_descriptor),
as illustrated in Fig. 25, following a descriptor tag
(descriptor_tag) and a descriptor length
15 (descriptor_length), the number (num_of_views) of
viewpoints of an image that is an encoding target, a
compatible mode (compatible mode), the information
(compatible_view) that designates a compatible image, and
the like are described as the compatibility information.
20 [0201]
In addition, in the descriptor
(depth_map_structure_descriptor), as illustrated in Fig.
26, following the descriptor tag and the descriptor
length, the number (num_of_depth_map) of disparity images,
25 a flag (is_frame_packing) that represents whether or not
disparity images are multiplexed, a multiplexing mode
(frame_packing_mode) of the disparity images, information
(comb_frame_packing_views) that designates the disparity
images that are multiplexed, and the like are described
30 as the disparity image information.
[0202]
73
•
In addition, the image information is the same as
the disparity image information except that the image
information is not information of the disparity image but
information of the compatible image and the auxiliary
5 image, and thus is not illustrated in the figure.
[0203]
Fig. 27 is a diagram that illustrates an example of
the description of the inter-viewpoint distance
information in a case where the compatibility information,
10 the disparity image information, the image information,
and the inter-viewpoint distance information are
described in the descriptors of the PMT.
[0204]
As illustrated in Fig. 27, in a case where the
15 inter-viewpoint distance information is described in the
descriptor of the PMT, a descriptor
(3DV_view_info_descriptor) in which the inter-viewpoint
distance information is arranged is disposed as the
descriptor of the PMT.
20 [0205]
In the descriptor (3DV_view_info_descriptor),
following the descriptor tag and the descriptor length,
the number (num_of_views) of viewpoints of an image that
is an encoding target, the number (num_of_view_distance)
25 of detected inter-viewpoint distances, a combination
(view[O,l]) of two images that correspond to the interviewpoint
distance, an inter-viewpoint distance
(view_distance[view_distance_id]) that corresponds to the
combination, and the like are described as the inter-
30 viewpoint distance information.
[0206]
74
•
For example, the decoding device according to the
present technology can be applied to a display device or
a reproduction device such as a television receiver.
[0207 ]
5 The encoding devices and the decoding devices
according to the first to third embodiments may be
combined. In such a case, the decoding device selects
the compatible decoder in accordance with a compatible
mode that is included in the compatibility information.
10 [0208]
In addition, in the first to third embodiments,
although the resolutions of the auxiliary image and the
disparity image are lowered and the images are
multiplexed, only the resolutions thereof may be lowered
15 without multiplexing the images. Furthermore, the
resolution of the disparity image may not be lowered.
[0209]
In addition, the image information, the interviewpoint
distance information, the disparity image
20 information, and the compatibility information may be
transmitted in a system that is different from that of
the encoded stream without being encoded. Furthermore,
the image information, the inter-viewpoint distance
information, the disparity image information, and the
25 compatibility information may be encoded and transmitted
in a system that is different from that of the encoded
stream.
[0210]
In addition, the image information, the inter30
viewpoint distance information, the disparity image
information, and the compatibility information may be
75
•
described in an area (the header, user data, or the like)
other than the above-described area of the encoded stream
without being encoded and may be encoded and described in
a predetermined area of the encoded stream. Furthermore,
5 the image information, the inter-viewpoint distance
information, the disparity image information, and the
compatibility information may be transmitted as auxiliary
information other than the encoded stream.
[0211]
10
[Configuration Example of Encoding Device According to
Fourth Embodiment]
Fig. 28 is a block diagram that illustrates a
configuration example of an encoding device according to
15 a fourth embodiment of the present technology.
[0212]
In the configuration illustrated in Fig. 28, the
same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 5.
20 Duplicate description will not be presented as is
appropriate.
[0213]
The configuration of the encoding device 300
illustrated in Fig. 28 is mainly different from the
25 configuration illustrated in Fig. 5 in that an image
converting unit 301, a disparity image generating unit
302, an image information generating unit 303, and a
disparity image information generating unit 304 are
arranged instead of the image converting unit 52, the
30 disparity image generating unit 53, the image information
generating unit 54, and the disparity image information
76
•
generating unit 57.
[0214]
The encoding device 300 transmits the image
information and the disparity image information with not
5 information relating to multiplexing but information
relating to the resolution lowering process being
included therein.
[0215]
More specifically, the image converting unit 301 of
10 the encoding device 300, similarly to the image
converting unit 52 illustrated in Fig. 5, determines an
image A1 supplied from an imaging unit 51A as a
compatible image. The image converting unit 301,
similarly to the image converting unit 52, supplies
15 information designating the image A1 as a compatible
image to the compatibility information generating unit 55.
Then, the image converting unit 301, similarly to the
image converting unit 52, directly supplies the image A1
that is the compatible image to the encoder 58.
20 [0216]
In addition, the image converting unit 301 sets
images B1 and C1 other than the image A1 as auxiliary
images, lowers the resolution of the images in accordance
with a horizontal half mode, and multiplexes the images,
25 thereby acquiring a multiplexed image of which the
resolution is the same as that of the compatible image.
Here, the horizontal half mode is a mode in which the
resolution in the horizontal direction is set as a half
of the resolution of the compatible image. The image
30 converting unit 301 supplies the multiplexed image of the
auxiliary images to the encoder 58 and supplies
77
i
:1
•
information that represents the conversion mode of the
resolution of the auxiliary images to the image
information generating unit 303.
[0217]
5 The disparity image generating unit 302, similarly
to the disparity image generating unit 53 illustrated in
Fig. 5, detects the disparity of each pixel of the images
Al to Cl by using the images Al to Cl that are supplied
from the imaging units 5lA to 5lC. The disparity image
10 generating unit 302, similarly to the disparity image
generating unit 53, generates a disparity image AI' and
directly supplies the generated disparity image to the
encoder 58. In addition, the disparity image generating
unit 302 generates disparity images Bl' and Cl' and
15 lowers the resolutions of the disparity images in
accordance with the horizontal half mode. Then, the
disparity image generating unit 302 multiplexes the
disparity images Bl' and Cl' of which the resolutions are
lowered, thereby acquiring a multiplexed image of which
20 the resolution is the same as that of the compatible
image. The disparity image generating unit 302 supplies
the multiplexed image that is acquired as a result
thereof to the encoder 58 and supplies information that
represents the conversion mode of the resolution the
25 disparity image of the auxiliary image to the disparity
image information generating unit 304.
[0218]
The image information generating unit 303 generates
the information representing the conversion mode of the
30 resolution of the auxiliary image and the like as image
information based on the information that is supplied
78
•
from the image converting unit 301. The image
information generating unit 303 supplies the image
information to the encoder 58.
[0219]
5 The disparity image information generating unit 304
generat~s information that represents the conversion mode
of the resolution of the disparity image of the auxiliary
image and the like as disparity image information based
on the information that is supplied from the disparity
10 image generating unit 302. The disparity image
information generating unit 304 supplies the disparity
image information to the encoder 58.
[0220]
[Example of Description of Disparity Image Information]
15 Fig. 29 is a diagram that illustrates an example of
the description of the disparity image information
included in the auxiliary stream in the encoding device
300 illustrated in Fig. 28 in a case where a mode that is
compliant with the AVC mode or the MVC mode is used as
20 the encoding mode.
[0221]
As illustrated in Fig. 29, as the disparity image
information (depth_map_structure), the number
(num_of_depth_map) of disparity images, a low-resolution
25 flag (is_decimated) that represents whether or not the
resolution of the disparity image has been lowered, a
conversion mode (decimation_mode) (conversion mode
information) of the resolution of the disparity image,
information (comb_ decimated_views) that designates a
30 disparity image of which the resolution has been lowered,
and the like are described. As conversion modes of the
79
•
resolution other than the horizontal half mode, there are
a vertical half mode in which the resolution in the
vertical direction is halved, a horizontal-vertical 1/4
mode in which the resolution in the horizontal direction
5 and the resolution in the vertical direction are halved,
and the like.
[0222]
In addition, in the encoding device 300, the image
information is the same as the disparity image
10 information except that the image information is not
information of the disparity image but information of the
compatible image and the auxiliary image, and thus the
description thereof will not be presented.
[0223]
15 [Description of Process of Encoding Device]
Figs. 30 and 31 are flowcharts that illustrate the
encoding process performed by the encoding device 300
illustrated in Fig. 28. This encoding process, for
example, is started when the images A1 to C1 are output
20 from the imaging units 51A to 51C.
[0224]
The encoding process illustrated in Figs. 30 and 31
is the same as the encoding process illustrated in Figs.
10 and 11 except that a conversion mode of the resolution
25 is determined instead of the multiplexing mode, and
information relating to the resolution lowering process
that includes the conversion mode is included in the
image information and the disparity image information.
[0225]
30 More specifically, the process of Steps S211 and
S212 illustrated in Fig. 30 is the same as the process of
80
•
Steps Sll and S12 illustrated in Fig. 10, and, in Step
S213, the image converting unit 301 determines the image
A1 supplied from the imaging unit 51A as a compatible
image and determines the conversion mode of the
5 resolution of the auxiliary image. The image converting
unit 301 supplies information that designates the image
A1 as a compatible image to the compatibility information
generating unit 55 and supplies the conversion mode of
the resolution of the auxiliary image to the image
10 information generating unit 303.
[0226]
The process of Steps S214 and S215 is the same as
the process of Steps S14 and S15 illustrated in Fig. 10.
In Step S216, the image converting unit 301 sets images
15 B1 and C1 other than the image A1 as auxiliary images and
lowers the resolutions of the auxiliary images based on
the conversion mode of the resolution of the auxiliary
image that is determined in Step S213 and multiplexes
resultant auxiliary images.
20 [0227]
The process of Steps S217 and S218 (Fig. 31) is the
same as the process of Steps S17 and S18 (Fig. 11), and,
in Step S219, the disparity image generating unit 302
determines the conversion mode of the resolution of the
25 disparity images of the auxiliary images and supplies
information that represents the conversion mode to the
disparity image information generating unit 304.
[0228]
The process of Step S220 (Fig. 31) is the same as
30 the process of Step S20 illustrated in Fig. 11, and, in
Step S221, the disparity image generating unit 302 lowers
81
•
the resolution of the disparity images of the auxiliary
images based on the conversion mode of the resolution of
the disparity image of the auxiliary image that is
determined in Step S219 and multiplexes resultant
5 disparity images.
[0229]
The process of Steps S222 to S226 is the same as
the process of Steps S22 to S26 illustrated in Fig. 11.
[0230]
10 As above, since the encoding device 300 lowers the
resolutions of the auxiliary image and the disparity
image, the information amount of the encoding result can
be suppressed without increasing the compression rate of
the compatible image. As a result, an increase in the
15 effect of the compression strain on the compatible image
due to an increase in the compression rate is prevented,
whereby the image quality of the compatible image can be
maintained to be high. This similarly applies to the
encoding devices 50, 140, and 180.
20 [0231]
[Configuration Example of Decoding Device]
Fig. 32 is a diagram that illustrates a
configuration example of a decoding device that decodes a
multiplexed stream that is transmitted from the encoding
25 device 300 illustrated in Fig. 28.
[0232]
In the configuration illustrated in Fig. 32, the
same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 12.
30 Duplicate description will not be presented as is
appropriate.
82
•
[0233]
The configuration of the decoding device 320
illustrated in Fig. 32 is mainly different from the
configuration illustrated in Fig. 12 in that an image
5 generating unit 321 is disposed instead of the image
generating unit 127. The decoding device 320 decodes a
mUltiplexed stream that is transmitted from the encoding
device 300, generates an image of one viewpoint or a
multi-viewpoint image, and displays the generated image
10 on a display device not illustrated in the figure.
[0234]
More specifically, the image generating unit 321 of
the decoding device 320 outputs the image in accordance
with an instruction made by a viewer, thereby displaying
15 the image on a display device not illustrated in the
figure. More specifically, the image generating unit 321
generates images of three or more viewpoints, which
correspond to a display device not illustrated in the
figure, each having the resolution that is the same as
20 the resolution of the compatible image or the auxiliary
image by using the compatible image, the multiplexed
image of the auxiliary images, the multiplexed image of
the disparity images of the compatible images, and the
multiplexed image of the disparity images of the
25 auxiliary images in accordance with a viewer's
instruction for displaying a 3D image of the multiviewpoint
mode based on the image information supplied
from the image information acquiring unit 123, the interviewpoint
distance information supplied from the inter-
30 viewpoint distance information acquiring unit 124, the
disparity image information supplied from the disparity
83
•
image information acquiring unit 125, the compatibility
information supplied from the compatibility information
acquiring unit 126, and the like.
[0235]
5 Described in detail, the image generating unit 321
(a separation unit and an image converting unit)
separates the disparity image of each auxiliary image
from the multiplexed image of the disparity images of the
auxiliary images and increases the resolution thereof
10 based on the conversion mode of the resolution that is
included in the disparity image information supplied from
the disparity image information acquiring unit 125. As a
result, the image generating unit 321 acquires the
disparity image of each auxiliary image having the same
15 resolution as the resolution of the compatible image. In
addition, the image generating unit 321 separates each
auxiliary image from the multiplexed image of the
auxiliary images, increases the resolution thereof, and
acquires each auxiliary image having the same resolution
20 as the resolution of the compatible image based on the
conversion mode of the resolution that is included in the
image information supplied from the image information
acquiring unit 123.
[0236]
25 Furthermore, the image generating unit 321
determines the position of each viewpoint of a multiviewpoint
image to be generated based on the interviewpoint
distance information and the number of
viewpoints that corresponds to a display device not
30 illustrated in the figure. Then, the image generating
unit 321 generates an image of each viewpoint of which
84
•
the position is determined by using the compatible image,
each auxiliary image having the same resolution as the
resolution of the compatible image, the disparity image
of the compatible image, and the disparity image of each
5 auxiliary image. Then, the image generating unit 321
converts the resolution of the generated image of each
viewpoint into resolution that is "1/ the number of
viewpoints" of the resolution of the compatible image or
the auxiliary image, synthesizes the converted images,
10 and displays a resultant image on a display device not
illustrated in the figure.
[0237]
At this time, the multi-viewpoint images after the
synthesis are displayed such that the viewing angles are
15 different for each viewpoint, and a viewer can view a 3D
image without wearing glasses by seeing images of
arbitrary two viewpoints with his left and right eyes.
[0238]
In addition, the image generating unit 321 outputs
20 an image A1 that is a compatible image supplied from the
compatible decoder 131 of the decoder 122 in accordance
with an instruction for displaying a 2D image that is
supplied from a viewer, thereby displaying the image on a
display device not illustrated in the figure.
25 Accordingly, the viewer can view the 2D image.
[0239]
[Description of Process of Decoding Device]
Fig. 33 is a flowchart that illustrates a decoding
process performed by the decoding device 320 illustrated
30 in Fig. 32. This decoding process, for example, is
started when a multiplexed stream that is transmitted
85
•
from the encoding device 300 illustrated in Fig. 28 is
input to the decoding device 320.
[0240]
The decoding process illustrated in Fig. 33 is the
5 same as the decoding process illustrated in Fig. 13
except that the resolutions of the auxiliary image and
the disparity image of the auxiliary image are increased
based on the image information and the disparity image
information that are included in the auxiliary stream.
10 [0241]
More specifically, the process of Steps S231 to
S239 illustrated in Fig. 33 is the same as the process of
Steps S31 to 839 illustrated in Fig. 13, and, in Step
S240, the image generating unit 321 separates the
15 multiplexed image of the auxiliary images based on the
conversion mode of the resolution that is included in the
image information supplied from the image information
acquiring unit 123 and increases the resolution of the
separated multiplexed image. In this way, the image
20 generating unit 321 acquires images B1 and C1 that are
auxiliary images having the resolution that is the same
as the compatible image.
[0242]
In Step S241, the image generating unit 321
25 separates the multiplexed image of the disparity images
of the auxiliary images based on the conversion mode of
the resolution that is included in the disparity image
information supplied from the disparity image information
acquiring unit 125 and increases the resolution of the
30 separated multiplexed image. In this way, the image
generating unit 321 acquires images B1' and C1' that are
86
•
the disparity images of the auxiliary images having the
resolution that is the same as the compatible image.
[0243]
The process of Step S242 is the same as the process
5 of Step S40 illustrated in Fig. 13, and, in Step S243,
the image generating unit 321 generates an image of each
viewpoint having the resolution that is the same as the
resolution of the compatible image or the auxiliary image
based· on the position of each viewpoint that is
10 determined in Step S242, the compatibility information
supplied from the compatibility information acquiring
unit 126, and the like by using the compatible image, the
auxiliary image having the resolution that is the same as
the resolution of the compatible image, the disparity
15 image of the compatible image, and the disparity image of
the auxiliary image.
[0244]
The process of Steps S244 to S246 is the same as
the process of Steps S42 to S44 illustrated in Fig. 13.
20 [0245]
In a decoding device that can decode only a
compatible stream that has compatibility with the
decoding device 320, the TS of the auxiliary stream is
ignored, and only the process of Steps S233 and S246 is
25 performed.
[0246]
As above, the decoding device 320 can decode a
compatible stream and an auxiliary stream that are
encoded in accordance with a mode having compatibility
30 with an existing mode by the encoding device 300.
[0247]
87
•
[Another Example of Arrangement of Disparity Image
Information]
In the description presented above, in the fourth
embodiment, although the compatibility information, the
5 disparity image information, the image information, and
the inter-viewpoint distance information are included in
the auxiliary stream, similarly to the first to third
embodiments, such information may be included in the TS.
In such a case, .for example, in descriptors of the PMT or
10 the SIT that is included in the packet of the TS of the
auxiliary stream, the compatibility information, the
disparity image information, the image information, and
the inter-viewpoint distance information are described.
[0248]
15 Fig. 34 is a diagram that illustrates an example of
the description of a descriptor
(depth_map_structure_descriptor) in which the disparity
image information is arranged in a case where the
compatibility information, the disparity image
20 information, the image information, and the interviewpoint
distance information are described in
descriptors of the PMT.
[0249]
As illustrated in Fig. 34, in the descriptor
25 (depth_map_structure_descriptor), following the
descriptor tag and the descriptor length, the number
(nurn_of_depth_map) of disparity images, a low-resolution
flag (is_decimated), a conversion mode (decimation_mode)
of the resolution of the disparity image, information
30 (comb_ decimated_views) that designates the disparity
image of which the resolution has been lowered, and the
88
•
like are described as the disparity image information.
[0250]
Although in the fourth embodiment, in the image
information and the disparity image information according
5 to the first embodiment, not the information relating to
the multiplexing but the information relating to the
resolution lowering is included, also in the image
information and the disparity image information according
to the second and third embodiments, not the information
10 relating to the multiplexing but the information relating
to resolution lowering may be included.
[0251]

[Configuration Example of Encoding Device According to
15 Fifth Embodiment]
Fig. 35 is a block diagram that illustrates a
configuration example of an encoding device according to
a fifth embodiment of the present technology.
[0252]
20 In the configuration illustrated in Fig. 35, the
same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 14.
Duplicate description will not be presented as is
appropriate.
25 [0253]
The configuration of the encoding device 440
illustrated in Fig. 35 is mainly different from the
configuration illustrated in Fig. 14 in that a disparity
image generating unit 441 and an encoder 442 are arranged
30 instead of the disparity image generating unit 143 and
the encoder 145. The encoding device 440 does not
89
•
multiplex the disparity images of compatible images in
the spatial direction but multiplexes the disparity
images of the compatible images in the time direction and
encodes the multiplexed disparity images.
5 [0254]
More specifically, the disparity image generating
unit 441 of the encoding device 440 detects the disparity
of each pixel of images A2to 02 using the images A2 to
02 that are supplied from the imaging units 141A to 1410.
10 The disparity image generating unit 441 generates a
disparity image A2' of the image A2 and a disparity image
B2' of the image B2, which are compatible images, and a
disparity image C2' of the image C2 and a disparity image
02' of the image 02, which are auxiliary images, based on
15 the detection result.
[0255]
In addition, the disparity image generating unit
441, similarly to the disparity image generating unit 143
illustrated in Fig. 14, multiplexes the disparity images
20 C2' and 02' in the spatial direction in accordance with a
predetermined multiplexing mode. In addition, the
disparity image generating unit 441 (multiplexing unit)
multiplexes disparity images A2' and B2' and a
multiplexed image of disparity images of the auxiliary
25 images in the time direction. The disparity image
generating unit 441 supplies a multiplexed image in which
disparity images A2' and B2' and the multiplexed image of
the disparity images of the auxiliary images
corresponding to one frame exist within one frame time,
30 which is acquired as a result thereof, to the encoder 442
as a time-multiplexed image.
90
•
[0256]
In addition, the disparity image generating unit
441 supplies information that represents a multiplexing
mode of the disparity images of the auxiliary images and
5 a multiplexing mode (hereinafter, referred to as a frame
sequential mode) in the time direction as a multiplexing
mode of the disparity images of the compatible images and
the auxiliary images to the disparity image information
generating unit 57.
10 [0257]
The encoder 442 is configured by a compatible
encoder 151 and an auxiliary encoder 451. The auxiliary
encoder 451 of the encoder 442 encodes the multiplexed
image of the auxiliary images that is supplied from the
15 image converting unit 142 and the time-multiplexed image
supplied from the disparity image generating unit 441 in
accordance with the 3DV mode. Here, the 3DV mode is a
mode that is used for encoding a display image in
accordance with a mode of multi-viewpoints that is
20 compliant with the AVe mode or the Mve mode. The
auxiliary encoder 451 supplies an encoded stream that is
acquired as a result of the encoding process to the
multiplexing unit 443 as an auxiliary stream.
[0258]
25 The multiplexing unit 443 generates TSs by using
the compatible stream supplied from the compatible
encoder 151, the auxiliary stream supplied from the
auxiliary encoder 451, the image information supplied
from the image information generating unit 54, the
30 compatibility information supplied from the compatibility
information generating unit 55, the inter-viewpoint
91
•
distance information supplied from the inter-viewpoint
distance information generating unit 56, the disparity
image information supplied from the disparity image
information generating unit 57, and the like. The
5 multiplexing unit 443 multiplexes the generated TSs and
transmits a multiplexed stream that is acquired as a
result of the multiplexing process.
[0259]
Hereinafter, the image information, the
10 compatibility information, the inter-viewpoint distance
information, and the disparity image information will be
collectively referred to as auxiliary information.
[0260]
[Example of Description of Disparity Image Information]
15 Fig. 36 is a diagram that illustrates an example of
the description of the disparity image information in the
encoding device 440 illustrated in Fig. 35 in a case
where the auxiliary information is described in a
descriptor of the PMT.
20 [0261]
As illustrated in Fig. 36, in a descriptor
(depth_map_structure_descriptor) in which the disparity
image information is arranged, similarly to the case of
Fig. 25, following a descriptor tag (descriptor_tag) and
25 a descriptor length (descriptor_length), the disparity
image information is described. In addition, as the
disparity image information, similarly to the case of Fig.
25, the number (num_of_depth_map) of disparity images, a
flag (is frame_packing) representing whether or not the
30 disparity images are multiplexed, a multiplexing mode
(frame_packing_mode) of the disparity images, information
92
•
(cornb_frame_packing_views) that designates the disparity
images that are multiplexed, and the like are described.
However, as the multiplexing mode of the disparity mode,
not only the side-by-side mode (SBS) or the top-and-
5 bottom mode (TOB) but also the frame-sequential mode is
described.
[0262]
In the encoding device 440 illustrated in Fig. 35,
although the auxiliary information is included in the TS,
10 the auxiliary information may be included in the
auxiliary stream.
[0263]
Fig. 37 is a diagram that illustrates an example of
the description of the disparity image information
15 included in the auxiliary stream in a case where the
auxiliary information is included in the auxiliary stream.
[0264]
In such a case, as illustrated in Fig. 6, the
disparity image information (depth_map_structure), for
20 example, is arranged in the SEI that is included in the
auxiliary stream.
[0265]
In addition, as the disparity image information
(depth_map_structure), as illustrated in Fig. 37, the
25 number (nurn_of_depth_map) of disparity images (depth map),
a flag (is_frame_packing) representing whether or not the
disparity images are multiplexed, a multiplexing system
(frame_packing_mode) of the disparity images, information
(cornb_frame_packing_views) that designates the disparity
30 images that are multiplexed, and the like are described.
However, as the multiplexing mode, similarly to the case
93
•
of Fig. 36, not only the side-by-side mode (SBS) or the
top-and-bottom mode (TOB), but also a frame-sequential
mode is described.
[0266]
5 Although not illustrated in the figure, the image
information is the same as the disparity image
information except that the image information is not a
disparity image but information of the compatible images
and the auxiliary images.
10 [0267]
[Description of Process of Encoding Device]
Figs. 38 and 39 are flowcharts that illustrate an
encoding process performed by the encoding device 440
illustrated in Fig. 35. This encoding process, for
15 example, is started when images A2 to D2 are output from
imaging units 141A to 141D.
[0268 ]
The process of Steps S451 to S455 illustrated in
Figs. 38 and 39 is the same as the process of Steps S51
20 to S55 illustrated in Figs. 15 and 16, and thus the
description thereof will not be presented.
[0269]
In Step S456, the image information generating unit
54 generates information that represents the multiplexing
25 modes of the compatible images and the auxiliary images
and the like as image information based on the
information supplied from the image converting unit 142
and inputs the generated image information to the
multiplexing unit 443. Then, the process proceeds to
30 Step S457.
[0270]
94
15
•
The process of Steps S457 to S459 is the same as
the process of Steps S57 to S59 illustrated in Figs. 15
and 16, and thus the description thereof will not be
presented.
5 [0271]
In Step S460 illustrated in Fig. 39, the disparity
image generating unit 441 determines the multiplexing
mode of the disparity images of the auxiliary images and
the multiplexing mode of the multiplexed image of the
10 disparity images of the compatible images and the
disparity images of the auxiliary image and supplies
information that represents the multiplexing modes to the
disparity image information generating unit 57.
[0272]
In Step S461, the disparity image information
generating unit 57 generates information that represents
the multiplexing mode of the disparity images of the
auxiliary images and the multiplexing mode of the
multiplexed image of the disparity images of the
20 compatible images and the disparity images of the
auxiliary images and the like as disparity image
information based on the information that is supplied
from the disparity image generating unit 441 and inputs
the generated disparity image information to the
25 multiplexing unit 443.
[0273]
In Step S462, the disparity image generating unit
441 multiplexes the disparity images C2' and D2' of the
auxiliary images in the spatial direction and multiplexes
30 the disparity images A2' and B2' of the compatible images
and the multiplexed image of the disparity images of the
95
•
auxiliary images in the time direction, based on the
multiplexing mode that is determined in Step S460.
[0274]
In Step S463, the disparity image generating unit
5 441 inputs the time-multiplexed image that is acquired as
a result of the multiplexing process of Step S462 to the
encoder 442.
[0275]
In Step S464, the compatible encoder 151 of the
10 encoder 442 encodes the multiplexed image of the
compatible images that is supplied from the image
converting unit 142 in accordance with the existing Ave
mode and supplies an encoded stream that is acquired as a
result thereof to the multiplexing unit 443 as a
15 compatible stream.
[0276]
In Step S465, the auxiliary encoder 451 encodes the
multiplexed image of the auxiliary images that is
supplied from the image converting unit 142 and the time-
20 multiplexed image that is supplied from the disparity
image generating unit 441 in accordance with the 3DV mode.
The auxiliary encoder 451 supplies an encoded stream that
is acquired as a result of the encoding process to the
multiplexing unit 443 as an auxiliary stream.
25 [0277]
In Step S466, the multiplexing unit 443 generates
TSs from the compatible stream supplied from the
compatible encoder 151 and the auxiliary stream and the
auxiliary information that are supplied from the
30 auxiliary encoder 451, performs a multiplexing process
thereof, and transmits a multiplexed stream acquired as a
96
•
result thereof. This multiplexed stream, for example, is
recorded on a BD or the like or is transmitted as a
broadcast stream. Then, the process ends.
[0278 ]
5 [Configuration Example of Decoding Device]
Fig. 40 is a diagram that illustrates a
configuration example of a decoding device that decodes a
multiplexed stream that is transmitted from the encoding
deviCe 440 illustrated in Fig. 35.
10 [0279]
In the configuration illustrated in Fig. 40, the
same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 17.
Duplicate description will not be presented as is
15 appropriate.
[0280 ]
The configuration of the decoding device 460
illustrated in Fig. 40 is mainly different from the
configuration illustrated in Fig. 17 in that a separation
20 unit 461, a decoder 462, and an image generating unit 463
are disposed instead of the separation unit 121, the
decoder 122, and the image generating unit 171. The
decoding device 460 generates images of two viewpoints or
multi-viewpoint images by decoding the multiplexed stream
25 that is transmitted from the encoding device 440 and
displays the generated images on a display device not
illustrated in the figure.
[0281]
More specifically, the separation unit 461 of the
30 decoding device 460 receives a multiplexed stream that is
transmitted from the encoding device 440 and separates
97
•
each TS. The separation unit 461 extracts a compatible
stream and an auxiliary stream from the separated TS and
supplies the extracted streams to the decoder 462. In
addition, the separation unit 461 extracts auxiliary
5 information that is included in the TS, supplies image
information included in the auxiliary information to the
image information acquiring unit 123, and supplies
viewpoint distance information to the inter-viewpoint
distance information acquiring unit 124. Furthermore,
10 the separation unit 461 supplies disparity image
information included in the auxiliary information to the
disparity image information acquiring unit 125 and
supplies compatibility information to the compatibility
information acquiring unit 126 and the decoder 462.
15 [0282]
The decoder 462 is configured by a compatible
decoder 471 and an auxiliary decoder 472. The compatible
decoder 471 of the decoder 462 identifies the compatible
stream out of the compatible stream and the auxiliary
20 stream supplied from the separation unit 461 based on the
compatibility information that is supplied from the
separation unit 461. The compatible decoder 471 decodes
the encoded compatible image included in the compatible
stream in accordance with a mode that corresponds to the
25 Ave mode and supplies a multiplexed image of the
compatible images acquired as a result of the decoding
process to the image generating unit 463.
[0283]
The auxiliary decoder 472 identifies the auxiliary
30 stream out of the compatible stream and the auxiliary
stream that are supplied from the separation unit 461
98
•
based on the compatibility information that is supplied
from the separation unit 461. The auxiliary decoder 472
decodes the multiplexed image of auxiliary images and the
time-multiplexed image, which are encoded, included in
5 the auxiliary stream supplied from the separation unit
461 in accordance with a mode that corresponds to the
auxiliary encoder 451 illustrated in Fig. 35. The
auxiliary decoder 472 supplies the multiplexed image of
the auxiliary images and the time-multiplexed image that
10 are acquired as a result of the decoding process to the
image generating unit 463.
[0284]
The image generating unit 463 outputs an image in
accordance with an instruction supplied from a viewer and
15 displays the image on a display device not illustrated in
the figure. Described in more detail, the image
generating unit 463 generates images of three or more
viewpoints, which correspond to a display device not
illustrated in the figure, each having the resolution
20 that is a half of the resolution of the compatible image
or the auxiliary image by using the multiplexed image of
compatible images, the multiplexed image of the auxiliary
images, and the time-multiplexed image in accordance with
a viewer's instruction for displaying a 3D image of the
25 multi-viewpoint mode based on the image information
supplied from the image information acquiring unit 123,
the inter-viewpoint distance information supplied from
the inter-viewpoint distance information acquiring unit
124, the disparity image information supplied from the
30 disparity image information acquiring unit 125, the
compatibility information supplied from the compatibility
99
•
information acquiring unit 126, and the like.
[0285]
Described in more detail, the image generating unit
463 (separation unit) separates disparity images A2' and
5 B2' that are the disparity images of the compatible
images and the multiplexed image of the disparity images
of the auxiliary images from the time-multiplexed image
based on the information representing the frame
sequential mode as the multiplexing mode of the
10 multiplexed image of the disparity images of the
compatible images and the disparity images of the
auxiliary images included in the disparity image
information that is supplied from the disparity image
information acquiring unit 125. Then, the image
15 generating unit 463 separates the disparity images C2'
and D2' from the multiplexed image of the disparity
images of the auxiliary images based on the multiplexing
mode of the disparity images of the auxiliary images that
is included in the disparity image information.
20 [0286]
In addition, the image generating unit 463,
similarly to the image generating unit 171 illustrated in
Fig. 17, separates each auxiliary image from the
multiplexed image of the auxiliary images based on the
25 information that represents the multiplexing mode of the
auxiliary images that is included in the image
information supplied from the image information acquiring
unit 123. Furthermore, the image generating unit 463,
similarly to the image generating unit 171, separates
30 each compatible image from the multiplexed image of the
compatible images based on the information that
100 ;1
,:i
:1
1I
I
•
represents the multiplexing mode of the compatible images
that is included in the image information.
[0287]
Furthermore, the image generating unit 463,
5 similarly to the image generating unit 171, determines
the position of each viewpoint of multi-viewpoint images
to be generated based on the inter-viewpoint distance
information and the number of viewpoints corresponding to
a display device not illustrated in the figure. Then,
10 the image generating unit 463, similarly to the image
generating unit 171, generates the image of each
viewpoint of which the position is determined by using
each compatible image, each auxiliary image, the
disparity image of each compatible image, and the
15 disparity image of each auxiliary image. Then, the image
generating unit 463, similarly to the image generating
unit 171, converts the resolution of the generated multiviewpoint
images into resolution that is ~1 / the number
of viewpoints" of the resolution of the compatible image
20 or the auxiliary image, synthesizes the images, and
displays the synthesized image on a display device not
illustrated in the figure.
[0288]
At this time, the multi-viewpoint images after the
25 synthesis are displayed such that the viewing angles are
different for each viewpoint, and a viewer can view a 3D
image without wearing glasses by seeing images of
arbitrary two viewpoints with his left and right eyes.
[0289]
30 In addition, the image generating unit 463,
similarly to the image generating unit 171 separates the
101
•
multiplexed image of the compatible images that is
supplied from the decoder 462 into images A2 and B2 of
which the resolution is a half of the resolution of the
compatible image based on the image information supplied
5 from the image information acquiring unit 123 in
accordance with an instruction for displaying a 3D image
of the two-viewpoint mode that is supplied from a viewer.
Then, the image generating unit 463, similarly to the
image generating unit 171, alternately outputs the
10 separated images A2 and B2 of which the resolution is a
half of the resolution of the compatible image, thereby
displaying the separated images on a display device not
illustrated in the figure. At this time, the viewer can
view a 3D image by wearing glasses in which one a left-
15 eye shutter and a right-eye shutter is open at the time
of displaying the image A2, and the other is open at the
time of displaying the image B2 and viewing the images A2
and B2 that are alternately displayed on the display
device.
20 [0290]
[Description of Decoding Process]
Fig. 41 is a flowchart that illustrates a decoding
process performed by the decoding device 460 illustrated
in Fig. 40. This decoding process, for example, is
25 started when a multiplexed stream that is transmitted
from the encoding device 440 illustrated in Fig. 35 is
input to the decoding device 460.
[0291]
In Step S471 illustrated in Fig. 41, the separation
30 unit 461 of the decoding device 460 receives the
multiplexed stream that is transmitted from the encoding
102
•
device 50 and separates each TS. The separation unit 461
supplies the compatible stream and the auxiliary stream
that are included in the separated TS to the decoder 462.
In addition, the separation unit 461 extracts the
5 auxiliary information included in the TS, supplies the
image information included in the auxiliary information
to the image information acquiring unit 123, and supplies
the viewpoint distance information to the inter-viewpoint
distance information acquiring unit 124. Furthermore,
10 the separation unit 461 supplies the disparity image
information that is included in the auxiliary information
to the disparity image information acquiring unit 125 and
supplies the compatibility information to the
compatibility information acquiring unit 126 and the
15 decoder 462.
[0292]
In Step S472, the compatible decoder 471 identifies
the compatible stream out of the compatible stream and
the auxiliary stream that are supplied from the
20 separation unit 461 based on the compatibility
information that is supplied from the separation unit 461.
[0293]
In Step S473, the compatible decoder 471 decodes
the compatible image that is included in the compatible
25 stream in accordance with a mode that corresponds to the
Ave mode and supplies a multiplexed image of the
compatible images that is acquired as a result of the
decoding process to the image generating unit 463.
[0294]
30 In Step S474, the image generating unit 463
determines whether or not an instruction for displaying a
103
•
3D image of the two-viewpoint mode has been made by a
viewer. In a case where it is determined that the
instruction for displaying a 3D image of the twoviewpoint
mode has not been made by the viewer in Step
5 S474, in other words, in a case where an instruction for
displaying a 3D image of the multi-viewpoint mode has
been made, the auxiliary decoder 472 identifies the
auxiliary stream out of the compatible stream and the
auxiliary stream that are supplied from the separation
10 unit 461 based on the compatibility information.
[0295]
Then, in Step S475, the auxiliary decoder 472
extracts a multiplexed image of the auxiliary images and
a time-multiplexed image, which are encoded, from the
15 auxiliary stream and decodes the extracted multiplexed
images in accordance with a mode that corresponds to the
auxiliary encoder 62 illustrated in Fig. 2. The
auxiliary decoder 472 supplies the multiplexed image of
the auxiliary images and the time-multiplexed image that
20 are acquired as a result of the decoding process to the
image generating unit 127, and the process proceeds to
Step S476.
[0296]
In Step S476, the image information acquiring unit
25 123 acquires the image information that is supplied from
the separation unit 461 and inputs the acquired image
information to the image generating unit 463. In Step
S477, the inter-viewpoint distance information acquiring
unit 124 acquires the inter-viewpoint distance
30 information that is supplied from the separation unit 461
and inputs the acquired inter-viewpoint distance
104
•
information to the image generating unit 463.
[0297 ]
In Step S478, the disparity image information
acquiring unit 125 acquires the disparity image
5 information that is supplied from the separation unit 461
and inputs the acquired disparity image information to
the image generating unit 463. In Step S479, the
compatibility information acquiring unit 126 acquires the
compatibility information that is supplied from the
10 separation unit 461 and inputs the acquired compatibility
information to the image generating unit 463.
[0298 ]
In Step S480, the image generating unit 463
determines the position of each viewpoint of a multi-
15 viewpoint image to be generated based on the interviewpoint
distance information supplied from the interviewpoint
distance information acquiring unit 124 and the
number of viewpoints that corresponds to a display device
not illustrated in the figure.
20 [0299]
In Step S481, the image generating unit 463
generates images of the viewpoints each having the
resolution that is a half of the resolution of the
compatible image or the auxiliary image based on the
25 position of each viewpoint determined in Step S480, the
image information supplied from the image information
acquiring unit 123, the disparity image information
supplied from the disparity image information acquiring
unit 125, the compatibility information supplied from the
30 compatibility information acquiring unit 126, and the
like by using the multiplexed image of the compatible
105
r
•
images, the multiplexed image of the auxiliary images,
and the time-multiplexed image.
[0300 ]
Steps S482 and S483 are the same as Steps S92 and
5 S93 illustrated in Fig. 18, and thus the description
thereof will not be presented.
[0301]
In a case where it is determined that the
instruction for displaying'a 3D image of the two-
10 viewpoint mode has been made by a viewer in Step S474, in
Step S484, the image information acquiring unit 123
acquires the image information that is supplied from the
separation unit 461 and inputs the acquired image
information to the image generating unit 463.
15 [0302]
The process of Steps S485 and S486 is the same as
the process of Steps S94 and S95 illustrated in Fig. 18,
and thus the description thereof will not be presented.
[0303]
20 In addition, in a decoding device that can decode
only a compatible stream that has compatibility with the
decoding device 460, the auxiliary stream is ignored, and
only the process of Steps S473, S484, and S485 is
performed. However, in such a case, in the process of
25 Step S484, a multiplexed image of the compatible images
is separated based on the multiplexing mode determined in
advance.
[0304]

30 [Configuration Example of Encoding Device According to
Sixth Embodiment]
106
•
Fig. 42 is a block diagram that illustrates a
configuration example of an encoding device according to
a sixth embodiment of the present technology.
[0305]
5 In the configuration illustrated in Fig. 42, the
same reference numeral i~ assigned to a configuration
that is the same as that illustrated in Fig. 35.
Duplicate description will not be presented as is
appropriate.
10 [0306]
The configuration of the encoding device 480
illustrated in Fig. 42 is mainly different from the
configuration illustrated in Fig. 35 in that a disparity
image generating unit 481, an encoder 482, and a
15 disparity image information generating unit 483 are
disposed instead of the disparity image generating unit
143, the encoder 145, and the disparity image information
generating unit 57. The encoding device 480 encodes a
common disparity image that shows a disparity value that
20 is common to the compatible images and a common disparity
image of the auxiliary images.
[0307]
More specifically, the disparity image generating
unit 481 of the encoding device 480 detects the disparity
25 of each pixel between images A2 and B2 and the disparity
of each pixel between images C2 and D2 using the images
A2 to D2 that are supplied from the imaging units 141A to
141D. The disparity image generating unit 481 generates
a disparity image formed by disparity values that
30 represent the disparity of each pixel between the images
A2 and B2 as a common disparity image AB2' of the
107
•
compatible images and supplies the generated common
disparity image to the encoder 482. In addition, the
disparity image generating unit 481 generates a disparity
image formed by disparity values that represent the
5 disparity of each pixel between the images C2 and 02 as a
common disparity image C02' of the auxiliary images and
supplies the generated common disparity image to the
encoder 482.
[0308]
10 Furthermore, the disparity image generating unit
481 supplies information that represents that the
disparity images of the compatible images and the
auxiliary images are common disparity images to the
disparity image information generating unit 483.
15 [0309]
The encoder 482 is configured by a compatible
encoder 151 and an auxiliary encoder 491. The auxiliary
encoder 491 encodes the multiplexed image of auxiliary
images that is supplied from the image converting unit
20 142 and the common disparity image AB2' of the compatible
images and the common disparity image C02' of the
auxiliary images that are supplied from the disparity
image generating unit 481 in accordance with the 30V mode.
The auxiliary encoder 491 supplies an encoded stream
25 acquired as a result thereof to the multiplexing unit 443
as an auxiliary stream.
[0310]
The disparity image information generating unit 483
generates information representing that the disparity
30 images of the compatible images and the auxiliary images
are common disparity images and the like as disparity
108
•
image information based on the information that is
supplied from the disparity image generating unit 53 and
supplies the generated disparity image information to the
multiplexing unit 443.
5 [0311]
[Description of Process of Encoding Device]
Figs. 43 and 44 represent a flowchart that
illustrates an encoding process that is performed by the
encoding device 480 illustrated i~ Fig.42~ This
10 encoding process, for example, is started when images A2
to D2 are output from the imaging units 141A to 141D.
[0312]
The process of Steps S491 to S498 illustrated in
Fig. 43 is similar to that of Steps S451 to S458
15 illustrated in Fig. 38, and thus, the description thereof
will not be presented.
[0313]
In Step S499 illustrated in Fig. 44, the disparity
image generating unit 481 detects the disparity of each
20 pixel between images A2 and B2 and the disparity of each
pixel between images C2 and D2 using the images A2 to D2
that are supplied from the imaging units 141A to 141D.
Then, the disparity image generating unit 481 generates a
common disparity image AB2' that is formed by disparity
25 values that represent the disparity of each pixel between
the images A2 and B2 and a common disparity image CD2'
that is formed by disparity values that represent the
disparity of each pixel between the images C2 and D2.
[0314]
30 In Step S500, the disparity image information
generating unit 483 generates information representing
109
•
that the disparity images of the compatible images and
the auxiliary images are common disparity images and the
like as disparity image information based on the
information that is supplied from the disparity image
5 generating unit 481 and inputs the generated disparity
image information to the multiplexing unit 443.
[0315]
In Step S501, the disparity image generating unit
48i inputs the common disparity images CD2' of the
10 auxiliary images and the common disparity images AB2' of
the compatible images, which are generated in Step S499,
to the encoder 482.
[0316]
In Step S502, the compatible encoder 151 of the
15 encoder 482 encodes the multiplexed image of the
compatible images that is supplied from the image
converting unit 142 in accordance with the existing AVC
mode and supplies an encoded stream acquired as a result
thereof to the multiplexing unit 443 as a compatible
20 stream.
[0317]
In Step S503, the auxiliary encoder 491 encodes the
multiplexed image of the auxiliary images supplied from
the image converting unit 142 and the common disparity
25 image of the compatible images and the common disparity
image of the auxiliary images that are supplied from the
disparity image generating unit 481 in accordance with
the 3DV mode. The auxiliary encoder 491 supplies an
encoded stream acquired as a result of the encoding
30 process to the multiplexing unit 443 as an auxiliary
stream.
110
•
[0318]
In Step S504, the multiplexing unit 443 generates
TSs from the compatible stream that is supplied from the
compatible encoder 151 and the auxiliary stream and the
5 auxiliary information that are supplied from the
auxiliary encoder 491, performs a multiplexing process
thereof, and transmits a multiplexed stream acquired as a
result thereof. This multiplexed stream, for example, is
recorded on a BD or the like or is transmitted as a
10 broadcast stream. Then, the process ends.
[0319]
[Configuration Example of Decoding Device]
Fig. 45 is a diagram that illustrates a
configuration example of a decoding device that decodes
15 the multiplexed stream transmitted from the encoding
device 480 illustrated in Fig. 42.
[0320]
In the configuration illustrated in Fig. 45, the
same reference numeral is assigned to a configuration
20 that is the same as that illustrated in Fig. 40.
Duplicate description will not be presented as is
appropriate.
[0321]
The configuration of the decoding device 500
25 illustrated in Fig. 45 is mainly different from the
configuration illustrated in Fig. 40 in that a decoder
501 and an image generating unit 502 are disposed instead
of the decoder 122 and the image generating unit 171.
The decoding device 500 generates images of two
30 viewpoints or multi-viewpoint images by decoding the
multiplexed stream that is transmitted from the encoding
111
•
device 480 and displays the generated images on a display
device not illustrated in the figure.
[0322]
More specifically, the decoder 501 of the decoding
5 device 500 is configured by a compatible decoder 471 and
an auxiliary decoder 511. The auxiliary decoder 511 of
the decoder 501 identifies the compatible stream out of
the compatible stream and the auxiliary stream that are
supplied from the separation unit 461 based on the
10 compatibility information that is supplied from the
separation unit 461. In addition, the auxiliary decoder
511 decodes the multiplexed image of auxiliary images
that is included in the auxiliary stream, the common
disparity image AB2' of the compatible images, and the
15 common disparity image CD2' of the auxiliary images in
accordance with a mode that corresponds to the auxiliary
encoder 491 illustrated in Fig. 42. The auxiliary
decoder 511 supplies the multiplexed image of the
auxiliary images and the common disparity images AB2' and
20 CD2' that are acquired as a result of the decoding
process to the image generating unit 502.
[0323]
The image generating unit 502 outputs an image in
accordance with an instruction supplied from a viewer and
25 displays the image on a display device not illustrated in
the figure. When described in more detail, the image
generating unit 502 generates images of three or more
viewpoints, which correspond to a display device not
illustrated in the figure, each having the resolution
30 that is a half of the resolution of the compatible image
or the auxiliary image by using the multiplexed image of
112
•
compatible images, the multiplexed image of the auxiliary
images, and the common disparity images AB2' and CD2' in
accordance with a viewer's instruction for displaying a
3D image of the multi-viewpoint mode based on the image
5 information supplied from the image information acquiring
unit 123, the inter-viewpoint distance information
supplied from the inter-viewpoint distance information
acquiring unit 124, the disparity image information
supplied from the disparity image information acquiring
10 unit 125, the compatibility information supplied from the
compatibility information acquiring unit 126, and the
like.
[0324]
Described in more detail, the image generating unit
15 502 maintains the common disparity images AB2' and CD2'
without being changed based on the information
representing that the disparity images of the compatible
images and the auxiliary images are common disparity
images that is included in the disparity image
20 information supplied from the disparity image information
acquiring unit 125.
[0325]
In addition, the image generating unit 502,
similarly to the image generating unit 171 illustrated in
25 Fig. 17, separates each auxiliary image from the
multiplexed image of the auxiliary images based on the
information that represents the multiplexing mode of the
auxiliary images that is included in the image
information supplied from the image information acquiring
30 unit 123. Furthermore, the image generating unit 502,
similarly to the image generating unit 171, separates
113
•
each compatible image from the multiplexed image of the
compatible images based on the information that
represents the multiplexing mode of the compatible images
that is included in the image information.
5 [0326]
Furthermore, the image generating unit 502,
similarly to the image generating unit 171, determines
the position of each viewpoint of the multi-viewpoint
images to be generated based on the inter-viewpoint
10 distance information and the number of viewpoints that
corresponds to a display device not illustrated in the
figure. Then, the image generating unit 502 generates
the image of each viewpoint of which the position is
determined by using each compatible image, each auxiliary
15 image, the common disparity image AB2', and the common
disparity image CD2'. Then, the image generating unit
502, similarly to the image generating unit 171, converts
the resolution of the generated multi-viewpoint images
into resolution that is "lithe number of viewpoints" of
20 the resolution of the compatible image or the auxiliary
image, synthesizes the images, and displays the
synthesized image on a display device not illustrated in
the figure.
[0327]
25 At this time, the multi-viewpoint images after the
synthesis are displayed such that the viewing angles are
different for each viewpoint, and a viewer can view a 3D
image without wearing glasses by seeing images of
arbitrary two viewpoints with his left and right eyes.
30 [0328]
In addition, the image generating unit 502,
114
•
similarly to the image generating unit 171, separates
from the multiplexed image of the compatible images that
is supplied from the decoder 501 into images A2 and B2 of
which the resolution is a half of the resolution of the
5 compatible image based on the image information supplied
from the image information acquiring unit 123 in
accordance with an instruction for displaying a 3D image
of the two-viewpoint mode that is supplied from a viewer.
Then, the image generating unit 502, similarly to the
10 image generating unit 171, alternately outputs the
separated images A2 and B2 of which the resolution is a
half of the resolution of the compatible image, thereby
displaying the separated images on a display device not
illustrated in the figure. At this time, the viewer can
15 view a 3D image by wearing glasses in which one a lefteye
shutter and a right-eye shutter is open at the time
of displaying the image A2, and the other is open at the
time of displaying the image B2 and viewing the images A2
and B2 that are alternately displayed on the display
20 device.
[0329]
[Description of Process of Decoding Device]
Fig. 46 is a flowchart that illustrates a decoding
process performed by the decoding device 500 illustrated
25 in Fig. 45. This decoding process, for example, is
started when a multiplexed stream that is transmitted
from the encoding device 480 illustrated in Fig. 42 is
input to the decoding device 500.
[0330]
30 The process of Steps S511 to S514 illustrated in
Fig. 46 is similar to that of Steps S471 to S474
115
•
illustrated in Fig. 41, and thus, the description thereof
will not be presented.
[0331]
In Step S515, the auxiliary decoder 511 extracts
5 the multiplexed image of the auxiliary images, the common
disparity image AB2' of the compatible images, and the
common disparity image CD2' of the auxiliary images,
which are encoded, and decodes the extracted images in
accordance with a mode that corresponds to the auxiliary
10 encoder 491 illustrated in Fig. 42. The auxiliary
decoder 511 supplies the multiplexed image of the
auxiliary images, the common disparity image AB2', and
the common disparity image C02', which are acquired as a
result of the decoding process, to the image generating
15 unit 502.
[0332]
The process of Steps S516 to S520 is similar to
that of Steps S476 to S480 illustrated in Fig. 41, and
thus, the description thereof will not be presented.
20 [0333]
After the process of Step S520, in Step S521, the
image generating unit 502 generates an image of each
viewpoint of which the resolution is a half of the
resolution of the compatible image or the auxiliary image
25 by using the compatible images, the multiplexed image of
the auxiliary images, the common disparity image AB2',
and the common disparity image C02' based on the position
of each viewpoint determined in Step S519, the image
information supplied from the image information acquiring
30 unit 123, the disparity image information supplied from
the disparity image information acquiring unit 125, and
116
•
the compatibility information supplied from the
compatibility information acquiring unit 126, and the
like.
[0334]
5 The process of Steps S522 to S526 is similar to
that of Steps S482 to S486 illustrated in Fig. 41, and
thus, the description thereof will not be presented.
[0335]
Although the encoding devices 440 and 480,
10 similarly to the encoding device 140, multiplex and
encode compatible images of two view points, similarly to
the encoding device 180 illustrated in Fig. 19, the
encoding devices 440 and 480 may encode compatible images
of two viewpoints without multiplexing them. In addition,
15 the encoding devices 440 and 480, similarly to the
encoding device 50 illustrated in Fig. 5, may encode a
compatible image of one viewpoint.
[0336]
In addition, the encoding devices 140 and 180 may
20 encode the disparity images of the compatible images and
the auxiliary images without multiplexing the abovedescribed
disparity images. Furthermore, the encoding
device 50 may encode the disparity images of the
auxiliary images without multiplexing the above-described
25 disparity images.
[0337]

Fig. 47 is a diagram that illustrates an example of
a multiplexing pattern of an encoding target in a case
30 where the number of viewpoints of the compatible images
is two, and the number of viewpoints of the auxiliary
117
•
images is two.
[0338]
As illustrated in (1) of Fig. 47, the encoding
device 140 illustrated Fig. 14 multiplexes images A2 and
5 B2 that are compatible images in the spatial direction
and encodes the multiplexed image in accordance with the
AVC mode. In addition, the encoding device 140
multiplexes the images C2 and D2 that are auxiliary
images, the disparity imagesA2' andB2' that are
10 disparity images of the compatible images, and the
disparity images C2' and D2' that are disparity images of
the auxiliary images in the spatial direction and encodes
the multiplexed image in accordance with the 3DV mode
that is compliant with the MVC mode.
15 [0339]
Furthermore, as illustrated in (2) of Fig. 47, the
encoding device 140 may encode the disparity images A2'
to D2' in accordance with the 3DV mode that is compliant
with the MVC mode without multiplexing the disparity
20 images. In addition, as illustrated in (3) of Fig. 47,
the encoding device 480 illustrated in Fig. 42 may encode
the common disparity image AB2' instead of the disparity
images A2' and B2' that are disparity images of the
compatible images and encode the common disparity image
25 CD2' instead of the disparity images C2' and D2' that are
disparity images of the auxiliary images.
[0340]
Furthermore, as illustrated in (4) of Fig. 47, the
encoding device 440 illustrated in Fig. 35 multiplexes
30 the disparity images A2' and B2' and the multiplexed
image of the disparity images of the auxiliary images in
118
•
accordance with the frame sequential mode without
multiplexing the disparity images A2' and B2' that are
disparity images of the compatible images in the spatial
direction and encodes the multiplexed image in accordance
5 with the 3DV mode that is compliant with the MVC mode.
[0341]
As illustrated in (5) of Fig. 47, the encoding
device 180 illustrated in Fig. 19 encodes the image A2 in
accordance with the AVC mode and encodes· the image B2
10 with the image A2 set as a base view in accordance with
the MVC mode without multiplexing the images A2 and B2
that are compatible images. In addition, the encoding
device 180 encodes the images C2 and D2 that are
auxiliary images, the disparity images A2' and B2' that
15 are the disparity images of the compatible images, and
the disparity images C2' and D2' that are the disparity
images of the auxiliary images in the spatial direction
and encodes a resultant multiplexed image in accordance
with a 3DV mode that is in compliant with the MVC mode.
20 [0342]
In addition, as illustrated in (6) of Fig. 47, the
encoding device 180 may encode the disparity images A2'
to D2' in accordance with the 3DV mode that is compliant
with the MVC mode without multiplexing them. Furthermore,
25 as illustrated in (7) of Fig. 47, similarly to the
encoding device 480, the encoding device 180 may encode
the common disparity image AB2' instead of the disparity
images A2' and B2' and encode the common disparity image
CD2' instead of the disparity images C2' and D2' .
30 [0343]
Furthermore, as illustrated in (8) of Fig. 47, the
119
•
encoding device 180, similarly to the encoding device 440,
may multiple the disparity images A2' and B2' and the
multiplexed image of the disparity images of the
auxiliary images in accordance with the frame sequential
5 mode and encode a resultant multiplexed image in
accordance with the 3DV mode that is compliant with the
MVC mode without multiplexing the disparity images A2'
and B2' in the spatial direction.
[0344]
10 In addition, the multiplexing pattern is not
limited to the example illustrated in Fig. 47. For
example, the multiplexed image of the auxiliary images,
the disparity image of the compatible image, and the
multiplexed image of the disparity images of the
15 auxiliary images may be further multiplexed in accordance
with a frame sequential mode.
[0345]
Fig. 48 is a diagram that illustrates features of
the effects of multiplexing in the multiplexing patterns
20 illustrated in (1) to (8) of Fig. 47.
[0346]
In the table represented in Fig. 48, items of the
effects "Compatibility", "Image Quality", and "Data
Amount" are arranged, and the degrees of the effects of
25 the items for the multiplexing patterns illustrated in
(1) to (8) of Fig. 47 are represented. In the table
represented in Fig. 48, a circle represents the presence
of an effect, and a double circle represents a remarkable
effect.
30 [0347]
In a case where multiplexing is performed using the
120
•
multiplexing pattern illustrated in (1) of Fig. 47, the
multiplexing mode and the encoding mode of compatible
images are the same as those of an existing mode, whereby
the compatibility is secured. In addition, since
5 disparity images, similarly to the compatible images, are
multiplexed in the spatial direction, for example, the
disparity images can be separated using a separation unit,
which separates compatible images; prepared on the
decoding device side. Accordingly, it is assured that
10 the disparity images can be separated on the decoding
device side. Therefore, in this case, there is a
remarkable effect on the compatibility, and a double
circle is described in correspondence with the item
"Compatibility" illustrated in Fig. 48.
15 [0348]
In a case where multiplexing is performed using the
multiplexing pattern illustrated in (2) of Fig. 47, the
multiplexing mode and the encoding mode of compatible
images are the same as those of an existing mode, whereby
20 the compatibility is secured. In addition, since the
resolution of the disparity image is the same as that of
the image before being multiplexed, the accuracy is high.
As a result, in the decoding device, the accuracy of an
image of a predetermined viewpoint that is generated
25 using the disparity image is improved. Therefore, in
this case, there are effects on the compatibility and the
image quality of an image that is generated using the
disparity image, and accordingly, circles are described
in correspondence with items "Compatibility" and "Image
30 Quality" represented in Fig. 48.
[0349]
121
•
In a case where multiplexing is performed using the
multiplexing pattern illustrated in (3) of Fig. 47, the
multiplexing mode and the encoding mode of compatible
images are the same as those of an existing mode, whereby
5 the compatibility is secured. In addition, the amount of
data of the disparity images of the compatible images and
the auxiliary images each having the same resolution as
that of the image before multiplexing decreases to be the
amount of data of the disparity images of two viewpoints.
10 Accordingly, in this case, since there are effects on the
compatibility and the data amount, circles are described
in correspondence with items ~Compatibility" and ~Data
Amount" represented in Fig. 48.
[0350]
15 In a case where multiplexing is performed using the
multiplexing pattern illustrated in (4) of Fig. 47, the
multiplexing mode and the encoding mode of compatible
images are the same as those of an existing mode, whereby
the compatibility is secured. In addition, since the
20 disparity images are multiplexed in the time direction,
the amount of data of the disparity images at each ~ime
is less than that of the case of (3) of Fig. 47 , whereby
the amount of data that can be transmitted increases.
Accordingly, even in a situation in which there is no
25 room for a transmission bandwidth for transmitting a
disparity image having the same resolution as the image
before multiplexing, a disparity image of which the
resolution is the same as the resolution of the image
before multiplexing can be transmitted, whereby the
30 accuracy of images of predetermined viewpoints to be
generated in the decoding device using the disparity
122
•
image is improved. Accordingly, in this case, there are
effects on the compatibility and the image quality of an
image generated using the disparity image, and there is a
remarkable effect on the data amount. Therefore, circles
5 are described in correspondence with items
~Compatibility" and ~Image Quality" illustrated in Fig.
48, and a double circle is described in correspondence
with ~Data Amount".
[0351]
10 The multiplexing process using the multiplexing
patterns illustrated in (1) to (4) of Fig. 47 is
performed, for example, when an image used for
broadcasting, the IP (internet protocol) that is
compliant with broadcasting, ATSC (Advanced Television
15 Systems Committee) 2.0 specifications and the like, nextgeneration
broadcasting that is converged broadcasting,
or internet delivery application is set as an encoding
target.
[0352]
20 In a case where multiplexing is performed using the
multiplexing pattern illustrated in (5) of Fig. 47, the
multiplexing mode and the encoding mode of compatible
images are the same as those of an existing mode, whereby
the compatibility is secured. In addition, since the
25 resolution of each disparity image is a half of the
resolution of the image, the amount of data of the
disparity images of the compatible image and the
auxiliary image decreases to be the amount of data of the
disparity images of two viewpoints. Accordingly, in this
30 case, there are effects on the compatibility and the data
amount, and circles are described in correspondence with
123
•
items "Compatibility" and "Data Amount" illustrated in
Fig. 48.
[0353]
In a case where multiplexing is performed using the
5 multiplexing pattern illustrated in (6) of Fig. 47,
similarly to the case of (2) of Fig. 47, the
compatibility is secured, and the accuracy of images of
predetermined viewpoints generated in the decoding device
using the disparity images is improved. Accordingly, in
10 this case, there are effects on the compatibility and the
image quality of an image generated using disparity
images, and circles are described in correspondence with
items "Compatibility" and "Image Quality" illustrated in
Fig. 48.
15 [0354]
In a case where multiplexing is performed using the
multiplexing pattern illustrated in (7) of Fig. 47,
similarly to the case of (3) of Fig. 47, the
compatibility is secured, and the amount of data of the
20 disparity image decreases. Accordingly, in this case,
there are effects on the compatibility and the data
amount, and circles are described in correspondence with
items "Compatibility" and "Data Amount" illustrated in
Fig. 48.
25 [0355]
In a case where multiplexing is performed using the
multiplexing pattern illustrated in (8) of Fig. 47,
similarly to the case of (4) of Fig. 47, the
compatibility is secured. In addition, similarly to the
30 case of (4) of Fig. 47, the amount of data of the
disparity images at each time is less than that of the
124
•
case of (7) of Fig. 47, and as a result, the accuracy of
images of predetermined viewpoints generated in the
decoding device using the disparity images is improved.
Accordingly, in this case, there are effects on the
5 compatibility and the image quality of an image generated
using the disparity images, and there is a remarkable
effect on the data amount. Therefore, circles are
described in correspondence with items "Compatibility"
and "Image Quality" illustrated in Fig. 48, and a double
10 circle is described in correspondence with "Data Amount".
[0356]
The multiplexing process using the multiplexing
patterns illustrated in (5), (7), and (8) of Fig. 47 is
performed, for example, when an image used for an
15 application for the BD, broadcasting, next-generation
broadcasting or internet delivery is set as an encoding
target. In addition, the multiplexing process using the
multiplexing pattern illustrated in (6) of Fig. 47 is
performed, for example, when an image used for an
20 application for the BD, next-generation broadcasting or
internet delivery is set as an encoding target.
[0357]
Fig. 49 is a diagram that illustrates an example of
the multiplexing pattern of the encoding target in a case
25 where the number of viewpoints of the compatible image is
one, and the number of viewpoints of the auxiliary image
is two.
[0358]
As illustrated in (1) of Fig. 49, the encoding
30 device 50 illustrated in Fig. 5 encodes an image A1 that
is a compatible image in accordance with the Ave mode.
125
•
In addition, the encoding device 50 multiplexes images B1
and C1 that are auxiliary images and disparity images B1'
and C1' that are the disparity images of the auxiliary
images in the time direction. Then, the encoding device
5 50 encodes a disparity image A1' that is the disparity
image of the compatible image, a multiplexed image of the
auxiliary images, and a multiplexed image of the
disparity images of the auxiliary images in accordance
with the 3DV mode that is compliant with the MVC mode.
10 [0359]
The encoding device 50, as illustrated in (2) of
Fig. 49, may encode the disparity images A1' to C1' in
accordance with the 3DV mode that is compliant with the
MVC mode without multiplexing the disparity images. In
15 addition, as illustrated in (3) of Fig. 49, the encoding
device 50, similarly to the encoding device 480, may
encode a cornmon disparity image BC1' of images Band C
that are auxiliary images instead of the multiplexed
image of the disparity images of the auxiliary images.
20 [0360]
In addition, as illustrated in (4) of Fig. 49, the
encoding device 50, similarly to the encoding device 440,
may multiplex the disparity images A1' to C1' in
accordance with the frame sequential mode and encode a
25 resultant multiplexed image in accordance with the 3DV
mode that is compliant with the MVC mode without
multiplexing the disparity images B1' and C1' in the
spatial direction.
[0361]
30 The effects of the multiplexing processes using the
multiplexing patterns illustrated in (1) to (4) of Fig.
126
•
49 and the encoding targets at the time of performing the
multiplexing processes using the multiplexing patterns
are the same as the effects and the encoding targets of
the multiplexing process using the multiplexing patterns
5 illustrated in (5) to (8) of Fig. 47. However, in the
multiplexing process using the multiplexing pattern
illustrated in (1) of Fig. 49, since the resolution of
the disparity image of the compatible image is the same
as the resolution of the compatible image, a decrease in
10 the amount of data of the disparity image as an effect of
this multiplexing process is an effect only for the
disparity image of the auxiliary image.
[0362]
Fig. 50 is a diagram that illustrates an example of
15 the multiplexing pattern of an encoding target in a case
where the number of viewpoints of the compatible image is
two, and the number of viewpoints of the auxiliary image
is zero.
[0363]
20 In a case where the number of viewpoints of the
compatible images is two, and the number of viewpoints of
the auxiliary image is zero, as illustrated in (1) of Fig.
50, similarly to the case of the encoding device 140
illustrated in Fig. 14, the images A2 and B2 that are
25 compatible images are multiplexed in the spatial
direction and are encoded in accordance with the Ave mode.
In addition, the disparity images A2' and B2' that are
the disparity images of the compatible images are
multiplexed in the spatial direction and are encoded in
30 accordance with the 3DV mode that is compliant with the
Ave mode.
127
•
[0364]
As illustrated in (2) of Fig. 50, the disparity
images A2' and B2' may be encoded in accordance with the
3DV mode that is compliant with the MVC mode without
5 being multiplexed. In addition, as illustrated in (3) of
Fig. 50, similarly to the case of the encoding device 4BO,
the common disparity image AB2' may be encoded instead of
the disparity images A2' and B2' .
[0365]
10 Furthermore, as illustrated in (4) of Fig. 50,
similarly to the case of the encoding device 440, the
disparity images A2' and B2' may be multiplexed in the
frame sequential mode and be encoded without being
multiplexed in the time direction.
15 [0366]
In addition, as illustrated in (5) of Fig. 50,
similarly to the case of the encoding device lBO, it may
be configured such that the image A2 is encoded in
accordance with the AVC mode, and the image B2 is encoded
20 with the MVC mode in which the image A2 is set as a base
view without multiplexing the images A2 and B2 that are
compatible images.
[0367]
In this case, as illustrated in (6) of Fig. 50,
25 similarly to the case of (2) of Fig. 50, the disparity
images may be encoded without being multiplexed, or, as
illustrated in (7) of Fig. 50, similarly to the case of
(3) of Fig. 50, the cornmon disparity image AB2' may be
encoded. In addition, as illustrated in (B) of Fig. 50,
30 similarly to the case of (4) illustrated in Fig. 50, the
disparity images of the compatible images may be
12B
•
multiplexed in accordance with the frame sequential mode
and be encoded.
[0368]
In addition, the effects of the multiplexing
5 processes using the multiplexing patterns illustrated in
(1) to (8) of Fig. 49 and the encoding targets at the
time of performing the multiplexing processes using the
multiplexing patterns are the same as the effects and the
encoding targets of the multiplexing processes using the
10 multiplexing patterns illustrated in (1) to (8) of Fig.
47.
[0369]
In the description presented above, although the
resolution of the disparity images that are multiplexed
15 in accordance with the frame sequential mode has been
described to be the same as the resolution of the image
before the multiplexing process, the resolution may be
lower than the resolution of the image before the
multiplexing process. In addition, the auxiliary images,
20 similarly to the disparity images, may be also
multiplexed in accordance with the frame sequential mode.
[0370]
Furthermore, in the description presented above,
although the information representing the multiplexing
25 mode of the images and the information representing the
multiplexing mode of the disparity images have been
described to be transmitted in the encoding device,
information used for identifying the multiplexing
patterns illustrated in Figs. 47, 49, and 50 may be
30 transmitted.
[0371]
129
•
In addition, the encoding device may transmit a
flag that is used for identifying an application that
corresponds to an image that is an encoding target.
[0372]
5
[Configuration Example of Encoding Device According to
Seventh Embodiment]
Fig. 51 is a block diagram that illustrates a
configuration example of an encoding device according to
10 a seventh embodiment of the present technology.
[0373]
In the configuration illustrated in Fig. 51, the
same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 35.
15 Duplicate description will not be presented as is
appropriate.
[0374]
The configuration of the encoding device 520
illustrated in Fig. 51 is mainly different from the
20 configuration illustrated in Fig. 35 in that an encoder
523 and a transmission unit 524 are disposed instead of
the encoder 145 and the multiplexing unit 443, and a
multiplexing unit 521 and a multiplexing information
generating unit 522 are newly arranged.
25 [0375]
The encoding device 520 multiplexes a multiplexed
image of compatible images, a multiplexed image of
auxiliary images, a multiplexed image of disparity images
of the compatible images, and a multiplexed image of
30 disparity images of the auxiliary images in accordance
with the frame sequential mode and encodes a resultant
130
•
multiplexed image.
[0376]
More specifically, the multiplexing unit 521 of the
encoding device 520 multiplexes a multiplexed image of a
5 compatible image and an auxiliary image that is acquired
as a result of the multiplexing process performed by an
image converting unit 142 and a multiplexed image of the
disparity images of the compatible image and the
auxiliary image that is acquired as a result of the
10 multiplexing process performed by a disparity image
generating unit 143 in accordance with the frame
sequential mode.
[0377]
Then, the multiplexing unit 521 supplies a
15 multiplexed image in which a multiplexed image of
compatible images, a multiplexed image of the disparity
images of the compatible images, a multiplexed image of
auxiliary images, and a multiplexed image of the
disparity images of the auxiliary images, which can be
20 acquired as a result of the multiplexing process, are
sequentially present to the encoder 523 as a timedirection
multiplexed image within one frame time.
[0378]
In addition, the multiplexing unit 521 supplies
25 information representing that multiplexed images of the
compatible images and the auxiliary images and a
multiplexed image of the disparity images of the
compatible image and the auxiliary image are multiplexed
in accordance with the frame sequential mode and
30 representing the arrangement order of images in the timemultiplexed
image to the multiplexing information
131
•
generating unit 522 and the encoder 523.
[0379]
The multiplexing information generating unit 522,
based on the information that is supplied from the
5 multiplexing unit 521, generates the information and the
like as whole multiplexing information relating to the
multiplexing of the compatible images and the auxiliary
images and the disparity images of the compatible images
and the auxiliary images and supplies the generated whole
10 multiplexing information to the transmission unit 524.
[0380]
The encoder 523 is configured by a compatible
encoder 531 and an auxiliary encoder 532. The encoder
523 designates the multiplexed image of the compatible
15 images that is included in the time-multiplexed image
supplied from the multiplexing unit 521 based on the
information supplied from the multiplexing unit 521 and
supplies the multiplexed image of the compatible images
to the compatible encoder 531. In addition, the encoder
20 523 supplies the multiplexed image of the auxiliary
images and the multiplexed images of the disparity images
of the compatible images and the auxiliary images that
are included in the time-multiplexed image to the
auxiliary encoder 532 based on the information that is
25 supplied from the multiplexing unit 521.
[0381]
The compatible encoder 531 of the encoder 523
encodes the multiplexed image of the compatible images
that are included in the time-multiplexed image in
30 accordance with the existing Ave mode. The auxiliary
encoder 532 encodes the multiplexed image of the
132
•
auxiliary images and the multiplexed images of the
disparity images of the compatible images and the
auxiliary images that are included in the timemultiplexed
image in accordance with the 3DV mode. At
5 this time, the multiplexed image of the auxiliary images
are encoded by referring to the multiplexed image of the
compatible images, and the multiplexed image of the
disparity images of the auxiliary images is encoded by
referring to the multiplexed image of the disparity
10 images of the compatible images.
[0382]
The encoder 523 supplies a bit stream that is
formed from encoded data of the time-multiplexed image
that is acquired as a result of the encoding process
15 performed by the compatible encoder 531 or the auxiliary
encoder 532 to the transmission unit 524.
[0383]
The transmission unit 524 generates a TS using the
bit stream supplied from the encoder 523, image
20 information supplied from the image information
generating unit 54, compatibility information supplied
from the compatibility information generating unit 55,
inter-viewpoint distance information supplied from the
inter-viewpoint distance information generating unit 144,
25 disparity image information supplied from the disparity
image information generating unit 57, the whole
multiplexing information supplied from the multiplexing
information generating unit 522, and the like. The
multiplexing unit 443 transmits the generated TS.
30 [0384]
[Description of Process of Encoding Device]
133
•
Figs. 52 and 53 are a flowchart that illustrates an
encoding process that is performed by the encoding device
520 illustrated in Fig. 51. This encoding process, for
example, is started when images A2 to D2 are output from
5 the imaging units 141A to 141D.
[0385]
The process of Steps S531 to S537 illustrated in
Fig. 52 is the same as the process of Steps S451 to S457
illustrated in Fig. 38, and thus the description thereof
10 will not be presented.
[0386]
After the process of Step S537, in Step S538, the
image converting unit 142 inputs a multiplexed image of
the compatible images and a multiplexed image of the
15 auxiliary images to the multiplexing unit 521, and the
process proceeds to Step S539 illustrated in Fig. 53.
[0387]
The process of Steps S539 to S542 illustrated in
Fig. 53 is the same as the process of Steps S459 to S462
20 illustrated in Fig. 39, and thus the description thereof
will not be presented.
[0388]
After the process of Step S543, in Step S544, the
multiplexing unit 521 multiplexes the multiplexed images
25 of the compatible images and the auxiliary images that
are acquired as a result of the multiplexing process
performed by the image converting unit 142 and the
multiplexed images of the disparity images of the
compatible images and the auxiliary images that are
30 acquired as a result of the multiplexing process
performed by the disparity image generating unit 143 in
134
•
accordance with the frame sequential mode. The
multiplexing unit 521 supplies a multiplexed image that
is acquired as a result of the multiplexing process to
the encoder 523 as a time-direction multiplexed image.
5 [0389]
In addition, the multiplexing unit 521 supplies
information representing that multiplexed images of the
compatible images and the auxiliary images and
multiplexed images of the disparity images are
10 multiplexed in accordance with the frame sequential mode
and representing the arrangement order of images in the
time-multiplexed image to the multiplexing information
generating unit 522 and the encoder 523.
[0390 ]
15 In Step S545, the multiplexing information
generating unit 522, based on the information that is
supplied from the multiplexing unit 521, generates the
information and the like as whole multiplexing
information and supplies the generated whole multiplexing
20 information to the transmission unit 524.
[0391]
In Step S546, the compatible encoder 531 encodes
the multiplexed image of the compatible images that is
included in the time-multiplexed image input based on the
25 information supplied from the multiplexing unit 521 by
the encoder 523 in accordance with the existing Ave mode.
[0392]
In Step S547, the auxiliary encoder 532 encodes the
multiplexed image of the auxiliary images, the
30 multiplexed image of the disparity images of the
compatible images, and the multiplexed image of the
135
•
disparity images of the auxiliary images that are
included in the time-multiplexed image that is input
based on the information supplied from the multiplexing
unit 521 by the encoder 523 in accordance with the 3DV
5 mode. The encoder 523 supplies a bit stream that is
formed from encoded data of the time-multiplexed image
that is acquired as a result of the encoding process of
Steps S546 and S547 to the transmission unit 524.
[0393]
10 In Step S548, the transmission unit 524 generates a
TS from the bit stream supplied from the encoder 523, the
auxiliary information, and the whole multiplexing
information supplied from the multiplexing information
generating unit 522 and transmits the generated TS. This
15 TS, for example, is recorded on a BD or the like or is
transmitted as a broadcast stream. Then, the process
ends.
[0394]
As above, the encoding device 520 generates one bit
20 stream from encoded data of the multiplexed image of the
compatible images, the multiplexed image of the auxiliary
images, the multiplexed image of the disparity images of
the compatible images, and the multiplexed image of the
disparity images of the auxiliary images. Accordingly, a
25 decoding device that includes a decoder that can decode
only one bit stream can decode the bit stream that is
generated by the encoding device 520.
[0395]
In the description presented above, although the
30 disparity images of the compatible images, the auxiliary
images and the disparity images of the auxiliary images
136
•
have been described to be encoded in accordance with the
3DV mode that is compliant with the encoding mode of the
compatible images, the images may be encoded in
accordance with MPEG2 (Moving Picture Experts Group phase
5 2) or the like that is not compliant with an encoding
mode of compatible images.
[0396]
[Configuration Example of Decoding Device]
Fig. 54 is a diagram that illustrates a
10 configuration example of a decoding device that decodes
the TS transmitted from the encoding device 520
illustrated in Fig. 51.
[0397]
In the configuration illustrated in Fig. 54, the
15 same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 40.
Duplicate description will not be presented as is
appropriate.
[0398]
20 The configuration of the decoding device 540
illustrated in Fig. 54 is mainly different from the
configuration illustrated in Fig. 40 in that a reception
unit 541, a decoder 542, and an image generating unit 544
are disposed instead of the separation unit 121, the
25 decoder 122, and the image generating unit 171, and a
multiplexing information acquiring unit 543 is newly
arranged. The decoding device 540 decodes a bit stream
of a time-multiplexed image that is included in a TS
transmitted from the encoding device 520, generates
30 images of two viewpoints or multi-viewpoint images, and
displays the generated images on a display device not
137
•
illustrated in the figure.
[0399]
More specifically, the reception unit 541 of the
decoding device 540 receives a TS that is transmitted
5 from the encoding device 520. The reception unit 541
extracts a bit stream of a time-multiplexed image that is
included in the TS and supplies the bit stream to the
decoder 542. In addition, the reception unit 541
extracts auxiliary information that is included in the TS
10 and supplies image information that is included in the
auxiliary information to the image information acquiring
unit 123 and supplies viewpoint distance information to
an inter-viewpoint distance information acquiring unit
124. Furthermore, the reception unit 541 supplies
15 disparity image information included in the auxiliary
information to a disparity image information acquiring
unit 125 and supplies compatibility information to a
compatibility information acquiring unit 126 and the
decoder 501. In addition, the reception unit 541
20 extracts whole multiplexing information that is included
in the TS and supplies the extracted whole multiplexing
information to the multiplexing information acquiring
unit 543.
[0400]
25 The decoder 542 is configured by a compatible
decoder 551 and an auxiliary decoder 552. The compatible
decoder 551 of the decoder 542 extracts encoded data of
the multiplexed image of the compatible images from the
bit stream that is supplied from the reception unit 541
30 based on the compatibility information that is supplied
from the separation unit 461. The compatible decoder 551
138
•
decodes the extracted encoded data of the multiplexed
image of the compatible images in accordance with a mode
that corresponds to the Ave mode and supplies the decoded
data to the image generating unit 544.
5 [0401]
The auxiliary decoder 552 extracts encoded data of
the multiplexed image of the auxiliary images, the
multiplexed image of the disparity images of the
compatible images, and the multiplexed image of the
10 disparity images of the auxiliary images from the bit
stream that is supplied from the reception unit 541 based
on the compatibility information supplied from the
separation unit 461. The auxiliary decoder 552 decodes
encoded data of the multiplexed image of the auxiliary
15 images, the multiplexed image of the disparity images of
compatible images, and the multiplexed image of the
disparity images of the auxiliary images, that are
extracted, in accordance with a mode that corresponds to
the decoding mode of the auxiliary encoder 532
20 illustrated in Fig. 51 and supplies the decoded data to
the image generating unit 544.
[0402]
The multiplexing information acquiring unit 543
acquires the whole multiplexing information that is
25 supplied from the reception unit 541 and supplies the
acquired whole multiplexing information to the image
generating unit 544.
[0403]
The image generating unit 544 outputs an image in
30 accordance with a viewer's display instruction, thereby
displaying the image on a display device not illustrated
139
•
in the figure. More specifically, the image generating
unit 544, in accordance with a viewer's instruction for
displaying a 3D image of the multi-viewpoint system,
generates images of three or more viewpoints, which
5 correspond to a display device not illustrated in the
figure, each having the resolution that is a half of the
resolution of the compatible image or the auxiliary image
by using the multiplexed image of the compatible images,
the multiplexed image of the auxiliary images, the
10 multiplexed image of the disparity images of the
compatible images, and the multiplexed image of the
disparity images of the auxiliary images based on the
image information supplied from the image information
acquiring unit 123, the inter-viewpoint distance
15 information supplied from the inter-viewpoint distance
information acquiring unit 124, the disparity image
information supplied from the disparity image information
acquiring unit 125, the compatibility information
supplied from the compatibility information acquiring
20 unit 126, the whole multiplexing information supplied
from the multiplexing information acquiring unit 543,and
the like.
[0404]
Described in more detail, the image generating unit
25 544 identifies the multiplexed image of the auxiliary
image, the multiplexed image of the disparity image of
the compatible image, and the multiplexed image of the
disparity image of the auxiliary image, which are
supplied from the auxiliary decoder 552, based on the
30 whole multiplexing information supplied from the
multiplexing information acquiring unit 543. In addition,
140
•
the image generating unit 544, similarly to the image
generating unit 171 illustrated in Fig. 17, separates the
disparity image of each auxiliary image from the
multiplexed image of the disparity images of the
5 auxiliary images based on the information that represents
the multiplexing mode of the disparity image of the
auxiliary image that is included in the disparity image
information supplied from the disparity image information
acquiring unit 125. Furthermore, the image generating
10 unit 544, similarly to the image generating unit 171,
separates the disparity image of each compatible image
from the multiplexed image of the disparity images of the
compatible images based on the information that
represents the multiplexing mode of the disparity images
15 of the compatible images that is included in the
disparity image information.
[0405]
Furthermore, the image generating unit 544,
similarly to the image generating unit 171, separates
20 each auxiliary image from the multiplexed image of the
auxiliary images based on the information that represents
the multiplexing mode of the auxiliary images that is
included in the image information supplied from the image
information acquiring unit 123. In addition, the image
25 generating unit 544, similarly to the image generating
unit 171, separates each compatible image from the
multiplexed image of the compatible images based on the
information that represents the multiplexing mode of the
compatible images that is included in the image
30 information.
[0406]
141
•
In addition, the image generating unit 544,
similarly to the image generating unit 171, determines
the position of each viewpoint of the multi-viewpoint
images to be generated based on the inter-viewpoint
5 distance information and the number of viewpoints that
corresponds to a display device not illustrated in the
figure. Then, the image generating unit 544, similarly
to the image generating unit 171, generates an image of
each viewpoint of which the position is determined by
10 using each compatible image, each auxiliary image, the
disparity image of each compatible image, and the
disparity image of each auxiliary image. Then, the image
generating unit 544 converts the resolution of the
generated multi-viewpoint images to the resolution that
15 is ~1/the number of viewpoints" of the resolution of the
compatible image or the auxiliary image, synthesizes the
converted images, and displays the synthesized image on a
display device not illustrated in the figure.
[0407]
20 At this time, the multi-viewpoint images after the
synthesis are displayed such that the viewing angles are
different for each viewpoint, and a viewer can view a 3D
image without wearing glasses by seeing images of
arbitrary two viewpoints with his left and right eyes.
25 [0408]
In addition, the image generating unit 544,
similarly to the image generating unit 171, separates the
multiplexed image of the compatible images that is
supplied from the decoder 542 into images A2 and B2 of
30 which the resolution is a half of the resolution of the
compatible image based on the image information supplied
142
•
from the image information acquiring unit 123 in
accordance with an instruction for displaying a 3D image
of the two-viewpoint mode that is supplied from a viewer.
Then, the image generating unit 544, similarly to the
5 image generating unit 171, alternately outputs the
separated images A2 and B2 of which the resolution is a
half of the resolution of the compatible image, thereby
displaying the separated images on a display device not
illustrated in the figure. At this time, the viewer can
10 view a 3D image by wearing glasses in which one a lefteye
shutter and a right-eye shutter is open at the time
of displaying the image A2, and the other is open at the
time of displaying the image B2 and viewing the images A2
and B2 that are alternately displayed on the display
15 device.
[0409]
[Description of Process of Decoding Device]
Fig. 55 is a flowchart that illustrates a decoding
process performed by the decoding device 540 illustrated
20 in Fig. 54. This decoding process, for example, is
started when a TS that is transmitted from the encoding
device 520 illustrated in Fig. 51 is input to the
decoding device 540.
[0410]
25 In Step S551 illustrated in Fig. 55, the reception
unit 541 of the decoding device 540 receives a TS that is
transmitted from the encoding device 520. The reception
unit 541 extracts a bit stream included in the TS and
supplies the extracted bit stream to the decoder 542. In
30 addition, the reception unit 541 extracts auxiliary
information that is included in the TS, supplies image
143
•
information included in the auxiliary information to the
image information acquiring unit 123, and supplies
viewpoint distance information to the inter-viewpoint
distance information acquiring unit 124. Furthermore,
5 the reception unit 541 supplies disparity image
information included in the auxiliary information to the
disparity image information acquiring unit 125 and
supplies compatibility information to the compatibility
information acquiring unit 126. In addition, the
10 reception unit 541 extracts whole multiplexing
information that is included in the TS and supplies the
extracted whole multiplexing information to the
multiplexing information acquiring unit 543.
[0411]
15 In Step S552, similarly to the process of Step S474
illustrated in Fig. 41, the image generating unit 544
determines whether or not an instruction for displaying a
3D image of the two-viewpoint mode has been made by a
viewer. In a case where it is determined that the
20 instruction for displaying a 3D image of the twoviewpoint
mode has not been made by the viewer in Step
S552, in other words, in a case where an instruction for
displaying a 3D image of the multi-viewpoint mode has
been made, the process proceeds to Step S553.
25 [0412]
In Step S553, the compatible decoder 551 of the
decoder 542 extracts encoded data of the multiplexed
image of the compatible images from the bit stream based
on the compatibility information that is supplied from
30 the reception unit 541 and decodes the extracted encoded
data in a mode that corresponds to the Ave mode. Then,
144
•
the compatible decoder 551 supplies a multiplexed image
of the compatible images that is acquired as a result of
the decoding process to the image generating unit 544.
[0413]
5 In Step S554, the auxiliary decoder 552 extracts
encoded data of the multiplexed image of the auxiliary
images, the multiplexed image of the disparity images of
the compatible images, and the multiplexed image of the
disparity images of the auxiliary images from the bit
10 stream based on the compatibility information that is
supplied from the reception unit 541 and decodes the
extracted encoded data in a mode that corresponds to the
auxiliary encoder 532 illustrated in Fig. 51. The
auxiliary decoder 552 supplies the multiplexed image of
15 the auxiliary images, the multiplexed image of the
disparity images of the compatible images, and the
multiplexed image of the disparity images of the
auxiliary images, which are acquired as a result of the
decoding process, to the image generating unit 544, and
20 the process proceeds to Step S555.
[0414]
The process of Steps S555 to S558 is the same as
that of Steps S476 to S479 illustrated in Fig. 41, and
thus description thereof will not be presented.
25 [0415]
After the process of Step S558, in Step S559, the
multiplexing information acquiring unit 543 acquires the
whole multiplexing information that is supplied from the
reception unit 541 and inputs the acquired whole
30 multiplexing information to the image generating unit 544.
[0416]
145
•
In Step S560, the image generating unit 544,
similarly to the process of Step S480 illustrated in Fig.
41, determines the position of each viewpoint of multiviewpoint
images to be generated based on the inter-
S viewpoint distance information supplied from the interviewpoint
distance information acquiring unit 124 and the
number of viewpoints that corresponds to a display device
not illustrated in the figure.
[0417]
10 In Step S561, the image generating unit 544
generates an image of each viewpoint of which the
resolution is a half of the resolution of the compatible
image or the auxiliary image by using the multiplexed
image of the compatible images, the multiplexed image of
15 the auxiliary images, the multiplexed image of the
disparity images of the compatible images, and the
multiplexed image of the disparity images of the
auxiliary images based on the position of each viewpoint
that is determined in Step S560, the image information
20 supplied from the image information acquiring unit 123,
the disparity image information supplied from the
disparity image information acquiring unit 125, the
compatibility information supplied from the compatibility
information acquiring unit 126, the whole multiplexing
25 information supplied from the multiplexing information
acquiring unit 543, and the like. Then, the process
proceeds to Step S562.
[0418]
The process of Steps S562 and 8563 is the same as
30 that of 8teps 8482 and 8483 illustrated in Fig. 41, and
thus description thereof will not be presented.
146
•
[0419]
In a case where it is determined that the
instruction for displaying a 3D image of the twoviewpoint
mode has been made by a viewer in Step S552, in
5 Step S564, the compatible decoder 551 extracts encoded
data of the multiplexed image of the compatible images
from the bit stream based on the compatibility
information that is supplied from the reception unit 541
and decodes the extracted encoded data in accordance with
10 a mode that corresponds to the Ave mode. Then, the
compatible decoder 551 supplies a multiplexed image of
the compatible images that is acquired as a result of the
decoding process to the image generating unit 544.
[0420]
15 The process of Steps S565 to S567 is the same as
that of Steps S484 to S486 illustrated in Fig. 41, and
thus description thereof will not be presented.
[0421]
In addition, in a decoding device that can decode
20 only a compatible image that has compatibility with the
decoding device 540, encoded data other than the encoded
data of the compatible images that can be processed is
ignored, and the process of Steps S564, S566, and S567 is
performed. However, in such a case, in the process of
25 Step S566, a multiplexed image of the compatible images
is separated in accordance with a multiplexing mode that
is determined in advance.
[0422]
As above, the decoding device 540 can decode one
30 bit stream that is generated from the encoded data of the
multiplexed image of the compatible images, the
147
•
multiplexed image of the auxiliary images, the
multiplexed image of the disparity images of the
compatible images, and the multiplexed image of the
disparity images of the auxiliary images.
5 [0423]

[Configuration Example of Bit Stream]
Fig. 56 is a diagram that illustrates a
configuration example of a bit stream (encoded stream) in
10 units of access units that is generated in accordance
with this embodiment.
[0424]
In the example illustrated in Fig. 56, the
compatible images are an L image and an R image of 1920 x
15 1080 pixels, and the auxiliary image is an 0 image of
1920 x 1080 pixels. In addition, the size of the L
disparity image, the R disparity image, and the 0
disparity image is 960 x 1080 pixels. Furthermore, the L
image is encoded in accordance with the AVC mode, the R
20 image is encoded in accordance with the MVC mode, and the
o image, the L disparity image, the R disparity image,
and the 0 disparity image are encoded in according with
the 3DV mode. In addition, the view IDs of the L image,
the R image, the 0 image, the L disparity image, the R
25 disparity image, and the 0 disparity image are 0, 1, 2, 3,
4, and 5, respectively. Here, the view ID is an ID that
is unique to the image of each viewpoint and the
disparity image.
[0425]
30 As illustrated in Fig. 56, in the bit stream in
units of access units, for example, an access unit
148
•
delimiter (AUD), an SPS (sequence parameter set), a
Subset SPS (Subset SPS1) according to the Mve mode for an
image, a Subset SPS (Subset SPS2) according to the 3DV
mode for a disparity image, a PPS (picture parameter set),
5 SEI according to the Ave mode, SEI according to the Mve
mode, SEI according to the 3DV mode, and an NAL (network
abstraction layer) unit of encoded data are arranged in
order from the head.
[0426]
10 The NAL unit of the access unit delimiter is an NAL
unit that represents a boundary of an access unit. The
NAL unit of the SPS is an NAL unit of the SPS that
includes a profile_ide (100 in the example illustrated in
Fig. 56) that represents a profile of the L image out of
15 profiles defined in accordance with the Ave mode. The
NAL unit of the Subset SPS for an image is an NAL unit of
the Subset SPS including profile_ide (128 in the example
illustrated in Fig. 56) that represents profiles of the R
image and the 0 image out of profiles defined in
20 accordance with the Mve mode. The NAL unit of the Subset
SPS for a disparity image is an NAL unit of the Subset
SPS that includes profile_ide (138 in the example of Fig.
56) that represents profiles of the L disparity image,
the R disparity ~mage, and the 0 disparity image out of
25 profiles defined as profiles for a disparity image in
accordance with the 3DV mode.
[0427]
The NAL unit of the SEI according to the Ave mode
is an NAL unit of the SEI of the L image. The NAL unit
30 of the SEI according to the Mve mode is an NAL unit of
the SEI of the L image or the R image. The NAL unit of
149
•
the SEI of the 30V mode is an NAL unit of the SEI of the
L image, the R image, the 0 image, the L disparity image,
the R disparity image, or the 0 disparity image.
[0428]
5 As NAL units of the encoded data, NAL units of
encoded data of an L image, a delimiter (MVC DO), encoded
data of an R image, a delimiter (30V DO), encoded data of
an 0 image, a delimiter (30V DO), encoded data of an L
disparity image, a delimiter (30V DO), encoded data of an
10 R disparity image, a delimiter (30V DO), encoded data of
an 0 disparity image are arranged in order from the head.
[0429]
In the NAL unit of the encoded data of the L image,
an NAL header that includes 1 or 5 as the type of the NAL
15 unit representing a picture according to the AVC mode is
added. In addition, the NAL unit of the delimiter (MVC
DO) is an NAL unit that represents a head of the encoded
data of the MVC mode. In the NAL unit of the encoded
data of the R image, an NAL header that includes 20
20 representing encoded data according to the MVC mode as
the type of the NAL unit is added. In addition, the NAL
unit of the delimiter (30V DO) is an NAL unit that
represents a head of the encoded data of the 30V mode.
Furthermore, in the NAL units of the encoded data of the
25 0 image, the L disparity image, the R disparity image,
and the 0 disparity image, an NAL header that includes 21
representing encoded data according to the 30V mode as
the type of the NAL unit is added.
[0430]
30 [Example of Description of Subset SPS According to 3DV
Mode for Disparity Image]
150
•
Fig. 57 is a diagram that illustrates an example of
the description of a Subset SPS according to the 3DV mode
for a disparity image illustrated in Fig. 56.
[0431]
5 As illustrated in Fig. 57, in a Subset SPS
according to the 3DV mode for a disparity image, SPS
(seq_parameter_set_data) that includes profile_ide (138
in the example illustrated in Fig. 57) representing the
profiles of the L disparity image, the R disparity image,
10 and the 0 disparity image and information for each
profile_ide that is defined in accordance with the 3DV
mode are described.
[0432]
More specifically, in the Subset SPS of the 3DV
15 mode for a disparity image, as information when
profile_ide is 138, extension information
(seq_parameter_set_depth_extension) for a disparity image,
a VUI information flag
(depth_vui_parameters_present_flag) for a disparity image
20 that represents whether or not VUI extension information
for a disparity image is included and the like are
described. In addition, in a case where the VUI
information flag for a disparity image represents that
the VUI extension information for a disparity image is
25 included, VUI extension information
(depth_vui_parameters__extension) for a disparity image
is also described.
[0433]
In addition, in a case where the Subset SPS of the
30 3DV mode for a disparity image is referred to in a
decoding process, in other words, in a case where the L
151
•
disparity image, the R disparity image, and the 0
disparity image are decoded, similarly to a case where an
IDR picture is decoded, the reference image is reset.
[0434]
5 Fig. 58 is a diagram that illustrates an example of
the description of extension information for a disparity
image that is illustrated in Fig. 57.
[0435]
As illustrated in Fig. 58, the extension
10 information for disparity information, similarly to the
extension information (seq_parameter_set_mvc_extension)
within the Subset SPS of the MVC mode, is formed by
described information and a view ID (ref_view_id) of an
image corresponding to each disparity image.
15 [0436]
In Fig. 58, similarly to the extension information
within the Subset SPS of the MVC mode, the described
information is expanded and is included in the extension
information for a disparity image. Accordingly, the view
20 ID (view_id) of each disparity view and the view ID of an
image that corresponds to each disparity image can be
described together for each disparity image. In other
words, information (num_views_minusl) that represents the
number of disparity images is described, and a
25 description for reading out the view ID of a disparity
image and the view ID of an image that corresponds to the
disparity image can be made as many times as the number.
[0437]
In contrast to this, similarly to the extension
30 information within the Subset SPS of the MVC mode, in a
case where the described information is not expanded and
152
•
is included in the extension information for a disparity
image, in addition to the extension information for a
disparity image, information representing the number of
disparity images is described, and a description for
5 reading out a view ID of an image that corresponds to the
disparity image needs to be made as many times as the
number. As a result, the description of the information
that represents the number of disparity images and the
description for reading out the information as many times
10 as the number overlap each other.
[0438]
Accordingly, as illustrated in Fig. 58, in a case
where described information is expanded and is included
in the extension information for a disparity image,
15 similarly to the extension information within the Subset
SPS of the MVC mode, the amount of data of the extension
information for a disparity image can be less than that
of a case where the information is not expanded and is
included in the extension information for a disparity
20 image.
[0439]
Fig. 59 is a diagram that illustrates an example of
the description of VUI extension information for a
disparity image illustrated in Fig. 57.
25 [0440]
As illustrated in Fig. 59, the VUI extension
information for a disparity image is described in the
same manner as the VUI extension information
(mvc_vui_parameters__extension) of the MVC mode except
30 for the following points. In other words, in the VUI
information for a disparity image, for each pixel of a
153
•
disparity image, a position type flag
(depth_loc_info_present_flag) that represents whether or
not a position type representing the type of a position
of an image that corresponds to the disparity image is
5 included and a before-conversion size information flag
(video_src_info_present_flag) representing whether or not
before-conversion size information representing the size
of the disparity image before resolution conversion is
included are included. In addition, in a case where the
10 position type flag represents that the position type is
included, the position type is also included in the VUI
information for a disparity image, and, in a case where
the before-conversion size information flag represents
that the before-conversion size information is included,
15 the before-conversion size information is also included
in the VUI information for a disparity image.
[0441]
The position type is formed by a top field position
type (depth_sample_loc_type_top_field) and a bottom field
20 position type (depth_sample_loc_type_bottom_field). The
top field and bottom field position types are described
similarly to the top field position type
(chroma_sample_loc_type_top_field) and the bottom field
position type (chroma_sample_loc_type_bottom_field) that
25 are included in the VUI extension information of the MVC
mode.
[0442]
In addition, the before-conversion size information
is configured by information (pic_width_in_mbs_minus1)
30 that represents the number of macro blocks of the
disparity image before resolution conversion in the
154
horizontal direction, information
(pic_height_in_mbs_minusl) that represents the number of
macro blocks in the vertical direction, an aspect ratio
flag (aspect_ratio_info_present_flag) that represents
5 whether or not aspect information representing an aspect
ratio is included, and the like. In a case where the
aspect ratio flag represents that the aspect information
is included, the aspect information is also included in
the before-conversion size information.
10 [0443]
The aspect information is formed by an aspect ratio
ID (aspect_ratio_idc) that is an ID unique to an aspect
ratio and the like. This aspect ratio IDs are assigned
not only to aspect ratios that are defined in advance but
15 also to all the aspect ratios that are not defined. In a
case where an aspect ratio ID that is included in the
aspect information is an aspect ratio ID (Extended_SAR)
that is given to all the aspect ratios that are not
defined, values (sar_width and sar_height) of the aspect
20 ratio of the disparity image before resolution conversion
in the horizontal and vertical directions are also
included in the aspect information.
[0444]
[Example of Description of NAL Header of Encoded Data of
25 3DV Mode]
Fig. 60 is a diagram that illustrates an example of
the description of an NAL header of an NAL unit of
encoded data according to the 3DV mode in which 21 is
included as the type of the NAL unit.
30 [0445]
As illustrated in Fig. 60, in the NAL header of the
155
•
encoded data according to the 3DV mode, as information at
a time when the type of the NAL unit is 21, a disparity
image header extension information flag
(depth_extension_flag) representing whether or not
5 disparity image header extension information is included
and the like are described. Here, the disparity image
header extension information is also described in the NAL
header of the encoded data of the disparity image
according to the 3DV mode.
10 [0446]
In a case where the disparity image header
extension information flag represents that the disparity
image header extension information is included, in the
NAL header of the encoded data of the 3DV mode, the
15 disparity image header extension information
(nal_unit_header_depth_extension) is also described. On
the other hand, in a case where the disparity image
header extension information flag represents that the
disparity image header extension information is not
20 included, in the NAL header of the encoded data of the
3DV mode, MVC mode header extension information
(nal_unit_header mvc extension) is also described.
[0447]
Fig. 61 is a diagram that illustrates an example of
25 the description of the disparity image header extension
information illustrated in Fig. 60.
[0448]
As illustrated in Fig. 61, the disparity image
header extension information is configured to be the same
30 as the MVC mode header extension information except that
a view ID (ref_view_id) of an image corresponding to the
156
disparity image is included.
[0449]
[Example of Description of SEI according to 3DV Mode]
Fig. 62 is a diagram that illustrates an example of
5 the description of SEI according to the 3DV mode
illustrated in Fig. 56.
[0450]
As illustrated in Fig. 62, in the SEI according to
the 3DV mode, similarly to the SEI according to the MVC
10 mode, a message of SEI is described.
[0451]
In other words, in the SEI according to the 3DV
mode, an operation point flag (operation_point_flag)
representing whether or not an operation point is
15 designated is described, and, in a case where the
operation point flag represents that the operation point
is not designated, an all-component flag
(all_view_components_in_au_flag) representing whether or
not the message of the SEI is adapted to all the images
20 and the disparity images within the access unit is
described. On the other hand, in a case where the allcomponent
flag represents that the message of the SEI is
not adapted to all the images and the disparity images
within the access unit, the number
25 (num_view_components_minus1) of view IDs and the view IDs
(sei_view_id) of the images and the disparity images to
which the message of the SEI is adapted are described.
[0452]
On the other hand, in a case where the operation
30 point flag represents that the operation point is
designated, out of operation points to which the message
157
of the SEI is adapted, view IDs (sei_op_view_id) of the
images and the disparity images that are adaptation
targets and the number (sei_op_temporal_id) of the
operation points are described. Then, the message
5 (sei_rbsp) of the SEI is described. In addition, in the
SEI according to the 3DV mode, a plurality of messages of
the SEI can be described.
[0453]
In the example illustrated in Fig. 56, although the
10 profiles of the Rand 0 images are assumed to be the same,
the profiles of the Rand 0 images may be different from
each other.
[0454]
[Another Configuration Example of Bit stream]
15 Fig. 63 is a diagram that illustrates a
configuration example of the bit stream in units of
access units in a case where profiles of the Rand 0
images illustrated in Fig. 56 are different from each
other.
20 [0455]
The configuration of the bit stream illustrated in
Fig. 63 is different from the configuration of the bit
stream illustrated in Fig. 56 in that a Subset SPS
according to the 3DV mode for an image other than the
25 Subset SPS according to the MVC mode for an image and the
Subset SPS according to the 3DV mode for a disparity
image is arranged.
[0456]
The NAL unit (Subset SPSl) of the Subset SPS
30 according to the MVC mode for an image is an NAL unit of
the Subset SPS that includes profile_ide (128 in the
158
•
example illustrated in Fig. 63) representing the profile
of the R image out of profiles defined in the MVC mode.
The NAL unit of the Subset SPS (Subset SPS2) according to
the 3DV mode for an image is an NAL unit of the Subset
5 SPS that includes profile_ide (148 in the example
illustrated in Fig. 63) representing the profile of the 0
image out of profiles defined as profiles for an image in
the 3DV mode. The NAL unit of the Subset SPS (Subset
SPS3) for a disparity image is an NAL unit of the Subset
10 SPS that includes profile_ide (138 in the example
illustrated in Fig. 63) representing the profiles of the
L disparity image, the R disparity image, and the 0
disparity image out of profiles defined as profiles for a
disparity image in the 3DV mode.
15 [0457]
[Example of Description of Subset SPS According to 3DV
Mode for Image]
Fig. 64 is a diagram that illustrates an example of
the description of Subset SPS according to the 3DV mode
20 for an image that is illustrated in Fig. 63.
[0458]
As illustrated in Fig. 64, in the Subset SPS
according to the 3DV mode for an image, SPS
(seq_parameter_set_data) that includes profile_ide (148
25 in the example illustrated in Fig. 64) representing the
profile of the image 0 and information for each
profile_ide that is defined in the 3DV mode are described.
[0459]
More specifically, in the Subset SPS according to
30 the 3DV mode for an image, similarly to the Subset SPS
according to the MVC mode as information at a time when
159
•
profile_idc is 148, the extension information
(seq_parameter_set_mvc_extension) according to the MVC
mode, the VUI information flag
(mvc_vui_parameters_present_flag) according to the MVC
5 mode that represents whether or not the VUI extension
information according to the MVC mode is included, and
the like are described. In addition, in a case where the
VUI information flag according to the MVC mode represents
that the VUI extension information according to the MVC
10 mode is included, the VUI extension information
(mvc_vui_parameters_extension) according to the MVC mode
is also described. Furthermore, as the information at a
time when profile_idc is 138, information similar to that
illustrated in Fig. 57 is described.
15 [0460]
Here, the 3DV mode may be a mode for encoding a
display image according to a multi-viewpoint mode that is
compliant with an HEVC (High Efficiency Video Coding)
mode other than the AVC mode and the MVC mode. A bit
20 stream of such a case will be described as below. In
this specification, it is assumed that the HEVC mode is
based on HEVC Working Draft: Thomas Wiegand, Woo-jin Han,
Benjamin Bross, Jens-Rainer Ohm, Gary J. Sullivian, "WD3:
Working Draft3 of High-Efficiency Video Coding", JCTVc-
25 E603 d5 (versionS), written on May 20, 2011.
[0461]

[Configuration Example of Bit Stream]
30 Fig. 65 is a diagram that illustrates a
configuration example of a bit stream in units of access
160
units in a case where the 3DV mode is a mode that is
compliant with the HEVe mode.
[0462]
In the example illustrated in Fig. 65, it is
5 assumed that the L viewpoint image, the R viewpoint image,
and the 0 viewpoint image as in the example illustrated
in Fig. 56 are encoding targets, the L image is encoded
in accordance with the Ave mode, and the L viewpoint
image, the R viewpoint image, and the 0 viewpoint image
10 are encoded in accordance with the 3DV mode.
[0463]
The bit stream illustrated in Fig. 65 is different
from the bit stream illustrated in Fig. 56 in that the
Subset SPS (Subset SPSl) according to the 3DV mode for an
15 image and the Subset SPS (Subset SPS2) according to the
3DV mode for a disparity image can be described within
the SPS, and NAL headers including mutually-different
types of the NAL units are added to the NAL units of the
SEI according to the HEve mode and the SEI according to
20 the 3DV mode.
[0464]
In the bit stream illustrated in Fig. 65, the
Subset SPS according to the 3DV mode for an image and the
Subset SPS according to the 3DV mode for a disparity
25 image may be described only within the SPS, may be
described to be separate from the SPS, or may be
described in the SPS and described to be separate from
the SPS. Here, the description of the Subset SPS
according to the 3DV mode for an image and the Subset SPS
30 according to the 3DV mode for a disparity image separated
from the SPS is compliant with the BD standard and is
161
I
appropriate for a case where the encoded data according
to the HEVC mode and the encoded data according to the
3DV mode are generated as ES (elementary streams)
different from each other.
5 [0465]
In the bit stream illustrated in Fig. 65, since the
types of the NAL units that are included in the NAL
headers added to the NAL unit of the SEI according to the
HEVC mode and the NAL unit of the SEI according to the
10 3DV mode are different from each other, an NAL unit of
the SEI according to the HEVC mode or the SEI according
to the 3DV mode can be easily extracted in a decoding
process.
[0466]
15 [Example of Description of SPS]
Fig. 66 is a diagram that illustrates an example of
the description of SPS illustrated in Fig. 65.
[0467]
The description of the SPS illustrated in Fig. 66
20 is the same as the description of the SPS according to
the HEVC mode except that a Subset SPS information flag
(subset_seq_present_flag) representing whether or not
information of the Subset SPS is included is described,
and the information of the Subset SPS is described in a
25 case where the Subset SPS information flag represents
that the information of the Subset SPS is included.
[0468]
As illustrated in Fig. 66, the information of the
Subset SPS includes the number (nurn_subset_seq) of Subset
30 SPS and a Subset SPS flag (subset_seq_info_present_flag)
that represents whether or not the Subset SPS is included.
162
•
In addition, in a case where the Subset SPS flag
represents that the Subset SPS is included, the Subset
SPS (subset_seq_parameter_set_data) is also included in
the information of the Subset SPS.
5 [0469]
As above, since the number of Subset SPS is
described as the information of the Subset SPS, it can be
recognized whether or the Subset SPS is present by only
reading out the description of the SPS in the decoding
10 process. In addition, since the Subset SPS flag is
described, the Subset SPS can be described separated from
the SPS without being described within the SPS, whereby
duplicate on the description of the Subset SPS can be
prevented.
15 [0470]
[Example of Description of Subset SPS]
Fig. 67 is a diagram that illustrates an example of
the description of the Subset SPS illustrated in Fig. 66.
[0471]
20 The description of the Subset SPS illustrated in
Fig. 67 is the same as the description of the SPS
according to the HEVC mode except that images to which
the Subset SPS is adapted, the number
(num_subset_seq_views) of view IDs of disparity images, a
25 disparity image flag (depth_extension_flag) that
represents whether or not the adaptation target of the
Subset SPS is a disparity image, and an invalidness flag
(seq param_override_flag) that represents whether or not
the SPS is invalidated at the time of adapting the Subset
30 SPS are described.
[0472]
163
•
In a case where the disparity image flag represents
that the adaptation target of the Subset SPS is a
disparity image, a view ID (ref_view_id) of an image that
corresponds to the disparity image that is the adaption
5 target is described in the Subset SPS. In addition, in a
case where the invalidness flag represents that the SPS
is invalidated at the time of adapting the Subset SPS,
similarly to the SPS, information
(subset_seq_profile_idc) that represents profiles and the
10 like are described in the Subset SPS.
[0473]
Of the descriptions of the Subset SPS, as
descriptions similar to the descriptions of the SPS, for
example, there are information
15 (subset_seq_frame_cropping_flag) that represents whether
or not an adapted image and a disparity image are cropped,
a VUI information flag
(subset_seq_vui_parameters_present_flag) that represents
whether or not the VUI information
20 (subset_seq_vui_parameters) is included, and the like.
In a case whether VUI information flag represents that
the VUI information is included, similarly to the SPS,
the VUI information is also described. On the other hand,
in a case where the VUI information flag represents that
25 the VUI information is not included, similarly to the SPS,
the VUI information is not described. In such a case,
the VUI information of the SPS is adapted as the VUI
information.
[0474]
30 [Example of Description of VUI information of Subset SPS]
Fig. 68 is a diagram that illustrates an example of
164
•
the description of the VUI information of the Subset SPS
illustrated in Fig. 67.
[0475]
The description of the VUI information of the
5 Subset SPS illustrated in Fig. 68 is the same as the
description of the VUI information of the SPS according
to the HEVC mode except that a before-conversion size
information flag (video_src_info_present_flag)
representing whether or not before-conversion size
10 information that is an adaptation target is included is
described.
[0476]
In a case where the before-conversion size
information flag represents that the before-conversion
15 size information of an adaption target is included, the
before-conversion size information is described in the
VUI information illustrated in Fig. 68. In other words,
information (src_pic_width_in_mbs_minus1) that represents
the number of macro blocks of the adaptation target
20 before resolution conversion in the horizontal direction,
information (src_pic_height_in_mbs_minus1) that
represents the number of macro blocks in the vertical
direction, an aspect ratio flag
(src_aspect_ratio_info_present_flag) that represents
25 whether or not aspect information representing an aspect
ratio is included, and the like are described.
[0477]
In a case where the aspect ratio flag represents
that the aspect information is included, the aspect
30 information that is formed by an aspect ratio ID
(src_aspect_ratio_idc) and the like are also included in
165
•
the before-conversion size information. In a case where
the aspect ratio ID included in the aspect information is
an aspect ratio ID (Extended_SAR) given to all the aspect
ratios that are not defined, values (sar_width and
5 sar_height) of the aspect ratio of the adaption target
before resolution conversion in the horizontal and
vertical directions are also included in the aspect
information.
[0478]
10 In addition, in the VUI information of the Subset
SPS illustrated in Fig. 68, points different from those
of the VUI information of the SPS may be described. In
such a case, for information that is not described in the
VUI information of the Subset SPS, information that is
15 included in the VUI information of the SPS is applied.
[0479]
[Example of Description of SEll
Fig. 69 is a diagram that illustrates an example of
the description of the SEI according to the 3DV mode that
20 is illustrated in Fig. 65.
[0480]
The description of the SEI according to the 3DV
mode illustrated in Fig. 69 is the same as the
description of the SEI message of the type of the SEI
25 message except that the type (nesting_type) of the SEI
message, an all-component flag
(all_view_components_in_au_flag), and the like are
described.
[0481]
30 As the types of the SEI message, there are SEI
message according to the MVC mode, SEI message according
166
•
to the 3DV mode, SEI message defined by a user, and the
like. In a case where the all-component flag represents
that adaptation to all the images and the disparity
images within the access unit is not performed, the
5 number of view IDs (num_view_components_minus1) of the
images and the disparity images to which the SEI message
is adapted and view IDs (nesting_sei_view_id)
corresponding to the number are also described in the SEI
according to the 3DV mode.
10 [0482]

[Description of Computer to Which Present Technology Is
Applied]
Next, a series of processes described above may be
15 performed by either hardware or software. In a case
where the series of processes is performed by software, a
program configuring the software is installed to a
general-purpose computer or the like.
[0483]
20
25
Fig. 70 illustrates a configuration example of a
computer according to an embodiment to which a program
that executes the above-described series of processes is
installed.
[0484]
The program may be recorded in a storage unit 808
as a recording medium that is built in the computer or a
ROM (Read Only Memory) 802 in advance.
[0485]
Alternatively, the program may be stored (recorded)
30 on a removable medium 811. Such removable medium 811 may
be provided as so-called package software. Here,
i
167 I
I
•
examples of the removable medium 811 include a flexible
disk, a compact disc read only memory (CD-ROM), a magneto
optical (MO) disk, a digital versatile disc (DVD), a
magnetic disk, and a semiconductor memory.
5 [0486]
In addition to installing the program on the
computer from a removable medium 811 as described above
through a drive 810, the program may be downloaded into
the computer through a communication network or a
10 broadcast network and be installed to the storage unit
808 that is built therein. In other words, the program
may be transmitted to the computer in a wireless manner,
for example, from a download site through a satellite
used for digital satellite broadcasting or be transmitted
15 to the computer in a wired manner through a network such
as a local area network (LAN) or the Internet.
[0487]
The computer includes a CPU (central processing
unit) 801 therein, and an input/output interface 805 is
20 connected to the CPU 801 through a bus 804.
[0488]
When an instruction is input from a user through
the input/output interface 805 by operating an input unit
806, the CPU 801 executes a program that is stored in the
25 ROM 802 in accordance with the instruction.
Alternatively, the CPU 801 loads a program that is stored
in the storage unit 808 into a RAM (random access memory)
803 and executes the program.
[0489]
30 Accordingly, the CPU 801 performs the process
according to the above-described flowchart or the process
168
t,
•
that is performed by the configuration of the abovedescribed
block diagram. Then, the CPU 801 outputs a
processing result from an output unit 807, for example,
through the input/output interface 805, transmits the
5 processing result from a communication unit 809, or
records the processing result in the storage unit 808 as
is needed.
[0490]
Here, the input unit 806 is configured by a
10 keyboard, a mouse, a microphone, and the like. In
addition, the output unit 807 is configured by an LCD
(liquid crystal display), a speaker, and the like.
[0491]
Here, in this specification, the process that is
15 performed by a computer in accordance with a program does
not need to be performed necessarily in a time series in
accordance with the sequence described in the flowchart.
In other words, the process that is performed by the
computer in accordance with the program includes a
20 process (for example, a parallel process or a process
using an object) that is performed in a parallel manner
or in an individual manner.
[0492]
In addition, the program may be processed by one
25 computer (processor) or may be processed by a plurality
of computers in a distributed manner. Furthermore, the
program may be transmitted to a remote computer and be
executed.
[0493]
30 The present technology can be applied to an
encoding device and a decoding device that are used when
169
data is received through a network medium such as
satellite broadcasting, cable TV (television), the
Internet, or a cellular phone or data is processed on a
storage medium such as an optical disc, a magnetic disk,
5 or a flash memory.
[0494]
In addition, the encoding device and the decoding
device described above can be applied to an arbitrary
electronic apparatus. Hereinafter, the examples thereof
10 will be described.
[0495]

[Configuration Example of Television Apparatus]
Fig. 71 illustrates a schematic configuration of a
15 television apparatus according to the present technology
as an example. The television apparatus 900 includes an
antenna 901, a tuner 902, a demultiplexer 903, a decoder
904, a video signal processing unit 905, a display unit
906, an audio signal processing unit 907, a speaker 908,
20 and an external interface unit 909. In addition, the
television apparatus 900 includes a control unit 910, a
user interface unit 911, and the like.
[0496]
The tuner 902 selects a desired channel from among
25 broadcasting signals received by the antenna 901,
demodulates a corresponding broadcasting signal, and
outputs an acquired decoded bit stream to the
demultiplexer 903.
[0497]
30 The demultiplexer 903 extracts packets of a video
and an audio of a program that is a viewing target from
170
•
the encoded bit stream and outputs data of the extracted
packets to the decoder 904. In addition, the
demultiplexer 903 supplies packets of data such as EPG
(electronic program guide) to the control unit 910.
5 Furthermore, in a case where scrambling is performed, the
scrambling is cancelled by the demultiplexer.
[0498]
The decoder 904 performs a decoding process of the
packets, outputs video data generated by the decoding
10 process to the video signal processing unit 905, and
outputs audio data to the audio signal processing unit
907.
[0499]
The video signal processing unit 905 performs noise
15 elimination or video processing according to a user's
setting for the video data. The video signal processing
unit 905 generates video data of a program to be
displayed on the display unit 906, image data acquired by
a process that is based on an application supplied
20 through a network, and the like. In addition, the video
signal processing unit 905 generates video data used for
displaying a menu screen of item selection or the like
and superimposes the generated video data on the video
data of the program. The video signal processing unit
25 905 generates a driving signal based on the video data
generated as above and drives the display unit 906.
[0500]
The display unit 906 drives a display device (for
example, a liquid crystal display device or the like)
30 based on the driving signal supplied from the video
signal processing unit 905, thereby displaying a video of
171
III
•
the program or the like.
[0501]
The audio signal processing unit 907 performs
predetermined processing such as noise elimination for
5 the audio data, performs a D/A conversion process or an
amplification process for the audio data after processing,
and supplies resultant audio data to the speaker 90B,
thereby performing an audio output process.
[0502]
10 The external interface unit 909 is an interface for
a connection with an external device or a network and
performs data transmission or data reception of video
data, audio data, and the like.
[0503]
15 The user interface unit 911 is connected to the
control unit 910. The user interface unit 911 is
configured by an operating switch, a remote control
signal reception unit, and the like and supplies an
operating signal according to a user's operation to the
20 control unit 910.
[0504]
The control unit 910 is configured by a CPU
(central processing unit), a memory, and the like. The
memory stores various kinds of data, EPG data, data
25 acquired through the network, and the like that are
necessary for a program that is executed by the CPU or
the CPU to performing a process. The program stored in
the memory is read out and executed by the CPU at
predetermined timing such as start-up of the television
30 apparatus 900. The CPU controls each unit so as to
enable the television apparatus 900 to perform an
172
•
operation according to a user's operation by executing
the program.
[0505]
In addition, in the television apparatus 900, a bus
5 912 is disposed so as to connect the tuner 902, the
demultiplexer 903, the video signal processing unit 905,
the audio signal processing unit 907, the external
interface unit 909, and the like and the control unit 910
to each other.
10 [0506]
In the television apparatus configured as above,
the function of the decoding device (decoding method)
according to this application is arranged in the decoder
904. Accordingly, multi-viewpoint images can be decoded
15 in accordance with a mode having compatibility with an
existing mode.
[0507]

[Configuration Example of Cellular Phone]
20 Fig. 72 illustrates a schematic configuration of a
cellular phone according to the present technology. The
cellular phone 920 includes a communication unit 922, an
audio codec 923, a camera unit 926, an image processing
unit 927, a demultiplexing unit 928, a
25 recording/reproducing unit 929, a display unit 930, and a
control unit 931. These are interconnected through a bus
933.
[0508]
In addition, an antenna 921 is connected to the
30 communication unit 922, and a speaker 924 and a
microphone 925 are connected to the audio codec 923.
173
•
Furthermore, an operating unit 932 is connected to the
control unit 931.
[0509]
The cellular phone 920 performs various operations
5 such as transmission and reception of an audio signal,
transmission and reception of an electronic mail or image
data, image capturing, and data recording in various
modes such as a voice phone mode and a data communication
mode.
10 [0510]
In the voice phone mode, an audio signal generated
by the microphone 925 is converted into audio data and is
compressed by the audio codec 923, and resultant audio
data is supplied to the communication unit 922. The
15 communication unit 922 performs a modulation process, a
frequency converting process, and the like for the audio
data, thereby generating a transmission signal. In
addition, the communication unit 922 supplies the
transmission signal to the antenna 921, thereby
20 transmitting the transmission signal to a base station
not illustrated in the figure. Furthermore, the
communication unit 922 performs an amplification process,
a frequency converting process, a demodulation process,
and the like for the reception signal received by the
25 antenna 921 and supplies acquired audio data to the audio
codec 923. The audio codec 923 performs data
decompression of the audio data, converts the audio data
into an analog audio signal, and outputs the audio signal
to the speaker 924.
30 [0511]
In the data communication mode, in a case where
174
•
mail transmission is performed, the control unit 931
receives character data input by an operation of the
operating unit 932 and displays the input character on
the display unit 930. In addition, the control unit 931
5 generates mail data based on a user's instruction or the
like supplied from the operating unit 932 and supplies
the generated mail data to the communication unit 922.
The communication unit 922 performs a modulation process,
a frequency converting process, and the like for the mail
10 data and transmits an acquired transmission signal from
the antenna 921. In addition, the communication unit 922
performs an amplification process, a frequency converting
process, a demodulation process, and the like for the
reception signal received by the antenna 921, thereby
15 restoring the mail data. This mail data is supplied to
the display unit 930, whereby the content of the mail is
displayed.
[0512]
In addition, the cellular phone 920 may record the
20 received mail data on a storage medium by using the
recording/reproducing unit 929. The storage medium is an
arbitrary rewritable storage medium. For example, the
storage medium is a semiconductor memory such as a RAM or
a built-in-type flash memory or a removable medium such
25 as a hard disk, a magnetic disk, a magneto-optical disk,
an optical disc, a USB memory, or a memory card.
[0513]
In a case where image data is transmitted in the
data communication mode, the image data generated by the
30 camera unit 926 is supplied to the image processing unit
927. The image processing unit 927 performs an encoding
175
•
process of the image data, thereby generating encoded
data.
[0514]
The demultiplexing unit 928 multiplexes the encoded
5 data generated by the image processing unit 927 and the
audio data supplied from the audio codec 923 in a
predetermined mode and supplies multiplexed data to the
communication unit 922. The communication unit 922
performs a modulation process, a frequency converting
10 process, and the like of the multiplexing data and
transmits an acquired transmission signal from the
antenna 921. In addition, the communication unit 922
performs an amplification process, a frequency converting
process, a demodulation process, and the like for the
15 reception signal received by the antenna 921, thereby
restoring the multiplexed data. This multiplexed data is
supplied to the demultiplexing unit 928. The
demultiplexing unit 928 demultiplexes the multiplexed
data, supplies encoded data to the image processing unit
20 927, and supplies audio data to the audio codec 923. The
image processing unit 927 decodes the encoded data,
thereby generating image data. This image data is
supplied to the display unit 930, and the received image
is displayed. The audio codec 923 converts audio data
25 into an analog audio signal and supplies the analog audio
signal to the speaker 924, thereby outputting the
received audio.
[0515]
In the cellular phone device configured as above,
30 the function of the decoding device (decoding method)
according to this application is arranged in the image
176
10
•
processing unit 927. Accordingly, in communication of
the image data, when decoded image data is generated by
decoding an encoded stream, multi-viewpoint images can be
decoded in accordance with a mode that has compatibility
5 with an existing mode.
[0516J

[Configuration Example of Recording and Reproducing
Device]
Fig. 73 illustrates a schematic configuration of a
recording and reproducing device according to the present
technology. The recording and reproducing device 940,
for example, records audio data and video data of a
received broadcasting program on a recording medium and
15 supplies the recorded data to a user at timing according
to a user's instruction. In addition, the recording and
reproducing device 940, for example, may acquire audio
data and video data from another device and record the
data on a recording medium. Furthermore, the recording
20 and reproducing device 940 decodes and outputs the audio
data and the video data that are recorded on the
recording medium, whereby an image display or audio
output in a monitor device or the like can be performed.
[0517J
25 The recording and reproducing device 940 includes a
tuner 941, an external interface unit 942, an encoder 943,
an HDD (Hard Disk Drive) unit 944, a disk driver 945, a
selector 946, a decoder 947, an OSD (On-Screen Display)
unit 948, a control unit 949, and a user interface unit
30 950.
[0518]
177
•
The tuner 941 selects a desired channel from among
broadcasting signals received by an antenna not
illustrated in the figure. The tuner 941 outputs an
encoded bit stream that is acquired by demodulating a
5 reception signal of the desired channel to the selector
946.
[0519]
The external interface unit 942 is configured as at
least one of an IEEE 1394 interface, a network interface
10 unit, a USB interface, a flash memory interface, and the
like. The external interface unit 942 is an interface
used for a connection with an external device, a network,
a memory card, or the like and performs data reception of
video data, audio data, and the like to be recorded.
15 [0520]
When the video data and the audio data supplied
from the external interface unit 942 are not encoded, the
encoder 943 encodes the data in accordance with a
predetermined mode and outputs the encoded bit stream to
20 the selector 946.
[0521]
The HDD unit 944 records content data such as a
video and an audio, various programs, and other data on a
built-in hard disk and reads out the data from the hard
25 disk at the time of reproducing the data or the like.
[0522]
The disk driver 945 performs signal recording and
signal reproduction for the installed optical disc.
Examples of the optical disc include a DVD disc (a DVD30
Video, a DVD-RAM, a DVD-R, a DVD-RW, a DVD+R, DVD+RW, and
the like), a Blu-ray disc, and the like.
178
•
[0523]
The selector 946 selects one of encoded bit streams
supplied from the tuner 941 or the encoder 943 at the
time of recording a video or audio and supplies the
5 encoded bit stream to one of the HDD unit 944 and the
disk driver 945. In addition, the selector 946 supplies
an encoded bit stream that is output from the HDD unit
944 or the disk driver 945 to the decoder 947 at the time
of reproducing a video or an audio.
10 [0524]
The decoder 947 performs a decoding process of the
encoded bit stream. The decoder 947 supplies the video
data that is generated by performing the decoding process
to the OSD unit 948. In addition, the decoder 947
15 outputs audio data that is generated by performing a
decoding process.
[0525]
The OSD unit 948 generates video data used for
displaying a menu screen of item selection or the like
20 and outputs the generated video data while being
superimposed on the video data output from the decoder
947.
[0526]
The user interface unit 950 is connected to the
25 control unit 949. The user interface unit 950 is
configured by an operating switch, a remote control
signal reception unit, and the like and supplies an
operating signal according to a user's operation to the
control unit 949.
30 [0527]
The control unit 949 is configured by a CPU, a
179
•
memory, and the like. The memory stores various kinds of
data that is necessary for a program executed by the CPU
or the CPU to perform a process. The program that is
stored in the memory is read out and executed by the CPU
5 at predetermined timing such as start-up of the recording
and reproducing device 940. The CPU controls each unit
such that the recording and reproducing device 940
performs an operation according to a user's operation by
executing the program.
10 [0528]
In the recording and reproducing device configured
as above, the function of the encoding device (encoding
method) according to this application is arranged in the
encoder 943. Accordingly, when decoded image data is
15 generated by decoding an encoded stream, multi-viewpoint
images can be encoded in accordance with a mode that has
compatibility with an existing mode.
[0529]

20 [Configuration Example of Imaging Apparatus]
Fig. 74 is a diagram that illustrates a schematic
configuration of an imaging apparatus according to the
present technology. The imaging apparatus 960 images a
subject, displays an image of the subject on the display
25 unit or records the image on the recording medium as
image data.
[0530]
The imaging apparatus 960 includes an optical block
961, an imaging unit 962, a camera signal processing unit
30 963, an image data processing unit 964, a display unit
965, an external interface unit 966, a memory unit 967, a
180
•
medium drive 968, an OSD unit 969, and a control unit 970.
In addition, a user interface unit 971 is connected to
the control unit 970. Furthermore, the image data
processing unit 964, the external interface unit 966, the
5 memory unit 967, the medium drive 968, the OSD unit 969,
the control unit 970, and the like are connected through
a bus 972.
[0531]
The optical block 961 is configured by using a
10 focus lens, a diaphragm mechanism, and the like. The
optical block 961 forms an optical image of a subject on
an imaging face of the imaging unit 962. The imaging
unit 962 is configured by using a CCD or a CMOS image
sensor, generates an electric signal in accordance with
15 an optical image through photoelectric conversion, and
supplies the electric signal to the camera signal
processing unit 963.
[0532]
The camera signal processing unit 963 performs
20 various camera signal processes such as knee correction,
gamma correction, and color correction for the electric
signal supplied from the imaging unit 962. The camera
signal processing unit 963 supplies the image data after
the camera signal processing to the image data processing
25 unit 964.
[0533]
The image data processing unit 964 performs an
encoding process for the image data supplied from the
camera signal processing unit 963. The image data
30 processing unit 964 supplies the encoded data generated
by performing the encoding process to the external
181
5
10
•
interface unit 966 or the medium drive 968. In addition,
the image data processing unit 964 performs a decoding
process for the encoded data that is supplied from the
external interface unit 966 or the medium drive 968. The
image data processing unit 964 supplies the image data
that is generated by performing the decoding process to
the display unit 965. In addition, the image data
processing unit 964 supplies display data that is
acquired by the process of supplying the image data
supplied from the camera signal processing unit 963 to
the display unit 965 or is acquired from the OSD unit 969
to the display unit 965 while being superimposed on the
image data.
[0534]
15 The OSD unit 969 generates display data of a menu
screen, an icon, or the like that is formed from a symbol,
a character, or a graphic and outputs the display data to
the image data processing unit 964.
[0535]
20 The external interface unit 966, for example, is
configured by USB input/output terminals and the like and
is connected to a printer in a case where an image is
printed. In addition, a drive is connected to the
external interface unit 966 as is necessary, a removable
25 medium such as a magnetic disk or an optical disc is
appropriately installed therein, and a computer program
read from the removable medium is installed as is
necessary. In addition, the external interface unit 966
includes a network interface that is connected to a
30 predetermined network such as a LAN or the Internet. The
control unit 970, for example, in accordance with an
182
r
•
instruction supplied from the user interface unit 971,
can read encoded data from the memory unit 967 and supply
the encoded data from the external interface unit 966 to
another device that is connected through the network. In
5 addition, the control unit 970 can acquire the encoded
data or the image data, which is supplied from another
device through the network, through the external
interface unit 966 and supply the data to the image data
processing unit 964.
10 [0536]
As the recording medium that is driven by the
medium drive 968, for example, an arbitrary readable and
writable removable medium such as a magnetic disk, a
magneto-optical disk, an optical disc, or a semiconductor
15 memory is used. In addition, the type of the recording
medium as the removable medium is arbitrary and may be a
tape device, a disk device, or a memory card.
Furthermore, the recording medium may be a non-contact IC
card, or the like.
20 [0537]
In addition, the medium drive 968 and the recording
medium may be integrated together and configured by a
non-portable recording medium such as a built-in type
hard disk drive or an SSD (Solid State Drive).
25 [0538]
The control unit 970 is configured by a CPU, a
memory, and the like. The memory stores various kinds of
data that is necessary for a program executed by the CPU
or the CPU to perform a process. The program that is
30 stored in the memory is read out and executed by the CPU
at predetermined timing such as start-up of the imaging
183
•
apparatus 960. The CPU controls each unit such that the
imaging apparatus 960 performs an operation according to
a user's operation by executing the program.
[0539]
5 In the imaging apparatus configured as above, the
function of the decoding device (decoding method)
according to this application is arranged in the image
data processing unit 964. Accordingly, when decoded
image data is generated by decoding encoded data that is
10 recorded in a memory unit 967, a recording medium, or the
like, multi-viewpoint images can be decoded in accordance
with a mode that has compatibility with an existing mode.
[0540]
In addition, an embodiment of the present
15 technology is not limited to the above-described
embodiments, and various changes can be made therein in a
range not departing from the concept of the present
technology.
20 REFERENCE SIGNS LIST
[0541]
50 Encoding device
52 Image converting unit
53 Disparity image generating unit
25 55 Compatibility information generating unit
56 Inter-viewpoint distance information generating
unit
30
59
61
62
120
Multiplexing unit
Compatible encoder
Auxiliary encoder
Decoding device
184
•
121 Separation unit
127 Image generating unit
131 Compatible decoder
132 Auxiliary decoder
5 140 Encoding device
142 Image converting unit
143 Disparity image generating unit
144 Inter-viewpoint distance information generating
unit
10 151 Compatible encoder
152 Auxiliary encoder
170 Decoding device
171 Image generating unit
180 Encoding device
15 181 Image converting unit
191 Compatible encoder
200 Decoding device
202 Image Generating Unit
211 Compatible Decoder

•
CLAIMS
1. An encoding device comprising:
a compatible image encoding unit that generates a
5 first encoded stream by designating a compatible image
out of multi-viewpoint images and encoding the designated
compatible image;
an image converting unit that converts resolution
of an auxiliary image used when multi-viewpoint images
10 are generated from the compatible image;
an auxiliary image encoding unit that generates a
second encoded stream by encoding the auxiliary image
converted by the image converting unit;
a setting unit that sets compatible image
15 designating information designating the compatible image;
and
a transmission unit that transmits the first
encoded stream generated by the compatible image encoding
unit, the second encoded stream generated by the
20 auxiliary image encoding unit, and the compatible image
designating information set by the setting unit.
2. The encoding device according to claim 1, further
comprising a disparity image encoding unit that generates
25 a first disparity encoded stream by encoding a compatible
disparity image that is a disparity image of the
compatible image and generates a second disparity encoded
stream by encoding an auxiliary disparity image that is a
disparity image of the auxiliary image,
30 wherein the transmission unit transmits the first
disparity encoded stream and the second disparity encoded
186
stream that are generated by the disparity image encoding
unit.
3. The encoding device according to claim 2,
5 wherein the image converting unit converts the
resolution of the auxiliary disparity image, and
the disparity image encoding unit encodes the
auxiliary disparity image of which the resolution is
converted by the image converting unit.
10
4. The encoding device according to claim 3, further
comprising a multiplexing unit that multiplexes the
auxiliary disparity image of which the resolution is
converted by the image converting unit,
15 wherein the setting unit sets multiplexing mode
information that represents a multiplexing mode of the
auxiliary disparity image,
the disparity image encoding unit encodes the
auxiliary disparity image multiplexed by the multiplexing
20 unit, and
the transmission unit transmits the multiplexing
mode information set by the setting unit.
5. The encoding device according to claim 3,
25 wherein the setting unit sets conversion mode
information that represents a conversion mode of the
resolution of the auxiliary disparity image, and
the transmission unit transmits the conversion mode
information set by the setting unit.
30
6. The encoding device according to claim 3, further
187
comprising a multiplexing unit that multiplexes the
compatible disparity image in a time direction,
wherein the setting unit sets multiplexing mode
information that represents a multiplexing mode of the
5 compatible disparity image,
the disparity image encoding unit encodes the
compatible disparity image multiplexed ~n the time
direction by the multiplexing unit, and
the transmission unit transmits the multiplexing
10 mode information set by the setting unit.
7. The encoding device according to claim 6,
wherein the multiplexing unit multiplexes the
auxiliary disparity image in a spatial direction,
15 the setting unit sets the multiplexing mode
information of the compatible disparity image and the
auxiliary disparity image,
the disparity image encoding unit encodes the
compatible disparity image multiplexed in the time
20 direction by the multiplexing unit and the auxiliary
disparity image multiplexed in the spatial direction, and
the transmission unit transmits the multiplexing
mode information set by the setting unit.
25 8. The encoding device according to claim 2, wherein
the compatible disparity image is a disparity image that
is common to the compatible images.
9. The encoding device according to claim 2, further
30 comprising a multiplexing unit that multiplexes the
auxiliary image and the compatible disparity image in a
188
time direction,
wherein the setting unit sets multiplexing mode
information that represents a multiplexing mode in the
multiplexing unit,
5 the auxiliary image encoding unit encodes the
auxiliary image multiplexed by the multiplexing unit,
the disparity image encoding unit encodes the
compatible disparity image multiplexed by the
multiplexing unit, and
10 the transmission unit transmits the multiplexing
mode information set by the setting unit.
10. The encoding device according to claim 1, wherein
the setting unit sets distance information that
15 represents a distance between viewpoints of the
compatible image and the auxiliary image, and
the transmission unit transmits the distance
information set by the setting unit.
20 11. The encoding device according to claim 1,
wherein the image converting unit multiplexes the
auxiliary image of which the resolution is converted,
the setting unit sets auxiliary image multiplexing
mode information that represents a multiplexing mode of
25 the auxiliary image,
the auxiliary image encoding unit encodes the
auxiliary image multiplexed by the image converting unit,
and
the transmission unit transmits the auxiliary image
30 multiplexing mode information set by the setting unit.
189
12. The encoding device according to claim 11,
wherein the setting unit sets auxiliary image
resolution converting mode information that represents a
conversion mode of the resolution of the auxiliary image,
5 and
the transmission unit transmits the auxiliary image
resolution converting mode information set by the setting
unit.
10 13. An encoding method to be performed by an encoding
device, the encoding method comprising:
a compatible image encoding step of generating a
first encoded stream by designating a compatible image
out of multi-viewpoint images and encoding the designated
15 compatible image;
an image converting step of converting resolution
of an auxiliary image used when multi-viewpoint images
are generated from the compatible image;
an auxiliary image encoding step of generating a
20 second encoded stream by encoding the auxiliary image
converted in a process of the image converting step;
a setting step of setting compatible image
designating information designating the compatible image;
and
25 a transmitting step of transmitting the first
encoded stream generated in a process of the compatible
image encoding step, the second encoded stream generated
in a process of the auxiliary image encoding step, and
the compatible image designating information set in a
30 process of the setting step.
190
14. A decoding device comprising:
a reception unit that receives a first encoded
stream that is acquired as a result of encoding a
compatible image designated out of multi-viewpoint images,
5 a second encoded stream that is acquired as a result of
converting resolution of an auxiliary image used when a
multi-viewpoint image is generated from the compatible
image and encoding the auxiliary image, and compatible
image designating information that designates the
10 compatible image;
a compatible image decoding unit that identifies
the first encoded stream based on the compatible image
designating information and decodes the first encoded
stream;
15 an auxiliary image decoding unit that decodes the
second encoded stream; and
an image converting unit that converts the
resolution of the auxiliary image, of which the
resolution is converted, that is acquired as a result of
20 decoding by the auxiliary image decoding unit.
15. The decoding device according to claim 14, further
comprising a disparity image decoding unit that decodes a
first disparity encoded stream that is acquired as a
25 result of encoding a compatible disparity image that is a
disparity image of the compatible image and a second
disparity encoded stream that is acquired as a result of
encoding an auxiliary disparity image that is a disparity
image of the auxiliary image,
30 wherein the reception unit receives the first
disparity encoded stream and the second disparity encoded
191
stream.
16. The decoding device according to claim 15,
wherein the second disparity encoded stream is
5 acquired as a result of encoding the auxiliary disparity
image of which the resolution is converted, and
the image converting unit converts the resolution
of the auxiliary disparity image, of which the resolution
is converted, that is acquired as a result of decoding by
10 the disparity image decoding unit.
17. The decoding device according to claim 16, further
comprising a separation unit that separates the
multiplexed auxiliary disparity image,
15 wherein the second disparity encoded stream is
acquired as a result of encoding the multiplexed
auxiliary disparity image of which the resolution is
converted,
the reception unit receives multiplexing mode
20 information that represents a multiplexing mode of the
auxiliary disparity image,
the separation unit separates the multiplexed
auxiliary disparity image, of which the resolution is
converted, that is acquired as a result of decoding by
25 the disparity image decoding unit based on the
multiplexing mode information received by the reception
unit, and
the image converting unit converts the resolution
of the auxiliary disparity image, of which the resolution
30 is converted, that is acquired as a result of separation
by the separation unit.
192
18. The decoding device according to claim 16,
wherein the reception unit receives conversion mode
information that represents a conversion mode of the
5 resolution of the auxiliary disparity image, and
the image converting unit converts the resolution
of the auxiliary disparity image, of which the resolution
is converted, based on the conversion mode information
received by the reception unit.
10
19. The decoding device according to claim 16, further
comprising a separation unit that separates the
compatible disparity image multiplexed in a time
direction,
15 wherein the first disparity encoded stream is
acquired as a result of encoding the compatible disparity
image multiplexed in the time direction,
the reception unit receives multiplexing mode
information that represents a multiplexing mode of the
20 compatible disparity image, and
the separation unit separates the compatible
disparity image multiplexed in the time direction that is
acquired as a result of decoding by the disparity image
decoding unit based on the multiplexing mode information
25 received by the reception unit.
20. The decoding device according to claim 19,
wherein the second disparity encoded stream is
acquired as a result of encoding the auxiliary disparity
30 image, of which the resolution is converted, multiplexed
in a spatial direction,
193
the reception unit receives the multiplexing mode
information of the compatible disparity image and the
auxiliary disparity image,
the separation unit separates the compatible
5 disparity image, which is multiplexed in the time
direction, acquired as a result of decoding by the
disparity image decoding unit based on the multiplexing
mode information of the compatible disparity image that
is received by the reception unit and separates the
10 auxiliary disparity image, of which the resolution is
converted, multiplexed in the spatial direction that is
acquired as a result of decoding by the disparity image
decoding unit based on the multiplexing mode information
of the auxiliary disparity image received by the
15 reception unit, and
the image converting unit converts the resolution
of the auxiliary disparity image, of which the resolution
is converted, separated by the separation unit.
20 21. The decoding device according to claim 15, wherein
the compatible disparity image is a disparity image that
is common to the compatible images.
22. The decoding device according to claim 15, further
25 comprising a separation unit that separates the auxiliary
image and the compatible disparity image that are
multiplexed in a time direction,
wherein the second encoded stream is acquired as a
result of encoding the auxiliary image, of which the
30 resolution is converted, multiplexed in the time
direction,
194
the first disparity encoded stream is acquired as a
result of encoding the compatible disparity image
multiplexed in the time direction,
the reception unit receives multiplexing mode
5 information that represents multiplexing modes of the
auxiliary image and the compatible disparity image,
the separation unit, based on the multiplexing mode
information, separates the auxiliary image, of which the
resolution is converted, multiplexed in the time
10 direction and acquired as a result of decoding by the
auxiliary image decoding unit, and separates the
compatible disparity image multiplexed in the time
direction and acquired as a result of decoding by the
disparity image decoding unit, and
15 the image converting unit converts the resolution
of the auxiliary image, of which the resolution is
converted, separated by the separation unit.
23. The decoding device according to claim 14, further
20 comprising a generation unit that generates the multiviewpoint
image using the compatible image acquired as a
result of decoding by the compatible image decoding unit
and the auxiliary image of which the resolution is
converted by the image converting unit,
25 wherein the reception unit receives distance
information that represents a distance between viewpoints
of the compatible image and the auxiliary image, and
the generation unit generates the multi-viewpoint
image by using the compatible image, the auxiliary image,
30 and the distance information received by the reception
unit.
195
24. The decoding device according to claim 14, further
comprising a separation unit that separates the
multiplexed auxiliary image,
5 wherein the second encoded stream is acquired as a
result of encoding the multiplexed auxiliary image, of
which the resolution is converted,
the reception unit receives auxiliary image
multiplexing mode information that represents a
10 multiplexing mode of the auxiliary image,
the separation unit separates the multiplexed
auxiliary image, of which the resolution is converted,
that is acquired as a result of decoding by the auxiliary
image decoding unit based on the auxiliary image
15 multiplexing mode information received by the reception
unit, and
the image converting unit converts the resolution
of the auxiliary image, of which the resolution is
converted, separated by the separation unit.
20
25. The decoding device according to claim 24,
wherein the reception unit receives auxiliary image
resolution converting mode information that represents a
conversion mode of the resolution of the auxiliary image,
25 and
the image converting unit converts the resolution
of the auxiliary image, of which the resolution is
converted, based on the auxiliary image resolution
converting mode information received by the reception
30 unit.
196
26. A decoding method to be performed by a decoding
device, the decoding method comprising:
a receiving step of receiving a first encoded
stream that is acquired as a result of encoding a
5 compatible image designated from multi-viewpoint images,
a second encoded stream that is acquired as a result of
converting resolution of an auxiliary image used when a
multi-viewpoint image is generated from the compatible
image and encoding the auxiliary image, and compatible
10 image designating information that designates the
compatible image;
a compatible image decoding step of identifying the
first encoded stream based on the compatible image
designating information and decoding the first encoded
15 stream;
an auxiliary image decoding step of decoding the
second encoded stream; and
an image converting step of converting the
resolution of the auxiliary image, of which the
20 resolution is converted, that is acquired as a result of
decoding in a process of the auxiliary image decoding
step.
Dated this February 25, 2013
DEEPAKKUMAR
OF REMFRY & SAGAR
197