DESCRIPTION
ENCODING DEVICE, ENCODING METHOD, DECODING DEVICE, AND
DECODING METHOD
5 TECHNICAL FIELD
[0001]
The present technology relates to an encoding
device, an encoding method, a decoding device, and a
decoding method, and more particularly, to an encoding
10 device, an encoding method, a decoding device, and a
decoding method capable of suppressing the amount of
information without degrading the image quality of an
image of a viewpoint that is used when the compatibility
is secured, particularly, in a case where a multi15
viewpoint image is encoded.
BACKGROUND ART
[0002]
Currently, as a 3D image viewing mode, generally,
20 there is a mode (hereinafter, referred to as a twoviewpoint
mode) in which two-viewpoint images alternately
displayed are seen by wearing glasses of which a left-eye
shutter is open at the time of displaying one image out
of two-viewpoint images, and a right-eye shutter is open
25 at the time of displaying the other image.
[0003]
However, in such a two-viewpoint mode, a viewer
needs to purchase glasses in addition to a 3D image
display device, and accordingly, the viewer's willingness
30 to buy reduces. In addition, since a viewer needs to
wear glasses for viewing, it annoys the viewer.
1
Accordingly, a demand for a viewing mode (hereinafter,
referred to as a multi-viewpoint mode) increases in which
a 3D image can be viewed without wearing glasses.
[0004]
5 In the multi-viewpoint mode, multi-viewpoint images
are displayed such that a viewable angle is different for
each viewpoint, and, a 3D image can be seen by a viewer
viewing images of arbitrary two viewpoints with left and
right eyes without wearing glasses.
10 [0005]
A display device that provides viewing of a multiviewpoint
mode, for example, generates multi-viewpoint
images for a multi-viewpoint mode based on images of two
viewpoints for a two-viewpoint mode and displays the
15 generated multi-viewpoint images. More specifically, the
display device acquires disparity (depth) of twoviewpoint
images for a two-viewpoint mode using an image
disparity estimating technology (Depth Estimation). Then,
the display device generates a synthesized image of
20 multi-viewpoint images adjacent to a viewpoint
corresponding to the images of two viewpoints for a twoviewpoint
mode using a multi-viewpoint image generating
technology (View Generation) using the disparity between
images of two viewpoints and a synthesis technology (View
25 Synthesis) and displays the synthesized image.
[0006]
Existing encoding modes include an advanced video
coding (AVC) mode and a multi-view video coding (MVC)
mode.
30 [0007]
Fig. 1 is a diagram that illustrates an example of
2
an image processing system that encodes a 2D image in
accordance with the Ave mode and decodes the encoded
image.
[0008]
5 The image processing system 10 illustrated in Fig.
1 is configured by an imaging unit 11, an Ave encoder 12,
and an Ave decoder 13.
[0009]
The imaging unit 11 of the image processing system
10 10 captures an image A of a predetermined viewpoint and
supplies the captured image to the AVe encoder 12. The
AVe encoder 12 encodes the image A that is supplied from
the imaging unit 11 in accordance with the Ave mode and
supplies the encoded image to the Ave decoder 13. The
15 Ave decoder 13 decodes the image A after encoding that is
supplied from the Ave encoder 12 in accordance with a
mode that corresponds to the Ave mode. The Ave decoder
13 outputs the image A that is acquired as a result of
the decoding process, thereby displaying the image. In
20 this way, the viewer can see a 2D image.
[0010]
Fig. 2 is a diagram that illustrates an example of
an image processing system that encodes an image of two
viewpoints in accordance with the Ave mode and decodes
25 the image.
[0011]
The image processing system 20 illustrated in Fig.
2 is configured by an imaging unit 21A, an imaging unit
21B, an Ave encoder 22, and an Ave decoder 23.
30 [0012]
The imaging unit 21A captures an image A of a
3
predetermined viewpoint, halves the resolution of the
image, and supplies a resultant image to the Ave encoder
22 as a left half image of the screen. In addition, the
imaging unit 21B captures an image B of a viewpoint that
5 is different from the viewpoint of the image A, halves
the resolution of the image, and supplies a resultant
image to the Ave encoder 22 as a right half image of the
screen. The AVe encoder 22 encodes an image of which the
left half is configured by the image A (hereinafter
10 referred to as a half-resolution image A) of which the
resolution has halved and of which the right half is
configured by an image B (hereinafter, referred to as a
half-resolution image B) of which the resolution has
halved in accordance with the Ave mode and supplies the
15 encoded image to the Ave decoder 23.
[0013]
The AVe decoder 23 decodes the images after
encoding that are supplied from the Ave encoder 22 in
accordance with a mode that corresponds to the Ave mode.
20 The Ave decoder 23 outputs images acquired as a result of
the decoding process and alternately displays the halfresolution
images A and B. At this time, a viewer, for
example, wears glasses of which the left-eye shutter is
open at the time of displaying the half-resolution image
25 A and of which the right-eye shutter is open at the time
of displaying the half-resolution image B, views the
half-resolution image A only with the left eye, and views
the half-resolution image B only with the right eye. In
this way, the viewer can see a 3D image.
30 [0014]
Fig. 3 is a diagram that illustrates an image
4
processing system that encodes an image of two viewpoints
in accordance with the MVC mode and decodes the encoded
image.
[0015]
5 The image processing system 30 illustrated in Fig.
3 is configured by an imaging unit 31A, an imaging unit
31B, an MVC encoder 32, and an MVC decoder 33.
[0016]
The imaging unit 31A captures an image A of a
10 predetermined viewpoint and supplies the captured image
to the MVC encoder 32. In addition, the imaging unit 31B
captures an image B of a viewpoint that is different from
the viewpoint of the image A and supplies the captured
image to the MVC encoder 32. The MVC encoder 32 encodes
15 the image A that is supplied from the imaging unit 31A as
a base image in accordance with the AVC mode and decodes
the image B that is supplied from the imaging unit 31B as
a dependent image in accordance with the MVC mode. The
MVC encoder 32 supplies the images A and B after encoding
20 to the MVC decoder 33.
[0017]
The MVC decoder 33 decodes the images A and B after
encoding that are supplied from the MVC encoder 32 in
accordance with a mode that corresponds to the MVC mode.
25 The MVC decoder 33 outputs the images A and B that are
acquired as a result of the decoding process and
alternately displays the images. At this time, a viewer,
for example, wears glasses of which the left-eye shutter
is open at the time of displaying the image A and of
30 which the right-eye shutter is open at the time of
displaying the image B, views the image A only with the
5
left eye, and views the image B only with the right eye.
In this way, the viewer can see a 3D image.
[0018]
Fig. 4 is a diagram that illustrates an example of
5 an image processing system that generates an image of
multi-viewpoints by encoding an image of two viewpoints
in accordance with the MVC mode and decoding the image.
[0019]
The image processing system 40 illustrated in Fig.
10 4 is configured by imaging units 31A and 31B, an MVC
encoder 32, an MVC decoder 33, a disparity detecting unit
41, and an image generating unit 42. In Fig. 4, the same
reference numeral is assigned to a configuration that is
the same as that illustrated in Fig. 3, and description
15 thereof will not be appropriately presented.
[0020]
Images A and B that are acquired as a result of
decoding performed by the MVC decoder 33 are supplied to
the disparity detecting unit 41 of the image processing
20 system 40. The disparity detecting unit 41 detects the
disparity of each pixel of the images A and B using the
images A and B. The disparity detecting unit 41
generates a disparity image A' that represents the
disparity of each pixel of the image A and a disparity
25 image B' that represents the disparity of each pixel of
the image B and supplies the generated disparity images
to the image generating unit 42. The image generating
unit 42 generates an image of multi-viewpoints of which
the number corresponding to a display device not
30 illustrated in the figure by using the images A and B
acquired as a result of decoding performed by the MVC
6
decoder 33 and the disparity images A' and B' supplied
from the image generating unit 42. Then, the image
generating unit 42 converts the resolution of the
generated image of each viewpoint into ~1/ the number of
5 viewpoints" of the resolution, synthesizes the images,
and outputs the synthesized image so as to be displayed.
[0021]
At this time, the multi-viewpoint images after the
synthesis are displayed such that the viewing angles are
10 different for each viewpoint, and a viewer can view a 3D
image without wearing glasses by seeing images of
arbitrary two viewpoints with his left and right eyes.
[0022]
Meanwhile, a mode for encoding a multi-viewpoint
15 image has also been devised (for example, see Patent
Document 1).
CITATION LIST
PATENT DOCUMENT
20 [0023]
Patent Document 1: Japanese Patent Application Laid-Open
No. 2008-182669
SUMMARY OF THE INVENTION
25 PROBLEMS TO BE SOLVED BY THE INVENTION
[0024]
However, there has been no mode for encoding a
multi-viewpoint image, which suppresses the amount of
information without degrading the image quality of an
30 image (for example, an image of one viewpoint or images
of two viewpoints) of a viewpoint that is used when the
7
compatibility is secured.
[0025]
The present technology has been contrived in
consideration of such situations, and, the amount of
5 information can be suppressed without degrading the image
quality of an image of a viewpoint that is used when the
compatibility is secured in a case where a multiviewpoint
image is encoded.
10 SOLUTIONS TO PROBLEMS
[0026]
According to a first aspect of the present
technology, there is provided an encoding device
including: a compatible image encoding unit that
15 generates a first encoded stream by designating a
compatible image from a multi-viewpoint image and
encoding the designated compatible stream; an image
converting unit that lowers the resolution of an
auxiliary image that is used when the multi-viewpoint
20 image is generated from the compatible image; an
auxiliary image encoding unit that generates a second
encoded stream by encoding the auxiliary image of which
the resolution is lowered by the image converting unit;
and a transmission unit that transmits the first encoded
25 stream generated by the compatible image encoding unit
and the second encoded stream generated by the auxiliary
image encoding unit.
[0027]
An encoding method according to the first aspect of
30 the present technology corresponds to the encoding device
according to the first aspect of the present technology.
8
[0028J
In the first aspect of the present technology, a
compatible image is designated from a multi-viewpoint
image, a first encoded stream is generated by encoding
5 the designated compatible stream, the resolution of an
auxiliary image that is used when the multi-viewpoint
image is generated from the compatible image is lowered,
the auxiliary image of which the resolution is lowered is
encoded, a second encoded stream is generated, and the
10 first encoded stream and the second encoded stream are
transmitted.
[0029J
According to a second aspect of the present
technology, there is provided a decoding device
15 including: a reception unit that receives a first encoded
stream that is acquired as a result of encoding a
compatible image designated from a multi-viewpoint and a
second encoded stream that is acquired as a result of
lowering the resolution of an auxiliary image used when
20 the multi-viewpoint image is generated from the
compatible image and encoding the auxiliary image; a
compatible image decoding unit that decodes the first
encoded stream received by the reception unit; an
auxiliary image decoding unit that decodes the second
25 encoded stream received by the reception unit; and an
image converting unit that increases the resolution of
the auxiliary image, of which the resolution is lowered,
acquired as a result of decoding by the auxiliary image
decoding unit.
30 [0030J
A decoding method according to the second aspect of
9
the present technology corresponds to the decoding device
according to the second aspect of the present technology.
[0031]
In the second aspect of the present technology, a
5 first encoded stream that is acquired as a result of
encoding a compatible image designated from a multiviewpoint
and a second encoded stream that is acquired as
a result of lowering the resolution of an auxiliary image
used when the multi-viewpoint image is generated from the
10 compatible image and encoding the auxiliary image are
received, the first encoded stream is decoded, the second
encoded stream is decoded, and the resolution of the
auxiliary image, of which the resolution is lowered,
acquired as a result of decoding is increased.
15 [0032]
In addition, the encoding device according to the
first aspect and the decoding device according to the
second aspect may be realized by causing a computer to
execute a program.
20 [0033]
Furthermore, the program that is executed by a
computer so as to realize the encoding device according
to the first aspect and the decoding device according to
the second aspect may be provided while being transmitted
25 through a transmission medium or being recorded on a
recording medium.
EFFECTS OF THE INVENTION
[0034]
30 According to the first aspect of the present
technology, the amount of information can be suppressed
10
•
without degrading the image quality of an image of a
viewpoint that is used when the compatibility is secured
in a case where a multi-viewpoint image is encoded.
[0035]
5 In addition, according to the second aspect of the
present technology, a multi-viewpoint image that is
encoded so as to suppress the amount of information
without degrading the image quality of an image of a
viewpoint used when the compatibility is secured can be
10 decoded.
BRIEF DESCRIPTION OF DRAWINGS
[0036]
Fig. 1 is a diagram that illustrates an example of
15 an image processing system that encodes a 2D image in
accordance with an AVC mode and decodes the encoded image.
Fig. 2 is a diagram that illustrates an example of
an image processing system that encodes an image of two
viewpoints in accordance with an AVC mode and decodes the
20 encoded image.
Fig. 3 is a diagram that illustrates an example of
an image processing system that encodes an image of two
viewpoints in accordance with an MVC mode and decodes the
encoded image.
25 Fig. 4 is a diagram that illustrates an example of
an image processing system that generates an image of
multi-viewpoints by encoding an image of two viewpoints
in accordance with the MVC mode and decoding the encoded
image.
30 Fig. 5 is a block diagram that illustrates a
configuration example of an encoding device according to
11
•
a first embodiment of the present technology.
Fig. 6 is a diagram that illustrates an example of
the description of compatibility information and
disparity image information.
5 Fig. 7 is a diagram that illustrates a detailed
example of the description of the compatibility
information.
Fig. 8 is a diagram that illustrates a detailed
example of the description of the disparity image
10 information.
Fig. 9 is a diagram that illustrates an example of
the description of inter-viewpoint distance information.
Fig. 10 is a flowchart that illustrates an encoding
process performed by the encoding device illustrated in
15 Fig. 5.
Fig. 11 is a flowchart that illustrates an encoding
process performed by the encoding device illustrated in
Fig. 5.
Fig. 12 is a diagram that illustrates a
20 configuration example of a decoding device that
corresponds to the encoding device illustrated in Fig. 5.
Fig. 13 is a flowchart that illustrates a decoding
process performed by the decoding device illustrated in
Fig. 12.
25 Fig. 14 is a block diagram that illustrates a
configuration example of an encoding device according to
a second embodiment of the present technology.
Fig. 15 is a flowchart that illustrates an encoding
process performed by the encoding device illustrated in
30 Fig. 14.
Fig. 16 is a flowchart that illustrates an encoding
12
process performed by the encoding device illustrated in
Fig. 14.
Fig. 17 is a diagram that illustrates a
configuration example of a decoding device that
5 corresponds to the encoding device illustrated in Fig. 14.
Fig. 18 is a flowchart that illustrates a decoding
process performed by the decoding device illustrated in
Fig. 17.
Fig. 19 is a block diagram that illustrates a
10 configuration example of an encoding device according to
a third embodiment of the present technology.
Fig. 20 is a flowchart that illustrates an encoding
process performed by the encoding device illustrated in
Fig. 19.
15 Fig. 21 is a flowchart that illustrates the
encoding process performed by the encoding device
illustrated in Fig. 19.
Fig. 22 is a diagram that illustrates a
configuration example of a decoding device that
20 corresponds to the encoding device illustrated in Fig. 19.
Fig. 23 is a flowchart that illustrates a decoding
process performed by the decoding device illustrated in
Fig. 22.
Fig. 24 is a diagram that illustrates another
25 example of the description of compatibility information
and disparity image information.
Fig. 25 is a diagram that illustrates another
detailed example of the description of the compatibility
information.
30 Fig. 26 is a diagram that illustrates another
detailed example of the description of the disparity
13
image information.
Fig. 27 is a diagram that illustrates another
example of the description of the inter-viewpoint
distance information.
5 Fig. 28 is a block diagram that illustrates a
configuration example of an encoding device according to
a fourth embodiment of the present technology.
Fig. 29 is a diagram that illustrates a further
another example of the description of the disparity image
10 information.
Fig. 30 is a flowchart that illustrates an encoding
process performed by the encoding device illustrated in
Fig. 28.
Fig. 31 is a flowchart that illustrates the
15 encoding process performed by the encoding device
illustrated in Fig. 28.
Fig. 32 is a diagram that illustrates a
configuration example of a decoding device that
corresponds to the encoding device illustrated in Fig. 28.
20 Fig. 33 is a flowchart that illustrates a decoding
process performed by the decoding device illustrated in
Fig. 32.
Fig. 34 is a diagram that illustrates further
another example of the description of the disparity image
25 information.
Fig. 35 is a block diagram that illustrates a
configuration example of an encoding device according to
a fifth embodiment of the present technology.
Fig. 36 is a diagram that illustrates a detailed
30 example of the description of the disparity image
information.
14
I
•
Fig. 37 is a diagram that illustrates a detailed
example of the description of the disparity image
information.
Fig. 38 is a flowchart that illustrates an encoding
5 process performed by the encoding device illustrated in
Fig. 35.
Fig. 39 is a flowchart that illustrates the
encoding process performed by the encoding device
illustrated in Fig. 35.
10 Fig. 40 is a diagram that illustrates a
configuration example of a decoding device that
corresponds to the encoding device illustrated in Fig. 35.
Fig. 41 is a flowchart that illustrates a decoding
process performed by the decoding device illustrated in
15 Fig. 40.
Fig. 42 is a block diagram that illustrates a
configuration example of an encoding device according to
a sixth embodiment of the present technology.
Fig. 43 is a flowchart that illustrates an encoding
20 process performed by the encoding device illustrated in
Fig. 42.
Fig. 44 is a flowchart that illustrates the
encoding process performed by the encoding device
illustrated in Fig. 42.
25 Fig. 45 is a diagram that illustrates a
configuration example of a decoding device that
corresponds to the encoding device illustrated in Fig. 42.
Fig. 46 is a flowchart that illustrates a decoding
process performed by the decoding device illustrated in
30 Fig. 45.
Fig. 47 is a diagram that illustrates examples of a
15
multiplexed pattern of an encoding target.
Fig. 48 is a diagram that illustrates features of
the effect according to multiplexing.
Fig. 49 is a diagram that illustrates other
5 examples of the multiplexed pattern of the encoding
target.
Fig. 50 is a diagram that illustrates further other
examples of the multiplexed pattern of the encoding
target.
10 Fig. 51 is a block diagram that illustrates a
configuration example of an encoding device according to
a seventh embodiment of the present technology.
Fig. 52 is a flowchart that illustrates an encoding
process performed by the encoding device illustrated in
15 Fig. 5l.
Fig. 53 is a flowchart that illustrates the
encoding process performed by the encoding device
illustrated in Fig. 51.
Fig. 54 is a flowchart that illustrates an encoding
20 process performed by the encoding device illustrated in
Fig. 5l.
Fig. 55 is a flowchart that illustrates an encoding
process performed by the encoding device illustrated in
Fig. 54.
25 Fig. 56 is a diagram that illustrates a
configuration example of a bit stream.
Fig. 57 is a diagram that illustrates an example of
the description of a Subset SPS according to the 3DV mode
for a disparity image illustrated in Fig. 56.
30 Fig. 58 is a diagram that illustrates an example of
the description of extension information for a disparity
16
image that is illustrated in Fig. 57.
Fig. 59 is a diagram that illustrates an example of
the description of VUl extension information for a
disparity image illustrated in Fig. 57.
5 Fig. 60 is a diagram that illustrates an example of
the description of an NAL header of encoded data
according to the 3DV mode.
Fig. 61 is a diagram that illustrates an example of
the description of disparity image header extension
10 information illustrated in Fig. 60.
Fig. 62 is a diagram that illustrates an example of
the description of SEl according to the 3DV mode that is
illustrated in Fig. 56.
Fig. 63 is a diagram that illustrates another
15 configuration example of the bit stream.
Fig. 64 is a diagram that illustrates an example of
the description of a Subset SPS according to the 3DV mode
for an image that is illustrated in Fig. 63.
Fig. 65 is a diagram that illustrates a
20 configuration example of a bit stream in a case where the
3DV mode is a mode that is compliant with an HEVC mode.
Fig. 66 is a diagram that illustrates an example of
the description of an SPS illustrated in Fig. 65.
Fig. 67 is a diagram that illustrates an example of
25 the description of the Subset SPS illustrated in Fig. 66.
Fig. 68 is a diagram that illustrates an example of
the description of VUl information of the Subset SPS
illustrated in Fig. 67.
Fig. 69 is a diagram that illustrates an example of
30 the description of SEl according to the 3DV mode that is
illustrated in Fig. 65.
17
Fig. 70 is a diagram that illustrates a
configuration example of a computer according to an
embodiment.
Fig. 71 is a diagram that illustrates a schematic
5 configuration of a television apparatus according to the
present technology.
Fig. 72 is a diagram that illustrates a schematic
configuration of a cellular phone according to the
present technology.
10 Fig. 73 is a diagram that illustrates a schematic
configuration of a recording and reproducing device
according to the present technology.
Fig. 74 is a diagram that illustrates a schematic
configuration of an imaging apparatus according to the
15 present technology.
MODE FOR CARRYING OUT THE INVENTION
[0037]

20 [Configuration Example of Encoding Device According to
First Embodiment]
Fig. 5 is a block diagram that illustrates a
configuration example of an encoding device according to
a first embodiment of the present technology.
25 [0038]
The encoding device 50 illustrated in Fig. 5 is
configured by imaging units 51A to 51C, an image
converting unit 52, a disparity image generating unit 53,
an image information generating unit 54, a compatibility
30 information generating unit 55, an inter-viewpoint
distance information generating unit 56, a disparity
18
t
image information generating unit 57, an encoder 58, and
a multiplexing unit 59.
[0039]
The encoding device 50 encodes an image of one
5 viewpoint that is included in a multi-viewpoint image as
a 2D image in accordance with an existing encoding mode,
thereby securing the compatibility with an existing
encoding device that encodes a 2D image.
[0040]
10 Hereinafter, out of multi-viewpoint images, an
image that is encoded in an existing encoding mode so as
to secure the compatibility with an existing encoding
device will be referred to as a compatible image, and an
image that is used for generating images of viewpoints
15 more than the number of viewpoints of a compatible image
using the compatible image will be referred to as an
auxiliary image.
[0041]
In the encoding device 50, the imaging unit 51A
20 captures an HD (high definition) image of a predetermined
viewpoint as an image Al and supplies the captured image
to the image converting unit 52, the disparity image
generating unit 53, and the inter-viewpoint distance
information generating unit 56. The imaging unit SIB
25 captures an HD image of a viewpoint that is different
from the viewpoint of the image Al as an image Bl at a
position that is separate from the imaging unit 51A by a
distance ~dl~ in a horizontal direction and supplies the
captured image to the image converting unit 52, the
30 disparity image generating unit 53, and the interviewpoint
distance information generating unit 56. The
19
imaging unit 51C captures an HD image of a viewpoint
different from the viewpoints of the images A1 and B1 as
an image C1 at a position separate from the imaging unit
51A by a distance ~d1Ac in the horizontal direction that
5 is opposite to the imaging unit 51B and supplies the
captured image to the image converting unit 52, the
disparity image generating unit 53, and the interviewpoint
distance information generating unit 56.
[0042]
10 The viewpoints that correspond to the images B1 and
C1 are viewpoints that are located on the outer side out
of viewpoints of images that can be recognized as 3D
images. In this way, a decoding device that corresponds
to the encoding device 50 can generate multi-viewpoint
15 images by interpolating images of viewpoints located on
the further inner side than the viewpoints of the images
B1 and C1 by using the images A1 to C1. As a result, the
multi-viewpoint images can be generated with precision
that is higher than that of a case where images of
20 viewpoints located on the outer side are interpolated
using images of viewpoints located on the inner side.
The distances ~d1~ and ~d1Ac may be configured to be
either fixed or changed each time.
[0043]
25 The image converting unit 52 determines the image
A1, which is supplied from the imaging unit 51A of which
the position in the horizontal direction is located on
the inner side out of the imaging units 51A to 51C, as a
compatible image. The image converting unit 52 supplies
30 information that designates the image A1 as a compatible
image to the compatibility information generating unit 55.
20
,
Then, the image converting unit 52 directly supplies the
image Al that is the compatible image to the encoder 58.
[0044]
In addition, the image converting unit 52 sets the
5 images B1 and C1 other than the image A1 as auxiliary
images and multiplexes the images B1 and C1 in accordance
with a predetermined multiplexing mode. More
specifically, for example, in a case where the
multiplexing mode is a side-by-side mode, the image
10 converting unit 52 (image converting unit) halves the
resolution of each one of the images B1 and C1. Then,
the image converting unit 52 (multiplexing unit)
multiplexes the image B1 (hereinafter, referred to as a
half-resolution image B1) of which the resolution has
15 halved and the image C1 (hereinafter, referred to as a
half-resolution image C1) of which the resolution has
halved such that the half-resolution image B1 becomes a
left-half image of the screen, and the half-resolution
image C1 becomes a right-half image of the screen. The
20 image converting unit 52 supplies a multiplexed image
that is acquired as a result of the multiplexing process
to the encoder 58 and supplies information that
represents a multiplexing mode of the auxiliary images to
the image information generating unit 54.
25 [0045]
The disparity image generating unit 53 detects the
disparity of each pixel of the images A1 to C1 using the
images A1 to C1 that are supplied from the imaging units
51A to 51C. The disparity image generating unit 53
30 generates a disparity image A1' that represents the
disparity of each pixel of the image A1 that is a
21
compatible image and directly supplies the generated
disparity image to the encoder 58. In addition, the
disparity image generating unit 53 (multiplexing unit)
generates a disparity image B1' that represents the
5 disparity of each pixel of the image B1 that is an
auxiliary image and a disparity image C1' that represents
the disparity of each pixel of the image C1 that is an
auxiliary image and multiplexes the generated disparity
images in accordance with a predetermined multiplexing
10 mode. The disparity image generating unit 53 supplies a
multiplexed image that is acquired as a result of the
multiplexing process to the encoder 58. In addition, the
disparity image generating unit 53 supplies information
that represents the multiplexing mode of the disparity
15 images of the auxiliary images to the disparity image
information generating unit 57.
[0046]
The image information generating unit 54 generates
information that represents the multiplexing mode of the
20 auxiliary images and the like as image information, which
is information relating to a compatible image and
auxiliary images, based on the information that is
supplied form the image converting unit 52 and supplies
the generated image information to the encoder 58.
25 [0047]
The compatibility information generating unit 55
generates information designating a compatible image, a
compatible mode, and the like as compatibility
information, which is information relating to the
30 compatibility, based on the information supplied from the
image converting unit 52 and supplies the generated
22
compatibility information to the encoder 58.
[0048]
Here, the compatible mode is a mode that represents
a method of encoding the compatible image. As examples
5 of the compatible mode, there are a mono mode that
represents an encoding method in which a compatible image
of one viewpoint is encoded in accordance with the AVC
mode, a frame packing mode that represents an encoding
method in which compatible images of two viewpoints are
10 multiplexed and encoded in accordance with the AVC mode,
and a stereo mode that represents an encoding method in
which compatible images of two viewpoints are encoded in
accordance with the MVC mode.
[0049]
15 The inter-viewpoint distance information generating
unit 56 detects each inter-viewpoint distance
(hereinafter, referred to as an inter-viewpoint distance)
between two images out of the images Al to CI using the
images Al to Cl that are supplied from the imaging units
20 5IA to SIC. For example, the inter-viewpoint distance
information generating unit 56 detects the distance ddl~
between the imaging units 5IA and SIB in the horizontal
direction and the distance ddlAc between the imaging
units 5IA and SIC in the horizontal direction as inter-
25 viewpoint distances. The inter-viewpoint distance
information generating unit 56 generates information that
represents the inter-viewpoint distances and the like as
inter-viewpoint distance information, which is
information relating to the inter-viewpoint distance
30 information, and supplies the generated inter-viewpoint
distance information to the encoder 58.
23
•
[0050]
The disparity image information generating unit 57
generates information that represents the multiplexing
mode of the disparity images of the auxiliary images and
5 the like as disparity image information, which is
information relating to the disparity images, based on
the information that is supplied from the disparity image
generating unit 53 and supplies the generated disparity
image information to the encoder 58.
10 [0051]
The encoder 58 is configured by a compatible
encoder 61 and an auxiliary encoder 62. The compatible
encoder 61 (compatible image encoding unit) designates
the image A1 that is a compatible image from the
15 multiplexed image of the compatible image and the
auxiliary images supplied from the image converting unit
52, encodes the image A1 in accordance with the existing
Ave mode, adds various information to the encoded image,
and supplies an encoded stream that is acquired as a
20 result thereof to the multiplexing unit 59 as a
compatible stream (first encoded stream) .
[0052]
The auxiliary encoder 62 (an auxiliary image
encoding unit and a disparity image encoding unit)
25 encodes the multiplexed image of the auxiliary images
that is supplied from the image converting unit 52 and
the multiplexed images of the disparity images A1' of the
compatible images and the disparity images of the
auxiliary images that are supplied from the disparity
30 image generating unit 53 in accordance with a
predetermined mode. As an encoding mode employed by the
24
•
auxiliary encoder 62, the AVe mode, the Mve mode, an
MPEG2 (Moving Picture Experts Group phase 2), or the like
can be used.
[0053]
5 In addition, the auxiliary encoder 62 adds the
image information supplied from the image information
generating unit 54, the compatibility information
supplied from compatibility information generating unit
55, the inter-viewpoint distance information supplied
10 from the inter-viewpoint distance information generating
unit 56, the disparity image information supplied from
the disparity image information generating unit 57, and
the like to an encoded image (the second stream and the
disparity encoded stream) that is acquired as a result of
15 the encoding process, thereby generating an encoded
stream. The auxiliary encoder 62 supplies the encoded
stream to the multiplexing unit 59 as an auxiliary stream.
[0054]
The multiplexing unit 59 respectively generates TSs
20 (transport streams) from the compatible stream supplied
from the compatible encoder 61 and the auxiliary stream
supplied from the auxiliary encoder 62, and multiplexes
the generated TSs. The multiplexing unit 59
(transmission unit) transmits a multiplexed stream that
25 is acquired as a result of the multiplexing process.
[0055]
[Example of Description of Information Included in
Auxiliary Stream]
Figs. 6 to 8 are diagrams that illustrate examples
30 of the description of the compatibility information and
the disparity image information that are included in the
25
•
auxiliary stream in a case where the AVC mode or the MVC
mode is used as an encoding mode.
[0056]
As illustrated in Fig. 6, the compatibility
5 information (3DV_view_structure) and the disparity image
information (depth_map_structure) are arranged in the SEI
(supplemental enhancement information) that is included
in the auxiliary stream.
[0057]
10 As the compatibility information
(3DV_view structure), as illustrated in Fig. 7, the
number (num_of_views) of viewpoints (views) of an image
that is an encoding target, a compatible mode,
information (compatible_view) that designates a
15 compatible image, and the like are described.
[0058]
In addition, as the disparity image information
(depth_map_structure), as illustrated in Fig. 8, the
number (num_of_depth_map) of disparity images (depth map),
20 a flag (is frame_packing) representing whether or not the
disparity images are multiplexed, a multiplexing system
(frame_packing_mode) of the disparity images, information
(comb_frame_packing_views) that designates the disparity
images that are multiplexed, and the like are described.
25 As the multiplexing modes, there are a side-by-side mode
(SBS), a top-and-bottom mode (TOB) , and the like. Here,
the top-and-bottom mode is a mode in which one image out
of two images to be multiplexed is set as an upper image
of the screen, and the other image is set as a lower
30 image of the screen.
[0059]
26
The image information is similar to the disparity
image information except that the image information is
information of the compatible image and the auxiliary
image other than the disparity image and thus is not
5 illustrated in the figure.
[0060]
Fig. 9 is a diagram that illustrates an example of
the description of the inter-viewpoint distance
information that is included in the auxiliary stream in a
10 case where the Ave mode or the Mve mode is used as an
encoding mode.
[0061]
As illustrated in Fig. 9, the inter-viewpoint
distance information (3DV_view_info), similarly to the
15 compatibility information, the disparity image
information, and the image information, is arranged in
the SEI that is included in the auxiliary stream.
[0062]
As the inter-viewpoint distance information
20 (3DV_view_info), as illustrated in Fig. 9, the number
(num_of_views) of viewpoints of an image that is an
encoding target, the number (num_of_view_distance) of
detected inter-viewpoint distances, a combination
(view[O,l]) of two images corresponding to the inter-
25 viewpoint distance, an inter-viewpoint distance
(view_distance[view_distance_id]) that corresponds to the
combination, and the like are described.
[0063]
[Description of Process of Encoding Device]
30 Figs. 10 and 11 are flowcharts that illustrate an
encoding process that is performed by the encoding device
27
t
50 illustrated in Fig. 5. This encoding process, for
example, is started when the images A1 to C1 are output
from the imaging units 51A to 51C.
[0064]
5 In Step Sll illustrated in Fig. 10, the interviewpoint
distance information generating unit 56 detects
distances ~d1~ and ~d1Ac as inter-viewpoint distances
using the images A1 to C1 that are supplied from the
imaging units 51A to 51C.
10 [0065]
In Step S12, the inter-viewpoint distance
information generating unit 56 generates information that
represents the inter-viewpoint distances detected in Step
Sll and the like as inter-viewpoint distance information
15 and inputs the generated inter-viewpoint distance
information to the encoder 58.
[0066]
In Step S13, the image converting unit 52
determines the image A1 that is supplied from the imaging
20 unit 51A, of which the position in the horizontal
direction is located on the inner side out of the imaging
units 51A to 51C, as a compatible image and determines a
multiplexing mode of auxiliary images. The image
converting unit 52 supplies information that designates
25 the image A1 as a compatible image to the compatibility
information generating unit 55 and supplies the
multiplexing mode of the auxiliary images to the image
information generating unit 54.
[0067]
30 In Step S14, the compatibility information
generating unit 55 generates the information designating
28
t
the image A1 as a compatible image, a mono mode as a
compatible mode, and the like as compatible image based
on the information that is supplied from the image
converting unit 52 and inputs the generated compatibility
5 information to the encoder 58.
[0068]
In Step S15, the image information generating unit
54 generates information that represents the multiplexing
mode of auxiliary images and the like as image
10 information based on the information that is supplied
from the image converting unit 52 and inputs the
generated image information to the encoder 58.
[0069]
In Step S16, the image converting unit 52 sets
15 images B1 and C1 other than the image A1 as auxiliary
images and multiplexes the auxiliary images based on the
multiplexing mode of auxiliary images that is determined
in Step S13, thereby acquiring a multiplexed image of the
auxiliary images.
20 [0070]
In Step S17, the image converting unit 52 inputs
the multiplexed image of the image A1, which is a
compatible image, and the auxiliary image to the encoder
58.
25 [0071]
In Step S18 illustrated in Fig. 11, the disparity
image generating unit 53 detects the disparity of each
pixel of the images A1 to C1 using the images A1 to C1
that are supplied from the imaging units 51A to 51C and
30 generates disparity images A1' to Cl'.
[0072]
29
f
In Step S19, the disparity image generating unit 53
determines a multiplexing mode of the disparity images of
auxiliary images and supplies information that represents
the multiplexing mode to the disparity image information
5 generating unit 57.
[0073]
In Step S20, the disparity image information
generating unit 57 generates information that represents
the multiplexing mode of the disparity images of
10 auxiliary images and the like as disparity image
information based on the information that is supplied
from the disparity image generating unit 53 and inputs
the generated disparity image information to the encoder
58.
15 [0074]
In Step S21, the disparity image generating unit 53
multiplexes the disparity images of the auxiliary images
based on the multiplexing mode of the disparity images of
the auxiliary images that is determined in Step S19,
20 thereby acquiring a multiplexed image of the disparity
images of the auxiliary images.
[0075]
In Step S22, the disparity image generating unit 53
inputs the disparity image A1' of the compatible image
25 and the multiplexed image of the disparity images of the
auxiliary images to the encoder 58.
[0076]
In Step S23, the compatible encoder 61 of the
encoder 58 encodes the image A1 that is a compatible
30 image supplied from the image converting unit 52 in
accordance with the existing Ave mode and supplies an
30
encoded stream acquired as a result thereof to the
multiplexing unit 59 as a compatible stream.
[0077]
In Step S24, the auxiliary encoder 62 encodes the
5 multiplexed image of the auxiliary images that is
supplied from the image converting unit 52, the disparity
image A1' of the compatible image that is supplied from
the disparity image generating unit 53, and the
multiplexed image of the disparity images of the
10 auxiliary images in accordance with a predetermined mode.
[0078]
In Step S25, the auxiliary encoder 62 adds the
image information supplied from the image information
generating unit 54, the compatibility information
15 supplied from the compatibility information generating
unit 55, the inter-viewpoint distance information
supplied from the inter-viewpoint distance information
generating unit 56, the disparity image information
supplied from the disparity image information generating
20 unit 57, and the like to the encoded image that is
acquired as a result of the encoding process by a process
of Step S24, thereby generating an encoded stream. The
auxiliary encoder 62 supplies the encoded stream to the
multiplexing unit 59 as an auxiliary stream.
25 [0079]
In Step S26, the multiplexing unit 59 generates TSs,
respectively, from the compatible stream that is supplied
from the compatible encoder 61 and the auxiliary stream
that is supplied from the auxiliary encoder 62,
30 multiplexes the generated TSs, and transmits a resultant
stream. Then, the process ends.
31
[0080]
As above, since the encoding device 50 performs
encoding in accordance with an existing encoding mode
with an image of one viewpoint that is included in the
5 multi-viewpoint image set as a compatible image, the
compatibility with an existing encoding device that
encodes a 2D image can be secured. In other words, the
encoding device 50 can perform encoding for multiviewpoint
images in accordance with a mode that has
10 compatibility with an existing encoding mode.
[0081]
In addition, since the encoding device 50 encodes
the images A1 to C1 of three viewpoints, a decoding
device that corresponds to the encoding device 50 can
15 generate multi-viewpoint images from the images A1 to C1
of three viewpoints. In this way, according to the
decoding device, compared to a case where multi-viewpoint
images are generated from images of two viewpoints, the
viewpoints of images that can be generated are not
20 limited, and multi-viewpoint images having relatively
high precision can be generated.
[0082]
Furthermore, since the encoding device 50 encodes
images with the resolution of the auxiliary images being
25 lowered, the processing cost of the encoding process and
the decoding process can be lower than that of a case
where encoding is performed without lowering the
resolution. For example, in a case where auxiliary
images of two viewpoints are encoded without lowering the
30 resolution thereof, while the processing cost of the
encoding process and the decoding process is the same as
32
that of an encoding process and a decoding process for
two HD images, the processing cost of the encoding
process for auxiliary images of two viewpoints, which is
performed by the encoding device 50, is the same as the
5 processing cost of the encoding process or the decoding
process for one HD image. As a result, it can be
prevented that the performance of the decoding process
performed by the decoding device greatly affects the
image quality of multi-viewpoint images.
10 [0083]
Furthermore, when multi-viewpoint images are
synthesized, as will be described later, a decoding
device that corresponds to the encoding device 50 lowers
the resolution at the rate of the reciprocal of the
15 number of viewpoints of the multi-viewpoint images,
whereby the lowering of the resolution of the auxiliary
images that is performed by the encoding device 50 does
not affect the image quality of the multi-viewpoint
images after the synthesis.
20 [0084]
In addition, since the encoding device 50 performs
encoding with the resolution of auxiliary images being
halved and encodes the disparity images of the auxiliary
images while being halved, the amount of information of
25 an encoding target can be configured to be approximately
an amount that corresponds to 4 HD images (1080i x 4) in
the Ave mode.
[0085]
Here, in comprehensive view of the processing speed,
30 the processing performance with respect to power
consumption, a transmission data rate, a transmission
33
bandwidth, a bandwidth of a memory, a memory access speed,
and the like of a current display device that displays
multi-viewpoint images and a decoding device according
thereto, a value that is adequate as the amount of
5 information that can be processed by the decoding device
is considered to be about two times an amount that
corresponds to an HD image in the current state of the
Mve mode, that is, an amount that corresponds to 4 HD
images in the Ave mode. Accordingly, a decoding device
10 (decoding method) that corresponds to the encoding device
50 of which the amount of information of the encoding
target is about an amount that corresponds to 4 HD images
(1080i x 4) in the Ave mode can be realized at a
reasonable processing cost by a reasonable approach.
15 [0086]
In addition, according to the encoding device 50,
the amount of information of the encoding target is
reduced to an amount that corresponds to about 4 HD
images (1080i x 4) in the Ave mode, and accordingly, the
20 encoding device can be easily operated as a ED or a
broadcast application that has a limitation of a usable
bandwidth.
[0087]
Furthermore, since the encoding device 50 generates
25 disparity images and transmits the disparity images while
being included in an encoded stream, a decoding device
that corresponds to the encoding device 50 does not need
to generate a disparity image for generating multiviewpoint
images, and accordingly, the processing load of
30 the decoding device can be reduced. As a result, the
cost of the decoding device can be reduced. In addition,
34
it can be prevented that the disparity detecting
performance of the decoding device greatly affects the
image quality of multi-viewpoint images.
[0088]
5 In view of the situations of the display device and
the decoding device as described above, a main object of
the present technology is, by using various kinds of
information such as the compatibility information on the
premise of restrictions in consideration of a valid
10 processing cost and a reasonable realization technique,
to realize the generation of a multi-viewpoint image
having an improved image quality with high efficiency
while the compatibility with an existing format is
secured.
15 [0089]
[Configuration Example of Decoding Device]
Fig. 12 is a diagram that illustrates a
configuration example of a decoding device that decodes a
multiplexed stream that is transmitted from the encoding
20 device 50 illustrated in Fig. 5.
[0090]
The decoding device 120 illustrated in Fig. 12 is
configured by a separation unit 121, a decoder 122, an
image information acquiring unit 123, an inter-viewpoint
25 distance information acquiring unit 124, a disparity
image information acquiring unit 125, a compatibility
information acquiring unit 126, and an image generating
unit 127. The decoding device 120 decodes a multiplexed
stream that is transmitted from the encoding device 50,
30 generates an image of one viewpoint or images of
viewpoints, and displays the generated image on a display
35
•
device that is not illustrated in the figure.
[0091]
More specifically, the separation unit 121
(reception unit) of the decoding device 120 receives a
5 multiplexed stream that is transmitted from the encoding
device 50 and separates each TS. The separation unit 121
extracts a compatible stream and an auxiliary stream from
the separated TS and supplies the extracted streams to
the decoder 122.
10 [0092]
The decoder 122 is configured by a compatible
decoder 131 and an auxiliary decoder 132. The compatible
decoder 131 (compatible image decoding unit) of the
decoder 122 identifies the compatible stream out of the
15 compatible stream and the auxiliary stream supplied from
the separation unit 121 based on the information used for
specifying the compatible stream that is supplied from
the auxiliary decoder 132. The compatible decoder 131
decodes the encoded compatible image included in the
20 compatible stream in accordance with a mode that
corresponds to the Ave mode and supplies an image A1 that
is acquired as a result thereof to the image generating
unit 127.
[0093]
25 The auxiliary decoder 132 supplies the information
used for specifying the compatible stream to the
compatible decoder 131 based on the compatibility
information that is included in the auxiliary stream
supplied from the separation unit 121. The auxiliary
30 decoder 132 identifies the auxiliary stream out of the
compatible stream and the auxiliary stream that are
36
•
supplied from the separation unit 121 based on the
compatibility information. The auxiliary decoder 132
(auxiliary image decoding unit) decodes the multiplexed
image of auxiliary images, the disparity image A1' of the
5 compatible image, and the multiplexed image of disparity
images of the auxiliary images, which are encoded,
included in the auxiliary stream that is supplied from
the separation unit 121 in accordance with a mode that
corresponds to the auxiliary encoder 62 illustrated in
10 Fig. 5.
[0094]
The auxiliary decoder 132 supplies the multiplexed
image of the auxiliary images, the disparity image A1' of
the compatible image, and the multiplexed image of the
15 disparity images of the auxiliary images, which are
acquired as a result of the decoding process, to the
image generating unit 127. In addition, the auxiliary
decoder 132 supplies the image information included in
the auxiliary stream to the image information acquiring
20 unit 123 and supplies the inter-viewpoint distance
information to the inter-viewpoint distance information
acquiring unit 124. Furthermore, the auxiliary decoder
132 supplies the disparity image information included in
the auxiliary stream to the disparity image information
25 acquiring unit 125 and supplies the compatibility
information to the compatibility information acquiring
unit 126.
[0095]
The image information acquiring unit 123 acquires
30 the image information that is supplied from the auxiliary
decoder 132 and supplies the acquired image information
37
•
to the image generating unit 127. The inter-viewpoint
distance information acquiring unit 124 acquires the
inter-viewpoint distance information that is supplied
from the auxiliary decoder 132 and supplies the acquired
5 inter-viewpoint distance information to the image
generating unit 127.
[0096]
The disparity image information acquiring unit 125
acquires the disparity image information that is supplied
10 from the auxiliary decoder 132 and supplies the acquired
disparity image information to the image generating unit
127. The compatibility information acquiring unit 126
acquires the compatibility information that is supplied
from the auxiliary decoder 132 and supplies the acquired
15 compatibility information to the image generating unit
127.
[0097]
The image generating unit 127 outputs an image in
accordance with an instruction supplied from a viewer and
20 displays the image on a display device not illustrated in
the figure. More specifically, the image generating unit
127 generates images of three or more viewpoints, which
correspond to a display device not illustrated in the
figure, each having the resolution that is a half of the
25 resolution of the compatible image or the auxiliary image
by using the compatible image, the multiplexed image of
the auxiliary images, the disparity image of the
compatible image, and the multiplexed image of the
disparity images of the auxiliary images in accordance
30 with a viewer's instruction for displaying a 3D image of
the multi-viewpoint mode based on the image information
38
•
supplied from the image information acquiring unit 123,
the inter-viewpoint distance information supplied from
the inter-viewpoint distance information acquiring unit
124, the disparity image information supplied from the
5 disparity image information acquiring unit 125, the
compatibility information supplied from the compatibility
information acquiring unit 126, and the like.
[0098]
More specifically, the image generating unit 127
10 (separation unit) separates the disparity image of each
auxiliary image from the multiplexed image of the
disparity images of the auxiliary images based on the
information that represents a multiplexing mode of the
disparity images of the auxiliary images that is included
15 in the disparity image information supplied from the
disparity image information acquiring unit 125. In
addition, the image generating unit 127 separates each
auxiliary image from the multiplexed image of the
auxiliary images based on the information that represents
20 a multiplexing mode of auxiliary images that is included
in the image information supplied from the image
information acquiring unit 123.
[0099]
Furthermore, the image generating unit 127
25 determines the position of each viewpoint of the multiviewpoint
images to be generated based on the interviewpoint
distance information and the number of
viewpoints that corresponds to a display device not
illustrated in the figure. Then, the image generating
30 unit 127 generates the image of each viewpoint of which
the position is determined by using the compatible image,
39
•
each auxiliary image, the disparity image of the
compatible image, and the disparity image of each
auxiliary images. Then, the image generating unit 127
converts the resolution of the generated image of each
5 viewpoint into resolution that is "lithe number of
viewpoints" of the resolution of the compatible image or
the auxiliary image, synthesizes the images, and displays
the synthesized image on a display device not illustrated
in the figure.
10 [0100]
At this time, the multi-viewpoint images after the
synthesis are displayed such that the viewing angles are
different for each viewpoint, and a viewer can view a 3D
image without wearing glasses by seeing images of
15 arbitrary two viewpoints with his left and right eyes.
[0101]
In addition, the image generating unit 127 outputs
the image A1 that is the compatible image supplied from
the compatible decoder 131 of the decoder 122 in
20 accordance with a viewer's instruction for displaying a
20 image, thereby displaying the image on a display
device not illustrated in the figure. Accordingly, the
viewer can view the 20 image.
[0102]
25 [Description of Process of Decoding Device]
Fig. 13 is a flowchart that illustrates a decoding
process performed by the decoding device 120 illustrated
in Fig. 12. This decoding process, for example, is
started when a multiplexed stream that is transmitted
30 from the encoding device 50 illustrated in Fig. 5 is
input to the decoding device 120.
40
[0103]
In Step S31 illustrated in Fig. 13, the separation
unit 121 of the decoding device 120 receives the
multiplexed stream that is transmitted from the encoding
5 device 50 and separates each TS. The separation unit 121
extracts the compatible stream and the auxiliary stream
from the separated TS and supplies the extracted streams
to the decoder 122. The auxiliary decoder 132 of the
decoder 122 supplies the information used for specifying
10 the compatible stream to the compatible decoder 131 based
on the compatibility information included in the
auxiliary stream that is supplied from the separation
unit 121.
[0104]
15 In Step S32, the compatible decoder 131 identifies
the compatible stream out of the compatible stream and
the auxiliary stream that are supplied from the
separation unit 121 based on the information used for
specifying the compatible stream that is supplied from
20 the auxiliary decoder 132.
[0105]
In Step S33, the compatible decoder 131 decodes the
compatible image that is included in the compatible
stream in accordance with a mode that corresponds to the
25 Ave mode and supplies an image A1 that is acquired as a
result of the decoding process to the image generating
unit 127.
[0106]
In Step S34, the image generating unit 127
30 determines whether or not an instruction for displaying a
20 image has been made by a viewer. In a case where it
41
is determined that the instruction for displaying a 2D
image has not been made by the viewer in Step S34, in
other words, in a case where an instruction for
displaying a 3D image of the multi-viewpoint mode has
5 been made by the viewer, the auxiliary decoder 132
identifies the auxiliary stream out of the compatible
stream and the auxiliary stream that are supplied from
the separation unit 121 based on the compatibility
information.
10 [0107]
In Step S35, the auxiliary decoder 132 decodes the
multiplexed image of auxiliary images, the disparity
image A1' of the compatible image, and the multiplexed
image of the disparity images of the auxiliary images,
15 which are encoded, included in the auxiliary stream in
accordance with a mode that corresponds to the auxiliary
encoder 62 illustrated in Fig. 5. The auxiliary decoder
132 supplies the multiplexed image of the auxiliary
images, the disparity image A1' of the compatible image,
20 and the multiplexed image of the disparity images of the
auxiliary images that are acquired as a result of the
decoding process to the image generating unit 127. In
addition, the auxiliary decoder 132 supplies the image
information that is included in the auxiliary stream to
25 the image information acquiring unit 123 and supplies the
inter-viewpoint distance information to the interviewpoint
distance information acquiring unit 124.
Furthermore, the auxiliary decoder 132 supplies the
disparity image information that is included in the
30 auxiliary stream to the disparity image information
acquiring unit 125 and supplies the compatibility
42
information to the compatibility information acquiring
unit 126.
[0108]
In Step S36, the image information acquiring unit
5 123 acquires the image information that is supplied from
the auxiliary decoder 132 and inputs the image
information to the image generating unit 127. In Step
S37, the inter-viewpoint distance information acquiring
unit 124 acquires the inter-viewpoint distance
10 information that is supplied from the auxiliary decoder
132 and inputs the inter-viewpoint distance information
to the image generating unit 127.
[0109]
In Step S38, the disparity image information
15 acquiring unit 125 acquires the disparity image
information that is supplied from the auxiliary decoder
132 and inputs the disparity image information to the
image generating unit 127. In Step S39, the
compatibility information acquiring unit 126 acquires the
20 compatibility information that is supplied from the
auxiliary decoder 132 and inputs the compatibility
information to the image generating unit 127.
[0110]
In Step S40, the image generating unit 127
25 determines the position of each viewpoint of a multiviewpoint
image to be generated based on the interviewpoint
distance information supplied from the interviewpoint
distance information acquiring unit 124 and the
number of viewpoints that corresponds to a display device
30 not illustrated in the figure. For example, in a case
where an inter-viewpoint distance that is included in the
43
•
inter-viewpoint distance information is short, the image
generating unit 127 also determines the position of a
viewpoint that is located on the outer side of the
viewpoints of images B1 and C1 as the position of a
5 viewpoint of the multi-viewpoint 3D image to be generated.
On the other hand, in a case where the inter-viewpoint
distance that is included in the inter-viewpoint distance
information is long, the image generating unit 127
determines only positions of viewpoints located on·the
10 inner side of the images B1 and C1 as the positions of
viewpoints of the multi-viewpoint 3D image to be
generated.
[0111]
In Step S41, the image generating unit 127
15 generates images of the viewpoints each having the
resolution that is a half of the resolution of the
compatible image or the auxiliary image based on the
position of each viewpoint determined in Step S40, the
image information supplied from the image information
20 acquiring unit 123, the disparity image information
supplied from the disparity image information acquiring
unit 125, the compatibility information supplied from the
compatibility information acquiring unit 126, and the
like by using the compatible image, the multiplexed image
25 of the auxiliary images, the disparity image of the
compatible image, and the multiplexed image of the
disparity images of the auxiliary images.
[0112]
In Step S42, the image generating unit 127 converts
30 the resolution of the image of each viewpoint that is
generated in Step S41 into resolution that is "1/ the
44
~I
:.,'.:
t
(
•
number of viewpoints" of the resolution of the compatible
image or the auxiliary image and synthesizes the images
of each viewpoint after the conversion based on the
positions of the viewpoints.
5 [0113]
In Step S43, the image generating unit 127 outputs
the multi-viewpoint image after the synthesis that is
acquired by the process of Step S42 to a display device
that is not illustrated in the figure, thereby displaying
10 the multi-viewpoint image such that viewing angles are
different for each viewpoint. Then, the process ends.
[0114]
On the other hand, in a case where it is determined
that an instruction for displaying a 20 image has been
15 made by a viewer in Step S34, in Step S44, the image
generating unit 127 outputs an image A1 that is a
compatible image supplied from the compatible decoder 131
to a display device not illustrated in the figure,
thereby displaying the image. Then, the process ends.
20 [0115]
In a decoding device that can decode only a
compatible stream that has compatibility with the
decoding device 120, the TS of the auxiliary stream is
ignored, and only the processes of Steps S33 and S44 are
25 performed.
[0116]
As above, the decoding device 120 can decode a
compatible stream and an auxiliary stream that are
encoded in accordance with a mode having compatibility
30 with an existing mode by the encoding device 50.
[0117]
45
•
In addition, since the decoding device 120
generates a multi-viewpoint image using a compatible
image of one viewpoint and an auxiliary image of two
. viewpoints, the encoding device 50 may prepare only two
5 imaging units 51B and 51C in addition to the existing
imaging unit 51A that captures a compatible image of one
viewpoint as imaging units that generate multi-viewpoint
images. Accordingly, the installation of the imaging
units for generating multi-viewpoint images can be
10 performed in an easy manner at a low cost.
[0118]

[Configuration Example of Encoding Device According to
Second Embodiment]
15 Fig. 14 is a block diagram that illustrates a
configuration example of an encoding device according to
a second embodiment of the present technology.
[0119]
In the configuration illustrated in Fig. 14, the
20 same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 5.
Duplicate description will not be presented as is
appropriate.
[0120]
25 The configuration of the encoding device 140
illustrated in Fig. 14 is mainly different from the
configuration illustrated in Fig. 5 in that imaging units
141A to 1410, an image converting unit 142, a disparity
image generating unit 143, an inter-viewpoint distance
30 information generating unit 144, and an encoder 145 are
disposed instead of the imaging units 51A to 51C, the
46
•
image converting unit 52, the disparity image generating
unit 53, the inter-viewpoint distance information
generating unit 56, and the encoder 58.
[0121]
5 The encoding device 140 performs encoding in
accordance with the AVe mode with images of two
viewpoints out of multi-viewpoint images set as
compatible images, thereby securing the compatibility
with an existing encoding device that encodes a 3D image
10 of the two viewpoints mode.
[0122]
More specifically, the imaging unit 141A of the
encoding device 140 captures an HD image of a
predetermined viewpoint as an image A2 and supplies the
15 captured image to the image converting unit 142, the
disparity image generating unit 143, and the interviewpoint
distance information generating unit 144. The
imaging unit 141B captures an HD image of a viewpoint
that is different from the viewpoint of the image A2 as
20 an image B2 at a position that is separate from the
imaging unit 141A by a distance dd2~ in a horizontal
direction and supplies the captured image to the image
converting unit 142, the disparity image generating unit
143, and the inter-viewpoint distance information
25 generating unit 144.
[0123]
The imaging unit 141e captures an HD image of a
viewpoint different from the viewpoints of the images A2
and B2 as an image e2 at a position separate from the
30 imaging unit 141B by a distance dd2 Bc in the horizontal
direction that is opposite to the imaging unit 141A and
47
•
supplies the captured image to the image converting unit
142, the disparity image generating unit 143, and the
inter-viewpoint distance information generating unit 144.
The imaging unit 1410 captures an HO image of a viewpoint
5 different from the viewpoints of the images A2 to C2 as
an image 02 at a position separate from the imaging unit
141A by a distance ~d2AD in the horizontal direction that
is opposite to the imaging unit 141B and supplies the
captured image to the image converting unit 142, the
10 disparity image generating unit 143, and the interviewpoint
distance information generating unit 144.
[0124]
The viewpoints that correspond to the images C2 and
02 are viewpoints that are located on the outer side out
15 of viewpoints of images that can be recognized as 30
images. In this way, a decoding device that corresponds
to the encoding device 140 can generate multi-viewpoint
images by interpolating images of viewpoints located on
the further inner side than the viewpoints of the images
20 C2 and 02 by using the images A2 to 02. As a result, the
multi-viewpoint images can be generated with precision
that is higher than that of a case where images of
viewpoints located on the outer side are interpolated
using images of viewpoints located on the inner side.
25 The distances ~d2~, ~d2&, and ~d2AD may be configured to
be either fixed or changed each time.
[0125]
The image converting unit 142 determines the image
A2 that is supplied from the imaging unit 141A of which
30 the position in the horizontal direction is located on
the inner side out of the imaging units 141A to 1410 and
48
the image B2 that is supplied from the imaging unit 141B
as compatible images. Then, the image converting unit
142 multiplexes the images A2 and B2 that are the
compatible images in accordance with a predetermined
5 multiplexing mode and supplies the multiplexed image to
the encoder 145. In addition, the image converting unit
142 supplies information that designates the images A2
and B2 as compatible images to the compatibility
information generating unit 55.
10 [0126]
In addition, the image converting unit 142 sets the
images C2 and 02 other than the images A2 and B2 as
auxiliary images and multiplexes the images C2 and 02 in
accordance with a predetermined multiplexing mode. The
15 image converting unit 142 supplies a multiplexed image
that is acquired as a result of the multiplexing process
to the encoder 145. The image converting unit 142
supplies information that represents a multiplexing mode
of compatible images and auxiliary images to the image
20 information generating unit 54.
[0127]
The disparity image generating unit 143 detects the
disparity of each pixel of the images A2 to D2 using the
images A2 to 02 that are supplied from the imaging units
25 141A to 141D. The disparity image generating unit 143
generates a disparity image A2' that represents the
disparity of each pixel of the image A2 that is a
compatible image and a disparity image B2' that
represents the disparity of each pixel of the image B2
30 and multiplexes the disparity images A2' and B2' in
accordance with a predetermined multiplexing mode. The
49
disparity image generating unit 143 supplies a
multiplexed image that is acquired as a result thereof to
the encoder 145.
[0128]
5 In addition, the disparity image generating unit
143 generates a disparity image C2' that represents the
disparity of each pixel of the image C2 that is an
auxiliary image and a disparity image D2' that represents
the disparity of each pixel of the image D2 that is an
10 auxiliary image and multiplexes the disparity images C2'
and D2' in accordance with a predetermined multiplexing
mode. The disparity image generating unit 143 supplies a
multiplexed image that is acquired as a result thereof to
the encoder 145. The disparity image generating unit 143
15 supplies information that represents the multiplexing
mode of disparity images of compatible images and
auxiliary images to the disparity image information
generating unit 57.
[0129]
20 The inter-viewpoint distance information generating
unit 144 detects each inter-viewpoint distance between
the images A2 to D2 using the images A2 to 02 that are
supplied from the imaging units 141A to 141D. For
example, the inter-viewpoint distance information
25 generating unit 144 detects a distance ~d2~ between the
imaging units 141A and 141B in the horizontal direction,
a distance ~d2Bc between the imaging units 141B and 141C
in the horizontal direction, and a distance ~d2AD between
the imaging units 141A and 141D in the horizontal
30 direction as inter-viewpoint distances. The interviewpoint
distance information generating unit 144
50
generates information that represents the inter-viewpoint
distances and the like as inter-viewpoint distance
information and supplies the generated inter-viewpoint
distance information to the encoder 145.
5 [0130]
The encoder 145 is configured by a compatible
encoder 151 and an auxiliary encoder 152. The compatible
10
encoder 151 designates a multiplexed image of the
compatible images out of the multiplexed image of the
compatible images and the multiplexed image of the
auxiliary images that are supplied from the image
converting unit 142, encodes the multiplexed image of the
compatible images in accordance with the existing Ave
mode, adds various kinds of information thereto, and
15 supplies an acquired resultant encoded stream to the
multiplexing unit 59 as a compatible stream.
[0131]
The auxiliary encoder 152 encodes the multiplexed
image of the auxiliary images that is supplied from the
20 image converting unit 142, and the multiplexed image of
the disparity images of the compatible images and the
multiplexed image of the disparity images of the
auxiliary images that are supplied from the disparity
image generating unit 143 in accordance with a
25 predetermined mode. As the encoding mode employed by the
auxiliary encoder 152, for example, the Ave mode, the Mve
mode, or the like can be used.
[0132]
In addition, the auxiliary encoder 152 adds the
30 image information supplied from the image information
generating unit 54, the compatibility information
51
supplied from the compatibility information generating
unit 55, the inter-viewpoint distance information
supplied from the inter-viewpoint distance information
generating unit 144, the disparity image information
5 supplied from the disparity image information generating
unit 57, and the like, to an encoded image that is
acquired as a result of the encoding process, thereby
generating an encoded stream. The auxiliary encoder 152
supplies the encoded stream to the multiplexing unit 59
10 as an auxiliary stream.
[0133]
[Description of Process of Encoding Device]
Figs. 15 and 16 represent a flowchart that
illustrates an encoding process that is performed by the
15 encoding device 140 illustrated in Fig. 14. This
encoding process, for example, is started when the images
A2 to D2 are output from the imaging units 141A to 141D.
[0134]
In Step S51 illustrated in Fig. 15, the inter-
20 viewpoint distance information generating unit 144
detects distances ~d2~, ~d2Bc, and ~d2AD as interviewpoint
distances using the images A2 to D2 that are
supplied from the imaging units 141A to 141D.
[0135]
25 In Step S52, the inter-viewpoint distance
information generating unit 144 generates information
that represents the inter-viewpoint distances detected in
Step S5l and the like as inter-viewpoint distance
information and inputs the generated inter-viewpoint
30 distance information to the encoder 145.
[0136]
52
•
In Step S53, the image converting unit 142
determines the image A2 that is supplied from the imaging
unit 141A, of which the position in the horizontal
direction is located on the inner side out of the imaging
5 units 141A to 141D, and the image B2 that is supplied
from the imaging unit 141B as compatible images, and
determines multiplexing modes of the compatible images
and the auxiliary images. The image converting unit 142
supplies information that designates the images A2 and B2
10 as compatible images to the compatibility information
generating unit 55 and supplies the multiplexing modes of
the compatible images and the auxiliary images to the
image information generating unit 54.
[0137]
15 In Step S54, the compatibility information
generating unit 55 generates the information designating
the images A2 and B2 as compatible images, a frame
packing mode as a compatible mode, and the like as
compatibility information based on the information that
20 is supplied from the image converting unit 142 and inputs
the generated compatibility information to the encoder
145.
[0138]
In Step S55, the image converting unit 142
25 multiplexes the images A2 and B2 that are compatible
images based on the multiplexing mode of compatible
images that is determined in Step S53 and supplies the
multiplexed image to the encoder 145.
[0139]
30 In Step S56, the image information generating unit
54 generates information that represents the multiplexing
53
modes of compatible images and auxiliary images and the
like as image information based on the information that
is supplied from the image converting unit 142 and inputs
the generated image information to the encoder 145.
5 [0140]
In Step S57, the image converting unit 142 sets
images C2 and D2 other than the images A2 and B2 as
auxiliary images and multiplexes the auxiliary images
based on the multiplexing mode of auxiliary images that
10 is determined in Step S53, thereby acquiring a
multiplexed image of the auxiliary images.
[0141]
In Step S58, the image converting unit 142 inputs
the multiplexed image of the compatible images and the
15 multiplexed image of the auxiliary images to the encoder
145.
[0142]
In Step S59 illustrated in Fig. 16, the disparity
image generating unit 143 detects the disparity of each
20 pixel of the images A2 to 02 using the images A2 to 02
that are supplied from the imaging units 141A to 1410 and
generates disparity images A2' to D2' .
[0143]
In Step S60, the disparity image generating unit
25 143 determines multiplexing modes of the disparity images
of the compatible images and the disparity images of the
auxiliary images and supplies information that represents
the multiplexing modes to the disparity image information
generating unit 57.
30 [0144]
In Step S61, the disparity image information
54
generating unit 57 generates information that represents
the multiplexing modes of the disparity images of the
compatible images and the disparity images of the
auxiliary images and the like as disparity image
5 information based on the information that is supplied
from the disparity image generating unit 143 and inputs
the generated disparity image information to the encoder
145.
[0145]
10 In Step S62, the disparity image generating unit
143 multiplexes the disparity images A2' and B2' of the
compatible images based on the multiplexing mode of the
disparity images of compatible images that is determined
in Step S60 and multiplexes the disparity images e2' and
15 D2' of the auxiliary images based on the multiplexing
mode of the disparity images of auxiliary images.
[0146]
In Step S63, the disparity image generating unit
143 inputs the multiplexed image of the disparity images
20 of the compatible images and the multiplexed image of the
disparity images of the auxiliary images that are
acquired as a result of the multiplexing process
illustrated in Step S62 to the encoder 145.
[0147]
25 In Step S64, the compatible encoder 151 of the
encoder 145 encodes the multiplexed image of the
compatible images that is supplied from the image
converting unit 142 in accordance with the existing Ave
mode and supplies an encoded stream acquired as a result
30 thereof to the multiplexing unit 59 as a compatible
stream.
55
[0148]
In Step S65, the auxiliary encoder 152 encodes the
multiplexed image of the auxiliary images that is
supplied from the image converting unit 142 and the
5 multiplexed image of the disparity images of the
compatible images and the multiplexed image of the
disparity images of the auxiliary images that are
supplied from the disparity image generating unit 53 in
accordance with a predetermined mode. Then, the process
10 proceeds to Step S66.
[0149]
The process of Steps S66 and S67 is the same as the
process of Steps S25 and S26 illustrated in Fig. 11, and
thus the description thereof will not be presented.
15 [0150]
As above, since the encoding device 140 encodes
images of two viewpoints out of multi-viewpoint images as
compatibles image in accordance with an existing encoding
mode, the compatibility with an existing encoding device
20 that encodes a 30 image of the two-viewpoint mode can be
secured.
[0151]
In addition, since the encoding device 140 encodes
the images A2 to 02 of four viewpoints, a decoding device
25 that corresponds to the encoding device 140 can generate
multi-viewpoint images from the images A2 to 02 of four
viewpoints. In this way, according to the decoding
device, compared to a case where multi-viewpoint images
are generated from images of two viewpoints, the
30 viewpoints of images that can be generated are not
limited, and multi-viewpoint images having relatively
56
•
high precision can be generated.
[0152]
[Configuration Example of Decoding Device]
Fig. 17 is a diagram that illustrates a
5 configuration example of a decoding device that decodes
the multiplexed stream transmitted from the encoding
device 140 illustrated in Fig. 14.
[0153]
In the configuration illustrated in Fig. 17, the
10 same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 12.
Duplicate description will not be presented as is
appropriate.
[0154]
15 The configuration of the decoding device 170
illustrated in Fig. 17 is mainly different from the
configuration illustrated in Fig. 12 in that an image
generating unit 171 is disposed instead of the image
generating unit 127. The decoding device 170 generates
20 images of two viewpoints or multi-viewpoint images by
decoding the multiplexed stream that is transmitted from
the encoding device 140 and displays the generated images
on a display device not illustrated in the figure.
[0155]
25 More specifically, the image generating unit 171 of
the decoding device 170 outputs images in accordance with
a display instruction supplied from a viewer, thereby
displaying the images on a display device (not
illustrated). When described in more detail, the image
30 generating unit 171 generates images of three or more
viewpoints, which correspond to a display device (not
57
•
illustrated), each having the resolution that is a half
of the resolution of the compatible image or the
auxiliary image by using the multiplexed image of the
compatible images, the multiplexed image of the auxiliary
5 images, the multiplexed image of the disparity images of
the compatible images, and the multiplexed image of the
disparity images of the auxiliary images in accordance
with a viewer's instruction for displaying a 3D image of
multi-viewpoint mode based on the image information
10 supplied from the image information acquiring unit 123,
the inter-viewpoint distance information supplied from
the inter-viewpoint distance information acquiring unit
124, the disparity image information supplied from the
disparity image information acquiring unit 125, the
15 compatibility information supplied from the compatibility
information acquiring unit 126, and the like.
[0156]
More specifically, the image generating unit 171
separates the disparity image of each auxiliary image
20 from the multiplexed image of the disparity images of the
auxiliary images based on the information that represents
a multiplexing mode of the disparity images of the
auxiliary images that is included in the disparity image
information supplied from the disparity image information
25 acquiring unit 125. In addition, the image generating
unit 171 separates the disparity image of each compatible
image from the multiplexed image of the disparity images
of the compatible images based on the information, which
represents the multiplexing mode of the disparity images
30 of the compatible images, included in the disparity image
information.
58
•
[0157 ]
Furthermore, the image generating unit 171
separates each auxiliary image from the multiplexed image
of the auxiliary images based on the information that
5 represents a multiplexing mode of the auxiliary images
that is included in the image information supplied from
the image information acquiring unit 123. In addition,
the image generating unit 171 separates each compatible
image from the multiplexed image of the compatible images
10 based on the information that represents a multiplexing
mode of the compatible images that is included in the
image information.
[0158]
Furthermore, the image generating unit 171
15 determines the position of each viewpoint of the multiviewpoint
images to be generated based on the interviewpoint
distance information and the number of
viewpoints that corresponds to a display device not
illustrated in the figure. Then, the image generating
20 unit 171 generates the image of each viewpoint of which
the position is determined by using each compatible image,
each auxiliary image, the disparity image of each
compatible image, and the disparity image of each
auxiliary image. Then, the image generating unit 171
25 converts the resolution of the generated multi-viewpoint
images into resolution that is ~l/the number of
viewpoints" of the resolution of the compatible image or
the auxiliary image, synthesizes the images, and displays
the synthesized image on a display device not illustrated
30 in the figure.
[0159]
59
• At this time, the multi-viewpoint images after the
synthesis are displayed such that the viewing angles are
different for each viewpoint, and a viewer can view a 3D
image without wearing glasses by seeing images of
5 arbitrary two viewpoints with his left and right eyes.
[0160]
In addition, the image generating unit 171
separates the multiplexed image of the compatible images
that is supplied from the decoder 122 into images A2 and
10 B2 of which the resolution is a half of the resolution of
the compatible image based on the image information
supplied from the image information acquiring unit 123 in
accordance with an instruction for displaying a 3D image
of the two-viewpoint mode that is supplied from a viewer.
15 Then, the image generating unit 171 alternately outputs
the separated images A2 and B2 of which the resolution is
a half of the resolution of the compatible image, thereby
displaying the separated images on a display device not
illustrated in the figure. At this time, the viewer can
20 view a 3D image by wearing glasses in which one of a
left-eye shutter and a right-eye shutter is open at the
time of displaying the image A2, and the other is open at
the time of displaying the image B2 and viewing the
images A2 and B2 that are alternately displayed on the
25 display device.
[0161]
[Description of Process of Decoding Device]
Fig. 18 is a flowchart that illustrates a decoding
process performed by the decoding device 170 illustrated
30 in Fig. 17. This decoding process, for example, is
started when a multiplexed stream that is transmitted
60
•
from the encoding device 140 illustrated in Fig. 14 is
input to the decoding device 170.
[0162]
The process of Steps S81 to S83 illustrated in Fig.
5 18 is the same as the process of Steps S31 to S33
illustrated in Fig. 13, and thus the description thereof
will not be presented.
[0163]
After the process of Step S83, in Step S84, the
10 image generating unit 171 determines whether or not an
instruction for displaying a 3D image of the twoviewpoint
mode has been made by a viewer. In a case
where it is determined that the instruction for
displaying a 3D image of the two-viewpoint mode has not
15 been made by the viewer in Step S84, in other words, in a
case where an instruction for displaying a 3D image of
the multi-viewpoint mode has been made, the process
proceeds to Step S85. The process of Steps S85 to S93 is
the same as the process of Steps S35 to S43 illustrated
20 in Fig. 13, and thus the description thereof will not be
presented.
[0164]
On the other hand, in a case where it is determined
that the instruction for displaying a 3D image of the
25 two-viewpoint mode has been made by the viewer in Step
S84, the process proceeds to Step S94.
[0165]
In Step S94, the image generating unit 171
separates the multiplexed image of the compatible images
30 that is acquired as result of the decoding process
performed by the compatible decoder 131 based on the
61
information that represents a multiplexing mode of the
compatible image that is included in the image
information supplied from the image information acquiring
unit 123.
5 [0166]
In Step S95, the image generating unit 171
alternately outputs the images A2 and B2 that are the
compatible images, of which the resolution is a half of
the resolution of the separated compatible image to a
10 display device not illustrated in the figure, thereby
displaying the images. Then, the process ends.
[0167]
In addition, in a decoding device that can decode
only a compatible stream that has compatibility with the
15 decoding device 170, a TS of the auxiliary stream is
ignored, and only the process of Steps S83, S94, and S95
is performed. However, in such a case, in the process of
Step S94, a multiplexed image of compatible images is
separated based on not the multiplexing mode of the
20 compatible image that is included in the image
information but the multiplexing mode of the compatible
image that is determined in advance.
[0168]
As above, the decoding device 170 can decode a
25 compatible stream and an auxiliary stream that are
encoded by the encoding device 140 in accordance with a
mode that has compatibility with an existing mode.
[0169]
In addition, since the decoding device 170
30 generates multi-viewpoint images using compatible images
of two viewpoints and auxiliary images of two viewpoints,
62
•
the encoding device 140 may prepare only two imaging
units 141C and 141D other than the existing imaging units
141A and 141B that capture compatible images of two
viewpoints as imaging units used for generating multi-
5 viewpoint images. Accordingly, the installation of the
imaging units for generating multi-viewpoint images can
be easily performed at a low cost.
[0170]

10 [Configuration Example of Encoding Device According to
Third Embodiment]
Fig. 19 is a block diagram that illustrates a
configuration example of an encoding device according to
a third embodiment of the present technology.
15 [0171]
In the configuration illustrated in Fig. 19, the
same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 5 or 14.
Duplicate description will not be presented as is
20 appropriate.
[0172]
The configuration of the encoding device 180
illustrated in Fig. 19 is mainly different from the
configuration illustrated in Fig. 14 in that an image
25 converting unit 181 and an encoder 182 are disposed
instead of the image converting unit 142 and the encoder
145. The encoding device 180 secures the compatibility
with an existing encoding device that encodes a 3D image
of the two-viewpoint mode by encoding images of two
30 viewpoints out of multi-viewpoint images in accordance
with the MVC mode as compatible images.
63
•
[0173]
More specifically, the image converting unit 181 of
the encoding device 180, similarly to the image
converting unit 142 illustrated in Fig. 14, determines an
5 image A2 that is supplied from the imaging unit 141A, of
which the position in the horizontal direction is located
on the inner side out of the imaging units 141A to 1410,
and an image B2 that is supplied from the imaging unit
141B as compatible images. Then, the image converting
10 unit 181 directly supplies the images A2 and B2 that are
compatible images to the encoder 182. In addition, the
image converting unit 181, similarly to the image
converting unit 142, supplies information that designates
the images A2 and B2 as compatible images to the
15 compatibility information generating unit 55.
[0174]
In addition, the image converting unit 181,
similarly to the image converting unit 142, sets images
C2 and 02 other that the images A2 and B2 as auxiliary
20 images and multiplexes the auxiliary images in accordance
with a predetermined multiplexing mode. The image
converting unit 181 supplies a multiplexed image that is
acquired as a result of the multiplexing process to the
encoder 182 and supplies information that represents the
25 multiplexing mode of the auxiliary images to the image
information generating unit 54.
[0175]
The encoder 182 is configured by a compatible
encoder 191 and an auxiliary encoder 152. The compatible
30 encoder 191 of the encoder 182 designates compatible
images from among the compatible images and the
64
•
multiplexed image of the auxiliary images that are
supplied from the image converting unit 181 and encodes
the image A2 out of the compatible images as a base image
in accordance with the existing Ave mode, and encodes the
5 image B2 as a dependent image in accordance with the
existing Mve mode. The compatible encoder 191 generates
an encoded stream by adding various kinds of information
to an encoded image that is acquired as a result of the
encoding process and supplies the encoded stream to the
10 multiplexing unit 59 as a compatible stream.
[0176]
[Description of Process of Encoding Device]
Figs. 20 and 21 are flowcharts that illustrate an
encoding process that is performed by the encoding device
15 180 illustrated in Fig. 19. This encoding process, for
example, is started when the images A2 to D2 are output
from the imaging units 141A to 141D.
[0177]
The process of Steps Slll and Sl12 illustrated in
20 Fig. 20 is the same as the process of Steps S51 and S52
illustrated in Fig. 15, and thus the description thereof
will not be presented.
[0178]
After the process of Step Sl12, in Step Sl13, the
25 image converting unit 181 determines the image A2 that is
supplied from the imaging unit 141A, of which the
position in the horizontal direction is located on the
inner side out of the imaging units 141A to 141D, and the
image B2 that is supplied from the imaging unit 141B as
30 compatible images and determines a multiplexing mode of
the auxiliary images. The image converting unit 181
65
•
supplies information that designates the images A2 and B2
as compatible images to the compatibility information
generating unit 55 and supplies the multiplexing mode of
the auxiliary images to the image information generating
5 unit 54. Then, the process proceeds to Step Sl14.
[0179]
The process of Steps Sl14 to Sl17 is the same as
the process of Steps S14 to S17 illustrated in Fig. 10,
and thus the description thereof will not be presented.
10 The process of Steps Sl18 to S122 is the same as the
process of Steps S59 to S63 illustrated in Fig. 16, and
thus the description thereof will not be presented.
[0180]
After the process of Step S122, in Step S123, the
15 compatible encoder 191 of the encoder 182 encodes the
image A2 out of the compatible images supplied from the
image converti~g unit 181 as a base image in accordance
with the existing Ave mode and encodes the image B2 as a
dependent image in accordance with the existing Mve mode.
20 The compatible encoder 191 generates an encoded stream by
adding various kinds of information to an encoded image
that is acquired as a result of the encoding process and
supplies the encoded stream to the multiplexing unit 59
as a compatible stream. Then, the process proceeds to
25 Step S124.
[0181]
The process of Steps S124 to S126 is the same as
the process of Steps S65 to S67 illustrated in Fig. 16,
and thus the description thereof will not be presented.
30 [0182]
As above, since the encoding device 180 encodes
66
•
images of two viewpoints out of multi-viewpoint images as
compatible images in accordance with an existing encoding
mode, the compatibility with an existing encoding device
that encodes a 3D image of the two-viewpoint mode can be
5 secured.
[0183]
[Configuration Example of Decoding Device]
Fig. 22 is a diagram that illustrates a
configuration example of a decoding device that decodes
10 the multiplexed stream transmitted from the encoding
device 180 illustrated in Fig. 19.
[0184]
In the configuration illustrated in Fig. 22, the
same reference numeral is assigned to a configuration
15 that is the same as that illustrated in Fig. 12.
Duplicate description will not be presented as is
appropriate.
[0185]
The configuration of the decoding device 200
20 illustrated in Fig. 22 is mainly different from the
configuration illustrated in Fig. 12 in that a decoder
201 and an image generating unit 202 are disposed instead
of the decoder 122 and the image generating unit 127.
The decoding device 200 generates images of two
25 viewpoints, or multi-viewpoint images by decoding the
multiplexed stream that is transmitted from the encoding
device 180 and displays the generated images on a display
device not illustrated in the figure.
[0186]
30 More specifically, the decoder 201 of the decoding
device 200 is configured by a compatible decoder 211 and
67
•
an auxiliary decoder 132. The compatible decoder 211 of
the decoder 201, similarly to the compatible decoder 131
illustrated in Fig. 12, identifies the compatible stream
out of the compatible stream and the auxiliary stream
5 that are supplied from the separation unit 121 based on
the information used for specifying a compatible stream
that is supplied from the auxiliary decoder 132. The
compatible decoder 211 decodes an encoded compatible
image that is included in the compatible stream in
10 accordance with a mode that corresponds to the MVC mode
and supplies images A2 and B2 that are acquired as a
result of the decoding process to the image generating
unit 202.
[0187]
15 The image generating unit 202 outputs the image in
accordance with a display instruction supplied from a
viewer, thereby displaying the image on a display device
not illustrated in the figure. Described in more detail,
the image generating unit 202 generates images of three
20 or more viewpoints, which correspond to a display device
not illustrated in the figure, each having the resolution
that is a half of the resolution of the compatible image
or the auxiliary image by using the compatible image, the
multiplexed image of the auxiliary images, the
25 multiplexed image of the disparity images of the
compatible images, and the multiplexed image of the
disparity images of the auxiliary images in accordance
with a viewer's instruction for displaying a 3D image of
the multi-viewpoint mode based on the image information
30 supplied from the image information acquiring unit 123,
the inter-viewpoint distance information supplied from
68
•
the inter-viewpoint distance information acquiring unit
124, the disparity image information supplied from the
disparity image information acquiring unit 125, the
compatibility information supplied from the compatibility
5 information acquiring unit 126, and the like.
[0188]
More specifically, the image generating unit 202
separates the disparity image of each auxiliary image
from the multiplexed image of the disparity images of the
10 auxiliary images based on the information that represents
a multiplexing mode of the disparity images of the
auxiliary images that is included in the disparity image
information supplied from the disparity image information
acquiring unit 125. In addition, the image generating
15 unit 202 separates the disparity image of each compatible
image from the multiplexed image of the disparity images
of the compatible images based on the information that
represents a multiplexing mode of the disparity images of
the compatible images that is included in the disparity
20 image information.
[0189]
Furthermore, the image generating unit 202
separates each auxiliary image from the multiplexed image
of the auxiliary images based on the information that
25 represents a multiplexing mode of the auxiliary images
that is included in the image information supplied from
the image information acquiring unit 123. In addition,
the image generating unit 202 determines the positions of
viewpoints of multi-viewpoint images to be generated
30 based on the inter-viewpoint distance information and the
number of viewpoints that corresponds to a display device
69
•
not illustrated in the figure. Then, the image
generating unit 202 generates an image of each viewpoint
of which the position is determined by using each
compatible image, each auxiliary image, the disparity
5 image of each compatible image, and the disparity image
of each auxiliary image. Then, the image generating unit
202, similarly to the image generating unit 127, converts
the resolution of the generated multi-viewpoint images
into the resolution that is "l/the number of viewpoints"
10 of the resolution of the compatible image or the
auxiliary image, synthesizes the converted images, and
displays the synthesized image on a display device not
illustrated in the figure.
[0190]
15 At this time, the multi-viewpoint images after the
synthesis are displayed such that the viewing angles are
different for each viewpoint, and a viewer can view a 3D
image without wearing glasses by seeing images of
arbitrary two viewpoints with his/her left and right eyes.
20 [0191]
In addition, the image generating unit 202
alternately outputs the images A2 and B2 as compatible
images supplied from the decoder 122 in accordance with
an instruction for displaying a 3D image of the two-
25 viewpoint mode that is supplied from a viewer, thereby
displaying the images on a display device not illustrated
in the figure. At this time, the viewer can view a 3D
image by wearing glasses in which one of a left-eye
shutter and a right-eye shutter is open at the time of
30 displaying the image A2, and the other is open at the
time of displaying the image B2 and viewing the images A2
70
•
and B2 that are alternately displayed on the display
device.
[0192]
[Description of Process of Decoding Device]
5 Fig. 23 is a flowchart that illustrates a decoding
process performed by the decoding device 200 illustrated
in Fig. 22. This decoding process, for example, is
started when a multiplexed stream that is transmitted
from the encoding device 180 illustrated in Fig. 19 is
10 input to the decoding device 200.
[0193]
The process of Steps S141 to S153 illustrated in
Fig. 23 is the same as the process of Steps S81 to S93
illustrated in Fig. 18, and thus the description thereof
15 will not be presented.
[0194]
In a case where it is determined that the
instruction for displaying a 3D image of the twoviewpoint
mode has been made by the viewer in Step S144,
20 the process proceeds to Step S154. In Step S154, the
image generating unit 202 alternately outputs images A2
and B2 that are compatible images acquired as a result of
the decoding process performed by the compatible decoder
211 to a display device not illustrated in the figure,
25 thereby displaying the images.
[0195]
In a decoding device that can decode only a
compatible stream that has compatibility with the
decoding device 200, the TS of the auxiliary stream is
30 ignored, and only the process of Steps S143 and S154 is
performed.
71
!I
t
•
[0196]
As above, the decoding device 200 can decode a
compatible stream and an auxiliary stream that are
encoded in accordance with a mode having compatibility
5 with an existing mode by the encoding device 180.
[0197]

10 In the description presented above, although the
compatibility information, the disparity image
information, the image information, and the interviewpoint
distance information have been described to be
included in the auxiliary stream, the above-described
15 information may be included in the TS. In such a case,
for example, the compatibility information, the disparity
image information, the image information, and the interviewpoint
distance information are described in
descriptors of a PMT (program map table) or an SIT
20 (selection information table) within the packet of the TS
of the auxiliary stream.
[0198]
Figs. 24 to 26 are diagrams that illustrate
examples of the description of the compatibility
25 information and the disparity image information in a case
where the compatibility information, the disparity image
information, the image information, and the interviewpoint
distance information are described in the
descriptor of the PMT.
30 [0199]
As illustrated in Fig. 24, in a case where the
72
•
compatibility information and the disparity image
information are described in the descriptor of the PMT, a
descriptor (3DV_view_structure_descriptor) in which
compatibility information is arranged and a descriptor
5 (depth_map_structure_descriptor) in which the disparity
image information is arranged are disposed as the
descriptor of the PMT.
[0200]
In the descriptor (3DV_view_structure_descriptor),
10 as illustrated in Fig. 25, following a descriptor tag
(descriptor_tag) and a descriptor length
(descriptor_length), the number (num_of_views) of
viewpoints of an image that is an encoding target, a
compatible mode (compatible mode), the information
15 (compatible_view) that designates a compatible image, and
the like are described as the compatibility information.
[0201]
In addition, in the descriptor
(depth_map_structure_descriptor), as illustrated in Fig.
20 26, following the descriptor tag and the descriptor
length, the number (num_of_depth_map) of disparity images,
a flag (is frame_packing) that represents whether or not
disparity images are multiplexed, a multiplexing mode
(frame_packing_mode) of the disparity images, information
25 (comb_frame_packing_views) that designates the disparity
images that are multiplexed, and the like are described
as the disparity image information.
[0202]
In addition, the image information is the same as
30 the disparity image information except that the image
information is not information of the disparity image but
73
•
information of the compatible image and the auxiliary
image, and thus is not illustrated in the figure.
[0203]
Fig. 27 is a diagram that illustrates an example of
5 the description of the inter-viewpoint distance
information in a case where the compatibility information,
the disparity image information, the image information,
and the inter-viewpoint distance information are
described in the descriptors of the PMT.
10 [0204]
As illustrated in Fig. 27, in a case where the
inter-viewpoint distance information is described in the
descriptor of the PMT, a descriptor
(3DV_view_info_descriptor) in which the inter-viewpoint
15 distance information is arranged is disposed as the
descriptor of the PMT.
[0205]
In the descriptor (3DV_view_info_descriptor),
following the descriptor tag and the descriptor length,
20 the number (nurn_of_views) of viewpoints of an image that
is an encoding target, the number (nurn_of_view_distance)
of detected inter-viewpoint distances, a combination
(view[0,1]) of two images that correspond to the interviewpoint
distance, an inter-viewpoint distance
25 (view_distance[view_distance_id]) that corresponds to the
combination, and the like are described as the interviewpoint
distance information.
[0206]
For example, the decoding device according to the
30 present technology can be applied to a display device or
a reproduction device such as a television receiver.
74
1t
•
[0207]
The encoding devices and the decoding devices
according to the first to third embodiments may be
combined. In such a case, the decoding device selects
5 the compatible decoder in accordance with a compatible
mode that is included in the compatibility information.
[0208]
In addition, in the first to third embodiments,
although the resolutions of the auxiliary image and the
10 disparity image are lowered and the images are
multiplexed, only the resolutions thereof may be lowered
without multiplexing the images. Furthermore, the
resolution of the disparity image may not be lowered.
[0209]
15 In addition, the image information, the interviewpoint
distance information, the disparity image
information, and the compatibility information may be
transmitted in a system that is different from that of
the encoded stream without being encoded. Furthermore,
20 the image information, the inter-viewpoint distance
information, the disparity image information, and the
compatibility information may be encoded and transmitted
in a system that is different from that of the encoded
stream.
25 [0210]
In addition, the image information, the interviewpoint
distance information, the disparity image
information, and the compatibility information may be
described in an area (the header, user data, or the like)
30 other than the above-described area of the encoded stream
without being encoded and may be encoded and described in
75
•
a predetermined area of the encoded stream. Furthermore,
the image information, the inter-viewpoint distance
information, the disparity image information, and the
compatibility information may be transmitted as auxiliary
5 information other than the encoded stream.
[0211]

[Configuration Example of Encoding Device According to
Fourth Embodiment]
10 Fig. 28 is a block diagram that illustrates a
configuration example of an encoding device according to
a fourth embodiment of the present technology.
[0212]
In the configuration illustrated in Fig. 28, the
15 same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 5.
Duplicate description will not be presented as is
appropriate.
[0213]
20 The configuration of the encoding device 300
illustrated in Fig. 28 is mainly different from the
configuration illustrated in Fig. 5 in that an image
converting unit 301, a disparity image generating unit
302, an image information generating unit 303, a
25 disparity image information generating unit 304, and an
encoder 305 are arranged instead of the image converting
unit 52, the disparity image generating unit 53, the
image information generating unit 54, the disparity image
information generating unit 57, and the encoder 58.
30 [0214]
The encoding device 300 encodes the auxiliary image
76 I
•
and the disparity image of the auxiliary image, of which
the resolutions are lowered, without multiplexing the
images and transmits information representing that the
resolution at the time of encoding the auxiliary image
5 and the disparity image of the auxiliary image is
different from the resolution of the compatible image and
the like.
[0215]
More specifically, the image converting unit 301 of
10 the encoding device 300, similarly to the image
converting unit 52 illustrated in Fig. 5, determines an
image A1 supplied from an imaging unit 51A as a
compatible image. The image converting unit 301,
similarly to the image converting unit 52, supplies
15 information designating the image A1 as a compatible
image to the compatibility information generating unit 55.
Then, the image converting unit 301, similarly to the
image converting unit 52, directly supplies the image A1
that is the compatible image to the encoder 305.
20 [0216]
In addition, the image converting unit 301 sets
images B1 and C1 other than the image A1 as auxiliary
images and halves the resolutions thereof in the
horizontal direction. Then, the image converting unit
25 301 supplies a half-resolution image B1 and a halfresolution
image C1 that are acquired as a result thereof
to the encoder 305. In addition, the image converting
unit 301 supplies the information representing a
conversion mode of the resolution of the auxiliary image
30 and the information representing that the resolution of
the auxiliary image at the time of encoding is different
77
•
from the resolution of the compatible image to the image
information generating unit 303.
[0217]
The disparity image generating unit 302, similarly
5 to the disparity image generating unit 53 illustrated in
Fig. 5, detects the disparity of each pixel of the images
A1 to C1 by using the images A1 to C1 that are supplied
from the imaging units 51A to 51C. The disparity image
generating unit 302, similarly to the disparity image
10 generating unit 53, generates a disparity image A1' and
directly supplies the generated disparity image to the
encoder 305. In addition, the disparity image generating
unit 302 generates disparity images B1' and C1' and
halves the resolutions thereof in the horizontal
15 direction. The disparity image generating unit 302
supplies the disparity images B1' (hereinafter, referred
to as a half-resolution image B1') and C1' (hereinafter
referred to as a half-resolution image C1') of which the
resolutions are halved to the encoder 305. In addition,
20 the disparity image generating unit 302 supplies the
information representing the conversion mode of the
resolution of the disparity image of the auxiliary image
and the information representing that the resolution at
the time of encoding the disparity image of the auxiliary
25 image is different from the resolution of the compatible
image to the disparity image information generating unit
304.
[0218]
The image information generating unit 303 (setting
30 unit) generates the information representing the
conversion mode of the resolution of the auxiliary image,
78
•
the information (auxiliary image resolution identifying
information) representing whether or not the resolution
of the auxiliary image at the time of encoding is
different from the resolution of the compatible image,
5 and the like as image information based on the
information that is supplied from the image converting
unit 301. The image information generating unit 303
supplies the image information to the encoder 305.
[0219]
10 The disparity image information generating unit 304
generates information that represents the conversion mode
of the resolution of the disparity image of the auxiliary
image, information (auxiliary disparity image converting
mode information) representing whether or not the
15 resolution of the disparity image at the time of encoding
is different from the resolution of the compatible image,
and the like as disparity image information based on the
information that is supplied from the disparity image
generating unit 302. The disparity image information
20 generating unit 304 supplies the disparity image
information to the encoder 305.
[0220]
The encoder 305 is configured by a compatible
encoder 61 and an auxiliary encoder 311. The auxiliary
25 encoder 311 encodes the half-resolution image B1 and the
half-resolution image e1 that are supplied from the image
converting unit 301 and the disparity image A1' of the
compatible image, the half-resolution image B1', and the
half-resolution image e1' that are supplied from the
30 disparity image generating unit 302 in accordance with a
predetermined mode. As an encoding mode employed by the
79
•
auxiliary encoder 311, the Ave mode, the MVe mode, a mode
that is compliant with MPEG2, or the like can be used.
[0221]
In addition, the auxiliary encoder 311, similarly
5 to the auxiliary encoder 62 illustrated in Fig. 5, adds
the image information supplied from the image information
generating unit 303, the compatibility information
supplied from compatibility information generating unit
55, the inter-viewpoint distance information supplied
10 from the inter-viewpoint distance information generating
unit 56, the disparity image information supplied from
the disparity image information generating unit 304, and
the like to an encoded image that is acquired as a result
of the encoding process, thereby generating an encoded
15 stream. The auxiliary encoder 311 supplies the encoded
stream to the multiplexing unit 59 as an auxiliary stream.
[0222]
[Example of Description of Disparity Image Information]
Fig. 29 is a diagram that illustrates an example of
20 the description of the disparity image information
included in the auxiliary stream in the encoding device
300 illustrated in Fig. 28 in a case where a mode that is
compliant with the Ave mode or the Mve mode is used as
the encoding mode.
25 [0223]
As illustrated in Fig. 29, as the disparity image
information (depth_map_structure), the number
(num_of_depth_map) of disparity images, a flag
(is frame_packing) representing whether or not the
30 disparity image is multiplexed, the conversion mode
(decimation_mode) of the resolution of the disparity
80
•
image, information (comb_ decimated_views) designating
the disparity image of which the resolution is lowered,
an encoded resolution flag (is_decimated) (auxiliary
disparity image resolution identifying information) that
5 is a flag representing whether or not the resolution at
the time of encoding is different from the resolution of
the compatible image, and the like are described. As
conversion modes of the resolution, there are a
horizontal half mode in which the resolution in the
10 horizontal direction is halved, a vertical half mode in
which the resolution in the vertical direction is halved,
a horizontal-vertical 1/4 mode in which the resolution in
the horizontal direction and the resolution in the
vertical direction are halved, and the like.
15 [0224]
In addition, in the encoding device 300, the image
information is the same as the disparity image
information except that the image information is not
information of the disparity image but information of the
20 compatible image and the auxiliary image, and thus the
description thereof will not be presented.
[0225]
[Description of Process of Encoding Device]
Figs. 30 and 31 are flowcharts that illustrate the
25 encoding process performed by the encoding device 300
illustrated in Fig. 28. This encoding process, for
example, is started when the images Al to Cl are output
from the imaging units 51A to SIC.
[0226]
30 The encoding process illustrated in Figs. 30 and 31
is the same as the encoding process illustrated in Figs.
81
•
10 and 11 except that a conversion mode of the resolution
is determined instead of the multiplexing mode, and the
auxiliary image and the disparity image of the auxiliary
image are not multiplexed.
5 [0227]
More specifically, the process of Steps S211 and
S212 illustrated in Fig. 30 is the same as the process of
Steps Sll and S12 illustrated in Fig. 10, and, in Step
S213, the image converting unit 301 determines the image
10 Al supplied from the imaging unit 51A as a compatible
image and determines the conversion mode of the
resolution of the auxiliary image. The image converting
unit 301 supplies information that designates the image
A1 as a compatible image to the compatibility information
15 generating unit 55 and supplies the conversion mode of
the resolution of the auxiliary image to the image
information generating unit 303. In addition, the image
converting unit 301 determines that the auxiliary image
is not multiplexed and supplies information representing
20 that the resolution of the auxiliary image at the time of
encoding is different from the resolution of the
compatible image to the image information generating unit
303.
[0228]
25 The process of Steps S214 and S215 is the same as
the process of Steps S14 and S15 illustrated in Fig. 10,
and, in Step S216, the image converting unit 301 lowers
the resolutions of the auxiliary images based on the
conversion mode of the resolution of the auxiliary image
30 that is determined in Step S213 with the images B1 and C1
other than the image A1 set as the auxiliary images. In
82
•
Step S217, the image converting unit 301 inputs the image
Al that is a compatible image and the auxiliary images of
which the resolutions are lowered to the encoder 305.
[0229]
5 The process of Step S218 illustrated in Fig. 31 is
the same as the process of Step S18 illustrated in Fig.
11, and, in Step S219, the disparity image generating
unit 302 determines the conversion mode of the resolution
of the disparity images of the auxiliary images and
10 supplies information that represents the conversion mode
to the disparity image information generating unit 304.
In addition, the disparity image generating unit 302
determines that the disparity images of the auxiliary
images are not multiplexed and supplies the information
15 representing that the resolution of the disparity image
of the auxiliary image at the time of encoding is
different from the resolution of the compatible image to
the disparity image information generating unit 304.
[0230]
20 The process of Step S220 is the same as the process
of Step S20 illustrated in Fig. 11, and, in Step S221,
the disparity image generating unit 302 lowers the
resolution of the disparity images of the auxiliary
images based on the conversion mode of the resolution of
25 the disparity image of the auxiliary image that is
determined in Step S219. In Step S222, the disparity
image generating unit 302 inputs the disparity image AI'
of the compatible image and the disparity images of the
auxiliary images of which the resolutions are lowered to
30 the encoder 305.
[0231]
83
•
The process of Steps S223 to S226 is the same as
the process of Steps S23 to S26 illustrated in Fig. 11.
[0232]
As above, since the resolutions of the auxiliary
5 images and the disparity images are lowered, the encoding
device 300 can suppress the information amount of the
encoding result without increasing the compression rate.
As a result, it can be prevented that the effect of the
compression strain increases due to an increase in the
10 compression rate so as to degrade the image quality. In
addition, since the compatible image is not lowered, the
image quality of the compatible image can be maintained
to be a high image quality. This similarly applies to
the encoding devices 50, 140, and 180.
15 [0233]
[Configuration Example of Decoding Device]
Fig. 32 is a diagram that illustrates a
configuration example of a decoding device that decodes a
multiplexed stream that is transmitted from the encoding
20 device 300 illustrated in Fig. 28.
[0234]
In the configuration illustrated in Fig. 32, the
same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 12.
25 Duplicate description will not be presented as is
appropriate.
[0235]
The configuration of the decoding device 320
illustrated in Fig. 32 is mainly different from the
30 configuration illustrated in Fig. 12 in that a decoder
321 and an image generating unit 322 are disposed instead
84
of the decoder 122 and the image generating unit 127.
The decoding device 320 decodes a multiplexed stream that
is transmitted from the encoding device 300, generates an
image of one viewpoint or a multi-viewpoint image, and
5 displays the generated image on a display device not
illustrated in the figure.
[0236]
More specifically, the decoder 321 of the decoding
device 320 is configured by a compatible decoder 131 and
10 an auxiliary decoder 331. The auxiliary decoder 331,
similarly to the auxiliary decoder 132 illustrated in Fig.
12, supplies information for specifying a compatible
stream to the compatible decoder 131 based on the
compatibility information that is included in the
15 auxiliary stream supplied from the separation unit 121.
The auxiliary decoder 331, similarly to the auxiliary
decoder 132, identifies the auxiliary stream out of the
compatible stream and the auxiliary stream that are
supplied from the separation unit 121 based on the
20 compatibility information.
[0237]
The auxiliary decoder 331 decodes the halfresolution
image B1 and the half-resolution image e1 as
encoded auxiliary images included in the auxiliary stream,
25 the disparity image A1' of the compatible image, and the
half-resolution image B1' and the half-resolution image
e1' as the disparity images of the auxiliary images in
accordance with a mode that corresponds to the auxiliary
encoder 311 illustrated in Fig. 28 based on the image
30 information and the disparity image information that are
included in the auxiliary stream.
85
•
[0238]
The auxiliary decoder 331 supplies the auxiliary
image and the disparity image A1' of the compatible image,
of which the resolutions are lowered, and the disparity
5 image of the auxiliary image, of which the resolution is
lowered, which are acquired as a result of the decoding
process to the image generating unit 322. In addition,
the auxiliary decoder 331, similarly to the auxiliary
decoder 132, supplies the image information included in
10 the auxiliary stream to the image information acquiring
unit 123 and supplies the inter-viewpoint distance
information to the inter-viewpoint distance information
acquiring unit 124. Furthermore, the auxiliary decoder
331, similarly to the auxiliary decoder 132, supplies the
15 disparity image information included in the auxiliary
stream to the disparity image information acquiring unit
125 and supplies the compatibility information to the
compatibility information acquiring unit 126.
[0239]
20 The image generating unit 322 outputs the image in
accordance with a display instruction made by a viewer,
thereby displaying the image on a display device not
illustrated in the figure. More specifically, the image
generating unit 322 generates images of three or more
25 viewpoints, which correspond to a display device not
illustrated in the figure, each having the resolution
that is the same as the resolution of the compatible
image in accordance with a viewer's instruction for
displaying a 3D image of the multi-viewpoint mode by
30 using the compatible image, the auxiliary image of which
the resolution is lowered, the disparity image of the
86
•
compatible image, and the disparity image of the
auxiliary image of which the resolution is lowered based
on the image information supplied from the image
information acquiring unit 123, the inter-viewpoint
5 distance information supplied from the inter-viewpoint
distance information acquiring unit 124, the disparity
image information supplied from the disparity image
information acquiring unit 125, the compatibility
information supplied from the compatibility information
10 acquiring unit 126, and the like.
[0240]
Described in detail, the image generating unit 322
(image converting unit) acquires a disparity image of the
auxiliary image having the same resolution as that of the
15 compatible image by increasing the resolution of the
disparity image of the auxiliary image that has been
lowered based on the conversion mode of the resolution
and the information representing whether or not the
resolution of the disparity image at the time of encoding
20 is different from the resolution of the compatible image
that are included in the disparity image information
supplied from the disparity image information acquiring
unit 125. In addition, the image generating unit 322
acquires the auxiliary image having the same resolution
25 as that of the compatible image by increasing the
resolution of the auxiliary image, which has been lowered,
based on the conversion mode of the resolution and the
information representing whether or not the resolution of
the auxiliary image at the time of encoding is different
30 from the resolution of the compatible image that are
included in the image information supplied from the image
87
information acquiring unit 123.
[0241]
Furthermore, the image generating unit 322
determines the position of each viewpoint of a multi-
5 viewpoint image to be generated based on the number of
viewpoints that corresponds to a display device not
illustrated in the figure and the inter-viewpoint
distance information. Then, the image generating unit
322 generates an image of each viewpoint of which the
10 position is determined by using the compatible image,
each auxiliary image having the same resolution as the
resolution of the compatible image, the disparity image
of the compatible image, and the disparity image of each
auxiliary image. Then, the image generating unit 322
15 converts the resolution of the generated image of each
viewpoint into resolution that is "1/ the number of
viewpoints" of the resolution of the compatible image,
synthesizes the converted images, and displays a
resultant image on a display device not illustrated in
20 the figure.
[0242]
At this time, the multi-viewpoint images after the
synthesis are displayed such that the viewing angles are
different for each viewpoint, and a viewer can view a 3D
25 image without wearing glasses by seeing images of
arbitrary two viewpoints with his left and right eyes.
[0243]
In addition, the image generating unit 322 outputs
an image Al that is a compatible image supplied from the
30 compatible decoder 131 of the decoder 321 in accordance
with an instruction for displaying a 2D image that is
88
supplied from a viewer, thereby displaying the image on a
display device not illustrated in the figure.
Accordingly, the viewer can view the 2D image.
[0244]
5 [Description of Process of Decoding Device]
Fig. 33 is a flowchart that illustrates a decoding
process performed by the decoding device 320 illustrated
in Fig. 32. This decoding process, for example, is
started when a multiplexed stream that is transmitted
10 from the encoding device 300 illustrated in Fig. 28 is
input to the decoding device 320.
[0245]
The decoding process illustrated in Fig. 33 is the
same as the decoding process illustrated in Fig. 13
15 except that the auxiliary image and the disparity image
are decoded based on the image information and the
disparity image information that are included in the
auxiliary stream, and the resolutions of the auxiliary
images and the disparity images of the auxiliary images
20 are increased based on the image information and the
disparity image information.
[0246]
More specifically, the process of Steps S231 to
S234 illustrated in Fig. 33 is the same as the process of
25 Steps S31 to S34 illustrated in Fig. 13, and, in Step
S235, the auxiliary decoder 331 decodes the encoded
auxiliary image included in the auxiliary stream, the
disparity image AI' of the compatible image, and the
disparity images of the auxiliary images in accordance
30 with a mode that corresponds to the auxiliary encoder 311
illustrated in Fig. 28 based on the image information and
89
•
the disparity image information that are included in the
auxiliary stream.
[0247]
More specifically, in a case where the encoded
5 resolution flag included in the image information is a
flag representing that the resolution at the time of
encoding is different from the resolution of the
compatible image, the auxiliary decoder 331 sets the
resolution of the encoding target based on the conversion
10 mode of the resolution. For example, in a case where the
conversion mode of the resolution is the horizontal half
mode, a resolution that is a half of the resolution of
the compatible image in the horizontal direction is set
as the resolution of the encoding target. On the other
15 hand, in a case where the encoded resolution flag
included in the image information is a flag representing
that the resolution at the time of encoding is not
different from the resolution of the compatible image,
the auxiliary decoder 331 sets the resolution of the
20 encoding target to a resolution that is the same as the
resolution of the compatible image. Similarly, the
auxiliary decoder 331 sets the resolution of the encoding
target based on the encoded resolution flag that is
included in the disparity image information.
25 [0248]
The auxiliary decoder 331 decodes the auxiliary
image, the disparity image A1' of the compatible image,
and the disparity image of the auxiliary image based on
the set resolution of the encoding target. The auxiliary
30 decoder 331 supplies the auxiliary image, the disparity
image A1' of the compatible image, and the disparity
90
•
image of the auxiliary image of which the resolution is
lowered, which are acquired as a result of decoding, to
the image generating unit 322. In addition, the
auxiliary decoder 331 supplies the image information
5 included in the auxiliary stream to the image information
acquiring unit 123 and supplies the inter-viewpoint
distance information to the inter-viewpoint distance
information acquiring unit 124. Furthermore, the
auxiliary decoder 331 supplies the disparity image
10 information included in the auxiliary stream to the
disparity image information acquiring unit 125 and
supplies the compatibility information to the
compatibility information acquiring unit 126.
[0249]
15 The process of Steps S236 to S239 is the same as
the process of Steps S36 to S39 illustrated in Fig. 13,
and, in Step S240, the image generating unit 322
increases the resolution of the auxiliary image, which
has been lowered, based on the conversion mode of the
20 resolution and the information representing whether or
not the resolution of the auxiliary image at the time of
encoding is different from the resolution of the
compatible image, which are included in the image
information supplied from the image information acquiring
25 unit 123, thereby acquiring the auxiliary image having
the same resolution as the resolution of the compatible
image.
[0250]
In Step S241, the image generating unit 322
30 increases the resolution of the disparity image of the
auxiliary image, which has been lowered, based on the
91
conversion mode of the resolution and the information
representing whether or not the resolution of the
disparity image of the auxiliary image at the time of
encoding is different from the resolution of the
5 compatible image that are included in the disparity image
information supplied from the disparity image information
acquiring unit 125, thereby acquiring the disparity image
of the auxiliary image having the same resolution as the
resolution of the compatible image.
10 [0251]
The process of Step S242 is the same as the process
of Step S40 illustrated in Fig. 13, and, in Step S243,
the image generating unit 322 generates an image of each
viewpoint having the resolution that is the same as the
15 resolution of the compatible image based on the position
of each viewpoint that is determined in Step S242, the
compatibility information supplied from the compatibility
information acquiring unit 126, and the like by using the
compatible image, the auxiliary image having the
20 resolution that is the same as the resolution of the
compatible image, the disparity image of the compatible
image, and the disparity image of the auxiliary image.
[0252]
The process of Steps S244 to S246 is the same as
25 the process of Steps S42 to S44 illustrated in Fig. 13.
[0253]
In a decoding device that can decode only a
compatible stream that has compatibility with the
decoding device 320, the TS of the auxiliary stream is
30 ignored, and only the process of Steps S233 and S246 is
performed.
92
•
[0254]
As above, the decoding device 320 can decode a
compatible stream and an auxiliary stream that are
encoded in accordance with a mode having compatibility
5 with an existing mode by the encoding device 300.
[0255]
[Another Example of Arrangement of Disparity Image
Information]
In the description presented above, in the fourth
10 embodiment, although the compatibility information, the
disparity image information, the image information, and
the inter-viewpoint distance information are included in
the auxiliary stream, similarly to the first to third
embodiments, such information may be included in the TS.
15 In such a case, for example, in descriptors of the PMT or
the SIT that is included in the packet of the TS of the
auxiliary stream, the compatibility information, the
disparity image information, the image information, and
the inter-viewpoint distance information are described.
20 [0256]
Fig. 34 is a diagram that illustrates an example of
the description of a descriptor
(depth_map_structure_descriptor) in which the disparity
image information is arranged in a case where the
25 compatibility information, the disparity image
information, the image information, and the interviewpoint
distance information are described in
descriptors of the PMT.
[0257]
30 As illustrated in Fig. 34, in the descriptor
(depth_map_structure_descriptor), following the
93
descriptor tag and the descriptor length, the number
(num_of_depth_map) of disparity images, a flag
(is_frame_packing) representing whether or not the
disparity image is multiplexed, a conversion mode
5 (decimation_mode) of the resolution of the disparity
image, information (comb_ decimated_views) that
designates the disparity image of which the resolution
has been lowered, the encoded resolution flag
(is_decimated), and the like are described as the
10 disparity image information.
[0258]
In the fourth embodiment, although the auxiliary
image and the disparity image of the auxiliary image are
not multiplexed, the images may be multiplexed. In such
15 a case, in the image information and the disparity image
information, the information representing the
multiplexing mode and the information designating the
multiplexed disparity image are included. Then, the
decoding device 320 separates the auxiliary image and the
20 disparity image of the auxiliary image that are acquired
as a result of the decoding process based on the image
information and the disparity image information,
increases the resolutions of the auxiliary image and the
disparity image, which are separated, and uses resultant
25 images for generating an image of each viewpoint.
[0259]
In the fourth embodiment, although the encoded
resolution flag and the like are included in the image
information and the disparity image information according
30 to the first embodiment, similarly, the encoded
resolution flag and the like may be included also in the
94
t
image information and the disparity image information
according to the second and third embodiments. In
addition, in the fourth embodiment, although the
resolutions of the auxiliary image and the disparity
5 image of the auxiliary image are increased when an image
of each viewpoint is generated, the conversion of the
resolution performed at the time of generating an image
of each viewpoint may be a conversion of increasing or
decreasing· the resolution as long as the conversion is a
10 conversion from which all the images used for the
generation process have the same resolution. Also in the
first to third embodiments, similarly to the fourth
embodiment, the resolution is converted when an image of
each viewpoint is generated. In the first to third
15 embodiments, although an image of each viewpoint having
the resolution that is a half of the resolution of the
compatible image is generated, similarly to the fourth
embodiment, an image of each viewpoint having the same
resolution as the resolution of the compatible image may
20 be generated.
[0260]

[Configuration Example of Encoding Device According to
Fifth Embodiment]
25 Fig. 35 is a block diagram that illustrates a
configuration example of an encoding device according to
a fifth embodiment of the present technology.
[0261]
In the configuration illustrated in Fig. 35, the
30 same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 14.
95
t
Duplicate description will not be presented as is
appropriate.
[0262]
The configuration of the encoding device 440
5 illustrated in Fig. 35 is mainly different from the
configuration illustrated in Fig. 14 in that a disparity
image generating unit 441 and an encoder 442 are arranged
instead of the disparity image generating unit 143 and
the encoder 145. The encoding device 440 does not
10 multiplex the disparity images of compatible images in
the spatial direction but multiplexes the disparity
images of the compatible images in the time direction and
encodes the multiplexed disparity images.
[0263]
15 More specifically, the disparity image generating
unit 441 of the encoding device 440 detects the disparity
of each pixel of images A2 to D2 using the images A2 to
D2 that are supplied from the imaging units 141A to 141D.
The disparity image generating unit 441 generates a
20 disparity image A2' of the image A2 and a disparity image
B2' of the image B2, which are compatible images, and a
disparity image C2' of the image C2 and a disparity image
D2' of the image D2, which are auxiliary images, based on
the detection result.
25 [0264]
In addition, the disparity image generating unit
441, similarly to the disparity image generating unit 143
illustrated in Fig. 14, multiplexes the disparity images
C2' and D2' in the spatial direction in accordance with a
30 predetermined multiplexing mode. In addition, the
disparity image generating unit 441 multiplexes disparity
96
images A2' and B2' and a multiplexed image of disparity
images of the auxiliary images in the time direction.
The disparity image generating unit 441 supplies a
multiplexed image in which disparity images A2' and B2'
5 and the multiplexed image of the disparity images of the
auxiliary images corresponding to one frame exist within
one frame time, which is acquired as a result thereof, to
the encoder 442 as a time-multiplexed image.
[0265]
10 In addition, the disparity image generating unit
441 supplies information that represents a multiplexing
mode of the disparity images of the auxiliary images and
a multiplexing mode (hereinafter, referred to as a frame
sequential mode) in the time direction as a multiplexing
15 mode of the disparity images of the compatible images and
the auxiliary images to the disparity image information
generating unit 57.
[0266]
The encoder 442 is configured by a compatible
20 encoder 151 and an auxiliary encoder 451. The auxiliary
encoder 451 of the encoder 442 encodes the multiplexed
image of the auxiliary images that is supplied from the
image converting unit 142 and the time-multiplexed image
supplied from the disparity image generating unit 441 in
25 accordance with the 3DV mode. Here, the 3DV mode is a
mode that is used for encoding a display image in
accordance with a mode of multi-viewpoints that is
compliant with the Ave mode or the Mve mode. The
auxiliary encoder 451 supplies an encoded stream that is
30 acquired as a result of the encoding process to the
multiplexing unit 443 as an auxiliary stream.
97
[0267]
The multiplexing unit 443 generates TSs by using
the compatible stream supplied from the compatible
encoder 151, the auxiliary stream supplied from the
5 auxiliary encoder 451, the image information supplied
from the image information generating unit 54, the
compatibility information supplied from the compatibility
information generating unit 55, the inter-viewpoint
distance information supplied from the inter-viewpoint
10 distance information generating unit 56, the disparity
image information supplied from the disparity image
information generating unit 57, and the like. The
multiplexing unit 443 multiplexes the generated TSs and
transmits a multiplexed stream that is acquired as a
15 result of the multiplexing process.
[0268]
Hereinafter, the image information, the
compatibility information, the inter-viewpoint distance
information, and the disparity image information will be
20 collectively referred to as auxiliary information.
[0269]
[Example of Description of Disparity Image Information]
Fig. 36 is a diagram that illustrates an example of
the description of the disparity image information in the
25 encoding device 440 illustrated in Fig. 35 in a case
where the auxiliary information is described in a
descriptor of the PMT.
[0270]
As illustrated in Fig. 36, in a descriptor
30 (depth_map_structure_descriptor) in which the disparity
image information is arranged, similarly to the case of
98
Fig. 25, following a descriptor tag (descriptor_tag) and
a descriptor length (descriptor_length), the disparity
image information is described. In addition, as the
disparity image information, similarly to the case of Fig.
5 25, the number (nurn_of_depth_map) of disparity images, a
flag (is_frame_packing) representing whether or not the
disparity images are multiplexed, a multiplexing mode
(frame_packing_mode) of the disparity images, information
(cornb_frame_packing_views) that designates the disparity
10 images that are multiplexed, and the like are described.
However, as the multiplexing mode of the disparity mode,
not only the side-by-side mode (SBS) or the top-andbottom
mode (TOB) but also the frame-sequential mode is
described.
15 [0271]
In the encoding device 440 illustrated in Fig. 35,
although the auxiliary information is included in the TS,
the auxiliary information may be included in the
auxiliary stream.
20 [0272]
Fig. 37 is a diagram that illustrates an example of
the description of the disparity image information
included in the auxiliary stream in a case where the
auxiliary information is included in the auxiliary stream.
25 [0273]
In such a case, as illustrated in Fig. 6, the
disparity image information (depth_map_structure), for
example, is arranged in the SEI that is included in the
auxiliary stream.
30 [0274]
In addition, as the disparity image information
99
(depth_map_structure), as illustrated in Fig. 37, the
number (num_of_depth_map) of disparity images (depth map),
a flag (is_frame_packing) representing whether or not the
disparity images are multiplexed, a multiplexing system
5 (frame_packing_mode) of the disparity images, information
(comb_frame_packing_views) that designates the disparity
images that are multiplexed, and the like are described.
However, as the multiplexing mode, similarly to the case
of Fig. 36, not only the side-by-side mode (SBS) or the
10 top-and-bottom mode (TOB), but also a frame-sequential
mode is described.
[0275]
Although not illustrated in the figure, the image
information is the same as the disparity image
15 information except that the image information is not a
disparity image but information of the compatible images
and the auxiliary images.
[0276]
[Description of Process of Encoding Device]
20 Figs. 38 and 39 are flowcharts that illustrate an
encoding process performed by the encoding device 440
illustrated in Fig. 35. This encoding process, for
example, is started when images A2 to D2 are output from
imaging units 141A to 141D.
25 [0277]
The process of Steps S451 to S455 illustrated in
Figs. 38 and 39 is the same as the process of Steps S51
to S55 illustrated in Figs. 15 and 16, and thus the
description thereof will not be presented.
30 [0278]
In Step S456, the image information generating unit
100
54 generates information that represents the multiplexing
modes of the compatible images and the auxiliary images
and the like as image information based on the
information supplied from the image converting unit 142
5 and inputs the generated image information to the
multiplexing unit 443. Then, the process proceeds to
Step S457.
[0279]
The process of Steps S457 to S459 is the same as
10 the process of Steps S57 to S59 illustrated in Figs. 15
and 16, and thus the description thereof will not be
presented.
[0280]
In Step S460 illustrated in Fig. 39, the disparity
15 image generating unit 441 determines the multiplexing
mode of the disparity images of the auxiliary images and
the multiplexing mode of the multiplexed image of the
disparity images of the compatible images and the
disparity images of the auxiliary image and supplies
20 information that represents the multiplexing modes to the
disparity image information generating unit 57.
[0281]
In Step S461, the disparity image information
generating unit 57 generates information that represents
25 the multiplexing mode of the disparity images of the
auxiliary images and the multiplexing mode of the
multiplexed image of the disparity images of the
compatible images and the disparity images of the
auxiliary images and the like as disparity image
30 information based on the information that is supplied
from the disparity image generating unit 441 and inputs
101
•
the generated disparity image information to the
multiplexing unit 443.
[0282]
In Step S462, the disparity image generating unit
5 441 multiplexes the disparity images C2' and 02' of the
auxiliary images in the spatial direction and multiplexes
the disparity images A2' and B2' of the compatible images
and the multiplexed image of the disparity images of the
auxiliary images in the time direction, based on the
10 multiplexing mode that is determined in Step S460.
[0283]
In Step S463, the disparity image generating unit
441 inputs the time-multiplexed image that is acquired as
a result of the multiplexing process of Step S462 to the
15 encoder 442.
[0284]
In Step S464, the compatible encoder 151 of the
encoder 442 encodes the multiplexed image of the
compatible images that is supplied from the image
20 converting unit 142 in accordance with the existing AVC
mode and supplies an encoded stream that is acquired as a
result thereof to the multiplexing unit 443 as a
compatible stream.
[0285]
25 In Step 8465, the auxiliary encoder 451 encodes the
multiplexed image of the auxiliary images that is
supplied from the image converting unit 142 and the timemultiplexed
image that is supplied from the disparity
image generating unit 441 in accordance with the 30V mode.
30 The auxiliary encoder 451 supplies an encoded stream that
is acquired as a result of the encoding process to the
102
•
multiplexing unit 443 as an auxiliary stream.
[0286]
In Step S466, the multiplexing unit 443 generates
TSs from the compatible stream supplied from the
5 compatible encoder 151 and the auxiliary stream and the
auxiliary information that are supplied from the
auxiliary encoder 451, performs a multiplexing process
thereof, and transmits a multiplexed stream acquired as a
result thereof. This multiplexed stream, for example, is
10 recorded on a BD or the like or is transmitted as a
broadcast stream. Then, the process ends.
[0287]
[Configuration Example of Decoding Device]
Fig. 40 is a diagram that illustrates a
15 configuration example of a decoding device that decodes a
multiplexed stream that is transmitted from the encoding
device 440 illustrated in Fig. 35.
[0288]
In the configuration illustrated in Fig. 40, the
20 same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 17.
Duplicate description will not be presented as is
appropriate.
[0289]
25 The configuration of the decoding device 460
illustrated in Fig. 40 is mainly different from the
configuration illustrated in Fig. 17 in that a separation
unit 461, a decoder 462, and an image generating unit 463
are disposed instead of the separation unit 121, the
30 decoder 122, and the image generating unit 171. The
decoding device 460 generates images of two viewpoints or
103
•
multi-viewpoint images by decoding the multiplexed stream
that is transmitted from the encoding device 440 and
displays the generated images on a display device not
illustrated in the figure.
5 [0290]
More specifically, the separation unit 461 of the
decoding device 460 receives a multiplexed stream that is
transmitted from the encoding device 440 and separates
each TS. The separation unit 461 extracts a compatible
10 stream and an auxiliary stream from the separated TS and
supplies the extracted streams to the decoder 462. In
addition, the separation unit 461 extracts auxiliary
information that is included in the TS, supplies image
information included in the auxiliary information to the
15 image" information acquiring unit 123, and supplies
viewpoint distance information to the inter-viewpoint
distance information acquiring unit 124. Furthermore,
the separation unit 461 supplies disparity image
information included in the auxiliary information to the
20 disparity image information acquiring unit 125 and
supplies compatibility information to the compatibility
information acquiring unit 126 and the decoder 462.
[0291]
The decoder 462 is configured by a compatible
25 decoder 471 and an auxiliary decoder 472. The compatible
decoder 471 of the decoder 462 identifies the compatible
stream out of the compatible stream and the auxiliary
stream supplied from the separation unit 461 based on the
compatibility information that is supplied from the
30 separation unit 461. The compatible decoder 471 decodes
the encoded compatible image included in the compatible
104
" stream in accordance with a mode that corresponds to the
Ave mode and supplies a multiplexed image of the
compatible images acquired as a result of the decoding
process to the image generating unit 463.
5 [0292]
The auxiliary decoder 472 identifies the auxiliary
stream out of the compatible stream and the auxiliary
stream that are supplied from the separation unit 461
based on the compatibility information that is supplied
10 from the separation unit 461. The auxiliary decoder 472
decodes the multiplexed image of auxiliary images and the
time-multiplexed image, which are encoded, included in
the auxiliary stream supplied from the separation unit
461 in accordance with a mode that corresponds to the
15 auxiliary encoder 451 illustrated in Fig. 35. The
auxiliary decoder 472 supplies the multiplexed image of
the auxiliary images and the time-multiplexed image that
are acquired as a result of the decoding process to the
image generating unit 463.
20 [0293]
The image generating unit 463 outputs an image in
accordance with an instruction supplied from a viewer and
displays the image on a display device not illustrated in
the figure. Described in more detail, the image
25 generating unit 463 generates images of three or more
viewpoints, which correspond to a display device not
illustrated in the figure, each having the resolution
that is a half of the resolution of the compatible image
or the auxiliary image by using the multiplexed image of
30 compatible images, the multiplexed image of the auxiliary
images, and the time-multiplexed image in accordance with
105
•
a viewer's instruction for displaying a 3D image of the
multi-viewpoint mode based on the image information
supplied from the image information acquiring unit 123,
the inter-viewpoint distance information supplied from
5 the inter-viewpoint distance information acquiring unit
124, the disparity image information supplied from the
disparity image information acquiring unit 125, the
compatibility information supplied from the compatibility
information acquiring unit 126, and the like.
10 [0294]
Described in more detail, the image generating unit
463 separates disparity images A2' and B2' that are the
disparity images of the compatible images and the
multiplexed image of the disparity images of the
15 auxiliary images from the time-multiplexed image based on
the information representing the frame sequential mode as
the multiplexing mode of the multiplexed image of the
disparity images of the compatible images and the
disparity images of the auxiliary images included in the
20 disparity image information that is supplied from the
disparity image information acquiring unit 125. Then,
the image generating unit 463 separates the disparity
images C2' and D2' from the multiplexed image of the
disparity images of the auxiliary images based on the
25 multiplexing mode of the disparity images of the
auxiliary images that is included in the disparity image
information.
[0295]
In addition, the image generating unit 463,
30 similarly to the image generating unit 171 illustrated in
Fig. 17, separates each auxiliary image from the
106
multiplexed image of the auxiliary images based on the
information that represents the multiplexing mode of the
auxiliary images that is included in the image
information supplied from the image information acquiring
5 unit 123. Furthermore, the image generating unit 463,
similarly to the image generating unit 171, separates
each compatible image from the multiplexed image of the
compatible images based on the information that
represents the multiplexing mode of the compatible images
10 that is included in the image information.
[0296]
Furthermore, the image generating unit 463,
similarly to the image generating unit 171, determines
the position of each viewpoint of multi-viewpoint images
15 to be generated based on the inter-viewpoint distance
information and the number of viewpoints corresponding to
a display device not illustrated in the figure. Then,
the image generating unit 463, similarly to the image
generating unit 171, generates the image of each
20 viewpoint of which the position is determined by using
each compatible image, each auxiliary image, the
disparity image of each compatible image, and the
disparity image of each auxiliary image. Then, the image
generating unit 463, similarly to the image generating
25 unit 171, converts the resolution of the generated multiviewpoint
images into resolution that is ~1 / the number
of viewpoints" of the resolution of the compatible image
or the auxiliary image, synthesizes the images, and
displays the synthesized image on a display device not
30 illustrated in the figure.
[0297]
107
•
At this time, the multi-viewpoint images after the
synthesis are displayed such that the viewing angles are
different for each viewpoint, and a viewer can view a 3D
image without wearing glasses by seeing images of
5 arbitrary two viewpoints with his left and right eyes.
[0298]
In addition, the image generating unit 463,
similarly to the image generating unit 171 separates the
multiplexed image of the compatible images that is
10 supplied from the decoder 462 into images A2 and B2 of
which the resolution is a half of the resolution of the
compatible image based on the image information supplied
from the image information acquiring unit 123 in
accordance with an instruction for displaying a 3D image
15 of the two-viewpoint mode that is supplied from a viewer.
Then, the image generating unit 463, similarly to the
image generating unit 171, alternately outputs the
separated images A2 and B2 of which the resolution is a
half of the resolution of the compatible image, thereby
20 displaying the separated images on a display device not
illustrated in the figure. At this time, the viewer can
view a 3D image by wearing glasses in which one a lefteye
shutter and a right-eye shutter is open at the time
of displaying the image A2, and the other is open at the
25 time of displaying the image B2 and viewing the images A2
and B2 that are alternately displayed on the display
device.
[0299]
[Description of Decoding Process]
30 Fig. 41 is a flowchart that illustrates a decoding
process performed by the decoding device 460 illustrated
108
•
in Fig. 40. This decoding process, for example, is
started when a multiplexed stream that is transmitted
from the encoding device 440 illustrated in Fig. 35 is
input to the decoding device 460.
5 [0300]
In Step S471 illustrated in Fig. 41, the separation
unit 461 of the decoding device 460 receives the
multiplexed stream that is transmitted from the encoding
device 50 and separates each TS. The separation unit 461
10 supplies the compatible stream and the auxiliary stream
that are included in the separated TS to the decoder 462.
In addition, the separation unit 461 extracts the
auxiliary information included in the TS, supplies the
image information included in the auxiliary information
15 to the image information acquiring unit 123, and supplies
the viewpoint distance information to the inter-viewpoint
distance information acquiring unit 124. Furthermore,
the separation unit 461 supplies the disparity image
information that is included in the auxiliary information
20 to the disparity image information acquiring unit 125 and
supplies the compatibility information to the
compatibility information acquiring unit 126 and the
decoder 462.
[0301]
25 In Step S472, the compatible decoder 471 identifies
the compatible stream out of the compatible stream and
the auxiliary stream that are supplied from the
separation unit 461 based on the compatibility
information that is supplied from the separation unit 461.
30 [0302]
In Step S473, the compatible decoder 471 decodes
109
•
the compatible image that is included in the compatible
stream in accordance with a mode that corresponds to the
Ave mode and supplies a multiplexed image of the
compatible images that is acquired as a result of the
5 decoding process to the image generating unit 463.
[0303]
In Step S474, the image generating unit 463
determines whether or not an instruction for displaying a
3D image of the two-viewpoint mode has been made by a
10 viewer. In a case where it is determined that the
instruction for displaying a 3D image of the twoviewpoint
mode has not been made by the viewer in Step
S474, in other words, in a case where an instruction for
displaying a 3D image of the multi-viewpoint mode has
15 been made, the auxiliary decoder 472 identifies the
auxiliary stream out of the compatible stream and the
auxiliary stream that are supplied from the separation
unit 461 based on the compatibility information.
[0304]
20 Then, in Step S475, the auxiliary decoder 472
extracts a multiplexed image of the auxiliary images and
a time-multiplexed image, which are encoded, from the
auxiliary stream and decodes the extracted multiplexed
images in accordance with a mode that corresponds to the
25 auxiliary encoder 62 illustrated in Fig. 2. The
auxiliary decoder 472 supplies the multiplexed image of
the auxiliary images and the time-multiplexed image that
are acquired as a result of the decoding process to the
image generating unit 127, and the process proceeds to
30 Step S476.
[0305]
110
•
In Step S476, the image information acquiring unit
123 acquires the image information that is supplied from
the separation unit 461 and inputs the acquired image
information to the image generating unit 463. In Step
5 S477, the inter-viewpoint distance information acquiring
unit 124 acquires the inter-viewpoint distance
information that is supplied from the separation unit 461
and inputs the acquired inter-viewpoint distance
information to the image generating unit 463.
10 [0306]
In Step 8478, the disparity image information
acquiring unit 125 acquires the disparity image
information that is supplied from the separation unit 461
and inputs the acquired disparity image information to
15 the image generating unit 463. In Step S479, the
compatibility information acquiring unit 126 acquires the
compatibility information that is supplied from the
separation unit 461 and inputs the acquired compatibility
information to the image generating unit 463.
20 [0307]
In Step S480, the image generating unit 463
determines the position of each viewpoint of a multiviewpoint
image to be generated based on the interviewpoint
distance information supplied from the inter-
25 viewpoint distance information acquiring unit 124 and the
number of viewpoints that corresponds to a display device
not illustrated in the figure.
[0308]
In Step 8481, the image generating unit 463
30 generates images of the viewpoints each having the
resolution that is a half of the resolution of the
111
•
compatible image or the auxiliary image based on the
position of each viewpoint determined in Step S480, the
image information supplied from the image information
acquiring unit 123, the disparity image information
5 supplied from the disparity image information acquiring
unit 125, the compatibility information supplied from the
compatibility information acquiring unit 126, and the
like by using the multiplexed image of the compatible
images, the multiplexed image of the auxiliary images,
10 and the time-multiplexed image.
[0309]
Steps S482 and S483 are the same as Steps S92 and
S93 illustrated in Fig. 18, and thus the description
thereof will not be presented.
15 [0310]
In a case where it is determined that the
instruction for displaying a 3D image of the twoviewpoint
mode has been made by a viewer in Step S474, in
Step S484, the image information acquiring unit 123
20 acquires the image information that is supplied from the
separation unit 461 and inputs the acquired image
information to the image generating unit 463.
[0311]
The process of Steps S485 and S486 is the same as
25 the process of Steps S94 and S95 illustrated in Fig. 18,
and thus the description thereof will not be presented.
[0312]
In addition, in a decoding device that can decode
only a compatible stream that has compatibility with the
30 decoding device 460, the auxiliary stream is ignored, and
only the process of Steps S473, S484, and S485 is
112
•
performed. However, in such a case, in the process of
Step S484, a multiplexed image of the compatible images
is separated based on the multiplexing mode determined in
advance.
5 [0313]

[Configuration Example of Encoding Device According to
Sixth Embodiment]
Fig. 42 is a block diagram that illustrates a
10 configuration example of an encoding device according to
a sixth embodiment of the present technology.
[0314]
In the configuration illustrated in Fig. 42, the
same reference numeral is assigned to a configuration
15 that is the same as that illustrated in Fig. 35.
Duplicate description will not be presented as is
appropriate.
[0315]
The configuration of the encoding device 480
20 illustrated in Fig. 42 is mainly different from the
configuration illustrated in Fig. 35 in that a disparity
image generating unit 481, an encoder 482, and a
disparity image information generating unit 483 are
disposed instead of the disparity image generating unit
25 143, the encoder 145, and the disparity image information
generating unit 57. The encoding device 480 encodes a
common disparity image that shows a disparity value that
is common to the compatible images and a common disparity
image of the auxiliary images.
30 [0316]
More specifically, the disparity image generating
113
•
unit 481 of the encoding device 480 detects the disparity
of each pixel between images A2 and B2 and the disparity
of each pixel between images C2 and 02 using the images
A2 to 02 that are supplied from the imaging units 141A to
5 1410. The disparity image generating unit 481 generates
a disparity image formed by disparity values that
represent the disparity of each pixel between the images
A2 and B2 as a common disparity image AB2' of the
compatible images and supplies the generated common
10 disparity image to the encoder 482. In addition, the
disparity image generating unit 481 generates a disparity
image formed by disparity values that represent the
disparity of each pixel between the images C2 and 02 as a
common disparity image C02' of the auxiliary images and
15 supplies the generated common disparity image to the
encoder 482.
[0317]
Furthermore, the disparity image generating unit
481 supplies information that represents that the
20 disparity images of the compatible images and the
auxiliary images are common disparity images to the
disparity image information generating unit 483.
[0318]
The encoder 482 is configured by a compatible
25 encoder 151 and an auxiliary encoder 491. The auxiliary
encoder 491 encodes the multiplexed image of auxiliary
images that is supplied from the image converting unit
142 and the common disparity image AB2' of the compatible
images and the common disparity image C02' of the
30 auxiliary images that are supplied from the disparity
image generating unit 481 in accordance with the 30V mode.
114
The auxiliary encoder 491 supplies an encoded stream
acquired as a result thereof to the multiplexing unit 443
as an auxiliary stream.
[0319]
5 The disparity image information generating unit 483
generates information representing that the disparity
images of the compatible images and the auxiliary images
are common disparity images and the like as disparity
image information based on the information that is
10 supplied from the disparity image generating unit 53 and
supplies the generated disparity image information to the
multiplexing unit 443.
[0320]
[Description of Process of Encoding Device]
15 Figs. 43 and 44 represent a flowchart that
illustrates an encoding process that is performed by the
encoding device 480 illustrated in Fig. 42. This
encoding process, for example, is started when images A2
to D2 are output from the imaging units 141A to 141D.
20 [0321]
The process of Steps S491 to S498 illustrated in
Fig. 43 is similar to that of Steps S451 to S458
illustrated in Fig. 38, and thus, the description thereof
will not be presented.
25 [0322]
In Step S499 illustrated in Fig. 44, the disparity
image generating unit 481 detects the disparity of each
pixel between images A2 and B2 and the disparity of each
pixel between images C2 and D2 using the images A2 to D2
30 that are supplied from the imaging units 141A to 141D.
Then, the disparity image generating unit 481 generates a
115
•
common disparity image AB2' that is formed by disparity
values that represent the disparity of each pixel between
the images A2 and B2 and a common disparity image C02'
that is formed by disparity values that represent the
5 disparity of each pixel between the images C2 and 02.
[0323]
In Step S500, the disparity image information
generating unit 483 generates information representing
that the disparity images of the compatible images and
10 the auxiliary images are common disparity images and the
like as disparity image information based on the
information that is supplied from the disparity image
generating unit 481 and inputs the generated disparity
image information to the multiplexing unit 443.
15 [0324]
In Step S501, the disparity image generating unit
481 inputs the common disparity images C02' of the
auxiliary images and the common disparity images AB2' of
the compatible images, which are generated in Step S499,
20 to the encoder 482.
[0325]
In Step S502, the compatible encoder 151 of the
encoder 482 encodes the multiplexed image of the
compatible images that is supplied from the image
25 converting unit 142 in accordance with the existing AVC
mode and supplies an encoded stream acquired as a result
thereof to the multiplexing unit 443 as a compatible
stream.
[0326]
30 In Step S503, the auxiliary encoder 491 encodes the
multiplexed image of the auxiliary images supplied from
116
•
the image converting unit 142 and the cornmon disparity
image of the compatible images and the cornmon disparity
image of the auxiliary images that are supplied from the
disparity image generating unit 481 in accordance with
5 the 3DV mode. The auxiliary encoder 491 supplies an
encoded stream acquired as a result of the encoding
process to the multiplexing unit 443 as an auxiliary
stream.
[0327]
10 In Step S504, the multiplexing unit 443 generates
TSs from the compatible stream that is supplied from the
compatible encoder 151 and the auxiliary stream and the
auxiliary information that are supplied from the
auxiliary encoder 491, performs a multiplexing process
15 thereof, and transmits a multiplexed stream acquired as a
result thereof. This multiplexed stream, for example, is
recorded on a BD or the like or is transmitted as a
broadcast stream. Then, the process ends.
[0328]
20 [Configuration Example of Decoding Device]
Fig. 45 is a diagram that illustrates a
configuration example of a decoding device that decodes
the multiplexed stream transmitted from the encoding
device 480 illustrated in Fig. 42.
25 [0329]
In the configuration illustrated in Fig. 45, the
same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 40.
Duplicate description will not be presented as is
30 appropriate.
[0330]
117
•
The configuration of the decoding device 500
illustrated in Fig. 45 is mainly different from the
configuration illustrated in Fig. 40 in that a decoder
501 and an image generating unit 502 are disposed instead
5 of the decoder 122 and the image generating unit 171.
The decoding device 500 generates images of two
viewpoints or multi-viewpoint images by decoding the
multiplexed stream that is transmitted from the encoding
device 480 and displays the generated images on a display
10 device not illustrated in the figure.
[0331]
More specifically, the decoder 501 of the decoding
device 500 is configured by a compatible decoder 471 and
an auxiliary decoder 511. The auxiliary decoder 511 of
15 the decoder 501 identifies the compatible stream out of
the compatible stream and the auxiliary stream that are
supplied from the separation unit 461 based on the
compatibility information that is supplied from the
separation unit 461. In addition, the auxiliary decoder
20 511 decodes the multiplexed image of auxiliary images
that is included in the auxiliary stream, the common
disparity image AB2' of the compatible images, and the
common disparity image CD2' of the auxiliary images in
accordance with a mode that corresponds to the auxiliary
25 encoder 491 illustrated in Fig. 42. The auxiliary
decoder 511 supplies the multiplexed image of the
auxiliary images and the common disparity images AB2' and
CD2' that are acquired as a result of the decoding
process to the image generating unit 502.
30 [0332]
The image generating unit 502 outputs an image in
118
accordance with an instruction supplied from a viewer and
displays the image on a display device not illustrated in
the figure. When described in more detail, the image
generating unit 502 generates images of three or more
5 viewpoints, which correspond to a display device not
illustrated in the figure, each having the resolution
that is a half of the resolution of the compatible image
or the auxiliary image by using the multiplexed image of
compatible images, the multiplexed image of the auxiliary
10 images, and the common disparity images AB2' and CD2' in
accordance with a viewer's instruction for displaying a
3D image of the multi-viewpoint mode based on the image
information supplied from the image information acquiring
unit 123, the inter-viewpoint distance information
15 supplied from the inter-viewpoint distance information
acquiring unit 124, the disparity image information
supplied from the disparity image information acquiring
unit 125, the compatibility information supplied from the
compatibility information acquiring unit 126, and the
20 like.
[0333 ]
Described in more detail, the image generating unit
502 maintains the common disparity images AB2' and CD2'
without being changed based on the information
25 representing that the disparity images of the compatible
images and the auxiliary images are common disparity
images that is included in the disparity image
information supplied from the disparity image information
acquiring unit 125.
30 [0334]
In addition, the image generating unit 502,
119
•
similarly to the image generating unit 171 illustrated in
Fig. 17, separates each auxiliary image from the
multiplexed image of the auxiliary images based on the
information that represents the multiplexing mode of the
5 auxiliary images that is included in the image
information supplied from the image information acquiring
unit 123. Furthermore, the image generating unit 502,
similarly to the image generating unit 171, separates
each compatible image from the multiplexed image of the
10 compatible images based on the information that
represents the multiplexing mode of the compatible images
that is included in the image information.
[0335]
Furthermore, the image generating unit 502,
15 similarly to the image generating unit 171, determines
the position of each viewpoint of the multi-viewpoint
images to be generated based on the inter-viewpoint
distance information and the number of viewpoints that
corresponds to a display device not illustrated in the
20 figure. Then, the image generating unit 502 generates
the image of each viewpoint of which the position is
determined by using each compatible image, each auxiliary
image, the common disparity image AB2', and the common
disparity image CD2'. Then, the image generating unit
25 502, similarly to the image generating unit 171, converts
the resolution of the generated multi-viewpoint images
into resolution that is "lithe number of viewpoints" of
the resolution of the compatible image or the auxiliary
image, synthesizes the images, and displays the
30 synthesized image on a display device not illustrated in
the figure.
120
•
[0336]
At this time, the multi-viewpoint images after the
synthesis are displayed such that the viewing angles are
different for each viewpoint, and a viewer can view a 3D
5 image without wearing glasses by seeing images of
arbitrary two viewpoints with his left and right eyes.
[0337]
In addition, the image generating unit 502,
similarly to the image generating unit 171, separates
10 from the multiplexed image of the compatible images that
is supplied from the decoder 501 into images A2 and B2 of
which the resolution is a half of the resolution of the
compatible image based on the image information supplied
from the image information acquiring unit 123 in
15 accordance with an instruction for displaying a 3D image
of the two-viewpoint mode that is supplied from a viewer.
Then, the image generating unit 502, similarly to the
image generating unit 171, alternately outputs the
separated images A2 and B2 of which the resolution is a
20 half of the resolution of the compatible image, thereby
displaying the separated images on a display device not
illustrated in the figure. At this time, the viewer can
view a 3D image by wearing glasses in which one a lefteye
shutter and a right-eye shutter is open at the time
25 of displaying the image A2, and the other is open at the
time of displaying the image B2 and viewing the images A2
and B2 that are alternately displayed on the display
device.
[0338]
30 [Description of Process of Decoding Device]
Fig. 46 is a flowchart that illustrates a decoding
121
•
process performed by the decoding device 500 illustrated
in Fig. 45. This decoding process, for example, is
started when a multiplexed stream that is transmitted
from the encoding device 480 illustrated in Fig. 42 is
5 input to the decoding device 500.
[0339]
The process of Steps S511 to S514 illustrated in
Fig. 46 is similar to that of Steps S471 to S474
illustrated in Fig. 41, and thus, the description thereof
10 will not be presented.
[0340]
In Step S515, the auxiliary decoder 511 extracts
the multiplexed image of the auxiliary images, the common
disparity image AB2' of the compatible images, and the
15 common disparity image CD2' of the auxiliary images,
which are encoded, and decodes the extracted images in
accordance with a mode that corresponds to the auxiliary
encoder 491 illustrated in Fig. 42. The auxiliary
decoder 511 supplies the multiplexed image of the
20 auxiliary images, the common disparity image AB2', and
the common disparity image CD2', which are acquired as a
result of the decoding process, to the image generating
unit 502.
[0341]
25 The process of Steps S516 to S520 is similar to
that of Steps S476 to S480 illustrated in Fig. 41, and
thus, the description thereof will not be presented.
[0342]
After the process of Step S520, in Step S521, the
30 image generating unit 502 generates an image of each
viewpoint of which the resolution is a half of the
122
•
resolution of the compatible image or the auxiliary image
by using the compatible images, the multiplexed image of
the auxiliary images, the common disparity image AB2' ,
and the common disparity image CD2' based on the position
5 of each viewpoint determined in Step S519, the image
information supplied from the image information acquiring
unit 123, the disparity image information supplied from
the disparity image information acquiring unit 125, and
the compatibility information supplied from the
10 compatibility information acquiring unit 126, and the
like.
[0343]
The process of Steps S522 to S526 is similar to
that of Steps S482 to S486 illustrated in Fig. 41, and
15 thus, the description thereof will not be presented.
[0344]
Although the encoding devices 440 and 480,
similarly to the encoding device 140, multiplex and
encode compatible images of two view points, similarly to
20 the encoding device 180 illustrated in Fig. 19, the
encoding devices 440 and 480 may encode compatible images
of two viewpoints without multiplexing them. In addition,
the encoding devices 440 and 480, similarly to the
encoding device 50 illustrated in Fig. 5, may encode a
25 compatible image of one viewpoint.
[0345]
In addition, the encoding devices 140 and 180 may
encode the disparity images of the compatible images and
the auxiliary images without multiplexing the above-
30 described disparity images. Furthermore, the encoding
device 50 may encode the disparity images of the
123
•
auxiliary images without multiplexing the above-described
disparity images.
[0346]

5 Fig. 47 is a diagram that illustrates an example of
a multiplexing pattern of an encoding target in a case
where the number of viewpoints of the compatible images
is two, and the number of viewpoints of the auxiliary
images is two.
10 [0347]
As illustrated in (1) of Fig. 47, the encoding
device 140 illustrated Fig. 14 multiplexes images A2 and
B2 that are compatible images in the spatial direction
and encodes the multiplexed image in accordance with the
15 Ave mode. In addition, the encoding device 140
multiplexes the images e2 and D2 that are auxiliary
images, the disparity images A2' and B2' that are
disparity images of the compatible images, and the
disparity images e2' and D2' that are disparity images of
20 the auxiliary images in the spatial direction and encodes
the multiplexed image in accordance with the 3DV mode
that is compliant with the Mve mode.
[0348]
Furthermore, as illustrated in (2) of Fig. 47, the
25 encoding device 140 may encode the disparity images A2'
to D2' in accordance with the 3DV mode that is compliant
with the Mve mode without multiplexing the disparity
images. In addition, as illustrated in (3) of Fig. 47,
the encoding device 480 illustrated in Fig. 42 may encode
30 the cornmon disparity image AB2' instead of the disparity
images A2' and B2' that are disparity images of the
124
•
compatible images and encode the common disparity image
eD2' instead of the disparity images e2' and D2' that are
disparity images of the auxiliary images.
[0349]
5 Furthermore, as illustrated in (4) of Fig. 47, the
encoding device 440 illustrated in Fig. 35 multiplexes
the disparity images A2' and B2' and the multiplexed
image of the disparity images of the auxiliary images in
accordance with the frame sequential mode without
10 multiplexing the disparity images A2' and B2' that are
disparity images of the compatible images in the spatial
direction and encodes the multiplexed image in accordance
with the 3DV mode that is compliant with the Mve mode.
[0350]
15 As illustrated in (5) of Fig. 47, the encoding
device 180 illustrated in Fig. 19 encodes the image A2 in
accordance with the Ave mode and encodes the image B2
with the image A2 set as a base view in accordance with
the Mve mode without multiplexing the images A2 and B2
20 that are compatible images. In addition, the encoding
device 180 encodes the images e2 and D2 that are
auxiliary images, the disparity images A2' and B2' that
are the disparity images of the compatible images, and
the disparity images e2' and D2' that are the disparity
25 images of the auxiliary images in the spatial direction
and encodes a resultant multiplexed image in accordance
with a 3DV mode that is in compliant with the Mve mode.
[0351]
In addition, as illustrated in (6) of Fig. 47, the
30 encoding device 180 may encode the disparity images A2'
to D2' in accordance with the 3DV mode that is compliant
125
•
with the MVC mode without multiplexing them. Furthermore,
as illustrated in (7) of Fig. 47, similarly to the
encoding device 480, the encoding device 180 may encode
the common disparity image AB2' instead of the disparity
5 images A2' and B2' and encode the common disparity image
C02' instead of the disparity images C2' and 02' .
[0352]
Furthermore, as illustrated in (8) of Fig. 47, the
encoding device 180, similarly to the encoding device 440,
10 may multiple the disparity images A2' and B2' and the
multiplexed image of the disparity images of the
auxiliary images in accordance with the frame sequential
mode and encode a resultant multiplexed image in
accordance with the 30V mode that is compliant with the
15 MVC mode without multiplexing the disparity images A2'
and B2' in the spatial direction.
[0353]
Fig. 48 is a diagram that illustrates features of
the effects of multiplexing in the multiplexing patterns
20 illustrated in (1) to (8) of Fig. 47.
[0354]
In the table represented in Fig. 48, items of the
effects ~Compatibility", ~Image Quality", and ~Data
Amount" are arranged, and the degrees of the effects of
25 the items for the multiplexing patterns illustrated in
(1) to (8) of Fig. 47 are represented. In the table
represented in Fig. 48, a circle represents the presence
of an effect, and a double circle represents a remarkable
effect.
30 [0355]
In a case where multiplexing is performed using the
126
•
multiplexing pattern illustrated in (1) of Fig. 47, the
multiplexing mode and the encoding mode of compatible
images are the same as those of an existing mode, whereby
the compatibility is secured. In addition, since
5 disparity images, similarly to the compatible images, are
multiplexed in the spatial direction, for example, the
disparity images can be separated using a separation unit,
which separates compatible images, prepared on the
decoding device side. Accordingly, it is assured that
10 the disparity images can be separated on the decoding
device side. Therefore, in this case, there is a
remarkable effect on the compatibility, and a double
circle is described in correspondence with the item
"Compatibility" illustrated in Fig. 48.
15 [0356]
In a case where multiplexing is performed using the
multiplexing pattern illustrated in (2) of Fig. 47, the
multiplexing mode and the encoding mode of compatible
images are the same as those of an existing mode, whereby
20 the compatibility is secured. In addition, since the
resolution of the disparity image is the same as that of
the image before being multiplexed, the accuracy is high.
As a result, in the decoding device, the accuracy of an
image of a predetermined viewpoint that is generated
25 using the disparity image is improved. Therefore, in
this case, there are effects on the compatibility and the
image quality of an image that is generated using the
disparity image, and accordingly, circles are described
in correspondence with items "Compatibility" and "Image
30 Quality" represented in Fig. 48.
[0357]
127
•
In a case where multiplexing is performed using the
multiplexing pattern illustrated in (3) of Fig. 47, the
multiplexing mode and the encoding mode of compatible
images are the same as those of an existing mode, whereby
5 the compatibility is secured. In addition, the amount of
data of the disparity images of the compatible images and
the auxiliary images each having the same resolution as
that of the image before multiplexing decreases to be the
amount of data of the disparity images of two viewpoints.
10 Accordingly, in this case, since there are effects on the
compatibility and the data amount, circles are described
in correspondence with items ~Compatibility" and ~Data
Amount" represented in Fig. 48.
[0358 ]
15 In a case where multiplexing is performed using the
multiplexing pattern illustrated in (4) of Fig. 47, the
multiplexing mode and the encoding mode of compatible
images are the same as those of an existing mode, whereby
the compatibility is secured. In addition, since the
20 disparity images are multiplexed in the time direction,
the amount of data of the disparity images at each time
is less than that of the case of (3) of Fig. 47 , whereby
the amount of data that can be transmitted increases.
Accordingly, even in a situation in which there is no
25 room for a transmission bandwidth for transmitting a
disparity image having the same resolution as the image
before multiplexing, a disparity image of which the
resolution is the same as the resolution of the image
before multiplexing can be transmitted, whereby the
30 accuracy of images of predetermined viewpoints to be
generated in the decoding device using the disparity
128
•
image is improved. Accordingly, in this case, there are
effects on the compatibility and the image quality of an
image generated using the disparity image, and there is a
remarkable effect on the data amount. Therefore, circles
5 are described in correspondence with items
~Compatibility" and ~Image Quality" illustrated in Fig.
48, and a double circle is described in correspondence
with ~Data Amount".
[0359]
10 The multiplexing process using the multiplexing
patterns illustrated in (1) to (4) of Fig. 47 is
performed, for example, when an image used for
broadcasting, the IP (internet protocol) that is
compliant with broadcasting, ATSC (Advanced Television
15 Systems Committee) 2.0 specifications and the like, nextgeneration
broadcasting that is converged broadcasting,
or internet delivery application is set as an encoding
target.
[0360]
20 In a case where multiplexing is performed using the
multiplexing pattern illustrated in (5) of Fig. 47, the
multiplexing mode and the encoding mode of compatible
images are the same as those of an existing mode, whereby
the compatibility is secured. In addition, since the
25 resolution of each disparity image is a half of the
resolution of the image, the amount of data of the
disparity images of the compatible image and the
auxiliary image decreases to be the amount of data of the
disparity images of two viewpoints. Accordingly, in this
30 case, there are effects on the compatibility and the data
amount, and circles are described in correspondence with
129
•
items "Compatibility" and "Data Amount" illustrated in
Fig. 48.
[0361]
In a case where multiplexing is performed using the
5 multiplexing pattern illustrated in (6) of Fig. 47,
similarly to the case of (2) of Fig. 47, the
compatibility is secured, and the accuracy of images of
predetermined viewpoints generated in the decoding device
using the disparity images is improved. Accordingly, in
10 this case, there are effects on the compatibility and the
image quality of an image generated using disparity
images, and circles are described in correspondence with
items "Compatibility" and "Image Quality" illustrated in
Fig. 48.
15 [0362]
In a case where multiplexing is performed using the
multiplexing pattern illustrated in (7) of Fig. 47,
similarly to the case of (3) of Fig. 47, the
compatibility is secured, and the amount of data of the
20 disparity image decreases. Accordingly, in this case,
there are effects on the compatibility and the data
amount, and circles are described in correspondence with
items "Compatibility" and "Data Amount" illustrated in
Fig. 48.
25 [0363]
In a case where multiplexing is performed using the
multiplexing pattern illustrated in (8) of Fig. 47,
similarly to the case of (4) of Fig. 47, the
compatibility is secured. In addition, similarly to the
30 case of (4) of Fig. 47, the amount of data of the
disparity images at each time is less than that of the
130
•
case of (7) of Fig. 47, and as a result, the accuracy of
images of predetermined viewpoints generated in the
decoding device using the disparity images is improved.
Accordingly, in this case, there are effects on the
5 compatibility and the image quality of an image generated
using the disparity images, and there is a remarkable
effect on the data amount. Therefore, circles are
described in correspondence with items "Compatibility"
and "Image Quality" illustrated in Fig. 48, and a double
10 circle is described in correspondence with "Data Amount".
[0364 ]
The multiplexing process using the multiplexing
patterns illustrated in (5), (7), and (8) of Fig. 47 is
performed, for example, when an image used for an
15 application for the BD, broadcasting, next-generation
broadcasting or internet delivery is set as an encoding
target. In addition, the multiplexing process using the
multiplexing pattern illustrated in (6) of Fig. 47 is
performed, for example, when an image used for an
20 application for the BD, next-generation broadcasting or
internet delivery is set as an encoding target.
[0365 ]
Fig. 49 is a diagram that illustrates an example of
the multiplexing pattern of the encoding target in a case
25 where the number of viewpoints of the compatible image is
one, and the number of viewpoints of the auxiliary image
is two.
[0366]
As illustrated in (1) of Fig. 49, the encoding
30 device 50 illustrated in Fig. 5 encodes an image A1 that
is a compatible image in accordance with the AVC mode.
131
•
In addition, the encoding device 50 multiplexes images B1
and C1 that are auxiliary images and disparity images B1'
and C1' that are the disparity images of the auxiliary
images in the time direction. Then, the encoding device
5 50 encodes a disparity image A1' that is the disparity
image of the compatible image, a multiplexed image of the
auxiliary images, and a multiplexed image of the
disparity images of the auxiliary images in accordance
with the 3DV mode that is compliant with the MVC mode.
10 [0367]
The encoding device 50, as illustrated in (2) of
Fig. 49, may encode the disparity images A1' to C1' in
accordance with the 3DV mode that is compliant with the
MVC mode without multiplexing the disparity images. In
15 addition, as illustrated in (3) of Fig. 49, the encoding
device 50, similarly to the encoding device 480, may
encode a cornmon disparity image BC1' of images Band C
that are auxiliary images instead of the multiplexed
image of the disparity images of the auxiliary images.
20 [0368]
In addition, as illustrated in (4) of Fig. 49, the
encoding device 50, similarly to the encoding device 440,
may multiplex the disparity images A1' to C1' in
accordance with the frame sequential mode and encode a
25 resultant multiplexed image in accordance with the 3DV
mode that is compliant with the MVC mode without
multiplexing the disparity images B1' and C1' in the
spatial direction.
[0369]
30 The effects of the multiplexing processes using the
multiplexing patterns illustrated in (1) to (4) of Fig.
132
•
49 and the encoding targets at the time of performing the
multiplexing processes using the multiplexing patterns
are the same as the effects and the encoding targets of
the multiplexing process using the multiplexing patterns
5 illustrated in (5) to (8) of Fig. 47. However, in the
multiplexing process using the multiplexing pattern
illustrated in (1) of Fig. 49, since the resolution of
the disparity image of the compatible image is the same
as the resolution of the compatible image, a decrease in
10 the amount of data of the disparity image as an effect of
this multiplexing process is an effect only for the
disparity image of the auxiliary image.
[0370]
Fig. 50 is a diagram that illustrates an example of
15 the multiplexing pattern of an encoding target in a case
where the number of viewpoints of the compatible image is
two, and the number of viewpoints of the auxiliary image
is zero.
[0371]
20 In a case where the number of viewpoints of the
compatible images is two, and the number of viewpoints of
the auxiliary image is zero, as illustrated in (1) of Fig.
50, similarly to the case of the encoding device 140
illustrated in Fig. 14, the images A2 and B2 that are
25 compatible images are multiplexed in the spatial
direction and are encoded in accordance with the Ave mode.
In addition, the disparity images A2' and B2' that are
the disparity images of the compatible images are
multiplexed in the spatial direction and are encoded in
30 accordance with the 3DV mode that is compliant with the
Ave mode.
133
•
[0372]
As illustrated in (2) of Fig. 50, the disparity
images A2' and B2' may be encoded in accordance with the
3DV mode that is compliant with the MVC mode without
5 being multiplexed. In addition, as illustrated in (3) of
Fig. 50, similarly to the case of the encoding device 4BO,
the common disparity image AB2' may be encoded instead of
the disparity images A2' and B2' .
[0373]
10 Furthermore, as illustrated in (4) of Fig. 50,
similarly to the case of the encoding device 440, the
disparity images A2' and B2' may be multiplexed in the
frame sequential mode and be encoded without being
multiplexed in the time direction.
15 [0374]
In addition, as illustrated in (5) of Fig. 50,
similarly to the case of the encoding device lBO, it may
be configured such that the image A2 is encoded in
accordance with the AVC mode, and the image B2 is encoded
20 with the MVC mode in which the image A2 is set as a base
view without multiplexing the images A2 and B2 that are
compatible images.
[0375]
In this case, as illustrated in (6) of Fig. 50,
25 similarly to the case of (2) of Fig. 50, the disparity
images may be encoded without being multiplexed, or, as
illustrated in (7) of Fig. 50, similarly to the case of
(3) of Fig. 50, the common disparity image AB2' may be
encoded. In addition, as illustrated in (B) of Fig. 50,
30 similarly to the case of (4) illustrated in Fig. 50, the
disparity images of the compatible images may be
134
•
multiplexed in accordance with the frame sequential mode
and be encoded.
[0376]
In addition, the effects of the multiplexing
5 processes using the multiplexing patterns illustrated in
(1) to (8) of Fig. 49 and the encoding targets at the
time of performing the multiplexing processes using the
multiplexing patterns are the same as the effects and the
encoding targets of the multiplexing processes using the
10 multiplexing patterns illustrated in (1) to (8) of Fig.
47.
[0377]
In the description presented above, although the
resolution of the disparity images that are multiplexed
15 in accordance with the frame sequential mode has been
described to be the same as the resolution of the image
before the multiplexing process, the resolution may be
lower than the resolution of the image before the
multiplexing process. In addition, the auxiliary images,
20 similarly to the disparity images, may be also
multiplexed in accordance with the frame sequential mode.
[0378]
Furthermore, in the description presented above,
although the information representing the multiplexing
25 mode of the images and the information representing the
multiplexing mode of the disparity images have been
described to be transmitted in the encoding device,
information used for identifying the multiplexing
patterns illustrated in Figs. 47, 49, and 50 may be
30 transmitted.
[0379]
135
•
In addition, the encoding device may transmit a
flag that is used for identifying an application that
corresponds to an image that is an encoding target.
[0380]
5
[Configuration Example of Encoding Device According to
Seventh Embodiment]
Fig. 51 is a block diagram that illustrates a
configuration example of an encoding device according to
10 a seventh embodiment of the present technology.
[0381]
In the configuration illustrated in Fig. 51, the
same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 35.
15 Duplicate description will not be presented as is
appropriate.
[0382]
The configuration of the encoding device 520
illustrated in Fig. 51 is mainly different from the
20 configuration illustrated in Fig. 35 in that an encoder
523 and a transmission unit 524 are disposed instead of
the encoder 145 and the multiplexing unit 443, and a
multiplexing unit 521 and a multiplexing information
generating unit 522 are newly arranged.
25 [0383]
The encoding device 520 multiplexes a multiplexed
image of compatible images, a multiplexed image of
auxiliary images, a multiplexed image of disparity images
of the compatible images, and a multiplexed image of
30 disparity images of the auxiliary images in accordance
with the frame sequential mode and encodes a resultant
136
•
multiplexed image.
[0384]
More specifically, the multiplexing unit 521 of the
encoding device 520 multiplexes a multiplexed image of a
5 compatible image and an auxiliary image that is acquired
as a result of the multiplexing process performed by an
image converting unit 142 and a multiplexed image of the
disparity images of the compatible image and the
auxiliary image that is acquired as a result of the
10 multiplexing process performed by a disparity image
generating unit 143 in accordance with the frame
sequential mode.
[0385]
Then, the multiplexing unit 521 supplies a
15 multiplexed image in which a multiplexed image of
compatible images, a multiplexed image of the disparity
images of the compatible images, a multiplexed image of
auxiliary images, and a multiplexed image of the
disparity images of the auxiliary images, which can be
20 acquired as a result of the multiplexing process, are
sequentially present to the encoder 523 as a timedirection
multiplexed image within one frame time.
[0386]
In addition, the multiplexing unit 521 supplies
25 information representing that multiplexed images of the
compatible images and the auxiliary images and a
multiplexed image of the disparity images of the
compatible image and the auxiliary image are multiplexed
in accordance with the frame sequential mode and
30 representing the arrangement order of images in the timemultiplexed
image to the multiplexing information
137
•
generating unit 522 and the encoder 523.
[0387]
The multiplexing information generating unit 522,
based on the information that is supplied from the
5 multiplexing unit 521, generates the information and the
like as whole multiplexing information relating to the
multiplexing of the compatible images and the auxiliary
images and the disparity images of the compatible images
and the auxiliary images and supplies the generated whole
10 multiplexing information to the transmission unit 524.
[0388]
The encoder 523 is configured by a compatible
encoder 531 and an auxiliary encoder 532. The encoder
523 designates the multiplexed image of the compatible
15 images that is included in the time-multiplexed image
supplied from the multiplexing unit 521 based on the
information supplied from the multiplexing unit 521 and
supplies the multiplexed image of the compatible images
to the compatible encoder 531. In addition, the encoder
20 523 supplies the multiplexed image of the auxiliary
images and the multiplexed images of the disparity images
of the compatible images and the auxiliary images that
are included in the time-multiplexed image to the
auxiliary encoder 532 based on the information that is
25 supplied from the multiplexing unit 521.
[0389]
The compatible encoder 531 of the encoder 523
encodes the multiplexed image of the compatible images
that are included in the time-multiplexed image in
30 accordance with the existing Ave mode. The auxiliary
encoder 532 encodes the multiplexed image of the
138
•
auxiliary images and the multiplexed images of the
disparity images of the compatible images and the
auxiliary images that are included in the timemultiplexed
image in accordance with the 3DV mode. At
5 this time, the multiplexed image of the auxiliary images
are encoded by referring to the multiplexed image of the
compatible images, and the multiplexed image of the
disparity images of the auxiliary images is encoded by
referring to the multiplexed image of the disparity
10 images of the compatible images.
[0390]
The encoder 523 supplies a bit stream that is
formed from encoded data of the time-multiplexed image
that is acquired as a result of the encoding process
15 performed by the compatible encoder 531 or the auxiliary
encoder 532 to the transmission unit 524.
[0391]
The transmission unit 524 generates a TS using the
bit stream supplied from the encoder 523, image
20 information supplied from the image information
generating unit 54, compatibility information supplied
from the compatibility information generating unit 55,
inter-viewpoint distance information supplied from the
inter-viewpoint distance information generating unit 144,
25 disparity image information supplied from the disparity
image information generating unit 57, the whole
multiplexing information supplied from the multiplexing
information generating unit 522, and the like. The
multiplexing unit 443 transmits the generated TS.
30 [0392]
[Description of Process of Encoding Device]
139
•
Figs. 52 and 53 are a flowchart that illustrates an
encoding process that is performed by the encoding device
520 illustrated in Fig. 51. This encoding process, for
example, is started when images A2 to D2 are output from
5 the imaging units 141A to 141D.
[0393]
The process of Steps S531 to S537 illustrated in
Fig. 52 is the same as the process of Steps S451 to S457
illustrated in Fig. 38, and thus the description thereof
10 will not be presented.
[0394]
After the process of Step S537, in Step S538, the
image converting unit 142 inputs a multiplexed image of
the compatible images and a multiplexed image of the
15 auxiliary images to the multiplexing unit 521, and the
process proceeds to Step S539 illustrated in Fig. 53.
[0395]
The process of Steps S539 to S542 illustrated in
Fig. 53 is the same as the process of Steps S459 to S462
20 illustrated in Fig. 39, and thus the description thereof
will not be presented.
[0396]
After the process of Step S543, in Step S544, the
multiplexing unit 521 multiplexes the multiplexed images
25 of the compatible images and the auxiliary images that
are acquired as a result of the multiplexing process
performed by the image converting unit 142 and the
multiplexed images of the disparity images of the
compatible images and the auxiliary images that are
30 acquired as a result of the multiplexing process
performed by the disparity image generating unit 143 in
140
•
accordance with the frame sequential mode. The
multiplexing unit 521 supplies a multiplexed image that
is acquired as a result of the multiplexing process to
the encoder 523 as a time-direction multiplexed image.
5 [0397]
In addition, the multiplexing unit 521 supplies
information representing that multiplexed images of the
compatible images and the auxiliary images and
multiplexed images of the disparity images are
10 multiplexed in accordance with the frame sequential mode
and representing the arrangement order of images in the
time-multiplexed image to the multiplexing information
generating unit 522 and the encoder 523.
[0398]
15 In Step S545, the multiplexing information
generating unit 522, based on the information that is
supplied from the multiplexing unit 521, generates the
information and the like as whole multiplexing
information and supplies the generated whole multiplexing
20 information to the transmission unit 524.
[0399]
In Step S546, the compatible encoder 531 encodes
the multiplexed image of the compatible images that is
included in the time-multiplexed image input based on the
25 information supplied from the multiplexing unit 521 by
the encoder 523 in accordance with the existing Ave mode.
[0400]
In Step S547, the auxiliary encoder 532 encodes the
multiplexed image of the auxiliary images, the
30 multiplexed image of the disparity images of the
compatible images, and the multiplexed image of the
141
•
disparity images of the auxiliary images that are
included in the time-multiplexed image that is input
based on the information supplied from the multiplexing
unit 521 by the encoder 523 in accordance with the 3DV
5 mode. The encoder 523 supplies a bit stream that is
formed from encoded data of the time-multiplexed image
that is acquired as a result of the encoding process of
Steps S546 and S547 to the transmission unit 524.
[0401]
10 In Step S548, the transmission unit 524 generates a
TS from the bit stream supplied from the encoder 523, the
auxiliary information, and the whole multiplexing
information supplied from the multiplexing information
generating unit 522 and transmits the generated TS. This
15 TS, for example, is recorded on a BD or the like or is
transmitted as a broadcast stream. Then, the process
ends.
[0402]
As above, the encoding device 520 generates one bit
20 stream from encoded data of the multiplexed image of the
compatible images, the multiplexed image of the auxiliary
images, the multiplexed image of the disparity images of
the compatible images, and the multiplexed image of the
disparity images of the auxiliary images. Accordingly, a
25 decoding device that includes a decoder that can decode
only one bit stream can decode the bit stream that is
generated by the encoding device 520.
[0403]
In the description presented above, although the
30 disparity images of the compatible images, the auxiliary
images and the disparity images of the auxiliary images
142
•
have been described to be encoded in accordance with the
3DV mode that is compliant with the encoding mode of the
compatible images, the images may be encoded in
accordance with MPEG2 (Moving Picture Experts Group phase
5 2) or the like that is not compliant with an encoding
mode of compatible images.
[0404]
[Configuration Example of Decoding Device]
Fig. 54 is a diagram that illustrates a
10 configuration example of a decoding device that decodes
the TS transmitted from the encoding device 520
illustrated in Fig. 51.
[0405]
In the configuration illustrated in Fig. 54, the
15 same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 40.
Duplicate description will not be presented as is
appropriate.
[0406]
20 The configuration of the decoding device 540
illustrated in Fig. 54 is mainly different from the
configuration illustrated in Fig. 40 in that a reception
unit 541, a decoder 542, and an image generating unit 544
are disposed instead of the separation unit 121, the
25 decoder 122, and the image generating unit 171, and a
multiplexing information acquiring unit 543 is newly
arranged. The decoding device 540 decodes a bit stream
of a time-multiplexed image that is included in a TS
transmitted from the encoding device 520, generates
30 images of two viewpoints or multi-viewpoint images, and
displays the generated images on a display device not
143
•
illustrated in the figure.
[0407]
More specifically, the reception unit 541 of the
decoding device 540 receives a TS that is transmitted
5 from the encoding device 520. The reception unit 541
extracts a bit stream of a time-multiplexed image that is
included in the TS and supplies the bit stream to the
decoder 542. In addition, the reception unit 541
extracts auxiliary information that is included in the TS
10 and supplies image information that is included in the
auxiliary information to the image information acquiring
unit 123 and supplies viewpoint distance information to
an inter-viewpoint distance information acquiring unit
124. Furthermore, the reception unit 541 supplies
15 disparity image information included in the auxiliary
information to a disparity image information acquiring
unit 125 and supplies compatibility information to a
compatibility information acquiring unit 126 and the
decoder 501. In addition, the reception unit 541
20 extracts whole multiplexing information that is included
in the TS and supplies the extracted whole multiplexing
information to the multiplexing information acquiring
unit 543.
[0408]
25 The decoder 542 is configured by a compatible
decoder 551 and an auxiliary decoder 552. The compatible
decoder 551 of the decoder 542 extracts encoded data of
the multiplexed image of the compatible images from the
bit stream that is supplied from the reception unit 541
30 based on the compatibility information that is supplied
from the separation unit 461. The compatible decoder 551
144
•
decodes the extracted encoded data of the multiplexed
image of the compatible images in accordance with a mode
that corresponds to the Ave mode and supplies the decoded
data to the image generating unit 544.
5 [0409]
The auxiliary decoder 552 extracts encoded data of
the multiplexed image of the auxiliary images, the
multiplexed image of the disparity images of the
compatible images, and the multiplexed image of the
10 disparity images of the auxiliary images from the bit
stream that is supplied from the reception unit 541 based
on the compatibility information supplied from the
separation unit 461. The auxiliary decoder 552 decodes
encoded data of the multiplexed image of the auxiliary
15 images, the multiplexed image of the disparity images of
compatible images, and the multiplexed image of the
disparity images of the auxiliary images, that are
extracted, in accordance'with a mode that corresponds to
the decoding mode of the auxiliary encoder 532
20 illustrated in Fig. 51 and supplies the decoded data to
the image generating unit 544.
[0410]
The multiplexing information acquiring unit 543
acquires the whole multiplexing information that is
25 supplied from the reception unit 541 and supplies the
acquired whole multiplexing information to the image
generating unit 544.
[0411]
The image generating unit 544 outputs an image in
30 accordance with a viewer's display instruction, thereby
displaying the image on a display device not illustrated
145
•
in the figure. More specifically, the image generating
unit 544, in accordance with a viewer's instruction for
displaying a 3D image of the multi-viewpoint system,
generates images of three or more viewpoints, which
5 correspond to a display device not illustrated in the
figure, each having the resolution that is a half of the
resolution of the compatible image or the auxiliary image
by using the multiplexed image of the compatible images,
the multiplexed image of the auxiliary images, the
10 multiplexed image of the disparity images of the
compatible images, and the multiplexed image of the
disparity images of the auxiliary images based on the
image information supplied from the image information
acquiring unit 123, the inter-viewpoint distance
15 information supplied from the inter-viewpoint distance
information acquiring unit 124, the disparity image
information supplied from the disparity image information
acquiring unit 125, the compatibility information
supplied from the compatibility information acquiring
20 unit 126, the whole multiplexing information supplied
from the multiplexing information acquiring unit 543,and
the like.
[0412]
Described in more detail, the image generating unit
25 544 identifies the multiplexed image of the auxiliary
image, the multiplexed image of the disparity image of
the compatible image, and the multiplexed image of the
disparity image of the auxiliary image, which are
supplied from the auxiliary decoder 552, based on the
30 whole multiplexing information supplied from the
multiplexing information acquiring unit 543. In addition,
146
•
the image generating unit 544, similarly to the image
generating unit 171 illustrated in Fig. 17, separates the
disparity image of each auxiliary image from the
multiplexed image of the disparity images of the
5 auxiliary images based on the information that represents
the multiplexing mode of the disparity image of the
auxiliary image that is included in the disparity image
information supplied from the disparity image information
acquiring unit 125. Furthermore, the image generating
10 unit 544, similarly to the image generating unit 171,
separates the disparity image of each compatible image
from the multiplexed image of the disparity images of the
compatible images based on the information that
represents the multiplexing mode of the disparity images
15 of the compatible images that is included in the
disparity image information.
[0413]
Furthermore, the image generating unit 544,
similarly to the image generating unit 171, separates
20 each auxiliary image from the multiplexed image of the
auxiliary images based on the information that represents
the multiplexing mode of the auxiliary images that is
included in the image information supplied from the image
information acquiring unit 123. In addition, the image
25 generating unit 544, similarly to the image generating
unit 171, separates each compatible image from the
multiplexed image of the compatible images based on the
information that represents the multiplexing mode of the
compatible images that is included in the image
30 information.
[0414]
147
•
In addition, the image generating unit 544,
similarly to the image generating unit 171, determines
the position of each viewpoint of the multi-viewpoint
images to be generated based on the inter-viewpoint
5 distance information and the number of viewpoints that
corresponds to a display device not illustrated in the
figure. Then, the image generating unit 544, similarly
to the image generating unit 171, generates an image of
each viewpoint of which the position is determined by
10 using each compatible image, each auxiliary image, the
disparity image of each compatible image, and the
disparity image of each auxiliary image. Then, the image
generating unit 544 converts the resolution of the
generated multi-viewpoint images to the resolution that
15 is "lithe number of viewpoints" of the resolution of the
compatible image or the auxiliary image, synthesizes the
converted images, and displays the synthesized image on a
display device not illustrated in the figure.
[0415]
20 At this time, the multi-viewpoint images after the
synthesis are displayed such that the viewing angles are
different for each viewpoint, and a viewer can view a 3D
image without wearing glasses by seeing images of
arbitrary two viewpoints with his left and right eyes.
25 [0416]
In addition, the image generating unit 544,
similarly to the image generating unit 171, separates the
multiplexed image of the compatible images that is
supplied from the decoder 542 into images A2 and B2 of
30 which the resolution is a half of the resolution of the
compatible image based on the image information supplied
148
•
from the image information acquiring unit 123 in
accordance with an instruction for displaying a 3D image
of the two-viewpoint mode that is supplied from a viewer.
Then, the image generating unit 544, similarly to the
5 image generating unit 171, alternately outputs the
separated images A2 and B2 of which the resolution is a
half of the resolution of the compatible image, thereby
displaying the separated images on a display device not
illustrated in the figure. At this time, the viewer can
10 view a 3D image by wearing glasses in which one a lefteye
shutter and a right-eye shutter is open at the time
of displaying the image A2, and the other is open at the
time of displaying the image B2 and viewing the images A2
and B2 that are alternately displayed on the display
15 device.
[0417]
[Description of Process of Decoding Device]
Fig. 55 is a flowchart that illustrates a decoding
process performed by the decoding device 540 illustrated
20 in Fig. 54. This decoding process, for example, is
started when a TS that is transmitted from the encoding
device 520 illustrated in Fig. 51 is input to the
decoding device 540.
[0418]
25 In Step S551 illustrated in Fig. 55, the reception
unit 541 of the decoding device 540 receives a TS that is
transmitted from the encoding device 520. The reception
unit 541 extracts a bit stream included in the TS and
supplies the extracted bit stream to the decoder 542. In
30 addition, the reception unit 541 extracts auxiliary
information that is included in the TS, supplies image
149
•
information included in the auxiliary information to the
image information acquiring unit 123, and supplies
viewpoint distance information to the inter-viewpoint
distance information acquiring unit 124. Furthermore,
5 the reception unit 541 supplies disparity image
information included in the auxiliary information to the
disparity image information acquiring unit 125 and
supplies compatibility information to the compatibility
information acquiring unit 126. In addition, the
10 reception unit 541 extracts whole multiplexing
information that is included in the TS and supplies the
extracted whole multiplexing information to the
multiplexing information acquiring unit 543.
[0419]
15 In Step S552, similarly to the process of Step S474
illustrated in Fig. 41, the image generating unit 544
determines whether or not an instruction for displaying a
3D image of the two-viewpoint mode has been made by a
viewer. In a case where it is determined that the
20 instructiori for displaying a 3D image of the twoviewpoint
mode has not been made by the viewer in Step
S552, in other words, in a case where an instruction for
displaying a 3D image of the multi-viewpoint mode has
been made, the process proceeds to Step S553.
25 [0420]
In Step S553, the compatible decoder 551 of the
decoder 542 extracts encoded data of the multiplexed
image of the compatible images from the bit stream based
on the compatibility information that is supplied from
30 the reception unit 541 and decodes the extracted encoded
data in a mode that corresponds to the Ave mode. Then,
150
•
the compatible decoder 551 supplies a multiplexed image
of the compatible images that is acquired as a result of
the decoding process to the image generating unit 544.
[0421]
5 In Step S554, the auxiliary decoder 552 extracts
encoded data of the multiplexed image of the auxiliary
images, the multiplexed image of the disparity images of
the compatible images, and the multiplexed image of the
disparity images of the auxiliary images from the bit
10 stream based on the compatibility information that is
supplied from the reception unit 541 and decodes the
extracted encoded data in a mode that corresponds to the
auxiliary encoder 532 illustrated in Fig. 51. The
auxiliary decoder 552 supplies the multiplexed image of
15 the auxiliary images, the multiplexed image of the
disparity images of the compatible images, and the
multiplexed image of the disparity images of the
auxiliary images, which are acquired as a result of the
decoding process, to the image generating unit 544, and
20 the process proceeds to Step S555.
[0422]
The process of Steps S555 to S558 is the same as
that of Steps S476 to S479 illustrated in Fig. 41, and
thus description thereof will not be presented.
25 [0423]
After the process of Step S558, in Step S559, the
multiplexing information acquiring unit 543 acquires the
whole multiplexing information that is supplied from the
reception unit 541 and inputs the acquired whole
30 multiplexing information to the image generating unit 544.
[0424]
151
•
In Step S560, the image generating unit 544,
similarly to the process of Step S480 illustrated in Fig.
41, determines the position of each viewpoint of multiviewpoint
images to be generated based on the inter-
5 viewpoint distance information supplied from the interviewpoint
distance information acquiring unit 124 and the
number of viewpoints that corresponds to a display device
not illustrated in the figure.
[0425]
10 In Step S561, the image generating unit 544
generates an image of each viewpoint of which the
resolution is a half of the resolution of the compatible
image or the auxiliary image by using the multiplexed
image of the compatible images, the multiplexed image of
15 the auxiliary images, the multiplexed image of the
disparity images of the compatible images, and the
multiplexed image of the disparity images of the
auxiliary images based on the position of each viewpoint
that is determined in Step S560, the image information
20 supplied from the image information acquiring unit 123,
the disparity image information supplied from the
disparity image information acquiring unit 125, the
compatibility information supplied from the compatibility
information acquiring unit 126, the whole multiplexing
25 information supplied from the multiplexing information
acquiring unit 543, and the like. Then, the process
proceeds to Step S562.
[0426]
The process of Steps S562 and S563 is the same as
30 that of Steps S482 and S483 illustrated in Fig. 41, and
thus description thereof will not be presented.
152
•
[0427]
In a case where it is determined that the
instruction for displaying a 3D image of the twoviewpoint
mode has been made by a viewer in Step S552, in
5 Step S564, the compatible decoder 551 extracts encoded
data of the multiplexed image of the compatible images
from the bit stream based on the compatibility
information that is supplied from the reception unit 541
and decodes the extracted encoded data in accordance with
10 a mode that corresponds to the Ave mode. Then, the
compatible decoder 551 supplies a multiplexed image of
the compatible images that is acquired as a result of the
decoding process to the image generating unit 544.
[0428]
15 The process of Steps S565 to S567 is the same as
that of Steps S484 to S486 illustrated in Fig. 41, and
thus description thereof will not be presented.
[0429]
In addition, in a decoding device that can decode
20 only a compatible image that has compatibility with the
decoding device 540, encoded data other than the encoded
data of the compatible images that can be processed is
ignored, and the process of Steps S564, S566, and S567 is
performed. However, in such a case, in the prooess of
25 Step S566, a multiplexed image of the compatible images
is separated in accordance with a multiplexing mode that
is determined in advance.
[0430]
As above, the decoding device 540 can decode one
30 bit stream that is generated from the encoded data of the
multiplexed image of the compatible images, the
153
•
multiplexed image of the auxiliary images, the
multiplexed image of the disparity images of the
compatible images, and the multiplexed image of the
disparity images of the auxiliary images.
5 [0431]

[Configuration Example of Bit Stream]
Fig. 56 is a diagram that illustrates a
configuration example of a bit stream (encoded stream) in
10 units of access units that is generated in accordance
with this embodiment.
[0432]
In the example illustrated in Fig. 56, the
compatible images are an L image and an R image of 1920 x
15 1080 pixels, and the auxiliary image is an 0 image of
1920 x 1080 pixels. In addition, the size of the L
disparity image, the R disparity image, and the 0
disparity image is 960 x 1080 pixels. Furthermore, the L
image is encoded in accordance with the AVC mode, the R
20 image is encoded in accordance with the MVC mode, and the
o image, the L disparity image, the R disparity image,
and the 0 disparity image are encoded in according with
the 3DV mode. In addition, the view IDs of the L image,
the R image, the 0 image, the L disparity image, the R
25 disparity image, and the 0 disparity image are 0, 1, 2, 3,
4, and 5, respectively. Here, the view ID is an ID that
is unique to the image of each viewpoint and the
disparity image.
[0433]
30 As illustrated in Fig. 56, in the bit stream in
units of access units, for example, an access unit
154
•
delimiter (AUD), an SPS (sequence parameter set), a
Subset SPS (Subset SPS1) according to the Mve mode for an
image, a Subset SPS (Subset SPS2) according to the 3DV
mode for a disparity image, a PPS (picture parameter set),
5 SEI according to the Ave mode, SEI according to the Mve
mode, SEI according to the 3DV mode, and an NAL (network
abstraction layer) unit of encoded data are arranged in
order from the head.
[0434]
10 The NAL unit of the access unit delimiter is an NAL
unit that represents a boundary of an access unit. The
NAL unit of the SPS is an NAL unit of the SPS that
includes a profile_ide (100 in the example illustrated in
Fig. 56) that represents a profile of the L image out of
15 profiles defined in accordance with the Ave mode. The
NAL unit of the Subset SPS for an'image is an NAL unit of
the Subset SPS including profile_ide (128 in the example
illustrated in Fig. 56) that represents profiles of the R
image and the a image out of profiles defined in
20 accordance with the Mve mode. The NAL unit of the Subset
SPS for a disparity image is an NAL unit of the Subset
SPS that includes profile_ide (138 in the example of Fig.
56) that represents profiles of the L disparity image,
the R disparity image, and the a disparity image out of
25 profiles defined as profiles for a disparity image in
accordance with the 3DV mode.
[0435]
The NAL unit of the SEI according to the Ave mode
is an NAL unit of the SEI of the L image. The NAL unit
30 of the SEI according to the Mve mode is an NAL unit of
the SEI of the L image or the R image. The NAL unit of
155
•
the SEI of the 30V mode is an NAL unit of the SEI of the
L image, the R image, the 0 image, the L disparity image,
the R disparity image, or the 0 disparity image.
[0436]
5 As NAL units of the encoded data, NAL units of
encoded data of an L image, a delimiter (MVC DO), encoded
data of an R image, a delimiter (30V 00), encoded data of
an 0 image, a delimiter (30V 00), encoded data of an L
disparity image, a delimiter (30V 00), encoded data of an
10 R disparity image, a delimiter (30V DO), encoded data of
an 0 disparity image are arranged in order from the head.
[0437]
In the NAL unit of the encoded data of the L image,
an NAL header that includes 1 or 5 as the type of the NAL
15 unit representing a picture according to the AVC mode is
added. In addition, the NAL unit of the delimiter (MVC
DO) is an NAL unit that represents a head of the encoded
data of the MVC mode. In the NAL unit of the encoded
data of the R image, an NAL header that includes 20
20 representing encoded data according to the MVC mode as
the type of the NAL unit is added. In addition, the NAL
unit of the delimiter (30V DD) is an NAL unit that
represents a head of the encoded data of the 30V mode.
Furthermore, in the NAL units of the encoded data of the
25 0 image, the L disparity image, the R disparity image,
and the 0 disparity image, an NAL header that includes 21
representing encoded data according to the 30V mode as
the type of the NAL unit is added.
[0438]
30 [Example of Description of Subset SPS According to 3DV
Mode for Disparity Image]
156
•
Fig. 57 is a diagram that illustrates an example of
the description of a Subset SPS according to the 3DV mode
for a disparity image illustrated in Fig. 56.
[0439]
5 As illustrated in Fig. 57, in a Subset SPS
according to the 3DV mode for a disparity image, SPS
(seq_parameter_set_data) that includes profile_ide (138
in the example illustrated in Fig. 57) representing the
profiles of the L disparity image, the R disparity image,
10 and the 0 disparity image and information for each
profile_ide that is defined in accordance with the 3DV
mode are described.
[0440]
More specifically, in the Subset SPS of the 3DV
15 mode for a disparity image, as information when
profile_ide is 138, extension information
(seq_parameter_set_depth_extension) for a disparity image,
a VUI information flag
(depth_vui_parameters_present_flag) for a disparity image
20 that represents whether or not VUI extension information
for a disparity image is included and the like are
described. In addition, in a case where the VUI
information flag for a disparity image represents that
the VUI extension information for a disparity image is
25 included, VUI extension information
(depth_vui_parameters__extension) for a disparity image
is also described.
[0441]
In addition, in a case where the Subset SPS of the
30 3DV mode for a disparity image is referred to in a
decoding process, in other words, in a case where the L
157
•
disparity image, the R disparity image, and the 0
disparity image are decoded, similarly to a case where an
lOR picture is decoded, the reference image is reset.
[0442]
5 Fig. 58 is a diagram that illustrates an example of
the description of extension information for a disparity
image that is illustrated in Fig. 57.
[0443]
As illustrated in Fig. 58, the extension
10 information for disparity information, similarly to the
extension information (seq_parameter_set_mvc_extension)
within the Subset SPS of the MVC mode, is formed by
described information and a view 10 (ref_view_id) of an
image corresponding to each disparity image.
15 [0444]
In Fig. 58, similarly to the extension information
within the Subset SPS of the MVC mode, the described
information is expanded and is included in the extension
information for a disparity image. Accordingly, the view
20 10 (view_id) of each disparity view and the view 10 of an
image that corresponds to each disparity image can be
described together for each disparity image. In other
words, information (num_views_minusl) that represents the
number of disparity images is described, and a
25 description for reading out the view 10 of a disparity
image and the view 10 of an image that corresponds to the
disparity image can be made as many times as the number.
[0445]
In contrast to this, similarly to the extension
30 information within the Subset SPS of the MVC mode, in a
case where the described information is not expanded and
158
is included in the extension information for a disparity
image, in addition to the extension information for a
disparity image, information representing the number of
disparity images is described, and a description for
5 reading out a view ID of an image that corresponds to the
disparity image needs to be made as many times as the
number. As a result, the description of the information
that represents the number of disparity images and the
description for reading out the information as many times
10 as the number overlap each other.
[0446]
Accordingly, as illustrated in Fig. 58, in a case
where described information is expanded and is included
in the extension information for a disparity image,
15 similarly to the extension information within the Subset
SPS of the MVC mode, the amount of data of the extension
information for a disparity image can be less than that
of a case where the information is not expanded and is
included in the extension information for a disparity
20 image.
[0447]
Fig. 59 is a diagram that illustrates an example of
the description of VUI extension information for a
disparity image illustrated in Fig. 57.
25 [0448]
As illustrated in Fig. 59, the VUI extension
information for a disparity image is described in the
same manner as the VUI extension information
(mvc_vui_parameters__extension) of the MVC mode except
30 for the following points. In other words, in the VUI
information for a disparity image, for each pixel of a
159
disparity image, a position type flag
(depth_loc_info_present_flag) that represents whether or
not a position type representing the type of a position
of an image that corresponds to the disparity image is
5 included and a before-conversion size information flag
(video_src_info_present_flag) representing whether or not
before-conversion size information representing the size
of the disparity image before resolution conversion is
included are included. In addition, in a case where the
10 position type flag represents that the position type is
included, the position type is also included in the VUI
information for a disparity image, and, in a case where
the before-conversion size information flag represents
that the before-conversion size information is included,
15 the before-conversion size information is also included
in the VUI information for a disparity image.
[0449]
The position type is formed by a top field position
type (depth_sarnple_loc_type_top_field) and a bottom field
20 position type (depth_sample_loc_type_bottom_field). The
top field and bottom field position types are described
similarly to the top field position type
(chroma_sample_loc_type_top_field) and the bottom field
position type (chroma_sample_loc_type_bottom_field) that
25 are included in the VUI extension information of the MVC
mode.
[0450 ]
In addition, the before-conversion size information
is configured by information (pic_width_in_rnbs_minus1)
30 that represents the number of macro blocks of the
disparity image before resolution conversion in the
160
horizontal direction, information
(pic_height_in_mbs_minus1) that represents the number of
macro blocks in the vertical direction, an aspect ratio
flag (aspect_ratio_info_present_flag) that represents
5 whether or not aspect information representing an aspect
ratio is included, and the like. In a case where the
aspect ratio flag represents that the aspect information
is included, the aspect information is also included in
the before-conversion size information.
10 [0451]
The aspect information is formed by an aspect ratio
ID (aspect_ratio_idc) that is an ID unique to an aspect
ratio and the like. This aspect ratio IDs are assigned
not only to aspect ratios that are defined in advance but
15 also to all the aspect ratios that are not defined. In a
case where an aspect ratio ID that is included in the
aspect information is an aspect ratio ID (Extended_SAR)
that is given to all the aspect ratios that are not
defined, values (sar_width and sar_height) of the aspect
20 ratio of the disparity image before resolution conversion
in the horizontal and vertical directions are also
included in the aspect information.
[0452]
[Example of Description of NAL Header of Encoded Data of
25 3DV Mode]
Fig. 60 is a diagram that illustrates an example of
the description of an NAL header of an NAL unit of
encoded data according to the 3DV mode in which 21 is
included as the type of the NAL unit.
30 [0453]
As illustrated in Fig. 60, in the NAL header of the
161
encoded data according to the 30V mode, as information at
a time when the type of the NAL unit is 21, a disparity
image header extension information flag
(depth_extension_flag) representing whether or not
5 disparity image header extension information is included
and the like are described. Here, the disparity image
header extension information is also described in the NAL
header of the encoded data of the disparity image
according to the 30V mode.
10 [0454]
In a case where the disparity image header
extension information flag represents that the disparity
image header extension information is included, in the
NAL header of the encoded data of the 30V mode, the
15 disparity image header extension information
(nal_unit_header_depth_extension) is also described. On
the other hand, in a case where the disparity image
header extension information flag represents that the
disparity image header extension information is not
20 included, in the NAL header of the encoded data of the
30V mode, MVC mode header extension information
(nal unit_header_mvc_extension) is also described.
[0455]
Fig. 61 is a diagram that illustrates an example of
25 the description of the disparity image header extension
information illustrated in Fig. 60.
[0456]
As illustrated in Fig. 61, the disparity image
header extension information is configured to be the same
30 as the MVC mode header extension information except that
a view 10 (ref_view_id) of an image corresponding to the
162
disparity image is included.
[0457]
[Example of Description of SEI according to 3DV Mode]
Fig. 62 is a diagram that illustrates an example of
5 the description of SEI according to the 3DV mode
illustrated in Fig. 56.
[0458]
As illustrated in Fig. 62, in the SEI according to
the 3DV mode, similarly to the SEI according to the MVC
10 mode, a message of SEI is described.
[0459]
In other words, in the SEI according to the 3DV
mode, an operation point flag (operation_point_flag)
representing whether or not an operation point is
15 designated is described, and, in a case where the
operation point flag represents that the operation point
is not designated, an all-component flag
(all_view_components_in_au_flag) representing whether or
not the message of the SEI is adapted to all the images
20 and the disparity images within the access unit is
described. On the other hand, in a case where the allcomponent
flag represents that the message of the SEI is
not adapted to all the images and the disparity images
within the access unit, the number
25 (num_view_components_minus1) of view IDs and the view IDs
(sei_view_id) of the images and the disparity images to
which the message of the SEI is adapted are described.
[0460]
On the other hand, in a case where the operation
30 point flag represents that the operation point is
designated, out of operation points to which the message
163
15
of the SEI is adapted, view IDs (sei_op_view_id) of the
images and the disparity images that are adaptation
targets and the number (sei_op_temporal_id) of the
operation points are described. Then, the message
5 (sei_rbsp) of the SEI is described. In addition, in the
SEI according to the 3DV mode, a plurality of messages of
the SEI can be described.
[0461]
In the example illustrated in Fig. 56, although the
10 profiles of the Rand 0 images are assumed to be the same,
the profiles of the Rand 0 images may be different from
each other.
[0462]
[Another Configuration Example of Bit Stream]
Fig. 63 is a diagram that illustrates a
configuration example of the bit stream in units of
access units in a case where profiles of the Rand 0
images illustrated in Fig. 56 are different from each
other.
20 [0463]
The configuration of the bit stream illustrated in
25
Fig. 63 is different from the configuration of the bit
stream illustrated in Fig. 56 in that a Subset SPS
according to the 3DV mode for an image other than the
Subset SPS according to the MVC mode for an image and the
Subset SPS according to the 3DV mode for a disparity
image is arranged.
[0464]
The NAL unit (Subset SPSl) of the Subset SPS
30 according to the MVC mode for an image is an NAL unit of
the Subset SPS that includes profile_ide (128 in the
164
example illustrated in Fig. 63) representing the profile
of the R image out of profiles defined in the MVC mode.
The NAL unit of the Subset SPS (Subset SPS2) according to
the 3DV mode for an image is an NAL unit of the Subset
5 SPS that includes profile_ide (148 in the example
illustrated in Fig. 63) representing the profile of the 0
image out of profiles defined as profiles for an image in
the 3DV mode. The NAL unit of the Subset SPS (Subset
SPS3) for a disparity image is an NAL unit of the Subset
10 SPS that includes profile_ide (138 in the example
illustrated in Fig. 63) representing the profiles of the
L disparity image, the R disparity image, and the 0
disparity image out of profiles defined as profiles for a
disparity image in the 3DV mode.
15 [0465]
[Example of Description of Subset SPS According to 3DV
Mode for Image]
Fig. 64 is a diagram that illustrates an example of
the description of Subset SPS according to the 3DV mode
20 for an image that is illustrated in Fig. 63.
[0466]
As illustrated in Fig. 64, in the Subset SPS
according to the 3DV mode for an image, SPS
(seq_parameter_set_data) that includes profile_ide (148
25 in the example illustrated in Fig. 64) representing the
profile of the image 0 and information for each
profile_ide that is defined in the 3DV mode are described.
[0467]
More specifically, in the Subset SPS according to
30 the 3DV mode for an image, similarly to the Subset SPS
according to the MVC mode as information at a time when
165
t
profile_idc is 148, the extension information
(seq_parameter_set_mvc_extension) according to the MVC
mode, the VUI information flag
(mvc_vui_parameters_present_flag) according to the MVC
5 mode that represents whether or not the VUI extension
information according to the MVC mode is included, and
the like are described. In addition, in a case where the
VUI information flag according to the MVC mode represents
that the VUI extension information according to the MVC
10 mode is included, the VUI extension information
(mvc_vui_parameters_extension) according to the MVC mode
is also described. Furthermore, as the information at a
time when profile_idc is 138, information similar to that
illustrated in Fig. 57 is described.
15 [0468]
Here, the 3DV mode may be a mode for encoding a
display image according to a multi-viewpoint mode that is
compliant with an HEVC (High Efficiency Video Coding)
mode other than the AVC mode and the MVC mode. A bit
20 stream of such a case will be described as below. In
this specification, it is assumed that the HEVC mode is
based on HEVC Working Draft: Thomas Wiegand, Woo-jin Han,
Benjamin Bross, Jens-Rainer Ohm, Gary J. Sullivian, "WD3:
Working Draft3 of High-Efficiency Video Coding", JCTVc-
25 E603 dS (versionS), written on May 20, 2011.
[0469]

[Configuration Example of Bit Stream]
30 Fig. 65 is a diagram that illustrates a
configuration example of a bit stream in units of access
166
units in a case where the 3DV mode is a mode that is
compliant with the HEve mode.
[0470]
In the example illustrated in Fig. 65, it is
5 assumed that the L viewpoint image, the R viewpoint image,
and the 0 viewpoint image as in the example illustrated
in Fig. 56 are encoding targets, the L image is encoded
in accordance with the Ave mode, and the L viewpoint
image, the R viewpoint image, and the 0 viewpoint image
10 are encoded in accordance with the 3DV mode.
[0471]
The bit stream illustrated in Fig. 65 is different
from the bit stream illustrated in Fig. 56 in that the
Subset SPS (Subset SPSl) according to the 3DV mode for an
15 image and the Subset SPS (Subset SPS2) according to the
3DV mode for a disparity image can be described within
the SPS, and NAL headers including mutually-different
types of the NAL units are added to the NAL units of the
SEI according to the HEve mode and the SEI according to
20 the 3DV mode.
[0472]
In the bit stream illustrated in Fig. 65, the
25
Subset SPS according to the 3DV mode for an image and the
Subset SPS according to the 3DV mode for a disparity
image may be described only within the SPS, may be
described to be separate from the SPS, or may be
described in the SPS and described to be separate from
the SPS. Here, the description of the Subset SPS
according to the 3DV mode for an image and the Subset SPS
30 according to the 3DV mode for a disparity image separated
from the SPS is compliant with the BD standard and is
167
appropriate for a case where the encoded data according
to the HEVC mode and the encoded data according to the
3DV mode are generated as ES (elementary streams)
different from each other.
5 [0473]
In the bit stream illustrated in Fig. 65, since the
types of the NAL units that are included in the NAL
headers added to the NAL unit of the SEI according to the
HEVC mode and the NAL unit of the SEI according to the
10 3DV mode are different from each other, an NAL unit of
the SEI according to the HEVC mode or the SEI according
to the 3DV mode can be easily extracted in a decoding
process.
[0474]
15 [Example of Description of SPS]
Fig. 66 is a diagram that illustrates an example of
the description of SPS illustrated in Fig. 65.
[0475]
The description of the SPS illustrated in Fig. 66
20 is the same as the description of the SPS according to
the HEVC mode except that a Subset SPS information flag
(subset_seq_present_flag) representing whether or not
information of the Subset SPS is included is described,
and the information of the Subset SPS is described in a
25 case where the Subset SPS information flag represents
that the information of the Subset SPS is included.
[0476]
As illustrated in Fig. 66, the information of the
Subset SPS includes the number (num_subset seq) of Subset
30 SPS and a Subset SPS flag (subset_seq_info_present_flag)
that represents whether or not the Subset SPS is included.
168
In addition, in a case where the Subset SPS flag
represents that the Subset SPS is included, the Subset
SPS (subset_seq_parameter_set_data) is also included in
the information of the Subset SPS.
5 [0477]
As above, since the number of Subset SPS is
described as the information of the Subset SPS, it can be
recognized whether or the Subset SPS is present by only
reading out the description of the SPS in the decoding
10 process. In addition, since the Subset SPS flag is
described, the Subset SPS can be described separated from
the SPS without being described within the SPS, whereby
duplicate on the description of the Subset SPS can be
prevented.
15 [0478]
[Example of Description of Subset SPS]
Fig. 67 is a diagram that illustrates an example of
the description of the Subset SPS illustrated in Fig. 66.
[0479]
20 The description of the Subset SPS illustrated in
Fig. 67 is the same as the description of the SPS
according to the HEVC mode except that images to which
the Subset SPS is adapted, the number
(num_subset_seq_views) of view IDs of disparity images, a
25 disparity image flag (depth_extension_flag) that
represents whether or not the adaptation target of the
Subset SPS is a disparity image, and an invalidness flag
(seq param_override_flag) that represents whether or not
the SPS is invalidated at the time of adapting the Subset
30 SPS are described.
[0480]
169
In a case where the disparity image flag represents
that the adaptation target of the Subset SPS is a
disparity image, a view ID (ref_view_id) of an image that
corresponds to the disparity image that is the adaption
5 target is described in the Subset SPS. In addition, in a
case where the invalidness flag represents that the SPS
is invalidated at the time of adapting the Subset SPS,
similarly to the SPS, information
(subset_seq_profile_idc) that represents profiles and the
10 like are described in the Subset SPS.
[0481]
Of the descriptions of the Subset SPS, as
descriptions similar to the descriptions of the SPS, for
example, there are information
15 (subset_seq_frame_cropping_flag) that represents whether
or not an adapted image and a disparity image are cropped,
a VUI information flag
(subset_seq_vui_parameters_present flag) that represents
whether or not the VUI information
20 (subset_seq_vui_parameters) is included, and the like.
In a case whether VUI information flag represents that
the VUI information is included, similarly to the SPS,
the VUI information is also described. On the other hand,
in a case where the VUI information flag represents that
25 the VUI information is not included, similarly to the SPS,
the VUI information is not described. In such a case,
the VUI information of the SPS is adapted as the VUI
information.
[0482]
30 [Example of Description of VUI information of Subset SPS]
Fig. 68 is a diagram that illustrates an example of
170
the description of the VUI information of the Subset SPS
illustrated in Fig. 67.
[0483]
The description of the VUI information of the
5 Subset SPS illustrated in Fig. 68 is the same as the
description of the VUI information of the SPS according
to the HEVC mode except that a before-conversion size
information flag (video_src_info_present_flag)
representing whether or not before-conversion size
10 information that is an adaptation target is included is
described.
[0484]
In a case where the before-conversion size
information flag represents that the before-conversion
15 size information of an adaption target is included, the
before-conversion size information is described in the
VUI information illustrated in Fig. 68. In other words,
information (src_pic_width_in_mbs_minus1) that represents
the number of macro blocks of the adaptation target
20 before resolution conversion in the horizontal direction,
information (src_pic_height_in_mbs_minus1) that
represents the number of macro blocks in the vertical
direction, an aspect ratio flag
(src_aspect_ratio_info_present_flag) that represents
25 whether or not aspect information representing an aspect
ratio is included, and the like are described.
[0485]
In a case where the aspect ratio flag represents
that the aspect information is included, the aspect
30 information that is formed by an aspect ratio 10
(src_aspect_ratio_idc) and the like are also included in
171
the before-conversion size information. In a case where
the aspect ratio ID included in the aspect information is
an aspect ratio ID (Extended_SAR) given to all the aspect
ratios that are not defined, values (sar_width and
5 sar_height) of the aspect ratio of the adaption target
before resolution conversion in the horizontal and
vertical directions are also included in the aspect
information.
[0486]
10 In addition, in the VUI information of the Subset
SPS illustrated in Fig. 68, points different from those
of the VUI information of the SPS may be described. In
such a case, for information that is not described in the
VUI information of the Subset SPS, information that is
15 included in the VUI information of the SPS is applied.
[0487]
[Example of Description of SEI]
Fig. 69 is a diagram that illustrates an example of
the description of the SEI according to the 3DV mode that
20 is illustrated in Fig. 65.
[0488]
The description of the SEI according to the 3DV
mode illustrated in Fig. 69 is the same as the
description of the SEI message of the type of the SEI
25 message except that the type (nesting_type) of the SEI
message, an all-component flag
(all_view_components in_au_flag), and the like are
described.
[0489]
30 As the types of the SEI message, there are SEI
message according to the MVC mode, SEI message according
172
to the 3DV mode, SEI message defined by a user, and the
like. In a case where the all-component flag represents
that adaptation to all the images and the disparity
images within the access unit is not performed, the
5 number of view IDs (num_view_components_minus1) of the
images and ~he disparity images to which the SEI message
is adapted and view IDs (nesting_sei_view_id)
corresponding to the number are also described in the SEI
according to the 3DV mode.
10 [0490]

[Description of Computer to Which Present Technology Is
Applied]
Next, a series of processes described above may be
15 performed by either hardware or software. In a case
where the series of processes is performed by software, a
program configuring the software is installed to a
general-purpose computer or the like.
[0491]
20 Fig. 70 illustrates a configuration example of a
computer according to an embodiment to which a program
that executes the above-described series of processes is
installed.
[0492]
25 The program may be recorded in a storage unit 808
as a recording medium that is built in the computer or a
ROM (Read Only Memory) 802 in advance.
[0493]
Alternatively, the program may be stored (recorded)
30 on a removable medium 811. Such removable medium 811 may
be provided as so-called package software. Here,
173
examples of the removable medium 811 include a flexible
disk, a compact disc read only memory (CD-ROM), a magneto
optical (MO) disk, a digital versatile disc (DVD), a
magnetic disk, and a semiconductor memory.
5 [0494]
In addition to installing the program on the
computer from a removable medium 811 as described above
through a drive 810, the program may be downloaded into
the computer through a communication network or a
10 broadcast network and be installed to the storage unit
808 that is built therein. In other words, the program
may be transmitted to the computer in a wireless manner,
for example, from a download site through a satellite
used for digital satellite broadcasting or be transmitted
15 to the computer in a wired manner through a network such
as a local area network (LAN) or the Internet.
[0495]
The computer includes a CPU (central processing
unit) 801 therein, and an input/output interface 805 is
20 connected to the CPU 801 through a bus 804.
[0496]
When an instruction is input from a user through
the input/output interface 805 by operating an input unit
806, the CPU 801 executes a program that is stored in the
25 ROM 802 in accordance with the instruction.
Alternatively, the CPU 801 loads a program that is stored
in the storage unit 808 into a RAM (random access memory)
803 and executes the program.
[0497]
30 Accordingly, the CPU 801 performs the process
according to the above-described flowchart or the process
174
that is performed by the configuration of the abovedescribed
block diagram. Then, the CPU 801 outputs a
processing result from an output unit 807, for example,
through the input/output interface 805, transmits the
5 processing result from a communication unit 809, or
records the processing result in the storage unit 808 as
is needed.
[0498]
Here, the input unit 806 is configured by a
10 keyboard, a mouse, a microphone, and the like. In
addition, the output unit 807 is configured by an LCD
(liquid crystal display), a speaker, and the like.
[0499]
Here, in this specification, the process that is
15 performed by a computer in accordance with a program does
not need to be performed necessarily in a time series in
accordance with the sequence described in the flowchart.
In other words, the process that is performed by the
computer in accordance with the program includes a
20 process (for example, a parallel process or a process
using an object) that is performed in a parallel manner
or in an individual manner.
[0500]
In addition, the program may be processed by one
25 computer (processor) or may be processed by a plurality
of computers in a distributed manner. Furthermore, the
program may be transmitted to a remote computer and be
executed.
[0501]
30 The present technology can be applied to an
encoding device and a decoding device that are used when
175
•
data is received through a network medium such as
satellite broadcasting, cable TV (television), the
Internet, or a cellular phone or data is processed on a
storage medium such as an optical disc, a magnetic disk,
5 or a flash memory.
[0502]
In addition, the encoding device and the decoding
device described above can be applied to an arbitrary
electronic apparatus. Hereinafter, the examples thereof
10 will be described.
[0503]

[Configuration Example of Television Apparatus]
Fig. 71 illustrates a schematic configuration of a
15 television apparatus according to the present technology
as an example. The television apparatus 900 includes an
antenna 901, a tuner 902, a demultiplexer 903, a decoder
904, a video signal processing unit 905, a display unit
906, an audio signal processing unit 907, a speaker 908,
20 and an external interface unit 909. In addition, the
television apparatus 900 includes a control unit 910, a
user interface unit 911, and the like.
[0504]
The tuner 902 selects a desired channel from among
25 broadcasting signals received by the antenna 901,
demodulates a corresponding broadcasting signal, and
outputs an acquired decoded bit stream to the
demultiplexer 903.
[0505]
30 The demultiplexer 903 extracts packets of a video
and an audio of a program that is a viewing target from
176
•
the encoded bit stream and outputs data of the extracted
packets to the decoder 904. In addition, the
demultiplexer 903 supplies packets of data such as EPG
(electronic program guide) to the control unit 910.
5 Furthermore, in a case where scrambling is performed, the
scrambling is cancelled by the demultiplexer.
[0506]
The decoder 904 performs a decoding process of the
packets, outputs video data generated by the decoding
10 process to the video signal processing unit 905, and
outputs audio data to the audio signal processing unit
907.
[0507]
The video signal processing unit 905 performs noise
15 elimination or video processing according to a user's
setting for the video data. The video signal processing
unit 905 generates video data of a program to be
displayed on the display unit 906, image data acquired by
a process that is based on an application supplied
20 through a network, and the like. In addition, the video
signal processing unit 905 generates video data used for
displaying a menu screen of item selection or the like
and superimposes the generated video data on the video
data of the program. The video signal processing unit
25 905 generates a driving signal based on the video data
generated as above and drives the display unit 906.
[0508]
The display unit 906 drives a display device (for
example, a liquid crystal display device or the like)
30 based on the driving signal supplied from the video
signal processing unit 905, thereby displaying a video of
177
•
the program or the like.
[0509]
The audio signal processing unit 907 performs
predetermined processing such as noise elimination for
5 the audio data, performs a D/A conversion process or an
amplification process for the audio data after processing,
and supplies resultant audio data to the speaker 908,
thereby performing an audio output process.
[0510]
10 The external interface unit 909 is an interface for
a connection with an external device or a network and
performs data transmission or data reception of video
data, audio data, and the like.
[0511]
15 The user interface unit 911 is connected to the
control unit 910. The user interface unit 911 is
configured by an operating switch, a remote control
signal reception unit, and the like and supplies an
operating signal according to a user's operation to the
20 control unit 910.
[0512]
The control unit 910 is configured by a CPU
(central processing unit), a memory, and the like. The
memory stores various kinds of data, EPG data, data
25 acquired through the network, and the like that are
necessary for a program that is executed by the CPU or
the CPU to performing a process. The program stored in
the memory is read out and executed by the CPU at
predetermined timing such as start-up of the television
30 apparatus 900. The CPU controls each unit so as to
enable the television apparatus 900 to perform an
178
•
operation according to a user's operation by executing
the program.
[0513]
In addition, in the television apparatus 900, a bus
5 912 is disposed so as to connect the tuner 902, the
demultiplexer 903, the video signal processing unit 905,
the audio signal processing unit 907, the external
interface unit 909, and the like and the control unit 910
to each other.
10 [0514]
In the television apparatus configured as above,
the function of the decoding device (decoding method)
according to this application is arranged in the decoder
904. Accordingly, multi-viewpoint images can be decoded
15 in accordance with a mode having compatibility with an
existing mode.
[0515]

[Configuration Example of Cellular Phone]
20 Fig. 72 illustrates a schematic configuration of a
cellular phone according to the present technology. The
cellular phone 920 includes a communication unit 922, an
audio codec 923, a camera unit 926, an image processing
unit 927, a demultiplexing unit 928, a
25 recording/reproducing unit 929, a display unit 930, and a
control unit 931. These are interconnected through a bus
933.
[0516]
In addition, an antenna 921 is connected to the
30 communication unit 922, and a speaker 924 and a
microphone 925 are connected to the audio codec 923.
179
•
Furthermore, an operating unit 932 is connected to the
control unit 931.
[0517]
The cellular phone 920 performs various operations
5 such as transmission and reception of an audio signal,
transmission and reception of an electronic mail or image
data, image capturing, and data recording in various
modes such as a voice phone mode and a data communication
mode.
10 [0518]
In the voice phone mode, an audio signal generated
by the microphone 925 is converted into audio data and is
compressed by the audio codec 923, and resultant audio
data is supplied to the communication unit 922. The
15 communication unit 922 performs a modulation process, a
frequency converting process, and the like for the audio
data, thereby generating a transmission signal. In
addition, the communication unit 922 supplies the
transmission signal to the antenna 921, thereby
20 transmitting the transmission signal to a base station
not illustrated in the figure. Furthermore, the
communication unit 922 performs an amplification process,
a frequency converting process, a demodulation process,
and the like for the reception signal received by the
25 antenna 921 and supplies acquired audio data to the audio
codec 923. The audio codec 923 performs data
decompression of the audio data, converts the audio data
into an analog audio signal, and outputs the audio signal
to the speaker 924.
30 [0519]
In the data communication mode, in a case where
180
•
mail transmission is performed, the control unit 931
receives character data input by an operation of the
operating unit 932 and displays the input character on
the display unit 930. In addition, the control unit 931
5 generates mail data based on a user's instruction or the
like supplied from the operating unit 932 and supplies
the generated mail data to the communication unit 922.
The communication unit 922 performs a modulation process,
a frequency converting process, and the like for the mail
10 data and transmits an acquired transmission signal from
the antenna 921. In addition, the communication unit 922
performs an amplification process, a frequency converting
process, a demodulation process, and the like for the
reception signal received by the antenna 921, thereby
15 restoring the mail data. This mail data is supplied to
the display unit 930, whereby the content of the mail is
displayed.
[0520]
In addition, the cellular phone 920 may record the
20 received mail data on a storage medium by using the
recording/reproducing unit 929. The storage medium is an
arbitrary rewritable storage medium. For example, the
storage medium is a semiconductor memory such as a RAM or
a built-in-type flash memory or a removable medium such
25 as a hard disk, a magnetic disk, a magneto-optical disk,
an optical disc, a USB memory, or a memory card.
[0521]
In a case where image data is transmitted in the
data communication mode, the image data generated by the
30 camera unit 926 is supplied to the image processing unit
927. The image processing unit 927 performs an encoding
181
•
process of the image data, thereby generating encoded
data.
[0522]
The demultiplexing unit 928 multiplexes the encoded
5 data generated by the image processing unit 927 and the
audio data supplied from the audio codec 923 in a
predetermined mode and supplies multiplexed data to the
communication unit 922. The communication unit 922
performs a modulation process, a frequency converting
10 process, and the like of the multiplexing data and
transmits an acquired transmission signal from the
antenna 921. In addition, the communication unit 922
performs an amplification process, a frequency converting
process, a demodulation process, and the like for the
15 reception signal received by the antenna 921, thereby
restoring the multiplexed data. This multiplexed data is
supplied to the demultiplexing unit 928. The
demultiplexing unit 928 demultiplexes the multiplexed
data, supplies encoded data to the image processing unit
20 927, and supplies audio data to the audio codec 923. The
image processing unit 927 decodes the encoded data,
thereby generating image data. This image data is
supplied to the display unit 930, and the received image
is displayed. The audio codec 923 converts audio data
25 into an analog audio signal and supplies the analog audio
signal to the speaker 924, thereby outputting the
received audio.
[0523]
In the cellular phone device configured as above,
30 the function of the decoding device (decoding method)
according to this application is arranged in the image
182
•
processing unit 927. Accordingly, in communication of
the image data, when decoded image data is generated by
decoding an encoded stream, multi-viewpoint images can be
decoded in accordance with a mode that has compatibility
5 with an existing mode.
[0524]

[Configuration Example of Recording and Reproducing
Device]
10 Fig. 73 illustrates a schematic configuration of a
recording and reproducing device according to the present
technology. The recording and reproducing device 940,
for example, records audio data and video data of a
received broadcasting program on a recording medium and
15 supplies the recorded data to a user at timing according
to a user's instruction. In addition, the recording and
reproducing device 940, for example, may acquire audio
data and video data from another device and record the
data on a recording medium. Furthermore, the recording
20 and reproducing device 940 decodes and outputs the audio
data and the video data that are recorded on the
recording medium, whereby an image display or audio
output in a monitor device or the like can be performed.
[0525]
25 The recording and reproducing device 940 includes a
tuner 941, an external interface unit 942, an encoder 943,
an HDD (Hard Disk Drive) unit 944, a disk driver 945, a
selector 946, a decoder 947, an OSD (On-Screen Display)
unit 948, a control unit 949, and a user interface unit
30 950.
[0526]
183
•
The tuner 941 selects a desired channel from among
broadcasting signals received by an antenna not
illustrated in the figure. The tuner 941 outputs an
encoded bit stream that is acquired by demodulating a
5 reception signal of the desired channel to the selector
946.
[0527]
The external interface unit 942 is configured as at
least one of an IEEE 1394 interface, a network interface
10 unit, a USB interface, a flash memory interface, and the
like. The external interface unit 942 is an interface
used for a connection with an external device, a network,
a memory card, or the like and performs data reception of
video data, audio data, and the like to be recorded.
15 [0528]
When the video data and the audio data supplied
from the external interface unit 942 are not encoded, the
encoder 943 encodes the data in accordance with a
predetermined mode and outputs the encoded bit stream to
20 the selector 946.
[0529]
The HDD unit 944 records content data such as a
video and an audio, various programs, and other data on a
built-in hard disk and reads out the data from the hard
25 disk at the time of reproducing the data or the like.
[0530]
The disk driver 945 performs signal recording and
signal reproduction for the installed optical disc.
Examples of the optical disc include a DVD disc (a DVD30
Video, a DVD-RAM, a DVD-R, a DVD-RW, a DVD+R, DVD+RW, and
the like), a Blu-ray disc, and the like.
184
•
[0531]
The selector 946 selects one of encoded bit streams
supplied from the tuner 941 or the encoder 943 at the
time of recording a video or audio and supplies the
5 encoded bit stream to one of the HDD unit 944 and the
disk driver 945. In addition, the selector 946 supplies
an encoded bit stream that is output from the HDD unit
944 or the disk driver 945 to the decoder 947 at the time
of reproducing a video or an audio.
10 [0532]
The decoder 947 performs a decoding process of the
encoded bit stream. The decoder 947 supplies the video
data that is generated by performing the decoding process
to the OSD unit 948. In addition, the decoder 947
15 outputs audio data that is generated by performing a
decoding process.
[0533]
The OSD unit 948 generates video data used for
displaying a menu screen of item selection or the like
20 and outputs the generated video data while being
superimposed on the video data output from the decoder
947.
[0534]
The user interface unit 950 is connected to the
25 control unit 949. The user interface unit 950 is
configured by an operating switch, a remote control
signal reception unit, and the like and supplies an
operating signal according to a user's operation to the
control unit 949.
30 [0535]
The control unit 949 is configured by a CPU, a
185
•
memory, and the like. The memory stores various kinds of
data that is necessary for a program executed by the CPU
or the CPU to perform a process. The program that is
stored in the memory is read out and executed by the CPU
5 at predetermined timing such as start-up of the recording
and reproducing device 940. The CPU controls each unit
such that the recording and reproducing device 940
performs an operation according to a user's operation by
executing the program.
10 [0536]
In the recording and reproducing device configured
as above, the function of the encoding device (encoding
method) according to this application is arranged in the
encoder 943. Accordingly, when decoded image data is
15 generated by decoding an encoded stream, multi-viewpoint
images can be encoded in accordance with a mode that has
compatibility with an existing mode.
[0537]

20 [Configuration Example of Imaging Apparatus]
Fig. 74 is a diagram that illustrates a schematic
configuration of an imaging apparatus according to the
present technology. The imaging apparatus 960 images a
subject, displays an image of the subject on the display
25 unit or records the image on the recording medium as
image data.
[0538]
The imaging apparatus 960 includes an optical block
961, an imaging unit 962, a camera signal processing unit
30 963, an image data processing unit 964, a display unit
965, an external interface unit 966, a memory unit 967, a
186
•
medium drive 968, an OSO unit 969, and a control unit 970.
In addition, a user interface unit 971 is connected to
the control unit 970. Furthermore, the image data
processing unit 964, the external interface unit 966, the
5 memory unit 967, the medium drive 968, the aso unit 969,
the control unit 970, and the like are connected through
a bus 972.
[0539]
The optical block 961 is configured by using a
10 focus lens, a diaphragm mechanism, and the like. The
optical block 961 forms an optical image of a subject on
an imaging face of the imaging unit 962. The imaging
unit 962 is configured by using a CCO or a CMOS image
sensor, generates an electric signal in accordance with
15 an optical image through photoelectric conversion, and
supplies the electric signal to the camera signal
processing unit 963.
[0540]
The camera signal processing unit 963 performs
20 various camera signal processes such as knee correction,
gamma correction, and color correction for the electric
signal supplied from the imaging unit 962. The camera
signal processing unit 963 supplies the image data after
the camera signal processing to the image data processing
25 unit 964.
[0541]
The image data processing unit 964 performs an
encoding process for the image data supplied from the
camera signal processing unit 963. The image data
30 processing unit 964 supplies the encoded data generated
by performing the encoding process to the external
187
•
interface unit 966 or the medium drive 968. In addition,
the image data processing unit 964 performs a decoding
process for the encoded data that is supplied from the
external interface unit 966 or the medium drive 968. The
5 image data processing unit 964 supplies the image data
that is generated by performing the decoding process to
the display unit 965. In addition, the image data
processing unit 964 supplies display data that is
acquired by the process of supplying the image data
10 supplied from the camera signal processing unit 963 to
the display unit 965 or is acquired from the OSD unit 969
to the display unit 965 while being superimposed on the
image data.
[0542]
15 The OSD unit 969 generates display data of a menu
screen, an icon, or the like that is formed from a symbol,
a character, or a graphic and outputs the display data to
the image data processing unit 964.
[0543]
20 The external interface unit 966, for example, is
configured by USB input/output terminals and the like and
is connected to a printer in a case where an image is
printed. In addition, a drive is connected to the
external interface unit 966 as is necessary, a removable
25 medium such as a magnetic disk or an optical disc is
appropriately installed therein, and a computer program
read from the removable medium is installed as is
necessary. In addition, the external interface unit 966
includes a network interface that is connected to a
30 predetermined network such as a LAN or the Internet. The
control unit 970, for example, in accordance with an
188
•
instruction supplied from the user interface unit 971,
can read encoded data from the memory unit 967 and supply
the encoded data from the external interface unit 966 to
another device that is connected through the network. In
5 addition, the control unit 970 can acquire the encoded
data or the image data, which is supplied from another
device through the network, through the external
interface unit 966 and supply the data to the image data
processing unit 964.
10 [0544]
As the recording medium that is driven by the
medium drive 968, for example, an arbitrary readable and
writable removable medium such as a magnetic disk, a
magneto-optical disk, an optical disc, or a semiconductor
15 memory is used. In addition, the type of the recording
medium as the removable medium is arbitrary and may be a
tape device, a disk device, or a memory card.
Furthermore, the recording medium may be a non-contact IC
card, or the like.
20 [0545]
In addition, the medium drive 968 and the recording
medium may be integrated together and configured by a
non-portable recording medium such as a built-in type
hard disk drive or an SSD (Solid State Drive).
25 [0546]
The control unit 970 is configured by a CPU, a
memory, and the like. The memory stores various kinds of
data that is necessary for a program executed by the CPU
or the CPU to perform a process. The program that is
30 stored in the memory is read out and executed by the CPU
at predetermined timing such as start-up of the imaging
189
•
apparatus 960. The CPU controls each unit such that the
imaging apparatus 960 performs an operation according to
a user's operation by executing the program.
[0547]
5 In the imaging apparatus configured as above, the
function of the decoding device (decoding method)
according to this application is arranged in the image
data processing unit 964. Accordingly, when decoded
image data is generated by decoding encoded data that is
10 recorded in a memory unit 967, a recording medium, or the
like, multi-viewpoint images can be decoded in accordance
with a mode that has compatibility with an existing mode.
[0548]
In addition, an embodiment of the present
15 technology is not limited to the above-described
embodiments, and various changes can be made therein in a
range not departing from the concept of the present
technology.
20 REFERENCE SIGNS LIST
[0549]
50 Encoding device
52 Image converting unit
53 Disparity image generating unit
25 55 Compatibility information generating unit
56 Inter-viewpoint distance information generating
unit
30
59
61
62
120
Multiplexing unit
Compatible encoder
Auxiliary encoder
Decoding device
190
•
121 Separation unit
127 Image generating unit
131 Compatible decoder
132 Auxiliary decoder
5 140 Encoding device
142 Image converting unit
143 Disparity image generating unit
144 Inter-viewpoint distance information generating
unit
10 151 Compatible encoder
152 Auxiliary encoder
170 Decoding device
171 Image generating unit
180 Encoding device
15 181 Image converting unit
191 Compatible encoder
200 Decoding device
202 Image Generating Unit
211 Compatible Decoder
20

•
CLAIMS
1. An encoding device comprising:
a compatible image encoding unit that generates a
5 first encoded stream by designating a compatible image
from a multi-viewpoint image and encoding the designated
compatible image;
an image converting unit that lowers the resolution
of an auxiliary image that is used when the multi-
10 viewpoint image is generated from the compatible image;
an auxiliary image encoding unit that generates a
second encoded stream by encoding the auxiliary image of
which the resolution is lowered by the image converting
unit; and
15 a transmission unit that transmits the first
encoded stream generated by the compatible image encoding
unit and the second encoded stream generated by the
auxiliary image encoding unit.
20 2. The encoding device according to claim 1, further
comprising a setting unit that sets auxiliary image
resolution identifying information identifying that the
resolution of the auxiliary image at the time of encoding
is different from the resolution of the compatible image,
25 wherein the transmission unit transmits the
auxiliary image resolution identifying information set by
the setting unit.
3. The encoding device according to claim 2,
30 wherein the setting unit sets auxiliary image
converting mode information that represents a conversion
192
•
mode of the resolution of the auxiliary image, and
the transmission unit transmits the auxiliary image
converting mode information set by the setting unit.
5 4. The encoding device according to claim 1, further
comprising a disparity image encoding unit that encodes
an auxiliary disparity image representing disparity of
the auxiliary image,
wherein the image converting unit lowers the
10 resolution of the auxiliary disparity image, and
the disparity image encoding unit encodes the
auxiliary disparity image of which the resolution is
lowered by the image converting unit.
15 5. The encoding device according to claim 4, further
comprising a setting unit that sets auxiliary disparity
image resolution identifying information identifying that
the resolution of the auxiliary disparity image at the
time of encoding is different from the resolution of the
20 compatible image,
wherein the transmission unit transmits the
auxiliary disparity image resolution identifying
information set by the setting unit.
25 6. The encoding device according to claim 5,
wherein the setting unit sets auxiliary disparity
image converting mode information that represents a
conversion mode of the resolution of the auxiliary
disparity image, and
30 the transmission unit transmits the auxiliary
disparity image converting mode information set by the
193
•
setting unit.
7. The encoding device according to claim 4, further
comprising:
5 a multiplexing unit that multiplexes the auxiliary
disparity image of which the resolution is lowered by the
image converting unit; and
a setting unit that sets auxiliary disparity image
multiplexing mode information representing a multiplexing
10 mode of the auxiliary disparity image,
wherein the disparity image encoding unit encodes
the auxiliary disparity image multiplexed by the
multiplexing unit, and
the transmission unit transmits the auxiliary
15 disparity image multiplexing mode information set by the
setting unit.
8. The encoding device according to claim 2,
wherein the image converting unit multiplexes the
20 auxiliary image of which the resolution is lowered,
the setting unit sets auxiliary image multiplexing
mode information representing a multiplexing mode of the
auxiliary image,
the auxiliary image encoding unit encodes the
25 auxiliary image multiplexed by the image converting unit,
and
the transmission unit transmits the auxiliary image
multiplexing mode information set by the setting unit.
30 9. An encoding method to be performed by an encoding
device, the encoding method comprising:
194
•
a compatible image encoding step of generating a
first encoded stream by designating a compatible image
from a multi-viewpoint image and encoding the designated
compatible image;
5 an image converting step of lowering the resolution
of an auxiliary image that is used when the multiviewpoint
image is generated from the compatible image;
an auxiliary image encoding step of generating a
second encoded stream by encoding the auxiliary image of
10 which the resolution is lowered in a process of the image
converting step; and
a transmitting step of transmitting the first
encoded stream generated in a process of the compatible
image encoding step and the second encoded stream
15 generated in a process of the auxiliary image encoding
step.
10. A decoding device comprising:
a reception unit that receives a first encoded
20 stream that is acquired as a result of encoding a
compatible image designated from a multi-viewpoint image
and a second encoded stream that is acquired as a result
of lowering the resolution of an auxiliary image used
when the multi-viewpoint image is generated from the
25 compatible image and encoding the auxiliary image;
a compatible image decoding unit that decodes the
first encoded stream received by the reception unit;
an auxiliary image decoding unit that decodes the
second encoded stream received by the reception unit; and
30 an image converting unit that increases the
resolution of the auxiliary image, of which the
195
resolution is lowered, acquired as a result of decoding
by the auxiliary image decoding unit.
11. The decoding device according to claim 10,
5 wherein the reception unit receives auxiliary image
resolution identifying information identifying that the
resolution of the auxiliary image at the time of encoding
is different from the resolution of the compatible image,
and
10 the image converting unit increases the resolution
of the auxiliary image of which the resolution is lowered,
based on the auxiliary image resolution identifying
information received by the reception unit.
15 12. The decoding device according to claim 11,
wherein the reception unit receives auxiliary image
converting mode information that represents a conversion
mode of the resolution of the auxiliary image, and
the image converting unit increases the resolution
20 of the auxiliary image that is lowered, based on the
auxiliary image converting mode information received by
the reception unit.
13. The decoding device according to claim 10, further
25 comprising a disparity image decoding unit that decodes a
disparity encoded stream that is acquired as a result of
lowering the resolution of the auxiliary disparity image
representing disparity of the auxiliary image and
encoding the auxiliary disparity image,
30 wherein the reception unit receives the disparity
encoded stream, and
196
the image converting unit increases the resolution
of the auxiliary disparity image, of which the resolution
is lowered, acquired as a result of decoding by the
disparity image decoding unit.
5
14. The decoding device according to claim 13,
wherein the reception unit receives auxiliary
disparity image resolution identifying information
identifying that the resolution of the auxiliary
10 disparity image at the time of encoding is different from
the resolution of the compatible image, and
the image converting unit increases the resolution
of the auxiliary disparity image, of which the resolution
is lowered, based on the auxiliary disparity image
15 resolution identifying information received by the
reception unit.
15. The decoding device according to claim 14,
wherein the reception unit receives auxiliary
20 disparity image converting mode information that
represents a conversion mode of the resolution of the
auxiliary disparity image, and
the image converting unit increases the resolution
of the auxiliary disparity image, of which the resolution
25 is lowered, based on the auxiliary disparity image
converting mode information received by the reception
unit.
16. The decoding device according to claim 13, further
30 comprising a separation unit that separates the
multiplexed auxiliary disparity image,
197
wherein the disparity encoded stream is acquired as
a result of encoding the multiplexed auxiliary disparity
image of which the resolution is lowered,
the reception unit receives auxiliary disparity
5 image multiplexing mode information that represents a
multiplexing mode of the auxiliary disparity image,
the separation unit separates the multiplexed
auxiliary disparity image, of which the resolution is
lowered, acquired as a result of decoding by the
10 disparity image decoding unit based on the auxiliary
disparity image multiplexing mode information received by
the reception unit, and
the image converting unit increases the resolution
of the auxiliary disparity image, of which the resolution
15 is lowered, acquired as a result of separation by the
separation unit.
17. The decoding device according to claim 11,
wherein the second encoded stream is acquired as a
20 result of encoding the multiplexed auxiliary image of
which the resolution is lowered,
the reception unit receives auxiliary image
multiplexing mode information that represents a
multiplexing mode of the auxiliary image,
25 the separation unit separates the multiplexed
auxiliary image, of which the resolution is lowered,
acquired as a result of decoding by the auxiliary image
decoding unit based on the auxiliary image multiplexing
mode information received by the reception unit, and
30 the image converting unit increases the resolution
of the auxiliary image, of which the resolution is
198
lowered, acquired as a result of separation by the
separation unit.
18. A decoding method to be performed by a decoding
5 device, the decoding method comprising:
a receiving step of receiving a first encoded .
stream that is acquired as a result of encoding a
compatible image designated from a multi-viewpoint image
and a second encoded stream that is acquired as a result
10 of lowering the resolution of an auxiliary image used
when the multi-viewpoint image is generated from the
compatible image and encoding the auxiliary image;
a compatible image decoding step of decoding the
first encoded stream received in a process of the
15 receiving step;
an auxiliary image decoding step of decoding the
second encoded stream received in the process of the
receiving step; and
an image converting step of increasing the
20 resolution of the auxiliary image, of which the
resolution is lowered, acquired as a result of decoding
in a process of the auxiliary image decoding step.
•
Dated this 25.02.2013 ~~~\V-.-"""',
DEEPAK KUMAR
OFREMFRY&SAGAR
ATTORNEY FOR THE APPLICANT[S]