SP30B970WOOO
DESCRIPTION
ENCODING DEVICE, ENCODING METHOD, DECODING DEVICE, AND
DECODING METHOD
5 TECHNICAL FIELD
[0001]
The present technology relates to an encoding
device, an encoding method, a decoding device, and a
decoding method, and more particularly, to an encoding
10 device, an encoding method, a decoding device, and a
decoding method capable of encoding and decoding multiviewpoint
images in accordance with a mode that is
compatible with an existing mode.
15 BACKGROUND ART
[0002]
Currently, as a 3D image viewing mode, generally,
there is a mode (hereinafter, referred to as a twoviewpoint
mode) in which two-viewpoint images alternately
20 displayed are seen by wearing glasses of which a left-eye
shutter is open at the time of displaying one image out
of two-viewpoint images, and a right-eye shutter is open
at the time of displaying the other image.
[0003]
25 However, in such a two-viewpoint mode, a viewer
needs to purchase glasses in addition to a 3D image
display device, and accordingly, the viewer's willingness
to buy reduces. In addition, since a viewer needs to
wear glasses for viewing, it annoys the viewer.
30 Accordingly, a demand for a viewing mode (hereinafter,
referred to as a multi-viewpoint mode) increases in which
1
SP308970WOOO
a 3D image can be viewed without wearing glasses.
[0004]
In the multi-viewpoint mode, multi-viewpoint images
are displayed such that a viewable angle is different for
5 each viewpoint, and, a 3D image can be seen by a viewer
viewing images of arbitrary two viewpoints with left and
right eyes without wearing glasses.
[0005]
A display device that provides viewing of a multi-
10 viewpoint mode, for example, generates multi-viewpoint
images for a multi-viewpoint mode based on images of two
viewpoints for a two-viewpoint mode and displays the
generated multi-viewpoint images. More specifically, the
display device acquires parallax (depth) of two-viewpoint
15 images for a two-viewpoint mode using an image parallax
estimating technology (Depth Estimation). Then, the
display device generates a synthesized image of multiviewpoint
images adjacent to a viewpoint corresponding to
the images of two viewpoints for a two-viewpoint mode
20 using a multi-viewpoint image generating technology (View
Generation) using the parallax between images of two
viewpoints and a synthesis technology (View Synthesis)
and displays the synthesized image.
[0006]
25 Existing encoding modes include an advanced video
coding (AVC) mode and a multi-view video coding (MVC)
mode.
[0007]
Fig. 1 is a diagram that illustrates an example of
30 an encoding device that encodes two-viewpoint images in
the MVC mode and multiplexes the encoded images.
2
SP308970WOOO
[0008]
The encoding device 10 illustrated in Fig. 1 is
configured by an imaging unit 11A, an imaging unit 11B,
an MVC encoder 12, and a multiplexing unit 13.
5 [0009]
The imaging unit 11A captures an image A of a
predetermined viewpoint and supplies the captured image
to the MVC encoder 12. In addition, the imaging unit 11B
captures an image B of a viewpoint that is different from
10 that of the image A and supplies the captured image to
the MVC encoder 12. The MVC encoder 12 performs encoding
in accordance with the MVC mode with the image A supplied
from the imaging unit 11A set as a base image and the
image B supplied from the imaging unit 11B set as a
15 dependent image. The MVC encoder 12 supplies the images
A and B after the encoding to the multiplexing unit 13.
The multiplexing unit 13 generates a first TS (transport
stream) (hereinafter, referred to as TS1) based on the
image A after the encoding, generates a second TS
20 (hereinafter, referred to as TS2) based on the image B
after the encoding, and multiplexes the TS1 and TS2.
[0010]
The multiplexed TS1 and TS2 are separated by a
decoding device, and the images A and B after the
25 encoding are decoded in accordance with a mode that
corresponds to the MVC mode. Then, the images A and B
that are acquired as a result of the decoding are
alternately displayed. At this time, a viewer, for
example, wears glasses of which the left-eye shutter is
30 open at the time of displaying the image A and of which
the right-eye shutter is open at the time of displaying
3
•
5
10
SP308970WOOO
the image B, views the image A only with the left eye,
and views the image B only with the right eye. In this
way, the viewer can see a 3D image. In a case where a 2D
image is desired to be displayed, only the image A is
displayed.
[0011]
Meanwhile, a mode for encoding a multi-viewpoint
image has also been devised (for example, see Patent
Document 1).
CITATION LIST
PATENT DOCUMENT
[0012]
Patent Document 1: Japanese Patent Application Laid-Open
15 No. 2008-182669
SUMMARY OF THE INVENTION
PROBLEMS TO BE SOLVED BY THE INVENTION
[0013]
20 However, a mode, which encodes and decodes multiviewpoint
images, having compatibility with an existing
mode for images of two viewpoints or less has not been
considered.
[0014]
25 The present technology has been contrived in view
of such situations and enables encoding and decoding
multi-viewpoint images according to a mode that is
compatible with an existing mode.
30 SOLUTIONS TO PROBLEMS
[0015]
4
5
10
15
SP308970WOOO
An encoding device according to a first aspect of
the present technology includes: a compatible image
encoding unit that generates a first encoded stream by
designating a compatible image from among multi-viewpoint
images and encoding the designated compatible image in
units of access units; an auxiliary image encoding unit
that generates a second encoded stream by encoding
auxiliary images used when multi-viewpoint images are
generated from the compatible image in units of the
access units; a setting unit that sets boundary
information representing a boundary of a unit; and a
transmission unit that transmits the first encoded stream
generated by the compatible image encoding unit, the
boundary information set by the setting unit, and the
second encoded stream encoded by the auxiliary image
encoding unit.
[0016]
An encoding method according to the first aspect of
the present technology corresponds to the encoding device
20 according to the first aspect of the present technology.
[0017]
In the first aspect of the present technology, a
first encoded stream is generated by designating a
compatible image from among multi-viewpoint images and
25 encoding the designated compatible image in units of
access units, a second encoded stream is generated by
encoding auxiliary images used when multi-viewpoint
images are generated from the compatible image in units
of the access units, boundary information representing a
30 boundary of a unit is set, and the first encoded stream,
the boundary information, and the second encoded stream
5
SP308970WOOO
are transmitted.
[0018]
A decoding device according to a second aspect of
the present technology includes: a separation unit that
5 receives a first encoded stream that is acquired as a
result of encoding a compatible image designated from
multi-viewpoint images in units of access units, boundary
information that represents a boundary of a unit, and a
second encoded stream that is acquired as a result of
10 encoding auxiliary images used at the time of generating
the multi-viewpoint images from the compatible image in
units of access units and separates the first encoded
stream and the second encoded stream based on the
boundary information; a compatible image decoding unit
15 that decodes the first encoded stream separated by the
separation unit; and an auxiliary image decoding unit
that decodes the second encoded stream separated by the
separation unit.
[0019]
20 A decoding method according to the second aspect of
the present technology corresponds to the decoding device
according to the second aspect of the present technology.
[0020]
In the second aspect of the present technology, a
25 first encoded stream that is acquired as a result of
encoding a compatible image designated from multiviewpoint
images in units of access units, boundary
information that represents a boundary of a unit, and a
second encoded stream that is acquired as a result of
30 encoding auxiliary images used at the time of generating
the multi-viewpoint images from the compatible image in
6
SP308970WOOO
units of access units are received, the first encoded
stream and the second encoded stream are separated based
on the boundary information; the separated first encoded
stream is decoded, and the separated second encoded
5 stream is decoded.
[0021]
In addition, the encoding device according to the
first aspect and the decoding device according to the
second aspect may be realized by causing a computer to
10 execute a program.
[0022]
Furthermore, the program that is executed by a
computer so as to realize the encoding device according
to the first aspect and the decoding device according to
15 the second aspect may be provided while being transmitted
through a transmission medium or being recorded on a
recording medium.
EFFECTS OF THE INVENTION
20 [0023]
According to the first aspect of the present
technology, multi-viewpoint images can be encoded in a
mode having compatibility with an existing mode.
[0024]
25 In addition, according to the second aspect of the
present technology, multi-viewpoint images that have been
encoded in a mode having compatibility with an existing
mode can be decoded.
30 BRIEF DESCRIPTION OF DRAWINGS
[0025]
7
SP308970WOOO
Fig. 1 is a diagram that illustrates an example of
a conventional encoding device.
Fig. 2 is a block diagram that illustrates a
configuration example of an encoding device according to
5 a first embodiment of the present technology.
Fig. 3 is a diagram that illustrates a
configuration example of a TSs that are generated by a
multiplexing unit illustrated in Fig. 2.
Fig. 4 is a diagram that illustrates a detailed
10 configuration example of a TS2 illustrated in Fig. 3.
Fig. 5 is a diagram that illustrates an example of
a method of multiplexing the TS1 and TS2.
Fig. 6 is a flowchart that illustrates an encoding
process performed by the encoding device illustrated in
15 Fig. 2.
Fig. 7 is a flowchart that illustrates the encoding
process performed by the encoding device illustrated in
Fig. 2.
Fig. 8 is a diagram that illustrates a
20 configuration example of a decoding device that
corresponds to the encoding device illustrated in Fig. 2.
Fig. 9 is a flowchart that illustrates a decoding
process performed by the decoding device illustrated in
Fig. 8.
25 Fig. 10 is a diagram that illustrates another
configuration example of the TSs that are generated by
the multiplexing unit illustrated in Fig. 2.
Fig. 11 is a diagram that illustrates a detailed
configuration example of the TS2 illustrated in Fig. 10.
30 Fig. 12 is a diagram that illustrates a detailed
example of a TS3 illustrated in Fig. 10.
8
•
5
10
15
20
25
30
SP308970WOOO
Fig. 13 is a flowchart that illustrates another
decoding process performed by the decoding device
illustrated in Fig. 8.
Fig. 14 is a block diagram that illustrates a
configuration example of an encoding device according to
a second embodiment of the present technology.
Fig. 15 is a diagram that illustrates a
configuration example of TSs that are generated by a
multiplexing unit illustrated in Fig. 14.
Fig. 16 is a flowchart that illustrates an encoding
process performed by the encoding device illustrated in
Fig. 14.
Fig. 17 is a flowchart that illustrates the
encoding process performed by the encoding device
illustrated in Fig. 14.
Fig. 18 is a diagram that illustrates a
configuration example of a decoding device that
corresponds to the encoding device illustrated in Fig. 14.
Fig. 19 is a flowchart that illustrates a decoding
process performed by the decoding device illustrated in
Fig. 18.
Fig. 20 is a diagram that illustrates another
configuration example of the TSs that are generated by
the multiplexing unit illustrated in Fig. 14.
Fig. 21 is a flowchart that illustrates another
decoding process performed by the decoding device
illustrated in Fig. 18.
Fig. 22 is a block diagram that illustrates a
configuration example of an encoding device according to
a third embodiment of the present technology.
Fig. 23 is a diagram that illustrates a first
9
SP308970WOOO
configuration example of TSs that are generated by a
multiplexing unit illustrated in Fig. 22.
Fig. 24 is a flowchart that illustrates an encoding
process performed by the encoding device illustrated in
5 Fig. 22.
Fig. 25 is a flowchart that illustrates the
encoding process performed by the encoding device
illustrated in Fig. 22.
Fig. 26 is a diagram that illustrates a
10 configuration example of a decoding device that
corresponds to the encoding device illustrated in Fig. 22.
Fig. 27 is a flowchart that illustrates a decoding
process performed by the decoding device illustrated in
Fig. 26.
15 Fig. 28 is a diagram that illustrates a second
configuration example of the TSs that are generated by
the multiplexing unit illustrated in Fig. 22.
Fig. 29 is a diagram that illustrates an example of
a method of multiplexing the TSI to TS3 illustrated in
20 Fig. 28.
Fig. 30 is a flowchart that illustrates a second
decoding process performed by the decoding device
illustrated in Fig. 26.
Fig. 31 is a diagram that illustrates a third
25 configuration example of the TSs that are generated by
the multiplexing unit illustrated in Fig. 22.
Fig. 32 is a flowchart that illustrates a third
decoding process performed by the decoding device 200
illustrated in Fig. 26.
30 Fig. 33 is a block diagram that illustrates a
configuration example of an encoding device according to
10
SP308970WOOO
a fourth embodiment of the present technology.
Fig. 34 is a diagram that illustrates a
configuration example of TSs that are generated by a
multiplexing unit illustrated in Fig. 33.
5 Fig. 35 is a diagram that illustrates a detailed
configuration example of a TS2 illustrated in Fig. 34.
Fig. 36 is a diagram that illustrates a
configuration example of a decoding device that
corresponds to the encoding device illustrated in Fig. 33.
10 Fig. 37 is a block diagram that illustrates a
configuration example of an encoding device according to
a fifth embodiment of the present technology.
Fig. 38 is a diagram that illustrates a
configuration example of an encoding processing unit
15 illustrated in Fig. 37.
Fig. 39 is a diagram that illustrates the reference
relation in an encoding process performed by the encoding
unit illustrated in Fig. 38.
Fig. 40 is a diagram that illustrates configuration
20 examples of an NAL unit.
Fig. 41 is a diagram that illustrates a
configuration example of a bit stream.
Fig. 42 is a diagram that illustrates a
configuration example of SEI according to a 3DV mode.
25 Fig. 43 is a diagram that illustrates an example of
the description of parallax information.
Fig. 44 is a flowchart that illustrates a multiviewpoint
encoding process performed by the encoding
device illustrated in Fig. 37.
30 Fig. 45 is a flowchart that illustrates a detailed
encoding process of Step S257 represented in Fig. 44.
11
SP308970WOOO
Fig. 46 is a block diagram that illustrates a
configuration example of a decoding device according to
the AVC mode that corresponds to the encoding device
illustrated in Fig. 37.
5 Fig. 47 is a flowchart that illustrates a decoding
process performed by a decoding device illustrated in Fig.
46.
Fig. 48 is a block diagram that illustrates a
configuration example of the decoding device according to
10 the MVC mode that corresponds to the encoding device
illustrated in Fig. 37.
Fig. 49 is a flowchart that illustrates a decoding
process performed by the decoding device illustrated in
Fig. 48.
15 Fig. 50 is a block diagram that illustrates a
configuration example of a decoding device according to
the 3DV mode that corresponds to the encoding device
illustrated in Fig. 37.
Fig. 51 is a flowchart that illustrates a decoding
20 process performed by the decoding device 360 illustrated
in Fig. 50.
Fig. 52 is a diagram that illustrates encoded data
that is a decoding target.
Fig. 53 is a block diagram that illustrates a
25 configuration example of an encoding device according to
a sixth embodiment of the present technology.
Fig. 54 is a block diagram that illustrates a
configuration example of an encoding processing unit
illustrated in Fig. 53.
30 Fig. 55 is a diagram that illustrates a
configuration example of an NAL unit.
12
• SP308970WOOO
5
20
15
10
Fig. 56 is a diagram that illustrates a
configuration example of a bit stream.
Fig. 57 is a flowchart that illustrates a multiviewpoint
encoding process performed by the encoding
device illustrated in Fig. 53.
Fig. 58 is a flowchart that illustrates a detailed
encoding process of Step S369 represented in Fig. 57.
Fig. 59 is a block diagram that illustrates a
configuration example of a decoding device according to
the 3DV mode that corresponds to the encoding device
illustrated in Fig. 53.
Fig. 60 is a flowchart that illustrates a decoding
process performed by the decoding device illustrated in
Fig. 59.
Fig. 61 is a diagram that illustrates an example of
the description of parallax information.
Fig. 62 is a block diagram that illustrates a
configuration example of an encoding device according to
a seventh embodiment of the present technology.
Fig. 63 is a diagram that illustrates an example of
the description of compatibility information and parallax
image information.
Fig. 64 is a diagram that illustrates an example of
detailed description of the parallax image information
25 illustrated in Fig. 63.
Fig. 65 is a diagram that illustrates an example of
the description of compatibility information and parallax
image information included in an auxiliary stream.
Fig. 66 is a diagram that illustrates an example of
30 the detailed description of the parallax image
information illustrated in Fig. 65.
13
•
5
10
15
20
25
30
SP308970WOOO
Fig. 67 is a flowchart that illustrates an encoding
process performed by the encoding device illustrated in
Fig. 62.
Fig. 68 is a flowchart that illustrates the
encoding process performed by the encoding device
illustrated in Fig. 62.
Fig. 69 is a diagram that illustrates a
configuration example of a decoding device that
corresponds to the encoding device illustrated in Fig. 62.
Fig. 70 is a flowchart that illustrates a decoding
process performed by the decoding device illustrated in
Fig. 69.
Fig. 71 is a block diagram that illustrates a
configuration example of an encoding device according to
an eighth embodiment of the present technology.
Fig. 72 is a flowchart that illustrates an encoding
process performed by the encoding device illustrated in
Fig. 71.
Fig. 73 is a flowchart that illustrates the
encoding process performed by the encoding device
illustrated in Fig. 71.
Fig. 74 is a diagram that illustrates a
configuration example of a decoding device that
corresponds to the encoding device illustrated in Fig. 71.
Fig. 75 is a flowchart that illustrates a decoding
process performed by the decoding device illustrated in
Fig. 74.
Fig. 76 is a diagram that illustrates an example of
a multiplexing pattern of an encoding target.
Fig. 77 is a diagram that illustrates features of
the effect of multiplexing.
14
•
5
10
15
20
25
30
SP308970WOOO
Fig. 78 is a diagram that illustrates another
example of the multiplexing pattern of the encoding
target.
Fig. 79 is a diagram that illustrates further
another example of the multiplexing pattern of an
encoding target.
Fig. 80 is a block diagram that illustrates a
configuration example of an encoding device according to
a ninth embodiment of the present technology.
Fig. 81 is a flowchart that illustrates an encoding
process performed by the encoding device illustrated in
Fig. 80.
Fig. 82 is a flowchart that illustrates the
encoding process performed by the encoding device
illustrated in Fig. 80.
Fig. 83 is a diagram that illustrates a
configuration example of a decoding device that
corresponds to the encoding device illustrated in Fig. 80.
Fig. 84 is a flowchart that illustrates the
decoding process performed by the decoding device
illustrated in Fig. 83.
Fig. 85 is another configuration example of the bit
stream.
Fig. 86 is a diagram that illustrates an example of
the description of Subset SPS according to a 3DV mode for
a parallax image illustrated in Fig. 85.
Fig. 87 is a diagram that illustrates an example of
the description of extension information for a parallax
image that is illustrated in Fig. 86.
Fig. 88 is a diagram that illustrates an example of
the description of VUl extension information for a
15
•
5
10
15
20
25
30
SP308970WOOO
parallax image illustrated in Fig. 86.
Fig. 89 is a diagram that illustrates an example of
the description of an NAL header of encoded data
according to the 3DV mode.
Fig. 90 is a diagram that illustrates an example of
the description of parallax image header extension
information illustrated in Fig. 89.
Fig. 91 is a diagram that illustrates an example of
the description of SEl according to the 3DV mode that is
illustrated in Fig. 85.
Fig. 92 is a diagram that illustrates further
another configuration example of the bit stream.
Fig. 93 is a diagram that illustrates an example of
the description of Subset SPS according to the 3DV mode
for an image that is illustrated in Fig. 92.
Fig. 94 is a diagram that illustrates a
configuration example of a bit stream in a case where the
3DV mode is a mode that is compliant with an HEVC mode.
Fig. 95 is a diagram that illustrates an example of
the description of an SPS illustrated in Fig. 94.
Fig. 96 is a diagram that illustrates an example of
the description of the Subset SPS illustrated in Fig. 95.
Fig. 97 is a diagram that illustrates an example of
the description of VUl information of the Subset SPS
illustrated in Fig. 96.
Fig. 98 is a diagram that illustrates an example of
the description of SEl according to the 3DV mode that is
illustrated in Fig. 94.
Fig. 99 is a diagram that illustrates a
configuration example of a computer according to an
embodiment.
16
SP308970WOOO
Fig. 100 is a diagram that illustrates a schematic
configuration of a television apparatus according to the
present technology.
Fig. 101 is a diagram that illustrates a schematic
5 configuration of a cellular phone according to the
present technology.
Fig. 102 is a diagram that illustrates a schematic
configuration of a recording and reproducing device
according to the present technology.
10 Fig. 103 is a diagram that illustrates a schematic
configuration of an imaging apparatus according to the
present technology.
MODE FOR CARRYING OUT THE INVENTION
15 [0026]

[Configuration Example of Encoding Device According to
Embodiment]
Fig. 2 is a block diagram that illustrates a
20 configuration example of an encoding device according to
a first embodiment of the present technology.
[0027]
The encoding device 50 illustrated in Fig. 2 is
configured by imaging units 51A to 51C, an image
25 converting unit 52, a parallax image generating unit 53,
an image information generating unit 54, a compatibility
information generating unit 55, an inter-viewpoint
distance information generating unit 56, a parallax image
information generating unit 57, an encoder 58, and a
30 multiplexing unit 59.
[0028]
17
SP308970WOOO
The encoding device 50 independently generates TSs
by performing encoding with an image of one viewpoint out
of multi-viewpoint images used as a 20 image in
accordance with an existing encoding mode, thereby
5 securing the compatibility with an existing encoding
device that encodes a 20 image.
[0029]
Hereinafter, out of multi-viewpoint images, an
image that is encoded in an existing encoding mode so as
10 to secure the compatibility with an existing encoding
device will be referred to as a compatible image, and an
image that is used for generating images of viewpoints
more than the number of viewpoints of a compatible image
using the compatible image will be referred to as an
15 auxiliary image.
[0030]
In the encoding device 50, the imaging unit 51A
captures an HO (high definition) image of a predetermined
viewpoint as an image A1 and supplies the captured image
20 to the image converting unit 52, the parallax image
generating unit 53, and the inter-viewpoint distance
information generating unit 56. The imaging unit 51B
captures an HO image of a viewpoint that is different
from the viewpoint of the image A1 as an image B1 at a
25 position that is separate from the imaging unit 51A by a
distance dd1~ in a horizontal direction, which has the
same distance to a subject in the depth direction, and
supplies the captured image to the image converting unit
52, the parallax image generating unit 53, and the inter-
30 viewpoint distance information generating unit 56. The
imaging unit 51C captures an HO image of a viewpoint
18
SP308970WOOO
different from the viewpoints of the images Al and Bl as
an image Cl at a position separate from the imaging unit
51A by a distance AdlAc in the horizontal direction that
is opposite to the imaging unit SIB and supplies the
5 captured image to the image converting unit 52, the
parallax image generating unit 53, and the interviewpoint
distance information generating unit 56.
[0031]
The viewpoints that correspond to the images Bl and
10 Cl are viewpoints that are located on the outer side out
of viewpoints of images that can be recognized as 3D
images. In this way, a decoding device that corresponds
to the encoding device 50 can generate multi-viewpoint
images by interpolating images of viewpoints located on
15 the further inner side than the viewpoints of the images
Bl and Cl by using the images Al to Cl. As a result, the
multi-viewpoint images can be generated with precision
that is higher than that of a case where images of
viewpoints located on the outer side are interpolated
20 using images of viewpoints located on the inner side.
The distances Adl~ and AdlAc may be configured to be
either fixed or changed each time.
[0032]
The image converting unit 52 determines the image
25 AI, which is supplied from the imaging unit 51A of which
the position in the horizontal direction is located on
the inner side out of the imaging units 51A to SIC, as a
compatible image. The image converting unit 52 supplies
information that designates the image Al as a compatible
30 image to the compatibility information generating unit 55.
Then, the image converting unit 52 directly supplies the
19
SP308970WOOO
image Al that is the compatible image to the encoder 58.
[0033]
In addition, the image converting unit 52 sets the
images Bl and Cl other than the image Al as auxiliary
5 images and multiplexes the images Bl and Cl in accordance
with a predetermined multiplexing mode. More
specifically, for example, in a case where the
multiplexing mode is a side-by-side mode, the image
converting unit 52 halves the resolution of each one of
10 the images Bl and Cl. Then, the image converting unit 52
multiplexes the image Bl (hereinafter, referred to as a
half-resolution image Bll of which the resolution has
halved and the image Cl (hereinafter, referred to as a
half-resolution image Cll of which the resolution has
15 halved such that the half-resolution image Bl becomes a
left-half image of the screen, and the half-resolution
image Cl becomes a right-half image of the screen. The
image converting unit 52 supplies a multiplexed image
that is acquired as a result of the multiplexing process
20 to the encoder 58 and supplies information that
represents a multiplexing mode of the auxiliary images to
the image information generating unit 54.
[0034]
The parallax image generating unit 53 detects the
25 disparity of each pixel of the images Al to Cl using the
images Al to Cl that are supplied from the imaging units
51A to 51C. The parallax image generating unit 53
generates a parallax image Al' that represents the
disparity of each pixel of the image Al that is a
30 compatible image and directly supplies the generated
parallax image to the encoder 58. In addition, the
20
SP308970WOOO
parallax image generating unit 53 generates a parallax
image B1' that represents the disparity of each pixel of
the image B1 that is an auxiliary image and a parallax
image e1' that represents the disparity of each pixel of
5 the image e1 that is an auxiliary image and multiplexes
the generated parallax images in accordance with a
predetermined multiplexing mode. The parallax image
generating unit 53 supplies a multiplexed image that is
acquired as a result of the multiplexing process to the
10 encoder 58. In addition, the parallax image generating
unit 53 supplies information that represents the
multiplexing mode of the parallax images of the auxiliary
images to the parallax image information generating unit
57.
15 [0035]
The image information generating unit 54 generates
information that represents the multiplexing mode of the
auxiliary images and the like as image information, which
is information relating to a compatible image and
20 auxiliary images, based on the information that is
supplied from the image converting unit 52 and supplies
the generated image information to the multiplexing unit
59.
[0036]
25 The compatibility information generating unit 55
generates information designating the compatible image, a
compatible mode, and the like as compatibility
information, which is information relating to' the
compatibility, based on the information supplied from the
30 image converting unit 52 and supplies the generated
compatibility information to the multiplexing unit 59.
21
, SP308970WOOO
[0037]
Here, the compatible mode is a mode that represents
a method of encoding the compatible image. As examples
of the compatible mode, there are a mono mode that
5 represents an encoding method in which a compatible image
of one viewpoint is encoded in accordance with the AVC
mode, a frame packing mode that represents an encoding
method in which compatible images of two viewpoints are
multiplexed and encoded in accordance with the AVC mode;
10 and a stereo mode that represents an encoding method in
which compatible images of two viewpoints are encoded in
accordance with the MVC mode.
[0038]
The inter-viewpoint distance information generating
15 unit 56 detects each inter-viewpoint distance
(hereinafter, referred to as an inter-viewpoint distance)
between two images out of the images A1 to C1 using the
images A1 to C1 that are supplied from the imaging units
51A to 51C. For example, the inter-viewpoint distance
20 information generating unit 56 detects the distance ~d1~
between the imaging units 51A and 51B in the horizontal
direction and the distance ~d1Ac between the imaging
units 51A and 51C in the horizontal direction as interviewpoint
distances. The inter-viewpoint distance
25 information generating unit 56 generates information that
represents the inter-viewpoint distances and the like as
inter-viewpoint distance information, which is
information relating to the inter-viewpoint distance
information, and supplies the generated inter-viewpoint
30 distance information to the multiplexing unit 59.
[0039]
22
SP308970WOOO
The parallax image information generating unit 57
generates information that represents the multiplexing
mode of the parallax images of the auxiliary images and
the like as parallax image information, which is
5 information relating to the parallax images, based on the
information that is supplied from the parallax image
generating unit 53 and supplies the generated parallax
image information to the multiplexing unit 59.
[0040]
10 The encoder 58 is configured by a compatible
encoder 61 and an auxiliary encoder 62. The compatible
encoder 61 (compatible image encoding unit) designates
the image Al that is the compatible image from the
multiplexed image of the compatible image and the
15 auxiliary images supplied from the image converting unit
52 and encodes the image Al in units of access units in
accordance with the existing Ave mode. The compatible
encoder 61 supplies an encoded stream that is acquired as
a result thereof to the multiplexing unit 59 as a
20 compatible stream (first encoded stream) .
[0041]
The auxiliary encoder 62 (auxiliary image encoding
unit) encodes the multiplexed image of the auxiliary
images that is supplied from the image converting unit 52
25 and the multiplexed images of the parallax images Al' of
the compatible images and the parallax images of the
auxiliary images that are supplied from the parallax
image generating unit 53 in units of access units in
accordance with a predetermined mode. The auxiliary
30 encoder 62 supplies encoded streams (a second encoded
stream, a first parallax encoded stream, and a second
23
SP308970WOOO
parallax encoded stream) acquired as a result thereof to
the multiplexing unit 59 as auxiliary streams. As an
encoding mode employed by the auxiliary encoder 62, the
AVC mode, the MVC mode, an MPEG2 (Moving Picture Experts
5 Group phase 2), or the like can be used.
[0042]
The multiplexing unit 59 (a setting unit and a
transmission unit) generates a TS using the compatible
stream supplied from the compatible encoder 61, the
10 auxiliary streams supplied from the auxiliary encoder 62,
the image information supplied from the image information
generating unit 54, the compatibility information
supplied from compatibility information generating unit
55, the inter-viewpoint distance information supplied
15 from the inter-viewpoint distance information generating
unit 56, the parallax image information supplied from the
parallax image information generating unit 57, and the
like. The multiplexing unit 59 multiplexes the generated
TS and transmits a multiplexed stream that is acquired as
20 a result thereof.
[0043]
Hereinafter, the image information, the
compatibility information, the inter-viewpoint distance
information, and the parallax image information will be
25 collectively referred to as auxiliary information.
[0044]
[Configuration Example of TS]
Fig. 3 is a diagram that illustrates a
configuration example of the TSs that are generated by
30 the multiplexing unit 59 illustrated in Fig. 2.
[0045 ]
24
SP308970WOOO
In the example illustrated in Fig. 3, a TSl is
generated from the compatible stream by the multiplexing
unit 59. In addition, a TS2 is generated from the
auxiliary stream that includes the multiplexed image of
5 the encoded auxiliary images, the parallax image A' of
the compatible image, and the multiplexed image of the
parallax images of the auxiliary images and the auxiliary
information.
[0046]
10 In the example illustrated in Fig. 3, since the
compatible stream and the stream other than the
compatible stream are stored in TSs different from each
other, the amount of the information of the compatible
stream does not need to be reduced. Accordingly, the
15 image quality of the compatible image can be configured
to be equal to the image quality of the 2D image that is
encoded in accordance with the existing AVC method.
[0047]
Fig. 4 is a diagram that illustrates a detailed
20 configuration example of the TS2 illustrated in Fig. 3.
[0048]
As illustrated in A of Fig. 4 to C of Fig. 4, in
the TS2 illustrated in Fig. 3, data is arranged in units
of units, and, at the head of each unit, a delimiter
25 (Del) (boundary information) that represents the
separation (boundary) of the unit is inserted.
[0049]
In the example illustrated in A of Fig. 4, a
multiplexed image (Bl+Cl) of encoded auxiliary images and
30 auxiliary information (Aux Inf) of the multiplexed image
of the auxiliary images, a parallax image (Al') of an
25
SP308970WOOO
encoded compatible image and auxiliary information of the
parallax image of the compatible image, or a multiplexed
image (B1'+C1') of parallax images of encoded auxiliary
images and auxiliary information of the multiplexed image
5 of the parallax images of the auxiliary images arranged
in units of access units that are decoding units are
arranged in each unit as data.
[0050]
In such a case, a decoding device that receives the
10 TS2 can independently extract the multiplexed image of
the encoded auxiliary images, the parallax image of the
compatible image, and the multiplexed image of the
parallax images of the auxiliary images, which are
arranged in units of access units, by extracting data in
15 units of the units. As a result, for example, in a case
where the decoding device generates multi-viewpoint
images using only the compatible image and the parallax
image of the compatible image, images that are used for
generating multi-viewpoint images can be easily extracted.
20 In addition, since each image of the multiplexed image of
the auxiliary images, the parallax image of the
compatible image, and the multiplexed image of the
parallax images of the auxiliary images and the auxiliary
information of the image are arranged in the same unit,
25 an image that is arranged in each unit can be
independently processed in an easy manner.
[0051]
In the example illustrated in B of Fig. 4, a
multiplexed image of encoded auxiliary images, a parallax
30 image of a compatible image, a multiplexed image of
parallax images of the auxiliary images, and auxiliary
26
SP308970WOOO
information, which are arranged in units of access units,
are arranged altogether in each unit. In such a case, a
decoding device that receives the TS2 can extract the
multiplexed image of the encoded auxiliary images, the
5 parallax image of the compatible image, the multiplexed
image of the parallax images of the auxiliary images, and
the auxiliary information, which are arranged in units of
access units, altogether by extracting data in units of
units.
10 [0052]
In the example illustrated in C of Fig. 4, a
multiplexed image of encoded auxiliary images and
auxiliary information of the multiplexed image of the
auxiliary images or a parallax image of an encoded
15 compatible image, a multiplexed image of parallax images
of the auxiliary images, and auxiliary information of the
parallax image of the compatible image and the parallax
image of the auxiliary images, which are arranged in
units of access units, are arranged in each unit.
20 [0053]
In such a case, the decoding device that receives
the TS2 can independently extract the multiplexed image
of the encoded auxiliary images, the parallax image of
the compatible image, and the multiplexed image of the
25 parallax images of the auxiliary images, which are
arranged in units of access units, by extracting data in
units of the units. As a result, for example, in a case
where the decoder for the image and the decoder for the
parallax image are separately prepared in the decoding
30 device, data can be efficiently supplied to each decoder
Furthermore, an image and a parallax image can be
27
SP308970WOOO
independently processed by the decoding device in an easy
manner.
[0054]
[Description of Multiplexing Method]
5 Fig. 5 is a diagram that illustrates an example of
the method of multiplexing the TS1 and the TS2.
[0055]
As illustrated in Fig. 5, the TS1 and TS2 are
multiplexed by being arranged from the head in order of
10 the TS2 and the TS1 in units of access units. In
addition, a 3DV representation delimiter (boundary
information) representing the head of the TS that
includes information other than a compatible stream is
set and added to the head of the TS2. In other words,
15 the 3DV representation delimiter is arranged on the
boundary between a TS2 of a specific access unit and a
TS1 of the previous access unit. Accordingly, a decoding
device that corresponds to the encoding device 50 can
easily recognize a compatible stream, and an auxiliary
20 stream and an auxiliary image synchronized therewith by
extracting data from a specific 3DV representation
delimiter to a next 3DV representation delimiter.
[0056]
[Description of Process of Encoding Device]
25 Figs. 6 and 7 are flowcharts that illustrate an
encoding process that is performed by the encoding device
50 illustrated in Fig. 2. This encoding process, for
example, is started when the images A1 to C1 are output
from the imaging units 51A to 51C.
30 [0057]
In Step Sll illustrated in Fig. 6, the inter-
28
• SP308970WOOO
viewpoint distance information generating unit 56 detects
distances ~d1~ and ~d1Ac as inter~viewpoint distances
using the images A1 to C1 that are supplied from the
imaging units 51A to 51C.
5 [0058]
In Step S12, the inter-viewpoint distance
information generating unit 56 generates information that
represents the inter-viewpoint distances detected in Step
Sll and the like as inter-viewpoint distance information
10 and inputs the generated inter-viewpoint distance
information to the multiplexing unit 59.
[0059]
In Step S13, the image converting unit 52
determines the image A1 that is supplied from the imaging
15 unit 51A, of which the position in the horizontal
direction is located on the inner side out of the imaging
units 51A to 51C, as a compatible image and determines a
multiplexing mode of auxiliary images. The image
converting unit 52 supplies information that designates
20 the image A1 as a compatible image to the compatibility
information generating unit 55 and supplies the
multiplexing mode of the auxiliary images to the image
information generating unit 54.
[0060]
25 In Step S14, the compatibility information
generating unit 55 generates the information designating
the image A1 as a compatible image, a mono mode as a
compatible mode, and the like as compatible information
based on the information that is supplied from the image
30 converting unit 52 and inputs the generated compatibility
information to the multiplexing unit 59.
29
• SP308970WOOO
[0061]
In Step S15, the image information generating unit
54 generates information that represents the multiplexing
mode of auxiliary images and the like as image
5 information based on the information that is supplied
from the image converting unit 52 and inputs the
generated image information to the multiplexing unit 59.
[0062]
In Step S16, the image converting unit 52 sets
10 images B1 and e1 other than the image Al as auxiliary
images and multiplexes the auxiliary images based on the
multiplexing mode of auxiliary images that is determined
in Step S13, thereby acquiring a multiplexed image of the
auxiliary images.
15 [0063]
In Step S17, the image converting unit 52 inputs
the multiplexed image of the image A1, which is a
compatible image, and the auxiliary image to the encoder
58.
20 [0064]
In Step S18 illustrated in Fig. 7, the parallax
image generating unit 53 detects the disparity of each
pixel of the images A1 to e1 using the images A1 to e1
that are supplied from the imaging units 51A to 51e and
25 generates parallax images A1' to e1'.
[0065]
In Step S19, the parallax image generating unit 53
determines a multiplexing mode of the parallax images of
auxiliary images and supplies information that represents
30 the multiplexing mode to the parallax image information
generating unit 57.
30
• SP308970WOOO
[0066]
In Step S20, the parallax image information
generating unit 57 generates information that represents
the multiplexing mode of the parallax images of auxiliary
5 images and the like as parallax image information based
on the information that is supplied from the parallax
image generating unit 53 and inputs the generated
parallax image information to the multiplexing unit 59.
[0067]
10 In Step S21, the parallax image generating unit 53
multiplexes the parallax images of the auxiliary images
based on the multiplexing mode of the parallax images of
the auxiliary images that is determined in Step S19,
thereby acquiring a multiplexed image of the parallax
15 images of the auxiliary images.
[0068]
In Step S22, the parallax image generating unit 53
inputs the parallax image A1' of the compatible image and
the multiplexed image of the parallax images of the
20 auxiliary images to the encoder 58.
[0069]
In Step S23, the compatible encoder 61 of the
encoder 58 encodes the image A1 that is a compatible
image supplied from the image converting unit 52 in
25 accordance with the existing Ave mode and supplies an
encoded stream acquired as a result thereof to the
multiplexing unit 59 as a compatible stream.
[0070]
In Step S24, the auxiliary encoder 62 encodes the
30 multiplexed image of the auxiliary images that is
supplied from the image converting unit 52, the parallax
31
5
15
10
• SP308970WOOO
image A1' of the compatible image that is supplied from
the parallax image generating unit 53, and the
multiplexed image of the parallax images of the auxiliary
images in accordance with a predetermined mode. The
auxiliary encoder 62 supplies an encoded stream acquired
as a result of the encoding process to the multiplexing
unit 59 as an auxiliary stream.
[0071]
In Step S25, the multiplexing unit 59 generates a
TS1 from the compatible stream that is supplied from the
compatible encoder 61, generates a TS2 from the auxiliary
stream and the auxiliary information supplied from the
auxiliary encoder 62, performs a multiplexing process,
and transmits a multiplexed stream acquired as a result
thereof. This multiplexed stream, for example, is
recorded on a BD (Blu-Ray (registered trademark) Disc) or
the like or is transmitted as a broadcast stream. Then,
the process ends.
[0072]
20 As above, since the encoding device 50 performs the
multiplexing process with the compatible stream and the
auxiliary stream and the auxiliary information being
stored in TSs different from each another, the
compatibility with an encoding device that encodes an
25 existing 2D image in multiplexing can be secured. In
other words, the encoding device 50 can perform a
multiplexing process for multi-viewpoint images in
accordance with a mode that has the compatibility with an
existing mode.
30 [0073]
In addition, since the encoding device 50 encodes
32
SP308970WOOO
an image of one viewpoint out of multi-viewpoint images
as a compatible image in accordance with an existing
encoding mode, the compatibility with an existing
encoding device that encodes a 2D image in the encoding
5 process can be secured. In other words, the encoding
device 50 can perform encoding for multi-viewpoint images
in accordance with a mode that has compatibility with an
existing encoding mode.
[0074]
10 Furthermore, since the encoding device 50 encodes
the images Al to Cl of three viewpoints, a decoding
device that corresponds to the encoding device 50 can
generate multi-viewpoint images from the images Al to Cl
of three viewpoints. In this way, according to the
15 decoding device, compared to a case where multi-viewpoint
images are generated from images of two viewpoints, the
viewpoints of images that can be generated are not
limited, and multi-viewpoint images having relatively
high precision can be generated.
20 [0075]
In addition, since the encoding device 50 encodes
images with the resolution of the auxiliary images being
lowered, the processing cost of the encoding process and
the decoding process is lower than that of a case where
25 encoding is performed without lowering the resolution.
For example, in a case where auxiliary images of two
viewpoints are encoded without lowering the resolution
thereof, while the processing cost of the encoding
process and the decoding process is the same as that of
30 an encoding process and a decoding process for two HD
images, the processing cost of the encoding process for
33
SP308970WOOO
auxiliary images of two viewpoints, which is performed by
the encoding device 50, is the same as the processing
cost of the encoding process or the decoding process for
one HD image. As a result, it can be prevented that the
5 performance of the decoding process performed by the
decoding device greatly affects the image quality of
multi-viewpoint images.
[0076]
Furthermore, when multi-viewpoint images are
10 synthesized, as will be described later, a decoding
device that corresponds to the encoding device 50 lowers
the resolution at the rate of the reciprocal of the
number of viewpoints of the multi-viewpoint images,
whereby the lowering of the resolution of the auxiliary
15 images that is performed by the encoding device 50 does
not affect the image quality of the multi-viewpoint
images after the synthesis.
[0077]
In addition, since the encoding device 50 performs
20 encoding with the resolution of auxiliary images being
halved and encodes the parallax images of the auxiliary
images while being halved, the amount of information of
an encoding target can be configured to be approximately
an amount that corresponds to 4 HD images (1080i x 4) in
25 the Ave mode.
[0078]
Here, in comprehensive view of the processing speed,
the processing performance with respect to power
consumption, a transmission data rate, a transmission
30 bandwidth, a bandwidth of a memory, a memory access speed,
and the like of a current display device that displays
34
SP308970WOOO
multi-viewpoint images and a decoding device according
thereto, a value that is adequate as the amount of
information that can be processed by the decoding device
is considered to be about two times an amount that
5 corresponds to an HD image in the current state of the
Mve mode, that is, an amount that corresponds to 4 HD
images in the Ave mode. Accordingly, a decoding device
(decoding method) that corresponds to the encoding device
50 of which the amount of information of the encoding
10 target is about an amount that corresponds to 4 HD images
(1080i x 4) in the Ave mode can be realized at a
reasonable processing cost by a reasonable approach.
[0079]
In addition, according to the encoding device 50,
15 the amount of information of the encoding target is
reduced to an amount that corresponds to about 4 HD
images (1080i x 4) in the Ave mode, and accordingly, the
encoding device can be easily operated as a BD or a
broadcast application that has a limitation of a usable
20 bandwidth.
[0080]
Furthermore, since the encoding device 50 generates
parallax images and transmits the parallax images while
being included in an encoded stream, a decoding device
25 that corresponds to the encoding device 50 does not need
to generate a parallax image for generating multiviewpoint
images, and accordingly, the processing load of
the decoding device can be reduced. As a result, the
manufacturing cost of the decoding device can be reduced.
30 In addition, it can be prevented that the parallax
detecting performance of the decoding device greatly
35
SP308970WOOO
affects the image quality of multi-viewpoint images.
[0081]
[Configuration Example of Decoding Device]
Fig. 8 is a diagram that illustrates a
5 configuration example of a decoding device that decodes a
multiplexed stream that is transmitted from the encoding
device 50 illustrated in Fig. 2.
[0082]
The decoding device 120 illustrated in Fig. 8 is
10 configured by a separation unit 121, a decoder 122, an
image information acquiring unit 123, an inter-viewpoint
distance information acquiring unit 124, a parallax image
information acquiring unit 125, a compatibility
information acquiring unit 126, and an image generating
15 unit 127. The decoding device 120 separates and decodes
a multiplexed stream that is transmitted from the
encoding device 50, generates an image of one viewpoint
or multi-viewpoint images, and displays the generated
image on a display device that is not illustrated in the
20 figure.
[0083]
More specifically, the separation unit 121
(separation unit) of the decoding device 120 receives a
multiplexed stream that is transmitted from the encoding
25 device 50 and separates each TS. The separation unit 121
extracts a compatible stream included in the TS and an
auxiliary stream included in the TS and supplies
extracted streams to the decoder 122. In addition, the
separation unit 121 extracts auxiliary information
30 included in the TS, supplies image information included
in the auxiliary information to the image information
36
SP308970WOOO
acquiring unit 123, and supplies yiewpoint distance
information to the inter-viewpoint distance information
acquiring unit 124. Furthermore, the separation unit 121
supplies parallax image information included in the
5 auxiliary information to the parallax image information
acquiring unit 125 and supplies compatibility information
to the compatibility information acquiring unit 126.
[0084]
The decoder 122 is configured by a compatible
10 decoder 131 and an auxiliary decoder 132. The compatible
decoder 131 (compatible image decoding unit) of the
decoder 122 decodes a encoded compatible image that is
included in the compatible stream supplied from the
separation unit 121 in accordance with a mode that
15 corresponds to the Ave mode and supplies the decoded
compatible image to the image generating unit 127.
[0085]
The auxiliary decoder 132 (auxiliary image decoding
unit) decodes a multiplexed image of auxiliary images
20 included in the auxiliary stream that is supplied from
the separation unit 121, a parallax image of a compatible
image, and a multiplexed image of parallax images of the
auxiliary images in accordance with a mode that
corresponds to the auxiliary encoder 62 illustrated in
25 Fig. 2. The auxiliary decoder 132 supplies the
multiplexed image of the auxiliary images, the parallax
image A' of the compatible image, and the multiplexed
image of the parallax images of the auxiliary images,
which are acquired as a result of the decoding process,
30 to the image generating unit 127.
[0086]
37
SP308970WOOO
The image information acquiring unit 123 acquires
the image information that is supplied from the
separation unit 121 and supplies the acquired image
information to the image generating unit 127. The inter-
5 viewpoint distance information acquiring unit 124
acquires the inter-viewpoint distance information that is
supplied from the separation unit 121 and supplies the
acquired inter-viewpoint distance information to the
image generating unit 127.
10 [0087]
The parallax image information acquiring unit 125
acquires the parallax image information that is supplied
from the separation unit 121 and supplies the acquired
parallax image information to the image generating unit
15 127. The compatibility information acquiring unit 126
acquires the compatibility information that is supplied
from the separation unit 121 and supplies the acquired
compatibility information to the image generating unit
127.
20 [0088]
The image generating unit 127 outputs an image in
accordance with an instruction supplied from a viewer and
displays the image on a display device not illustrated in
the figure. More specifically, the image generating unit
25 127 (generation unit) generates images of three or more
viewpoints, which correspond to a display device not
illustrated in the figure, each having the resolution
that is a half of the resolution of the compatible image
or the auxiliary image by using the compatible image, the
30 multiplexed image of the auxiliary images, the parallax
image of the compatible image, and the multiplexed image
38
SP308970WOOO
of the parallax images of the auxiliary images in
accordance with a viewer's instruction for displaying a
3D image of multi-viewpoints based on the image
information supplied from the image information acquiring
5 unit 123, the inter-viewpoint distance information
supplied from the inter-viewpoint distance information
acquiring unit 124, the parallax image information
supplied from the parallax image information acquiring
unit 125, the compatibility information supplied from the
10 compatibility information acquiring unit 126, and the
like.
[0089]
More specifically, the image generating unit 127
separates the parallax image of each auxiliary image from
15 the multiplexed image of the parallax images of the
auxiliary images based on the information that represents
a multiplexing mode of the parallax images of the
auxiliary images that is included in the parallax image
information supplied from the parallax image information
20 acquiring unit 125. In addition, the image generating
unit 127 separates each auxiliary image from the
multiplexed image of the auxiliary images based on the
information that represents a multiplexing mode of
auxiliary images that is included in the image
25 information supplied from the image information acquiring
unit 123.
[0090]
Furthermore, the image generating unit 127
determines the position of each viewpoint of the multi30
viewpoint images to be generated based on the interviewpoint
distance information and the number of
39
SP308970WOOO
viewpoints that corresponds to a display device not
illustrated in the figure. Then, the image generating
unit 127 generates the image of each viewpoint of which
the position is determined by using the compatible image,
5 each auxiliary image, the parallax image of the
compatible image, and the parallax image of each
auxiliary images. Then, the image generating unit 127
converts the resolution of the generated image of each
viewpoint into resolution that is ~1/the number of
10 viewpoints" of the resolution of the compatible image or
the auxiliary image, synthesizes the images, and displays
the synthesized image on a display device not illustrated
in the figure.
[0091]
15 At this time, the multi-viewpoint images after the
synthesis are displayed such that the viewing angles are
different for each viewpoint, and a viewer can view a 3D
image without wearing glasses by seeing images of
arbitrary two viewpoints with his left and right eyes.
20 [0092]
In addition, the image generating unit 127 outputs
the image A1 that is the compatible image supplied from
the compatible decoder 131 of the decoder 122 in
accordance with a viewer's instruction for displaying a
25 2D image, thereby displaying the image on a display
device not illustrated in the figure. Accordingly, the
viewer can view the 2D image.
[0093]
[Description of Process of Decoding Device]
30 Fig. 9 is a flowchart that illustrates a decoding
process performed by the decoding device 120 illustrated
40
5
10
15
20
25
30
SP308970WOOO
in Fig. 8. This decoding process, for example, is
started when a multiplexed stream that is transmitted
from the encoding device 50 illustrated in Fig. 2 is
input to the decoding device 120.
[0094]
In Step 831 illustrated in Fig. 9, the image
generating unit 127 of the decoding device 120 determines
whether or not an instruction for displaying a 2D image
has been made by a viewer. In a case where it is
determined that the instruction for displaying a 2D image
has not been made by the viewer in Step S31, in other
words, in a case where an instruction for displaying a 3D
image of the multi-viewpoint mode has been made by the
viewer, the process proceeds to Step S32.
[0095]
In Step S32, the separation unit 121 receives the
multiplexed stream that is transmitted from the encoding
device 50 and separates TSI and TS2 from the multiplexed
stream. The separation unit 121 extracts a compatible
stream included in the TSI and an auxiliary stream
included in the TS2 by referring a delimiter and the like
and supplies the extracted streams to the decoder 122.
In addition, the separation unit 121 extracts auxiliary
information included in the TS2 by referring to the
delimiter and the like, supplies image information that
is included in the auxiliary information to the image
information acquiring unit 123, and supplies viewpoint
distance information to the inter-viewpoint distance
information acquiring unit 124. In addition, the
separation unit 121 supplies parallax image information
that is included in the auxiliary information to the
41
SP308970WOOO
parallax image information acquiring unit 125 and
supplies compatibility information to the compatibility
information acquiring unit 126.
[0096]
5 In Step S33, the compatible decoder 131 of the
decoder 122 extracts an encoded compatible image from the
compatible stream that is supplied from the separation
unit 121 and decodes the compatible image in accordance
with a mode that corresponds to the Ave mode. Then, the
10 compatible decoder 131 supplies an image Al that is
acquired as a result of the decoding process to the image
generating unit 127.
[0097]
In Step S34, the auxiliary decoder 132 extracts an
15 encoded multiplexed image of auxiliary images, a parallax
image A' of the compatible image, and a multiplexed image
of the parallax images of the auxiliary images from the
auxiliary stream and decodes the extracted images in
accordance with a mode that corresponds to the auxiliary
20 encoder 62 illustrated in Fig. 2. The auxiliary decoder
132 supplies the multiplexed image of the auxiliary
images, the parallax image A' of the compatible image,
and the multiplexed image of the parallax images of the
auxiliary images that are acquired as a result of the
25 decoding process to the image generating unit 127.
[0098]
In Step S35, the image information acquiring unit
123 acquires the image information that is supplied from
the separation unit 121 and inputs the image information
30 to the image generating unit 127. In Step S36, the
inter-viewpoint distance information acquiring unit 124
42
SP308970WOOO
acquires the inter-viewpoint distance information that is
supplied from the separation unit 121 and inputs the
inter-viewpoint distance information to the image
generating unit 127.
5 [0099]
In Step S37, the parallax image information
acquiring unit 125 acquires the parallax image
information that is supplied from the separation unit. 121
and inputs the parallax image information to the image
10 generating unit 127. In Step S38, the compatibility
information acquiring unit 126 acquires the compatibility
information that is supplied from the separation unit 121
and supplies the compatibility information to the image
generating unit 127.
15 [0100]
In Step S39, the image generating unit 127
determines the position of each viewpoint of a multiviewpoint
image to be generated based on the interviewpoint
distance information supplied from the inter-
20 viewpoint distance information acquiring unit 124 and the
number of viewpoints that corresponds to a display device
not illustrated in the figure. For example, in a case
where an inter-viewpoint distance that is included in the
inter-viewpoint distance information is short, the image
25 generating unit 127 also determines the position of a
viewpoint that is located on the outer side of the
viewpoints of images B1 and C1 as the position of a
viewpoint of the multi-viewpoint 3D image to be generated.
On the other hand, in a case where the inter-viewpoint
30 distance that is included in the inter-viewpoint distance
information is long, the image generating unit 127
43
SP308970WOOO
determines only positions of viewpoints located on the
inner side of the images B1 and C1 as the positions of
viewpoints of the multi-viewpoint 3D image to be
generated.
5 [0101]
In Step S40, the image generating unit 127
generates images of the viewpoints each having the
resolution that is a half of the resolution of the
compatible image or the auxiliary image based on the
10 position of each viewpoint determined in Step S39, the
image information supplied from the image information
acquiring unit 123, the parallax image information
supplied from the parallax image information acquiring
unit 125, the compatibility information supplied from the
15 compatibility information acquiring unit 126, and the
like by using the compatible image, the multiplexed image
of the auxiliary images, the parallax image of the
compatible image, and the multiplexed image of the
parallax images of the auxiliary images.
20 [0102]
In Step S41, the image generating unit 127 converts
the resolution of the image of each viewpoint that is
generated in Step S40 into resolution that is ~1/ the
number of viewpoints" of the resolution of the compatible
25 image or the auxiliary image and synthesizes the images
of each viewpoint after the conversion based on the
positions of the viewpoints.
[0103]
In Step S42, the image generating unit 127 outputs
30 the multi-viewpoint image after the synthesis that is
acquired by the process of Step S41 to a display device
44
• SP308970WOOO
that is not illustrated in the figure, thereby displaying
the multi-viewpoint image such that viewing angles are
different for each viewpoint. Then, the process ends.
[0104]
5 On the other hand, in a case where it is determined
that an instruction for displaying a 2D image has been
made by a viewer in Step S31, the separation unit 121
separates a TS1 from the multiplexed stream in Step S43.
More specifically, the separation unit 121 acquires the
10 TS1 other than a TS2, to which a 3DV representation
delimiter NAL unit is added, from the multiplexed stream.
Then, the separation unit 121 extracts a compatible
stream that is included in the TS1 by referring to the
delimiter and the like and supplies the extracted stream
15 to the decoder 122.
[0105]
In Step S44, the compatible decoder 131 of the
decoder 122 extracts an encoded compatible image from the
compatible stream that is supplied from the separation
20 unit 121 and decodes the encoded compatible image in
accordance with a mode that corresponds to the Ave mode.
The compatible decoder 131 supplies an image A that is
the compatible image acquired as a result of the decoding
process to the image generating unit 127.
25 [0106]
In Step S45, the image generating unit 127 outputs
an image A1 that is a compatible image supplied from the
compatible decoder 131 to a display device not
illustrated in the figure, thereby displaying the image.
30 Then, the process ends.
[0107]
45
•
5
10
15
SP308970WOOO
In a decoding device that can decode only a
compatible stream that has compatibility with the
decoding device 120, a TS2 to which the 3DV
representation delimiter NAL unit is added is ignored,
and the process of Steps S44 and S45 is performed.
[0108]
As above, the decoding device 120 can separate a
multiplexed stream that is multiplexed by the encoding
device 50 in accordance with a mode that has
compatibility with an existing mode. In addition, the
decoding device 120 can decode a compatible stream and an
auxiliary stream that are encoded in accordance with a
mode, which has compatibility with an existing mode and
requires relatively low process cost, by the encoding
device 50.
[0109]
Furthermore, since the decoding device 120
generates a multi-viewpoint image using a compatible
image of one viewpoint and an auxiliary image of two
20 viewpoints, the encoding device 50 may prepare only two
imaging units 51B and 51C in addition to the existing
imaging unit 51A that captures a compatible image of one
viewpoint as imaging units that generate multi-viewpoint
images. Accordingly, the installation of the imaging
25 units for generating multi-viewpoint images can be
performed in an easy manner at a low cost.
[0110]
[Another Configuration Example of TS]
Fig. 10 is a diagram that illustrates another
30 configuration example of the TSs that are generated by
the multiplexing unit 59 illustrated in Fig. 2.
46
5
10
15
20
25
30
SP308970WOOO
[0111]
In the example illustrated in Fig. 10, three TSs
are generated by the multiplexing unit 59, and an
auxiliary stream and auxiliary information are included
in TSs that are different from each other. More
specifically, a TS1 is generated from the compatible
stream, a TS2 is generated from the auxiliary stream, and
a TS3 is generated from the auxiliary information.
[0112]
In the example illustrated in Fig. 10, the TS3 is
independently generated only from the auxiliary
information of which the amount of information is
relatively small. Accordingly, a decoding device that
executes an application, in which there is a limitation
on the number of TSs that can be simultaneously processed,
such as a BD application can reproduce the TS1 and the
TS2 in synchronization with each other by preloading the
TS3 of which the amount of information is relatively
small.
[0113]
In addition, at the heads of the TS2 that is formed
from the auxiliary stream and the TS3 that is formed from
the auxiliary information, similarly to a case where the
auxiliary stream and the auxiliary information are
included in the same TS, a 3DV representation delimiter
illustrated in Fig. 5 is arranged.
[0114]
Fig. 11 is a diagram that illustrates a detailed
configuration example of the TS2 illustrated in Fig. 10.
[0115]
As illustrated in A of Fig. 11 to C of Fig. 11,
47
•
5
10
15
20
25
30
SP308970WOOO
data is arranged in units of units in the TS2 illustrated
in Fig. 10, and, at the head of each unit, a delimiter
that represents the separation of the unit is inserted.
[0116]
In the example illustrated in A of Fig. 11, in each
unit, a multiplexed image (B+C) of encoded auxiliary
images, a parallax image (A') of a compatible image, or a
multiplexed image (B'+C') of parallax images of auxiliary
images is arranged as data in units of access units.
[0117]
In such a case, a decoding device that receives the
TS2 can independently extract the multiplexed image of
the encoded auxiliary image, the parallax image of the
compatible image, and the multiplexed image of the
parallax images of the auxiliary images that are arranged
in units of access units by extracting data in units of
the units. As a result, for example, in a case where the
decoding device generates a multi-viewpoint image using
only a compatible image and a parallax image of the
compatible image, images used for generating the multiviewpoint
image can be easily extracted.
[0118]
In the example illustrated in B of Fig. 11, in each
unit, a multiplexed image of encoded auxiliary images, a
parallax image of a compatible image, and a multiplexed
image of parallax images of the auxiliary images are
arranged together in units of access units. In such a
case, a decoding device that receives the TS2 can extract
the multiplexed image of the encoded auxiliary images,
the parallax image of the compatible image, and the
multiplexed image of the parallax images of the auxiliary
48
5
• SP308970WOOO
images together that are arranged in units of access
units by extracting data in units of the units.
[0119]
In the example illustrated in C of Fig. 11, in each
unit, a multiplexed image of encoded auxiliary images or
a parallax image of an encoded compatible image and a
multiplexed image of parallax images of auxiliary images
are arranged together in units of access units.
[0120]
10 In such a case, a decoding device that receives the
TS2 can independently extract the multiplexed image of
the encoded auxiliary images, the parallax image of the
compatible image, and the multiplexed image of the
parallax images of the auxiliary images that are arranged
15 in units of access units by extracting data in units of
the units. As a result, for example, in a case where a
decoder used for an image and a decoder used for a
parallax image are separately prepared in the decoding
device, data can be efficiently supplied to each decoder.
20 In addition, an image and a parallax image can be
independently processed by the decoding device in an easy
manner.
[0121]
Fig. 12 is a diagram that illustrates a detailed
25 configuration example of the TS3 illustrated in Fig. 10.
[0122]
As illustrated in A of Fig. 12 to D of Fig. 12, in
the TS3 illustrated in Fig. 10, data is arranged in units
of units, and, at the head of each unit, a delimiter that
30 represents the separation of the unit is inserted.
[0123]
49
5
• SP308970WOOO
In the example illustrated in A of Fig. 12, in each
unit, auxiliary information (Aux Info(B+C)) of a
multiplexed image of auxiliary images, auxiliary
information (Aux Info(A')) of a parallax image of a
compatible image, or auxiliary information (Aux
Info(B'+C')) of a multiplexed image of parallax images of
auxiliary images is arranged as data in units of access
units.
[0124]
10 In such a case, a decoding device that receives the
TS2 can independently extract the auxiliary information
of the multiplexed image of the auxiliary images, the
auxiliary information of the parallax image of the
compatible image, and the auxiliary information of the
15 multiplexed image of the parallax images of the auxiliary
images, which are arranged in units of access units, by
extracting data in units of the units. As a result, the
multiplexed image of the encoded auxiliary images, the
parallax image of the compatible image, and the
20 multiplexed image of the parallax images of the auxiliary
images, which are arranged in units of the access units,
can be independently processed in an easy manner.
[0125]
In the example illustrated in B of Fig. 12, a
25 multiplexed image of auxiliary images, a parallax image
of a compatible image, and auxiliary information (Aux
Info (B+C, A', B'+C')) of a multiplexed image of parallax
images of auxiliary images, which are arranged in units
of access units, are arranged altogether in each unit.
30 In such a case, a decoding device that receives the TS2
can extract the multiplexed image of the auxiliary images,
50
SP308970WOOO
the parallax image of the compatible image, and the
auxiliary information of the multiplexed image of the
parallax images of the auxiliary images altogether which
are arranged in units of access units by extracting data
5 in units of units.
[0126]
In the example .illustrated in C of Fig. 12,
auxiliary information (Aux Info(B+C)) of a multiplexed
image of an auxiliary image or a parallax image of a
10 compatible image and auxiliary information (Aux Info (A' ,
B'+C')) of a multiplexed image of parallax images of
auxiliary images are arranged in each unit in units of
access units.
[0127]
15 In such a case, a decoding device that receives the
TS3 can independently extract the auxiliary information
of the multiplexed image of the auxiliary image, the
parallax image of the compatible image, and the auxiliary
information of the multiplexed image of the parallax
20 images of the auxiliary images, which are arranged in
units of access units, by extracting data in units of the
units. As a result, for example, in a case where a
decoder used for an image and a decoder used for a
parallax image are separately prepared in the decoding
25 device, the auxiliary information can be efficiently
supplied in synchronization with a decoding result.
[0128]
In the example illustrated in D of Fig. 12,
auxiliary information (Aux Info(All B+C)) of a
30 multiplexed image of auxiliary images corresponding to a
predetermined time (for example, two hours), auxiliary
51
5
10
15
20
25
30
SP308970WOOO
information (Aux Info(All A')) of a parallax image of a
compatible image, and auxiliary information (Aux Info(All
B'+C')) of a multiplexed image of parallax images of
auxiliary images are arranged in each unit. In such a
case, a decoding device that receives the TS3 can extract
and maintain the auxiliary information of the multiplexed
image of the auxiliary image corresponding to the
predetermined time, the auxiliary information of the
parallax image of the compatible image, and the auxiliary
information of the multiplexed image of the parallax
images of the auxiliary images together by extracting
data in units of units when the auxiliary information is
preloaded.
[0129]
[Description of Another Encoding Process]
The encoding process performed by the encoding
device 50 in a case where the TSs having the
configurations described with reference to Figs. 10 to 12
are generated is the same as the encoding process
illustrated in Figs. 6 and 7 except that, in Step S25
illustrated in Fig. 7, a TSI is generated from a
compatible stream, a TS2 is generated from an auxiliary
stream, a TS3 is generated from auxiliary information,
and multiplexing is performed, and thus, the description
thereof will not be presented.
[0130]
[Description of Another Decoding Process]
Fig. 13 is a flowchart that illustrates a decoding
process performed by the decoding device 120 illustrated
in Fig. 8 in a case where the configurations of the TSs,
which are multiplexed in a multiplexed stream, are the
52
SP308970WOOO
configurations described with reference to Figs. 10 to 12.
This decoding process, for example, is started when a
multiplexed stream that is transmitted from the encoding
device 50 illustrated in Fig. 2 is input to the decoding
5 device 120.
[0131]
In Step S51 illustrated in Fig. 13, the image
generating unit 127 of the decoding device 120 determines
whether or not an instruction for displaying a 2D image
10 has been made by a viewer. In a case where it is
determined that the instruction for displaying a 2D image
has not been made by the viewer in Step S31, in other
words, in a case where an instruction for displaying a 3D
image of the multi-viewpoint mode has been made by the
15 viewer, the process proceeds to Step S52.
[0132]
In Step S52, the separation unit 121 acquires the
multiplexed stream that is transmitted from the encoding
device 50 and separates a TS3 from the multiplexed stream.
20 Then, the separation unit 121 extracts auxiliary
information included in the TS3 by referring to the
delimiter and the like, supplies image information that
is included in the auxiliary information to the image
information acquiring unit 123 for the maintenance
25 thereof, and supplies viewpoint distance information to
the inter-viewpoint distance information acquiring unit
124 for 'the maintenance thereof. In addition, the
separation unit 121 supplies parallax image information
that is included in the auxiliary information to the
30 parallax image information acquiring unit 125 for the
maintenance thereof and supplies compatibility
53
SP308970WOOO
information to the compatibility information acquiring
unit 126 for the maintenance thereof. Then, the process
proceeds to Step S53.
[0133]
5 The process of Steps S53 to S66 is similar to that
of Steps S32 to S45 illustrated in Fig. 9, and thus, the
description thereof will not be presented.
[0134]
In a decoding device that can decode only a
10 compatible stream that has compatibility with the
decoding device 120, the TS2 and TS3 are ignored, and the
process of Steps S65 and S66 is performed.
[0135]

15 [Configuration Example of Encoding Device According to
Second Embodiment]
Fig. 14 is a block diagram that illustrates a
configuration example of an encoding device according to
a second embodiment of the present technology.
20 [0136]
In the configuration illustrated in Fig. 14, the
same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 2.
Duplicate description will not be presented as is
25 appropriate.
[0137]
The configuration of the encoding device 140
illustrated in Fig. 14 is mainly different from the
configuration illustrated in Fig. 2 in that imaging units
30 141A to 141D, an image converting unit 142, a parallax
image generating unit 143, an inter-viewpoint distance
54
•
5
SP308970WOOO
information generating unit 144, and an encoder 145 are
disposed instead of the imaging units 51A to 51C, the
image converting unit 52, the parallax image generating
unit 53, the inter-viewpoint distance information
generating unit 56, and the encoder 58.
[0138]
The encoding device 140 independently generates TSs
by performing encoding in accordance with the AVC mode
with images of two viewpoints out of multi-viewpoint
10 images used as compatible images, thereby securing the
compatibility with an existing encoding device that
encodes a 3D image of a two view-point mode in accordance
with the AVC mode.
[0139]
15 More specifically, the imaging unit 141A of the
encoding device 140 captures an HD image of a
predetermined viewpoint as an image A2 and supplies the
captured image to the image converting unit 142, the
parallax image generating unit 143, and the inter-
20 viewpoint distance information generating unit 144. The
imaging unit 141B captures an HD image of a viewpoint
that is different from the viewpoint of the image A2 as
an image B2 at a position that is separate from the
imaging unit 141A by a distance dd2~ in a horizontal
25 direction and supplies the captured image to the image
converting unit 142, the parallax image generating unit
143, and the inter-viewpoint distance information
generating unit 144.
[0140]
30 The imaging unit 141C captures an HD image of a
viewpoint different from the viewpoints of the images A2
55
5
10
15
20
25
30
SP308970WOOO
and B2 as an image C2 at a position separate from the
imaging unit 141B by a distance dd2Bc in the horizontal
direction that is opposite to the imaging unit 141A and
supplies the captured image to the image converting unit
142, the parallax image generating unit 143, and the
inter-viewpoint distance information generating unit 144.
The imaging unit 141D captures an HD image of a viewpoint
different from the viewpoints of the images A2 to C2 as
an image D2 at a position separate from the imaging unit
141A by a distance dd2AD in the horizontal direction that
is opposite to the imaging unit 141B and supplies the
captured image to the image converting unit 142, the
parallax image generating unit 143, and the interviewpoint
distance information generating unit 144.
[0141]
The viewpoints that correspond to the images C2 and
D2 are viewpoints that are located on the outer side out
of viewpoints of images that can be recognized as 3D
images. In this way, a decoding device that corresponds
to the encoding device 140 can generate multi-viewpoint
images by interpolating images of viewpoints located on
the further inner side than the viewpoints of the images
C2 and D2 by using the images A2 to D2. As a result, the
multi-viewpoint images can be generated with precision
that is higher than that of a case where images of
viewpoints located on the outer side are interpolated
using images of viewpoints located on the inner side.
The distances dd2~, Ad2&, and Ad2~ may be configured to
be either fixed or changed each time.
[0142]
The image converting unit 142 determines the image
56
,
5
10
SP308970WOOO
A2 that is supplied from the imaging unit 141A of which
the position in the horizontal direction is located on
the inner side out of the imaging units 141A to 141D and
the image B2 that is supplied from the imaging unit 141B
as compatible images. Then, the image converting unit
142 multiplexes the images A2 and B2 that are the
compatible images in accordance with a predetermined
multiplexing mode and supplies the multiplexed image to
the encoder 145. In addition, the image converting unit
142 supplies information that designates the images A2
and B2 as compatible images to the compatibility
information generating unit 55.
[0143]
In addition, the image converting unit 142 sets the
15 images C2 and D2 other than the images A2 and B2 as
auxiliary images and multiplexes the images C2 and D2 in
accordance with a predetermined multiplexing mode. The
image converting unit 142 supplies a multiplexed image
that is acquired as a result of the multiplexing process
20 to the encoder 145. The image converting unit 142
supplies information that represents a multiplexing mode
of compatible images and auxiliary images to the image
information generating unit 54.
[0144]
25 The parallax image generating unit 143 detects the
disparity of each pixel of the images A2 to D2 using the
images A2 to D2 that are supplied from the imaging units
141A to 141D. The parallax image generating unit 143
generates a parallax image A2' that represents the
30 disparity of each pixel of the image A2 that is a
compatible image and a parallax image B2' that represents
57
,
5
SP308970WOOO
the disparity of each pixel of the image B2 and
multiplexes the parallax images A2' and B2' in accordance
with a predetermined multiplexing mode. The parallax
image generating unit 143 supplies a multiplexed image
that is acquired as a result thereof to the encoder 145.
[0145]
In addition, the parallax image generating unit 143
generates a parallax image C2' that represents the
disparity of each pixel of the image C2 that is an
10 auxiliary image and a parallax image 02' that represents
the disparity of each pixel of the image 02 that is an
auxiliary image and multiplexes the parallax images C2'
and 02' in accordance with a predetermined multiplexing
mode. The parallax image generating unit 143 supplies a
15 multiplexed image that is acquired as a result thereof to
the encoder 145. The parallax image generating unit 143
supplies information that represents the multiplexing
mode of parallax images of compatible images and
auxiliary images to the parallax image information
20 generating unit 57.
[0146]
The inter-viewpoint distance information generating
unit 144 detects each inter-viewpoint distance between
the images A2 to 02 using the images A2 to 02 that are
25 supplied from the imaging units 141A to 1410. For
example, the inter-viewpoint distance information
generating unit 144 detects a distance ~d2~ between the
imaging units 141A and 141B in the horizontal direction,
a distance ~d2Bc between the imaging units 141B and 141C
30 in the horizontal direction, and a distance ~d2AD between
the imaging units 141A and 1410 in the horizontal
58
,
5
10
15
20
25
30
SP308970WOOO
direction as inter-viewpoint distances. The interviewpoint
distance information generating unit 144
generates information that represents the inter-viewpoint
distances and the like as inter-viewpoint distance
information and supplies the generated inter-viewpoint
distance information to the multiplexing unit 59.
[0147]
The encoder 145 is configured by a compatible
encoder 151 and an auxiliary encoder 152. The compatible
encoder 151 designates a multiplexed image of the
compatible images out of the multiplexed image of the
compatible images and the multiplexed image of the
auxiliary images that are supplied from the image
converting unit 142 and encodes the multiplexed image of
the compatible images in units of access units in
accordance with the existing Ave mode. The encoder 145
supplies an encoded stream that is acquired as a result
thereof to the multiplexing unit 59 as a compatible
stream.
[0148]
The auxiliary encoder 152 encodes the multiplexed
image of the auxiliary images that is supplied from the
image converting unit 142 and the multiplexed image of
the parallax images of the compatible images and the
multiplexed image of the parallax images of the auxiliary
images that are supplied from the parallax image
generating unit 143 in units of access units in
accordance with a predetermined mode. The auxiliary
encoder 152 supplies encoded streams acquired as a result
thereof to the multiplexing unit 59 as auxiliary streams.
As the encoding mode employed by the auxiliary encoder
59
• SP308970WOOO
152, for example, the AVC mode, the MVC mode, or the like
can be used.
[0149]
[Configuration Example of TS]
5 Fig. 15 is a diagram that illustrates a
configuration example of TSs that are generated by the
multiplexing unit 59 illustrated in Fig. 14.
[0150]
In the example illustrated in Fig. 15, in the
10 multiplexing unit 59, a TS1 is generated from the
compatible stream, and a TS2 is generated from the
auxiliary stream and the auxiliary information.
[0151]
Although not illustrated in the figure, the
15 configuration of the TS2 illustrated in Fig. 15 is the
same as the configuration described with reference to Fig.
4 except that a multiplexed image of the parallax images
of the compatible images is arranged instead of the
parallax image of the compatible image.
20 [0152]
[Description of Process of Encoding Device]
Figs. 16 and 17 represent a flowchart that
illustrates an encoding process that is performed by the
encoding device 140 illustrated in Fig. 14. This
25 encoding process, for example, is started when the images
A2 to D2 are output from the imaging units 141A to 141D.
[0153]
In Step S71 illustrated in Fig. 16, the interviewpoint
distance information generating unit 144
30 detects distances ~d2~, ~d2Bc, and ~d2AD as interviewpoint
distances using the images A2 to D2 that are
60
SP308970WOOO
supplied from the imaging units 141A to 141D.
[0154]
In Step S72, the inter-viewpoint distance
information generating unit 144 generates information
5 that represents the inter-viewpoint distances detected in
Step S71 and the like as inter-viewpoint distance
information and inputs the generated inter-viewpoint
distance information to the multiplexing unit 59.
[0155]
10 In Step S73, the image converting unit 142
determines the image A2 that is supplied from the imaging
unit 141A, of which the position in the horizontal
direction is located on the inner side out of the imaging
units 141A to 141D, and the image B2 that is supplied
15 from the imaging unit 141B as compatible images and
determines multiplexing modes of the compatible images
and the auxiliary images. The image converting unit 142
supplies information that designates the images A2 and B2
as compatible images to the compatibility information
20 generating unit 55 and supplies the multiplexing modes of
the compatible images and the auxiliary images to the
image information generating unit 54.
[0156]
In Step S74, the compatibility information
25 generating unit 55 generates the information designating
the images A2 and B2 as compatible images, a frame
packing mode as a compatible mode, and the like as
compatibility information based on the information that
is supplied from the image converting unit 142 and inputs
30 the generated compatibility information to the
multiplexing unit 59.
61
•
5
10
15
20
SP308970WOOO
[0157]
In Step S75, the image converting unit 142
multiplexes the images A2 and B2 that are compatible
images based on the multiplexing mode of compatible
images that is determined in Step S73 and supplies the
multiplexed image to the encoder 145.
[0158]
In Step S76, the image information generating unit
54 generates information that represents the multiplexing
modes of compatible images ad auxiliary images and the
like as image information based on the information that
is supplied from the image converting unit 142 and inputs
the generated image information to the multiplexing unit
59.
[0159]
In Step S77, the image converting unit 142 sets
images C2 and D2 other than the images A2 and B2 as
auxiliary images and multiplexes the auxiliary images
based on the multiplexing mode of auxiliary images that
is determined in Step S73, thereby acquiring a
multiplexed image of the auxiliary images.
[0160]
In Step S78, the image converting unit 142 inputs
the multiplexed image of the compatible images and the
25 multiplexed image of the auxiliary images to the encoder
145.
[0161]
In Step S79 illustrated in Fig. 17, the parallax
image generating unit 143 detects the disparity of each
30 pixel of the images A2 to D2 using the images A2 to D2
that are supplied from the imaging units 141A to 141D and
62
SP308970WOOO
generates parallax images A2' to 02'.
[0162]
In Step S80, the parallax image generating unit 143
determines multiplexing modes of the parallax images of
5 the compatible images and the parallax images of the
auxiliary images and supplies information that represents
the multiplexing modes to the parallax image information
generating unit 57.
[0163]
10 In Step S81, the parallax image information
generating unit 57 generates information that represents
the multiplexing modes of the parallax images of the
compatible images and the parallax images of the
auxiliary images and the like as parallax image
15 information based on the information that is supplied
from the parallax image generating unit 143 and inputs
the generated parallax image information to the
multiplexing unit 59.
[0164]
20 In Step S82, the parallax image generating unit 143
multiplexes the parallax images A2' and B2' of the
compatible images based on the multiplexing mode of the
parallax images of compatible images that is determined
in Step S80 and multiplexes the parallax images C2' and
25 02' of the auxiliary images based on the multiplexing
mode of the parallax images of auxiliary images.
[0165]
In Step S83, the parallax image generating unit 143
inputs the multiplexed image of the parallax images of
30 the compatible images and the multiplexed image of the
parallax images of the auxiliary images that are acquired
63
SP308970WOOO
as a result of the multiplexing process illustrated in
Step S82 to the encoder 145.
[0166]
In Step S84, the compatible encoder 151 of the
5 encoder 145 encodes the multiplexed image of the
compatible images that is supplied from the image
converting unit 142 in accordance with the existing Ave
mode and supplies an encoded stream acquired as a result
thereof to the multiplexing unit 59 as a compatible
10 stream.
[0167]
In Step S85, the auxiliary encoder 152 encodes the
multiplexed image of the auxiliary images that is
supplied from the image converting unit 142 and the
15 multiplexed image of the parallax images of the
compatible images and the multiplexed image of the
parallax images of the auxiliary images that are supplied
from the parallax image generating unit 143 in accordance
with a predetermined mode. The auxiliary encoder 152
20 supplies an encoded stream acquired as a result of the
encoding process to the multiplexing unit 59 as an
auxiliary stream.
[0168]
In Step S86, the multiplexing unit 59 generates a
25 TS1 from the compatible stream that is supplied from the
compatible encoder 151, generates a TS2 from the
auxiliary stream and the auxiliary information supplied
from the auxiliary encoder 152, performs a multiplexing
process, and transmits a multiplexed stream acquired as a
30 result thereof. This multiplexed stream, for example, is
recorded on a BD or the like or is transmitted as a
64
•
broadcast stream. Then, the process ends.
SP308970WOOO
[0169]
As above, since the encoding device 140 performs
the multiplexing process with the compatible stream and
5 the auxiliary stream and the auxiliary information being
stored in TSs different from each another, the
compatibility with an existing encoding device that
encodes a 3D image of two-viewpoint mode in accordance
with the Ave mode in multiplexing can be secured. In
10 other words, the encoding device 140 can perform a
multiplexing process for multi-viewpoint images in
accordance with a mode that has the compatibility with an
existing mode.
[0170]
15 In addition, since the encoding device 140 encodes
images of two viewpoints out of multi-viewpoint images as
compatible image in accordance with an existing encoding
mode, the compatibility with an existing encoding device
that encodes a 3D image of the two-viewpoint mode in the
20 encoding process can be secured.
[0171]
Furthermore, since the encoding device 140 encodes
the images A2 to D2 of four viewpoints, a decoding device
that corresponds to the encoding device 140 can generate
25 multi-viewpoint images from the images A2 to D2 of four
viewpoints. In this way, according to the decoding
device, compared to a case where multi-viewpoint images
are generated from images of two viewpoints, the
viewpoints of images that can be generated are not
30 limited, and multi-viewpoint images having relatively
high precision can be generated.
65
• SP308970WOOO
[0172]
[Configuration Example of Decoding Device]
Fig. 18 is a diagram that illustrates a
configuration example of a decoding device that decodes
5 the multiplexed stream transmitted from the encoding
device 140 illustrated in Fig. 14.
[0173]
In the configuration illustrated in Fig. 18, the
same reference numeral is assigned to a configuration
10 that is the same as that illustrated in Fig. 8.
Duplicate description will not be presented as is
appropriate.
[0174]
The configuration of the decoding device 170
15 illustrated in Fig. 18 is mainly different from the
configuration illustrated in Fig. 8 in that an image
generating unit 171 is disposed instead of the image
generating unit 127. The decoding device 170 generates
images of two viewpoints or multi-viewpoint images by
20 decoding the multiplexed stream that is transmitted from
the encoding device 140 and displays the generated images
on a display device not illustrated in the figure.
[0175]
More specifically, the image generating unit 171 of
25 the decoding device 170 outputs images in accordance with
a display instruction supplied from a viewer, thereby
displaying the images on a display device (not
illustrated). When described in more detail, the image
generating unit 171 generates images of three or more
30 viewpoints, which correspond to a display device (not
illustrated), each having the resolution that is a half
66
•
5
10
15
20
25
30
SP308970WOOO
of the resolution of the compatible image or the
auxiliary image by using the multiplexed image of the
compatible images, the multiplexed image of the auxiliary
images, the multiplexed image of the parallax images of
the compatible images, and the multiplexed image of the
parallax images of the auxiliary images in accordance
with a viewer's instruction for displaying a 3D image of
multi-viewpoint mode based on the image information
supplied from the image information acquiring unit 123,
the inter-viewpoint distance information supplied from
the inter-viewpoint distance information acquiring unit
124, the parallax image information supplied from the
parallax image information acquiring unit 125, the
compatibility information supplied from the compatibility
information acquiring unit 126, and the like.
[0176]
More specifically, the image generating unit 171
separates the parallax image of each auxiliary image from
the multiplexed image of the parallax images of the
auxiliary images based on the information that represents
a multiplexing mode of the parallax images of the
auxiliary images that is included in the parallax image
information supplied from the parallax image information
acquiring unit 125. In addition, the image generating
unit 171 separates the parallax image of each compatible
image from the multiplexed image of the parallax images
of the compatible images based on the information, which
represents the multiplexing mode of the parallax images
of the compatible images, included in the parallax image
information.
[0177]
67
•
5
10
SP308970WOOO
Furthermore, the image generating unit 171
separates each auxiliary image from the multiplexed image
of the auxiliary images based on the information that
represents a multiplexing mode of the auxiliary images
that is included in the image information supplied from
the image information acquiring unit 123. In addition,
the image generating unit 171 separates each compatible
image from the multiplexed image of the compatible images
based on the information that represents a multiplexing
mode of the compatible images that is included in the
image information.
[0178]
Furthermore, the image generating unit 171
determines the position of each viewpoint of the multi-
15 viewpoint images to be generated based on the interviewpoint
distance information and the number of
viewpoints that corresponds to a display device not
illustrated in the figure. Then, the image generating
unit 171 generates the image of each viewpoint of which
20 the position is determined by using each compatible image,
each auxiliary image, the parallax image of each
compatible image, and the parallax image of each
auxiliary image. Then, the image generating unit 171
converts the resolution of the generated multi-viewpoint
25 images into resolution that is "lithe number of
viewpoints" of the resolution of the compatible image or
the auxiliary image, synthesizes the images, and displays
the synthesized image on a display device not illustrated
in the figure.
30 [0179]
At this time, the multi-viewpoint images after the
68
5
10
15
20
25
30
SP308970WOOO
synthesis are displayed such that the viewing angles are
different for each viewpoint, and a viewer can view a 3D
image without wearing glasses by seeing images of
arbitrary two viewpoints with his left and right eyes.
[0180]
In addition, the image generating unit 171
separates the multiplexed image of the compatible images
that is supplied from the decoder 122 into images A2 and
B2 of which the resolution is a half of the resolution of
the compatible image based on the image information
supplied from the image information acquiring unit 123 in
accordance with an instruction for displaying a 3D image
of the two-viewpoint mode that is supplied from a viewer.
Then, the image generating unit 171 alternately outputs
the separated images A2 and B2 of which the resolution is
a half of the resolution of the compatible image, thereby
displaying the separated images on a display device not
illustrated in the figure. At this time, the viewer can
view a 3D image by wearing glasses in which one of a
left-eye shutter and a right-eye shutter is open at the
time of displaying the image A2, and the other is open at
the time of displaying the image B2 and viewing the
images A2 and B2 that are alternately displayed on the
display device.
[0181]
[Description of Process of Decoding Device]
Fig. 19 is a flowchart that illustrates a decoding
process performed by the decoding device 170 illustrated
in Fig. 18. This decoding process, for example, is
started when a multiplexed stream that is transmitted
from the encoding device 140 illustrated in Fig. 14 is
69
SP308970WOOO
input to the decoding device 170.
[0182]
In Step S91 illustrated in Fig. 19, the separation
unit 121 of the decoding device 170 receives the
5 multiplexed stream that is transmitted from the encoding
device 50 and separates TS1 and TS2 from the multiplexed
stream. The separation unit 121 extracts a compatible
stream included in the TS1 and an auxiliary stream
included in the TS2 by referring a delimiter and the like
10 and supplies the extracted streams to the decoder 122.
In addition, the separation unit 121 extracts auxiliary
information included in the TS2 by referring to the
delimiter and the like, supplies image information that
is included in the auxiliary information to the image
15 information acquiring unit 123, and supplies viewpoint
distance information to the inter-viewpoint distance
information acquiring unit 124. In addition, the
separation unit 121 supplies parallax image information
that is included in the auxiliary information to the
20 parallax image information acquiring unit 125 and
supplies compatibility information to the compatibility
information acquiring unit 126.
[0183]
In Step S92, the image generating unit 171
25 determines whether or not an instruction for displaying a
3D image of the two-viewpoint mode has been made by a
viewer. In a case where it is determined that the
instruction for displaying a 3D image of the twoviewpoint
mode has not been made by the viewer in Step
30 S92, in other words, in a case where an instruction for
displaying a 3D image of the multi-viewpoint mode has
70
SP308970WOOO
been made, the process proceeds to Step S93.
[0184]
The process of Steps 893 to S102 is the same as
that of Steps S33 to S42 illustrated in Fig. 9 except
5 that the compatible image is a multiplexed image of
compatible images, and the parallax image of the
compatible image is a multiplexed image of parallax
images of compatible images, and thus description thereof
will not be presented.
10 [0185]
On the other hand, in a case where it is determined
that the instruction for displaying a 3D image of the
two-viewpoint mode has been made by the viewer in Step
S92, the process proceeds to Step S103.
15 [0186]
In Step S103, the compatible decoder 131 of the
decoder 122 extracts a multiplexed image of encoded
compatible images from the compatible stream that is
supplied from the separation unit 121 and decodes the
20 encoded compatible image in accordance with a mode that
corresponds to the Ave mode. The compatible decoder 131
supplies the multiplexed image of the compatible image
that is acquired as a result of the decoding process to
the image generating unit 171.
25 [0187]
In Step S104, the image information acquiring unit
123 inputs the image information that is supplied from
the separation unit 121 to the image generating unit 171.
[0188]
30 In Step S105, the image generating unit 171
separates the multiplexed image of the compatible images
71
•
5
10
15
20
SP308970WOOO
that is acquired as result of the decoding process
performed by the compatible decoder 131 based on the
information that represents a multiplexing mode of the
compatible image that is included in the image
information supplied from the image information acquiring
unit 123.
[0189]
In Step 5106, the image generating unit 171
alternately outputs the compatible images A2 and B2 of
which the resolution is a half of the resolution of the
separated compatible image to a display device not
illustrated in the figure, thereby displaying the images.
Then, the process ends.
[0190]
In addition, in a decoding device that can decode
only a compatible stream that has compatibility with the
decoding device 170, a TS2 to which a 3DV Representation
Delimiter NAL unit is attached is ignored, and the
process of Steps 5103, 5105, and 5106 is performed.
However, in this case, in the process of Step 5105, a
multiplexed image of compatible images is separated based
on the multiplexing mode determined in advance.
[0191]
As above, the decoding device 170 can separate a
25 multiplexed stream that is multiplexed by the encoding
device 140 in accordance with a mode that has
compatibility with an existing mode. In addition, the
decoding device 170 can decode a compatible stream and an
auxiliary stream that are encoded by the encoding device
30 140 in accordance with a mode that has compatibility with
an existing mode and has a relative low processing cost.
72
• SP308970WOOO
[0192]
In addition, since the decoding device 170
generates multi-viewpoint images using compatible images
of two viewpoints and auxiliary images of two viewpoints,
5 the encoding device 140 may prepare only two imaging
units 141C and 141D other than the existing imaging units
141A and 141B that capture compatible images of two
viewpoints as- imaging units used for generating multiviewpoint
images. Accordingly, the installation of the
10 imaging units for generating multi-viewpoint images can
be easily performed at a low cost.
[0193]
[Another Configuration Example of TS]
Fig. 20 is a diagram that illustrates another
15 configuration example of the TSs that are generated by
the multiplexing unit 59 illustrated in Fig. 14.
[0194]
In the example illustrated in Fig. 20, three TSs
are generated by the multiplexing unit 59, and an
20 auxiliary stream and auxiliary information are included
in TSs that are different from each other. More
specifically, a TSI is generated from the compatible
stream, a TS2 is generated from the auxiliary stream, and
a TS3 is generated from the auxiliary information.
25 [0195]
In the example illustrated in Fig. 20, the TS3 is
independently generated only from the auxiliary
information of which the amount of information is
relatively small. Accordingly, a decoding device that
30 executes an application, in which there is a limitation
on the number of TSs that can be simultaneously processed,
73
SP308970WOOO
such as a BD application can reproduce the TSl and the
TS2 in synchronization with each other by preloading the
TS3 of which the amount of information is relatively
small.
5 [0196]
In addition, although not illustrated in the figure,
the configuration of the TS2 illustrated in Fig. 20 is
the same as the configuration described with reference to
Fig. 11 except that a multiplexed image of parallax
10 images of compatible images is arranged instead of the
parallax image of the compatible image. Furthermore, the
configuration of the TS3 is the same as the configuration
described with reference to Fig. 12 except that an
auxiliary image of the multiplexed image of the parallax
15 images of the compatible images is arranged instead of
the auxiliary image of the parallax image of the
compatible image.
[0197]
[Description of Another Encoding Process]
20 The encoding process performed by the encoding
device 140 in a case where the TSs having the
configurations described with reference to Fig. 20 are
generated is the same as the encoding process illustrated
in Figs. 16 and 17 except that, in Step S86 illustrated
25 in Fig. 17, a TS1 is generated from a compatible stream,
a TS2 is generated from an auxiliary stream, a TS3 is
generated from auxiliary information, and multiplexing is
performed, and thus, the description thereof will not be
presented.
30 [0198]
[Description of Another Decoding Process]
74
SP308970WOOO
Fig. 21 is a flowchart that illustrates a decoding
process performed by the decoding device 170 illustrated
in Fig. 18 in a case where the configurations of the TSs,
which are multiplexed in a multiplexed stream, are the
5 configurations described with reference to Fig. 20. This
decoding process, for example, is started when a
multiplexed stream that is transmitted from the encoding
device 140 illustrated in Fig. 14 is input to the
decoding device 170.
10 [0199]
In Step Slll illustrated in Fig. 21, the separation
unit 121 of the decoding device 120 acquires the
multiplexed stream that is transmitted from the encoding
device 50 and separates a TS3 from the multiplexed stream.
15 Then, the separation unit 121 extracts auxiliary
information included in the TS3 by referring to the
delimiter and the like. In addition, the separation unit
121 supplies image information that is included in the
auxiliary information to the image information acquiring
20 unit 123 for the maintenance thereof and supplies
viewpoint distance information to the inter-viewpoint
distance information acquiring unit 124 for the
maintenance thereof. Furthermore, the separation unit
121 supplies parallax image information included in the
25 auxiliary information to the parallax image information
acquiring unit 125 for the maintenance thereof and
supplies compatibility information to the compatibility
information acquiring unit 126 for the maintenance
thereof. Then, the process proceeds to Step Sl12.
30 [0200]
In Step Sl12, the image generating unit 127
75
•
5
10
15
20
25
30
SP308970WOOO
determines whether or not an instruction for displaying a
3D image of the two-viewpoint mode has been made by a
viewer. In a case where it is determined that the
instruction for displaying a 3D image of the twoviewpoint
mode has not been made by the viewer in Step
S112, in other words, in a case where an instruction for
displaying a 3D image of the multi-viewpoint mode has
been made, the process proceeds to Step S113.
[0201]
In Step S113, the separation unit 121 separates the
TS1 and the TS2 from the multiplexed stream. The
separation unit 121 extracts a compatible stream included
in the TS1 and an auxiliary stream included in the TS2 by
referring to the delimiter and the like and supplies the
extracted streams to the decoder 122. Then, the process
proceeds to Step S114.
[0202]
The process of Steps S114 to S123 is the same as
the process of Steps S93 to S102 illustrated in Fig. 19,
and thus the description thereof will not be presented.
[0203]
On the other hand, in a case where it is determined
that an instruction for displaying a 3D image of the twoviewpoint
mode has been made by the viewer in Step S112,
the separation unit 121 separates the TS1 from the
multiplexed stream in Step S124. Then, the separation
unit 121 supplies a compatible stream that is included in
the TS1 to the decoder 122 by referring to the delimiter
and the like, and the process proceeds to Step S125.
[0204]
The process of Steps S125 to S128 is the same as
76
• SP308970WOOO
the process of Steps S103 to S106 illustrated in Fig. 19,
and thus the description thereof will not be presented.
[0205J

5 [Configuration Example of Encoding Device According to
Third Embodiment]
Fig. 22 is a block diagram that illustrates a
configuration example of an encoding device according to
a third embodiment of the present technology.
10 [0206J
In the configuration illustrated in Fig. 22, the
same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 2 or 14.
Duplicate description will not be presented as is
15 appropriate.
[0207J
The configuration of the encoding device 180
illustrated in Fig. 22 is mainly different from the
configuration illustrated in Fig. 14 in that an image
20 converting unit 181 and an encoder 182 are disposed
instead of the image converting unit 142 and the encoder
145. The encoding device 180 secures compatibility with
an existing encoding device that encodes a 3D image of
two viewpoints in accordance with the MVC mode by
25 encoding images of two viewpoints out of multi-viewpoint
images in accordance with the MVC mode as compatible
images and storing the images altogether in one TS or
storing the images in TSs that are different for each
viewpoint.
30 [0208J
More specifically, the image converting unit 181 of
77
•
5
10
15
20
25
30
SP308970WOOO
the encoding device 180, similarly to the image
converting unit 142 illustrated in Fig. 14, determines an
image A2 that is supplied from the imaging unit 141A, of
which the position in the horizontal direction is located
on the inner side out of the imaging units 141A to 141D,
and an image B2 that is supplied from the imaging unit
141B as compatible images. Then, the image converting
unit 181 directly supplies the images A2 and B2 that are
compatible images to the encoder 182. In addition, the
image converting unit 181, similarly to the image
converting unit 142, supplies information that designates
the images A2 and B2 as compatible images to the
compatibility information generating unit 55.
[0209]
In addition, the image converting unit 181,
similarly to the image converting unit 142, sets images
C2 and D2 other that the images A2 and B2 as auxiliary
images and multiplexes the auxiliary images in accordance
with a predetermined multiplexing mode. The image
converting unit 181 supplies a multiplexed image that is
acquired as a result of the multiplexing process to the
encoder 182 and supplies information that represents the
multiplexing mode of the auxiliary images to the image
information generating unit 54.
[0210]
The encoder 182 is configured by a compatible
encoder 191 and an auxiliary encoder 152. The compatible
encoder 191 of the encoder 182 designates compatible
images from among the compatible images and the
multiplexed image of the auxiliary images that are
supplied from the image converting unit 181, encodes the
78
•
5
10
15
20
25
30
SP308970WOOO
image A2 out of the compatible images as a base image in
accordance with the existing AVC mode in units of access
units, and encodes the image B2 as a dependent image in
accordance with the existing MVC mode in units of the
access units. The compatible encoder 191 supplies an
encoded stream that is acquired as a result thereof to
the multiplexing unit 59 as a compatible stream.
[0211]
[First Configuration Example of TS]
Fig. 23 is a diagram that illustrates a first
configuration example of the TSs that are generated by
the multiplexing unit 59 illustrated in Fig. 22.
[0212]
In the example illustrated in Fig. 23, in the
multiplexing unit 59, a TS1 is generated from the image
A2 after encoding which is included in the compatible
stream, and a TS2 is generated from the image B2 after
encoding, the auxiliary stream, and the auxiliary
information.
[0213]
[Description of Process of Encoding Device]
Figs. 24 and 25 are flowcharts that illustrate an
encoding process that is performed by the encoding device
180 illustrated in Fig. 22. This encoding process, for
example, is started when the images A2 to D2 are output
from the imaging units 141A to 141D.
[0214]
The process of Steps S131 and S132 illustrated in
Fig. 24 is the same as the process of Steps S71 and S72
illustrated in Fig. 16, and thus the description thereof
will not be presented.
79
• SP308970WOOO
[0215]
After the process of Step S132, in Step S133, the
image converting unit 181 determines the image A2 that is
supplied from the imaging unit 141A, of which the
5 position in the horizontal direction is located on the
inner side out of the imaging units 141A to 141D, and the
image B2 that is supplied from the imaging unit 141B as
compatible images and determines a multiplexing mode of
the auxiliary images. The image converting unit 181
10 supplies information that designates the images A2 and B2
as compatible images to the compatibility information
generating unit 55 and supplies the multiplexing mode of
the auxiliary images to the image information generating
unit 54. Then, the process proceeds to Step S134.
15 [0216]
The process of Steps S134 to S137 is the same as
the process of Steps S14 to S17 illustrated in Fig. 6,
and thus the description thereof will not be presented.
The process of Steps S138 to S142 is the same as the
20 process of Steps S79 to S83 illustrated in Fig. 17, and
thus the description thereof will not be presented.
[0217]
After the process of Step S142, in Step S143, the
compatible encoder 191 of the encoder 182 encodes the
25 image A2 out of the compatible images supplied from the
image converting unit 181 as a base image in accordance
with the existing Ave mode and encodes the image B2 as a
dependent image in accordance with the existing Mve mode.
The compatible encoder 191 supplies an encoded stream
30 that is acquired as a result thereof to the multiplexing
unit 59 as a compatible stream.
80
•
5
10
15
20
SP308970WOOO
[0218]
In step S144, the auxiliary encoder 152 encodes the
multiplexed image of the auxiliary images that is
supplied from the image converting unit 142 and the
multiplexed image of the parallax images of the
compatible images and the multiplexed images of the
parallax images of the auxiliary images that are supplied
from the parallax image generating unit 53 in accordance
with a predetermined mode. The auxiliary encoder 152
supplies an encoded stream that is acquired as a result
of the encoding process to the multiplexing unit 59 as an
auxiliary stream.
[0219]
In step S145, the multiplexing unit 59 generates a
TS1 from the image A after encoding which is included in
the compatible steam, generates a TS2 from the image B
after encoding, the auxiliary stream, and the auxiliary
information, performs a multiplexing process, and
transmits a multiplexed stream that is acquired as a
result thereof. This multiplexed stream, for example, is
recorded on a BD or the like or is transmitted as a
broadcast stream. Then, the process ends.
[0220]
As above, the encoding device 180 performs
25 multiplexing with one of the compatible images after
encoding and the other being stored in TSs different from
each other, and accordingly, the compatibility with an
existing encoding device can be secured which encodes 3D
images of the two-viewpoint mode in accordance with the
30 MVC mode and stores the images in two TSs in multiplexing.
[0221]
81
.. SP308970WOOO
In addition, since the encoding device 180 encodes
images of two viewpoints out of multi-viewpoint images as
compatible images in accordance with an existing encoding
mode, the compatibility with an existing encoding device
5 can be secured in encoding which encodes 3D images of the
two-viewpoint mode.
[0222]
[Configuration Example of Decoding Device]
Fig. 26 is a diagram that illustrates a
10 configuration example of a decoding device that decodes
the multiplexed stream transmitted from the encoding
device 180 illustrated in Fig. 22.
[0223]
In the configuration illustrated in Fig. 26, the
15 same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 18.
Duplicate description will not be presented as is
appropriate.
[0224]
20 The configuration of the decoding device 200
illustrated in Fig. 26 is mainly different from the
configuration illustrated in Fig. 18 in that a decoder
201 and an image generating unit 202 are disposed instead
of the decoder 122 and the image generating unit 171.
25 The decoding device 200 generates an image of one
viewpoint, images of two viewpoints, or multi-viewpoint
images by decoding the multiplexed stream that is
transmitted from the encoding device 180 and displays the
generated images on a display device not illustrated in
30 the figure.
[0225]
82
5
10
15
20
25
30
SP308970WOOO
More specifically, the decoder 201 of the decoding
device 200 is configured by a compatible decoder 211 and
an auxiliary decoder 132. The compatible decoder 211 of
the decoder 201 decodes an encoded compatible image that
is included in the compatible stream supplied from the
separation unit 121 in accordance with a mode that
corresponds to the MVC mode and supplies the decoded
compatible image to the image generating unit 202.
[0226]
The image generating unit 202 outputs the image in
accordance with a display instruction supplied from a
viewer, thereby displaying the image on a display device
not illustrated in the figure. More specifically, the
image generating unit 202 generates images of three or
more viewpoints, which correspond to a display device not
illustrated in the figure, each having the resolution
that is a half of the resolution of the compatible image
or the auxiliary image by using the compatible image, the
multiplexed image of the auxiliary images, the
multiplexed image of the parallax images of the
compatible images, and the multiplexed image of the
parallax images of the auxiliary images in accordance
with a viewer's instruction for displaying a 3D image of
multi-viewpoints based on the image information supplied
from the image information acquiring unit 123, the interviewpoint
distance information supplied from the interviewpoint
distance information acquiring unit 124, the
parallax image information supplied from the parallax
image information acquiring unit 125, the compatibility
information supplied from the compatibility information
acquiring unit 126, and the like.
83
SP308970WOOO
[0227]
More specifically, the image generating unit 202
separates the parallax image of each auxiliary image from
the multiplexed image of the parallax images of the
5 auxiliary images based on the information that represents
a multiplexing mode of the parallax images of the
auxiliary images that is included in the parallax image
information supplied from the parallax image information
acquiring unit 125. In addition, the image generating
10 unit 202 separates the parallax image of each compatible
image from the multiplexed image of the parallax images
of the compatible images based on the information that
represents a multiplexing mode of the parallax images of
the compatible images that is included in the parallax
15 image information.
[0228]
Furthermore, the image generating unit 202
separates each auxiliary image from the multiplexed image
of the auxiliary images based on the information that
20 represents a multiplexing mode of the auxiliary images
that is included in the image information supplied from
the image information acquiring unit 123. In addition,
the image generating unit 202 determines the positions of
viewpoints of multi-viewpoint images to be generated
25 based on the inter-viewpoint distance information and the
number of viewpoints that corresponds to a display device
not illustrated in the figure. Then, the image
generating unit 202 generates an image of each viewpoint
of which the position is determined by using each
30 compatible image, each auxiliary image, the parallax
image of each compatible image, and the parallax image of
84
each auxiliary image.
SP308970WOOO
Then, the image generating unit
202, similarly to the image generating unit 127, converts
the resolution of the generated multi-viewpoint images
into the resolution that is "l/the number of viewpoints"
5 of the resolution of the compatible image or the
auxiliary image, synthesizes the converted images, and
displays the synthesized image on a display device not
illustrated in the figure.
[0229]
10 At this time, the multi-viewpoint images after the
synthesis are displayed such that the viewing angles are
different for each viewpoint, and a viewer can view a 3D
image without wearing glasses by seeing images of
arbitrary two viewpoints with his/her left and right eyes.
15 [0230]
In addition, the image generating unit 202
alternately outputs the images A2 and B2 as compatible
images supplied from the decoder 122 in accordance with
an instruction for displaying a 3D image of the two-
20 viewpoint mode that is supplied from a viewer, thereby
displaying the images on a display device not illustrated
in the figure. At this time, the viewer can view a 3D
image by wearing glasses in which one of a left-eye
shutter and a right-eye shutter is open at the time of
25 displaying the image A2, and the other is open at the
time of displaying the image B2 and viewing the images A2
and B2 that are alternately displayed on the display
device.
[0231]
30 Furthermore, the image generating unit 202 outputs
the image A2 out of the compatible images supplied from
85
5
• SP308970WOOO
the decoder 122 in accordance with an instruction for
displaying a 2D image that is supplied from a viewer,
thereby displaying the image on a display device not
illustrated in the figure. Accordingly, the viewer can
view the 2D image.
[0232]
[Description of Process of Decoding Device]
Fig. 27 is a flowchart that illustrates a decoding
process performed by the decoding device 200 illustrated
10 in Fig. 26. This decoding process, for example, is
started when a multiplexed stream that is transmitted
from the encoding device 180 illustrated in Fig. 22 is
input to the decoding device 200.
[0233]
15 In Step S151 illustrated in Fig. 27, the image
generating unit 202 of the decoding device 200 determines
whether or not an instruction for displaying a 2D image
has been made by a viewer. In a case where it is
determined that the instruction for displaying a 2D image
20 has been made by the viewer in Step S151, the process
proceeds to Step S152.
[0234]
In Step S152, the separation unit 121 receives the
multiplexed stream that is transmitted from the encoding
25 device 180 and separates a TS1 from the multiplexed
stream. Then, the separation unit 121 extracts a part of
the compatible stream that is included in the TS1 by
referring to the delimiter and the like and supplies the
extracted stream to the decoder 201.
30 [0235]
In Step S153, the compatible decoder 211 of the
86
5
10
15
20
25
30
SP308970WOOO
decoder 201 extracts an image A2, which is a base image
out of encoded compatible images, from a part of the
compatible stream that is supplied from the separation
unit 121 and decodes the extracted image in accordance
with a mode that corresponds to the Ave mode. The
compatible decoder 211 supplies the image A2 that is
acquired as a result of the decoding process to the image
generating unit 202.
[0236]
In Step S154, the image generating unit 202 outputs
the image A2 that is supplied from the compatible decoder
211 to a display device not illustrated in the figure,
thereby displaying the image. Then, the process ends.
[0237]
On the other hand, in a case where it is determined
that an instruction for displaying a 20 image has not
been made by the viewer in Step S151, the separation unit
121 separates TSI and TS2 from the multiplexed stream in
Step S155. Then, the separation unit 121 extracts a part
of the compatible stream that is included in the TSI by
referring to the delimiter and the like and supplies the
extracted stream to the decoder 122. In addition, the
separation unit 121 extracts another part of the
compatible stream included in the TS2 and the auxiliary
stream and supplies the extracted streams to the decoder
122. In addition, the separation unit 121 extracts the
auxiliary information that is included in the TS2 by
referring to the delimiter and the like, supplies the
image information that is included in the auxiliary
information to the image information acquiring unit 123,
and supplies the viewpoint distance information to the
87
•
5
10
15
20
25
30
SP30B970WOOO
inter-viewpoint distance information acquiring unit 124.
Furthermore, the separation unit 121 supplies the
parallax image information that is included in the
auxiliary information to the parallax image information
acquiring unit 125 and supplies the compatibility
information to the compatibility information acquiring
unit 126.
[023B]
In step S156, the image generating unit 202
determines whether or not an instruction for displaying a
3D image of the two-viewpoint mode has been made by a
viewer. In a case where it is determined that the
instruction for displaying a 3D image of the twoviewpoint
mode has been made by the viewer in Step S156,
the process proceeds to Step S157.
[0239]
In Step S157, the compatible decoder 211 of the
decoder 122 extracts an encoded compatible image from the
compatible stream that is supplied from the separation
unit 121 and decodes the extracted compatible image.
More specifically, the compatible decoder 211 extracts an
encoded image A from a part of the compatible stream that
is supplied from the separation unit 121 and extracts an
encoded image B from another part of the compatible
stream. Then, the compatible decoder 211 decodes the
encoded image A as an encoded base image in accordance
with a mode that corresponds to the Ave mode and decodes
the encoded image B as an encoded dependent image in
accordance with a mode that corresponds to the Mve mode.
The compatible decoder 211 supplies the images A and B
that are compatible images acquired as a result of the
BB
• SP308970WOOO
decoding process to the image generating unit 202.
[0240]
In Step S158, the image generating unit 202
alternately outputs images Al and Bl out of the
5 compatible images that are supplied from the compatible
decoder 211, thereby displaying the images Al and Bl on a
display device not illustrated in the figure. Then, the
process ends.
[0241]
10 On the other hand, in a case where it is determined
that the instruction for displaying a 3D image of the
two-viewpoint mode has not been made by the viewer in
Step S156, in other words, in a case where an instruction
for displaying a 3D image of the multi-viewpoint mode has
15 been made by the viewer, the process proceeds to Step
S159.
[0242]
In Step S159, the compatible decoder 211, similarly
to the process of Step S157, extracts an encoded
20 compatible image from the compatible stream that is
supplied from the separation unit 121 and decodes the
extracted compatible image in accordance with a mode that
corresponds to the MVC mode. The compatible decoder 211
supplies images A and B that are the compatible images
25 acquired as a result of the decoding process to the image
generating unit 202.
[0243]
In Step S160, the auxiliary decoder 132 extracts
the encoded multiplexed image of the auxiliary images,
30 the multiplexed image of the parallax images of the
compatible images, and the multiplexed image of the
89
•
5
10
15
SP308970WOOO
parallax images of the auxiliary images from the
auxiliary stream and decodes the extracted images in
accordance with a mode that corresponds to the auxiliary
encoder 152 illustrated in Fig. 22. The auxiliary
decoder 132 supplies the multiplexed image of the
auxiliary images, the multiplexed image of the parallax
images of the compatible images, and the multiplexed
image of the parallax images of the auxiliary images,
which are acquired as a result of the decoding process,
to the image generating unit 202.
[0244]
The process of Steps S161 to S168 is the same as
the process of Steps S35 to S42 illustrated in Fig. 9
except that the parallax image of the compatible image is
a multiplexed image of the parallax images of the
compatible images, and thus the description thereof will
not be presented.
[0245]
In a decoding device that can decode only a
20 compatible stream that has compatibility with the
decoding device 200, the auxiliary stream and the
auxiliary information included in the TS2 are ignored,
and the process of Steps S151 to S155, S157, and S158 is
performed.
25 [0246]
As above, the decoding device 200 can separate the
multiplexed stream that is multiplexed by the encoding
device 180 in accordance with a mode that has
compatibility with an existing mode. In addition, the
30 decoding device 200 can decode a compatible stream and an
auxiliary stream that are encoded in accordance with a
90
• SP308970WOOO
mode, which has compatibility with an existing mode and
requires relatively low process cost, by the encoding
device 180.
[0247]
5 [Second Configuration Example of TS]
Fig. 28 is a diagram that illustrates a second
configuration example of the TSs that are generated by
the multiplexing unit 59 illustrated in Fig. 22.
[0248]
10 In the example illustrated in Fig. 28, three TSs
are generated by the multiplexing unit 59, and an image
B2 that is one of encoded compatible images included in
the compatible stream and an auxiliary stream and
auxiliary information are included in TSs that are
15 different from each other. More specifically, a TS1 is
generated from an image A2 that is one of encoded
compatible images included in the compatible stream, a
TS2 is generated from the other image B2, and a TS3 is
generated from an auxiliary stream and auxiliary
20 information.
[0249]
In the example illustrated in Fig. 28, since each
one of the images A1 and B2 after encoding, which are
included in the compatible stream, is stored in one TS,
25 the amount of information of the compatible stream does
not need to be decreased. Accordingly, the image quality
of the compatible image can be configured to be the same
as the image quality of the images of two viewpoints that
are encoded in accordance with the existing MVC mode.
30 [0250]
[Description of Multiplexing Method]
91
• SP308970WOOO
Fig. 29 is a diagram that illustrates an example of
the method of multiplexing the TS1 to the TS3 illustrated
in Fig. 28.
[0251]
5 As illustrated in Fig. 29, the TS1 to TS3 are
multiplexed by being arranged from the head in order of
the TS3, TS1, and the TS2 in units of access units. In
addition, a 3DV representation delimiter representing the
head of the TS that includes information other than a
10 compatible stream is added to the head of the TS3. In
other words, the 3DV representation delimiter is arranged
on the boundary between a TS3 of a specific access unit
and a TS2 of the previous access unit. Accordingly, the
decoding device 200 can easily recognize a compatible
15 stream and an auxiliary stream and an auxiliary image
synchronized therewith by extracting data from a specific
3DV representation delimiter to a next 3DV representation
delimiter.
[0252]
20 [Description of Second Encoding Process]
An encoding process that .is performed by the
encoding device 180 in a case where the TSs having the
configurations described with reference to Fig. 28 is the
same as the encoding process described with reference to
25 Figs. 24 and 25 except that the TS1 is generated from an
image A after encoding that is included in the compatible
stream, and a TS2 is generated from an image B after
encoding, and a TS3 is generated from the auxiliary
stream and the auxiliary information in Step S145
30 illustrated in Fig. 25, and the description thereof will
not be presented.
92
SP308970WOOO
[0253]
[Description of Second Decoding Process]
Fig. 30 is a flowchart that illustrates a decoding
process performed by the decoding device 200 illustrated
5 in Fig. 26 in a case where the configurations of the TSs,
which are multiplexed in a multiplexed stream, are the
configurations described with reference. to Fig. 28. This
decoding process, for example, is started when a
multiplexed stream that is transmitted from the encoding
10 device 180 illustrated in Fig. 22 is input to the
decoding device 200.
[0254]
The process of Steps S181 to S184 illustrated in
Fig. 30 is the same as the process of Steps S151 to S154
15 illustrated in Fig. 27, and thus the description thereof
will not be presented.
[0255]
In a case where it is determined that the
instruction for displaying a 2D image has not been made
20 by the viewer in Step S181, the image generating unit 202,
similarly to the process of Step S156 illustrated in Fig.
30, determines whether or not an instruction for
displaying a 3D image of the two-viewpoint mode has been
made by the viewer in Step S185. In a case where it is
25 determined that there is the instruction for displaying a
3D image of the two-viewpoint mode has been made by the
viewer in Step S185, the process proceeds to Step S186.
[0256]
In Step S186, the separation unit 121, similarly to
30 the process of Step S155 illustrated in Fig. 27,
separates the TSl and the TS2 from the multiplexed stream.
93
•
5
10
15
20
25
30
SP308970WOOO
Then, the separation unit 121 extracts a part of the
compatible stream that is included in the TS1 and a part
other than the compatible stream that is included in the
TS2 and supplies the extracted parts to the decoder 122.
Then, the process proceeds to Step S187.
[0257]
The process of Steps S187 and S188 is the same as
the process of Steps S157 and S158 illustrated in Fig. 27,
and thus the description thereof will not be presented.
[0258]
On the other hand, in a case where it is determined
. that the instruction for displaying a 3D image of the
two-viewpoint mode has not been made by the viewer in
Step S185, in other words, in a case where an instruction
for displaying a 3D image of the multi-viewpoint mode has
been made by the viewer, the process proceeds to Step
S189.
[0259]
In Step S189, the separation unit 121 separates the
TS3 from the multiplexed stream. Then, the separation
unit 121 extracts the auxiliary information included in
the TS3 by referring to the delimiter and the like,
supplies the image information included in the auxiliary
information to the image information acquiring unit 123
for the maintenance thereof, and supplies the viewpoint
distance information to the inter-viewpoint distance
information acquiring unit 124 for the maintenance
thereof. In addition, the separation unit 121 supplies
the parallax image information included in the auxiliary
information to the parallax image information acquiring
unit 125 for the maintenance thereof and supplies the
94
•
5
10
15
20
25
30
SP308970WOOO
compatibility information to the compatibility
information acquiring unit 126 for the maintenance
thereof. Furthermore, the separation unit 121 extracts
the auxiliary stream included in the TS3 by referring to
the delimiter and the like and supplies the extracted
auxiliary stream to the decoder 122.
[0260]
In Step S190, the separation unit 121, similarly to
the process of Step S186, separates the TS1 and the TS2
from the multiplexed stream. Then, the separation unit
121 extracts a part of the compatible stream that is
included in the TS1 and a part other than the compatible
stream that is included in the TS2 and supplies the
extracted parts to the decoder 122.
[0261]
The process of Steps S191 to S200 is the same as
the process of Steps S159 to S168 illustrated in Fig. 27,
and thus the description thereof will not be presented.
[0262]
In a decoding device that can decode only a
compatible stream that has compatibility with the
decoding device 200, the TS3 is ignored, and the process
of Steps S181 to S188 is performed.
[0263]
As above, the decoding device 200 can separate the
multiplexed stream that is multiplexed by the encoding
device 180 in accordance with a mode that has
compatibility with an existing mode.
[0264]
[Third Configuration Example of TS]
Fig. 31 is a diagram that illustrates a third
95
•
5
10
15
20
SP308970WOOO
configuration example of the TSs that are generated by
the multiplexing unit 59 illustrated in Fig. 22.
[0265]
In the example illustrated in Fig. 31, three TSs
are generated by the multiplexing unit 59, and a
compatible stream, an auxiliary stream, and auxiliary
information are included in TSs that are different from
each other. More specifically, a TSI is generated from
the compatible stream, a TS2 is generated from the
auxiliary stream, and a TS3 is generated from the
auxiliary information.
[0266]
In the example illustrated in Fig. 31, the TS3 is
independently generated only from the auxiliary
information of which the amount of information is
relatively small. Accordingly, a decoding device that
executes an application, in which there is a limitation
on the number of TSs that can be simultaneously processed,
such as a BD application can reproduce the TSI and the
TS2 in synchronization with each other by preloading the
TS3 of which the amount of information is relatively
small.
[0267]
In addition, in the example illustrated in Fig. 31,
25 since the compatible stream is stored in the TS that is
different from the TSs in which the auxiliary stream and
the auxiliary information are stored, the encoding device
can acquire the auxiliary stream and the auxiliary
information by downloading them from another device and
30 generate a multiplexed stream after the compatible stream
is generated.
96
• SP308970WOOO
[0268]
In addition, since the compatible stream is stored
in one TS, the encoding device 180 can secure the
compatibility with an encoding device that encodes a 3D
5 image of the existing two-viewpoint mode in accordance
with the MVC mode and stores the encoded image in one TS
in multiplexing.
[0269]
[Description of Third Encoding Process]
10 An encoding process that is performed by the
encoding device 180 in a case where the TSs having the
configurations described with reference to Fig. 31 is the
same as the encoding process described with reference to
Figs. 24 and 25 except that the TS1 is generated from the
15 compatible stream, the TS2 is generated from the
auxiliary stream, and the TS3 is generated from the
auxiliary information in Step S145 illustrated in Fig. 25,
and thus the description thereof will not be presented.
[0270]
20 [Description of Third Decoding Process]
Fig. 32 is a flowchart that illustrates a decoding
process performed by the decoding device 200 illustrated
in Fig. 26 in a case where the configurations of the TSs,
which are multiplexed in a multiplexed stream, are the
25 configurations described with reference to Fig. 31. This
decoding process, for example, is started when a
multiplexed stream that is transmitted from the encoding
device 180 illustrated in Fig. 22 is input to the
decoding device 200.
30 [0271]
In Step S211 illustrated in Fig. 32, the image
97
•
5
10
15
20
25
30
SP308970WOOO
generating unit 202 of the decoding device 200, similarly
to the process of Step S151 illustrated in Fig. 27,
determines whether or not an instruction for displaying a
2D image has been made by a viewer. In a case where it
is determined that the instruction for displaying a 2D
image has been made by a viewer in Step S211, the process
proceeds to Step S212.
[0272]
In Step S212, the separation unit 121 receives the
multiplexed stream that is transmitted from the encoding
device 180 and separates a TS1 from the multiplexed
stream. Then, the separation unit 121 extracts a
compatible stream that is included in the TS1 and
supplies the extracted compatible stream to the decoder
201.
[0273]
In Step S213, the compatible decoder 211 of the
decoder 201 extracts an encoded compatible image from the
compatible stream that is supplied from the separation
unit 121 and decodes the encoded compatible image in
accordance with a mode that corresponds to the MVC mode.
The compatible decoder 211 supplies an image A2 that is
acquired as a result of the decoding process to the image
generating unit 202.
[0274]
In Step S214, the image generating unit 202,
similarly to the process of Step S154 illustrated in Fig.
15, outputs the image A2 that is supplied from the
compatible decoder 211 to a display device not
illustrated in the figure, thereby displaying the image.
Then, the process ends.
98
• SP308970WOOO
[0275]
On the other hand, in a case where it is determined
that an instruction for displaying a 2D image has not
been made by the viewer in Step S211, the image
5 generating unit 202 determines whether nor not an
instruction for displaying a 3D image of the twoviewpoint
mode has been made by the viewer in. Step S215.
In a case where it is determined that the instruction for
displaying a 3D image of the two-viewpoint mode has been
10 made by the viewer in Step S215, the process proceeds to
Step S216.
[0276]
In Step S216, the separation unit 121 separates a
TS1 from the multiplexed stream. Then, the separation
15 unit 121 extracts a compatible stream that is included in
the TS1 and supplies the extracted compatible stream to
the decoder 122.
[0277]
In Step S217, the compatible decoder 211 of the
20 decoder 122 extracts an encoded compatible image from the
compatible stream that is supplied from the separation
unit 121 and decodes the encoded compatible image in
accordance with a mode that corresponds to the MVC mode.
The compatible decoder 211 supplies images A and B that
25 are compatible images acquired as a result of the
decoding process to the image generating unit 202.
[0278]
In Step S218, the image generating unit 202
alternately outputs images A1 and B1 out of the
30 compatible images that are supplied from the compatible
decoder 211, thereby displaying the images A1 and B1 on a
99
•
display device not illustrated in the figure.
process ends.
[0279]
SP308970WOOO
Then, the
On the other hand, in a case where it is determined
5 that the instruction for displaying a 3D image of the
two-viewpoint mode has been made by the viewer in Step
S215, in other words, in a case where an instruction for
displaying a 3D image of the multi-viewpoint mode has
been made by the viewer, the process proceeds to Step
10 S219.
[0280]
In Step S219, the separation unit 121 separates a
TS3 from the multiplexed stream that is output from the
encoding device 180. Then, the separation unit 121
15 extracts auxiliary information that is included in the
TS3 by referring to the delimiter and the like and
supplies image information included in the auxiliary
information to the image information acquiring unit 123
for the maintenance thereof and supplies viewpoint
20 distance information to the inter-viewpoint distance
information acquiring unit 124 for the maintenance
thereof. In addition, the separation unit 121 supplies
parallax image information included in the auxiliary
information to the parallax image information acquiring
25 unit 125 for the maintenance thereof and supplies
compatibility information to the compatibility
information acquiring unit 126 for the maintenance
thereof.
[0281]
30 In Step S220, the separation unit 121 separates a
TS1 and a TS2 from the multiplexed stream that is output
100
SP308970WOOO
from the encoding device 180. Then, the separation unit
121 supplies a compatible stream included in the TS1 and
an auxiliary stream included in the TS2 to the decoder
122.
5 [0282]
In Step S221, the compatible decoder 211 of the
decoder 122 extracts an encoded compatible image from the
compatible stream that is supplied from the separation
unit 121 and decodes the encoded compatible image in
10 accordance with a mode that corresponds to the MVC mode.
The compatible decoder 211 supplies images A and B that
are compatible images acquires as a result of the
decoding process to the image generating unit 202. Then,
the process proceeds to Step S222.
15 [0283]
The processes of Steps S222 to S230 are the same as
the processes of Steps S192 to S200 illustrated in Fig.
30, and thus the description thereof will not be
presented.
20 [0284]
In the description presented above, although the
decoding device separates the TS using the delimiter, in
a case where a PIO of each TS has a fixed value, the TS
may be separated using the PID. In such a case, since a
25 decoding device that can decode only a compatible stream
does not recognize any PIO other than the PIO of the TS
in which a compatible stream is included, the decoding
device ignores TSs other than the TS.
[0285]
30 In addition, the auxiliary information may be
configured to be encoded.
101
• SP308970WOOO
[0286]

[Configuration Example of Encoding Device According to
Fourth Embodiment]
5 Fig. 33 is a block diagram that illustrates a
configuration example of an encoding device according to
a fourth embodiment of the present technology.
[0287]
In the configuration illustrated in Fig. 33, the
10 same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 2.
Duplicate description will not be presented as is
appropriate.
[0288]
15 The configuration of the encoding device 230
illustrated in Fig. 33 is mainly different from the
configuration illustrated in Fig. 2 in that imaging units
231-1 to 231-N (here, N is a multiple of two that is four
or more), an image converting unit 232, a parallax image
20 generating unit 233, and an inter-viewpoint distance
information generating unit 234 are disposed instead of
the imaging units 51B and 51C, the image converting unit
52, the parallax image generating unit 53, and the interviewpoint
distance information generating unit 56. The
25 encoding device 230 sets an image of one viewpoint out of
multi-viewpoint images as a compatible image, sets images
of the remaining N viewpoints as auxiliary images, and
encodes the images.
[0289]
30 More specifically, in the encoding device 230, the
imaging units 231-1 to 231-N are sequentially aligned in
102
•
5
10
15
SP308970WOOO
the horizontal direction with the imaging unit 51A
disposed at the center thereof. Here, distances between
the imaging units 231-1 and 231-2 and the imaging units
231-2 and 231-3, ... , and the imaging units 231- (N-l) and
231-N are distances ~dl1, ~dI2' ... , ~dlN-l.
[0290]
The imaging units 231-1 to 231-N capture HD images
of a viewpoint different from the viewpoint of an image
Al as images PI to PN and supplies the captured images to
the image converting unit 232, the parallax image
generating unit 233, and the inter-viewpoint distance
information generating unit 234. Hereinafter, in a case
where the imaging units 231-1 to 231-N do not need to be
particularly discriminated from one another, the imaging
units will be collectively referred to as an imaging unit
231. Similarly, images PI to PN will be referred to as
an image P.
[0291 ]
A viewpoint that corresponds to the image P is a
20 viewpoint that is located on the outer side out of
viewpoints of images that can be recognized as 3D images.
Accordingly, a decoding device that corresponds to the
encoding device 230 can generate multi-viewpoint images
by interpolating images of viewpoints located on the
25 inner side from the viewpoint of the image P using the
images Al and P. As a result, multi-viewpoint images can
be generated with precision that is higher than that of a
case where images of viewpoints located on the outer side
are interpolated using the images of viewpoints located
30 on the inner side. The distances ~dl1 to ~dlN-l may
either be fixed or be changed for each time.
103
• SP308970WOOO
[0292]
The image converting unit 232 determines the image
A1, which is supplied from the imaging unit 51A of which
the position in the horizontal direction is located on
5 the inner side out of the imaging units 51A and 231, as a
compatible image. The image converting unit 232 supplies
information that designates the image A1 as a compatible
image to the compatibility information generating unit 55.
Then, the image converting unit 232 directly supplies the
10 image A1 that is the compatible image to the encoder 58.
[0293 ]
In addition, the image converting unit 232 sets
images P other than the image A1 as auxiliary images and
multiplexes the images in accordance with a predetermined
15 multiplexing mode. More specifically, for example, in a
case where the multiplexing mode is a side-by-side mode,
the image converting unit 232 halves the resolution of
the image P. Then, the image converting unit 232
multiplexes images P1, P3, ... , P(N-1) (hereinafter,
20 referred to as half-resolution odd images) of which the
resolution has halved and images P2, P4, ... , PN
(hereinafter, referred to as half-solution even images)
of which the resolution has halved such that the halfresolution
odd images are images of a left half of the
25 screen, and the half-resolution even images are images of
a right half of the screen. The image converting unit
232 supplies a multiplexed image that is acquired as a
result of the multiplexing process to the encoder 58 and
supplies information that represents a multiplexing mode
30 of the auxiliary images to the image information
generating unit 54.
104
SP308970WOOO
[0294]
The parallax image generating unit 233 detects the
disparity of each pixel of the images A1 and P using the
image A1 that is supplied from the imaging unit 51A and
5 the image P supplied from the imaging unit 231. The
parallax image generating unit 233 generates a parallax
image A1' of the image A1 that is a compatible image and
directly supplies the generated parallax image to the
encoder 58. In addition, the parallax image generating
10 unit 233 generates parallax images P1' to PN' of the
images P1 to PN that are auxiliary images and, similarly
to the image converting unit 232, multiplexes the
parallax images in accordance with a predetermined
multiplexing mode. The parallax image generating unit
15 233 supplies a multiplexed image that is acquired as a
result thereof to the encoder 58. The parallax image
generating unit 233 supplies information that represents
the multiplexing mode of the parallax images of the
auxiliary images to the parallax image information
20 generating unit 57.
[0295]
Hereinafter, in a case where the parallax images
P1' to PN' do not need to be particularly discriminated
from each other, the parallax images will be collectively
25 referred to as a parallax image P'.
[0296]
The inter-viewpoint distance information generating
unit 234 detects an inter-viewpoint distance between the
images A1 and P using the image A1 supplied from the
30 imaging unit 51A and the image P supplied from the
imaging unit 231. For example, the inter-viewpoint
105
8P308970WOOO
distance information generating unit 234 detects a
distance ddl1 between the imaging units 231-1 and 231-2
in the horizontal direction, ... , and a distance ddlN- 1
between the imaging units 231-(N-l) and 231-N in the
5 horizontal direction as inter-viewpoint distances. The
inter-viewpoint distance information generating unit 234
generates information that represents the inter-viewpoint
distances and the like as inter-viewpoint distance
information and supplies the inter-viewpoint distance
10 information to the multiplexing unit 59.
[0297 ]
[Configuration Example of T8]
Fig. 34 is a diagram that illustrates a
configuration example of T8s that are generated by a
15 multiplexing unit 59 illustrated in Fig. 33.
[0298]
In the example illustrated in Fig. 34, a T81 is
generated from the compatible stream by the multiplexing
unit 59. In addition, a T82 is generated from the
20 auxiliary stream that includes the multiplexed images of
the images P that are encoded auxiliary images, the
parallax image A' of the compatible image, and the
multiplexed image of the parallax images P' of the
auxiliary images and the auxiliary information.
25 [0299]
In the example illustrated in Fig. 34, similarly to
the case illustrated in Fig. 3, since the compatible
stream and the stream other than the compatible stream
are stored in TSs different from each other, the amount
30 of the information of the compatible stream does not need
to be reduced. Accordingly, the image quality of the
106
SP308970WOOO
compatible image can be configured to be equal to the
image quality of the 2D image that is encoded in
accordance with the existing Ave method.
[0300 ]
5 Fig. 35 is a diagram that illustrates a detailed
configuration example of the TS2 illustrated in Fig. 34.
[0301]
As illustrated in A of Fig. 35 to e of Fig. 35, in
the TS2 illustrated in Fig. 34, data is arranged in units
10 of units, and, at the head of each unit, a delimiter
(Del) is inserted.
[0302]
In the example illustrated in A of Fig. 35, a
multiplexed image (P1+P2, ... , P(N-1)+PN) of encoded
15 auxiliary images and auxiliary information (Aux Inf) of
the multiplexed image of the auxiliary images, a parallax
image (A1') of an encoded compatible image and auxiliary
information of the parallax image of the compatible image,
or a multiplexed image (P1' +P2', ... , P (N-1)' +PN') of
20 parallax images of encoded auxiliary images and auxiliary
information of the multiplexed image of the parallax
images of the auxiliary images arranged in units of
access units that are decoding units are arranged in each
unit as data.
25 [0303]
In such a case, similarly to the case illustrated
in A of Fig. 4, a decoding device that receives the TS2
can independently extract the multiplexed image of the
encoded auxiliary images, the parallax image of the
30 compatible image, and the multiplexed image of the
parallax images of the auxiliary images, which are
107
SP308970WOOO
arranged in units of access units, by extracting data in
units of the units. As a result, for example, in a case
where the decoding device generates multi-viewpoint
images using only the compatible image and the parallax
5 image of the compatible image, images that are used for
generating multi-viewpoint images can be easily extracted.
In addition, since each image of the multiplexed image of
the auxiliary images, the parallax image of the
compatible image, and the multiplexed image of the
10 parallax images of the auxiliary images and the auxiliary
information of the image are arranged in the same unit,
an image that is arranged in each unit can be
independently processed in an easy manner.
[0304]
15 In the example illustrated in B of Fig. 35, a
multiplexed image of encoded auxiliary images, a parallax
image of a compatible image, a multiplexed image of
parallax images of the auxiliary images, and auxiliary
information, which are arranged in units of access units,
20 are arranged altogether in each unit. In such a case,
similarly to the case of B of Fig. 4, a decoding device
that receives the TS2 can extract the multiplexed image
of the encoded auxiliary images, the parallax image of
the compatible image, the multiplexed image of parallax
25 images of the auxiliary images, and the auxiliary
information, which are arranged in units of access units,
altogether by extracting data in units of units.
[0305]
In the example illustrated in C of Fig. 35, a
30 multiplexed image of encoded auxiliary images and
auxiliary information of the multiplexed image of the
108
SP308970WOOO
auxiliary images or a parallax image of an encoded
compatible image, a multiplexed image of parallax images
of the auxiliary images, and auxiliary information of the
parallax image of the compatible image and the parallax
5 images of the auxiliary images, which are arranged in
units of access units, are arranged in each unit.
[0306]
In such a case, similarly to the case illustrated
in C of Fig. 4, the decoding device that receives the TS2
10 can independently extract the multiplexed image of the
encoded auxiliary images, the parallax image of the
compatible image, and the multiplexed image of the
parallax images of the auxiliary images, which are
arranged in units of access units, by extracting data in
15 units of the units. As a result, for example, in a case
where the decoder for images and the decoder for parallax
images are separately provided in the decoding device,
data can efficiently be supplied to the respective
decoders. In addition, images and parallax images can be
20 independently processed in an easy manner in the decoding
device.
[0307]
An encoding process performed by the encoding
device 230 illustrated in Fig. 33 is the same as the
25 encoding process illustrated in Figs. 6 and 7 except that
the number of viewpoints of the auxiliary images is not
two but N, and thus the description thereof will not be
presented.
[0308]
30 As above, the encoding device 230 performs
multiplexing with the compatible stream, the auxiliary
109
SP308970WOOO
stream, and the compatibility information being stored in
TSs different from each other, and accordingly, the
compatibility with an existing encoding device can be
secured which encodes a 2D image in multiplexing.
5 [0309]
In addition, since the encoding device 230 encodes
an image of one viewpoint out of multi-viewpoint images
as a compatible image in accordance with an existing
encoding mode, the compatibility with an existing
10 encoding device that encodes a 2D image in the encoding
process can be secured. In other words, the encoding
device 230 can perform encoding for multi-viewpoint
images in accordance with a mode that has compatibility
with an existing encoding mode.
15 [0310]
Furthermore, since the encoding device 230 encodes
the images A1 and the image P of N+1 viewpoints, a
decoding device that corresponds to the encoding device
230 can generate multi-viewpoint images from the images
20 A1 and the image P of N+1 viewpoints. In this way,
according to the decoding device, compared to a case
where multi-viewpoint images are generated from images of
two viewpoints, the viewpoints of images that can be
generated are not limited, and multi-viewpoint images
25 having relatively high prevision can be generated.
[0311]
In addition, since the encoding device 230 encodes
images with the resolution of the auxiliary images being
lowered, the processing cost of the encoding process and
30 the decoding process is lower than that of a case where
encoding is performed without lowering the resolution.
110
SP308970WOOO
[0312]
[Configuration Example of Decoding Device]
Fig. 36 is a diagram that illustrates a
configuration example of a decoding device that decodes a
5 multiplexed stream that is transmitted from the encoding
device 230 illustrated in Fig. 33.
[0313]
In the configuration illustrated in Fig. 36, the
same reference numeral is assigned to a configuration
10 that is the same as that illustrated in Fig. 8.
Duplicate description will not be presented as is
appropriate.
[0314]
The configuration of the decoding device 260
15 illustrated in Fig. 36 is mainly different from the
configuration illustrated in Fig. 8 in that a decoder 261
and an image generating unit 262 are disposed instead of
the decoder 122 and the image generating unit 127. The
decoding device 120 generates an image of one viewpoint
20 or multi-viewpoint images by separating and decoding the
multiplexed stream that is transmitted from the encoding
device 230 and displays the generated images on a display
device not illustrated in the figure.
[0315]
25 The decoder 261 of the decoding device 260
illustrated in Fig. 36 is configured by a compatible
decoder 131 and an auxiliary decoder 271. The auxiliary
decoder 271 (auxiliary image decoding unit) of the
decoder 261 decodes the multiplexed image of the
30 auxiliary images, the parallax image of the compatible
image, and the multiplexed image of the parallax images
111
•
5
10
SP308970WOOO
of the auxiliary images that are included in the
auxiliary stream supplied form the separation unit 121 in
accordance with a mode that corresponds to the auxiliary
encoder 62 illustrated in Fig. 33. The auxiliary decoder
271 supplies the multiplexed image of the images P that
are the auxiliary images acquired as a result of the
decoding process, the parallax image A' of the compatible
image, and the multiplexed image of the parallax images
P' that are parallax images of the auxiliary images to
the image generating unit 262.
[0316]
The image generating unit 262 outputs an image in
accordance with a displaying instruction supplied from a
viewer and displays the image on a display device not
15 illustrated in the figure. More specifically, the image
generating unit 262 (generation unit) generates images of
three or more viewpoints, which correspond to a display
device not illustrated in the figure, each having the
resolution that is a half of the resolution of the
20 compatible image or the auxiliary image by using the
image A that is a compatible image, the multiplexed image
of the images P that are auxiliary images, the parallax
image A' of the compatible image, and the multiplexed
image of the parallax images P' of the auxiliary images
25 in accordance with a viewer's instruction for displaying
a 3D image of multi-viewpoints based on the image
information supplied from the image information acquiring
unit 123, the inter-viewpoint distance information
supplied from the inter-viewpoint distance information
30 acquiring unit 124, the parallax image information
supplied from the parallax image information acquiring
112
• SP308970WOOO
unit 125, the compatibility information supplied from the
compatibility information acquiring unit 126, and the
like.
[0317]
5 When described in more detail, the image generating
unit 262 separates the parallax image P' of each
auxiliary image from the multiplexed image of the
parallax images PI' of the auxiliary images based on the
information that represents a multiplexing mode of the
10 parallax images of the auxiliary images that is included
in the parallax image information supplied from the
parallax image information acquiring unit 125. In
addition, the image generating unit 262 separates each
image P from the multiplexed image of the image P that is
15 the auxiliary images based on the information that
represents a multiplexing mode of auxiliary images that
is included in the image information supplied from the
image information acquiring unit 123.
[0318]
20 Furthermore, the image generating unit 262
determines the position of each viewpoint of the multiviewpoint
images to be generated based on the interviewpoint
distance information and the number of
viewpoints that corresponds to a display device not
25 illustrated in the figure. Then, the image generating
unit 262 generates the image of each viewpoint of which
the position is determined by using the compatible image,
each auxiliary image, the parallax image of the
compatible image, and the parallax image of each
30 auxiliary images. Then, the image generating unit 262
converts the resolution of the generated image of each
113
•
5
SP308970WOOO
viewpoint into resolution that is ~l/the number of
viewpoints" of the resolution of the compatible image or
the auxiliary image, synthesizes the images, and displays
the synthesized image on a display device not illustrated
in the figure.
[0319]
At this time, the multi-viewpoint images after the
synthesis are displayed such that the viewing angles are
different for each viewpoint, and a viewer can view a 3D
10 image without wearing glasses by seeing images of
arbitrary two viewpoints with his left and right eyes.
[0320]
In addition, the image generating unit 262 outputs
the image Al that is the compatible image supplied from
15 the compatible decoder 131 of the decoder 261 in
accordance with a viewer's instruction for displaying a
2D image, thereby displaying the image on a display
device not illustrated in the figure. Accordingly, the
viewer can view the 2D image.
20 [0321]
A decoding process performed by the decoding device
260 illustrated in Fig. 36 is the same as the decoding
process illustrated in Fig. 9 except that the auxiliary
images are images P of N viewpoints, and thus the
25 description thereof will not be presented.
[0322]
As above, the decoding device 260 can separate a
multiplexed stream that is multiplexed by the encoding
device 230 in accordance with a mode that has
30 compatibility with an existing mode. In addition, the
decoding device 260 can decode a compatible stream and an
114
• SP308970WOOO
auxiliary stream that are encoded in accordance with a
mode, which has compatibility with an existing mode and
requires relatively low process cost, by the encoding
device 230.
5 [0323]

[Configuration Example of Encoding Device According to
Fifth Embodiment]
Fig. 37 is a block diagram that illustrates a
10 configuration example of an encoding device according to
a fifth embodiment of the present technology.
[0324]
The encoding device 290 illustrated in Fig. 37 is
configured by an image conversion processing unit 291, a
15 parallax information generating unit 292, an encoding
processing unit 293, and a transmission unit 294. The
encoding device 290 encodes one of compatible images of
two viewpoints in accordance with the AVC mode, encodes
the other thereof in accordance with the MVC mode, and
20 encodes an auxiliary image of one viewpoint, a parallax
image of a compatible image, and a parallax image of the
auxiliary image in accordance with the 3DV mode. Here,
the 3DV mode is a mode that is used for encoding a
display image in accordance with a mode of multi-
25 viewpoints that is compliant with the AVC mode or the MVC
mode.
[0325]
More specifically, an L viewpoint image formed by
an L image that is a left-eye image in the two-viewpoint
30 mode as a compatible image of one viewpoint and a
parallax image (hereinafter, referred to as an L parallax
115
•
5
10
SP308970waoo
image) of the L image is input to the encoding device 290.
In addition, an R viewpoint image formed by an R image
that is a right-eye image in the two-viewpoint mode as a
compatible image of another one viewpoint and a parallax
image (hereinafter, referred to as an R parallax image)
of the R image is input to the encoding device 290.
Furthermore, an a-viewpoint image that is formed by an a
image as an auxiliary image and a parallax image
(hereinafter, referred to as an a parallax image) of the
a image is input to the encoding device 290. Here, it is
assumed that the resolutions of the L viewpoint image,
the R viewpoint image, and the a viewpoint image are the
same.
[0326]
15 The image conversion processing unit 291 of the
encoding device 290 lowers the resolutions by performing
a resolution lowering process such as a filter process
for the a image and the a parallax image that configure
the a viewpoint image that is input to the encoding
20 device 290. The image conversion processing unit 291
supplies the a viewpoint image after the resolution
lowering process to the encoding processing unit 293. In
addition, the image conversion processing unit 291
generates information that represents the type of a
25 filter used in the resolution lowering process and the
like as auxiliary image low-resolution converting
information that is information relating to the
resolution lowering process of the auxiliary image and
supplies the generated auxiliary image low-resolution
30 converting information to the transmission unit 294.
[0327]
116
5
• SP308970WOOO
The parallax information generating unit 292
(setting unit) generates parallax information that is
information relating to the parallax images of the L
parallax image, the R parallax image, and the 0 parallax
image based on information of the L image, the R image,
and the 0 image at the time of capturing the images and
the like and supplies the generated parallax information
to the encoding processing unit 293.
[0328]
10 The encoding processing unit 293 encodes the L
image included in the L viewpoint image that is input to
the encoding device 290 in accordance with the Ave mode.
In addition, the encoding processing unit 293 encodes the
R image included in the R viewpoint image with the L
15 image set as a base view (base image) and the R image set
as a non-base view (dependent image) in accordance with
the Mve mode.
[0329]
In addition, the encoding processing unit 293
20 encodes the L parallax image, the R viewpoint image, and
the 0 viewpoint image after the resolution lowering
process that is supplied from the image conversion
processing unit 291 in accordance with the 3DV mode. At
this time, in a case where the 0 viewpoint image after
25 the resolution lowering process is encoded, the encoding
processing unit 293 performs a resolution lowering
process such as a filter process for the L viewpoint
image and the R viewpoint image to be referred to and
uses the L viewpoint image and the R viewpoint image, of
30 which the resolutions are the same as the resolution of
the 0 viewpoint image, that are acquired as a result
117
• SP308970WOOO
thereof as reference images.
[0330]
The encoding processing unit 293 generates a bit
stream from encoded data of the L viewpoint image, the R
5 viewpoint image, and the 0 viewpoint image, which are
acquired as a result of the encoding process, and the
parallax information that is supplied from the parallax
information generating unit 292 and supplies the bit
stream to the transmission unit 294. In addition, the
10 encoding processing unit 293 generates information that
represents the type of a filter used in the resolution
lowering process at the time of generating the reference
images of the 0 viewpoint image and the like as reference
image low-resolution converting information that is
15 information relating to the resolution lowering process
of the reference images and supplies the generated
reference image low-resolution converting information to
the transmission unit 294.
[0331]
20 The transmission unit 294 generates TSs from the
auxiliary image low-resolution converting information
supplied from the image conversion processing unit 291
and the bit stream and the reference image low-resolution
converting information supplied from the encoding
25 processing unit 293 and transmits the generated TSs.
[0332]
[Configuration Example of Encoding Processing Unit]
Fig. 38 is a diagram that illustrates a
configuration example of the encoding processing unit 293
30 illustrated in Fig. 37.
[0333]
118
• SP308970WOOO
The encoding processing unit 293 illustrated in Fig.
38 is configured by an encoding unit 301, an additional
information generating unit 302, an addition unit 303,
and a bit stream generating unit 304.
5 [0334]
The encoding unit 301 of the encoding processing
unit 293 is configured by a compatible encoding unit 311
and an auxiliary encoding unit 312. The compatible
encoding unit 311 of the encoding unit 301 encodes the L
10 image out of the L image and the R image that are
compatible images in accordance with the Ave mode. In
addition, the compatible encoding unit 311 encodes the R
image in accordance with the Mve mode with the L image
set as a base view and the R image as a non-base view.
15 The compatible encoding unit 311 supplies encoded data of
the L image and the R image that is acquired as a result
of the encoding process to the addition unit 303.
[0335]
The auxiliary encoding unit 312 encodes the L
20 parallax image and the R parallax image, which are the
parallax images of compatible images, the 0 image that is
an auxiliary image, and an 0 parallax image that is the
parallax image of the auxiliary image in accordance with
the 3DV mode. More specifically, the auxiliary encoding
25 unit 312 encodes the 0 image in accordance with the 3DV
mode with the L image and the R image used as reference
images. At this time, the auxiliary encoding unit 312
performs a resolution lowering process for the L image
and the R image to be referred to, thereby lowering the
30 resolutions of the L image and the R image to be the same
as the resolution of the 0 image. Then, the auxiliary
119
• SP308970WOOO
encoding unit 312 encodes the 0 image by referring to the
L image and the R image after the resolution lowering
process.
[0336]
5 In addition, the auxiliary encoding unit 312
encodes the L parallax image in accordance with the 3DV
mode without referring to any image other than the
parallax image of the same viewpoint and encodes the R
parallax image in accordance with the 3DV mode by
10 referring to the L parallax image. Furthermore, the
auxiliary encoding unit 312 encodes the 0 parallax image
after the resolution lowering process in accordance with
the 3DV mode by referring to the L parallax image and the
R parallax image. At this time, the auxiliary encoding
15 unit 312 lowers the resolution of the L parallax image
and the R parallax image, which are referred to, to be
the same resolution as the resolution of the 0 parallax
image through a resolution lowering process and refers to
the L parallax image and the R parallax image after the
20 resolution lowering process.
[0337]
The auxiliary encoding unit 312 supplies encoded
data of the L parallax image, the R parallax image, and
the 0 viewpoint image that are acquired as a result of
25 the encoding process to the addition unit 303. In
addition, the auxiliary encoding unit 312 generates
information that represents the type of a filter used in
the resolution lowering process at the time of generating
the reference images of the 0 viewpoint image and the
30 like as reference image low-resolution converting
information and supplies the generated reference image
120
• SP308970WOOO
low-resolution converting information to the transmission
unit 294 illustrated in Fig. 37.
[0338]
The additional information generating unit 302
5 generates information of an SPS (sequence parameter set)
of the L image and a Subset SPS of the L parallax image,
the R viewpoint image, and the 0 viewpoint image. In
addition, the additional information generating unit 302
generates information of SEI (supplemental enhancement
10 information) of the L image, the R image, and the 0 image.
Furthermore, the additional information generating unit
302 generates information of the SEI of the L parallax
image including the parallax information of the L
parallax image, information of the SEI of the R parallax
15 image including the parallax information of the R
parallax image, and information of the SEI of the 0
parallax image including the parallax information of the
o parallax image that are supplied from the parallax
information generating unit 292 illustrated in Fig. 37.
20 The additional information generating unit 302 supplies
the information of the SPS and the SEI of the L image and
the information of the Subset SPS and the SEI of the L
parallax image, the R viewpoint image, and the 0
viewpoint image to the addition unit 303.
25 [0339]
The addition unit 303 adds an NAL header that
includes information representing the type of a picture
of the Ave mode as a type of an NAL (network abstraction
layer) unit to the encoded data of the L image that is
30 supplied from the compatible encoding unit 311, thereby
generating an NAL unit. In addition, the addition unit
121
•
5
10
15
20
25
30
SP308970WOOO
303 adds an NAL header that includes information
representing encoded data of the MVC mode as a type of
the NAL unit to the encoded data of the R image that is
supplied from the compatible encoding unit 311, thereby
generating an NAL unit.
[0340]
In addition, the addition unit 303 adds the NAL
header that includes the information representing the
encoded data of the 3DV mode as a type of the NAL unit
and includes a parallax flag representing that the image
is not a parallax image to the encoded data of the 0
image that is supplied from the auxiliary encoding unit
312, thereby generating an NAL unit. Furthermore, the
addition unit 303 adds an NAL header that includes the
information representing the encoded data of the 3DV mode
as a type of the NAL unit and includes information
relating to a parallax flag (parallax identification
information) that represents a parallax image,
information relating to a corresponding image, and the
like to the encoded data of the L parallax image, the R
parallax image, and the 0 parallax image supplied from
the auxiliary encoding unit 312, thereby generating an
NAL unit.
[0341]
Furthermore, the addition unit 303 adds the NAL
header that represents information as the type of the NAL
unit to the information of the SPS and the SEI of the L
image and the information of the Subset SPS and the SEI
of the L parallax image, the R viewpoint image, and the 0
viewpoint image that are supplied from the additional
information generating unit 302, thereby generating NAL
122
• SP308970WOOO
units. The addition unit 303 supplies the NAL units
generated as described above to the bit stream generating
unit 304.
[0342]
5 The bit stream generating unit 304 generates a bit
stream by arranging the NAL units that are supplied from
the addition unit 303 in a predetermined order and
supplies the generated bit stream to the transmission
unit 294 illustrated in Fig. 37.
10 [0343]
[Description of Encoding]
Fig. 39 is a diagram that illustrates the reference
relation in an encoding process performed by the encoding
unit 301 illustrated in Fig. 38.
15 [0344]
As illustrated in Fig. 39, the compatible encoding
unit 311 encodes the L image in accordance with the Ave
mode. In addition, the compatible encoding unit 311
encodes the R image in accordance with the Mve mode with
20 the L image set as a base view and the R image set as a
non-base view. Furthermore, the auxiliary encoding unit
312 encodes the 0 image after the resolution lowering
process in accordance with the 3DV mode by referring to
the L image and the R image after the resolution lowering
25 process.
[0345]
In addition, as illustrated in Fig. 39, the
auxiliary encoding unit 312 encodes the L parallax image
in accordance with the 3DV mode without referring to any
30 image other than the parallax image of the same viewpoint
and encodes the R parallax image in accordance with the
123
•
5
10
15
20
25
30
SP308970WOOO
3DV mode by referring to the L parallax image.
Furthermore, the auxiliary encoding unit 312 encodes the
o parallax image after the resolution lowering process in
accordance with the 3DV mode by referring to the L
parallax image and the R parallax image after the
resolution lowering process.
[0346]
[Configuration Example of NAL Unit]
Fig. 40 is a diagram that illustrates configuration
examples of the NAL units of the encoded data of the L
viewpoint image, the R viewpoint image, and the 0
viewpoint image that is included in the bit stream
generated by the encoding processing unit 293 illustrated
in Fig. 38.
[0347]
As illustrated in A of Fig. 40, for example, in a
case where an L image is encoded in accordance with the
existing AVC mode, and an L parallax image, an R
viewpoint image, and an 0 viewpoint image are encoded in
accordance with the MVC mode with the L image set as a
base view, the NAL unit of the encoded data of the L
image is configured by, an NAL header that includes
information representing the type of a picture of the AVC
mode of the L image as a type of the NAL unit and encoded
data of the L image.
[0348]
In addition, the NAL unit of encoded data of each
one of the R image, the 0 image, the L parallax image,
the R parallax image, and the 0 parallax image is
configured by an NAL header that includes information
representing the encoded data of the MVC as a type of the
124
•
5
SP308970WOOO
NAL unit (NAL unit type) and the encoded data.
Accordingly, the NAL headers of the NAL units of the
encoded data of the R image, the 0 image, the L parallax
image, the R parallax image, and the 0 parallax image,
which are encoded in accordance with the MVC mode, are
the same.
[0349]
Accordingly, the decoding device that decodes a bit
stream cannot identify the NAL units of the R image, the
10 0 image, the L parallax image, the R parallax image, and
the 0 parallax image. Therefore, a decoding device of
the existing MVC mode cannot extract and decode only an
NAL unit of the R image, which is necessary, but performs
an unnecessary decoding process.
15 [0350]
In contrast to this, as illustrated in B of Fig. 40,
in the encoding process performed by the encoding
processing unit 293, in the NAL headers of the NAL units
of the 0 image, the L parallax image, the R parallax
20 image, and the 0 parallax image, information representing
the encoded data of the 3DV mode is included as a type of
the NAL unit, and a parallax flag (is_depth) is included
as extension information (nal_unit_header_3dv_extension).
[0351]
25 More specifically, in the NAL header of the NAL
unit of the 0 image, a parallax flag that is "0"
representing that the image is not a parallax image is
included. Accordingly, the decoding device can identify
the 0 image. In addition, in the NAL headers of the NAL
30 units of the L parallax image, the R parallax image, and
the 0 parallax image, a parallax flag that is "1"
125
• SP308970WOOO
representing a parallax image is included. Accordingly,
the decoding device can identify the L parallax image,
the R parallax image, and the 0 parallax image.
[0352]
5 Furthermore, in the NAL headers of the NAL units of
the L parallax image, the R parallax image, and the 0
parallax image, information relating to corresponding
images (a compatible image and an auxiliary image) is
included as extension information. As the information
10 relating to a corresponding image, there is information
(cor_view_id) (compatible image identification
information or auxiliary image identification
information) that specifies the number of viewpoints
(num_cor_view_id) of the corresponding image and the
15 viewpoints of the corresponding image.
[0353]
Here, the L parallax image is a parallax image of
the L image of one viewpoint, the R parallax image is a
parallax image of the R image of one viewpoint, and the 0
20 parallax image is a parallax image of the 0 image of one
viewpoint. In the example illustrated in Fig. 40,
information that specifies the viewpoint of the L image
is 0, information that specifies the viewpoint of the R
image is 1, and information that specifies the viewpoint
25 of the 0 image is 2.
[0354]
Accordingly, the information relating to a
corresponding image that is included in the NAL header of
the NAL unit of the L parallax image is "1" that is the
30 number of viewpoints of the corresponding image and "0"
as information that specifies the viewpoint of the
126
•
5
10
15
20
25
30
SP308970WOOO
corresponding image. In addition, the information
relating to a corresponding image that is included in the
NAL header of the NAL unit of the R parallax image is ~1"
that is the number of viewpoints of the corresponding
image and ~1" as information that specifies the viewpoint
of the corresponding image. Furthermore, the information
relating to a corresponding image that is included in the
NAL header of the NAL unit of the 0 parallax image is ~1"
that is the number of viewpoints of the corresponding
image and ~2" as information that specifies the viewpoint
of the corresponding image.
[0355]
As above, by including the information relating to
a corresponding image as the extension information, the L
parallax image, the R parallax image, and the 0 parallax
image can be respectively identified. In addition, since
the information that specifies an image corresponding to
the parallax image, which represents correspondence
relation between the parallax image and the image, is
included not in the NAL header of the image but in the
NAL header of the parallax image, the compatibility of
the NAL unit of the compatible image can be maintained.
[0356]
In addition, the type of the NAL unit of the L
image that is a compatible image is information
representing a picture of the AVC mode, and the type of
the NAL unit of the R image is information that
represents encoded data of the MVC mode. Accordingly,
the type of the NAL unit that represents encoded data of
the 3DV mode may be described as information that
represents the head of information other than a
127
,
5
10
15
20
25
30
SP308970WOOO
compatible image, in other words, boundary information
that represents a boundary of information other than a
compatible image.
[0357]
[Configuration Example of Bit Stream]
Fig. 41 is a diagram that illustrates a
configuration example of the bit stream that is generated
by the bit stream generating unit 304 illustrated in Fig.
38.
[0358]
As illustrated in A of Fig. 41, the bit stream of
the L image, which is encoded in accordance with the AVC
mode, is configured by an NAL unit of the information of
the SPS of the AVC mode of the L image, an NAL unit of
information of the SEI of the AVC mode of the L image,
and an NAL unit of the encoded data (L color) of the L
image.
[0359]
In addition, as illustrated in B of Fig. 41, a bit
stream of the R image, which is encoded in accordance
with the MVC mode, is configured by an NAL unit of the
information of the Subset SPS of the MVC mode of the R
image, an NAL unit of the information of the SEI of the
MVC mode of the R image, and an NAL unit of the encoded
data (R color) of the R image.
[0360]
As illustrated in C of Fig. 41, the bit stream of
the 0 image, which is encoded in accordance with the 3DV
mode, is configured by an NAL unit of the information of
the Subset SPS of the 3DV mode of the 0 image, an NAL
unit of the information of the SEI of the 3DV mode of the
128
5
10
15
20
25
30
SP308970WOOO
o image, and an NAL unit of the encoded data (0 color) of
the 0 image.
[0361]
As illustrated in D of Fig. 41, the bit stream of
the L parallax image, which is encoded in accordance with
the 3DV mode, is configured by an NAL unit of the
information of the Subset SPS of the 3DV mode of the L
parallax image, an NAL unit of the information of the SEI
of the 3DV mode that includes the parallax information
(3DV_view_synthesis_info) of the L parallax image, and an
NAL unit of the encoded data (L Depth) of the L parallax
image.
[0362]
As illustrated in E of Fig. 41, the bit stream of
the R parallax image, which is encoded in accordance with
the 3DV mode, is configured by an NAL unit of the
information of the Subset SPS of the 3DV mode of the R
parallax image, an NAL unit of the information of the SEI
of the 3DV mode that includes the parallax information of
the R parallax image, and an NAL unit of the encoded data
(R Depth) of the R parallax image.
[0363]
Although not illustrated in the figure, the bit
stream of the 0 parallax image, which is encoded in
accordance with the 3DV mode, similarly to the bit stream
of the R parallax image, is configured by an NAL unit of
the information of the Subset SPS of the 3DV mode of the
o parallax image, an NAL unit of the information of the
SEI of the 3DV mode that includes the parallax
information of the 0 parallax image, and an NAL unit of
the encoded data of the 0 parallax image.
129
• SP308970WOOO
[0364]
As above, since the parallax information is
included in the SEI of the 3DV mode, a decoding device
that is not in correspondence with the 3DV mode does not
5 necessarily need to process the parallax information.
[0365]
[Configuration Example of SEI of 3DV Mode]
Fig. 42 is a diagram that illustrates a
configuration example of the SEI of the 3DV mode.
10 [0366]
As illustrated in Fig. 42, in the SEI of the 3DV
mode, parallax information (3DV_view_synthesis_info) is
included.
[0367]
15 Fig. 43 is a diagram that illustrates an example of
the description of the parallax information
(3DV_view_synthesis_info) .
[0368]
As illustrated in Fig. 43, as the parallax
20 information, a parallax type (depth_type) is described
which represents whether each pixel value of the parallax
image is a parallax value (Disparity) that represents the
parallax or a depth value that represents the depth
(Depth) of a subject corresponding to the parallax.
25 [0369]
In addition, as the parallax information, a
normalization flag (is_normalized) (normalization
identification information) is described which represents
whether or not each pixel value of the parallax image is
30 normalized within a predetermined range (for example, a
range of 0 to 255) .
130
• SP308970WOOO
[0370]
Furthermore, in a case where each pixel value of
the parallax image is a depth value, a maximal value
(z_near) of depth values that are common to all the
5 viewpoints and a minimal value (z_far) of the depth value
are described as the parallax information. On the other
hand, in a case where each pixel value of the parallax
image is a parallax value, a maximal value (max_disp) of
parallax values for each viewpoint and a minimal value
10 (min_disp) of the parallax values are described.
[0371]
In addition, as the parallax information, a
photographing mode (camera_mode) of a camera that
captures an image that corresponds to a parallax image is
15 described. As the photographing modes, there are a
parallel photographing mode (lD parallel mode) in which
photographing is performed such that distances between
cameras and a subject in the depth direction are the same
and a photographing mode with radiation (general mode) in
20 which photographing is performed with cameras being
arranged in the radiation direction from a subject.
[0372]
In addition, as the parallax information,
correspondence relation information (interval_view_id)
25 that represents an image, from which the parallax of an
image corresponding to a parallax image is the parallax
represented by the parallax image, is described.
Furthermore, as the parallax information, the number of
viewpoints (num_cor_view_id) of an image that corresponds
30 to the parallax image and information (cor_vie_id) that
specifies the image are described.
131
• SP308970WOOO
[0373]
In addition, as the parallax information,
parameters of cameras (camera_parameters) that capture
images corresponding to the parallax images are described.
5 The parameters of the cameras are configured by an
internal parameter and external parameters. The internal
parameter is formed by a focal distance of the camera in
the horizontal direction that is common to all the
viewpoints and a principal point that is the image center,
10 in other words, the position of the optical center of the
lens in the horizontal direction. The position of the
principal point in the horizontal direction may be
different for each viewpoint. In addition, the external
parameter is a parameter that defines the position of the
15 camera in the horizontal direction.
[0374]
[Description of Process of Encoding Device]
Fig. 44 is a flowchart that illustrates a multiviewpoint
encoding process performed by the encoding
20 device 290 illustrated in Fig. 37. This multi-viewpoint
image encoding process, for example, is started when an L
viewpoint image, an R viewpoint image, and an 0 viewpoint
image are input to the encoding device 290 as encoding
targets.
25 [0375]
In Step S251 illustrated in Fig. 44, the encoding
processing unit 293 of the encoding device 290 acquires
the L viewpoint image that is input to the encoding
device 290. In Step S252, the encoding processing unit
30 293 acquires the R viewpoint image that is input to the
encoding device 290. In Step S253, the image conversion
132
• SP308970WOOO
processing unit 291 acquires the 0 viewpoint image that
is input to the encoding device 290.
[0376]
In Step S254, the image conversion processing unit
5 291 performs a resolution lowering process for each of an
o image and an 0 parallax image out of images of 0
viewpoints. The image conversion processing unit 291
supplies the images of 0 viewpoints after the resolution
lowering process to the encoding processing unit 293.
10 [0377]
In Step S255, the image conversion processing unit
291 generates information that represents the type of a
filter used in the resolution lowering process of Step
S254 and the like as auxiliary image low-resolution
15 converting information and supplies the generated
auxiliary image low-resolution converting information to
the transmission unit 294.
[0378]
In Step S256, the parallax information generating
20 unit 292 generates parallax information of the L parallax
image, the R parallax image, and the 0 parallax image
based on information of the L image, the R image, and the
o image at the time of capturing the images and the like
and supplies the generated parallax information to the
25 encoding processing unit 293.
[0379]
In Step S257, the encoding processing unit 293
performs an encoding process in which the L viewpoint
image, the R viewpoint image, and the 0 viewpoint image
30 after the resolution lowering process that is supplied
from the image conversion processing unit 291 are encoded.
133
• SP308970WOOO
This encoding process will be described in detail with
reference to Fig. 45 to be described later.
[0380]
In Step S258, the transmission unit 294 generates
5 TSs from the auxiliary image low-resolution converting
information supplied from the image conversion processing
unit 291 and the reference image low-resolution
converting information and the bit stream that are
supplied from the encoding processing unit 293 and
10 transmits the TSs. Then, the process ends.
[0381]
Fig. 45 is a flowchart that illustrates the
detailed encoding process of Step S257 illustrated in Fig.
44.
15 [0382]
In Step S270 illustrated in Fig. 45, the encoding
unit 301 (Fig. 38) of the encoding processing unit 293
encodes an L viewpoint image, an R viewpoint image, and
an 0 viewpoint image after the resolution lowering
20 process. More specifically, the compatible encoding unit
311 of the encoding unit 301 encodes the L image in
accordance with the Ave mode and encodes the R image in
accordance with the Mve mode. In addition, the auxiliary
encoding unit 312 encodes an L parallax image, an R
25 parallax image, and the 0 viewpoint image after the
resolution lowering process in accordance with the 3DV
mode. The encoding unit 301 supplies encoded data of the
L viewpoint image, the R viewpoint image, and the 0
viewpoint image that are acquired as a result of the
30 encoding process to the addition unit 303.
[0383]
134
•
5
SP308970WOOO
In Step S271, the auxiliary encoding unit 312
generates reference image low-resolution converting
information and supplies the generated reference image
low-resolution converting information to the transmission
unit 294 illustrated in Fig. 37.
[0384]
20
15
10
In Step S272, the additional information generating
unit 302 generates information of the SPS and the SEI of
the L image and the information of the Subset SPS and the
SEI of the L parallax image, the R viewpoint image, and
the 0 viewpoint image using the parallax information that
is supplied from the parallax information generating unit
292 illustrated in Fig. 37 and supplies the generated
information to the addition unit 303.
[0385]
In Step S273, the addition unit 303 adds an NAL
header including the type of the NAL unit that represents
the type of a picture of the AVC mode to the encoded data
of the L image that is supplied from the compatible
encoding unit 311, thereby generating an NAL unit. The
addition unit 303 supplies the generated NAL unit of the
encoded data of the L image to the bit stream generating
unit 304.
[0386]
25 In Step S274, the addition unit 303 adds an NAL
header including the type of the NAL unit that represents
the encoded data of the MVC mode to the encoded data of
the R image that is supplied from the compatible encoding
unit 311, thereby generating an NAL unit. The addition
30 unit 303 supplies the generated NAL unit of the encoded
data of the R image to the bit stream generating unit 304.
135
•
5
10
15
20
25
30
SP308970WOOO
[0387]
In Step S275, the addition unit 303 adds an NAL
header including the type of the NAL unit that represents
the encoded data of the 3DV mode and a parallax flag
representing that the image is not a parallax image to
the encoded data of the 0 image that is supplied from the
auxiliary encoding unit 312, thereby generating an NAL
unit. The addition unit 303 supplies the generated NAL
unit of the encoded data of the 0 image to the bit stream
generating unit 304.
[0388]
In Step S276, the addition unit 303 respectively
adds an NAL header including the type of the NAL unit
that represents the encoded data of the 3DV mode, a
parallax flag representing a parallax image, and
information relating to a corresponding image to the
encoded data of the L parallax image, the R parallax
image, and the 0 parallax image that is supplied from the
auxiliary encoding unit 312, thereby generating an NAL
unit. The addition unit 303 supplies the generated NAL
unit of the encoded data of the L parallax image, the R
parallax image, and the 0 parallax image to the bit
stream generating unit 304.
[0389]
In Step S277, the addition unit 303 respectively
adds an NAL header including the type of the NAL unit
that represents corresponding information to the
information of the SPS and the SEI of the L image that is
supplied from the additional information generating unit
302 and the information of the Subset SPS and SEI of the
L parallax image, the R viewpoint image, and the 0
136
SP308970WOOO
viewpoint image, thereby generating NAL units. The
addition unit 303 supplies the generated NAL units of the
information of the SPS and SEI of the L image and the
information of the Subset SPS and the SEI of the L
5 parallax image, the R viewpoint image, and the 0
viewpoint image to the bit stream generating unit 304.
[0390]
In Step S278, the bit stream generating unit 304
generates a bit stream by arranging the NAL units
10 supplied from the addition unit 303 in a predetermined
order and supplies the generated bit stream to the
transmission unit 294 illustrated in Fig. 37. Then, the
process is returned to Step S257 illustrated in Fig. 44,
and the process proceeds to Step S258.
15 [0391]
As above, the encoding device 290 encodes
compatible images in accordance with an existing encoding
mode and adds the NAL header including the type of the
NAL unit that represents the encoded data of the 3DV mode
20 to the encoded data of the 0 image that is an auxiliary
image. Accordingly, since a decoding device that
corresponds to only an existing encoding mode can extract
only the encoded data of the compatible image that can be
decoded based on the NAL header and decode the extracted
25 data in accordance with a mode that corresponds to an
existing encoding mode, it can be stated that the
encoding device 290 performs encoding in accordance with
a mode that has compatibility with an existing mode. As
a result, for example, data for applications that
30 correspond to the Ave mode, the Mve mode, and the 3DV
mode can be encoded together and broadcasted.
137
SP308970WOOO
[Configuration Example of Decoding Device According To
AVe Mode]
[0392]
Fig. 46 is a block diagram that illustrates a
5 configuration example of a decoding device according to
the Ave mode that decodes a bit stream transmitted from
the encoding device 290 illustrated in Fig. 37.
[0393]
The decoding device 320 illustrated in Fig. 46 is
10 configured by a reception unit 321, a separation unit 322,
and an AVC decoding unit 323 and decodes the encoded data
of the L image that is included in the bit stream
transmitted from the encoding device 290.
[0394]
15 The reception unit 321 of the decoding device 320
receives a TS that is transmitted from the encoding
device 290 illustrated in Fig. 37 and supplies a bit
stream that is included in the TS to the separation unit
322.
20 [0395]
The separation unit 322 separates NAL units of the
SPS and the SEI and the encoded data of the L image and
NAL units other than the above-described NAL units from
the bit stream that is supplied from the reception unit
25 321 based on the NAL headers.
[0396]
More specifically, the separation unit 322 extracts
an NAL unit of the encoded data of the L image including
the type of the NAL unit that represents the type of the
30 picture of the AVC mode in the NAL header, an NAL unit of
the SPS of the L image including the type of the NAL unit
138
SP308970WOOO
that represents the SPS of the Ave mode in the NAL header,
and an NAL unit of the SEl of the L image including the
type of the NAL unit that represents the SEl of the Ave
mode in the NAL header from the bit stream. The
5 separation unit 322 supplies the information of the SPS
and the information of the SEl, and encoded data of the L
image that is data included in the separated NAL units to
the Ave decoding unit 323 together with type information
that represents that the data is one of the information
10 of the SPS, the information of the SEl, and the encoded
data. Here, the type information is generated based on
the type of the corresponding NAL unit.
[0397]
The AVe decoding unit 323 maintains the information
15 of the SPS and the SEl that is supplied in correspondence
with the type information representing the information of
the SPS and the information of the SEl based on the type
information that is supplied from the separation unit 322.
The AVe decoding unit 323 decodes the encoded data of the
20 L image that is supplied together with the type
information representing the encoded data from the
separation unit 322 based on the maintained information
of the SPS and the SEl in accordance with a mode that
corresponds to the AVe mode. The AVe decoding unit 323
25 outputs the L image that is acquired as a result of the
decoding process on a display device not illustrated in
the figure, thereby displaying the L image. Therefore, a
viewer can view a 2D image.
[0398]
30 [Description of Decoding Device According to Ave Mode]
Fig. 47 is a flowchart that illustrates a decoding
139
SP308970WOOO
process performed by the decoding device 320 illustrated
in Fig. 46. This decoding process, for example, is
started when a TS is transmitted from the encoding device
290 illustrated in Fig. 37.
5 [0399]
In Step S291 illustrated in Fig. 47, the reception
unit 321 of the decoding device 320 receives a TS that is
transmitted from the encoding device 290. The reception
unit 321 supplies a bit stream that is included in the TS
10 to the separation unit 322. The process of the
subsequent Steps S292 to S297 is performed for each NAL
unit that configures the bit stream.
[0400]
In Step S292, the separation unit 322 determines
15 whether or not the type of the NAL unit that is included
in the NAL header of the NAL unit is the type of the Ave
mode. In other words, the separation unit 322 determines
whether the type of the NAL unit is information
representing the type of the picture, the SPS, or the SEI
20 according to the Ave mode.
[0401]
In a case where the type of the NAL unit is
determined to be the type of the Ave mode in Step S292,
the process proceeds to Step S293. In Step S293, the
25 separation unit 322 inputs the data of the L image
included in the NAL unit, in other words, the information
of the SPS, the information of the SEI, or the encoded
data of the L image to the Ave decoding unit 323 together
with the type information.
30 [0402]
In Step S294, the Ave decoding unit 323 determines
140
SP308970WOOO
whether or not the data is encoded data of the L image
based on the type information corresponding to the data
of the L image that is supplied from the separation unit
322. In a case where the encoded data of the L image is
5 determined in Step S294, the Ave decoding unit 323
decodes the encoded data of' the L image in accordance
with a mode corresponding to the Ave mode based on the
maintained information of the SPS and the SEI in Step
S295.
10 [0403]
In Step S296, the AVe decoding unit 323 outputs the
L image that is acquired as a result of the decoding
process to a display device not illustrated in the figure,
thereby displaying the L image. Then, the process ends.
15 [0404]
On the other hand, in a case where it is determined
that the data is not the encoded data of the L image in
Step S294, in other words, the data of the L image that
is supplied from the separation unit 322 is the
20 information of the SPS or the SEI of the L image, the
process proceeds to Step S297.
[0405]
In Step S297, the Ave decoding unit 323 maintains
the information of the SEI or the SPS of the L image that
25 is supplied from the separation unit 322, and the process
ends.
[0406]
In addition, in a case where the type of the NAL
unit is determined not to be the type of the Ave mode in
30 Step S292, the NAL unit is not supplied to the Ave
decoding unit 323, and the process ends.
141
SP308970WOOO
[0407]
As described above, the decoding device 320
separates NAL units of the SPS, the SEI, and the encoded
data of the L image that is a compatible image that can
5 be decoded by the decoding device 320 and NAL units other
than the above-described NAL units from the bit stream
based on the NAL headers and decodes only encoded data
included in the separated NAL units in accordance with a
mode corresponding to an existing AVC mode. Accordingly,
10 the decoding device 320 can decode a bit stream that is
acquired as a result of an encoding process having
compatibility with an existing mode that is performed by
the encoding device 290.
[0408]
15 [Configuration Example of Decoding Device According To
MVC Mode]
Fig. 48 is a block diagram that illustrates a
configuration example of a decoding device according to
the MVC mode that decodes a bit stream transmitted from
20 the encoding device 290 illustrated in Fig. 37.
[0409]
In the configuration illustrated in Fig. 48, the
same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 46.
25 Duplicate description will not be presented as is
appropriate.
[0410]
The configuration of the decoding device 320
illustrated in Fig. 48 is mainly different from the
30 configuration illustrated in Fig. 46 in that a separation
unit 341 and an MVC decoding unit 342 are disposed
142
SP308970WOOO
instead of the separation unit 322 and the AVC decoding
unit 323. The decoding device 320 decodes the encoded
data of the L image and the R image that is included in
the bit stream transmitted from the encoding device 290.
5 [0411]
More specifically, the separation unit 341 of the
decoding device 320 separates NAL units of the SPS of the
L image, the Subset SPS of the R image, and the SEI and
the encoded data of the L image and the R image and NAL
10 units other than the above-described NAL units from the
bit stream that is supplied from the reception unit 321
based on the NAL headers.
[0412]
When described in more detail, the separation unit
15 341, similarly to the separation unit 322 illustrated in
Fig. 46, extracts NAL units of the encoded data, the SPS,
and the SEI of the L image from the bit stream. In
addition, the separation unit 341 extracts an NAL unit of
the encoded data of the R image including the type of the
20 NAL unit that represents encoded data of the MVC mode in
the NAL header, an NAL unit of the Subset SPS of the R
image including the type of the NAL unit that represents
the Subset SPS of the MVC mode in the NAL header, and an
NAL unit of the SEI of the R image including the type of
25 the NAL unit that represents the SEI of the MVC mode in
the NAL header from the bit stream.
[0413]
The separation unit 341 supplies the SPS of the L
image, the Subset SPS of the R image, and the SEI and the
30 encoded data of the L image and the R image that are data
included in the separated NAL units to the MVC decoding
143
• SP308970WOOO
unit 342 together with the type information of the data.
[0414]
The MVC decoding unit 342 maintains the information
of the SEI of the L image and the R image, the SPS of the
5 L image, and the Subset SPS of the R image based on the
type information that is supplied from the separation
unit 341. The MVC decoding unit 342, similarly to the
AVC decoding unit 323 illustrated in Fig. 46, decodes the
encoded data of the'L image that is supplied from the
10 separation unit 341 in accordance with a mode that
corresponds to the AVC mode based on the maintained
information of the SEI and the SPS of the L image.
[0415]
The MVC decoding unit 342 decodes the encoded data
15 of the R image that is supplied from the separation unit
341 together with the type information representing the
encoded data of the R image based on the maintained
information of the SEI and the Subset SPS of the R image
in accordance with a mode that corresponds to the MVC
20 mode with the L image set as a base view and the R image
set as a non-base view. The MVC decoding unit 342
alternately outputs the L image and the R image that are
acquired as a result of the decoding process on a display
device not illustrated in the figure, thereby displaying
25 the L image and the R image.
[0416]
At this time, a viewer can view a 3D image by
wearing glasses of which the left-eye shutter is open at
the time of displaying the L image and of which the
30 right-eye shutter is open at the time of displaying the R
image and seeing the L image and the R image that are
144
• SP308970WOOO
alternately displayed on the display device.
[0417]
[Description of Process of Decoding Device According to
Mve Mode]
5 Fig. 49 is a flowchart that illustrates a decoding
process performed by the decoding device 340 illustrated
in Fig. 48. This decoding process, for example, is
started when a TS is transmitted from the encoding device
290 illustrated in Fig. 37.
10 [0418]
In Step S311 illustrated in Fig. 49, the reception
unit 321 of the decoding device 340 receives the TS that
is transmitted from the encoding device 290. The
reception unit 321 supplies a bit stream that is included
15 in the TS to the separation unit 341. The process of the
subsequent Steps S312 to S317 is performed for each NAL
unit that configures the bit stream.
[0419]
In Step S312, the separation unit 341 determines
20 whether the type of the NAL unit that is included in the
NAL header of the NAL unit is the type of the AVe mode or
the type of the MVe mode. In other words, the separation
unit 341 determines whether the type of the NAL unit is
information representing the type of the picture, the SPS,
25 or the SEI according to the Ave mode or information
representing encoded data, the Subset SPS, or the SEI
according to the Mve mode.
[0420]
In a case where the type of the NAL unit is
30 determined to be the type of the Ave mode or the type of
the MVC mode in Step S312, the process proceeds to Step
145
SP308970WOOO
S313. In Step S313, the separation unit 341 inputs the
data of the L image or the R image that is included in
the NAL unit, in other words, the information of the SPS,
the information of the SEI, or the encoded data of the L
5 image, or the information of the Subset SPS, the SEI
information, or the encoded data of the R image to the
MVC decoding unit 342 together with the type information.
[0421]
In Step S314, the MVC decoding unit 342 determines
10 whether the data supplied from the separation unit 341 is
encoded data of the L image or encoded data of the R
image based on the type information. In a case where the
encoded data of the L image or the encoded data of the R
image is determined in Step S314, the MVC decoding unit
15 342 decodes the encoded data of the L image or the R
image based on the maintained SPS or the information of
the Subset SPS and the SEI in Step S315.
[0422]
In Step S316, the MVC decoding unit 342 outputs the
20 L image or the R image that is acquired as a result of
the decoding process to a display device not illustrated
in the figure so as to display the image, and the process
ends.
[0423]
25 On the other hand, in a case where it is determined
that the data is not the encoded data of the L image or
the R image in Step S314, in other words, the data
supplied from the separation unit 341 is the information
of the SPS or the SEI of the L image or the information
30 of the Subset SPS or the SEI of the R image, the process
proceeds to Step S317.
146
SP308970WOOO
[0424]
In Step S317, the MVC decoding unit 342 maintains
the information of the SEI or the SPS of the L image or
the information of the Subset SPS or the SEI of the R
5 image that is supplied from the separation unit 341, and
the process ends.
[0425]
In addition, in a case where the type of the NAL
unit is determined not to be the type of the AVC mode or
10 the type of the MVC type in Step S312, the NAL unit is
not supplied to the MVC decoding unit 342, and the
process ends.
[0426]
As described above, the decoding device 340
15 separates NAL units of the SPS, the SEI, and the encoded
data of the L image that is a compatible image that can
be decoded by the decoding device 340 and the Subset SPS,
the SEI, and the encoded data of the R image and NAL
units other than the above-described NAL units from the
20 bit stream based on the NAL headers and decodes only
encoded data included in the separated NAL units in
accordance with a mode corresponding to an existing AVC
mode or an MVC mode. Accordingly, the decoding device
340 can decode a bit stream that is acquired as a result
25 of an encoding process having compatibility with an
existing mode that is performed by the encoding device
290.
[0427]
[Configuration Example of Decoding Device According to
30 3DV Mode]
Fig. 50 is a block diagram that illustrates a
147
SP308970WOOO
configuration example of a decoding device according to
the 3DV mode that decodes a bit stream transmitted from
the encoding device 290 illustrated in Fig. 37.
[0428]
5 The decoding device 360 illustrated in Fig. 50 is
configured by a reception unit 361, a separation unit 362,
a 3DV decoding unit 363, a low-resolution image reverseconversion
processing unit 364, and an image generating
unit 365. The decoding device 360 decodes the encoded
10 data of the L viewpoint image, the R viewpoint image, and
o viewpoint image that is included in the bit stream
transmitted from the encoding device 290.
[0429]
More specifically, the reception unit 361 of the
15 decoding device 360 receives a TS that is transmitted
from the encoding device 290 illustrated in Fig. 37. The
reception unit 361 supplies the bit stream included in
the TS to the separation unit 362, supplies the reference
image low-resolution converting information to the 3DV
20 decoding unit 363, and supplies the auxiliary image lowresolution
converting information to the low-resolution
image reverse-conversion processing unit 364.
[0430]
The separation unit 362 separates each NAL unit
25 from the bit stream that is supplied from the reception
unit 361 based on the NAL headers. Described in more
detail, the separation unit 362, similarly to the
separation unit 341 illustrated in Fig. 48, extracts the
NAL units of the encoded data, the SPS, and the SEI of
30 the L image from the bit stream. In addition, the
separation unit 362, similarly to the separation unit 341,
148
SP308970WOOO
extracts the NAL units of the encoded data, the Subset
SPS, and the SEI of the R image from the bit stream.
[0431]
Furthermore, the separation unit 362 extracts NAL
5 units of the encoded data of the L parallax image, the R
parallax image, and the 0 viewpoint image that includes
the type of the NAL unit that represents the encoded data
of the 3DV mode in the NAL headers. In addition, the
separation unit 362 extracts NAL units of the Subset SPS
10 of the L parallax image, the R parallax image, and the 0
viewpoint image and the SEI of the L parallax image, the
R parallax image, and the 0 viewpoint image that include
the type of the NAL unit representing the Subset SPS and
the SEI of the 3DV mode in the NAL headers.
15 [0432]
In addition, the separation unit 362 further
separates NAL headers of the encoded data of the L
parallax image, the R parallax image, and the 0 viewpoint
image based on the extension information included in the
20 NAL headers. More specifically, the separation unit 362
extracts NAL units of which the parallax flag as the
extension information represents that the image is not a
parallax image as NAL units of the encoded data of the 0
parallax image. In addition, the separation unit 362
25 extracts NAL units of which the parallax flag as the
extension information represents that the image is a
parallax image and of which information specifies one as
the number of viewpoints of a corresponding image and
specifies the L image as information specifying the
30 viewpoint of a corresponding image as NAL units of the
encoded data of the L parallax image. Similarly, the
149
I
• SP308970WOOO
separation unit 362 extracts an NAL unit of the encoded
data of the R parallax image and an NAL unit of the
encoded data of the 0 parallax image.
[0433]
5 The separation unit 362 supplies the information of
the SPS of the L image, the information of the Subset SPS
of the L parallax image, the R viewpoint image, and the 0
viewpoint image, and the information of the SEI and the
encoded data of L viewpoint image, the R viewpoint image,
10 and the 0 viewpoint image, which are data included in the
separated NAL units, to the 3DV decoding unit 363
together with the type information of the data.
[0434]
The 3DV decoding unit 363 maintains the information
15 of the SPS and the SEI of the L image and the information
of the Subset SPS and the SEI of the L parallax image,
the R viewpoint image, and the 0 viewpoint image based on
the type information that is supplied from the separation
unit 362. The 3DV decoding unit 363, similarly to the
20 MVC decoding unit 342 illustrated in Fig. 48, decodes the
encoded data of the L image that is supplied from the
separation unit 362 based on the maintained information
of the SEI and the SPS of the L image in accordance with
a mode that corresponds to the AVC mode. In addition,
25 the 3DV decoding unit 363, similarly to the MVC decoding
unit 342, decodes the encoded data of the R image that is
supplied from the separation unit 362 based on the
maintained information of the SEI and the Subset SPS of
the R image in accordance with a mode that corresponds to
30 the MVC mode.
[0435]
150
SP308970WOOO
In addition, the 3DV decoding unit 363 decodes the
encoded data of the 0 image that is supplied from the
separation unit 362 together with the type information
representing the encoded data of the 0 image based on the
5 maintained information of the SEI and the Subset SPS of
the 0 image in accordance with a mode that corresponds to
the 3DV mode by referring to the L image and the R image.
At this time, the 3DV decoding unit 363 performs. a
resolution lowering process for the L image and the R
10 image that are referred to based on the reference image
low-resolution converting information that is supplied
from the reception unit 361 and refers to the L image and
the R image after the resolution lowering process.
[0436]
15 Furthermore, the 3DV decoding unit 363 decodes the
encoded data of the L parallax image that is supplied
form the separation unit 362 together with the type
information representing the encoded data of the L
parallax image based on the maintained information of the
20 SEI and the Subset SPS of the L parallax image in
accordance with a mode that corresponds to the 3DV mode
without referring to any image other than the parallax
image of the same viewpoint. In addition, the 3DV
decoding unit 363 decodes the encoded data of the R
25 parallax image that is supplied form the separation unit
362 together with the type information representing the
encoded data of the R parallax image based on the
maintained information of the SEI and the Subset SPS of
the R parallax image in accordance with a mode that
30 corresponds to the 3DV mode by referring to the L
parallax image.
151
•
5
10
15
SP308970WOOO
[0437]
Furthermore, the 3DV decoding unit 363 decodes the
encoded data of the 0 parallax image that is supplied
form the separation unit 362 together with the type
information representing the encoded data of the 0
parallax image based on the maintained information of the
SEI and the Subset SPS of the 0 parallax image in
accordance with a mode that corresponds to the 3DV mode
by referring to the L parallax image and the R parallax
image. At this time, the 3DV decoding unit 363 performs
a resolution lowering process for the L parallax image
and the R parallax image that are referred to based on
the reference image low-resolution converting information
that is supplied from the reception unit 361 and refers
to the L parallax image and the R parallax image after
the resolution lowering process.
[0438]
The 3DV decoding unit 363 supplies the L viewpoint
image and the R viewpoint image acquired as a result of
20 the decoding process to the image generating unit 365.
In addition, the 3DV decoding unit 363 supplies the 0
viewpoint image acquired as a result of the decoding
process to the low-resolution image reverse-conversion
processing unit 364. Furthermore, the 3DV decoding unit
25 363 supplies the parallax information that is included in
the maintained SEI of the L parallax image, the R
parallax image, and the 0 parallax image to the image
generating unit 365.
[0439]
30 The low-resolution image reverse-conversion
processing unit 364 performs a resolution increasing
152
•
5
10
15
20
SP308970WOOO
process such as a filter process that corresponds to the
resolution lowering process performed by the image
conversion processing unit 291 illustrated in Fig. 37 for
an 0 image and an 0 parallax image that configure the 0
viewpoint image supplied from the 3DV decoding unit 363
based on the auxiliary image low-resolution converting
information supplied from the reception unit 361. In
this way, the resolution of the 0 viewpoint image becomes
the same as the resolution of the L viewpoint image and
the R viewpoint image. The low-resolution image reverseconversion
processing unit 364 supplies the 0 viewpoint
image after the resolution increasing process to the
image generating unit 365.
[0440]
The image generating unit 365 performs conversion
for the L parallax image that is supplied from the 3DV
decoding unit 363 as is necessary based on the parallax
information of the L parallax image that is supplied from
the 3DV decoding unit 363.
[0441]
25
For example, in a case where the parallax type
included in the parallax information of the L parallax
image represents that each pixel value of the parallax
image is a depth value, and a value that represents
parallax that can be processed by the image generating
unit 365 is a parallax value, the image generating unit
365 converts each pixel value of the L parallax image
into a parallax value.
[0442]
30 In addition, in a case where the normalization flag
included in the parallax information of the L parallax
153
•
5
10
15
20
25
30
SP308970WOOO
image is a value representing that each pixel value is
not normalized, and the value representing parallax that
can be processed by the image generating unit 365 is a
value after normalization, the image generating unit 365
normalizes each pixel value of the" L parallax image,
thereby converting it into a value after normalization.
[0443]
Furthermore, in a case where the photographing mode
included in the parallax information of the L parallax
image is the photographing mode with radiation, and the
parallax image that can be processed by the image
generating unit 365 is a parallax image of an image
captured in the parallel photographing mode, the image
generating unit 365 converts the L parallax image into a
parallax image of the L image that is captured in the
parallel photographing mode.
[0444]
The image generating unit 365, similarly to the L
parallax image, converts the R parallax image based on
the parallax information of the R parallax image and
converts the 0 parallax image based on the parallax
information of the 0 parallax image as is necessary.
[0445]
The image generating unit 365 generates images
having the number of viewpoints that is three or more
corresponding to a display device not illustrated in the
figure based on the L parallax image, the R parallax
image, and the 0 parallax image after conversion and a
maximal value and a minimal value of the depth values or
a maximal value and a minimal value of the parallax
values that are included in the L image, the R image, the
154
•
5
10
15
20
25
30
SP308970WOOO
o image, and parallax information and, the correspondence
relation information. Then, the image generating unit
365 converts the resolution of the generated image of
each viewpoint into resolution that is "lithe number of
viewpoints" of the resolution of the compatible image or
the auxiliary image, synthesizes the images, and outputs
the synthesized image to a display device not illustrated
in the figure so as to display the image.
[0446]
At this time, the multi-viewpoint images after the
synthesis are displayed such that the viewing angles are
different for each viewpoint, and a viewer can view a 3D
image without wearing glasses by seeing images of
arbitrary two viewpoints with his left and right eyes.
[0447]
[Description of Process of Decoding Device According to
3DV Mode]
Fig. 51 is a flowchart that illustrates a decoding
process performed by the decoding device 360 illustrated
in Fig. 50. This decoding process, for example, is
started when a TS is transmitted from the encoding device
290 illustrated in Fig. 37.
[0448]
In Step S331 illustrated in Fig. 51, the reception
unit 361 of the decoding device 340 receives a TS that is
transmitted from the encoding device 290. The reception
unit 361 supplies the bit stream included in the TS to
the separation unit 362, supplies ·the reference image
low-resolution converting information to the 3DV decoding
unit 363, and supplies the auxiliary image low-resolution
converting information to the low-resolution image
155
• 8P308970WOOO
reverse-conversion processing unit 364. The process of
the subsequent 8teps 8332 to 8342 is performed for each
NAL unit that configures the bit stream.
[0449]
5 In Step S332, the separation unit 362 separates
each NAL unit based on the NAL header of the NAL unit and
supplies data of the L viewpoint image, the R viewpoint
image, or the 0 viewpoint image after the resolution
lowering process that is included in the NAL unit to the
10 3DV decoding unit 363 together with the type information.
[0450]
In Step S333, the 3DV decoding unit 363 determines
whether the data supplied from the separation unit 362
together with the type information is the encoded data of
15 the L viewpoint image, the R viewpoint image, or the 0
viewpoint image after the resolution lowering process
based on the type information. In a case where the data
is determined to be the encoded data of the L viewpoint
image, the R viewpoint image, or the 0 viewpoint image in
20 Step 8333, the process proceeds to Step 8334.
[0451 ]
In Step 8334, the 3DV decoding unit 363 decodes the
encoded data of the L viewpoint image, the R viewpoint
image, or the 0 viewpoint image after the resolution
25 lowering process based on the maintained information of
the 8PS, the Subset SPS, or the SEI.
[0452]
In Step 8335, the 3DV decoding unit 363 determines
whether the image acquired as a result of the decoding
30 process is the 0 image or the 0 parallax image after the
resolution lowering process based on the type information.
156
•
5
10
15
20
25
30
SP308970WOOO
In a case where the image acquired as a result of the
decoding process is determined to be the 0 image or the 0
parallax image after the resolution lowering process in
Step S335, the 3DV decoding unit 363 supplies the 0 image
or the 0 parallax image to the low-resolution image
reverse-conversion processing unit 364.
[0453]
In Step S336, the low-resolution image reverseconversion
processing unit 364 performs a resolution
increasing process for the 0 image or the 0 parallax
image after the resolution lowering process that is
supplied from the 3DV decoding unit 363 based on the
auxiliary image low-resolution converting information
that is supplied from the reception unit 361. In this
way, the resolution of the 0 image or the 0 parallax
image becomes the same as that of the L viewpoint image
and the R viewpoint image.
[0454]
In Step S337, the low-resolution image reverseconversion
processing unit 364 outputs the 0 image or the
o parallax image after the resolution increasing process
to the image generating unit 365, and the process ends.
[0455]
On the other hand, in a case where the image
acquired as a result of the decoding process is
determined not to be the 0 image or the 0 parallax image
after the resolution lowering process in Step S335, in
other words, the image acquired as a result of the
decoding process is the L image, the L parallax image,
the R image, or the R parallax image, the process
proceeds to Step S338.
157
•
5
SP308970WOOO
[0456]
In Step S338, the 3DV decoding unit 363 outputs the
L image, the L parallax image, the R image, or the R
parallax image that is acquired as a result of the
decoding process to the image generating unit 365, and
the process ends.
[0457]
In addition, in a case where the data is determined
not to be the encoded data of the L viewpoint image, the
10 R viewpoint image, or the 0 viewpoint image in Step S333,
in other words, the data supplied from the separation
unit 362 is the information of the SPS, the Subset SPS,
or the SEI, the process proceeds to Step S339.
[0458]
15 In Step S339, the 3DV decoding unit 363 maintains
the information of the SPS, the Subset SPS, or the SEI
that is supplied from the separation unit 362.
[0459]
In Step S340, the 3DV decoding unit 363 determines
20 whether or not the data maintained in Step S339 is the
information of the SEI. In a case where the data
maintained in Step S340 is determined to be the
information of the SEI, the process proceeds to Step S341.
[0460]
25 In Step S341, the 3DV decoding unit 363 determines
whether there is parallax information in the information
of the SEI that is maintained in Step S339. In a case
where it is determined that there is the parallax
information in Step S341, in Step S342, the 3DV decoding
30 unit 363 extracts parallax information from the
information of the SEI and outputs the extracted parallax
158
, SP308970WOOO
information to the image generating unit 365, and the
process ends.
[0461]
On the other hand, in a case where the data
5 maintained in Step S340 is determined not to be the
information of the SEI or in a case where it is
determined that there is no parallax information in Step
S341, the process ends.
[0462]
10 As described above, the decoding device 360
separates each NAL unit from the bit stream based on the
NAL header. Then, the decoding device 360 decodes the
encoded data of the compatible image that is included in
the separated NAL unit in accordance with a mode that
15 corresponds to the existing Ave mode or the Mve mode and
decodes the encoded data of the auxiliary image and the
parallax image in accordance with a mode that corresponds
to the 3DV mode. Accordingly, the decoding device 340
can decode a bit stream that is acquired as a result of
20 an encoding process performed by the encoding device 290
in accordance with a mode that has compatibility with an
existing mode.
[0463]
[Description of Encoded Data That Is Decoding Target]
25 Fig. 52 is a diagram that illustrates encoded data
that is a decoding target for the decoding device 320
illustrated in Fig. 46, the decoding device 340
illustrated in Fig. 48, and the decoding device 360
illustrated in Fig. 50.
30 [0464]
In the example illustrated in Fig. 52, it is
159
• SP308970WOOO
assumed that a bit stream encoded in the reference
relation illustrated in Fig. 39 is transmitted from the
encoding device 290.
[0465]
5 In this case, as illustrated in Fig. 52, the
decoding device 320 decodes only encoded data (Coded
slice) that is included in the NAL unit of the L image of
which the type of the NAL unit (nal_unit_type) is
information (in the example illustrated in Fig. 52, one
10 or five) that represents the type of the picture of the
AVC mode in accordance with a mode that corresponds to
the AVC mode.
[0466]
Meanwhile, the decoding device 340, as illustrated
15 in Fig. 52, similarly to the decoding device 320, decodes
the encoded data included in the NAL unit of the L image
in accordance with a mode that corresponds to the AVC
mode. In addition, the decoding device 340 decodes the
encoded data included in the NAL unit of the R image of
20 which the type of the NAL unit is information (in the
example illustrated in Fig. 52, 20) representing the
encoded data of the MVC mode in accordance with a mode
that corresponds to the MVC mode with the L image set as
a base view and the R image set as a non-base view.
25 [0467]
Furthermore, the decoding device 360, as
illustrated in Fig. 52, similarly to the decoding device
320 and the decoding device 340, decodes encoded data
included in the NAL unit of the L image in accordance
30 with a mode that corresponds to the AVC mode. In
addition, the decoding device 360, similarly to the
160
,
5
SP308970WOOO
decoding device 340, decodes encoded data that is
included in the NAL unit of the R image in accordance
with a mode corresponding to the MVC mode with the L
image set as a base view and the R image set as a nonbase
view.
[0468]
The decoding device 360 decodes encoded data
included in the NAL unit of the 0 image after the
resolution lowering process of which the type of the NAL
10 unit is information (in the example illustrated in Fig.
52, 21) representing the encoded data of the 3DV mode and
of which the extension information
(nal_unit_header_3dv_extension) is a parallax flag
(is_depth=O) representing that the image is not a
15 parallax image in accordance with a mode that corresponds
to the 3DV mode by referring to the L image and the R
image after the resolution lowering process.
[0469]
In addition, the decoding device 360 decodes
20 encoded data included in the NAL unit of the L parallax
image of which the type of the NAL unit is information
representing the encoded data of the 3DV mode and of
which the extension information is information
(cor_view_id=O) that specifies the L image as information
25 specifying an image that corresponds to a parallax flag
(is_depth=l)) representing a parallax image in accordance
with a mode that corresponds to the 3DV mode without
referring to any image other than a parallax image of the
same viewpoint.
30 [0470]
Furthermore, the decoding device 360 decodes
161
SP308970WOOO
encoded data included in the NAL unit of the R parallax
image of which the type of the NAL unit is information
representing the encoded data of the 3DV mode and of
which the extension information is information
5 (cor_view_id=1) that specifies the R image as information
specifying an image that corresponds to a parallax flag
representing a parallax image in accordance with a mode
that corresponds to the 3DV mode by referring to the L
parallax image.
10 [0471]
In addition, the decoding device 360 decodes
encoded data included in the NAL unit of the 0 parallax
image after the resolution lowering process of which the
type of the NAL unit is information representing the
15 encoded data of the 3DV mode and of which the extension
information is information (cor_view_id=2) that specifies
the 0 image as information specifying an image that
corresponds to a parallax flag representing a parallax
image in accordance with a mode that corresponds to the
20 3DV mode by referring to the L parallax image and the R
parallax image after the resolution lowering process.
[0472]

[Configuration Example of Encoding Device According to
25 Sixth Embodiment]
Fig. 53 is a block diagram that illustrates a
configuration example of an encoding device according to
a sixth embodiment of the present technology.
[0473]
30 The encoding device 380 illustrated in Fig. 53 is
configured by a multiplexing processing unit 381, a
162
•
5
10
15
SP308970WOOO
multiplexing processing unit 382, a parallax information
generating unit 383, an encoding processing unit 384, and
a transmission unit 385. In the encoding device 380,
compatible images are an L image and an R image, and
auxiliary images are an 01 image to an ON image of N
(here, N is a multiple of two) viewpoints. The encoding
device 380 multiplexes and then encodes parallax images
of the compatible images, auxiliary images of each two
viewpoints, and parallax images of the auxiliary images.
[0474]
More specifically, an L parallax image out of L
viewpoint images input to the encoding device 380 and an
R parallax image out of R viewpoint images are input to
the multiplexing processing unit 381 of the encoding
device 380. The multiplexing processing unit 381
multiplexes the L parallax image and the R parallax image
in accordance with a predetermined multiplexing mode.
[0475]
More specifically, for example, in a case where the
20 multiplexing mode is a side-by-side mode, the
multiplexing processing unit 381 halves the resolution of
each one of the L parallax image and the R parallax image.
Then, the multiplexing processing unit 381 multiplexes
the L parallax image and the R parallax image of which
25 the resolution has halved such that the L parallax image
of which the resolution has halved becomes a left-half
image of the screen, and the R parallax image of which
the resolution has halved becomes a right-half image of
the screen. In addition, for example, in a case where
30 the multiplexing mode is a top-and-bottom mode, the
multiplexing processing unit 381 multiplexes the L
163
•
5
10
15
SP308970WOOO
parallax image and the R parallax image of which the
resolution has halved such that the L parallax image of
which the resolution has halved becomes an upper-half
image of the screen, and the R parallax image of which
the resolution has halved becomes a lower-half image of
the screen.
[0476]
The multiplexing processing unit 381 supplies a
multiplexed image that is acquired as a result of the
multiplexing process to the encoding processing unit 384
as a compatible parallax image. In addition, the
multiplexing processing unit 381 generates compatible
multiplexing information that is information representing
the multiplexing mode of the L parallax image and the R
parallax image that are parallax images of the compatible
images and supplies the generated compatible multiplexing
information to the transmission unit 385.
[0477]
To the multiplexing processing unit 382, an 01
20 viewpoint image that is formed by an 01 image and 01
parallax image that is a parallax image of the 01 image,
... , and an ON viewpoint image that is formed by an ON
image and an ON parallax image that is a parallax image
of the ON image, which are input to the encoding device
25 380, are input. The multiplexing processing unit 382
respectively multiplexes the 01 image and an 02 image, an
03 image and an 04 image, ... , an O(N-l) image and the ON
image in accordance with a predetermined multiplexing
mode. Then, the multiplexing processing unit 382
30 supplies a multiplexed image that is acquired as a result
of the multiplexing process to the encoding processing
164
•
5
10
15
20
25
30
SP308970WOOO
unit 384 as an auxiliary multiplexed image.
[0478]
Similarly, the multiplexing processing unit 382
respectively multiplexes the 01 parallax image and an 02
parallax image, an 03 parallax image and an 04 parallax
image, ... , an 0(N-1) parallax image and the ON parallax
image in accordance with a predetermined multiplexing
mode. Then, the multiplexing processing unit 382
supplies a multiplexed image that is acquired as a result
of the multiplexing process to the encoding processing
unit 384 as an auxiliary parallax multiplexed image. In
addition, the multiplexing processing unit 382 generates
auxiliary multiplexing information that is information
representing the multiplexing modes of the 01 image to
the ON image that are auxiliary images and the 01
parallax image to the ON parallax image that are parallax
images of auxiliary images and supplies the generated
auxiliary multiplexing information to the transmission
unit 385.
[0479]
Hereinafter, in a case where the 01 viewpoint image
to the ON viewpoint image do not need to be particularly
discriminated from each other, the 01 to ON viewpoint
images will be collectively referred to as an 0 multiviewpoint
image. Similarly, 01 to ON images will be
referred to as an 0 multiple image, and 01 to ON parallax
images will be referred to as an 0 multi-parallax image.
[0480]
The parallax information generating unit 383
generates parallax information of the L parallax image,
the R parallax image and the 0 multi-parallax image based
165
• SP308970WOOO
on information acquired at the time of capturing the L
image, the R image, and the 0 multi-image and supplies
the generated parallax information to the encoding
processing unit 384.
5 [0481]
The encoding processing unit 384, similarly to the
encoding processing unit 293 illustrated in Fig. 37,
encodes the L image included in the L viewpoint image
that is input to the encoding device 380 in accordance
10 with the Ave mode. In addition, the encoding processing
unit 384, similarly to the encoding processing unit 293,
encodes the R image included in the R viewpoint image
input to the encoding device 380 with the L image set as
a base view and the R image set as a non-base view in
15 accordance with the Mve mode.
[0482]
Furthermore, the encoding processing unit 384
encodes an auxiliary multiplexed image supplied from the
multiplexing processing unit 382 in accordance with the
20 3DV mode by referring to the L image and the R image. At
this time, the encoding processing unit 384 replicates
the L image and the R image that are referred to,
performs multiplexing in accordance with a multiplexing
mode that is the same as that of the auxiliary
25 multiplexed image, and refers to a multiplexed image that
is acquired as a result of the multiplexing process.
[0483]
In addition, the encoding processing unit 384
encodes the compatible multiplexed image that is supplied
30 from the multiplexing processing unit 381 in accordance
with the 3DV mode without referring to any image other
166
SP308970WOOO
than the compatible multiplexed image of the same
viewpoint. Furthermore, the encoding processing unit 384
encodes the auxiliary parallax multiplexed image that is
supplied from the multiplexing processing unit 382 in
5 accordance with the 3DV mode.
[0484]
The encoding processing unit 384 generates a bit
stream from the encoded data of the L image, the R image,
the compatible multiplexed image, the auxiliary
10 multiplexed image, and the auxiliary parallax multiplexed
image that is acquired as a result of the encoding
process and the parallax information that is supplied
from the parallax information generating unit 383 and
supplies the generated bit stream to the transmission
15 unit 385. In addition, the encoding processing unit 384
generates reference image multiplexing information that
is information representing the multiplexing mode at the
time of generating a reference image of the compatible
multiplexed image and supplies the generated reference
20 image multiplexing information to the transmission unit
385.
[0485]
The transmission unit 385 generates a TS from the
compatible multiplexing information that is supplied from
25 the multiplexing processing unit 381, the auxiliary
multiplexing information that is supplied from the
multiplexing processing unit 382, and the bit stream and
the reference image multiplexing information that are
supplied from the encoding processing unit 384 and
30 transmits the generated TS.
[0486]
167
SP308970WOOO
[Configuration Example of Encoding Processing Unit]
Fig. 54 is a block diagram that illustrates a
configuration example of the encoding processing unit 384
illustrated in Fig. 53.
5 [0487]
In the configuration illustrated in Fig. 54, the
same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 38.
Duplicate description will not be presented as is
10 appropriate.
[0488]
The configuration of the encoding processing unit
384 illustrated in Fig. 54 is mainly different from the
configuration illustrated in Fig. 38 in that an encoding
15 unit 401, an additional information generating unit 402,
and an addition unit 403 are disposed instead of the
encoding unit 301, the additional information generating
unit 302, and the addition unit 303.
[0489]
20 The encoding unit 401 of the encoding processing
unit 384 is configured by a compatible encoding unit 311
and an auxiliary encoding unit 411. The auxiliary
encoding unit 411 of the encoding unit 401 encodes the
compatible multiplexed image, the auxiliary multiplexed
25 image, and the auxiliary parallax multiplexed image in
accordance with the 3DV mode. The auxiliary encoding
unit 411 supplies encoded data of the compatible
multiplexed image, the auxiliary multiplexed image, and
the auxiliary parallax multiplexed image that is acquired
30 as a result of the encoding process to the addition unit
403. In addition, the encoding unit 401 generates
168
• SP308970WOOO
reference image multiplexing information and supplies the
generated reference image multiplexing information to the
transmission unit 385 illustrated in Fig. 53.
[0490]
5 The additional information generating unit 402
generates the information of the SPS of the L image and
the information of the Subset SPS of the R image, the
compatible multiplexed image, the auxiliary multiplexed
image, and the auxiliary parallax multiplexed image. In
10 addition, the additional information generating unit 402
generates the information of the SEI of the L image, the
R image, and the auxiliary multiplexed image.
Furthermore, the additional information generating unit
402 generates information of the SEI of the compatible
15 multiplexed image that includes the parallax information
of the L parallax image and the R parallax image supplied
from the parallax information generating unit 383
illustrated in Fig. 53.
[0491]
20 In addition, the additional information generating
unit 402 generates information of the SEI of the
auxiliary parallax multiplexed image that includes the
parallax information of the 0 multi-parallax image of two
viewpoints configuring the auxiliary parallax multiplexed
25 image supplied from the parallax information generating
unit 383. The additional information generating unit 402
supplies the information of the SPS of the L image, the
information of the Subset SPS of the R image, the
compatible multiplexed image, the auxiliary multiplexed
30 image, and the auxiliary parallax multiplexed image, and
the information of the SEI of the L image, the R image,
169
•
5
10
15
20
25
30
SP308970WOOO
the compatible multiplexed image, the auxiliary
multiplexed image, and the auxiliary parallax multiplexed
image to the addition unit 403.
[0492]
The addition unit 403, similarly to the addition
unit 303 illustrated in Fig. 38, adds an NAL header that
includes information representing the type of the picture
of the Ave mode as the type of the NAL unit to the
encoded data of the L image that is acquired as a result
of the encoding process performed by the compatible
encoding unit 311, thereby generating an NAL unit. In
addition, the addition unit 403, similarly to the
addition unit 303, adds an NAL header that includes
information representing encoded data of the Mve mode as
the type of the NAL unit to the encoded data of the R
image that is acquired as a result of the encoding
process performed by the compatible encoding unit 311,
thereby generating an NAL unit.
[0493]
In addition, the addition unit 403 adds an NAL
header that includes information representing the encoded
data of the 3DV mode as the type of the NAL unit and
includes a parallax flag that represents that the image
is not a parallax image to the encoded data of the
auxiliary multiplexed image supplied from the auxiliary
encoding unit 411, thereby generating an NAL unit.
Furthermore, the addition unit 403 respectively adds an
NAL header that includes information representing the
encoded data of the 3DV mode as the type of the NAL unit
and includes information relating to an image that
corresponds to the parallax flag representing a parallax
170
• SP308970WOOO
image to the encoded data of the compatible multiplexed
image and the auxiliary parallax multiplexed image that
is supplied from the auxiliary encoding unit 411, thereby
generating NAL units.
5 [0494]
In addition, the addition unit 403 respectively
adds a NAL header that represents information as the type
of the NAL unit to the information of the SPS of the L
image that is supplied from the additional information
10 generating unit 402, the information of the Subset SPS of
the R image, the compatible multiplexed image, the
auxiliary multiplexed image, and the auxiliary parallax
multiplexed image, and the information of the SEI of the
L image, the R image, the compatible multiplexed image,
15 the auxiliary multiplexed image, and the auxiliary
parallax multiplexed image, thereby generating NAL units.
The addition unit 403 supplies the NAL unit generated as
above to the bit stream generating unit 304.
[0495 ]
20 [Configuration Example of NAL Unit]
Fig. 55 is a diagram that illustrates configuration
examples of the NAL units of the encoded data of the L
image, the R image, the auxiliary multiplexed image, the
compatible multiplexed image, and the auxiliary parallax
25 multiplexed image in a bit stream that is generated by
the encoding processing unit 384 illustrated in Fig. 53.
[0496]
As illustrated in Fig. 55, the NAL units of the
encoded data of the L image and the encoded data of the R
30 image are the same as those of the case illustrated in
Fig. 40, and thus the description thereof will not be
171
30
• SP308970WOOO
presented.
[0497]
As illustrated in Fig. 55, in the encoding process
performed by the encoding processing unit 384, in the NAL
5 header of the NAL unit of the auxiliary multiplexed image,
information representing encoded data of the 3DV mode as
the type of the NAL unit is included, and a parallax flag
of "0" representing that the image is not a parallax
image is included as the extension information.
10 Accordingly, an auxiliary multiplexed image can be
identified by the decoding device.
[0498]
In addition, in the NAL headers of the NAL units of
the compatible multiplexed image and each auxiliary
15 parallax multiplexed image, a parallax flag of "1" that
represents a parallax image is included. Accordingly,
the compatible multiplexed image and the auxiliary
parallax multiplexed image can be identified by the
decoding device.
20 [0499]
Furthermore, in the NAL headers of the NAL units of
the compatible multiplexed image and each auxiliary
parallax multiplexed image, information relating to a
corresponding image is included as the extension
25 information. Here, the compatible multiplexed image is a
parallax image of the L image and the R image of two
viewpoints, and each auxiliary parallax multiplexed image
is a parallax image of the 0 multi-images of two
viewpoints. In the example illustrated in Fig. 55,
information that specifies the viewpoint of the L image
is 0, information that specifies the viewpoint of the R
172
fII
5
• SP308970WOOO
image is 1, and information that specifies viewpoints of
the 0 multi-image of two viewpoints corresponding to the
auxiliary parallax multiplexed image is 2 and 3.
[0500]
Accordingly, the information relating to a
corresponding image that is included in the NAL header of
the NAL unit of the compatible multiplexed image is 2
that is the number of viewpoints of the corresponding
image and 0 and 1 as the information that specifies the
10 viewpoint of the corresponding image. In addition, the
information relating to a corresponding image that is
included in the NAL header of the NAL unit of the
auxiliary parallax multiplexed image is 2 that is the
number of viewpoints of the corresponding image and 2 and
15 3 as information that specifies the viewpoints of the
corresponding image.
[0501]
As above, since the information relating to a
corresponding image is included as the extension
20 information, the compatible multiplexed image and each
auxiliary parallax multiplexed image can be respectively
identified.
[0502]
25
[Configuration Example of Bit Stream]
Fig. 56 is a diagram that illustrates a
configuration example of the bit stream that is generated
by the bit stream generating unit 304 illustrated in Fig.
54.
[0503]
30 A bit stream of the L image that is illustrated in
A of Fig. 56 and a bit stream of the R image that is
173
• SP308970WOOO
illustrated in B of Fig. 56 are the same as the bit
stream of the L image that is illustrated in A of Fig. 41
and the bit stream of the R image that is illustrated in
B of Fig. 41, and thus the description thereof will not
5 be presented.
[0504]
As illustrated in C of Fig. 56, a bit stream of the
auxiliary multiplexed image that is encoded in accordance
with the 3DV mode is configured by an NAL unit of the
10 information of the Subset SPS of the 3DV mode of the
auxiliary multiplexed image, an NAL unit of the
information of the SEI of the 3DV mode of the auxiliary
multiplexed image, and an NAL unit of encoded data of the
auxiliary multiplexed image.
15 [0505]
As illustrated in D of Fig. 56, a bit stream of the
compatible multiplexed image that is encoded in
accordance with the 3DV mode is configured by an NAL unit
of the information of the Subset SPS of the 3DV mode of
20 the compatible multiplexed image, an NAL unit of the
information of the SEI of the 3DV mode that includes
parallax information of the compatible multiplexed image,
and an NAL unit of encoded data of the compatible
multiplexed image.
25 [0506]
As illustrated in E of Fig. 56, a bit stream of the
auxiliary parallax multiplexed image that is encoded in
accordance with the 3DV mode is configured by an NAL unit
of the information of the Subset SPS of the 3DV mode of
I~.
~
I
30 the auxiliary parallax multiplexed image, an NAL unit of
the information of the SEI of the 3DV mode that includes
174
F
:f
If
fl f~ "ft,
II
25
SP308970WOOO
parallax information of the auxiliary parallax
multiplexed image, and an NAL unit of encoded data of the
auxiliary parallax multiplexed image.
[0507]
5 As above, since the parallax information is
included in the SEI of the 3DV mode, a decoding device
that is not in correspondence with the 3DV mode does not
unnecessarily need to process the parallax information.
[0508]
10 [Description of Process of Encoding Device]
Fig. 57 is a flowchart that illustrates a multiviewpoint
encoding process performed by the encoding
device 380 illustrated in Fig. 53. This multi-viewpoint
image encoding process, for example, is started when an L
15 viewpoint image, an R viewpoint image, and an 0 multiviewpoint
image are input to the encoding device 380 as
encoding targets.
[0509]
In Step S361 illustrated in Fig. 57, the encoding
20 processing unit 384 of the encoding device 380 acquires
an L image included in the L viewpoint image that is
input to the encoding device 380, and the multiplexing
processing unit 382 acquires an L parallax image.
[0510]
In Step S362, the encoding processing unit 384
acquires an R image included in the R viewpoint image
that is input to the encoding device 380, and the
multiplexing processing unit 382 acquires an R parallax
image. In Step S363, the multiplexing processing unit
30 382 acquires an 0 multi-viewpoint image that is input to
the encoding device 380.
175
I~
II!I
II
~~~~
I!
• SP308970WOOO
[0511]
In Step S364, the multiplexing processing unit 381
multiplexes the L parallax image acquired in Step S361
and the R parallax image acquired in Step S362 in
5 accordance with a predetermined multiplexing mode and
supplies a multiplexed image that is acquired as a result
of the multiplexing process to the encoding processing
unit 384 as a compatible parallax image. In Step S365,
the multiplexing processing unit 381 generates compatible
10 multiplexing information and supplies the generated
compatible multiplexing information to the transmission
unit 385.
[0512]
In Step S366, the multiplexing processing unit 382
15 multiplexes 0 multi-images that configure the 0 multiviewpoint
image and the 0 multi-parallax image in
accordance with a predetermined multiplexing mode for
each two viewpoints. The multiplexing processing unit
382 supplies the multiplexed image of the 0 multi-images
20 acquired as a result of the multiplexing process as an
auxiliary multiplexed image and the multiplexed image of
the 0 multi-parallax image as an auxiliary parallax
multiplexed image to the encoding processing unit 384.
[0513]
25 In Step S367, the multiplexing processing unit 382
generates auxiliary multiplexing information and supplies
the generated auxiliary multiplexing information to the
transmission unit 385.
[0514]
30 In Step S368, the parallax information generating
unit 383 generates parallax information of the L parallax
176
I
iLi!:
,
••
~
~III
,
.~
III
SP308970WOOO
image, the R parallax image, and the 0 multi-parallax
image based on the information acquired at the time of
capturing the L image, the R image, and the 0 multiimages
and the like and supplies the generated parallax
5 information to the encoding processing unit 384.
[0515]
In Step S369, the encoding processing unit 384
performs an encoding process in which the L image, the R
image, the compatible multiplexed image supplied from the
10 multiplexing processing unit 381, the auxiliary
multiplexed image and the auxiliary parallax multiplexed
image supplied from the multiplexing processing unit 382
are encoded. This encoding process will be described in
detail with reference to Fig. 58 to be described later.
15 [0516]
In Step S370, the transmission unit 385 generates a
TS from the compatible multiplexing information supplied
from the multiplexing processing unit 381, the auxiliary
multiplexing information supplied from the multiplexing
20 processing unit 382, and the reference image multiplexing
information and the bit stream that are supplied from the
encoding processing unit 384 and transmits the generated
TS. Then, the process ends.
[0517]
25 Fig. 58 is a flowchart that illustrates a detailed
encoding process of Step S369 represented in Fig. 57.
[0518]
In Step S390 illustrated in Fig. 58, the encoding
unit 401 (Fig. 54) of the encoding processing unit 384
30 encodes the L image, the R image, the compatible
multiplexed image, the auxiliary multiplexed image, and
177
•
5
10
15
20
25
30
SP308970WOOO
the auxiliary parallax multiplexed image. More
specifically, the compatible encoding unit 311 of the
encoding unit 401 encodes the L image in accordance with
the AVC mode and encodes the R image in accordance with
the MVC mode. In addition, the auxiliary encoding unit
411 encodes the compatible multiplexed image, the
auxiliary multiplexed image, and the auxiliary parallax
multiplexed image in accordance with the 3DV mode. The
encoding unit 401 supplies encoded data of the L image,
the R image, the compatible multiplexed image, the
auxiliary multiplexed image, and the auxiliary parallax
multiplexed image that is acquired as a result of the
encoding process to the addition unit 403.
[0519]
In Step S391, the auxiliary encoding unit 411
generates reference image multiplexing information and
supplies the generated reference image multiplexing
information to the transmission unit 385 illustrated in
Fig. 53.
[0520]
In Step S392, the additional information generating
unit 402 generates the information of the SPS and the SEI
of the L image and the information of the Subset SPS and
the SEI of the R image, the compatible multiplexed image,
the auxiliary multiplexed image, and the auxiliary
parallax multiplexed image using the parallax information
that is supplied from the parallax information generating
unit 383 illustrated in Fig. 53 and supplies the
generated information to the addition unit 303.
[0521]
The process of Steps S393 and S394 is the same as
178
SP308970WOOO
the process of Steps S273 and S274 illustrated in Fig. 45,
and thus the description thereof will not be presented.
[0522]
After the process of Step S394, in Step 8395, the
5 addition unit 403 adds an NAL header that includes the
type of the NAL unit that represents encoded data
according to the 3DV mode and a parallax flag that
represents that the image is not a parallax image to the
encoded data of the auxiliary multiplexed image that is
10 supplied from the auxiliary encoding unit 411, thereby
generating an NAL unit. The addition unit 403 supplies
the generated NAL unit of the encoded data of the
auxiliary multiplexed image to the bit stream generating
unit 304.
15 [0523]
In Step S396, the addition unit 403 respectively
adds an NAL header that includes the type of the NAL unit
that represents encoded data according to the 3DV mode, a
parallax flag that represents a parallax image, and
20 information relating to a corresponding image to the
encoded data of the compatible multiplexed image and the
encoded data of the auxiliary parallax multiplexed image
that are supplied from the auxiliary encoding unit 411,
thereby generating NAL units. The addition unit 403
25 supplies the generated NAL units of the encoded data of
the compatible multiplexed image and the auxiliary
parallax multiplexed image to the bit stream generating
unit 304.
[0524]
30 In Step S397, the addition unit 403 respectively
adds an NAL header that includes the type of the NAL unit
179
SP308970WOOO
represents corresponding information to the information
of the SPS and the SEI of the L image that is supplied
from the additional information generating unit 402 and
the information of the Subset SPS and the SEI of the R
5 image, the compatible multiplexed image, the auxiliary
multiplexed image, and the auxiliary parallax multiplexed
image, thereby generating NAL units. The addition unit
403 supplies the generated NAL units of the SPS and the
SEI of the L image and the Subset SPS and the SEI of the
10 R image, the compatible multiplexed image, the auxiliary
multiplexed image, and the auxiliary parallax multiplexed
image to the bit stream generating unit 304.
[0525]
In Step S398, the bit stream generating unit 304,
15 similarly to the process of Step S278 illustrated in Fig.
45, arranges the NAL units supplied from the addition
unit 403 in a predetermined order, thereby generating a
bit stream. Then, the bit stream generating unit 304
supplies the bit stream to the transmission unit 385
20 illustrated in Fig. 53 and returns the process to Step
S369 illustrated in Fig. 57. Accordingly, the process
proceeds to Step S370.
[0526]
As described above, the encoding device 380 encodes
25 compatible images in accordance with an existing encoding
mode and adds the NAL header that includes the type of
the NAL unit representing the encoded data according to
the 3DV mode to the encoded data of the 0 multi-images
that are auxiliary images. Accordingly, since a decoding
30 device that corresponds to only an existing encoding mode
can extract only the encoded data of the compatible image
180
SP308970WOOO
that can be decoded based on the NAL header and decode
the extracted data in accordance with a mode that
corresponds to an existing encoding mode, it can be
stated that the encoding device 380 performs encoding in
5 accordance with a mode that has compatibility with an
existing mode.
[0527]
Although the encoding device 290 and the encoding
device 380 described above encode the parallax image of
10 the compatible image without referring to any image other
than the parallax image of the same viewpoint, the
encoding may be performed by additionally referring to an
image of the same viewpoint.
[0528]
15 In addition, although the encoding device 380 has
been described to directly encode the compatible images,
the compatible image may be multiplexed and then encoded.
[0529]
[Configuration Example of Decoding Device According to
20 3DV Mode]
Fig. 59 is a block diagram that illustrates a
configuration example of the decoding device according to
the 3DV mode that decodes a bit stream transmitted from
the encoding device 380 illustrated in Fig. 53.
25 [0530]
The decoding device 420 illustrated in Fig. 59 is
configured by a reception unit 421, a separation unit 422,
a 3DV decoding unit 423, a separation unit 424, a
separation unit 425, and an image generating unit 426.
30 The decoding device 420 decodes the encoded data of the L
image, the R image, the auxiliary multiplexed image, the
181
SP308970WOOO
compatible multiplexed image, and the auxiliary parallax
multiplexed image that are included in the bit stream
transmitted from the encoding device 380.
[0531]
5 More specifically, the reception unit 421 of the
decoding device 420 receives a TS that is transmitted
from the encoding device 380 illustrated in Fig. 53. The
reception unit 421 supplies a bit stream included in the
TS to the separation unit 422 and supplies reference
10 image multiplexing information to the 3DV decoding unit
423. In addition, the reception unit 421 supplies
compatible multiplexing information that is included in
the TS to the separation unit 424 and supplies the
auxiliary multiplexing information to the separation unit
15 425.
[0532]
The separation unit 422 separates each NAL unit
from the bit stream that is supplied from the reception
unit 421 based on the NAL headers. When described in
20 more detail, the separation unit 422, similarly to the
separation unit 362 illustrated in Fig. 50, extracts the
NAL units of the encoded data, the SPS, and the SEI of
the L image from the bit stream. In addition, the
separation unit 422, similarly to the separation unit 362,
25 extracts the NAL units of the encoded data, the Subset
SPS, and the SEI of the R image from the bit stream.
[0533]
Furthermore, the separation unit 422 extracts NAL
units, which include the type of the NAL unit
30 representing the encoded data according to the 3DV in the
NAL header, of encoded data of the compatible multiplexed
182
,
5
10
SP308970WOOO
image, the auxiliary multiplexed image, and the auxiliary
parallax multiplexed image. In addition, the separation
unit 422 extracts NAL units, which include the type of
the NAL unit that represents the Subset SPS and the SEI
according to the 3DV in the NAL header, of the Subset SPS
of the compatible multiplexed image, the auxiliary
multiplexed image, and the auxiliary parallax multiplexed
image and the SEI of the compatible multiplexed image,
the auxiliary multiplexed image, and the auxiliary
parallax multiplexed image.
[0534]
Furthermore, the separation unit 422 further
separates the NAL headers of the encoded data of the
auxiliary mUltiplexed image, the compatible multiplexed
15 image, and the auxiliary parallax multiplexed image based
on the extension information included in the NAL header.
More specifically, the separation unit 422 extracts an
NAL unit of which the extension information is a parallax
flag that represents that the image is not a parallax
20 image as an NAL unit of the encoded data of the auxiliary
multiplexed image. In addition, the separation unit 422
extracts an NAL unit of which the extension information
is a parallax flag that represents a parallax image, of
which the number of viewpoints of a corresponding image
25 is 2, and of which information specifies the L image and
the R image as information that specifies the viewpoints
of a corresponding image as a NAL unit of the encoded
data of the compatible multiplexed image. Similarly, the
separation unit 422 extracts an NAL unit of the encoded
30 data of the auxiliary parallax multiplexed image.
[0535]
183
•
5
10
15
20
25
30
SP308970WOOO
The separation unit 422 supplies the information of
the SPS of the L image, the information of the Subset SPS
of the R image, the compatible multiplexed image, the
auxiliary multiplexed image, and the auxiliary parallax
multiplexed image, and the information of the SEI and the
encoded data of the L image, the R image, the auxiliary
multiplexed image, the compatible multiplexed image, and
the auxiliary parallax multiplexed image to the 3DV
decoding unit 423, which are data included in the
separated NAL units, together with the type information
of the data.
[0536]
The 3DV decoding unit 423 maintains the SPS of the
L image, the Subset SPS of the R image, the compatible
multiplexed image, the auxiliary multiplexed image, and
the auxiliary parallax multiplexed image, and the
information of the SEI of the L image, the R image, the
auxiliary multiplexed image, the compatible multiplexed
image, and the auxiliary parallax multiplexed image based
on the type information that is supplied from the
separation unit 422.
[0537]
The 3DV decoding unit 423, similarly to the 3DV
decoding unit 363 illustrated in Fig. 50, decodes the
encoded data of the L image that is supplied from the
separation unit 422 based on the maintained information
of the SEI and the SPS of the L image in accordance with
a mode that corresponds to the Ave mode. In addition,
the 3DV decoding unit 423, similarly to the 3DV decoding
unit 363, decodes the encoded data of the R image that is
supplied from the separation unit 422 based on the
184
• SP308970WOOO
maintained information of the SEI and the Subset SPS of
the R image in accordance with a mode that corresponds to
the MVC mode.
[0538]
5 Furthermore, the 3DV decoding unit 423 decodes the
encoded data of the auxiliary multiplexed image that is
supplied from the separation unit 422 together with the
type information representing the encoded data of the
auxiliary multiplexed image based on the maintained
10 information of the SEI and the Subset SPS of the
auxiliary multiplexed image in accordance with a mode
that corresponds to the 3DV mode by referring to the L
image and the R image. At this time, the 3DV decoding
unit 423 replicates and multiplexes the L image and the R
15 image that are referred to based on the reference image
multiplexing information that is supplied from the
reception unit 421 and refers to a multiplexed image that
is acquired as a result of the multiplexing process.
[0539]
20 Furthermore, the 3DV decoding unit 423 decodes the
encoded data of the compatible multiplexed image that is
supplied from the separation'unit 422 together with the
type information representing the encoded data of the
compatible multiplexed image based on the maintained
25 information of the SEI and the Subset SPS of the
compatible multiplexed image in accordance with a mode
that corresponds to the 3DV mode without referring to any
image other than the parallax image of the same viewpoint.
[0540]
30 In addition, the 3DV decoding unit 423 decodes the
encoded data of the auxiliary parallax multiplexed image
185
SP308970WOOO
that is supplied form the separation unit 422 together
with the type information representing the encoded data
of the auxiliary parallax multiplexed image based on the
maintained information of the SEI and the Subset SPS of
5 the auxiliary parallax multiplexed image in accordance
with a mode that corresponds to the 3DV mode by referring
to the compatible multiplexed image.
[0541]
The 3DV decoding unit 423 ~upplies the L image and
10 the R image acquired as a result of the decoding process
to the image generating unit 426, supplies the compatible
multiplexed image to the separation unit 424, and
supplies the auxiliary multiplexed image and the
auxiliary parallax multiplexed image to the separation
15 unit 425. In addition, the 3DV decoding unit 423
supplies the parallax information that is included in the
maintained SEI of the compatible multiplexed image and
the auxiliary parallax multiplexed image to the image
generating unit 426.
20 [0542]
The separation unit 424 separates an L parallax
image and an R parallax image of which the resolutions
are the same as those of the L image and the R image from
the compatible multiplexed image that is supplied from
25 the 3DV decoding unit 423 based on the compatible
multiplexing information supplied from the reception unit
421. More specifically, the separation unit 424
separates the L parallax image and the R parallax image
of which the resolutions have halved from the compatible
30 multiplexed image and performs a resolution increasing
process for the L parallax image and the R parallax image,
186
5
10
15
SP308970WOOO
thereby acquiring the L parallax image and the R parallax
image of which the resolutions are the same as those of
the L image and the R image. The separation unit 424
supplies the L parallax image and the R parallax image to
the image generating unit 426.
[0543]
The separation unit 425, similarly to the
separation unit 424, separates 0 multi-images from the
auxiliary multiplexed image that is supplied from the 3DV
decoding unit 423 and separates the 0 multi-parallax
images from the auxiliary parallax multiplexed image
based on the auxiliary image multiplexed information that
is supplied from the reception unit 421. The separation
unit 425 supplies 0 multi-viewpoint images acquired as a
result thereof to the image generating unit 426.
[0544]
The image generating unit 426, similarly to the
image generating unit 365 illustrated in Fig. 50 performs
conversion for the L parallax image and the R parallax
20 image that are supplied from the separation unit 424 and
the 0 multi-parallax images supplied from the separation
unit 425 as is necessary based on the parallax
information that is supplied from the 3DV decoding unit
423.
25 [0545]
The image generating unit 426 generates images
having the number of viewpoints that is three or more
corresponding to a display device not illustrated in the
figure based on the L parallax image, the R parallax
30 image, and the 0 multi-parallax image after conversion,
the L image, the R image, and the 0 multi-viewpoint image,
187
SP308970WOOO
and a maximal value and a minimal value of the depth
values or a maximal value and a minimal value of the
parallax values that are included in the parallax
information, and the correspondence relation information.
5 Then, the image generating unit 426 converts the
resolution of the generated image of each viewpoint into
resolution that is ~l/the number of viewpoints" of the
resolution of the compatible image or the auxiliary image,
synthesizes the images, and displays the synthesized
10 image on a display device not illustrated in the figure.
[0546]
At this time, the multi-viewpoint images after the
synthesis are displayed such that the viewing angles are
different for each viewpoint, and a viewer can view a 3D
15 image without wearing glasses by seeing images of
arbitrary two viewpoints with his left and right eyes.
[0547]
[Description of Process of Decoding Device According to
3DV Mode]
20 Fig. 60 is a flowchart that illustrates a decoding
process performed by the decoding device 420 illustrated
in Fig. 59. This decoding process, for example, is
started when a TS is transmitted from the encoding device
380 illustrated in Fig. 53.
25 [0548]
In Step S411 illustrated in Fig. 60, the reception
unit 421 of the decoding device 340 receives a TS that is
transmitted from the encoding device 380. The reception
unit 421 supplies the bit stream included in the TS to
30 the separation unit 422 and supplies the reference image
multiplexing information to the 3DV decoding unit 423.
188
•
5
10
15
20
25
30
SP308970WOOO
In addition, the reception unit 421 supplies the
compatible multiplexing information included in the TS to
the separation unit 424 and supplies the auxiliary
multiplexing information to the separation unit 425. The
process of the subsequent Steps S412 to S428 is performed
for each NAL unit that configures the bit stream.
[0549]
In Step S412, the separation unit 422 separates
each NAL unit based on the NAL header of the NAL unit and
inputs data of the L image and the R image, the auxiliary
multiplexed image, the compatible multiplexed image, or
the auxiliary parallax multiplexed image that is included
in the NAL unit to the 3DV decoding unit 423 together
with the type information.
[0550]
In Step S413, the 3DV decoding unit 423 determines
whether the data input together with the type information
is the encoded data of the L image, the R image, the
auxiliary multiplexed image, the compatible multiplexed
image, or the auxiliary parallax multiplexed image based
on the type information that is supplied from the
separation unit 422. In a case where the data is
determined to be the encoded data of the L image, the R
image, the auxiliary multiplexed image, the compatible
multiplexed image, or the auxiliary parallax multiplexed
image in Step S413, the process proceeds to Step S414.
[0551]
In Step S414, the 3DV decoding unit 423 decodes the
encoded data of the L image, the R image, the auxiliary
multiplexed image, the compatible multiplexed image, or
the auxiliary parallax multiplexed image based on the
189
•
5
10
15
20
25
30
SP308970WOOO
maintained information of the SPS, the Subset SPS, or the
SEI.
[0552]
In Step S415, the 3DV decoding unit 423 determines
whether or not an image acquired as a result of the
decoding process is a compatible multiplexed image based
on the type information. In a case where the image
acquired as a result of the decoding process is
determined to be a compatible multiplexed image in Step
S415, the 3DV decoding unit 423 supplies the compatible
multiplexed image to the separation unit 424.
[0553]
In Step S416, the separation unit 424 separates the
compatible multiplexed image supplied from the 3DV
decoding unit 423 into an L parallax image and an R
parallax jmage of which the resolution is the same as
that of the L image and the R image based on the
compatible multiplexing information supplied from the
reception unit 421. The separation unit 424 supplies the
L parallax image and the R parallax image to the image
generating unit 426.
[0554]
In Step S417, the separation unit 424 outputs the L
parallax image and the R parallax image to the image
generating unit 426, and the process ends.
[0555]
On the other hand, in a case where the image
acquired as a result of the decoding process is
determined not to be a compatible multiplexed image in
Step S415, the 3DV decoding unit 423 determines whether
or not the image acquired as the result of the decoding
190
• SP308970WOOO
process is an auxiliary multiplexed image based on the
type information in Step S418.
[0556]
In a case where the image acquired as the result of
5 the decoding process is determined to be an auxiliary
multiplexed image in Step S418, the 3DV decoding unit 423
supplies the auxiliary multiplexed image to the
separation unit 425, and the process proceeds to Step
S419.
10 [0557]
In Step S419, the separation unit 425 separates the
auxiliary multiplexed image supplied from the 3DV
decoding unit 423 into 0 multi-images of two viewpoints
of which the resolution is the same as that of the L
15 image and the R image based on the auxiliary multiplexing
information supplied from the reception unit 421. The
separation unit 425 supplies the 0 multi-images of the
two viewpoints to the image generating unit 426.
[0558]
20 In Step S420, the separation unit 425 outputs the 0
multi-images of the two viewpoints to the image
generating unit 426, and the process ends.
[0559]
In a case where the image acquired as the result of
25 the decoding process is determined not to be an auxiliary
multiplexed image in Step S418, the 3DV decoding unit 423
determines whether or not the image acquired as the
result of the decoding process is an auxiliary parallax
multiplexed image based on the type information in Step
30 S421.
[0560]
191
5
10
15
20
25
30
SP308970WOOO
In a case where the image acquired as the result of
the decoding process is determined to be an auxiliary
parallax multiplexed image in Step S421, the 3DV decoding
unit 423 supplies the auxiliary parallax multiplexed
image to the separation unit 425, and the process
proceeds to Step S422.
[0561]
In Step S422, the separation unit 425 separates the
auxiliary parallax multiplexed information supplied from
the 3DV decoding unit 423 into 0 multi-parallax images of
two viewpoints of which the resolution is the same as
that of the L image and the R image based on the
auxiliary multiplexing information. The separation unit
425 supplies the 0 multi-parallax images of the two
viewpoints to the image generating unit 426.
[0562]
In Step S423, the separation unit 424 outputs the 0
multi-parallax images of two viewpoints to the image
generating unit 426, and the process ends.
[0563]
On the other hand, in a case where the image
acquired as the result of the decoding process is
determined not to be an auxiliary parallax multiplexed
image in Step S421, in other words, in a case where the
image acquired as the result of the decoding process is
the L image or the R image, the process proceeds to Step
S424.
[0564]
In Step S424, the 3DV decoding unit 423 outputs the
L image or the R image that is acquired as the result of
the decoding process to the image generating unit 426,
192
SP308970WOOO
and the process ends.
[0565]
On the other hand, in a case where the data is
determined not to be the encoded data of the L image, the
5 R image, the auxiliary multiplexed image, the compatible
multiplexed image, or the auxiliary parallax multiplexed
image in Step S413, in other words, in a case where the
data supplied from the separation unit 422 is the
information of the SPS, the Subset SPS, or the SEl, the
10 process proceeds to Step S425.
[0566]
The process of Steps S425 to S428 is the same as
the process of Steps S339 to S342 illustrated in Fig. 51,
and thus the description thereof will not be presented.
15 [0567]
As above, the decoding device 420 separates the bit
stream into each NAL unit based on the NAL headers. Then,
the decoding device 420 decodes the encoded data of the
compatible image included in the separated NAL unit in
20 accordance with a mode that corresponds to the existing
AVC mode or the MVC mode and decodes the data of the
auxiliary image and the parallax image in accordance with
a mode that corresponds to the 3DV mode. Accordingly,
the decoding device 420 can decode the bit stream that is
25 acquired as a result of an encoding process having
compatibility with an existing mode using the encoding
device 380.
[0568]
Although not illustrated in the figure, the
30 decoding device according to the AVC mode and the
decoding device according to the MVC mode that decode the
193
5
10
• SP308970WOOO
bit stream that is generated by the encoding device 380
illustrated in Fig. 53 are the same as the decoding
device 320 illustrated in Fig. 46 and the decoding device
340 illustrated in Fig. 48.
[0569]
[Another Arrangement Example of Parallax Information]
In the description presented above, although the
parallax information has been described to be included in
the SEI, the parallax information may be included in the
TS. In such a case, for example, the parallax
information is described in a descriptor of a PMT
(program map table) or an SIT (selection information
table) within the TS.
[0570]
15 Fig. 61 is a diagram that illustrates an example of
the description of parallax information in a case where
the parallax information is described in a descriptor of
the PMT.
[0571]
20 As illustrated in Fig. 61, in a case where the
parallax information is described in a descriptor of the
PMT, the descriptor (3DV_view_synthesis_descriptor) in
which the parallax information is arranged is disposed as
the descriptor of the PMT. In this descriptor, the
25 parallax information (3DV_view_synthesis_info)
illustrated in Fig. 43 is described.
[0572]
In addition, the parallax information may be
included in both the PMT and the SIT within the SEI and
30 the TS. Furthermore, the extension information may be
described not in the NAL header but in the PMT, the SIT,
194
• SP308970WOOO
the SEI, or the like within the TS.
[0573]
Furthermore, the auxiliary image low-resolution
converting information, the compatible multiplexing
5 information, and the auxiliary multiplexing information
may be transmitted while being included in the SEI or the
like.
[0574]
In addition, in the description presented above,
10 the extension information has been described to be added
to the encoded data, the extension information may be
transmitted (recorded) separately from the image data (or
the bit stream). Furthermore, the extension information
may be connected to the image data (or the bit stream).
15 [0575]
In this embodiment, the "connection" is defined as
below. The "connection" represents a state in which the
image data (or the bit stream) and the extension
information are linked to each other. The image data and
20 the extension information as connection targets may be
transmitted through mutually-different transmission lines.
In addition, the image data (or the bit stream) and the
extension information as connection targets may be
recorded on mutually-different recording media (or
25 mutually-different areas of the same recording medium) .
Furthermore, the unit of linking the image data (or the
bit stream) and the extension information to each other,
for example, may be an encoding processing unit (one
frame, multiple frames, or the like).
30 [0576]
In addition, although the encoding device 290 and
195
•
5
10
15
SP308970WOOO
the encoding device 380 have been described to
respectively generate one TS, in the same manner as the
encoding device 50 illustrated in Fig. 2, the encoding
device 140 illustrated in Fig. 14, the encoding device
180 illustrated in Fig. 22, and the encoding device 230
illustrated in Fig. 33 except that the auxiliary
information replaces the auxiliary image low-resolution
converting information and the reference image lowresolution
converting information, a plurality of TSs may
be generated.
[0577]

[Configuration Example of Encoding Device According to
Seventh Embodiment]
Fig. 62 is a block diagram that illustrates a
configuration example of an encoding device according to
a seventh embodiment of the present technology.
[0578]
In the configuration illustrated in Fig. 62, the
20 same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 14.
Duplicate description will not be presented as is
appropriate.
[0579]
25 The configuration of the encoding device 440
illustrated in Fig. 62 is mainly different from the
configuration illustrated in Fig. 14 in that a parallax
image generating unit 441 and an encoder 442 are disposed
instead of the parallax image generating unit 143 and the
30 encoder 145. The encoding device 440 does not multiplex
the parallax images of compatible images in the spatial
196
• SP308970WOOO
direction but multiplexes the parallax images of the
compatible images in the time direction and encodes the
multiplexed parallax images.
[0580]
5 More specifically, the parallax image generating
unit 441 of the encoding device 440 detects the disparity
of each pixel of images A2 to 02 using the images A2 to
D2 that are supplied from the imaging units 141A to 1410.
The parallax image generating unit 441 generates a
10 parallax image A2' of the image A2 and a parallax image
B2' of the image B2, which are compatible images, and a
parallax image C2' of the image C2 and a parallax image
D2' of the image 02, which are auxiliary images, based on
the detection result.
15 [0581]
In addition, the parallax image generating unit 441,
similarly to the parallax image generating unit 143
illustrated in Fig. 14, multiplexes the parallax images
C2' and 02' in the spatial direction in accordance with a
20 predetermined multiplexing mode. Furthermore, the
parallax image generating unit 441 multiplexes the
parallax images A2' and B2' and the multiplexed image of
the parallax images of the auxiliary images in the time
direction. The parallax image generating unit 441
25 supplies a multiplexed image in which parallax images A2'
and B2' and the multiplexed image of the parallax images
of the auxiliary images corresponding to one frame exist
within one frame time, which is acquired as a result
thereof, to the encoder 442 as a time-multiplexed image.
30 [0582]
In addition, the parallax image generating unit 441
197
•
5
SP308970WOOO
supplies information that represents a multiplexing mode
of the parallax images of the auxiliary images and a
multiplexing mode (hereinafter, referred to as a frame
sequential mode) in the time direction as a multiplexing
mode of the parallax images of the compatible images and
the auxiliary images to the parallax image information
generating unit 57.
[0583]
The encoder 442 is configured'by a compatible
10 encoder 151 and an auxiliary encoder 451. The auxiliary
encoder 451 of the encoder 442 encodes the multiplexed
image of the auxiliary images that is supplied from the
image converting unit 142 and the time-multiplexed image
that is supplied from the parallax image generating unit
15 441 in accordance with the 3DV mode in units of access
units. The auxiliary encoder 451 supplies encoded
streams acquired as a result thereof to the multiplexing
unit 59 as auxiliary streams.
[0584]
20 [Example of Description of Auxiliary Information]
Fig. 63 is a diagram that illustrates an example of
the description of the compatibility information and the
parallax image information in a case where the auxiliary
information is described in a descriptor of the PMT.
25 [0585]
As illustrated in Fig. 63, in a case where the
auxiliary information is described in the descriptor of
the PMT, a descriptor (3DV_view_structure_descriptor) in
which compatibility information is arranged and a
30 descriptor (depth_map_structure_descriptor) in which the
parallax image information is arranged, and the like are
198
• SP308970WOOO
disposed as the descriptors of the PMT.
[0586]
In the descriptor (depth_map_structure_descriptor),
as illustrated in Fig. 64, following a descriptor tag
5 (descriptor_tag) and a descriptor length
(descriptor_length), the number (num_of_depth_map) of
parallax images, a flag (is_frame_packing) that
represents whether or not parallax images are multiplexed,
a multiplexing mode (frame_packing_mode) of the parallax
10 images, information (comb_frame_packing_views) that
designates the parallax images that are multiplexed, and
the like are described as the parallax image information.
[0587]
In addition, as examples of the multiplexing mode,
15 there are a side-by-side mode (SBS), a top-and-bottom
mode (TOB) , and a frame sequential mode.
[0588]
In this specification, although the auxiliary
information is configured to be included in the TS, the
20 auxiliary information may be included in the auxiliary
stream.
[0589]
Figs. 65 and 66 are diagrams that illustrate
examples of the description of the compatibility
25 information and the parallax image information included
in an auxiliary stream in a case where the auxiliary
information is included in the auxiliary stream.
[0590]
As illustrated in Fig. 65, the compatibility
30 information (3DV_view_structure) and the parallax image
information (depth_map_structure), for example, are
199
• SP308970WOOO
arranged in the SEI (Supplemental Enhancement
Information) included in the auxiliary stream.
[0591]
As the parallax image information
5 (depth_map_structure), as illustrated in Fig. 66, the
number (nurn_of_depth_map) of parallax images (depth map),
a flag (is_frame_packing) that represents whether or not
the parallax images are multiplexed, the multiplexing
mode (frame_packing_mode) of the parallax images,
10 information (cornb_frame_packing_views) that designates
the parallax images that are multiplexed, and the like
are described.
[0592]
Although not illustrated in the figure, the image
15 information is the same as the parallax image information
except that the image information is not a parallax image
but information of the compatible images and the
auxiliary images.
[0593]
20 [Description of Process of Encoding Device]
Figs. 67 and 68 represent a flowchart that
illustrates an encoding process that is performed by the
encoding device 440 illustrated in Fig. 62. This
encoding process, for example, is started when images A2
25 to D2 are output from imaging units 141A to 141D.
[0594]
The process of Steps S451 to S459 illustrated in
Figs. 67 and 68 is the same as the process of Steps S71
to S79 illustrated in Figs. 16 and 17, and thus the
30 description thereof will not be presented.
[0595]
200
5
• SP308970WOOO
In Step S460 illustrated in Fig. 68, the parallax
image generating unit 441 determines the multiplexing
mode of the parallax images of the auxiliary images and
the multiplexing mode of the multiplexed image of the
parallax images of the compatible images and the parallax
images of the auxiliary image and supplies information
that represents the multiplexing modes to the parallax
image information generating unit 57.
[0596]
10 In Step S461, the parallax image information
generating unit 57 generates information that represents
the multiplexing mode of the parallax images of the
auxiliary images and the multiplexing mode of the
multiplexed image of the parallax images of the
15 compatible images and the parallax images of the
auxiliary images and the like as parallax image
information based on the information that is supplied
from the parallax image generating unit 441 and inputs
the generated parallax image information to the
20 multiplexing unit 59.
[0597]
In Step S462, the parallax image generating unit
441 multiplexes the parallax images C2' and D2' of the
auxiliary images in the spatial direction and multiplexes
25 the parallax images A2' and B2' of the compatible images
and the multiplexed image of the parallax images of the
auxiliary images in the time direction, based on the
multiplexing mode that is determined in Step S460.
[0598]
30 In Step S463, the parallax image generating unit
441 inputs the time-multiplexed image that is acquired as
201
• SP308970WOOO
a result of the multiplexing process of Step S462 to the
encoder 442.
[0599]
In Step 8464, the compatible encoder 151 of the
5 encoder 442 encodes the multiplexed image of the
compatible images that is supplied from the image
converting unit 142 in accordance with the existing Ave
mode and supplies an encoded stream that is acquired as a
result thereof to the multiplexing unit 59 as a
10 compatible stream.
[0600]
In 8tep 8465, the auxiliary encoder 451 encodes the
multiplexed image of the auxiliary images that is
supplied from the image converting unit 142 and the time-
15 multiplexed image that is supplied from the parallax
image generating unit 441 in accordance with the 3DV mode.
The auxiliary encoder 451 supplies an encoded stream that
is acquired as a result of the encoding process to the
multiplexing unit 59 as an auxiliary stream.
20 [0601]
In Step S466, the multiplexing unit 59, similarly
to the process of Step S86 illustrated in Fig. 17,
generates a TSl from the compatible stream that is
supplied from the compatible encoder 151, generates a T82
25 from the auxiliary stream and the auxiliary information
supplied from the auxiliary encoder 451, performs a
multiplexing process, and transmits a multiplexed stream
acquired as a result thereof. This multiplexed stream,
for example, is recorded on a BD or the like or is
30 transmitted as a broadcast stream. Then, the process
ends.
202
•
5
10
15
20
25
30
SP308970WOOO
[0602]
[Configuration Example of Decoding Device]
Fig. 69 is a diagram that illustrates a
configuration example of a decoding device that decodes
the multiplexed stream that is transmitted from the
encoding device 440 illustrated in Fig. 62.
[0603]
In the configuration illustrated in Fig. 69, the
same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 18.
Duplicate description will not be presented as is
appropriate.
[0604]
The configuration of the decoding device 460
illustrated in Fig. 69 is mainly different from the
configuration illustrated in Fig. 18 in that a decoder
461 and an image generating unit 462 are disposed instead
of the decoder 122 and the image generating unit 171.
The decoding device 460 generates images of two
viewpoints or multi-viewpoint images by decoding the
multiplexed stream that is transmitted from the encoding
device 440 and displays the generated images on a display
device not illustrated in the figure.
[0605]
More specifically, the decoder 461 of the decoding
device 460 is configured by a compatible decoder 131 and
an auxiliary decoder 471. The auxiliary decoder 471 of
the decoder 461 decodes the multiplexed image of
auxiliary images and a time-multiplexed image that are
included in the auxiliary stream supplied from the
separation unit 121 in accordance with a mode that
203
•
5
SP308970WOOO
corresponds to the auxiliary encoder 451 illustrated in
Fig. 62. The auxiliary decoder 471 supplies the
multiplexed image of the auxiliary images and the timemultiplexed
image that are acquired as a result of the
decoding process to the image generating unit 462.
[0606]
The image generating unit 462 outputs an image in
accordance with an instruction supplied from a viewer and
displays the image on a display device not illustrated in
10 the figure. When described in more detail, the image
generating unit 462 generates images of three or more
viewpoints, which correspond to a display device not
illustrated in the figure, each having the resolution
that is a half of the resolution of the compatible image
15 or the auxiliary image by using the multiplexed image of
compatible images, the multiplexed image of the auxiliary
images, and the time-multiplexed image in accordance with
a viewer's instruction for displaying a 3D image of the
multi-viewpoint mode based on the image information
20 supplied from the image information acquiring unit 123,
the inter-viewpoint distance information supplied from
the inter-viewpoint distance information acquiring unit
124, the parallax image information supplied from the
parallax image information acquiring unit 125, the
25 compatibility information supplied from the compatibility
information acquiring unit 126, and the like.
[0607]
Described in more detail, the image generating unit
462 separates parallax images A2' and B2' that are the
30 parallax images of the compatible images and the
multiplexed image of the parallax images of the auxiliary
204
•
5
10
SP308970WOOO
images from the time-multiplexed image based on the
information representing the frame sequential mode as the
multiplexing mode of the multiplexed image of the
parallax images of the compatible images and the parallax
images of the auxiliary images included in the parallax
image information that is supplied from the parallax
image information acquiring unit 125. Then, the image
generating unit 462 separates the parallax images C2' and
D2' from the multiplexed image of the parallax images of
the auxiliary images based on the multiplexing mode of
the parallax images of the auxiliary images that is
included in the parallax image information.
[0608]
In addition, the image generating unit 462,
15 similarly to the image generating unit 171 illustrated in
Fig. 18, separates each auxiliary image from the
multiplexed image of the auxiliary images based on the
information that represents the multiplexing mode of the
auxiliary images that is included in the image
20 information supplied from the image information acquiring
unit 123. Furthermore, the image generating unit 462,
similarly to the image generating unit 171, separates
each compatible image from the multiplexed image of the
compatible images based on the information that
25 represents the multiplexing mode of the compatible images
that is included in the image information.
[0609]
Furthermore, the image generating unit 462,
similarly to the image generating unit 171, determines
30 the position of each viewpoint of multi-viewpoint images
to be generated based on the inter-viewpoint distance
205
•
5
10
15
20
25
30
SP308970WOOO
information and the number of viewpoints corresponding to
a display device not illustrated in the figure. Then,
the image generating unit 462, similarly to the image
generating unit 171, generates the image of each
viewpoint of which the position is determined by using
each compatible image, each auxiliary image, the parallax
image of each compatible image, and the parallax image of
each auxiliary image. Then, the image generating unit
462; similarly to the image generating unit 171, converts
the resolution of the generated multi-viewpoint images
into resolution that is "lithe number of viewpoints" of
the resolution of the compatible image or the auxiliary
image, synthesizes the images, and displays the
synthesized image on a display device not illustrated in
the figure.
[0610]
At this time, the multi-viewpoint images after the
synthesis are displayed such that the viewing angles are
different for each viewpoint, and a viewer can view a 3D
image without wearing glasses by seeing images of
arbitrary two viewpoints with his left and right eyes.
[0611]
In addition, the image generating unit 462,
similarly to the image generating unit 171 separates the
multiplexed image of the compatible images that is
supplied from the decoder 461 into images A2 and B2 of
which the resolution is a half of the resolution of the
compatible image based on the image information supplied
from the image information acquiring unit 123 in
accordance with an instruction for displaying a 3D image
of the two-viewpoint mode that is supplied from a viewer.
206
•
5
10
SP308970WOOO
Then, the image generating unit 462, similarly to the
image generating unit 171, alternately outputs the
separated images A2 and B2 of which the resolution is a
half of the resolution of the compatible image, thereby
displaying the separated images on a display device not
illustrated in the figure. At this time, the viewer can
view a 3D image by wearing glasses in which one a lefteye
shutter and a right-eye shutter is open at the time
of displaying the image A2, and the. other is open at the
time of displaying the image B2 and viewing the images A2
and B2 that are alternately displayed on the display
device.
[0612]
[Description of Decoding Process]
15 Fig. 70 is a flowchart that illustrates a decoding
process performed by the decoding device 460 illustrated
in Fig. 69. This decoding process, for example, is
started when a multiplexed stream that is transmitted
from the encoding device 440 illustrated in Fig. 62 is
20 input to the decoding device 460.
[0613]
The process of Steps S471 to S473 illustrated in
Fig. 70 is similar to that of Steps S91 to S93
illustrated in Fig. 19, and thus, the description thereof
25 will not be presented.
[0614]
In Step 8474, the auxiliary decoder 471 extracts a
multiplexed image of encoded auxiliary images and a timemultiplexed
image from the auxiliary stream and decodes
30 the extracted multiplexed images in accordance with a
mode that corresponds to the auxiliary encoder 62
207
•
5
10
15
20
25
30
SP308970WOOO
illustrated in Fig. 2. The auxiliary decoder 471
supplies the multiplexed image of the auxiliary images
and the time-multiplexed image that are acquired as a
result of the decoding process to the image generating
unit 127, and the process proceeds to Step S475.
[0615]
The process of Steps S475 to S479 is similar to
that of Steps S95 to S99 illustrated in Fig. 19, and thus,
the description thereof will not be presented.
[0616]
In Step S480, the image generating unit 462
generates images of the viewpoints each having the
resolution that is a half of the resolution of the
compatible image or the auxiliary image based on the
position of each viewpoint determined in Step S479, the
image information supplied from the image information
acquiring unit 123, the parallax image information
supplied from the parallax image information acquiring
unit 125, the compatibility information supplied from the
compatibility information acquiring unit 126, and the
like by using the multiplexed image of the compatible
images, the multiplexed image of the auxiliary images,
and the time-multiplexed image.
[0617]
The process of Steps S481 to S486 is similar to
that of Steps S101 to S106 illustrated in Fig. 19, and
thus, the description thereof will not be presented.
[0618]
In a decoding device that can decode only a
compatible stream that has compatibility with the
decoding device 460, a TS2 to which the 3DV
208
•
5
10
15
20
25
30
8P308970WOOO
representation delimiter NAL unit is added is ignored,
and the process of 8teps 8483, 8485, and 8486 is
performed. However, in this case, in the process of 8tep
8485, the multiplexed image of the compatible images is
separated based on the multiplexing mode that is
determined in advance.
[0619]

[Configuration Example of Encoding Device According to
Eighth Embodiment]
Fig. 71 is a block diagram that illustrates a
configuration example of an encoding device according to
an eighth embodiment of the present technology.
[0620]
In the configuration illustrated in Fig. 71, the
same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 14.
Duplicate description will not be presented as is
appropriate.
[0621]
The configuration of the encoding device 480
illustrated in Fig. 71 is mainly different from the
configuration illustrated in Fig. 14 in that a parallax
image generating unit 481, an encoder 482, and a parallax
image information generating unit 483 are disposed
instead of the parallax image generating unit 143, the
encoder 145, and the parallax image information
generating unit 57. The encoding device 480 encodes a
common parallax image that shows a parallax value that is
common to the compatible images and a common parallax
image of the auxiliary images.
209
• SP308970WOOO
[0622]
More specifically, the parallax image generating
unit 481 of the encoding device 480 detects the disparity
of each pixel between images A2 and B2 and the disparity
5 of each pixel between images C2 and D2 using the images
A2 to D2 that are supplied from the imaging units 141A to
141D. The parallax image generating unit 481 generates a
parallax image formed by disparity values that represent
the disparity of each pixel between the images A2 and B2
10 as a common parallax image AB2' of the compatible images
and supplies the generated common parallax image to the
encoder 482. In addition, the parallax image generating
unit 481 generates a parallax image formed by disparity
values that represent the disparity of each pixel between
15 the images C2 and D2 as a common parallax image CD2' of
the auxiliary images and supplies the generated common
parallax image to the encoder 482.
[0623]
Furthermore, the parallax image generating unit 481
20 supplies information that represents that the parallax
images of the compatible images and the auxiliary images
are common parallax images to the parallax image
information generating unit 483.
[0624]
25 The encoder 482 is configured by a compatible
encoder 151 and an auxiliary encoder 491. The auxiliary
encoder 491 encodes the multiplexed image of auxiliary
images that is supplied from the image converting unit
142 and the common parallax image AB2' of the compatible
30 images and the common parallax image CD2' of the
auxiliary images that are supplied from the parallax
210
•
5
10
15
20
25
30
SP308970WOOO
image generating unit 481 in accordance with the 3DV mode
in units of access units. The auxiliary encoder 491
supplies an encoded stream acquired as a result thereof
to the multiplexing unit 59 as an auxiliary stream.
[0625]
The parallax image information generating unit 483
generates information representing that the parallax
images of the compatible images and the auxiliary images
are common parallax images and the like as parallax image
information based on the information that is supplied
from the parallax image generating unit 53 and supplies
the generated parallax image information to the
multiplexing unit 59.
[0626]
[Description of Process of Encoding Device]
Figs. 72 and 73 represent a flowchart that
illustrates an encoding process that is performed by the
encoding device 480 illustrated in Fig. 71. This
encoding process, for example, is started when images A2
to D2 are output from the imaging units 141A to 141D.
[0627]
The process of Steps S491 to S498 illustrated in
Fig. 72 is similar to that of Steps S71 to S78
illustrated in Fig. 16, and thus, the description thereof
will not be presented.
[0628]
In Step S499 illustrated in Fig. 73, the parallax
image generating unit 481 detects the disparity of each
pixel between images A2 and B2 and the disparity of each
pixel between images C2 and D2 using the images A2 to D2
that are supplied from the imaging units 141A to 141D.
211
SP308970WOOO
Then, the parallax image generating unit 481 generates a
common parallax image AB2' that is formed by disparity
values that represent the disparity of each pixel between
the images A2 and B2 and a common parallax image CD2'
5 that is formed by disparity values that represent the
disparity of each pixel between the images C2 and D2.
[0629]
In Step 8500, The parallax image information
generating unit 483 generates information representing
10 that the parallax images of the compatible images and the
auxiliary images are common parallax images and the like
as parallax image information based on the information
that is supplied from the parallax image generating unit
481 and inputs the generated parallax image information
15 to the multiplexing unit 59.
[0630]
In Step 8501, the parallax image generating unit
481 inputs the common parallax images CD2' of the
auxiliary images and the common parallax images AB2' of
20 the compatible images, which are generated in Step S499,
to the encoder 482.
[0631]
In Step 8502, the compatible encoder 151 of the
encoder 482 encodes the multiplexed image of the
25 compatible images that is supplied from the image
converting unit 142 in accordance with the existing AVC
mode and supplies an encoded stream acquired as a result
thereof to the multiplexing unit 59 as a compatible
stream.
30 [0632]
In Step S503, the auxiliary encoder 491 encodes the
212
8P308970WOOO
multiplexed image of the auxiliary images supplied from
the image converting unit 142 and the common parallax
image of the compatible images and the common parallax
image of the auxiliary images that are supplied from the
5 parallax image generating unit 481 in accordance with the
3DV mode. The auxiliary encoder 491 supplies an encoded
stream acquired as a result of the encoding process to
the multiplexing unit 59 as an auxiliary stream.
[0633]
10 In 8tep 8504, the multiplexing unit 59 generates a
T81 from the compatible stream that is supplied from the
compatible encoder 151, generates a T82 from the
auxiliary stream and the auxiliary information supplied
from the auxiliary encoder 491, performs a multiplexing
15 process, and transmits a multiplexed stream acquired as a
result thereof. This multiplexed stream, for example, is
recorded on a BD or the like or is transmitted as a
broadcast stream. Then, the process ends.
[0634]
20 [Configuration Example of Decoding Device]
Fig. 74 is a diagram that illustrates a
configuration example of a decoding device that decodes
the multiplexed stream transmitted from the encoding
device 480 illustrated in Fig. 71.
25 [0635]
In the configuration illustrated in Fig. 74, the
same reference numeral is assigned to a configuration
that is the same as that illustrated in Fig. 18.
Duplicate description will not be presented as is
30 appropriate.
[0636]
213
SP308970WOOO
The configuration of the decoding device 500
illustrated in Fig. 74 is mainly different from the
configuration illustrated in Fig. 18 in that a decoder
501 and an image generating unit 502 are disposed instead
5 of the decoder 122 and the image generating unit 171.
The decoding device 500 generates images of two
viewpoints or multi-viewpoint images by decoding the
multiplexed stream that is transmitted from the encoding
device 480 and displays the generated images on a display
10 device not illustrated in the figure.
[0637]
More specifically, the decoder 501 of the decoding
device 500 is configured by a compatible decoder 131 and
an auxiliary decoder 511. The auxiliary decoder 511 of
15 the decoder 501 decodes the multiplexed image of
auxiliary images that is included in the auxiliary stream
that is supplied from the separation unit 121, the common
parallax image AB2' of the compatible images, and the
common parallax image CD2' of the auxiliary images in
20 accordance with a mode that corresponds to the auxiliary
encoder 491 illustrated in Fig. 71. The auxiliary
decoder 511 supplies the multiplexed image of the
auxiliary images and the common parallax images AB2' and
CD2' that are acquired as a result of the decoding
25 process to the image generating unit 502.
[0638]
The image generating unit 502 outputs an image in
accordance with an instruction supplied from a viewer and
displays the image on a display device not illustrated in
30 the figure. When described in more detail, the image
generating unit 502 generates images of three or more
214
SP308970WOOO
viewpoints, which correspond to a display device not
illustrated in the figure, each having the resolution
that is a half of the resolution of the compatible image
or the auxiliary image by using the multiplexed image of
5 compatible images, the multiplexed image of the auxiliary
images, and the common parallax images AB2' and CD2' in
accordance with a viewer's instruction for displaying a
3D image of the multi-viewpoint mode based on the image
information supplied from the image information acquiring
10 unit 123, the inter-viewpoint distance information
supplied from the inter-viewpoint distance information
acquiring unit 124, the parallax image information
supplied from the parallax image information acquiring
unit 125, the compatibility information supplied from the
15 compatibility information acquiring unit 126, and the
like.
[0639]
Described in more detail, the image generating unit
502 maintains the common parallax images AB2' and CD2'
20 without being changed based on the information
representing that the parallax images of the compatible
images and the auxiliary images are common parallax
images that is included in the parallax image information
supplied from the parallax image information acquiring
25 unit 125.
[0640]
In addition, the image generating unit 502,
similarly to the image generating unit 171 illustrated in
Fig. 18, separates each auxiliary image from the
30 multiplexed image of the auxiliary images based on the
information that represents the multiplexing mode of the
215
5
• SP308970WOOO
auxiliary images that is included in the image
information supplied from the image information acquiring
unit 123. Furthermore, the image generating unit 502,
similarly to the image generating unit 171, separates
each compatible image from the multiplexed image of the
compatible images based on the information that
represents the multiplexing mode of the compatible images
that is included in the image information.
[0641]
10 Furthermore, the image generating unit 502,
similarly to the image generating unit 171, determines
the position of each viewpoint of the multi-viewpoint
images to be generated based on the inter-viewpoint
distance information and the number of viewpoints that
15 corresponds to a display device not illustrated in the
figure. Then, the image generating unit 502 generates
the image of each viewpoint of which the position is
determined by using each compatible image, each auxiliary
image, the common parallax image AB2', and the common
20 parallax image CD2'. Then, the image generating unit 502,
similarly to the image generating unit 171, converts the
resolution of the generated multi-viewpoint images into
resolution that is "lithe number of viewpoints" of the
resolution of the compatible image or the auxiliary image,
25 synthesizes the images, and displays the synthesized
image on a display device not illustrated in the figure.
[0642]
At this time, the multi-viewpoint images after the
synthesis are displayed such that the viewing angles are
30 different for each viewpoint, and a viewer can view a 3D
image without wearing glasses by seeing images of
216
SP308970WOOO
arbitrary two viewpoints with his left and right eyes.
[0643]
In addition, the image generating unit 502,
similarly to the image generating unit 171, separates
5 from the multiplexed image of the compatible images that
is supplied from the decoder 501 into images A2 and B2 of
which the resolution is a half of the resolution of the
compatible image based on the image information supplied
from the image information acquiring unit 123 in
10 accordance with an instruction for displaying a 3D image
of the two-viewpoint mode that is supplied from a viewer.
Then, the image generating unit 502, similarly to the
image generating unit 171, alternately outputs the
separated images A2 and B2 of which the resolution is a
15 half of the resolution of the compatible image, thereby
displaying the separated images on a display device not
illustrated in the figure. At this time, the viewer can
view a 3D image by wearing glasses in which one a lefteye
shutter and a right-eye shutter is open at the time
20 of displaying the image A2, and the other is open at the
time of displaying the image B2 and viewing the images A2
and B2 that are alternately displayed on the display
device.
[0644]
25 [Description of Process of Decoding Device]
Fig. 75 is a flowchart that illustrates a decoding
process performed by the decoding device 500 illustrated
in Fig. 74. This decoding process, for example, is
started when a multiplexed stream that is transmitted
30 from the encoding device 480 illustrated in Fig. 71 is
input to the decoding device 500.
217
30
SP308970WOOO
[0645]
The process of Steps S511 to S513 illustrated in
Fig. 75 is similar to that of Steps S91 to S93
illustrated in Fig. 19, and thus, the description thereof
5 will not be presented.
[0646]
In Step S514, the auxiliary decoder 511 extracts
the encoded multiplexed image of the auxiliary images,
the common parallax image AB2' of the compatible images,
10 and the common parallax image C02' of the auxiliary
images and decodes the extracted images in accordance
with a mode that corresponds to the auxiliary encoder 491
illustrated in Fig. 71. The auxiliary decoder 511
supplies the multiplexed image of the auxiliary images,
15 the common parallax image AB2', and the common parallax
image C02', which are acquired as a result of the
decoding process, to the image generating unit 502.
[0647]
The process of Steps S515 to S519 is similar to
20 that of Steps S95 to S99 illustrated in Fig. 19, and thus,
the description thereof will not be presented.
[0648]
After the process of Step S519, in Step S520, the
image generating unit 502 generates an image of each
25 viewpoint of which the resolution is a half of the
resolution of the compatible image or the auxiliary image
by using the compatible images, the multiplexed image of
the auxiliary images, the common parallax image AB2', and
the common parallax images C02' based on the position of
each viewpoint determined in Step S519, the image
information supplied from the image information acquiring
218
• SP308970WOOO
unit 123, the parallax image information supplied from
the parallax image information acquiring unit 125, and
the compatibility information supplied from the
compatibility information acquiring unit 126, and the
5 like.
[0649]
The process of Steps S521 to S526 is similar to
that of Steps S101 to S106 illustrated in Fig. 19, and
thus, the description thereof will not be presented.
10 [0650]
Although the encoding devices 440 and 480,
similarly to the encoding device 140, multiplex and
encode compatible images of two view points, similarly to
the encoding device 180 illustrated in Fig. 22, the
15 encoding devices 440 and 480 may encode compatible images
of two viewpoints without multiplexing them. In addition,
the encoding devices 440 and 480, similarly to the
encoding device 50 illustrated in Fig. 2, may encode a
compatible image of one viewpoint.
20 [0651]
25
30
In addition, the encoding devices 140 and 180 may
encode the parallax images of the compatible images and
the auxiliary images without multiplexing the abovedescribed
parallax images. Furthermore, the encoding
device 50 may encode the parallax images of the auxiliary
images without multiplexing the above-described parallax
images.
[0652]

Fig. 76 is a diagram that illustrates an example of
a multiplexing pattern of an encoding target in a case
219
I!
l
i~
ii
if
F
iII
[
5
• SP308970WOOO
where the number of viewpoints of the compatible images
is two, and the number of viewpoints of the auxiliary
images is two.
[0653]
As illustrated in (1) of Fig. 76, the encoding
device 140 illustrated Fig. 14 multiplexes images A2 and
B2 that are compatible images in the spatial direction
and encodes the multiplexed image in accordance with the
Ave mode. In addition, the encoding device 140
10 multiplexes the images e2 and D2 that are auxiliary
images, the parallax images A2' and B2' that are parallax
images of the compatible images, and the parallax images
C2' and D2' that are parallax images of the auxiliary
images in the spatial direction and encodes the
15 multiplexed image in accordance with the 3DV mode that is
compliant with the Mve mode.
[0654]
Furthermore, as illustrated in (2) of Fig. 76, the
encoding device 140 may encode the parallax images A2' to
20 D2' in accordance with the 3DV mode that is compliant
with the MVe mode without multiplexing the parallax
images. In addition, as illustrated in (3) of Fig. 76,
the encoding device 480 illustrated in Fig. 71 may encode
the common parallax image AB2' instead of the parallax
25 images A2' and B2' that are parallax images of the
compatible images and encode the common parallax image
CD2' instead of the parallax images C2' and D2' that are
parallax images of the auxiliary images.
[0655]
30 Furthermore, as illustrated in (4) of Fig. 76, the
encoding device 440 illustrated in Fig. 62 multiplexes
220
III
II~I
ii
it
It r
I~
III
i.
~
• SP308970WOOO
the parallax images A2' and B2' and the multiplexed image
of the parallax images of the auxiliary images in
accordance with the frame sequential mode without
multiplexing the parallax images A2' and B2' that are
5 parallax images of the compatible images in the spatial
direction and encodes the multiplexed image in accordance
with the 3DV mode that is compliant with the MVC mode.
[0656]
As illustrated in (5) of Fig. 76, the encoding
10 device 180 illustrated in Fig. 22 encodes the image A2 in
accordance with the AVC mode and encodes the image B2
with the image A2 set as a base view in accordance with
the MVC mode without multiplexing the images A2 and B2
that are compatible images. In addition, the encoding
15 device 180 encodes the images C2 and D2 that are
auxiliary images, the parallax images A2' and B2' that
are the parallax images of the compatible images, and the
parallax images C2' and D2' that are the parallax images
of the auxiliary images in the spatial direction and
20 encodes a resultant multiplexed image in accordance with
a 3DV mode that is in compliant with the MVC mode.
[0657]
In addition, as illustrated in (6) of Fig. 76, the
encoding device 180 may encode the parallax images A2' to
25 D2' in accordance with the 3DV mode that is compliant
with the MVC mode without multiplexing them. Furthermore,
as illustrated in (7) of Fig. 76, similarly to the
encoding device 480, the encoding device 180 may encode
the common parallax image AB2' instead of the parallax
30 images A2' and B2' and encode the common parallax image
CD2' instead of the parallax images C2' and D2' .
221
5
• SP308970WOOO
[0658]
Furthermore, as illustrated in (8) of Fig. 76, the
encoding device 180, similarly to the encoding device 440,
may multiple the parallax images A2' and B2' and the
multiplexed image of the parallax images of the auxiliary
images in accordance with the frame sequential mode and
encode a resultant multiplexed image in accordance with
the 3DV mode that is compliant with the MVC mode without
multiplexing the parallax images A2' and B2' in the
10 spatial direction.
[0659]
Fig. 77 is a diagram that illustrates features of
the effects of multiplexing in the multiplexing patterns
illustrated in (1) to (8) of Fig. 76.
15 [0660]
In the table represented in Fig. 77, items of the
effects "Compatibility", "Image Quality", and "Data
Amount" are arranged, and the degrees of the effects of
the items for the multiplexing patterns illustrated in
20 (1) to (8) of Fig. 76 are represented. In the table
represented in Fig. 77, a circle represents the presence
of an effect, and a double circle represents a remarkable
effect.
[0661]
25 In a case where multiplexing is performed using the
multiplexing pattern illustrated in (1) of Fig. 76, the
multiplexing mode and the encoding mode of compatible
images are the same as those of an existing mode, whereby
the compatibility is secured. In addition, since
30 parallax images, similarly to the compatible images, are
multiplexed in the spatial direction, for example, the
222
r
I
II
iII
10
SP308970WOOO
parallax images can be separated using a separation unit,
which separates compatible images, prepared on the
decoding device side. Accordingly, it is assured that
the parallax images can be separated on the decoding
5 device side. Therefore, in this case, there is a
remarkable effect on the compatibility, and a double
circle is described in correspondence with the item
~Compatibility" illustrated in Fig. 77.
[0662]
In a case where multiplexing is performed using the
multiplexing pattern illustrated in (2) of Fig. 76, the
multiplexing mode and the encoding mode of compatible
images are the same as those of an existing mode, whereby
the compatibility is secured. In addition, since the
15 resolution of the parallax image is the same as that of
the image before being multiplexed, the accuracy is high.
As a result, in the decoding device, the accuracy of an
image of a predetermined viewpoint that is generated
using the parallax image is improved. Therefore, in this
20 case, there are effects on the compatibility and the
image quality of an image that is generated using the
parallax image, and accordingly, circles are described in
correspondence with items ~Compatibility" and ~Image
Quality" represented in Fig. 77.
25 [0663]
30
In a case where multiplexing is performed using the
multiplexing pattern illustrated in (3) of Fig. 76, the
multiplexing mode and the encoding mode of compatible
images are the same as those of an existing mode, whereby
the compatibility is secured. In addition, the amount of
data of the parallax images of the compatible images and
223
•
5
10
15
20
25
30
SP308970WOOO
the auxiliary images each having the same resolution as
that of the image before multiplexing decreases to be the
amount of data of the parallax images of two viewpoints.
Accordingly, in this case, since there are effects on the
compatibility and the data amount, circles are described
in correspondence with items "Compatibility" and "Data
Amount" represented in Fig. 77.
[0664]
In a case where multiplexing is performed using the
multiplexing pattern illustrated in (4) of Fig. 76, the
multiplexing mode and the encoding mode of compatible
images are the same as those of an existing mode, whereby
the compatibility is secured. In addition, since the
parallax images are multiplexed in the time direction,
the amount of data of the parallax images at each time is
less than that of the case of (3) of Fig. 76, whereby the
amount of data that can be transmitted increases.
Accordingly, even in a situation in which there is no
room for a transmission bandwidth for transmitting a
parallax image having the same resolution as the image
before multiplexing, a parallax image of which the
resolution is the same as the resolution of the image
before multiplexing can be transmitted, whereby the
accuracy of images of predetermined viewpoints to be
generated in the decoding device using the parallax image
is improved. Accordingly, in this case, there are
effects on the compatibility and the image quality of an
image generated using the parallax image, and there is a
remarkable effect on the data amount. Therefore, circles
are described in correspondence with items
"Compatibility" and "Image Quality" illustrated in Fig.
224
• SP308970WOOO
77, and a double circle is described in correspondence
with "Data Amount".
[0665]
The multiplexing process using the multiplexing
5 patterns illustrated in (1) to (4) of Fig. 76 is
performed, for example, when an image used for
broadcasting, the IP (Internet Protocol) that is
compliant with broadcasting, ATSC (Advanced Television
Systems Committee) 2.0 specifications, and ·the like,
10 next-generation broadcasting that is converged
broadcasting, or internet delivery is set as an encoding
target.
[0666]
In a case where multiplexing is performed using the
15 multiplexing pattern illustrated in (5) of Fig. 76, the
multiplexing mode and the encoding mode of compatible
images are the same as those of an existing mode, whereby
the compatibility is secured. In addition, since the
resolution of each parallax image is a half of the
20 resolution of the image, the amount of data of the
parallax images of the compatible image and the auxiliary
image decreases to be the amount of data of the parallax
images of two viewpoints. Accordingly, in this case,
there are effects on the compatibility and the data
25 amount, and circles are described in correspondence with
items "Compatibility" and "Data Amount" illustrated in
Fig. 77.
[ 0 667]
In a case where multiplexing is performed using the
30 multiplexing pattern illustrated in (6) of Fig. 76,
similarly to the case of (2) of Fig. 76, the
225
5
• SP308970WOOO
compatibility is secured, and the accuracy of images of
predetermined viewpoints generated in the decoding device
using the parallax images is improved. Accordingly, in
this case, there are effects on the compatibility and the
image quality of an image generated using parallax images,
and circles are described in correspondence with items
"Compatibility" and "Image Quality" illustrated in Fig.
77.
[0668]
10 In a case where multiplexing is performed using the
multiplexing pattern illustrated in (7) of Fig. 76,
similarly to the case of (3) of Fig. 76, the
compatibility is secured, and the amount of data of the
parallax image decreases. Accordingly, in this case,
15 there are effects on the compatibility and the data
amount, and circles are described in correspondence with
items "Compatibility" and "Data Amount" illustrated in
Fig. 77.
[0669]
20 In a case where multiplexing is performed using the
multiplexing pattern illustrated in (8) of Fig. 76,
similarly to the case of (4) of Fig. 76, the
compatibility is secured. In addition, similarly to the
case of (4) of Fig. 76, the amount of data of the
25 parallax images at each time is less than that of the
case of (7) of Fig. 76 , and as a result, the accuracy of
images of predetermined viewpoints generated using the
parallax images is improved. Accordingly, in this case,
there are effects on the compatibility and the image
30 quality of an image generated using the parallax images,
and there is a remarkable effect on the data amount.
226
• SP308970WOOO
Therefore, circles are described in correspondence with
items ~Compatibility" and ~Image Quality" illustrated in
Fig. 77, and a double circle is described in
correspondence with ~Data Amount".
5 [0670]
The multiplexing process using the multiplexing
patterns illustrated in (5), (7), and (8) of Fig. 76 is
performed, for example, when an image used for an
application for the BD, broadcasting, next-generation
10 broadcasting or internet delivery is set as an encoding
target. In addition, the multiplexing process using the
multiplexing pattern illustrated in (6) of Fig. 76 is
performed, for example, when an image used for an
application for the BD, next-generation broadcasting or
15 internet delivery is set as an encoding target.
20
25
30
[0671 ]
Fig. 78 is a diagram that illustrates an example of
the multiplexing pattern of the encoding target in a case
where the number of viewpoints of the compatible image is
one, and the number of viewpoints of the auxiliary image
is two.
[0672]
As illustrated in (1) of Fig. 78, the encoding
device 50 illustrated in Fig. 2 encodes an image A1 that
is a compatible image in accordance with the AVC mode.
In addition, the encoding device 50 multiplexes images B1
and C1 that are auxiliary images and parallax images B1'
and C1' that are the parallax images of the auxiliary
images in the time direction. Then, the encoding device
50 encodes a parallax image A1' that is the parallax
image of the compatible image, a multiplexed image of the
227
I
IIiII
Iii~
:K
¥
•
5
10
15
20
25
30
SP308970WOOO
auxiliary images, and a multiplexed image of the parallax
images of the auxiliary images in accordance with the 3DV
mode that is compliant with the MVe mode.
[0673]
The encoding device 50, as illustrated in (2) of
Fig. 78, may encode the parallax images A1' to e1' in
accordance with the 3DV mode that is compliant with the
Mve mode without multiplexing the parallax images. In
addition, as illustrated in (3) of Fig. 78, the encoding
device 50, similarly to the encoding device 480, may
encode a common parallax image Bel' of images Band e
that are auxiliary images instead of the multiplexed
image of the parallax images of the auxiliary images.
[0674]
In addition, as illustrated in (4) of Fig. 78, the
encoding device 50, similarly to the encoding device 440,
may multiplex the parallax images A1' to e1' in
accordance with the frame sequential mode and encode a
resultant multiplexed image in accordance with the 3DV
mode that is compliant with the Mve mode without
multiplexing the parallax images B1' and e1' in the
spatial direction.
[0675]
The effects of the multiplexing processes using the
multiplexing patterns illustrated in (1) to (4) of Fig.
78 and the encoding targets at the time of performing the
multiplexing processes using the multiplexing patterns
are the same as the effects and the encoding targets of
the multiplexing process using the multiplexing patterns
illustrated in (5) to (8) of Fig. 76. However, in the
multiplexing process using the multiplexing pattern
228
•
5
SP308970WOOO
illustrated in (1) of Fig. 78, the resolution of the
parallax image of the compatible image is the same as the
resolution of the compatible image, and a decrease in the
amount of dgta of the parallax image as an effect of this
multiplexing process is an effect only for the parallax
image of the auxiliary image.
[0676]
Fig. 79 is a diagram that illustrates an example of
the multiplexing pattern of an encoding target in a case
10 where the number of viewpoints of the compatible image is
two, and the number of viewpoints of the auxiliary image
is zero.
[0677]
In a case where the number of viewpoints of the
15 compatible images is two, and the number of viewpoints of
the auxiliary image is zero, as illustrated in (1) of Fig.
79, similarly to the case of the encoding device 140
illustrated in Fig. 14, the images A2 and B2 that are
compatible images are multiplexed in the spatial
20 direction and are encoded in accordance with the Ave mode.
In addition, the parallax images A2' and B2' that are the
parallax images of the compatible images are multiplexed
in the spatial direction and are encoded in accordance
with the 3DV mode that is compliant with the Ave mode.
25 [0678]
As illustrated in (2) of Fig. 79, the parallax
images A2' and B2' may be encoded in accordance with the
3DV mode that is compliant with the Mve mode without
being multiplexed. In addition, as illustrated in (3) of
30 Fig. 79, similarly to the case of the encoding device 480,
the common parallax image AB2' may be encoded instead of
229
• SP308970WOOO
the parallax images A2' and B2' .
[0679]
Furthermore, as illustrated in (4) of Fig. 79,
similarly to the case of the encoding device 440, the
5 parallax images A2' and B2' may be multiplexed in the
frame sequential mode and be encoded without being
multiplexed in the time direction.
[0680]
In addition, as illustrated in (5) of Fig. 79,
10 similarly to the encoding device 180, it may be
configured such that the image A2 is encoded in
accordance with the Ave mode, and the image B2 is encoded
with the Mve mode in which the image A2 is set as a base
view without multiplexing the images A2 and B2 that are
15 compatible images.
[0681]
In this case, as illustrated in (6) of Fig. 79,
similarly to the case of (2) of Fig. 79, the parallax
images may be encoded without being multiplexed, or, as
20 illustrated in (7) of Fig. 79, similarly to the case of
(3) of Fig. 79, the common parallax image AB2' may be
encoded. In addition, as illustrated in (8) of Fig. 79,
similarly to the case of (4) illustrated in Fig. 79, the
parallax images of the compatible images may be
25 multiplexed in accordance with the frame sequential mode
and be encoded.
[0682]
In addition, the effects of the multiplexing
processes using the multiplexing patterns illustrated in
30 (1) to (8) of Fig. 78 and the encoding targets at the
time of performing the multiplexing processes using the
230
•
5
10
15
SP308970WOOO
multiplexing patterns are the same as the effects and the
encoding targets of the multiplexing processes using the
multiplexing patterns illustrated in (1) to (8) of Fig.
76.
[0683]
In the description presented above, although the
resolution of the parallax images that are multiplexed in
accordance with the frame sequential mode has been
described to be the same as the resolution of the image
before the multiplexing process, the resolution may be
lower than the resolution of the image before the
multiplexing process. In addition, the auxiliary images,
similarly to the parallax images, may be also multiplexed
in accordance with the frame sequential mode.
[0684]
20
Furthermore, in the description presented above,
although the information representing the multiplexing
mode of the images and the information representing the
multiplexing mode of the parallax images have been
described to be transmitted in the encoding device,
information used for identifying the multiplexing
patterns illustrated in Figs. 76, 78, and 79 may be
transmitted.
[0685]
25 In addition, the encoding device may transmit a
flag that is used for identifying an application that
corresponds to an image that is an encoding target.
[0686]