SP308971WOOO
DESCRIPTION
IMAGE PROCESSING DEVICE AND METHOD
TECHNICAL FIELD
5 [0001]
The present technique relates to image processing
devices and methods, and more particularly, to image
processing devices and methods for correctly transmitting
image data in such a format as to generate multi-view
10 images.
BACKGROUND ART
[0002]
As video content, 3D (3 Dimensions) images that
15 allow stereoscopic viewing by using parallaxes have been
becoming common.
[0003]
As a popular technique for viewing such 3D images,
there is a technique by which a user sees images of two
20 viewpoints that are alternately displayed. In viewing
the images of two viewpoints, the user wears glasses that
open the shutter for the left eye when one of the images
of two viewpoints is displayed, and open the shutter for
the right eye when the other one of the images is
25 displayed (hereinafter referred to as the two-view
technique) .
[0004]
An example method of encoding and decoding 3D image
data for allowing stereoscopic viewing, or image data of
30 multiple viewpoints, is disclosed in Patent Document 1.
[0005]
1
SP308971WOOO
In recent years, there is an increasing demand for
a technique for stereoscopically viewing 3D images
without glasses for stereoscopic viewing (hereinafter
referred to a multi-view technique). In the case of a
5 multi-view technique, to display a 3D image, a multi-view
image is generated and displayed. A viewer can view an
image suitable for his/her location (viewpoint), without
glasses for stereoscopic viewing.
10 CITATION LIST
PATENT DOCUMENT
[0006]
Patent Document 1: Japanese Patent Application Laid-Open
No. 2008-182669
15
SUMMARY OF THE INVENTION
PROBLEMS TO BE SOLVED BY THE INVENTION
[0007]
There may be cases where such 3D images are
20 transmitted between devices, like other types of images.
For example, uncompressed 3D images might be transmitted
from a Blu-ray player (a reproducing device) or the like
to a digital television monitor (a display device) or the
like.
25 [0008]
However, there have been transmission methods only
for two-view 3D images, and no methods have been
suggested for transmitting multi-view 3D images.
[0009]
,f
30 Therefore, if such multi-view 3D images are
transmitted by a conventional transmission method for
2
SP308971WOOO
two-view 3D images, correct transmission cannot be
performed.
[0010]
The present technique has been developed in view of
5 those circumstances, and an object thereof is to allow
correct transmission of image data in such a format as to
generate multi-view images.
SOLUTIONS TO PROBLEMS
10 [0011]
One aspect of the present technique is an image
processing device that includes: a mode determining unit
that determines a compatible mode indicating
compatibility with an image data format by referring to
15 the format of stereoscopic image data to be used for
generating a multi-view image; a generating unit that
generates compatibility information as 3D information,
the compatibility information indicating the compatible
mode determined by the mode determining unit; and a
20 transmitting unit that transmits the stereoscopic image
data and the 3D information generated by the generating
unit to a transmission destination of the stereoscopic
image data.
[0012]
25 The generating unit can generate format information
as the 3D information, the format information indicating
the format of the stereoscopic image data.
[0013]
The generating unit can generate image multiplexing
30 information and parallax image multiplexing information
as the 3D information, the image multiplexing information
3
f
SP308971WOOO
indicating the multiplexing technique for the image data
in the stereoscopic image data, the parallax image
multiplexing information indicating the multiplexing
technique for the parallax image data in the stereoscopic
5 image data.
[0014]
The generating unit can generate identification
information as the 3D information, the identification
information being for identifying the multiplexing
10 pattern of the image data in the stereoscopic image data
or the multiplexing pattern of the parallax image data in
the stereoscopic image data.
[0015]
The transmitting unit can put the 3D information in
15 VSIFPC (Vendor Specific InfoFrame Packet Contents) of the
HDMI (High Definition Multimedia Interface) standard, and
transmit the 3D information.
[0016]
The transmitting unit can put the 3D information in
20 a free space in a region of the VSIFPC, and then transmit
the 3D information, the region of the VSIFPC being
designed to have information about 3D image data written
therein.
[0017]
25 Examples of the compatible mode may include mono
compatible mode indicating compatibility with
conventional two-dimensional image data, frame packing
compatible mode indicating compatibility with
conventional frame packing 3D image data, and
30 stereoscopic compatible mode indicating compatibility
with conventional stereoscopic 3D image data.
4
SP308971WOOO
[0018]
The transmitting unit can transmit data of one
frame of the stereoscopic image data, using an active
video interval equivalent to more than one cycle of a
5 vertical synchronization signal.
[0019]
The image processing device further includes an
interval setting unit that sets irregular cycles to the
vertical synchronization signal. The transmitting unit
10 can transmit data of one frame of the stereoscopic image
data, using the active video interval that is equivalent
to the cycles set by the interval setting unit.
[0020]
The image processing device further includes: a
15 capability information acquiring unit that acquires
capability information from a device at the transmission
destination of the stereoscopic image data, the
capability information indicating the processing
capability of the device; and a synchronization setting
20 unit that sets a synchronization method for the
stereoscopic image data, based on the processing
capability of the device indicated by the capability
information acquired by the capability information
acquiring unit. The generating unit can generate
25 synchronization information as the 3D information, the
synchronization information indicating the
synchronization method set by the synchronization setting
unit.
[0021]
,,
30 The one aspect of the present technique is also an
image processing method to be performed by an image
5
•
5
10
15
20
25
30
SP308971WOOO
processing device, the method including: determining a
compatible mode indicating compatibility with an image
data format by referring to the format of stereoscopic
image data to be used for generating a multi-view image,
the determining being performed by a mode determining
unit; generating compatibility information as 3D
information, the compatibility information indicating the
compatible mode determined by the mode determining unit,
the generating being performed by a generating unit; and
transmitting the stereoscopic image data and the 3D
information generated by the generating unit to a
transmission destination of the stereoscopic image data,
the transmitting being performed by a transmitting unit.
[0022]
The other aspect of the present technique is an
image processing device that includes: a receiving unit
that receives stereoscopic image data to be used for
generating a multi-view image, and receives a compatible
mode as 3D information, the compatible mode indicating
compatibility of the format of the stereoscopic image
data with an image data format; a mode determining unit
that determines the compatible mode of the stereoscopic
image data received by the receiving unit, based on the
3D information received by the receiving unit; and a
synchronization control unit that controls
synchronization of processing of the stereoscopic image
data received by the receiving unit, based on the
compatible mode determined by the mode determining unit.
[0023]
The receiving unit can receive format information
as the 3D information, the format information indicating
6
SP308971WOOO
the format of the stereoscopic image data.
[0024]
The receiving unit can generate image multiplexing
information and parallax image multiplexing information
5 as the 3D information, the image multiplexing information
indicating the multiplexing technique for the image data
in the stereoscopic image data, the parallax image
multiplexing information indicating the multiplexing
technique for the parallax image data in the stereoscopic
10 image data.
[0025]
The receiving unit can receive identification
information as the 3D information, the identification
information being for identifying the multiplexing
15 pattern of the image data in the stereoscopic image data
or the multiplexing pattern of the parallax image data in
the stereoscopic image data.
[0026]
The receiving unit can receive the 3D information
20 that is written in VSIFPC (Vendor Specific InfoFrame
Packet Contents) of the HDMI (High Definition Multimedia
Interface) standard, and is then transmitted.
[0027]
The receiving unit can receive the 3D information
25 that is written in a free space in a region of the VSIFPC
and is then transmitted, the region of the VSIFPC being
designed to have information about 3D image data written
therein.
[0028]
30 Examples of the compatible mode may include mono
compatible mode indicating compatibility with
7
•
5
10
SP308971WOOO
conventional two-dimensional image data, frame packing
compatible mode indicating compatibility with
conventional frame packing 3D image data, and
stereoscopic compatible mode indicating compatibility
with conventional stereoscopic 3D image data.
[0029]
The receiving unit can receive data of one frame of
the stereoscopic image data transmitted using an active
video interval equivalent to more than one cycle of a
vertical synchronization signal.
[0030]
r
The receiving unit can receive data of one frame of
the stereoscopic image data transmitted using the active
video interval equivalent to more than one cycle of the
15 vertical synchronization signal, the cycles of the
vertical synchronization signal being irregular.
[0031]
The image processing device may further include a
capability information providing unit that provides
20 capability information indicating the processing
capability of the image processing device to the device
as the transmitter of the stereoscopic image data.
[0032]
The other aspect of the present technique is also
25 an image processing method to be performed by an image
processing device, the method including: receiving
stereoscopic image data to be used for generating a
multi-view image, and receives a compatible mode as 3D
information, the compatible mode indicating compatibility
30 of the format of the stereoscopic image data with an
image data format, the receiving being performed by a
8
SP308971WOOO
receiving unit; determining the compatible mode of the
stereoscopic image data received by the receiving unit,
based on the 3D information received by the receiving
unit, the determining being performed by a mode
5 determining unit; and controlling synchronization of
processing of the stereoscopic image data received by the
receiving unit, based on the compatible mode determined
by the mode determining unit, the controlling being
performed by a synchronization control unit.
10 [0033]
In the one aspect of the present technique, the
format of stereoscopic image data to be used for
generating a multi-view image is referred to, and a
compatible mode indicating compatibility with an image
15 data format is determined. Compatibility information
indicating the compatible mode is generated as 3D
information, and the stereoscopic image data and the 3D
information are transmitted to a transmission destination
of the stereoscopic image data.
20 [0034]
In the other aspect of the present technique,
stereoscopic image data to be used for generating a
multi-view image is received, and a compatible mode
indicating compatibility of the format of the
25 stereoscopic image data with an image data format is also
received as 3D information. Based on the received 3D
information, the compatible mode of the stereoscopic
image data is determined. Based on the compatible mode,
synchronization of the processing of the stereoscopic
30 image data is controlled.
9
SP308971WOOO
EFFECTS OF THE INVENTION
[0035]
According to the present technique, communications
can be performed. Particularly, image data in such a
5 format as to generate multi-view images can be correctly
transmitted.
BRIEF DESCRIPTION OF DRAWINGS
[0036]
10 Fig. 1 is a block diagram showing a typical example
structure of an encoding device of a 3D image system.
Fig. 2 is a block diagram showing a typical example
structure of a decoding device corresponding to the
encoding device of the 3D image system shown in Fig. 1.
15 Fig. 3 is a block diagram showing another example
structure of an encoding device of a 3D image system.
Fig. 4 is a block diagram showing a typical example
structure of a decoding device corresponding to the
encoding device of the 3D image system shown in Fig. 3.
20 Fig. 5 is a block diagram showing yet another
example structure of an encoding device of a 3D image
system.
Fig. 6 is a block diagram showing a typical example
structure of a decoding device corresponding to the
25 encoding device of the 3D image system shown in Fig. 5.
Fig. 7 is a block diagram showing a typical example
structure of a transmission system to which the present
technique is applied.
Fig. 8 is a block diagram showing a typical example
30 structure of the reproducing device shown in Fig. 7.
Fig. 9 is a block diagram showing a typical example
10
f
SP308971WOOO
structure of the display device shown in Fig. 7.
Fig. 10 is a block diagram showing a typical
example structure of a HDMI transmission system.
Fig. 11 is a diagram showing an example structure
5 of Vendor Specific infoFrame Packet Contents.
Fig. 12 is a diagram for explaining an example
expansion of 3D_Structure.
Fig. 13 is a diagram for explaining an example of
conventional image transmission.
10 Fig. 14 is a diagram for explaining an example of
image transmission to which the present technique is
applied.
Fig. 15 is a diagram for explaining another example
of image transmission to which the present technique is
15 applied.
Fig. 16 is a diagram for explaining yet another
example of image transmission to which the present
technique is applied.
Fig. 17 is a diagram for explaining still another
20 example of image transmission to which the present
technique is applied.
Fig. 18 is a diagram for explaining yet another
example of image transmission to which the present
technique is applied.
25 Fig. 19 is a diagram for explaining still another
example of image transmission to which the present
technique is applied.
Fig. 20 is a diagram for explaining yet another
example of image transmission to which the present
30 technique is applied.
Fig. 21 is a flowchart for explaining an example of
11
SP308971WOOO
a transmission processing flow.
Fig. 22 is a flowchart for explaining an example of
a reception processing flow.
Fig. 23 is a diagram for showing an example
5 structure of E-EDID data.
Fig. 24 is a diagram showing an example structure
of Vendor-Specific Data Block.
Fig. 25 is a diagram for explaining an example
expansion of 3D_Structure_ALL_X.
10 Fig. 26 is a diagram for explaining an example
expansion of 3D_Structure_X.
Fig. 27 is a block diagram showing another example
structure of the reproducing device shown in Fig. 7.
Fig. 28 is a block diagram showing another example
15 structure of the display device shown in Fig. 7.
Fig. 29 is a flowchart for explaining another
example of a transmission processing flow.
Fig. 30 is a flowchart for explaining another
example of a reception processing flow.
20 Fig. 31 is a block diagram showing another example
structure of an encoding device of a 3D image system.
Fig. 32 is a diagram showing an example description
of compatibility information and parallax image
information.
25 Fig. 33 is a diagram showing an example specific
description of the parallax image information shown in
Fig. 32.
Fig. 34 is a diagram showing an example description
of compatibility information and parallax image
30 information in an auxiliary stream.
Fig. 35 is a diagram showing an example specific
12
f
SP308971WOOO
description of the parallax image information shown in
Fig. 34.
Fig. 36 is a flowchart for explaining an encoding
operation by the encoding device shown in Fig. 31.
5 Fig. 37 is a flowchart for explaining the encoding
operation by the encoding device shown in Fig. 31.
Fig. 38 is a diagram showing an example structure
of a decoding device corresponding to the encoding device
shown in Fig. 31.
10 Fig. 39 is a flowchart for explaining a decoding
operation by the decoding device shown in Fig. 38.
Fig. 40 is a block diagram showing another example
structure of an encoding device of a 3D image system.
Fig. 41 is a flowchart for explaining an encoding
15 operation by the encoding device shown in Fig. 40.
Fig. 42 is a flowchart for explaining the encoding
operation by the encoding device shown in Fig. 40.
Fig. 43 is a diagram showing an example structure
of a decoding device corresponding to the encoding device
20 shown in Fig. 40.
Fig. 44 is a flowchart for explaining a decoding
operation by the decoding device shown in Fig. 43.
Fig. 45 is a diagram showing examples of
multiplexing patterns for objects to be encoded.
25 Fig. 46 is a diagram showing the characteristics of
the effects of multiplexing.
Fig. 47 is a diagram showing other examples of
multiplexing patterns for objects to be encoded.
Fig. 48 is a diagram showing yet other examples of
30 multiplexing patterns for objects to be encoded.
Fig. 49 is a block diagram showing another example
13
SP308971WOOO
structure of an encoding device of a 3D image system.
Fig. 50 is a flowchart for explaining an encoding
operation by the encoding device shown in Fig. 49.
Fig. 51 is a flowchart for explaining the encoding
5 operation by the encoding device shown in Fig. 49.
Fig. 52 is a diagram showing an example structure
of a decoding device corresponding to the encoding device
shown in Fig. 49.
Fig. 53 is a flowchart for explaining a decoding
10 operation by the decoding device shown in Fig. 52.
Fig. 54 is a diagram showing an example structure
of a bit stream.
Fig. 55 is a diagram showing an example description
of the Subset SPS of the 3DV technique for parallax
15 images shown in Fig. 54.
Fig. 56 is a diagram showing an example description
of the extension information for parallax images shown in
Fig. 55.
Fig. 57 is a diagram showing an example description
20 of the VUI extension information for parallax images in
shown in Fig. 55.
Fig. 58 is a diagram showing an example description
of the NAL header of 3DV encoded data.
Fig. 59 is a diagram showing an example description
25 of the header extension information for parallax images
shown in Fig. 58.
Fig. 60 is a diagram showing an example description
of the SEI of the 3DV technique shown in Fig. 54.
Fig. 61 is a diagram showing another example
30 structure of a bit stream.
Fig. 62 is a diagram showing an example description
14
SP308971WOOO
of the Subset SPS of the 3DV technique for images shown
in Fig. 61.
Fig. 63 is a diagram showing an example structure
of a bit stream in a case where the 3DV technique is
5 compliant with the HEVC technique.
Fig. 64 is a diagram showing an example description
of the SPS shown in Fig. 63.
Fig. 65 is a diagram showing an example description
of the Subset SPS shown in Fig. 64.
10 Fig. 66 is a diagram showing an example description
of the VUI information in the Subset SPS shown in Fig. 65.
Fig. 67 is a diagram showing an example description
of the SEI of the 3DV technique shown in Fig. 63.
Fig. 68 is a block diagram showing a typical
15 example structure of a computer to which the present
technique is applied.
Fig. 69 is a diagram schematically showing the
structure of a television apparatus to which the present
technique is applied.
20 Fig. 70 is a diagram schematically showing the
structure of a portable telephone device to which the
present technique is applied.
Fig. 71 is a diagram schematically showing the
structure of a recording/reproducing device to which the
25 present technique is applied.
Fig. 72 is a diagram schematically showing the
structure of an imaging device to which the present
technique is applied.
30 MODE FOR CARRYING OUT THE INVENTION
[0037]
15
SP308971WOOO
The following is a description of modes for
carrying out the present technique (hereinafter referred
to as the embodiments). Explanation will be made in the
following order. 1. First Embodiment (Transmission
5 System) 2. Second Embodiment (Transmission System) 3.
Third Embodiment (Transmission System) 4. Fourth
Embodiment (Transmission System) 5. Fifth Embodiment
(Transmission System) 6. Bit Stream in the Embodiment 7.
Bit Stream in a Case Where the 3DV Format is Compliant
10 with HEVC Format 8. Sixth Embodiment (Computer) 9.
Seventh Embodiment (Television Apparatus) 10. Eighth
Embodiment (Portable Telephone Device) 11. Ninth
Embodiment (Recording/Reproducing Device) 12. Tenth
Embodiment (Imaging Device)
15 [0038]
<1. First Embodiment>
[Example of a 3D Image System]
First, a system that handles stereoscopic images
(3D images) that allow stereoscopic viewing by using the
20 parallax between images. The 3D image system described
below is a system that handles 3D images in a multi-view
format by generating and displaying multi-view images.
In the case of 3D images, a user can stereoscopically
view images with the naked eye (without glasses for
25 stereoscopic viewing) .
[0039]
Hereinafter, the multi-view 3D image format in
which multi-view images are generated and displayed will
be referred to as 3DV. Here, 3DV is a format (a
30 technique) compliant with AVC, MVC, the later described
HEVC (High Efficiency Video Coding), or the like.
16
rIt
SP308971WOOO
[0040]
In the 3DV format, 3D image data in a multi-view
format are encoded on the transmission side when the 3D
image data is transmitted, and the encoded data is
5 decoded on the reception side. The 3DV format has
compatibility with conventional other image formats.
[0041]
That is, in the 3DV format, multi-view images
formed as 3D images include images to be encoded by a
10 conventional encoding technique so as to secure the
compatibility with conventional encoding devices, and
other images. In the following, each image to be encoded
by a conventional encoding technique will be referred to
as a compatible image, and other images for generating,
15 by using the compatible images, images each having a
larger number of viewpoints than the number of viewpoints
of each compatible image will be referred to as auxiliary
images.
[0042]
fI
20 In the following, examples of the data transmission
will be described.
[0043]
[Example Structure of an Encoding Device]
First, an example in which 3D images are
25 transmitted in such a format as to secure compatibility
with conventional 20 images (Mono Compatible Mode) is
described.
[0044]
Fig. 1 is a block diagram showing a typical example
30 structure of an encoding device of a 3D image system.
The encoding device 50 shown in Fig. 1 is a device on the
17
25
SP30897lWOOO
transmission side of the 3D image system, and encodes 3D
images in Mono Compatible Mode.
[0045]
As shown in Fig. 1, in the encoding device 50, an
5 imaging unit 5lA captures a HD (High Definition) image of
a predetermined viewpoint as an image AI, and supplies
the image Al to an image converting unit 52, a parallax
image generating unit 53, and an inter-viewpoint distance
information generating unit 56. In a position at a
10 distance Adlu from the imaging unit 5lA in a horizontal
direction, an imaging unit 5lB captures a HD image, as an
image Bl, of a different viewpoint from that of the image
AI, and supplies the image Bl to the image converting
unit 52, the parallax image generating unit 53, and the
15 inter-viewpoint distance information generating unit 56.
In a position at a distance AdlAc from the imaging unit
5lA in the opposite horizontal direction from the imaging
unit 5lB, an imaging unit 5lC captures a HD image, as an
image Cl, of a different viewpoint from those of the
20 image Al and the image Bl, and supplies the image Cl to
the image converting unit 52, the parallax image
generating unit 53, and the inter-viewpoint distance
information generating unit 56.
[0046]
In the following, the viewpoints corresponding to
the image Bl and the image Cl are the outer viewpoints
among the viewpoints of an image that can be perceived as
a 3D image. Accordingly, using the images Al through Cl,
the decoding device corresponding to the encoding device
30 50 can generate a multi-view image by interpolating an
image of a viewpoint that is located further inside
18
'III
SP308971WOOO
compared with the viewpoints of the image Bl and the
image Cl. As a result, a multi-view image can be
generated with higher precision than in a case where an
image of an outer viewpoint is interpolated
5 (extrapolated) with the use of images of inner viewpoints.
[0047]
The decoding device corresponding to the encoding
device 50 may of course interpolate (extrapolate) an
image of a viewpoint that is located further outside
10 compared with those of the image Bl and the image Cl.
Also, the distance ~dl~ and the distance ~dlAc may be
constant, or may vary with time.
[0048]
The image converting unit 52 determines the image
15 Al to be a compatible image, as the image Al is supplied
from the inner imaging unit 51A of the imaging units 51A
through 51C in the horizontal direction. The image
converting unit 52 supplies information designating the
image Al as the compatible image to a compatibility
20 information generating unit 55. The image converting
unit 52 supplies the image Al as the compatible image as
it is to an encoder 58.
[0049]
The image converting unit 52 also determines the
25 image Bl and the image Cl to be auxiliary images, and
multiplexes those images other than the image Al by a
predetermined multiplexing technique. For example, the
image converting unit 52 reduces the horizontal
resolution of the image Bl and the image Cl by half
30 (hereinafter referred to as the 1/2 resolution image Bl
and the 1/2 resolution image Cl), and arranges the image
19
f,
•
5
10
15
20
25
30
SP308971WOOO
Bl and the image Cl having the halved resolution in the
horizontal direction, to achieve the original image size
of one frame, as shown in Fig. 1. This multiplexing
technique is called the Side By Side (SBS) technique.
[0050]
The image converting unit 52 supplies the
multiplexed image obtained as a result of the
multiplexing to the encoder 58, and supplies information
indicating the multiplexing technique for the auxiliary
images to an image information generating unit 54.
[0051]
Using the images Al through Cl supplied from the
imaging units 51A through 51C, the parallax image
generating unit 53 (the parallax image generating unit)
detects the parallax of each pixel of the images Al
through Cl. The parallax image generating unit 53
generates a parallax image Al' indicating the parallax of
each pixel of the image Al as the compatible image, and
supplies the parallax image Al' directly to the encoder
58.
[0052]
The parallax image generating unit 53 also
generates a parallax image Bl' indicating the parallax of
each pixel of the image Bl as an auxiliary image, and a
parallax image Cl' indicating the parallax of each pixel
of the image C as an auxiliary image. The parallax
images Bl' and Cl' are multiplexed by the same method
(the Side By Side technique) as the method used for
multiplexing the image Bl and the image Cl. The parallax
image generating unit 53 supplies the resultant
multiplexed image to the encoder 58. The parallax image
20
• SP308971WOOO
generating unit 53 supplies information indicating the
multiplexing technique for the parallax images of the
auxiliary images to a parallax image information
generating unit 57.
5 [0053]
Based on the information supplied from the image
converting unit 52, the image information generating unit
54 generates information indicating the multiplexing
technique for the auxiliary images and the like as image
10 information about the compatible image and the auxiliary
images, and supplies the image information to the encoder
58.
[0054]
Based on the information supplied from the image
15 converting unit 52, the compatibility information
generating unit 55 (the generating unit) generates
information designating the compatible image and a
compatible mode as compatibility information about
compatibility, and supplies the compatibility information
20 to the encoder 58.
[0055]
It should be noted that the compatible mode is the
mode indicating the encoding method or multiplexing
method for the compatible image (or indicating whether or
25 not multiplexing is to be performed, the type of
multiplexing technique, or the like). For example, the
compatible mode may be a mono mode indicating an encoding
method for encoding a compatible image of a single
viewpoint by the Ave technique, a frame packing mode
30 indicating an encoding method for multiplexing compatible
images of two viewpoints and encoding the multiplexed
21
,lI
I
5
SP308971WOOO
image by the AVC technique, a stereo mode indicating an
encoding method for encoding compatible images of two
viewpoints by the MVC technique, or the like.
[0056]
Using the images Al through Cl supplied from the
imaging unit 51A through SIC, the inter-viewpoint
distance information generating unit 56 (the interviewpoint
distance information generating unit) detects
each distance between two images among the images Al
10 through Cl (hereinafter referred to as the inter-
III 1
viewpoint distance). For example, the inter-viewpoint
distance information generating unit 56 detects inter-
15
viewpoint distances that are the distance ~dlAB between
the imaging unit 51A and the imaging unit SIB in the
horizontal direction, and the distance ~dlAc between the
imaging unit 51A and the imaging unit SIC in the
III
horizontal direction. The inter-viewpoint distance
information generating unit 56 generates information
indicating the inter-viewpoint distances and the like as
20 inter-viewpoint distance information about the interviewpoint
distances, and supplies the inter-viewpoint
distance information to the encoder 58.
[0057]
Based on the information supplied from the parallax
25 image generating unit 53, the parallax image information
generating unit 57 generates information indicating the
multiplexing technique for the parallax images of the
auxiliary images and the like as parallax image
information about parallax images, and supplies the
30 parallax image information to the encoder 58.
[0058]
22
10
30
SP308971WOOO
The encoder 58 is formed with a compatible encoder
61 and an auxiliary encoder 62. The compatible encoder
61 (the compatible image encoding unit) encodes the image
A1 as the compatible image supplied from the image
5 converting unit 52 by the conventional Ave technique, and
adds various kinds of information to the encoded image.
The resultant encoded stream is supplied as a compatible
stream to a multiplexing unit 59.
[0059]
The auxiliary encoder 62 (the auxiliary image
encoding unit) encodes, by a predetermined technique, the
multiplexed image of the auxiliary images from the image
converting unit 52, and the parallax image All of the
compatible image and the multiplexed image of the
15 parallax images of the auxiliary images from the parallax
image generating unit 53. The encoding technique used by
the auxiliary encoder 62 may be the Ave technique, the
Mve technique, the MPEG2 (Moving Picture Experts Group
phase 2) technique, or the like.
20 [0060]
The auxiliary encoder 62 also generates an encoded
stream by adding the image information from the image
information generating unit 54, the compatibility
information from the compatibility information generating
25 unit 55, the inter-viewpoint distance information from
the inter-viewpoint distance information generating unit
56, the parallax image information from the parallax
image information generating unit 57, and the like, to
the encoded images obtained as a result of the encoding.
The auxiliary encoder 62 supplies the encoded stream as
an auxiliary stream to the multiplexing unit 59.
23
SP308971WOOO
[0061]
The multiplexing unit 59 generates TSs (Transport
Streams) from the compatible stream supplied from the
compatible encoder 61 and the auxiliary stream supplied
5 from the auxiliary encoder 62, and multiplexes the TSs.
The multiplexing unit 59 (the transmitting unit)
transmits the multiplexed stream obtained as a result of
the multiplexing.
[0062]
10 [Example Structure of a Decoding Device]
Fig. 2 is a diagram showing an example structure of
a decoding device (of Mono Compatible Mode) that decodes
the multiplexed stream transmitted from the encoding
device 50 shown in Fig. 1.
15 [0063]
The decoding device 120 shown in Fig. 2 includes a
separating unit 121, a decoder 122, an image information
acquiring unit 123, an inter-viewpoint distance
information acquiring unit 124, a parallax image
20 information acquiring unit 125, a compatibility
information acquiring unit 126, and an image generating
unit 127. The decoding device 120 decodes the
multiplexed stream transmitted from the encoding device
50, generates a single-view image or a multi-view image,
25 and causes a display device (not shown) to display the
generated image.
[0064]
Specifically, the separating unit 121 (the
receiving unit) of the decoding device 120 receives the
30 multiplexed stream transmitted from the encoding device
50, and separates the TSs from each other. The
24
SP308971WOOO
separating unit 121 extracts the compatible stream and
the auxiliary stream from the separated TSs, and supplies
the extracted streams to the decoder 122.
5
10
15
[0065]
The decoder 122 is formed with a compatible decoder
131 and an auxiliary decoder 132.
Based on information that is supplied from the auxiliary
decoder 132 and is for identifying the compatible stream,
the compatible decoder 131 (the compatible image decoding
unit) of the decoder 122 distinguishes the compatible
stream from the auxiliary stream supplied from the
separating unit 121. The compatible decoder 131 decodes
the encoded compatible image contained in the compatible
stream by a technique corresponding to the Ave technique,
and supplies the resultant image A1 to the image
generating unit 127.
[0066]
fII[I!
20
Based on the compatibility information contained in
the auxiliary stream supplied from the separating unit
121, the auxiliary decoder 132 supplies information for
identifying a compatible stream to the compatible decoder
131. Based on the compatibility information, the
auxiliary decoder 132 identifies the auxiliary stream
between the compatible stream and the auxiliary stream
25 supplied from the separating unit 121. The auxiliary
30
decoder 132 (the auxiliary image decoding unit) decodes,
by a technique corresponding to the auxiliary encoder 62
shown in Fig. 5, the multiplexed image of the encoded
auxiliary images, the parallax image Al of the compatible
image, and the multiplexed image of the parallax images
of the auxiliary images, which are contained in the
25
SP308971WOOO
auxiliary stream supplied from the separating unit 121.
[0067]
The auxiliary decoder 132 supplies the multiplexed
image of the auxiliary images, the parallax image AI of
5 the compatible image, and the multiplexed image of the
parallax images of the auxiliary images, which are
obtained as a result of the decoding, to the image
generating unit 127. Also, the auxiliary decoder 132
supplies the image information contained in the auxiliary
10 stream to the image information acquiring unit 123, and
supplies the inter-viewpoint distance information to the
inter-viewpoint distance information acquiring unit 124.
Further, the auxiliary decoder 132 supplies the parallax
image information contained in the auxiliary stream to
15 the parallax image information acquiring unit 125, and
supplies the compatibility information to the
compatibility information acquiring unit 126.
[0068]
The image information acquiring unit 123 acquires
20 the image information supplied from the auxiliary decoder
132, and supplies the image information to the image
generating unit 127. The inter-viewpoint distance
information acquiring unit 124 acquires the interviewpoint
distance information supplied from the
25 auxiliary decoder 132, and supplies the inter-viewpoint
distance information to the image generating unit 127.
[0069]
The parallax image information acquiring unit 125
acquires the parallax image information supplied from the
30 auxiliary decoder 132, and supplies the parallax image
information to the image generating unit 127. The
26
SP308971WOOO
compatibility information acquiring unit 126 acquires the
compatibility information supplied from the auxiliary
decoder 132, and supplies the compatibility information
to the image generating unit 127.
5 [0070]
In accordance with a display instruction from a
viewer, the image generating unit 127 outputs images, and
causes the display device (not shown) to display the
images. Specifically, in accordance with a multi-view 3D
10 image display instruction from a viewer, the image
generating unit 127 (the generating unit) generates
images that are of three or more viewpoints corresponding
to the display device (not shown) and have half the
resolution of the compatible image and the auxiliary
15 images, using the compatible image, the multiplexed image
of the auxiliary images, the parallax image of the
compatible image, and the multiplexed image of the
parallax images of the auxiliary images, based on the
image information from the image information acquiring
20 unit 123, the inter-viewpoint distance information from
the inter-viewpoint distance information acquiring unit
124, the parallax image information from the parallax
image information acquiring unit 125, the compatibility
information from the compatibility information acquiring
25 unit 126, and the like.
[0071]
More specifically, based on the information that is
contained in the parallax image information from the
parallax image information acquiring unit 125 and
30 indicates the multiplexing technique for the parallax
images of the auxiliary images, the image generating unit
27
SP308971WOOO
127 separates the parallax image of each of the auxiliary
images from the multiplexed image of the parallax images
of the auxiliary images, which have been multiplexed by
the Side By Side technique, for example. Also, based on
5 the information that is contained in the image
information from the image information acquiring unit 123
and indicates the multiplexing technique for the
auxiliary images, the image generating unit 127 separates
each of the auxiliary images from the multiplexed image
10 of the auxiliary images, which have been multiplexed by
the Side By Side technique, for example.
[0072]
Further, based on the inter-viewpoint distance
information and the number of viewpoints corresponding to
15 the display device (not shown), the image generating unit
127 determines the location of each of the viewpoints of
the multi-view image to be generated. Using the
compatible image, the respective auxiliary images, the
parallax image of the compatible image, and the parallax
20 images of the respective auxiliary images, the image
generating unit 127 generates images of the respective
viewpoints having the locations determined. The image
generating unit 127 then converts the resolution of the
generated images of the respective viewpoints to the
25 resolution calculated by dividing the resolution of the
compatible image and the auxiliary images by the number
of viewpoints, and combines the generated images. The
resultant image is displayed on the display device (not
shown)
30 [0073]
At this point, the multi-view image after the
28
SP308971WOOO
combining is displayed in such a manner that the viewable
angle varies at each viewpoint, and the viewer can view a
3D image without glasses by seeing the images of any two
viewpoints with both eyes.
5 [0074]
Also, in accordance with a 2D image display
instruction from a viewer, the image generating unit 127
outputs the image Al as the compatible image supplied
from the compatible decoder 131 of the decoder 122, and
10 causes the display device (not shown) to display the
image AI. This enables the viewer to view a 2D image.
[0075]
[Example Structure of an Encoding Device]
Next, an example in which 3D images are transmitted
15 in such a format as to secure compatibility with
conventional frame packing 3D images (Frame Packing
Compatible Mode) is described.
[0076]
Fig. 3 is a block diagram showing another example
20 structure of an encoding device to which the present
technique is applied. In the structure illustrated in
Fig. 3, the same components as those in Fig. 1 are
denoted by the same reference numerals as those in Fig. 1.
The explanations that have already been made will not be
25 repeated. The encoding device 140 shown in Fig. 3 is
basically the same as the encoding device 50 shown in Fig.
1, except that one set of compatible images and one set
of auxiliary images are generated from four images in
this Frame Packing Compatible Mode. Like the auxiliary
30 images, the compatible images are also multiplexed by the
Side By Side technique, for example.
29
SP308971WOOO
[0077]
The structure of the encoding device 140 shown in
Fig. 3 differs from the structure shown in Fig. 5 mainly
in that the imaging units 51A through 51C, the image
5 converting unit 52, the parallax image generating unit 53,
the inter-viewpoint distance information generating unit
56, and the encoder 58 are replaced with imaging units
141A through 1410, an image converting unit 142, a
parallax image generating unit 143, an inter-viewpoint
10 distance information generating unit 144, and an encoder
145.
[0078]
The encoding device 140 determines images of two
viewpoints among images of a large number of viewpoints
15 to be compatible images, and encodes the compatible
images by the AVC technique, to secure compatibility with
encoding devices that encode 3D images in a conventional
two-view format.
[0079]
20 Specifically, the imaging unit 141A of the encoding
device 140 captures a HO image of a predetermined
viewpoint as an image A2, and supplies the image A2 to
the image converting unit 142, the parallax image
generating unit 143, and the inter-viewpoint distance
25 information generating unit 144. In a position at a
distance ~d2AB from the imaging unit 141A in a horizontal
direction, the imaging unit 141B captures a HO image, as
an image B2, of a different viewpoint from that of the
image A2, and supplies the image B2 to the image
30 converting unit 142, the parallax image generating unit
143, and the inter-viewpoint distance information
30
20
SP308971WOOO
generating unit 144.
[0080]
In a position at a distance ~d2Bc from the imaging
unit 141B in the opposite horizontal direction from the
5 imaging unit 141A, the imaging unit 141C captures a HO
image, as an image C2, of a different viewpoint from
those of the image A2 and the image B2, and supplies the
image C2 to the image converting unit 142, the parallax
image generating unit 143, and the inter-viewpoint
10 distance information generating unit 144. In a position
at a distance ~d2AD from the imaging unit 141A in the
opposite horizontal direction from the imaging unit 141B,
the imaging unit 1410 captures a HO image, as an image 02,
of a different viewpoint from those of the images A2
15 through C2, and supplies the image 02 to the image
converting unit 142, the parallax image generating unit
143, and the inter-viewpoint distance information
generating unit 144.
[0081]
In the following, the viewpoints corresponding to
the image C2 and the image 02 are the outer viewpoints
among the viewpoints of the image that can be perceived
as a 3D image. Accordingly, using the images A2 through
02, the decoding device corresponding to the encoding
25 device 140 can generate a multi-view image by
interpolating an image of a viewpoint that is located
further inside compared with the viewpoints of the image
C2 and the image 02. As a result, a multi-view image can
be generated with higher precision than in a case where
30 an image of an outer viewpoint is interpolated with the
use of images of inner viewpoints. The image
31
SP308971WOOO
interpolation may of course be extrapolation, as in the
case illustrated in Fig. 1. The distance dd2AB , the
distance dd2Bc , and the distance dd2AD may be constant, or
may vary with time.
5 [0082]
The image converting unit 142 determines the image
A2 and the image B2 to be compatible images, as the image
A2 and the image B2 are supplied from the imaging unit
141A and the imaging unit 141B, which are located in
10 inner positions among the imaging units 141A through 1410
in the horizontal direction. The image converting unit
142 multiplexes the image A2 and the image B2 as the
compatible images by a predetermined multiplexing
technique, and supplies the multiplexed image to the
15 encoder 145.
[0083]
For example, the image converting unit 142
multiplexes the image A2 and the image B2 by the Side By
Side technique, as shown in Fig. 3. The image converting
20 unit 142 also supplies information designating the image
A2 and the image B2 as the compatible images to the
compatibility information generating unit 55.
[0084]
The image converting unit 142 also determines the
25 image C2 and the image 02 to be auxiliary images, and
multiplexes those images other than the image A2 and the
image B2 by a predetermined multiplexing technique. For
example, the image converting unit 142 may also multiplex
the image C2 and the image 02 by the Side By Side
30 technique.
[0085]
32
SP308971WOOO
The image converting unit 142 supplies the
multiplexed images obtained as a result of the
multiplexing to the encoder 145. The image converting
unit 142 supplies information indicating the multiplexing
5 techniques for the compatible images and the auxiliary
images to the image information generating unit 54.
[0086]
Using the images A2 through D2 supplied from the
imaging units 141A through 141D, the parallax image
10 generating unit 143 detects the parallax of each pixel of
the images A2 through D2. The parallax image generating
unit 143 generates a parallax image A2' indicating the
parallax of each pixel of the image A2 as a compatible
image and a parallax image B2' indicating the parallax of
15 each pixel of the image B2 as a compatible image, and
multiplexes those parallax images by a predetermined
multiplexing technique. For example, the parallax image
generating unit 143 multiplexes the parallax image A2'
and the parallax image B2' by the Side By Side technique.
20 The parallax image generating unit 143 supplies the
resultant multiplexed image to the encoder 145.
[0087]
The parallax image generating unit 143 also
generates a parallax image C2' indicating the parallax of
25 each pixel of the image C2 as an auxiliary image and a
parallax image D2' indicating the parallax of each pixel
of the image D2 as an auxiliary image, and multiplexes
those parallax images by a predetermined multiplexing
technique. For example, the parallax image generating
30 unit 143 multiplexes the parallax image C2' and the
parallax image D2' by the Side By Side technique. The
33
SP308971WOOO
parallax image generating unit 143 supplies the resultant
multiplexed image to the encoder 145. The parallax image
generating unit 143 supplies information indicating the
multiplexing techniques for the parallax images of the
5 compatible images and the auxiliary images to the
parallax image information generating unit 57.
[0088]
Using the images A2 through 02 supplied from the
imaging units 141A through 1410, the inter-viewpoint
10 distance information generating unit 144 detects the
inter-viewpoint distances among the images A2 through 02.
For example, the inter-viewpoint distance information
generating unit 144 detects inter-viewpoint distances
that are the distance ~d2AB between the imaging unit 141A
15 and the imaging unit 141B in the horizontal direction,
the distance ~d2Bc between the imaging unit 141B and the
imaging unit 141C in the horizontal direction, and the
distance ~d2AD between the imaging unit 141A and the
imaging unit 1410 in the horizontal direction. The
20 inter-viewpoint distance information generating unit 144
generates information indicating the inter-viewpoint
distances and the like as inter-viewpoint distance
information, and supplies the inter-viewpoint distance
information to the encoder 145.
25 [0089]
The encoder 145 is formed with a compatible encoder
151 and an auxiliary encoder 152. The compatible encoder
151 encodes the multiplexed image of the compatible
images supplied from the image converting unit 142 by the
30 conventional AVC technique, and adds various kinds of
information to the encoded image. The resultant encoded
34
• SP308971WOOO
stream is supplied as a compatible stream to the
multiplexing unit 59.
[0090]
The auxiliary encoder 152 encodes, by a
5 predetermined technique, the multiplexed image of the
auxiliary images from the image converting unit 142, and
the multiplexed image of the parallax images of the
compatible images and the multiplexed image of the
parallax images of the auxiliary images from the parallax
10 image generating unit 143. The encoding technique used
by the auxiliary encoder 152 may be the AVC technique,
the MVC technique, or the like.
[0091]
The auxiliary encoder 152 also generates an encoded
15 stream by adding the image information from the image
information generating unit 54, the compatibility
information from the compatibility information generating
unit 55, the inter-viewpoint distance information from
the inter-viewpoint distance information generating unit
20 144, the parallax image information from the parallax
image information generating unit 57, and the like, to
the encoded images obtained as a result of the encoding.
The auxiliary encoder 152 supplies the encoded stream as
an auxiliary stream to the multiplexing unit 59.
25 [0092]
[Example Structure of a Decoding Device]
Fig. 4 is a diagram showing an example structure of
a decoding device (of Frame Packing Compatible Mode) that
decodes the multiplexed stream transmitted from the
30 encoding device 140 shown in Fig. 3. In the structure
illustrated in Fig. 4, the same components as those in
35
5
10
SP308971WOOO
Fig. 2 are denoted by the same reference numerals as
those in Fig. 2. The explanations that have already been
made will not be repeated.
[0093]
The structure of the decoding device 170 shown in
Fig. 4 differs from the structure shown in Fig. 2 mainly
in that the image generating unit 127 is replaced with an
image generating unit 171. The decoding device 170
decodes the multiplexed stream of the Frame Packing
Compatible Mode transmitted from the encoding device 140,
generates a double-viewpoint image or a multi-view image,
and causes a display device (not shown) to display the
generated image.
[0094]
15 Specifically, in accordance with a display
instruction from a viewer, the image generating unit 171
of the decoding device 170 outputs images, and causes the
display device (not shown) to display the images.
Particularly, in accordance with a multi-view 3D image
20 display instruction from a viewer, the image generating
unit 171 generates images that are of three or more
viewpoints corresponding to the display device (not
shown) and have half the resolution of the compatible
images and the auxiliary images, using the multiplexed
25 image of the compatible images, the multiplexed image of
the auxiliary images, the multiplexed image of the
parallax images of the compatible images, and the
multiplexed image of the parallax images of the auxiliary
images, based on the image information from the image
30 information acquiring unit 123, the inter-viewpoint
distance information from the inter-viewpoint distance
36
SP308971WOOO
information acquiring unit 124, the parallax image
information from the parallax image information acquiring
unit 125, the compatibility information from the
compatibility information acquiring unit 126, and the
5 like.
[0095]
More specifically, based on the information that is
contained in the parallax image information from the
parallax image information acquiring unit 125 and
10 indicates the multiplexing technique for the parallax
images of the auxiliary images, the image generating unit
171 separates the parallax image of each of the auxiliary
images from the multiplexed image of the parallax images
of the auxiliary images. In a case where the parallax
15 images of the auxiliary images have been multiplexed by
the Side By Side technique, for example, the image
generating unit 171 separates the parallax images into
two images lying side by side.
[0096]
20 Also, based on the information that is contained in
the parallax image information and indicates the
multiplexing technique for the parallax images of the
compatible images, the image generating unit 171
separates the parallax image of each of the compatible
25 images from the multiplexed image of the parallax images
of the compatible images. In a case where the parallax
images of the compatible images have been multiplexed by
the Side By Side technique, for example, the image
generating unit 171 separates the parallax images into
30 two images lying side by side.
[0097]
37
•
5
10
15
20
25
30
SP308971WOOO
Further, based on the information that is contained
in the image information from the image information
acquiring unit 123 and indicates the multiplexing
technique for the auxiliary images, the image generating
unit 171 separates each of the auxiliary images from the
multiplexed image of the auxiliary images. In a case
where the auxiliary images have been multiplexed by the
Side By Side technique, for example, the image generating
unit 171 separates the auxiliary images into two images
lying side by side.
[0098]
Also, based on the information that is contained in
the image information and indicates the multiplexing
technique for the compatible images, the image generating
unit 171 separates each of the compatible images from the
multiplexed image of the compatible images. In a case
where the compatible images have been multiplexed by the
Side By Side technique, for example, the image generating
unit 171 separates the compatible images into two images
lying side by side.
[0099]
Also, based on the inter-viewpoint distance
information and the number of viewpoints corresponding to
the display device (not shown), the image generating unit
171 determines the location of each of the viewpoints of
the multi-view image to be generated. Using the
respective compatible images, the respective auxiliary
images, the parallax images of the respective compatible
images, and the parallax images of the respective
auxiliary images, the image generating unit 171 generates
images of the respective viewpoints having the locations
38
1
.. SP308971WOOO
determined. The image generating unit 171 then converts
the resolution of the generated images of the respective
viewpoints to the resolution calculated by dividing the
resolution of the compatible images and the auxiliary
5 images by the number of viewpoints, and combines the
generated images. The resultant image is displayed on
the display device (not shown).
[0100]
At this point, the multi-view image after the
10 combining is displayed in such a manner that the viewable
angle varies at each viewpoint, and the viewer can view a
3D image without glasses by seeing the images of any two
viewpoints with both eyes.
[0101]
15 In accordance with a two-view 3D image display
instruction from a viewer, the image generating unit 171
separates the multiplexed image of the compatible images
supplied from the decoder 122 into the image A2 and the
image B2 with half the resolution of the compatible
20 images, based on the image information from the image
information acquiring unit 123. The image generating
unit 171 alternately outputs the image A2 and the image
B2 with half the resolution of the separated compatible
images, and causes the display device (not shown) to
25 display those images. At this point, the viewer puts on
glasses that open one of the shutter for the left eye and
the shutter for the right eye when the image A2 is
displayed, and open the other one of the shutters when
the image B2 is displayed. Wearing the glasses, the
30 viewer sees the image A2 and the image B2 that are
alternately displayed on the display device. In this
39
" SP308971WOOO
manner, the viewer can view a 3D image.
[0102]
In the examples illustrated in Figs. 3 and 4, the
compatible images, the auxiliary images, and the parallax
5 images of the compatible and auxiliary images may be
multiplexed by TAB (Top And Bottom) technique (also
referred to as Top Over Bottom (TaB) technique), instead
of the Side By Side technique.
[0103]
10 The TAB (Top And Bottom) technique is a
multiplexing technique for halving the vertical
resolution of two images to be multiplexed, arranging the
two images having the halved resolution in the vertical
direction, and achieving the original image size of one
15 frame.
[0104]
[Example Structure of an Encoding Device]
Next, an example in which 3D images are transmitted
in such a format as to secure compatibility with
20 conventional stereoscopic 3D images (Stereo Scopic
Compatible Mode) is described. In the case of a
conventional stereoscopic 3D image, an image for the left
eye and an image for the right eye, which form parallaxes,
are alternately displayed.
25 [0105]
Fig. 5 is a block diagram showing another example
structure of an encoding device to which the present
technique is applied. In the structure illustrated in
Fig. 5, the same components as those in Fig. 1 are
30 denoted by the same reference numerals as those in Fig. 1.
The explanations that have already been made will not be
40
~ SP308971WOOO
repeated. The encoding device 140 shown in Fig. 5 is
basically the same as the encoding device 50 shown in Fig.
1, except that two compatible images and one set of
auxiliary images are generated from four images in this
5 Stereo Scopic Compatible Mode. The compatible images are
not multiplexed, and the auxiliary images are multiplexed
by the Side By Side technique, for example.
[0106]
The structure of the encoding device 180 shown in
10 Fig. 5 differs from the structure shown in Fig. 3 mainly
in that the image converting unit 142 and the encoder 145
are replaced with an image converting unit 181 and an
encoder 182. The encoding device 180 determines images
of two viewpoints among images of a large number of
15 viewpoints to be compatible images, and encodes the
compatible images by the MVC technique, to secure
compatibility with encoding devices that encode 3D images
in a conventional two-view format.
[0107]
20 Specifically, like the image converting unit 142
shown in Fig. 14, the image converting unit 181 of the
encoding device 180 determines the image A2 and the image
B2 to be compatible images, as the image A2 and the image
B2 are supplied from the imaging unit 141A and the
25 imaging unit 141B, which are located in inner positions
among the imaging units 141A through 1410 in the
horizontal direction. For example, the image A2 is the
image for the left eye, and the image B2 is the image for
the right eye.
30 [0108]
The image converting unit 181 then supplies the
41
5
10
15
20
SP308971WOOO
image Al and the image A2 as the compatible images
directly (without multiplexing the compatible images) to
the encoder 182. Like the image converting unit 142, the
image converting unit 181 also supplies information
designating the image A2 and the image B2 as the
compatible images to the compatibility information
generating unit 55.
[0109]
Like the image converting unit 142, the image
converting unit 181 also determines the image e2 and the
image D2 to be auxiliary images, and multiplexes those
images other than the image A2 and the image B2 by a
predetermined multiplexing technique. For example, the
image converting unit 142 may multiplex the image e2 and
the image D2 by the Side By Side technique.
[0110]
The image converting unit 181 supplies the
multiplexed image obtained as a result of the
multiplexing to the encoder 182, and supplies information
indicating the multiplexing technique for the auxiliary
images to the image information generating unit 54.
[0111]
The encoder 182 is formed with a compatible encoder
191 and the auxiliary encoder 152. The compatible
25 encoder 191 of the encoder 182 determines the image A2 of
the compatible images supplied from the image converting
unit 181 to be a base image, and encodes the base image
by the conventional Ave technique. The encoder 182
determines the image B2 to be a dependent image, and
30 encodes the dependent image by the conventional Mve
technique. The compatible encoder 191 generates an
42
SP308971WOOO
encoded stream by adding various kinds of information to
the resultant encoded images, and supplies the encoded
stream as a compatible stream to the multiplexing unit 59.
[0112]
5 It should be noted that parallax information is
generated in the same manner as in the case of the
encoding device 140 shown in Fig. 3.
[0113]
[Example Structure of a Decoding Device]
10 Fig. 6 is a diagram showing an example structure of
a decoding device (of Frame Packing Compatible Mode) that
decodes the multiplexed stream transmitted from the
encoding device 180 shown in Fig. 5. In the structure
illustrated in Fig. 6, the same components as those in
15 Fig. 4 are denoted by the same reference numerals as
those in Fig. 4. The explanations that have already been
made will not be repeated.
[0114]
The structure of the decoding device 200 shown in
20 Fig. 6 differs from the structure shown in Fig. 2 mainly
in that the decoder 122 and the image generating unit 127
are replaced with a decoder 201 and an image generating
unit 202. The decoding device 200 decodes the
multiplexed stream transmitted from the encoding device
25 180, generates a two-view image or a multi-view image,
and causes a display device (not shown) to display the
generated image.
[0115]
Specifically, the decoder 201 of the decoding
30 device 200 is formed with a compatible decoder 211 and
the auxiliary decoder 132. Based on information that is
43
SP308971WOOO
supplied from the auxiliary decoder 132 and is for
identifying the compatible stream, the compatible decoder
211 of the decoder 201 distinguishes the compatible
stream from the auxiliary stream supplied from the
5 separating unit 121, like the compatible decoder 131
shown in Fig. 2. The compatible decoder 211 decodes the
encoded compatible images contained in the compatible
stream by a technique corresponding to the MVC technique,
and supplies the resultant images A2 and B2 to the image
10 generating unit 202.
[0116]
In accordance with a display instruction from a
viewer, the image generating unit 202 outputs images, and
causes the display device (not shown) to display the
15 images. Specifically, in accordance with a multi-view 3D
image display instruction from a viewer, the image
generating unit 202 generates images that are of three or
more viewpoints corresponding to the display device (not
shown) and have half the resolution of the compatible
20 images and the auxiliary images, using the compatible
images, the multiplexed image of the auxiliary images,
the multiplexed image of the parallax images of the
compatible images, and the multiplexed image of the
parallax images of the auxiliary images, based on the
25 image information from the image information acquiring
unit 123, the inter-viewpoint distance information from
the inter-viewpoint distance information acquiring unit
124, the parallax image information from the parallax
image information acquiring unit 125, the compatibility
30 information from the compatibility information acquiring
unit 126, and the like.
44
SP308971WOOO
[0117]
More specifically, based on the information that is
contained in the parallax image information from the
parallax image information acquiring unit 125 and
5 indicates the multiplexing technique for the parallax
images of the auxiliary images, the image generating unit
202 separates the parallax image of each of the auxiliary
images from the multiplexed image of the parallax images
of the auxiliary images. In a case where the parallax
10 images of the auxiliary images have been multiplexed by
the Side By Side technique, for example, the image
generating unit 202 separates the parallax images of the
auxiliary images into two images lying side by side.
[0118]
15 Also, based on the information that is contained in
the parallax image information and indicates the
multiplexing technique for the parallax images of the
compatible images, the image generating unit 202
separates the parallax image of each of the compatible
20 images from the multiplexed image of the parallax images
of the compatible images. In a case where the parallax
images of the compatible images have been multiplexed by
the Side By Side technique, for example, the image
generating unit 202 separates the parallax images of the
25 compatible images into two images lying side by side.
[0119]
Further, based on the information that is contained
in the image information from the image information
acquiring unit 123 and indicates the multiplexing
30 technique for the auxiliary images, the image generating
unit 202 separates each of the auxiliary images from the
45
5
10
15
20
25
30
SP308971WOOO
multiplexed image of the auxiliary images. In a case
where the compatible images have been multiplexed by the
Side By Side technique, for example, the image generating
unit 202 separates the compatible images into two images
lying side by side.
[0120]
Also, based on the inter-viewpoint distance
information and the number of viewpoints corresponding to
the display device (not shown), the image generating unit
202 determines the location of each of the viewpoints of
the multi-view image to be generated. Using the
respective compatible images, the respective auxiliary
images, the parallax images of the respective compatible
images, and the parallax images of the respective
auxiliary images, the image generating unit 202 generates
images of the respective viewpoints having the locations
determined. The image generating unit 202 then converts
the resolution of the generated images of the respective
viewpoints to the resolution calculated by dividing the
resolution of the compatible images and the auxiliary
images by the number of viewpoints, and combines the
generated images, like the image generating unit 127.
The resultant image is displayed on the display device
(not shown).
[0121]
At this point, the multi-view image after the
combining is displayed in such a manner that the viewable
angle varies at each viewpoint, and the viewer can view a
3D image without glasses by seeing the images of any two
viewpoints with both eyes.
[0122]
46
SP308971WOOO
Also, in accordance with a two-view 3D image
display instruction from a viewer, the image generating
unit 202 alternately outputs the image A2 and the image
B2 as the compatible images supplied from the decoder 122,
5 and causes the display device (not shown) to display
those images. At this point, the viewer puts on glasses
that open one of the shutter for the left eye and the
shutter for the right eye when the image A2 is displayed,
and open the other one of the shutters when the image B2
10 is displayed. Wearing the glasses, the viewer sees the
image A2 and the image B2 that are alternately displayed
on the display device. In this manner, the viewer can
view a 3D image.
[0123]
15 [Baseband Data Transmission]
In the above described decoding device 120 shown in
Fig. 2, for example, the left side (the separating unit
121 and the decoder 122) of the dashed line L1 is
designed as a reproducing device, and the right side (the
20 image generating unit 127) of the dashed line L1 is
designed as a display device.
[0124]
The reproducing device may be a player or a
recorder (with a playback function) that reads and
25 reproduces 3D image data recorded on a recording medium,
for example. Also, the reproducing device may be a
network device such as a set-top box that obtains 3D
image data from outside and reproduces the obtained image
data. The display device may be a CRT display, a LCD, an
30 organic EL display, a projector, or the like.
[0125]
47
J
JI
SP308971WOOO
The compatible image, the auxiliary images, and the
parallax images of those images need to be transmitted as
baseband data from the reproducing device to the display
device in this case.
5 [0126]
The same applies to the case of the decoding device
170 shown in Fig. 4. The structure on the left side of
the dashed line L2 is the reproducing device, and the
structure on the right side is the display device. In
10 this case, the compatible images, the auxiliary images,
and the parallax images of those images need to be
transmitted as baseband data between the reproducing
device and the display device.
[0127]
15 The same applies to the case of the decoding device
200 shown in Fig. 6. The structure on the left side of
the dashed line L3 is the reproducing device, and the
structure on the right side is the display device. In
this case, the compatible images, the auxiliary images,
41
1j
20 and the parallax images of those images need to be
transmitted as baseband data between the reproducing
device and the display device.
[0128]
There are many standards for communications between
25 audio-visual devices such as the reproducing device and
the display device. However, there have been no baseband
transmission methods for transmitting data in a 3D image
format compliant with the 3DV format described with
reference to Figs. 1 through 6 between audio-visual
30 devices.
[0129]
48
SP308971WOOO
For example, there is HDMI (High Definition
Multimedia Interface) as an interface that connects
digital devices.
[0130]
5
10
HDMI is an interface for digital appliances, and
was developed by modifying DVI (Digital Visual Interface),
which is a standard for connections between PCs (Personal
Computers) and displays. HDMI was developed for audiovisual
appliances by adding a voice transmitting function,
a copyright protecting function (a function to prevent
unauthorized copying of digital content and the like),
and a color-difference transmitting function, to DVI.
The HDMI 1.0 specification was formulated in December,
1I
·1111•!
1
1j
2002.
15 [0131]
HDMI has the advantage that uncompressed digital
sound and video are transmitted without image and sound
quality degradation. HDMI is currently based on the
function with the full-spec high definition (full HD: the
20 resolution being 1920 pixels in a horizontal line and
1080 pixels in a vertical line). As uncompressed video
and sound can be transferred from a player to a
television device, there is no need to prepare a specialpurpose
chip such as a decoder or software. HDMI also
25 has an intelligent function to make connected devices
recognize each other.
[0132]
Also, video, audio, and control signals are
transmitted through a single cable. Accordingly, the
30 wiring in audio-visual devices can be advantageously
simplified. As control signals and the like can also be
49
SP308971WOOO
transmitted, coordination between audio-visual devices is
easy.
[0133]
HDMI is currently compatible with conventional 3D
5 image formats such as the frame packing and stereoscopic
formats.
[0134]
However, 3DV format data contains information that
is not specified in conventional 3D image formats, as
10 described above. Therefore, there is a possibility that
3DV format data cannot be correctly transmitted even in
accordance with the HDMI standard.
[0135]
[Structure of a Transmission System]
15 In view of this, the transmission of (decoded) 3D
image data in the 3DV format between a reproducing device
and a display device is described below.
[0136]
Fig. 7 is a block diagram showing a typical example
20 structure of a transmission system to which the present
technique is applied. The transmission system 300 shown
in Fig. 7 is a system that transmits decoded 3D image
data in the 3DV format as described above.
[0137]
25 As shown in Fig. 7, the transmission system 300
includes a reproducing device 301 and a display device
302, which are connected by a HDMI cable 303 in such a
manner that data transmission can be performed.
[0138]
30 The reproducing device 301 reproduces 3D image data
that is read from a recording medium or is obtained from
50
SP308971WOOO
another device, and supplies the data to the display
device 302 via the HDMI cable 303. The reproducing
device 301 has the structure (the structure for the
compatible image(s), the auxiliary images, and the
5 parallax images of those images) on the left side of one
of the dashed lines L1 through L3 in the above described
decoding devices.
[0139]
The display device 302 generates a multi-view image
10 for display from the 3D image data transmitted from the
reproducing device 301 via the HDMI cable 303, and
displays the multi-view image. The display device 302
has the structure (the structure for the compatible
image(s), the auxiliary images, and the parallax images
15 of those images) on the right side of one of the dashed
lines L1 through L3 in the above described decoding
devices.
[0140]
In the transmission system 300, 3D image data is
20 transmitted from the reproducing device 301 to the
display device 302 via the HDMI cable 303 in accordance
with the HDMI standard.
[0141]
[Structure of the Reproducing Device]
25 Fig. 8 is a diagram showing a typical example
structure of the part involved in data transmission in
the reproducing device 301, which is a transmitting-end
device in data transmission in accordance with the HDMI
standard.
30 [0142]
As shown in Fig. 8, the reproducing device 301
51
SP308971WOOO
includes a decoder 311, a mode determining unit 312, a 3D
information generating unit 313, and a communication unit
314.
[0143]
5 The decoder 311 decodes encoded 3D image data, and
supplies the baseband 3D image data to the mode
determining unit 312 and the communication unit 314.
[0144]
The mode determining unit 312 examines the
10 structure (format) of the supplied 3D image data, and
determines whether the mode of the 3D image data is Mono
Compatible Mode, Frame Packing (Side By Side) Compatible
Mode, Frame Packing (Top and Bottom) Compatible Mode, or
Stereo Scopic Compatible Mode. The determination result
15 is supplied to the 3D information generating unit 313.
[0145]
The 3D information generating unit 313 generates 3D
information that contains information indicating that the
3D image data is image data in the 3DV format, and
20 information indicating the compatibility-related mode
determined by the mode determining unit 312. The 3D
information is inserted to a predetermined portion of the
3D image data to be transmitted. For example, the 3D
information generating unit 313 extends VSIFPC (Vendor
25 Specific infoFrame Packet Contents) of the HDMI standard,
and stores the 3D information thereinto.
[0146]
The communication unit 314 performs communications
with the display device 302 in accordance with the HDMI
30 standard. For example, the communication unit 314
transmits the 3D image data supplied from the decoder 311
52
•
5
10
15
20
SP308971WOOO
to the display device 302.
[0147]
[Structure of the Display Device]
Fig. 9 is a diagram showing a typical example
structure of the part involved in data transmission in
the display device 302, which is a receiving-end device
in data transmission in accordance with the HDMI standard.
[0148]
As shown in Fig. 9, the display device 302 includes
a communication unit 321, a mode determining unit 322, a
synchronization setting unit 323, a synchronization
control unit 324, an image generating unit 325, and a
display unit 326.
[0149]
The communication unit 321 performs communications
with the reproducing device 301 in accordance with the
HDMI standard. For example, the communication unit 321
receives the 3D image data transmitted from the
reproducing device 301, and supplies the received data to
the mode determining unit 322 and the image generating
unit 325.
[0150]
The mode determining unit 322 refers to the 3D
image information contained in the 3D image data supplied
25 from the communication unit 321, and determines the mode
of the 3D image data. For example, the mode determining
unit 322 determines whether the mode of the 3D image data
is Mono Compatible Mode, Frame Packing (Side By Side)
Compatible Mode, Frame Packing (Top and Bottom)
30 Compatible Mode, or Stereo Scopic Compatible Mode. The
synchronization setting unit 323 is notified of the
53
• SP308971WOOO
determination result.
[0151]
The synchronization setting unit 323 sets a
synchronization method in accordance with the determined
5 mode. In accordance with the synchronization method set
by the synchronization setting unit 323, the
synchronization control unit 324 supplies a
synchronization signal to the image generating unit 325
and the display unit 326, to control the operating
10 timings of the image generating unit 325 and the display
unit 326.
[0152]
Using the 3D image data (the compatible image(s),
the auxiliary images, and the parallax images of those
15 images) supplied via the communication unit 321, the
image generating unit 325 generates a display multi-view
image having a number of viewpoints that is designated by
a user or the like or can be handled by the display unit
326. The multi-view image is supplied to the display
20 unit 326, which then displays the multi-view image.
[0153]
The display unit 326 includes a display such as a
CRT display or a LCD, and displays the display multi-view
image supplied from the image generating unit 325 toward
25 the respective viewpoints.
[0154]
[HOMI Data Transmission]
Next, the communication unit 314 (the HOMI
transmitting unit) of the reproducing device 301 and the
30 communication unit 321 (the HOMI receiving unit) of the
display device 302 are described. Fig. 10 is a block
54
t SP308971WOOO
diagram showing specific example structures of the
communication unit 314 and the communication unit 321.
[0155]
In an effective image interval (hereinafter also
5 referred to as the active video interval, where
appropriate) that is an interval calculated by
subtracting a horizontal blanking interval and a vertical
blanking interval from the interval of a predetermined
number of vertical synchronization signals, the
10 communication unit 314 serving as the HDMI transmitting
unit unidirectionally transmits a differential signal
corresponding to uncompressed pixel data of an image of
one screen to the communication unit 321 serving as the
HDMI receiving unit through channels. In the horizontal
15 blanking interval or the vertical blanking interval, the
communication unit 314 also unidirectionally transmits a
differential signal corresponding to the audio data
accompanying at least the image, control data, other
auxiliary data, and the like, to the communication unit
20 321 through the channels.
[0156]
That is, the communication unit 314 includes a HDMI
transmitter 331. The HDMI transmitter 331 converts
encoded data into the corresponding differential signal,
25 and unidirectionally and serially transmits the
differential signal to the communication unit 321
connected thereto via the HDMI cable 303, through the
channels, which are three TMDS channels #0, #1, and #2.
[0157]
30 Also, the HDMI transmitter 331 converts the audio
data accompanying the encoded data, as well as necessary
55
t
5
10
15
20
25
30
SP308971WOOO
control data, other auxiliary data, and the like, into
the corresponding differential signal, and
unidirectionally and serially transmits the differential
signal to the communication unit 321 connected thereto
via the HOMI cable 303, through the three TMOS channels
#0, #1, and #2.
[0158]
Further, the HOMI transmitter 331 transmits pixel
clocks synchronized with the pixel data being transmitted
through the three TMOS channels #0, #1, and #2, to the
communication unit 321 connected thereto via the HOMI
cable 303, through a TMOS clock channel. Here, in one
TMOS channel #i (i 0, 1, 2), 10-bit data is transmitted
in one clock of the pixel clocks, for example.
[0159]
In the active video interval, the communication
unit 321 receives the differential signal corresponding
to the pixel data transmitted unidirectionally from the
communication unit 314 through the channels. In the
horizontal blanking interval or the vertical blanking
interval, the communication unit 321 also receives the
differential signal corresponding to the audio data and
the control data transmitted unidirectionally from the
communication unit 314 through the channels.
[0160]
That is, the communication unit 321 includes a HOMI
receiver 332. Through the TMOS channels #0, #1, and #2,
the HOMI receiver 332 receives the differential signal
corresponding to the pixel data and the differential
signal corresponding to the audio data and the control
data, which are transmitted unidirectionally from the
56
t SP308971WOOO
communication unit 314 connected thereto via the HOMI
cable 303, in synchronization with the pixel clocks also
transmitted from the communication unit 314 through the
TMOS clock channel.
5 [0161]
The transmission channels in the HOMI system formed
with the communication unit 314 and the communication
unit 321 include transmission channels such as a DOC
(Display Data Channel) 333 and a CEC line 334, as well as
10 the three TMOS channels #0 through #2 serving as the
transmission channels for unidirectionally and serially
transmitting encoded data and audio data from the
communication unit 314 to the communication unit 321 in
synchronization with the pixel clocks, and the TMOS clock
15 channel serving as the transmission channel for
transmitting the pixel clocks.
[0162]
The DOC 333 is formed with two signal lines (not
shown) included in the HOMI cable 303, and is used by the
20 communication unit 314 to read E-EOIO from the
communication unit 321 connected thereto via the HOMI
cable 303.
[0163]
That is, the communication unit 321 includes not
25 only the HOMI receiver 332 but also an EOIO ROM (Read
Only Memory) 335 storing the E-EOIO, which is performance
information about the performance
(configuration/capability) thereof. Through the DOC 333,
the communication unit 314 reads the E-EOIO of the
30 communication unit 321 from the communication unit 321
connected thereto via the HOMI cable 303. Based on the
57
t
5
10
15
20
25
30
SP308971WOOO
E-EDID, the communication unit 314 recognizes the image
format (profile) with which the electronic device
including the communication unit 321 is compatible, such
as RGB, YCbCr 4:4:4, YCbCr 4:2:2, or YCbCr 4:2:0.
[0164]
The CEC line 334 is formed with one signal line
(not shown) included in the HDMI cable 303, and is used
for bidirectional communication of control data between
the communication unit 314 and the communication unit 321.
[0165]
The HDMI cable 303 also includes a line 336
connected to a pin called HPD (Hot Plug Detect). A
source device can use the line 336 to detect a connection
with a sink device. The HDMI cable 303 also includes a
line 337 that is used for supplying power from a source
device to a sink device. The HDMI cable 303 further
includes a reserve line 338 for extensions.
[0166]
[3D Information Description]
In the HDMI standard data transmission described
above, the 3D information generating unit 313 extends
Vendor Specific infoFrame Packet Contents of the HDMI
standard as shown in Fig. 11, for example, and writes 3D
information indicating a mode.
[0167]
Vendor Specific infoFrame Packet Contents shown in
Fig. 11 is information that is transmitted, together with
data, from a transmitting end to a receiving end, and
indicates what kind of data is now being transmitted. A
receiving-end device can easily recognize what kind of
data is now being transmitted, by referring to this
58
t SP308971WOOO
Vendor Specific infoFrame Packet Contents.
[0168]
As shown in Fig. 11, the shaded portion of Vendor
Specific infoFrame Packet Contents is a field for a
5 description of the information about 3D image data
(3D Structure). The 3D information generating unit 313
extends the value of this region (3D_Structure), and
writes 3D information.
[0169]
10 For example, a value such as "0000", "0110", "0111",
"1000", or "1001" can be normally set in the field of
3D_Structure, as shown in the table in Fig. 12A. The 3D
information generating unit 313 uses the space (reserve)
of 3D_Structure, to set a value "1010" as the value
15 indicating the 3DV format, for example, as shown in Fig.
12A.
[0170]
The mode determining unit 322 on the receiving end
only has to know in advance that the value "1010"
20 indicates the 3DV format. For example, in a case where
the mode determining unit 322 has referred to
3D Structure of this Vendor Specific infoFrame Packet
Contents and has determined the value of to be "1010",
the mode determining unit 322 can recognize that the
25 image data to be transmitted is in the 3DV format.
[0171]
Also, as shown in Fig. 12B, designating respective
modes of the 3DV format may be allowed, for example. In
the example case shown in Fig. 12B, the value "1010" is
30 assigned to Mono Compatible Mode, the value "1011" is
assigned to Frame Packing (Side by Side) Compatible Mode,
59
t SP308971WOOO
the value "1100" is assigned to Frame Packing (Top and
Bottom) Compatible Mode, and the value "1101" is assigned
to Stereo Scopic Compatible Mode.
[0172]
5 The 3D information generating unit 313 sets the
value corresponding to the mode determined by the mode
determining unit 312 in 3D_Structure of Vendor Specific
infoFrame Packet Contents. Knowing those values in
advance, the mode determining unit 322 can easily
10 determine each mode by referring to 3D Structure at the
time of data reception.
[0173]
[Synchronization Method Extension]
In HDMI data transmission, image data of one frame
15 is normally transmitted in an active video interval that
is equivalent to one cycle of a horizontal
synchronization signal 351 and one cycle of a vertical
synchronization signal 352, as shown in Fig. 13.
[0174]
20 When transmitting 3D image data, the communication
unit 314 extends such a synchronization method in
accordance with each mode, for example.
[0175]
In the case of Mono Compatible Mode, a compatible
25 image, auxiliary images, and parallax images of those
images are transmitted, and therefore, data of four
frames in total needs to be transmitted as 3D image data
of one frame, for example, as shown in Fig. 14.
[0176]
30 Accordingly, as shown in Fig. 14, the communication
unit 314 may transmit the data of those four frames in an
60
SP308971WOOO
active video interval equivalent to one cycle of the
horizontal synchronization signal 351 and one cycle of
the vertical synchronization signal 352, for example.
[0177]
5 Alternatively, as shown in Fig. 15, the
communication unit 314 may use an active video interval
equivalent to two cycles of the vertical synchronization
signal in the transmission of the data of those four
frames, for example. It is of course possible to use an
10 active video interval equivalent to three or more cycles
of the vertical synchronization signal. However, to
maintain correct synchronization, both the reproducing
device 301 and the display device 302 need to know in
advance how many cycles of the vertical synchronization
15 signal is required in the active video interval to be
used in transmission of 3D image data of one frame.
[0178]
As 3D image data of one frame is transmitted in an
active video interval equivalent to cycles of the
20 vertical synchronization signal, more advanced
transmission control can be easily performed. In a case
where only part of information is transmitted, data
transmission is suspended in the other cycles, for
example.
25 [0179]
In the case of Frame Packing (Side by Side)
Compatible Mode, compatible images, auxiliary images, and
parallax images of those images are transmitted, and
therefore, data of four frames in total needs to be
30 transmitted as 3D image data of one frame, for example,
as shown in Fig. 16.
61
SP308971WOOO
[0180]
Accordingly, as shown in Fig. 16, the communication
unit 314 may transmit the data of those four frames in an
active video interval equivalent to one cycle of the
5 horizontal synchronization signal and one cycle of the
vertical synchronization signal, for example. In the
case of this Frame Packing (Side by Side) Compatible Mode,
3D image data of one frame may of course be transmitted
by using an active video interval equivalent to cycles of
10 the vertical synchronization signal, as in the above
described case of Mono Compatible Mode.
[0181]
In the case of Frame Packing (Top and Bottom)
Compatible Mode, compatible images, auxiliary images, and
15 parallax images of those images are transmitted, and
therefore, data of four frames in total needs to be
transmitted as 3D image data of one frame, for example,
as shown in Fig. 17.
[0182]
20 Accordingly, as shown in Fig. 17, the communication
unit 314 may transmit the data of those four frames in an
active video interval equivalent to one cycle of the
horizontal synchronization signal and one cycle of the
vertical synchronization signal, for example. In the
25 case of this Frame Packing (Top and Bottom) Compatible
Mode, 3D image data of one frame may of course be
transmitted by using an active video interval equivalent
to cycles of the vertical synchronization signal, as in
the above described case of Mono Compatible Mode.
30 [0183]
In the case of Stereo Scopic Compatible Mode,
62
SP308971WOOO
compatible images, auxiliary images, and parallax images
of those images are transmitted, and therefore, data of
four frames in total needs to be transmitted as 3D image
data of one frame, for example, as shown in Fig. 18.
5 [0184]
Accordingly, as shown in Fig. 18, the communication
unit 314 may transmit the data of those four frames in an
active video interval equivalent to one cycle of the
horizontal synchronization signal and one cycle of the
10 vertical synchronization signal, for example.
[0185]
In accordance with the current HDMI standard (Ver.
1.4), image data with a resolution of 4 K x 2 K or higher
cannot be transmitted in an active video interval
15 equivalent to one cycle of a vertical synchronization
signal. However, this restriction may be extended so
that 5 K x 2 K image data can be transmitted in an active
video interval equivalent to one cycle of a vertical
synchronization signal.
20 [0186]
Fig. 19 shows an example of Stereo Scopic
Compatible Mode transmission in that case. In this
manner, parallax information with a higher resolution
than in the example case shown in Fig. 18 can be
25 transmitted. Accordingly, in an image for display, a
sense of depth can be created with higher precision.
[0187]
In the case of this Stereo Scopic Compatible Mode,
3D image data of one frame may of course be transmitted
30 by using an active video interval equivalent to cycles of
the vertical synchronization signal, as in the above
63
SP308971WOOO
described case of Mono Compatible Mode.
[0188]
Further, as shown in Fig. 20, the cycles
(frequency) of the vertical synchronization signal may
5 not be constant but may be irregular, for example. In
the example case shown in Fig. 20, the first cycle of the
vertical synchronization signal has a length equivalent
to 2 K, the second cycle has a length equivalent to 1 K,
which is half the length of the first cycle, and the
10 third cycle has a length equivalent to 2 K, which is the
same as the first cycle.
[0189]
As described above, the frequency of the vertical
synchronization signal is appropriately changed to a
15 frequency suitable for the structure of 3D image data to
be transmitted. In this manner, more advanced transfer
control can be easily performed. In a case where only
compatibility information is transmitted, for example,
data transmission in the second cycle and the third cycle
20 are suspended.
[0190]
The cycle pattern in a case where the vertical
synchronization signal has irregular cycles is of course
not limited, and any pattern other than the example shown
25 in Fig. 20 may be used. The method that utilizes a
vertical synchronization signal having irregular cycles
can also be applied to Stereo Scopic Compatible Mode.
[0191]
[Transmission Processing Flow]
30 Various processes to be performed in the above
described 3D image data transmission are now described.
64
SP308971WOOO
[0192]
Referring to the flowchart shown in Fig. 21, an
example flow of a transmission process to be performed by
the reproducing device 301 shown in Fig. 8 is described.
5 [0193]
When the transmission process is started, the mode
determining unit 312 of the reproducing device 301 in
step S101 determines the mode of the stream (3D image
data) to be transmitted. In step S102, based on the
10 determination result of step S101, the 3D information
generating unit 313 generates the 3D information
indicating the mode of the stream.
[0194]
In step S103, the communication unit 314 transmits
15 the 3D information generated in step S103 to the display
device 302. In step S104, the communication unit 314
transmits the stream to the display device 302.
[0195]
By performing the above process, the reproducing
20 device 301 can supply the 3D information containing the
mode of the 3D image data to be transmitted, to the
display device 302 on the receiving end. Accordingly,
the reproducing device 301 can correctly transmit image
data in a format such as the above described 3DV format
25 in which a multi-view image can be generated.
[0196]
[Reception Processing Flow]
Referring now to the flowchart shown in Fig. 22, an
example flow of a reception process to be performed by
30 the display device 302 shown in Fig. 9 is described.
[0197]
65
,
5
10
15
20
25
30
8P308971WOOO
When the reception process is started, the
communication unit 321 of the display device 302 in step
8121 receives 3D information transmitted from the
reproducing device 301. In step 8122, the mode
determining unit 322 refers to the 3D information, and
determines the mode of the stream (3D image data) to be
transmitted.
[0198]
In step 8123, based on the mode determination
result of step 8122, the synchronization setting unit 323
sets the synchronization method described above with
reference to Figs. 14 through 20, for example. In step
8124, according to the synchronization method set in step
8123, the synchronization control unit 324 controls the
synchronization of the image generating unit 325 and the
display unit 326.
[0199]
In step 8125, the communication unit 321 receives
the stream (3D image data) transmitted from the
reproducing device 301. In step 8126, the image
generating unit 325 generates a multi-view image for
display, using the 3D image data received in step 8125.
[0200]
In step 8127, the display unit 326 displays the
multi-view image for display generated through the
procedure in step 8126.
[0201]
By performing the above process, the display device
302 can receive transmitted 3D image data, and 3D
information containing the mode of the 3D image data.
Further, the display device 302 easily determines the
66
,
5
10
SP308971WOOO
mode of the 3D image data from the 3D information, and
can set an appropriate synchronization method in
accordance with the mode. Accordingly, the display
device 302 can correctly transmit image data in a format
such as the above described 3DV format in which a multiview
image can be generated.
[0202]
Although data transmission in accordance with the
HDMI standard has been described above, the data
transmission can be performed in accordance with
transmission standards other than the HDMI standard.
[0203]
Also, in the above described example, the
reproducing device 301 is used as the device on the
15 transmitting end in data transmission of 3D image data,
and the display device 302 is used as the device on the
receiving end. However, any kinds of devices can be used
as the device on the transmitting end and the device on
the receiving end, as long as they can transmit and
20 receive 3D image data.
[0204]
<2. Second Embodiment>
[E-EDID Extension]
In the above described embodiment, 3D information
25 about the mode of 3D image data is transmitted together
with the 3D image data, so that 3DV data transmission is
realized. The present invention is not limited to that,
and the information about modes with which the display
device 302 on the receiving end is compatible may be
30 provided to the reproducing device 301 on the
transmitting end prior to the start of 3D image data
67
SP308971WOOO
transmission, for example.
[0205]
With this, the reproducing device 301 obtains
information about the capability of the display device
5 302, and, based on the information, can set a mode of 3D
image data. That is, the reproducing device 301 can
transmit 3D image data in a more appropriate manner.
[0206]
Through the HDMI cable 303, the communication unit
10 314 (the HDMI transmitting unit) of the reproducing
device 301 reads the E-EDID (Enhanced Extended Display
Identification Data) of the communication unit 321 from
the communication unit 321 (the HDMI receiving unit) of
the display device 302 via the DDC (Display Data Channel) .
15 This E-EDID contains information about the capability of
the display device 302, such as resolutions, decoding
delay time, bit depths, and frame rates that can be
handled in the display device 302.
[0207]
20 Fig. 23 shows an example data structure of the EEDID.
This E-EDID is formed with a basic block and an
extended block. At the top of the basic block, data that
is represented by "E-EDIDl.3 Basic Structure" and is
specified in the E-EDID 1.3 standard is provided,
25 followed by timing information that is represented by
"Preferred timing" and is used for maintaining
compatibility with the conventional EDID, and timing
information that is represented by "2nd timing", is used
for maintaining compatibility with the conventional EDID,
30 and differs from "Preferred timing".
[0208]
68
SP308971WOOO
In the basic block, "2nd timing" is followed
sequentially by information that is represented by
"Monitor NAME" and indicates the name of the display
device, and information that is represented by "Monitor
5 Range Limits" and indicates the number of pixels that can
be displayed in a case where the aspect ratio is 4:3 or
16:9.
[0209]
At the top of the extended block, information that
10 is represented by "Speaker Allocation" and concerns right
and left speakers is provided, sequentially followed by
data that is represented by "VIDEO SHORT" and contains
displayable image sizes (resolutions), frame rates,
information indicating whether it is of an interlace type
15 or a progressive type, information about aspect ratios,
and the like, data that is represented by "AUDIO SHORT"
and contains information about reproducible audio codec
formats, sampling frequencies, cutoff bands, the number
of codec bits, and the like, and information that is
20 represented by "Speaker Allocation" and concerns the
right and left speakers.
[0210]
In the extended block, "Speaker Allocation" is
followed by data that is represented by "Vendor Specific"
25 and is uniquely defined for each maker, timing
information that is represented by "3rd timing" and is
used for maintaining compatibility with the conventional
EDID, and timing information that is represented by "4th
timing" and is used for maintaining the compatibility
30 with the conventional EDID.
[0211]
69
SP308971WOOO
Fig. 24 shows an example structure of the data in
the shaded Vendor Specific region (Vendor Specific Data
Block) shown in Fig. 23. In this Vendor Specific region,
Oth through Nth blocks each being a 1-byte block are
5 provided.
[0212]
In the blocks after the 13th block in this Vendor
Specific region, the reserve region is extended, and
information about 3D formats with which the display
10 device 302 is compatible is written. Using the reserve
region, the display device 302 extends 3D Structure ALL_X,
3D_Structure_X, and 3D Detaile X of this Vendor Specific
Data Block, and defines the unit and cycles of a 3DV
stream and a vertical synchronization signal.
15 [0213]
For example, the value of 3D_Structure_ALL_X may be
extended as in the table shown in Fig. 25, and respective
modes may be assigned to the 9th through 13th blocks.
[0214]
20 Also, the value of 3D_Structure_X may be extended
as in the table shown in Fig. 12, and further, the value
of 3D Detail_X may be extended as in the table shown in
Fig. 26. Transmission of 3D image data of one frame
using an active video interval equivalent to cycles of a
25 vertical synchronization signal, the vertical
synchronization signal having irregular cycles, and the
like may be defined therein.
[0215]
[Structure of the Reproducing Device]
30 Fig. 27 is a block diagram showing a typical
example structure of the reproducing device 301 in the
70
SP308971WOOO
above case. As shown in Fig. 27, the reproducing device
301 in this case includes a receiving-end information
acquiring unit 411 and a synchronization setting unit 412,
as well as the components shown in Fig. 8.
5 [0216]
Before transmission of 3D image data is started,
the receiving-end information acquiring unit 411 acquires,
via the communication unit 314, receiving-end information
that is supplied from the display device 302 and
10 indicates the capability of the receiving-end device.
For example, the receiving-end information acquiring unit
411 receives the E-EDID data shown in Fig. 23 as the
receiving-end information. As described above, Vendor
Specific Data Block of this E-EDID data is extended as
15 shown in Fig. 24, and the values of 3D_Structure_ALL_X,
3D_Structure_X, 3D_Detaile_X, and the like are extended
as in the tables shown in Figs. 25 and 26, so that modes
that can be handled by the display device 302 are
indicated.
20 [0217]
After acquiring the receiving-end information, the
receiving-end information acquiring unit 411 supplies the
receiving-end information to the synchronization setting
unit 412. In accordance with the receiving-end
25 information acquired by the receiving-end information
acquiring unit 411 and the mode determined by the mode
determining unit 312, the synchronization setting unit
412 sets a synchronization method that can be implemented
by the display device 302, and notifies the 3D
30 information generating unit 313 of the synchronization
method.
71
,
5
10
SP308971WOOO
[0218]
The 3D information generating unit 313 generates 3D
information indicating the synchronization method set by
the synchronization setting unit 412, and inserts the 3D
information into the 3D image data.
[0219]
[Structure of the Display Device]
Fig. 28 is a block diagram showing a typical
example structure of the display device 302 in the above
case. As shown in Fig. 28, the display device 302 in
this case includes a receiving-end information providing
unit 421, as well as the components shown in Fig. 9.
[0220]
The receiving-end information providing unit 421
15 stores the E-EDID shown in Fig. 23, for example. Before
transmission of 3D image data is started, the receivingend
information providing unit 421 transmits the E-EDID
as negotiation information to the reproducing device 301
on the transmitting end via the communication unit 321.
20 [0221]
The receiving-end information providing unit 421
may be formed inside the communication unit 321, like the
EDID ROM 335 shown in Fig. 10.
[0222]
25 [Transmission Processing Flow]
Referring now to the flowchart shown in Fig. 29, an
example flow of a transmission process to be performed by
the reproducing device 301 shown in Fig. 27 is described.
[0223]
30 When the transmission process is started, the
receiving-end information acquiring unit 411 in step S201
72
,
5
10
15
20
25
30
8P308971WOOO
acquires the receiving-end information. In step 8202,
the mode determining unit 312 determines the mode of the
stream (3D image data) to be transmitted.
[0224]
In step 8203, based on the receiving-end
information acquired in step 8201 and the mode of the
stream determined in step 8202, the synchronization
setting unit 412 sets a synchronization method that can
be implemented by the display device 302 and is
compatible with the mode of the stream.
[0225]
In step 8204, the 3D information generating unit
313 generates the 3D information indicating the
synchronization method set in step 8203. In step 8205,
the communication unit 314 transmits the 3D information
generated in step 8204 to the display device 302. In
step 8206, the communication unit 314 also transmits the
stream to the display device 302.
[0226]
By performing the above process, the reproducing
device 301 can set a synchronization method in accordance
with the capability of the display device 302, and can
supply the 3D information about the method to the display
device 302. Accordingly, the reproducing device 301 can
correctly transmit image data in a format such as the
above described 3DV format in which a multi-view image
can be generated.
[0227]
[Reception Processing Flow]
Referring now to the flowchart shown in Fig. 30, an
example flow of a reception process to be performed by
73
• 8P308971WOOO
the display device 302 shown in Fig. 28 is described.
[0228]
When the reception process is started, the
receiving-end information providing unit 421 of the
5 display device 302 in step 8221 provides the receivingend
information to the reproducing device 301 via the
communication unit 321.
[0229]
In step 8222, the communication unit 321 receives
10 3D information from the reproducing device 301. In step8223,
the mode determining unit 322 refers to the 3D
information, and determines the mode of the stream (3D
image data) to be transmitted.
[0230]
15 In step 8224, based on the mode determination
result of step 8223, the synchronization setting unit 323
sets a synchronization method. In step 8225, according
to the synchronization method set in step 8224, the
synchronization control unit 324 controls the
20 synchronization of the image generating unit 325 and the
display unit 326.
[0231]
In step 8226, the communication unit 321 receives
the stream (3D image data) transmitted from the
25 reproducing device 301. In step 8227, the image
generating unit 325 generates a multi-view image for
display, using the 3D image data received in step 8226.
[0232]
In step 8228, the display unit 326 displays the
30 multi-view image for display generated through the
procedure in step 8227.
74
•
5
10
15
SP308971WOOO
[0233]
By performing the above process, the display device
302 can provide the receiving-end information indicating
the processing capability thereof to the reproducing
device 301 on the transmitting end, and can cause the
reproducing device 301 to set a synchronization method in
accordance with the capability of the display device 302.
The display device 302 can appropriately set the
synchronization method. Accordingly, the display device
302 can correctly transmit image data in a format such as
the above described 3DV format in which a multi-view
image can be generated.
[0234]
Although data transmission in accordance with the
HDMI standard has been described above, the data
transmission can be performed in accordance with
transmission standards other than the HDMI standard.
[0235]
Also, in the above described example, the
20 reproducing device 301 is used as the device on the
transmitting end in data transmission of 3D image data,
and the display device 302 is used as the device on the
receiving end. However, any kinds of devices can be used
as the device on the transmitting end and the device on
25 the receiving end, as long as they can transmit and
receive 3D image data.
[0236]
<3. Third Embodiment>
[Example Structure of an Encoding Device]
30 Fig. 31 is a block diagram showing an example
structure of an encoding device that encodes 3D images
75
• SP308971WOOO
transmitted in a third embodiment of a transmission
system to which the present technique is applied.
[0237]
In the structure illustrated in Fig. 31, the same
5 components as those in Fig. 3 are denoted by the same
reference numerals as those in Fig. 3. The explanations
that have already been made will not be repeated.
[0238]
The structure of the encoding device 440 shown in
10 Fig. 31 differs from the structure shown in Fig. 3 mainly
in that the parallax image generating unit 143, the
encoder 145, and the multiplexing unit 59 are replaced
with a parallax image generating unit 441, an encoder 442,
and a multiplexing unit 443. The encoding device 440
15 performs encoding by multiplexing parallax images of
compatible images in a temporal direction, instead of a
spatial direction.
[0239]
Specifically, using images A2 through D2 supplied
20 from imaging units 141A through 141D, the parallax image
generating unit 441 of the encoding device 440 detects
the parallax of each pixel of the images A2 through D2.
Based on the detection results, the parallax image
generating unit 441 generates a parallax image A2' of the
25 image A2 as a compatible image, a parallax image B2' of
the image B2 as a compatible image, a parallax image C2'
of the image C2 as an auxiliary image, and a parallax
image D2' of the image D2 as an auxiliary image.
[0240]
30 Like the parallax image generating unit 143 shown
in Fig. 3, the parallax image generating unit 441 also
76
5
10
• SP308971WOOO
multiplexes the parallax image C2' and the parallax image
D2' in a spatial direction by a predetermined
multiplexing technique. Further, the parallax image
generating unit 441 multiplexes the parallax image A2',
the parallax image B2', and the multiplexed image of the
parallax images of the auxiliary images in a temporal
direction. The parallax image generating unit 441
supplies the resultant multiplexed image as a temporal
multiplexed image to the encoder 442. In the temporal
multiplexed image, the parallax image A2', the parallax
image B2', and the multiplexed image of the parallax
images of the auxiliary images of one frame exist in one
frame time.
[0241]
15 Also, the parallax image generating unit 441
supplies information indicating the multiplexing
technique for the parallax images of the auxiliary images
and a technique for multiplexing images in a temporal
direction (hereinafter referred to as the frame
20 sequential technique) as the multiplexing technique for
the parallax images of the compatible images and the
auxiliary images, to a parallax image information
generating unit 57.
[0242]
25 The encoder 442 is formed with a compatible encoder
151 and an auxiliary encoder 451. The auxiliary encoder
451 of the encoder 442 encodes a multiplexed image of the
auxiliary images from an image converting unit 142, and
the temporal multiplexed image from the parallax image
30 generating unit 441, by the 3DV technique. The auxiliary
encoder 451 supplies the resultant encoded stream as an
77
• SP308971WOOO
auxiliary stream to the multiplexing unit 443.
[0243]
The multiplexing unit 443 generates TSs, using a
compatible stream from the compatible encoder 151, the
5 auxiliary stream from the auxiliary encoder 451, image
information from an image information generating unit 54,
compatibility information from a compatibility
information generating unit 55, inter-viewpoint distance
information from an inter-viewpoint distance information
10 generating unit 56, parallax image information from the
parallax image information generating unit 57, and the
like. The multiplexing unit 443 multiplexes the
generated TSs, and transmits the resultant multiplexed
stream.
15 [0244]
In the following, the image information, the
compatibility information, the inter-viewpoint distance
information, and the parallax image information will be
collectively referred to as auxiliary information.
20 [0245]
[Example Description of Auxiliary Information]
Fig. 32 is a diagram showing an example description
of the compatibility information and the parallax image
information in a case where the auxiliary information is
25 written into PMT descriptors.
[0246]
As shown in Fig. 32, in a case where the auxiliary
information is written into PMT descriptors, a descriptor
to which the compatibility information is set
30 (3DV_view_structure_descriptor), a descriptor to which
the parallax image information is set
78
• SP308971WOOO
(depth_map_structure_descriptor), and the like are
provided as the PMT descriptors.
[0247]
In the descriptor (depth_map_structure_descriptor),
5 a descriptor tag (descriptor tag) and a descriptor length
(descriptor_length) are followed by the number of
parallax images (num_of_depth_map), a flag indicating
whether the parallax images are multiplexed
(is frame_packing), the multiplexing technique for the
10 parallax images (frame_packing_mode), information
designating multiplexed parallax images
(comb_frame_packing_views), and the like, which are
written as the parallax image information, as shown in
Fig. 33.
15 [0248]
The multiplexing technique may be Side By Side
(SBS) technique, Top and Bottom (TaB) technique, the
frame sequential technique, or the like.
[0249]
20 Although the auxiliary information is contained in
the TSs in the encoding device 440, the auxiliary
information may be contained in the auxiliary stream.
[0250]
Figs. 34 and 35 are diagrams showing example
25 descriptions of the compatibility information and the
parallax image information in the auxiliary stream in a
case where the auxiliary information is contained in the
auxiliary stream.
[0251]
30 As shown in Fig. 34, the compatibility information
(3DV view structure) and the parallax image information
79
• SP308971WOOO
(depth_map_structure) are set in SEI (Supplemental
Enhancement Information) in the auxiliary stream, for
example.
[0252]
5 As the parallax image information
(depth_map_structure), the number (num of_depth_map) of
parallax images (depth map), the flag indicating whether
the parallax images are multiplexed (is frame_packing),
the multiplexing technique for the parallax images
10 (frame_packing_mode), and the information designating
multiplexed parallax images (comb_frame_packing_views),
and the like are written as shown in Fig. 35.
[0253]
Although not shown in the drawings, the image
15 information is the same as the parallax image information,
except for being the information about the compatible
images and the auxiliary images, instead of the parallax
images.
[0254]
20 [Description of an Operation of the Encoding Device]
Figs. 36 and 37 are flowcharts for explaining an
encoding operation by the encoding device 440 shown in
Fig. 31. This encoding operation is started when the
images A2 through 02 are output from the imaging units
25 141A through 1410, for example.
[0255]
In step S451 in Fig. 36, an inter-viewpoint
distance information generating unit 144 detects a
distance ~d2AB' a distance ~d2Bc, and a distance ~d2AD as
30 inter-viewpoint distances, using the images A2 through 02
supplied from the imaging units 141A through 1410.
80
• SP308971WOOO
[0256]
In step S452, the inter-viewpoint distance
information generating unit 144 generates information
indicating the inter-viewpoint distances detected in step
5 S451 and the like as the inter-viewpoint distance
information, and supplies the inter-viewpoint distance
information to the multiplexing unit 443.
[0257]
In step S453, the image converting unit 142
10 determines the image A2 and the image B2 to be compatible
images, as the image A2 and the image B2 are supplied
from the imaging unit 141A and the imaging unit 141B,
which are located in inner positions among the imaging
units 141A through 1410 in the horizontal direction. The
15 image converting unit 142 also determines the
multiplexing techniques for the compatible images and the
auxiliary images. The image converting unit 142 supplies
information designating the image A2 and the image B2 as
the compatible images to the compatibility information
20 generating unit 55, and supplies the multiplexing
techniques for the compatible images and the auxiliary
images to the image information generating unit 54.
[0258]
In step S454, based on the information supplied
25 from the image converting unit 142, the compatibility
information generating unit 55 generates the
compatibility information containing the information
designating the image A2 and the image B2 as the
compatible images, the frame packing mode as a compatible
30 mode, and the like, and inputs the compatibility
information to the multiplexing unit 443.
81
8P308971WOOO
[0259]
In step 8455, based on the multiplexing technique
for the compatible images determined in step 8453, the
image converting unit 142 multiplexes the image A2 and
5 the image B2 as the compatible images, and supplies the
multiplexed image to the encoder 442.
[0260]
In step 8456, based on the information supplied
from the image converting unit 142, the image information
10 generating unit 54 generates information indicating the
multiplexing techniques for the compatible images and the
auxiliary images and the like as image information, and
supplies the image information to the multiplexing unit
443.
15 [0261]
In step 8457, the image converting unit 142
determines the image C2 and the image 02 to be the
auxiliary images other than the image A2 and the image B2,
and, based on the multiplexing technique for the
20 auxiliary images determined in step 8453, multiplexes the
auxiliary images, to obtain a multiplexed image of the
auxiliary images.
[0262]
In step 8458, the image converting unit 142 inputs
25 the multiplexed image of the compatible images and the
multiplexed image of the auxiliary images to the encoder
442.
[0263]
In step 8459 in Fig. 37, using the images A2
30 through 02 supplied from the imaging units 141A through
1410, the parallax image generating unit 441 detects the
82
8P308971WOOO
parallax of each pixel of the images A2 through 02, and
generates the parallax images A2' through 02'.
[0264]
In step 8460, the parallax image generating unit
5 441 determines the multiplexing technique for the
parallax images of the auxiliary images, and the
multiplexing technique for the multiplexed images of the
parallax images of the compatible images and the parallax
images of the auxiliary images, and supplies information
10 indicating the multiplexing techniques to the parallax
image information generating unit 57.
[0265]
In step 8461, based on the information supplied
from the parallax image generating unit 441, the parallax
15 image information generating unit 57 generates
information indicating the multiplexing technique for the
parallax images of the auxiliary images and the
multiplexing technique for the multiplexed images of the
parallax images of the compatible images and the parallax
20 images of the auxiliary images as the parallax image
information, and inputs the parallax image information to
the multiplexing unit 443.
[0266]
25 In step 8462, based on the multiplexing technique
determined in step 8460, the parallax image generating
unit 441 multiplexes the parallax image C2' and the
parallax image 02' of the auxiliary images in a spatial
direction, and multiplexes the parallax image A2' and the
30 parallax image B2' of the compatible images and the
multiplexed image of the parallax images of the auxiliary
83
8P308971WOOO
images in a temporal direction.
[0267 ]
In step 8463, the parallax image generating unit
441 inputs the temporal multiplexed image obtained as a
5 result of the multiplexing in step 8462, to the encoder
442.
[0268]
In step 8464, the compatible encoder 151 of the
encoder 442 encodes the multiplexed image of the
10 compatible images supplied from the image converting unit
142 by the conventional Ave technique, and adds various
kinds of information to the encoded image. The resultant
encoded stream is supplied as the compatible stream to
the multiplexing unit 443.
15 [0269]
In step 8465, the auxiliary encoder 451 encodes the
multiplexed image of the auxiliary images from the image
converting unit 142, and the temporal multiplexed image
from the parallax image generating unit 441, by the 3DV
20 technique. The auxiliary encoder 451 supplies the
encoded stream as the auxiliary stream obtained as a
result of the encoding to the multiplexing unit 443.
[0270]
In step 8466, the multiplexing unit 443 generates
25 T8s from the compatible stream supplied from the
compatible encoder 151, the auxiliary stream supplied
from the auxiliary encoder 451, and the auxiliary
information, and multiplexes the T8s. The resultant
multiplexed stream is then transmitted. This multiplexed
30 stream is recorded on a BD or the like, or is transmitted
as a stream for broadcasting, for example. The operation
84
SP308971WOOO
then comes to an end.
[0271]
[Example Structure of a Decoding Device]
Fig. 38 is a diagram showing an example structure
5 of a decoding device that decodes the multiplexed stream
transmitted from the encoding device 440 shown in Fig. 31.
[0272]
In the structure illustrated in Fig. 38, the same
components as those in Fig. 4 are denoted by the same
10 reference numerals as those in Fig. 4. The. explanations
that have already been made will not be repeated.
[0273]
The structure of the decoding device 460 shown in
Fig. 38 differs from the structure shown in Fig. 4 mainly
15 in that the separating unit 121, the decoder 122, and the
image generating unit 171 are replaced with a decoder 462
and an image generating unit 463. The decoding device
460 decodes the multiplexed stream transmitted from the
encoding device 440, generates a two-view image or a
20 multi-view image, and causes a display device (not shown)
to display the generated image.
[0274]
Specifically, the separating unit 461 of the
decoding device 460 receives the multiplexed stream
25 transmitted from the encoding device 440, and separates
the TSs from each other. The separating unit 461
extracts the compatible stream contained in the TSs and
the auxiliary stream contained in the TSs, and supplies
the extracted streams to the decoder 462. The separating
30 unit 461 also extracts the auxiliary information
contained in the TSs, supplies the image information in
85
SP308971WOOO
the auxiliary information to an image information
acquiring unit 123, and supplies the inter-viewpoint
distance information to an inter-viewpoint distance
information acquiring unit 124. Further, the separating
5 unit 461 supplies the parallax image information in the
auxiliary information to a parallax image information
acquiring unit 125, and supplies the compatibility
information to a compatibility information acquiring unit
126.
10 [0275]
The decoder 462 is formed with a compatible decoder
131 and an auxiliary decoder 471. The auxiliary decoder
471 of the decoder 462 decodes the multiplexed image of
the auxiliary images and the temporal multiplexed image
15 contained in the auxiliary stream supplied from the
separating unit 461, by a technique corresponding to the
auxiliary encoder 451 shown in Fig. 31. The auxiliary
decoder 471 supplies the multiplexed image of the
auxiliary images and the temporal multiplexed image
20 obtained as a result of the decoding, to the image
generating unit 463.
[0276]
In accordance with a display instruction from a
viewer, the image generating unit 463 outputs images, and
25 causes the display device (not shown) to display the
images. Specifically, in accordance with a multi-view 3D
image display instruction from a viewer, the image
generating unit 463 generates images that are of three or
more viewpoints corresponding to the display device (not
30 shown) and have half the resolution of the compatible
images and the auxiliary images, using the multiplexed
86
SP308971WOOO
image of the compatible images, the multiplexed image of
the auxiliary images, and the temporal multiplexed image,
based on the image information from the image information
acquiring unit 123, the inter-viewpoint distance
5 information from the inter-viewpoint distance information
acquiring unit 124, the parallax image information from
the parallax image information acquiring unit 125, the
compatibility information from the compatibility
information acquiring unit 126, and the like.
10 [0277]
More specifically, based on the information that is
contained in the parallax image information from the
parallax image information acquiring unit 125 and
indicates the frame sequential technique as the
15 multiplexing technique for multiplexed images of the
parallax images of the compatible images and the parallax
images of the auxiliary images, the image generating unit
463 separates the parallax image A2' and the parallax
image B2', which are the parallax images of the
20 compatible images, and the multiplexed image of the
parallax images of the auxiliary images, from the
temporal multiplexed image. Based on the information
that is contained in the parallax image information and
indicates the multiplexing technique for the parallax
25 images of the auxiliary images, the image generating unit
463 separates the parallax image C2' and the parallax
image D2' from the multiplexed image of the parallax
images of the auxiliary images.
[0278]
30 Further, based on the information that is contained
in the image information from the image information
87
•
5
10
15
20
25
30
SP308971WOOO
acquiring unit 123 and indicates the multiplexing
technique for the auxiliary images, the image generating
unit 463 separates each of the auxiliary images from the
multiplexed image of the auxiliary images, like the image
generating unit 171 shown in Fig. 4. Also, based on the
information that is contained in the image information
and indicates the multiplexing technique for the
compatible images, the image generating unit 463
separates each of the compatible images from the
multiplexed image of the compatible images, like the
image generating unit 171.
[0279]
Also, based on the inter-viewpoint distance
information and the number of viewpoints corresponding to
the display device (not shown), the image generating unit
463 determines the location of each of the viewpoints of
the multi-view image to be generated, like the image
generating unit 171. Using the respective compatible
images, the respective auxiliary images, the parallax
images of the respective compatible images, and the
parallax images of the respective auxiliary images, the
image generating unit 463 generates images of the
respective viewpoints having the locations determined,
like the image generating unit 171. The image generating
unit 463 then converts the resolution of the generated
images of the respective viewpoints to the resolution
calculated by dividing the resolution of the compatible
images and the auxiliary images by the number of
viewpoints, and combines the generated images, like the
image generating unit 171. The resultant image is then
displayed on the display device (not shown).
88
• SP308971WOOO
[0280]
At this point, the multi-view image after the
combining is displayed in such a manner that the viewable
angle varies at each viewpoint, and the viewer can view a
5 3D image without glasses by seeing the images of any two
viewpoints with both eyes.
[0281]
In accordance with a two-view 3D image display
instruction from a viewer, the image generating unit 463
10 separates the multiplexed image of the compatible images
supplied from the decoder 462 into the image A2 and the
image B2 with half the resolution of the compatible
images, based on the image information from the image
information acquiring unit 123, like the image generating
15 unit 171. Like the image generating unit 171, the image
generating unit 463 alternately outputs the image A2 and
the image B2 with half the resolution of the separated
compatible images, and causes the display device (not
shown) to display those images. At this point, the
20 viewer puts on glasses that open one of the shutter for
the left eye and the shutter for the right eye when the
image A2 is displayed, and open the other one of the
shutters when the image B2 is displayed. Wearing the
glasses, the viewer sees the image A2 and the image B2
25 that are alternately displayed on the display device. In
this manner, the viewer can view a 3D image.
[0282]
[Description of a Decoding Operation]
Fig. 39 is a flowchart for explaining a decoding
30 operation by the decoding device 460 shown in Fig. 38.
This decoding operation is started when the multiplexed
89
•
5
10
15
20
25
30
8P308971WOOO
stream transmitted from the encoding device 440 shown in
Fig. 31 is input to the decoding device 460, for example.
[0283]
In step 8471 in Fig. 39, the separating unit 461 of
the decoding device 460 acquires the multiplexed stream
transmitted from the encoding device 50, and separates
the multiplexed stream into the respective T8s. The
separating unit 461 extracts the compatible stream and
the auxiliary stream contained in the T8s, and supplies
the extracted streams to the decoder 462. The separating
unit 461 also extracts the auxiliary information
contained in the T8s, supplies the image information in
the auxiliary information to the image information
acquiring unit 123, and supplies the inter-viewpoint
distance information to the inter-viewpoint distance
information acquiring unit 124. Further, the separating
unit 461 supplies the parallax image information in the
auxiliary information to the parallax image information
acquiring unit 125, and supplies the compatibility
information to the compatibility information acquiring
unit 126.
[0284]
In step 8472, the image generating unit 463
determines whether a two-view 3D image display
instruction has been issued from a viewer. If it is
determined in step 8472 that a two-view 3D image display
instruction has not been issued from a viewer, or that a
multi-view 3D image display instruction has been issued,
the operation moves on to step 8473.
[0285]
In step 8473, the compatible decoder 131 of the
90
•
5
8P308971WOOO
decoder 462 extracts the encoded compatible images from
the compatible stream supplied from the separating unit
461, and decodes the compatible images by a technique
corresponding to the AVe technique. The compatible
decoder 131 supplies the image A1 obtained as a result of
the decoding to the image generating unit 463.
[0286]
In step 8474, the auxiliary decoder 471 extracts
the encoded multiplexed image of the auxiliary images and
10 the temporal multiplexed image from the auxiliary stream,
and decodes the multiplexed image of the auxiliary images
and the temporal multiplexed image by a technique
corresponding to the auxiliary encoder 451 shown in Fig.
31. The auxiliary decoder 471 supplies the multiplexed
15 image of the auxiliary images and the temporal
multiplexed image obtained as a result of the decodipg,
to the image generating unit 127, and the operation moves
on to step 8475.
[0287]
20 In step 8475, the image information acquiring unit
123 acquires the image information supplied from the
separating unit 461, and inputs the image information to
the image generating unit 463. In step 8476, the interviewpoint
distance information acquiring unit 124
25 acquires the inter-viewpoint distance information
supplied from the separating unit 461, and supplies the
inter-viewpoint distance information to the image
generating unit 463.
[0288]
30 In step 8477, the parallax image information
acquiring unit 125 acquires the parallax image
91
•
5
10
15
20
25
30
8P308971WOOO
information supplied from the separating unit 461, and
inputs the parallax image information to the image
generating unit 463. In step 8478, the compatibility
information acquiring unit 126 acquires the compatibility
information supplied from the separating unit 461, and
supplies the compatibility information to the image
generating unit 463.
[0289]
In step 8479, based on the inter-viewpoint distance
information from the inter-viewpoint distance information
acquiring unit 124 and the number of viewpoints
corresponding to the display device (not shown), the
image generating unit 463 determines the location of each
of the viewpoints of the multi-view image to be generated.
[0290]
In step 8480, the image generating unit 463
generates images of the respective viewpoints that have
half the resolution of the compatible images and the
auxiliary images, using the multiplexed image of the
compatible images, the multiplexed image of the auxiliary
images, and the temporal multiplexed image, based on the
locations of the respective viewpoints determined in step
8479, the image information from the image information
acquiring unit 123, the parallax image information from
the parallax image information acquiring unit 125, the
compatibility information from the compatibility
information acquiring unit 126, and the like.
[0291]
In step 8481, the image generating unit 463
converts the resolution of the images of the respective
viewpoints generated in step 8480 to the resolution
92
•
5
10
15
20
25
30
8P308971WOOO
calculated by dividing the resolution of the compatible
images and the auxiliary images by the number of
viewpoints, and combines the images of the respective
viewpoints after the conversion, based on the locations
of the viewpoints.
[0292]
In step 8482, the image generating unit 463 outputs
the multi-view image obtained through the combining
process in step 8481 to the display device (not shown),
and causes the display device to display the multi-view
image so that the viewable angle varies at each viewpoint.
The operation then comes to an end.
[0293]
If it is determined in step 472 that a two-view 3D
image display instruction has been issued from a viewer,
on the other hand, the operation moves on to step 8483.
[0294]
In step 8483, the compatible decoder 131 of the
decoder 462 extracts the encoded multiplexed image of the
compatible images from the compatible stream supplied
from the separating unit 461, and decodes the multiplexed
image by a technique corresponding to the Ave technique.
The compatible decoder 131 supplies the multiplexed image
of the compatible images obtained as a result of the
decoding to the image generating unit 463.
[0295]
In step 8484, the image information acquiring unit
123 acquires the image information supplied from the
separating unit 461, and inputs the image information to
the image generating unit 463.
[0296]
93
• 8P308971WOOO
5
15
20
10
In step 8485, based on the information that is
contained in the image information supplied from the
image information acquiring unit 123 and indicates the
multiplexing technique for the compatible images, the
image generating unit 463 separates the multiplexed image
of the compatible images obtained as a result of the
decoding by the compatible decoder 131.
[0297]
In step 8486, the image generating unit 463
alternately outputs the image A2 and the image B2 with
half the resolution of the separated compatible images to
the display device, and causes the display device (not
shown) to display those images. The operation then comes
to an end.
[0298]
In a decoding device that can decode only the
compatible stream having compatibility with the decoding
device 460, the auxiliary stream is ignored, and the
procedures of steps 8483, 8485, and 8486 are carried out.
In this case, however, in the procedure in step 8485, the
multiplexed image of the compatible images is separated,
based on the multiplexing technique determined in advance.
[0299]
[Baseband Data Transmission]
25 In the above described decoding device 460 shown in
Fig. 38, if the left side (the separating unit 461 and
the decoder 462) of the dashed line L4 is designed as a
reproducing device, and the right side (the image
generating unit 463) of the dashed line L3 is designed as
30 a display device, the reproducing device and the display
device have the same structures as those of the above
94
• SP308971WOOO
described reproducing device 301 and the reproducing
device 302, respectively.
[0300]
<4. Fourth Embodiment>
5 [Example Structure of an Encoding Device]
Fig. 40 is a block diagram showing an example
structure of an encoding device that encodes 3D images
transmitted in a fourth embodiment of a transmission
system to which the present technique is applied.
10 [0301]
In the structure illustrated in Fig. 40, the same
components as those in Fig. 31 are denoted by the same
reference numerals as those in Fig. 31. The explanations
that have already been made will not be repeated.
15 [0302]
The structure of the encoding device 480 shown in
Fig. 40 differs from the structure shown in Fig. 31
mainly in that the parallax image generating unit 143,
the encoder 145, and the parallax image information
20 generating unit 57 are replaced with a parallax image
generating unit 481, an encoder 482, and a parallax image
information generating unit 483. The encoding device 480
encodes a common parallax image indicating parallax
values common to compatible images and a common parallax
25 image of auxiliary images.
[0303]
Specifically, using images A2 through 02 supplied
from imaging units 141A through 1410, the parallax image
generating unit 481 of the encoding device 480 detects
30 the parallax of each pixel between the image A2 and the
image B2, and the parallax of each pixel between the
95
~ SP308971WOOO
image C2 and the image 02. The parallax image generating
unit 481 generates a parallax image formed with parallax
values representing the parallaxes of the respective
pixels between the image A2 and the image B2, and
5 supplies the parallax image as a common parallax image
AB2' of the compatible images to the encoder 482. The
parallax image generating unit 481 also generates a
parallax image formed with parallax values representing
the parallaxes of the respective pixels between the image
10 C2 and the image 02, and supplies the parallax image as a
common parallax image C02' of the auxiliary images to the
encoder 482.
[0304]
The parallax image generating unit 481 supplies
15 information indicating that the parallax images of the
compatible images and the auxiliary images are common
parallax images, to the parallax image information
generating unit 483.
[0305]
20 The encoder 482 is formed with a compatible encoder
151 and an auxiliary encoder 491. The auxiliary encoder
491 encodes a multiplexed image of the auxiliary images
from an image converting unit 142, and the common
parallax image AB2' of the compatible images and the
25 common parallax image C02' of the auxiliary images from
the parallax image generating unit 481, by the 30V
technique. The auxiliary encoder 491 supplies the
resultant encoded stream as an auxiliary stream to a
multiplexing unit 443.
30 [0306]
Based on the information supplied from the parallax
96
•
5
10
15
20
25
30
8P308971WOOO
image generating unit 53, the parallax image information
generating unit 483 generates information indicating that
the parallax images of the compatible images and the
auxiliary images are common parallax images and the like
as parallax image information, and supplies the parallax
image information to the multiplexing unit 443.
[0307]
[Description of an Operation of the Encoding Device]
Figs. 41 and 42 are flowcharts for explaining an
encoding operation by the encoding device 480 shown in
Fig. 40. This encoding operation is started when the
images A2 through D2 are output from the imaging units
141A through 141D, for example.
[0308]
The procedures in steps 8491 through 8498 in Fig.
41 are the same as the procedures in steps 8451 through
8458 in Fig. 36, and therefore, explanation of them is
not repeated herein.
[0309]
In step 8499 in Fig. 42, using images A2 through D2
supplied from imaging units 141A through 141D, the
parallax image generating unit 481 detects the parallax
of each pixel between the image A2 and the image B2, and
the parallax of each pixel between the image C2 and the
image D2. The parallax image generating unit 481 then
generates the common parallax image AB2' formed with
parallax values representing the parallaxes of the
respective pixels between the image A2 and the image B2,
and the common parallax image CD2' formed with parallax
values representing the parallaxes of the respective
pixels between the image C2 and the image D2.
97
•
5
10
15
20
25
30
SP308971WOOO
[0310]
In step S500, based on the information supplied
from the parallax image generating unit 481, the parallax
image information generating unit 483 generates
information indicating that the parallax images of the
compatible images and the auxiliary images are common
parallax images and the like as parallax image
information, and inputs the parallax image information to
the multiplexing unit 443.
[0311]
In step S501, the parallax image generating unit
481 inputs the common parallax image C02' of the
auxiliary images and the common parallax image AB2' of
the compatible images generated in step S499, to the
encoder 482.
[0312]
In step S502, the compatible encoder 151 of the
encoder 482 encodes a multiplexed image of the compatible
images supplied from the image converting unit 142 by the
conventional AVC technique, and supplies the resultant
encoded stream as a compatible stream to the multiplexing
unit 443.
[0313]
In step S503, the auxiliary encoder 491 encodes the
multiplexed image of the auxiliary images from the image
converting unit 142, and the common parallax image of the
compatible images and the common parallax image of the
auxiliary images from the parallax image generating unit
481, by the 30V technique. The auxiliary encoder 491
supplies the encoded stream as the auxiliary stream
obtained as a result of the encoding to the multiplexing
98
SP308971WOOO
unit 443.
[0314]
In step S504, the multiplexing unit 443 generates
TSs from the compatible stream supplied from the
5 compatible encoder 151, the auxiliary stream supplied
from the auxiliary encoder 491, and auxiliary information,
and multiplexes the TSs. The resultant multiplexed
stream is then transmitted. This multiplexed stream is
recorded on a BD or the like, or is transmitted as a
10 stream for broadcasting, for example. The operation then
comes to an end.
[0315]
[Example Structure of a Decoding Device]
Fig. 43 is a diagram showing an example structure
15 of a decoding device that decodes the multiplexed stream
transmitted from the encoding device 480 shown in Fig. 40.
[0316]
In the structure illustrated in Fig. 43, the same
components as those in Fig. 38 are denoted by the same
20 reference numerals as those in Fig. 38. The explanations
that have already been made will not be repeated.
[0317]
The structure of the decoding device 500 shown in
Fig. 43 differs from the structure shown in Fig. 38
25 mainly in that the decoder 122 and the image generating
unit 171 are replaced with a decoder 501 and an image
generating unit 502. The decoding device 500 decodes the
multiplexed stream transmitted from the encoding device
480, generates a two-view image or a multi-view image,
30 and causes a display device (not shown) to display the
generated image.
99
•
5
10
15
20
25
30
SP308971WOOO
[0318]
Specifically, the decoder 501 of the decoding
device 500 is formed with a compatible decoder 131 and an
auxiliary decoder 511. The auxiliary decoder 511 of the
decoder 501 decodes the multiplexed image of the
auxiliary images, the common parallax image AB2' of the
compatible images, and the common parallax image C02' of
the auxiliary images, which are contained in the
auxiliary stream supplied from the separating unit 461,
by a technique corresponding to the auxiliary encoder 491
shown in Fig. 40. The auxiliary decoder 511 supplies the
multiplexed image of the auxiliary images, the common
parallax image AB2', and the common parallax image C02'
obtained as a result of the decoding, to the image
generating unit 502.
[0319]
In accordance with a display instruction from a
viewer, the image generating unit 502 outputs images, and
causes the display device (not shown) to display the
images. Specifically, in accordance with a multi-view 30
image display instruction from a viewer, the image
generating unit 502 generates images that are of three or
more viewpoints corresponding to the display device (not
shown) and have half the resolution of the compatible
images and the auxiliary images, using the multiplexed
image of the compatible images, the multiplexed image of
the auxiliary images, the common parallax image AB2', and
the common parallax image C02', based on image
information from an image information acquiring unit 123,
inter-viewpoint distance information from an interviewpoint
distance information acquiring unit 124,
100
SP308971WOOO
parallax image information from a parallax image
information acquiring unit 125, compatibility information
from a compatibility information acquiring unit 126, and
the like.
5 [0320]
More specifically, based on the information that is
contained in the parallax image information from the
parallax image information acquiring unit 125 and
indicates that the parallax images of the compatible
10 images and the auxiliary images are common parallax
images, the image generating unit 502 maintains the
common parallax image AB2' and the common parallax image
C02' as they are.
[0321]
15 Also, based on the information that is contained in
the image information from the image information
acquiring unit 123 and indicates the multiplexing
technique for the auxiliary images, the image generating
unit 502 separates each of the auxiliary images from the
20 multiplexed image of the auxiliary images, like the image
generating unit 171 shown in Fig. 3. Further, based on
the information that is contained in the image
information and indicates the multiplexing technique for
the compatible images, the image generating unit 502
25 separates each of the compatible images from the
multiplexed image of the compatible images, like the
image generating unit 171.
[0322]
Also, based on the inter-viewpoint distance
30 information and the number of viewpoints corresponding to
the display device (not shown), the image generating unit
101
SP308971WOOO
502 determines the location of each of the viewpoints of
the multi-view image to be generated, like the image
generating unit 171. Using the respective compatible
images, the respective auxiliary images, the common
5 parallax image AB2', and the common parallax image CD2',
the image generating unit 502 generates images of the
respective viewpoints having the locations determined.
Like the image generating unit 171, the image generating
unit 502 then converts the resolution of the generated
10 images of the respective viewpoints to the resolution
calculated by dividing the resolution of the compatible
images and the auxiliary images by the number of
viewpoints, and combines the generated images. The
resultant image is then displayed on the display device
15 (not shown).
[0323]
At this point, the multi-view image after the
combining is displayed in such a manner that the viewable
angle varies at each viewpoint, and the viewer can view a
20 3D image without glasses by seeing the images of any two
viewpoints with both eyes.
[0324]
In accordance with a two-view 3D image display
instruction from a viewer, the image generating unit 502
25 separates the multiplexed image of the compatible images
supplied from the decoder 501 into the image A2 and the
image B2 with half the resolution of the compatible
images, based on the image information from the image
information acquiring unit 123, like the image generating
30 unit 171. Like the image generating unit 171, the image
generating unit 502 alternately outputs the image A2 and
102
•
5
10
15
20
25
30
3P308971WOOO
the image B2 with half the resolution of the separated
compatible images, and causes the display device (not
shown) to display those images. At this point, the
viewer puts on glasses that open one of the shutter for
the left eye and the shutter for the right eye when the
image A2 is displayed, and open the other one of the
shutters when the image B2 is displayed. Wearing the
glasses, the viewer sees the image A2 and the image B2
that are alternately displayed on the display device. In
this manner, the viewer can view a 3D image.
[0325]
[Description of an Operation of the Decoding Device]
Fig. 44 is a flowchart for explaining a decoding
operation by the decoding device 500 shown in Fig. 43.
This decoding operation is started when the multiplexed
stream transmitted from the encoding device 480 shown in
Fig. 40 is input to the decoding device 500, for example.
[0326]
The procedures in steps 3511 through 3513 in Fig.
44 are the same as the procedures in steps 3471 through
3473 in Fig. 39, and therefore, explanation of them is
not repeated herein.
[0327]
In step 3514, the auxiliary decoder 511 extracts
the encoded multiplexed image of the auxiliary images,
the encoded common parallax image AB2' of the compatible
images, and the encoded common parallax image C02' of the
auxiliary images from the auxiliary stream, and decodes
the extracted images by a technique corresponding to the
encoding technique used in the auxiliary encoder 491
shown in Fig. 40. The auxiliary decoder 511 supplies the
103
•
5
10
15
20
25
8P308971WOOO
multiplexed image of the auxiliary images, the common
parallax image AB2', and the common parallax image C02'
obtained as a result of the decoding, to the image
generating unit 502.
[0328]
The procedures in steps 8515 through 8519 are the
same as the procedures in steps 8475 through 8479 in Fig.
39, and therefore, explanation of them is not repeated
herein.
[0329]
After the procedure in step 8519, the image
generating unit 502 in step 8520 generates images of the
respective viewpoints that have half the resolution of
the compatible images and the auxiliary images, using the
compatible images, the multiplexed image of the auxiliary
images, the common parallax image AB2', and the common
parallax image C02', based on the locations of the
respective viewpoints determined in step 8519, the image
information from the image information acquiring unit 123,
the parallax image information from the parallax image
information acquiring unit 125, the compatibility
information from the compatibility information acquiring
unit 126, and the like.
[0330]
The procedures in steps 8521 through 8526 are the
same as the procedures in steps 8481 through 8486 in Fig.
39, and therefore, explanation of them is not repeated
herein.
[0331]
30 [Baseband Data Transmission]
In the above described decoding device 500 shown in
104
•
5
10
15
20
25
30
SP308971WOOO
Fig. 43, if the left side (the separating unit 461 and
the decoder 501) of the dashed line L5 is designed as a
reproducing device, and the right side (the image
generating unit 502) of the dashed line L3 is designed as
a display device, the reproducing device and the display
device have the same structures as those of the above
described reproducing device 301 and the reproducing
device 302, respectively.
[0332]
The encoding device 440 and the encoding device 480
encode compatible images of two viewpoints after
multiplexing those compatible images, like the encoding
device 140. However, the encoding device 440 and the
encoding device 480 may encode the compatible images of
two viewpoints without multiplexing the compatible images,
like the encoding device 180 shown in Fig. 5. Also, the
encoding device 440 and the encoding device 480 may
encode a compatible image of one viewpoint, like the
encoding device 50 shown in Fig. 1.
[0333]
Also, the encoding device 140 and the encoding
device 180 may encode parallax images of compatible
images and auxiliary images, without multiplexing those
parallax images. Further, the encoding device 50 may
encode parallax images of auxiliary images, without
multiplexing those parallax images.
[0334]

Fig. 45 is a diagram showing examples of
multiplexing patterns for objects to be encoded in cases
105
• SP308971WOOO
where the number of compatible image viewpoints is 2, and
the number of auxiliary image viewpoints is 2.
[0335]
As shown in (1) in Fig. 45, the encoding device 140
5 shown in Fig. 3 multiplexes the image A2 and the image B2
as compatible images in a spatial direction, and encodes
those images by the AVC technique. The encoding device
140 also multiplexes, in a spatial direction, the image
C2 and the image D2 as the auxiliary images, the parallax
10 image A2' and the parallax image B2' as the parallax
images of the compatible images, and the parallax image
C2' and the parallax image D2' as the parallax images of
the auxiliary images, and encodes those multiplexed
images by the 3DV technique compliant with the MVC
15 technique.
[0336]
As shown in (2) in Fig. 45, the encoding device 140
may not multiplex the parallax images A2' through D2',
and may encode those parallax images by the 3DV technique
20 compliant with the MVC technique. Further, as shown in
(3) in Fig. 45, the encoding device 480 shown in Fig. 40
encodes the common parallax image AB2', instead of the
parallax image A2' and the parallax image B2' of the
parallax images of the compatible images, and encodes the
25 common parallax image CD2', instead of the parallax image
C2' and the parallax image D2' of the parallax images of
the auxiliary images.
[0337]
Also, as shown in (4) in Fig. 45, the encoding
30 device 440 shown in Fig. 31 does not multiplex the
parallax image A2' and the parallax image B2' as the
106
•
5
10
15
20
25
30
SP308971WOOO
parallax images of the compatible images in a spatial
direction, but multiplexes the parallax image A2', the
parallax image B2', and the multiplexed image of the
parallax images of the auxiliary images by the frame
sequential technique, and then encodes the multiplexed
image by the 30V technique compliant with the MVC
technique.
[0338]
Further, as shown in (5) in Fig. 45, the encoding
device 180 shown in Fig. 5 does not multiplex the image
A2 and the image B2 as the compatible images, but encodes
the image A2 by the AVC technique, and encodes the image
B2 by the MVC technique using the image A2 as the base
view. The encoding device 180 also multiplexes, in a
spatial direction, the image C2 and the image 02 as the
auxiliary images, the parallax image A2' and the parallax
image B2' as the parallax images of the compatible images,
and the parallax image C2' and the parallax image 02' as
the parallax images of the auxiliary images, and encodes
those multiplexed images by the 30V technique compliant
with the MVC technique.
[0339]
As shown in (6) in Fig. 45, the encoding device 180
may not multiplex the parallax images A2' through 02',
and may encode those parallax images by the 30V technique
compliant with the MVC technique. As shown in (7) in Fig.
45, the encoding device 180 may encode the common
parallax image AB2', instead of the parallax image A2'
and the parallax image B2', and encode the common
parallax image C02', instead of the parallax image C2'
and the parallax image 02', like the encoding device 480.
107
• SP30897lWOOO
[0340]
Further, as shown in (8) in Fig. 45, the encoding
device 180 may not multiplex the parallax image A2' and
the parallax image B2' in a spatial direction, but may
5 multiplex the parallax image A2', the parallax image B2',
and the multiplexed image of the parallax images of the
auxiliary images by the frame sequential technique, and
encode the multiplexed image by the 3DV technique
compliant with the MVC technique, like the encoding
10 device 440.
[0341]
Fig. 46 is a diagram showing the characteristics of
the effects of the multiplexing using the multiplexing
patterns shown in (1) through (8) in Fig. 45.
15 [0342]
In the table in Fig. 46, the items of effects,
"compatibility", "image quality", and "data amount", are
provided to indicate the levels of effects in the
respective items of the multiplexing patterns shown in
20 (1) through (8) in Fig. 45. It should be noted that, in
the table in Fig. 46, single circles represent "good",
and double circles represent "excellent".
[0343]
In a case where multiplexing is performed in the
25 multiplexing pattern shown in (1) in Fig. 45, the
multiplexing technique and the encoding technique for the
compatible images are the same as conventional techniques,
and compatibility is secured. Also, as the parallax
images are multiplexed in a spatial direction like the
30 compatible images, the parallax images can be separated
by using the separating unit that is prepared on the
108
•
5
SP308971WOOO
decoding device side and separates the compatible images,
for example. Accordingly, separation of the parallax
images on the decoding device side is guaranteed. Thus,
this case is particularly effective in terms of the
compatibility, and a double circle is put in the
"compatibility" column in Fig. 46.
[0344]
In a case where multiplexing is performed in the
multiplexing pattern shown in (2) in Fig. 45, the
10 multiplexing technique and the encoding technique for the
compatible images are also the same as conventional
techniques, and compatibility is secured. As the
resolution of the parallax images is the same as the
resolution of the pre-multiplex images, high precision is
15 achieved. As a result, the precision of the images of
predetermined viewpoints generated by using the parallax
images is increased in the decoding device. Accordingly,
this case is effective in terms of the compatibility and
the image quality of the images generated by using the
20 parallax images. Thus, single circles are put in the
"compatibility" and "image quality" columns in Fig. 46.
[0345 ]
Further, in a case where multiplexing is performed
in the multiplexing pattern shown in (3) in Fig. 45, the
25 encoding technique and the multiplexing technique for the
compatible images are the same as conventional techniques,
and compatibility is secured. Also, the data amount of
the parallax images of the compatible images and the
auxiliary images, the resolution of which is the same as
30 the resolution of the pre-multiplex images, is reduced to
the data amount of parallax images of two viewpoints.
109
I
5
10
15
20
25
SP308971WOOO
Accordingly, this case is effective in terms of the
compatibility and the data amount, and single circles are
put in the "compatibility" and "data amount" columns in
Fig. 46.
[0346]
In a case where multiplexing is performed in the
multiplexing pattern shown in (4) in Fig. 45, the
encoding technique and the multiplexing technique for the
compatible images are also the same as conventional
techniques, and compatibility is secured. Also, as the
parallax images are multiplexed in a temporal direction,
the data amount of the parallax images at each time
becomes smaller than in the case (3) in Fig. 45, and the
transmittable data amount increases. Accordingly, even
in a situation where the transmission band is not wide
enough to transmit parallax images having the same
resolution as pre-multiplex images, parallax images
having the same resolution as pre-multiplex images can be
transmitted. Accordingly, the precision of the images of
predetermined viewpoints generated by using the parallax
images becomes higher in the decoding device. This case
is effective in terms of the compatibility and the image
quality of the images generated by using the parallax
images, and is particularly effective in terms of the
data amount. Thus, single circles are put in the
"compatibility" and "image quality" columns, and a double
circuit is put in the "data amount" column in Fig. 46.
[0347]
Multiplexing in the multiplexing patterns shown in
30 (1) through (4) in Fig. 45 is performed when encoding is
performed on images for broadcasting such as next-
110
•
5
SP308971WOOO
generation broadcasting combined with the IP (Internet
Protocol) compliant with the ATSC (Advanced Television
Systems Committee) 2.0 standard or the like, or images
for applications distributed via the Internet, for
example.
[0348]
Further, in a case where multiplexing is performed
in the multiplexing pattern shown in (5) in Fig. 45, the
encoding technique and the multiplexing technique for the
10 compatible images are the same as conventional techniques,
and compatibility is secured. Also, as the resolution of
the respective parallax images is half the resolution of
the images, the data amount of the parallax images of the
compatible images and the auxiliary images is reduced to
15 the data amount of parallax images of two viewpoints.
Accordingly, this case is effective in terms of the
compatibility and the data amount, and single circles are
put in the "compatibility" and "data amount" columns in
Fig. 46.
20 [0349]
In a case where multiplexing is performed in the
multiplexing pattern shown in (6) in Fig. 45,
compatibility is secured, and the precision of the images
of predetermined viewpoints generated by using the
25 parallax images is increased in the decoding device, as
in (2) in Fig. 45. Accordingly, this case is effective
in terms of the compatibility and the image quality of
the images generated by using the parallax images. Thus,
single circles are put in the "compatibility" and "image
30 quality" columns in Fig. 46.
[ 0350]
111
•
5
SP308971WOOO
In a case where multiplexing is performed in the
multiplexing pattern shown in (7) in Fig. 45,
compatibility is secured, and the data amount of the
parallax images is reduced, as in (3) in Fig. 45.
Accordingly, this case is effective in terms of the
compatibility and the data amount, and single circles are
put in the "compatibility" and "data amount" columns in
Fig. 46.
[0351]
20
15
10 In a case where multiplexing is performed in the
multiplexing pattern shown in (8) in Fig. 45,
compatibility is secured, as in (4) in Fig. 45. Also, as
in (4) in Fig. 45, the data amount of the parallax images
at each time is made smaller than in the case (7) in Fig.
45. As a result, the precision of the images of
predetermined viewpoints generated by using the parallax
images in the decoding device becomes higher. This case
is effective in terms of the compatibility and the image
quality of the images generated by using the parallax
images, and is particularly effective in terms of the
data amount. Thus, single circles are put in the
"compatibility" and "image quality" columns, and a double
circuit is put in the "data amount" column in Fig. 46.
[0352]
25 Multiplexing in the multiplexing patterns shown in
(5), (7), and (8) in Fig. 45 is performed when encoding
is performed on images for BDs, broadcasting, nextgeneration
broadcasting, or applications distributed via
the Internet, for example. Further, multiplexing in the
30 multiplexing pattern shown in (6) in Fig. 45 is performed
when encoding is performed on images for BDs, next-
112
• SP308971WOOO
generation broadcasting, or applications distributed via
the Internet, for example.
[0353]
Fig. 47 is a diagram showing examples of
5 multiplexing patterns for objects to be encoded in cases
where the number of compatible image viewpoints is 1, and
the number of auxiliary image viewpoints is 2.
[0354]
As shown in (1) in Fig. 47, the encoding device 50
10 shown in Fig. 1 encodes the image A1 as the compatible
image by the AVC technique. The encoding device 50 also
multiplexes, in a temporal direction, the image B1 and
the image C1 as the auxiliary images, and the parallax
image B1' and the parallax image C1' as the parallax
15 images of the auxiliary images. The encoding device 50
then encodes the parallax image A1' as the parallax image
of the compatible image, the multiplexed image of the
auxiliary images, and the multiplexed image of the
parallax images of the auxiliary images by the 3DV
20 technique compliant with the MVC technique.
[0355]
As shown in (2) in Fig. 47, the encoding device 50
may not multiplex the parallax images A1' through C1',
and may encode those parallax images by the 3DV technique
25 compliant with the MVC technique. Also, as shown in (3)
in Fig. 47, the encoding device 50 may encode the common
parallax image BC1' of the image B and the image C as the
auxiliary images, instead of the multiplexed image of the
parallax images of the auxiliary images, like the
30 encoding device 480.
[0356]
113
•
5
10
15
20
25
SP308971WOOO
Further, as shown in (4) in Fig. 47, the encoding
device 50 may not multiplex the parallax image B1' and
the parallax image C1' in a spatial direction, but may
multiplex the parallax images A1' through C1' by the
frame sequential technique, and encode the multiplexed
image by the 3DV technique compliant with the MVC
technique, like the encoding device 440.
[0357]
The effects of the multiplexing in the multiplexing
patterns shown in (1) through (4) in Fig. 47, and the
objects to be encoded when the multiplexing is performed
in the multiplexing patterns are the same as the effects
of the multiplexing and the objects to be encoded in the
multiplexing patterns shown in (5) through (8) in Fig. 45,
respectively. However, in the multiplexing in the
multiplexing pattern shown in (1) in Fig. 47, the
resolution of the parallax image of the compatible image
is the same as the resolution of the compatible image,
and therefore, only the data amount of the parallax
images of the auxiliary images can be effectively reduced
by virtue of the multiplexing.
[0358]
Fig. 48 is a diagram showing examples of
multiplexing patterns for objects to be encoded in cases
where the number of compatible image viewpoints is 2, and
the number of auxiliary image viewpoints is O.
[0359]
In a case where the number of compatible image
viewpoints is 2 while the number of auxiliary image
30 viewpoints is 0, the image A2 and the image B2 as the
compatible images are multiplexed in a spatial direction
114
•
5
10
15
20
25
30
SP308971WOOO
shown in (1) in Fig. 48, and are encoded by the Ave
technique, as in the case of the encoding device 140
shown in Fig. 3. Also, the parallax image A2' and the
parallax image B2' as the parallax images of the
compatible images are multiplexed in a spatial direction,
and are encoded by the 3DV technique compliant with the
Ave technique.
[0360]
As shown in (2) in Fig. 48, the parallax image A2'
and the parallax image B2' may not be multiplexed, but
may be encoded by the 3DV technique compliant with the
Mve technique. Also, as shown in (3) in Fig. 48, the
common parallax image AB2' may be encoded, instead of the
parallax image A2' and the parallax image B2', as in the
case of the encoding device 480.
[0361]
Also, as shown in (4) in Fig. 48, the parallax
image A2' and the parallax image B2' may not be
multiplexed in a temporal direction, but may be
multiplexed by the frame sequential technique and be then
encoded, as in the case of the encoding device 440.
[0362]
Further, as shown in (5) in Fig. 48, the image A2
and the image B2 as the compatible images may not be
multiplexed, but the image A2 may be encoded by the Ave
technique, and the image B2 may be encoded by the Mve
technique using the image A2 as the base view, as in the
case of the encoding device 180.
[0363]
In this case, the parallax images may not be
multiplexed as shown in (6) in Fig. 48, but may be
115
•
5
10
15
20
25
30
SP308971WOOO
encoded, as in (2) in Fig. 48. Also, as shown in (7) in
Fig. 48, the common parallax image AB2' may be encoded
instead, as in (3) in Fig. 48. Also, as shown in (8) in
Fig. 48, the parallax images of the compatible images may
be multiplexed by the frame sequential technique, and be
then encoded, as in (4) in Fig. 48.
[0364]
The effects of the multiplexing in the multiplexing
patterns shown in (1) through (8) in Fig. 47, and the
objects to be encoded when the multiplexing is performed
in the multiplexing patterns are the same as the effects
of the multiplexing and the objects to be encoded in the
multiplexing patterns shown in (1) through (8) in Fig. 45,
respectively.
[0365]
In the above described examples, the resolution of
the parallax images multiplexed by the frame sequential
technique is the same as the resolution of the premultiplex
images. However, the resolution of those
parallax images may be lower than the resolution of the
pre-multiplex images. Also, the auxiliary images may be
multiplexed by the frame sequential technique, like the
parallax images.
[0366]
Further, in the above described examples, the
information indicating the multiplexing technique for the
images and the information indicating the multiplexing
technique for the parallax images are transmitted in the
encoding devices. However, information for identifying
the multiplexing patterns shown in Figs. 45, 47, and 48
may be transmitted.
116
• SP308971WOOO
[0367]
Also, the encoding devices may transmit a flag for
identifying an application suitable for the images to be
encoded.
5 [0368]
<5. Fifth Embodiment>
[Example Structure of an Encoding Device]
Fig. 49 is a block diagram showing an example
structure of an encoding device that encodes 3D images
10 transmitted in a fifth embodiment of a transmission
system to which the present technique is applied.
[0369]
In the structure illustrated in Fig. 49, the same
components as those in Fig. 3 are denoted by the same
15 reference numerals as those in Fig. 3. The explanations
that have already been made will not be repeated.
[0370]
The structure of the encoding device 520 shown in
Fig. 49 differs from the structure shown in Fig. 3 mainly
20 in that the encoder 145 and the multiplexing unit 443 are
replaced with an encoder 523 and a transmitting unit 524,
and a multiplexing unit 521 and a multiplexing
information generating unit 522 are added.
[0371]
25 The encoding device 520 multiplexes a multiplexed
image of compatible images, a multiplexed image of
auxiliary images, a multiplexed image of parallax images
of the compatible images, and a multiplexed image of
parallax images of the auxiliary images by the frame
30 sequential technique, and then encodes the resultant
multiplexed image.
117
•
5
10
15
20
25
30
SP308971WOOO
[0372]
Specifically, the multiplexing unit 521 of the
encoding device 520 multiplexes, by the frame sequential
technique, the multiplexed images of the compatible
images and the auxiliary images obtained as a result of
multiplexing performed by an image converting unit 142,
and the multiplexed images of the parallax images of the
compatible images and the auxiliary images obtained as a
result of multiplexing performed by a parallax image
generating unit 143.
[0373]
The multiplexing unit 521 then supplies the
multiplexed image obtained as a result of the
multiplexing as a temporal-direction multiplexed image to
the encoder 523. In this temporal-direction multiplexed
image, the multiplexed image of the compatible images,
the multiplexed image of the parallax images of the
compatible images, the multiplexed image of the auxiliary
images, and the multiplexed image of the parallax images
of the auxiliary images sequentially exist in one frame
time.
[0374]
The multiplexing unit 521 also supplies information
to the multiplexing information generating unit 522 and
the encoder 523. The information indicates that the
multiplexed images of the compatible images and the
auxiliary images, and the multiplexed images of the
parallax images of the compatible images and the
auxiliary images are multiplexed by the frame sequential
technique, and also indicates the sequential order of the
images in the temporal multiplexed image.
118
• SP308971WOOO
[0375]
Based on the information supplied from the
multiplexing unit 521, the multiplexing information
generating unit 522 generates the information and the
5 like as overall multiplexing information about the
multiplexing of the compatible images and the auxiliary
images, and the multiplexing of the parallax images of
the compatible images and the auxiliary images. The
overall multiplexing information is supplied to the
10 transmitting unit 524.
[0376]
The encoder 523 is formed with a compatible encoder
531 and an auxiliary encoder 532. Based on the
information supplied from the multiplexing unit 521, the
15 encoder 523 supplies the multiplexed image of the
compatible images in the temporal multiplexed image
supplied from the multiplexing unit 521, to the
compatible encoder 531. Based on the information
supplied from the multiplexing unit 521, the encoder 523
20 also supplies the multiplexed image of the auxiliary
images and the multiplexed images of the parallax images
of the compatible images and the auxiliary images in the
temporal multiplexed image, to the auxiliary encoder 532.
[0377]
25 The compatible encoder 531 of the encoder 523
encodes the multiplexed image of the compatible images in
the temporal multiplexed image by the conventional Ave
technique. The auxiliary encoder 532 encodes the
multiplexed image of the auxiliary images and the
30 multiplexed images of the parallax images of the
compatible images and the auxiliary images in the
119
•
5
10
15
20
25
30
SP308971WOOO
temporal multiplexed image, by the 3DV technique. At
this point, the multiplexed image of the auxiliary images
is encoded by referring to the multiplexed image of the
compatible images, and the multiplexed image of the
parallax images of the auxiliary images is encoded by
referring to the multiplexed image of the parallax images
of the compatible images.
[0378]
The encoder 523 supplies a bit stream to the
transmitting unit 524. The bit stream is formed with
encoded data of the temporal multiplexed image obtained
as a result of the encoding performed by the compatible
encoder 531 or the auxiliary encoder 532.
[0379]
Using the bit stream supplied from the encoder 523,
image information from an image information generating
unit 54, compatibility information from a compatibility
information generating unit 55, inter-viewpoint distance
information from an inter-viewpoint distance information
generating unit 144, parallax image information from a
parallax image information generating unit 57, the
overall multiplexing information from the multiplexing
information generating unit 522, and the like, the
transmitting unit 524 generates TSs. The multiplexing
unit 443 then transmits the generated TSs.
[0380]
[Description of an Operation of the Encoding Device]
Figs. 50 and 51 are flowcharts for explaining an
encoding operation by the encoding device 520 shown in
Fig. 49. This encoding operation is started when images
A2 through 02 are output from imaging units 141A through
120
•
5
10
15
20
25
30
8P308971WOOO
1410, for example.
[0381]
The procedures in steps 8531 through 8537 in Fig.
50 are the same as the procedures in steps 8451 through
8457 in Fig. 36, and therefore, explanation of them is
not repeated herein.
[0382]
After the procedure in step 8537, the image
converting unit 142 in step 8538 inputs the multiplexed
image of the compatible images and the multiplexed image
of the auxiliary images to the multiplexing unit 521, and
the operation moves on to step 8539 in Fig. 51.
[0383]
In step 8539 in Fig. 51, using the images A2
through 02 supplied from the imaging units 141A through
1410, the parallax image generating unit 143 detects the
parallax of each pixel of the images A2 through 02, and
generates parallax images A2' through 02'.
[0384]
In step 8540, the parallax image generating unit
143 determines the multiplexing techniques for the
parallax images of the compatible images and the parallax
images of the auxiliary images, and supplies information
indicating the multiplexing technique to the parallax
image information generating unit 57.
[0385]
In step 8541, based on the information supplied
from the parallax image generating unit 143, the parallax
image information generating unit 57 generates
information indicating the multiplexing techniques for
the parallax images of the compatible images and the
121
• 8P308971WOOO
parallax images of the auxiliary images and the like as
the parallax image information, and inputs the parallax
image information to the transmitting unit 524.
[0386]
5 In step 8542, based on the multiplexing technique
for the parallax images of the compatible images
determined in step 8540, the parallax image generating
unit 143 multiplexes the parallax image A2' and the
parallax image B2' of the compatible images. Also, based
10 on the multiplexing technique for the parallax images of
the auxiliary images, the parallax image generating unit
143 multiplexes the parallax image C2' and the parallax
image D2' of the auxiliary images.
[0387]
15 After the procedure in step 8543, the multiplexing
unit 521 in step 8544 multiplexes, by the frame
sequential technique, the multiplexed images of the
compatible images and the auxiliary images obtained as a
result of the multiplexing performed by the image
20 converting unit 142, and the multiplexed images of the
parallax images of the compatible images and the
auxiliary images obtained as a result of the multiplexing
performed by the parallax image generating unit 143. The
multiplexing unit 521 supplies the multiplexed image
25 obtained as a result of the multiplexing as the temporaldirection
multiplexed image to the encoder 523.
[0388]
The multiplexing unit 521 also supplies information
to the multiplexing information generating unit 522 and
30 the encoder 523. The information indicates that the
multiplexed images of the compatible images and the
122
•
5
10
15
20
25
30
8P308971WOOO
auxiliary images, and the multiplexed images of the
parallax images are multiplexed by the frame sequential
technique, and also indicates the sequential order of the
images in the temporal multiplexed image.
[0389]
In step 8545, based on the information supplied
from the multiplexing unit 521, the multiplexing
information generating unit 522 generates the information
and the like as the overall multiplexing information, and
inputs the overall multiplexing information to the
transmitting unit 524.
[0390]
In step 8546, the compatible encoder 531 encodes,
by the conventional Ave technique, the multiplexed image
of the compatible images in the temporal multiplexed
image, which is input by the encoder 523 based on the
information from the multiplexing unit 521.
[0391]
In step 8547, the auxiliary encoder 532 encodes, by
the 3DV technique, the multiplexed image of the auxiliary
images, the multiplexed image of the parallax images of
the compatible images, and the multiplexed image of the
parallax images of the auxiliary images in the temporal
multiplexed image, which is input by the encoder 523
based on the information from the multiplexing unit 521.
The encoder 523 supplies the bit stream formed with the
encoded data of the temporal multiplexed image obtained
as a result of the encoding in steps 8546 and 8547, to
the transmitting unit 524.
[0392]
In step 8548, the transmitting unit 524 generates
123
• SP308971WOOO
the TSs from the bit stream from the encoder 523,
auxiliary information, and the overall multiplexing
information from the multiplexing information generating
unit 522, and transmits the TSs. The TSs may be recorded
5 on a BD or the like, or may be transmitted as a stream
for broadcasting, for example. The operation then comes
to an end.
[0393]
10
As described above, the encoding device 520
generates one bit stream from the encoded data of the
multiplexed image of the compatible images, the
multiplexed image of the auxiliary images, the
multiplexed image of the parallax images of the
20
compatible images, and the multiplexed image of the
15 parallax images of the auxiliary images. Accordingly, in
a decoding device that includes a decoder capable of
decoding only one bit stream, the bit stream generated by
the encoding device 520 can be decoded.
[0394]
In the above described example, the parallax images
of the compatible images, the auxiliary images, and the
parallax images of the auxiliary images are encoded by
the 3DV technique compliant with the encoding technique
for the compatible images. However, those images may be
25 encoded by MPEG2 (Moving Picture Experts Group phase 2)
30
or the like, which is not compliant with the encoding
technique for the compatible images.
[0395]
[Example Structure of a Decoding Device]
Fig. 52 is a diagram showing an example structure
of a decoding device that decodes the TSs transmitted
124
• SP308971WOOO
from the encoding device 520 shown in Fig. 49.
[0396]
In the structure illustrated in Fig. 52, the same
components as those in Fig. 4 are denoted by the same
5 reference numerals as those in Fig. 4. The explanations
that have already been made will not be repeated.
[0397]
The structure of the decoding device 540 shown in
Fig. 52 differs from· the structure shown in Fig. 4 mainly
10 in that the separating unit 121, the decoder 122, and the
image generating unit 171 are replaced with a receiving
unit 541, a decoder 542, and an image generating unit 544,
and a multiplexing information acquiring unit 543 is
added. The decoding device 540 decodes the bit stream of
15 the temporal multiplexed image contained in the TSs
transmitted from the encoding device 520, generates a
two-view image or a multi-view image, and causes a
display device (not shown) to display the generated image.
[0398]
20 Specifically, the receiving unit 541 of the
decoding device 540 receives the TSs transmitted from the
encoding device 520. The receiving unit 541 extracts the
bit stream of the temporal multiplexed image contained in
the TSs, and supplies the bit stream to the decoder 542.
25 The receiving unit 541 also extracts the auxiliary
information contained in the TSs, supplies the image
information in the auxiliary information to an image
information acquiring unit 123, and supplies the interviewpoint
distance information to an inter-viewpoint
30 distance information acquiring unit 124. Further, the
receiving unit 541 supplies the parallax image
125
•
5
10
15
20
25
30
SP308971WOOO
information in the auxiliary information to a parallax
image information acquiring unit 125, and supplies the
compatibility information to a compatibility information
acquiring unit 126. The receiving unit 541 also extracts
the overall multiplexing information contained in the TSs,
and supplies the overall multiplexing information to the
multiplexing information acquiring unit 543.
[0399]
The decoder 542 is formed with a compatible decoder
551 and an auxiliary decoder 552. By a technique
corresponding to the Ave technique, the compatible
decoder 551 of the decoder 542 decodes the encoded data
of the multiplexed image of the compatible images
contained in the bit stream supplied from the receiving
unit 541, and supplies the decoded image to the image
generating unit 544.
[0400]
By a technique corresponding to the encoding
technique in the auxiliary encoder 532 shown in Fig. 49,
the auxiliary decoder 552 decodes the encoded data of the
multiplexed image of the auxiliary images, the
multiplexed image of the parallax images of the
compatible images, and the multiplexed image of the
parallax images of the auxiliary images, which are
contained in the bit stream supplied from the receiving
unit 541. The auxiliary decoder 552 supplies the
multiplexed image of the auxiliary images, the
multiplexed image of the parallax images of the
compatible images, and the multiplexed image of the
parallax images of the auxiliary images, which are
obtained as a result of the decoding, to the image
126
• SP308971WOOO
generating unit 544.
[0401]
The multiplexing information acquiring unit 543
acquires the overall multiplexing information supplied
5 from the receiving unit 541, and supplies the overall
multiplexing information to the image generating unit 544.
[0402]
In accordance with a display instruction from a
viewer, the image generating unit 544 outputs images, and
10 causes the display device (not shown) to display the
images. Specifically, in accordance with a multi-view 3D
image display instruction from a viewer, the image
generating unit 544 generates images that are of three or
more viewpoints corresponding to the display device (not
15 shown) and have half the resolution of the compatible
images and the auxiliary images, using the multiplexed
image of the compatible images, the multiplexed image of
the auxiliary images, the multiplexed image of the
parallax images of the compatible images, and the
20 multiplexed image of the parallax images of the auxiliary
images, based on the image information from the image
information acquiring unit 123, the inter-viewpoint
distance information from the inter-viewpoint distance
information acquiring unit 124, the parallax image
25 information from the parallax image information acquiring
unit 125, the compatibility information from the
compatibility information acquiring unit 126, the overall
multiplexing information from the multiplexing
information acquiring unit 543, and the like.
30 [0403]
More specifically, based on the overall
127
.. SP308971WOOO
multiplexing information from the multiplexing
information acquiring unit 543, the image generating unit
544 identifies the multiplexed image of the auxiliary
images, the multiplexed image of the parallax images of
5 the compatible images, and the multiplexed image of the
parallax images of the auxiliary images, which are
supplied from the auxiliary decoder 552. Based on the
information that is contained in the parallax image
information from the parallax image information acquiring
10 unit 125 and indicates the multiplexing technique for the
parallax images of the auxiliary images, the image
generating unit 544 separates the parallax image of each
of the auxiliary images from the multiplexed image of the
parallax images of the auxiliary images, like the image
15 generating unit 171 shown in Fig. 4. Based on the
information that is contained in the parallax image
information and indicates the multiplexing technique for
the parallax images of the compatible images, the image
generating unit 544 also separates the parallax image of
20 each of the compatible images from the multiplexed image
of the parallax images of the compatible images, like the
image generating unit 171.
[0404]
Further, based on the information that is contained
25 in the image information from the image information
acquiring unit 123 and indicates the multiplexing
technique for the auxiliary images, the image generating
unit 544 separates each of the auxiliary images from the
multiplexed image of the auxiliary images, like the image
30 generating unit 171. Based on the information that is
contained in the image information and indicates the
128
•
5
10
15
20
25
30
SP308971WOOO
multiplexing technique for the compatible images, the
image generating unit 544 also separates each of the
compatible images from the multiplexed image of the
compatible images, like the image generating unit 171.
[0405]
Based on the inter-viewpoint distance information
and the number of viewpoints corresponding to the display
device (not shown), the image generating unit 544 also
determines the location of each of the viewpoints of the
multi-view image to be generated, like the image
generating unit 171. Using the respective compatible
images, the respective auxiliary images, the parallax
images of the respective compatible images, and the
parallax images of the respective auxiliary images, the
image generating unit 544 generates images of the
respective viewpoints having the locations determined,
like the image generating unit 171. The image generating
unit 544 then converts the resolution of the generated
images of the viewpoints to the resolution calculated by
dividing the resolution of the compatible images and the
auxiliary images by the number of viewpoints, and
combines the generated images. The resultant image is
displayed on the display device (not shown).
[0406]
At this point, the multi-view image after the
combining is displayed in such a manner that the viewable
angle varies at each viewpoint, and the viewer can view a
3D image without glasses by seeing the images of any two
viewpoints with both eyes.
[0407]
In accordance with a two-view 3D image display
129
5
15
10
• SP308971WOOO
instruction from a viewer, the image generating unit 544
separates the multiplexed image of the compatible images
supplied from the decoder 542 into the image A2 and the
image 82 with half the resolution of the compatible
images, based on the image information from the image
information acquiring unit 123, like the image generating
unit 171. Like the image generating unit 171, the image
generating unit 544 alternately outputs the image A2 and
the image 82 with half the resolution of the separated
compatible images, and causes the display device (not
shown) to display those images. At this point, the
viewer puts on glasses that open one of the shutter for
the left eye and the shutter for the right eye when the
image A2 is displayed, and open the other one of the
shutters when the image 82 is displayed. Wearing the
glasses, the viewer sees the image A2 and the image 82
that are alternately displayed on the display device. In
this manner, the viewer can view a 3D image.
[0408]
20 [Description of an Operation of the Decoding Device]
Fig. 53 is a flowchart for explaining a decoding
operation by the decoding device 540 shown in Fig. 52.
This decoding operation is started when the TSs
transmitted from the encoding device 520 shown in Fig. 49
25 are input to the decoding device 540, for example.
[0409]
In step S551 in Fig. 53, the receiving unit 541 of
the decoding device 540 receives the TSs transmitted from
the encoding device 520. The receiving unit 541 extracts
30 the bit stream contained in the TSs, and supplies the bit
stream to the decoder 542. The receiving unit 541 also
130
•
5
10
15
20
25
30
8P308971WOOO
extracts the auxiliary information contained in the T8s,
supplies the image information in the auxiliary
information to the image information acquiring unit 123,
and supplies the inter-viewpoint distance information to
the inter-viewpoint distance information acquiring unit
124. Further, the receiving unit 541 supplies the
parallax image information in the auxiliary information
to the parallax image information acquiring unit 125, and
supplies the compatibility information to the
compatibility information acquiring unit 126. The
receiving unit 541 also extracts the overall multiplexing
information contained in the T8s, and supplies the
overall multiplexing information to the multiplexing
information acquiring unit 543.
[0410]
In step 8552, the image generating unit 544
determines whether a two-view 3D image display
instruction has been issued from a viewer. If it is
determined in step 8552 that a two-view 3D image display
instruction has not been issued from a viewer, or that a
multi-view 3D image display instruction has been issued,
the operation moves on to step 8553.
[0411]
In step 8553, the compatible decoder 551 of the
decoder 542 extracts the encoded data of the multiplexed
image of the compatible images from the bit stream
supplied from the receiving unit 541, and decodes the
extracted data by a technique corresponding to the Ave
technique. The compatible decoder 551 then supplies the
multiplexed image of the compatible images obtained as a
result of the decoding to the image generating unit 544.
131
4t 8P308971WOOO
[0412]
In step 8554, the auxiliary decoder 552 extracts
the encoded data of the multiplexed image of the
auxiliary images, the multiplexed image of the parallax
5 images of the compatible images, and the multiplexed
image of the parallax images of the auxiliary images from
the bit stream supplied from the receiving unit 541, and
decodes the extracted data by a technique corresponding
to the auxiliary encoder 532 shown in Fig. 49. The
10 auxiliary decoder 552 supplies the multiplexed image of
the auxiliary images, the multiplexed image of the
parallax images of the compatible images, and the
multiplexed image of the parallax images of the auxiliary
images, which are obtained as a result of the decoding,
15 to the image generating unit 544. The operation then
moves on to step 8555.
[0413]
The procedures in steps 8555 through 8558 are the
same as the procedures in steps 8475 through 8478 in Fig.
20 39, and therefore, explanation of them is not repeated
herein.
[0414]
After the procedure in step 8558, the multiplexing
information acquiring unit 543 in step 8559 acquires the
25 overall multiplexing information supplied from the
receiving unit 541, and inputs the overall multiplexing
information to the image generating unit 544.
[0415]
In step 8560, based on the inter-viewpoint distance
30 information from the inter-viewpoint distance information
acquiring unit 124 and the number of viewpoints
132
• 8P308971WOOO
corresponding to the display device (not shown), the
image generating unit 544 determines the location of each
of the viewpoints of the multi-view image to be generated.
[0416]
5 In step 8561, based on the locations of the
respective viewpoints determined in step 8560, the image
information from the image information acquiring unit 123,
the parallax image information from the parallax image
information acquiring unit 125, the compatibility
10 information from the compatibility information acquiring
unit 126, the overall multiplexing information from the
multiplexing information acquiring unit 543, and the like,
the image generating unit 544 generates images of the
respective viewpoints that have half the resolution of
15 the compatible images and the auxiliary images, using the
multiplexed image of the compatible images, the
multiplexed image of the auxiliary images, the
multiplexed image of the parallax images of the
compatible images, and the multiplexed image of the
20 parallax images of the auxiliary images. The operation
then moves on to step 8562.
[0417]
The procedures in steps 8562 through 8567 are the
same as the procedures in steps 8481 through 8486 in Fig.
25 39, and therefore, explanation of them is not repeated
herein. In the procedure in step 8564, however, the
compatible decoder 551 extracts the multiplexed image of
the compatible images from the bit stream supplied from
the receiving unit 541, instead of the compatible stream.
30 [0418]
In a decoding device that can decode only
133
•
5
SP308971WOOO
compatible images having compatibility with the decoding
device 540, encoded data other than the encoded data of
the processable compatible images is ignored, and the
procedures in steps S564, S566, and S567 are carried out.
In this case, however, in the procedure in step S566, the
multiplexed image of the compatible images is separated,
based on the multiplexing technique determined in advance.
[0419]
As described above, the decoding device 540 can
10 decode one bit stream generated from the encoded data of
the multiplexed image of the compatible images, the
multiplexed image of the auxiliary images, the
multiplexed image of the parallax images of the
compatible images, and the multiplexed image of the
15 parallax images of the auxiliary images.
[0420]
[Baseband Data Transmission]
In the above described decoding device 540 shown in
Fig. 52, if the left side (the receiving unit 541 and the
20 decoder 542) of the dashed line L5 is designed as a
reproducing device, and the right side (the image
generating unit 544) of the dashed line L5 is designed as
a display device, the reproducing device and the display
device have the same structures as those of the above
25 described reproducing device 301 and the reproducing
device 302, respectively.
[0421]
<6. Bit Streams in This Embodiment>
[Example Structure of a Bit Stream]
30 Fig. 54 is a diagram showing an example structure
of an access-unit bit stream (an encoded stream)
134
• SP308971WOOO
generated in this embodiment.
[0422]
In the example shown in Fig. 54, compatible images
are an L image and an R image of 1920 x 1080 pixels, and
5 an auxiliary image is an 0 image of 1920 x 1080 pixels.
An L parallax image, an R parallax image, and an 0
parallax image are 960 x 1080 in size. Further, the L
image is encoded by the Ave technique, the R image is
encoded by the Mve technique, and the 0 image, the L
10 parallax image, the R parallax image, and the 0 parallax
image are encoded by the 3DV technique. The view IDs of
the L image, the R image, the 0 image, the L parallax
image, the R parallax image, and the 0 parallax image are
0, 1, 2, 3, 4, and 5, respectively. It should be noted
15 that each view ID is an ID unique to an image or a
parallax image of a viewpoint.
[0423]
As shown in Fig. 54, the access-unit bit stream
includes an access unit delimiter (ADD), a SPS (Sequence
20 Parameter Set), a Subset SPS of the Mve technique for
images (Subset SPS1), a Subset SPS of the 3DV technique
for parallax images (Subset SPS2), a PPS (Picture
Parameter Set), SEI of the Ave technique, SEI of the Mve
technique, SEI of the 3DV technique, and a NAL(Network
25 Abstraction Layer) unit of encoded data, in this order.
[0424]
The NAL unit of the access unit delimiter is aNAL
unit representing the boundary of the access unit. The
NAL unit of the SPS is a NAL unit of a SPS containing
30 "profile ide" (100 in the example shown in Fig. 54)
representing the profile of the L image among the
135
•
5
10
15
20
25
30
SP308971WOOO
profiles defined by the Ave technique. The NAL unit of
the Subset SPS for images is a NAL unit of a Subset SPS
containing "profile ide" (128 in the example shown in Fig.
54) representing the profiles of the R image and the 0
image among the profiles defined by the Mve technique.
The NAL unit of the Subset SPS for parallax images is a
NAL unit of a Subset SPS containing "profile ide" (138 in
the example shown in Fig. 54) representing the profiles
of the L parallax image, the R parallax image, and the 0
parallax image among the profiles defined as the profiles
for parallax images by the 30V technique.
[0425]
The NAL unit of the SEI of the Ave technique is the
NAL unit of the SEI of the L image. The NAL unit of the
SEI of the Mve technique is the NAL unit of the SEI of
the L image and the R image. The NAL unit of the SEI of
the 30V technique is the NAL unit of the SEI of the L
image, the R image, the 0 image, the L parallax image,
the R parallax image, and the 0 parallax image.
[0426]
The NAL unit of encoded data includes encoded data
of the L image, a delimiter (MVe 00), encoded data of the
R image, a delimiter (30V 00), encoded data of the 0
image, a delimiter (30V 00), encoded data of the L
parallax image, a delimiter (30V 00), encoded data of the
R parallax image, a delimiter (30V 00), encoded data of
the 0 parallax image, in this order.
[0427]
A NAL header containing 1 or 5 as a type of NAL
unit, which represents an Ave picture, is attached to the
NAL unit of the encoded data of the L image. The NAL
136
5
10
• SP308971WOOO
unit of the delimiter (MVC DO) is a NAL unit representing
the top of MVC encoded data. A NAL header containing 20
as a type of NAL unit, which represents MVC encoded data,
is attached to the NAL unit of the encoded data of the R
image. The NAL unit of the delimiter (3DV DO) is aNAL
unit representing the top of 3DV encoded data. ANAL
header containing 21 as a type of NAL unit, which
represents 3DV encoded data, is attached to the NAL unit
of the encoded data of the 0 image, the L parallax image,
the R parallax image, and the 0 parallax image.
[0428]
[Example Description of the Subset SPS of the 3DV
Technique for Parallax Images]
Fig. 55 is a diagram showing an example description
15 of the Subset SPS of the 3DV technique for parallax
images shown in Fig. 54.
[0429]
As shown in Fig. 55, the Subset SPS of the 3DV
technique for parallax images includes aSPS
20 (seq_parameter_set_data) containing "profile ide" (138 in
the example shown in Fig. 55) representing the profiles
of the L parallax image, the R parallax image, and the 0
parallax image, and information about each "profile ide"
defined by the 3DV technique.
25 [0430]
Specifically, the Subset SPS of the 3DV technique
for parallax images includes, as the information shown
when "profile_ide" is 138, extension information for
parallax images (seq_parameter set_depth_extension), a
30 VUI information flag for parallax images indicating
whether VUI extension information for parallax images is
137
•
5
10
15
20
25
30
SP308971WOOO
contained (depth_vui_parameters_present_flag), and the
like. In a case where the VUI information flag for
parallax images indicates that the VUI extension
information for parallax images is contained, the VUI
extension information for parallax images
(depth_vui_parameters extension) is also written.
[0431]
In a case where the Subset SPS of the 3DV technique
for parallax images is referred to at the time of
decoding, or where the L parallax image, the R parallax
image, and the 0 parallax image are decoded, the
reference image is reset at the same time as decoding of
an lOR picture.
[0432]
Fig. 56 is a diagram showing an example description
of the extension information for parallax images shown in
Fig. 55.
[0433]
As shown in Fig. 56, the extension information for
parallax images contains information written like
extension information in the Subset SPS of the MVC
technique (seq_parameter_set_mvc_extension), and the view
IDs of the images corresponding to the respective
parallax images (ref_view_id).
[0434]
In Fig. 56, the information written like the
extension information in the Subset SPS of the MVC
technique is expanded and included in the extension
information for parallax images. Accordingly, the view
IDs (view_id) of the respective parallax images and the
view IDs of the images corresponding to the respective
138
SP308971WOOO
parallax images can be collectively written for each of
the parallax images. That is, information indicating the
number of parallax images (num_views_minus1) is written,
and a description can be provided so that the view IDs of
5 the same number of parallax images as the number
indicated in the information, and the view IDs of the
images corresponding to the parallax images are read out.
[0435]
In a case where the information written like the
10 extension information in the Subset SPS of the MVC
technique is not expanded but is contained in the
extension information for parallax images, on the other
hand, the information indicating the number of parallax
images is written together with the extension information
15 for parallax images, and a description need to be
provided so that the view IDs of the images corresponding
to the same number of parallax images as the number
indicated in the information can be read out. As a
result, the description of the information indicating the
20 number of parallax images overlaps the description for
reading out the information about the same number of
parallax images as the number indicated in the
information.
[0436]
25 Accordingly, in a case where the information
written like the extension information in the Subset SPS
of the MVC technique is expanded and is contained in the
extension information for parallax images, as shown in
Fig. 56, the data amount of the extension information for
30 parallax images can be made smaller than in a case where
the information is not expanded but is contained in the
139
SP308971WOOO
extension information for parallax images.
[0437]
Fig. 57 is a diagram showing an example description
of the VUI extension information for parallax images
5 shown in Fig. 55.
[0438]
As shown in Fig. 57, the VUI extension information
for parallax images is written like the VUI extension
information of the MVC technique
10 (mvc_vui_parameters__extension), except for the following
points. The VUI information for parallax images includes,
for each pixel of each parallax image, a location type
flag indicating whether the location type indicating the
type of location in the image corresponding to the
15 parallax image is contained (depth_loc_info_present flag),
and a pre-conversion size information flag indicating
whether pre-conversion size information indicating the
size of the parallax image prior to a resolution
conversion is contained (video src info_present_flag).
20 In a case where the location type flag indicates the
location type is contained, the VUI information for
parallax images also includes the location type. In a
case where the pre-conversion size information flag
indicates that the pre-conversion size information is
25 contained, the VUI information for parallax images also
includes the pre-conversion size information.
[0439]
The location type is formed with a top field
location type (depth_sample_loc type_top_field) and a
30 bottom field location type
(depth_sample loc_type_bottom_field). The top field and
140
•
5
SP308971WOOO
bottom field location types are written like the top
field location type (chroma_sample loc_type top_field)
and the bottom field location type
(chroma_sample_loc_type_bottom_field) contained in the
VUI extension information of the MVC technique.
[0440]
The pre-conversion size information is formed with
information indicating the number of macroblocks in the
transverse direction of the parallax image prior to the
10 resolution conversion (pic_width_in_mbs_minusl) and the
information indicating the number of macroblocks in the
vertical direction (pic_height_in_mbs_minusl), an aspect
ratio flag indicating whether aspect information
indicating the aspect ratio is contained
15 (aspect ratio info_present_flag), and the like. In a
case where the aspect ratio flag indicates that the
aspect information is contained, the pre-conversion size
information also contains the aspect information.
[0441]
20 The aspect information is formed with an aspect
ratio 10 that is an 10 unique to an aspect ratio
(aspect_ratio_idc) and the like. The aspect ratio 10 is
assigned to a predetermined aspect ratio, and is also
assigned to all undefined aspect ratios. In a case where
25 the aspect ratio 10 contained in the aspect information
is an aspect ratio 10 assigned to all undefined aspect
ratios (Extended_SAR), the aspect information also
contains the value in the transverse direction of the
parallax image (sar_width) and the value in the vertical
30 direction of the parallax image (sar_height) at the
aspect ratio prior to the resolution conversion.
141
• SP308971WOOO
[0442]
[Example Description of the NAL Header of 3DV Encoded
Data]
Fig. 58 is a diagram showing an example description
5 of the NAL header of the NAL unit of 3DV encoded data
containing 21 as the NAL unit type.
[0443]
As shown in Fig. 58, the NAL header of 3DV encoded
data contains a header extension information flag for
10 parallax images indicating whether the header extension
information for parallax images is contained
(depth_extension_flag), as the information shown when the
NAL unit type is 21. Here, the header extension
information for parallax images is written in the NAL
15 header of the 3DV encoded data of a parallax image.
[0444]
In a case where the header extension information
flag for parallax images indicates that the header
extension information for parallax images is contained,
20 the NAL header of the 3DV encoded data also contains the
header extension information for parallax images
(nal_unit_header_depth_extension). In a case where the
header extension information flag for parallax images
indicates that the header extension information for
25 parallax images is not contained, the NAL header of the
3DV encoded data also contains the header extension
information for the MVC technique
(nal_unit_header_mvc_extension) .
[0445]
30 Fig. 59 is a diagram showing an example description
of the header extension information for parallax images
142
• SP308971WOOO
shown in Fig. 58.
[0446]
As shown in Fig. 59, the header extension
information for parallax images is the same as the header
5 extension information for the MVC technique, except that
the view IDs of the images corresponding to the parallax
images (ref_view_id) are contained.
[0447]
[Example Description of the SEI of the 3DV Technique]
10 Fig. 60 is a diagram showing an example description
of the SEI of the 3DV technique shown in Fig. 54.
[0448]
As shown in Fig. 60, the SEI of the 3DV technique
includes a SEI message, like the SEI of the MVC technique.
15 [0449]
That is, the SEI of the 3DV technique contains an
operation point flag indicating whether to designate an
operation point (operation_point flag), and an overall
component flag indicating whether to apply the SEI
20 message to all the images and parallax images in the
access unit (all_view_components in_au_flag) in a case
where the operation point flag indicates that no
operation points are to be designated. Also, in a case
where the overall component flag indicates that the SEI
25 message is to be applied to not all the images and
parallax images in the access unit, the number of view
IDs of the images and parallax images to which the SEI
message is to be applied (num_view_components_minus1),
and the view IDs (sei_view_id) of those images and
30 parallax images are written.
[0450]
143
5
• SP308971WOOO
In a case where the operation point flag indicates
that operation points are to be designated, on the other
hand, the view IDs of the images and parallax images to
which the SEI message is to be applied among the
operation points (sei op_view_id), and the number of the
operation points (sei op_temporal id) are written. The
SEI message (sei rbsp) is then written. In the SEI of
the 3DV technique can contain two or more SEI messages.
[0451]
10 In the example shown in Fig. 54, the profiles of
the R image and the 0 image are the same. However, those
profiles may differ from each other.
[0452]
[Another Example Structure of a Bit Stream]
15 Fig. 61 is a diagram showing an example structure
of an access-unit bit stream in a case where the profiles
of the R image and the 0 image differ from each other in
Fig. 54.
[0453]
20 The structure of the bit stream shown in Fig. 61
differs from the structure of the bit stream shown in Fig.
54 in that a Subset SPS of the 3DV technique for images
is provided as well as the Subset SPS of the MVC
technique for images and the Subset SPS of the 3DV
25 technique for parallax images.
[0454]
The NAL unit of the Subset SPS of the MVC technique
for images (Subset SPSl) is a NAL unit of a Subset SPS
containing "profile idc" (128 in the example shown in Fig.
30 61) representing the profile of the R image among the
profiles defined by the MVC technique. The NAL unit of
144
,
5
10
15
20
25
30
SP308971WOOO
the Subset SPS of the 3DV technique for images (Subset
SPS2) is a NAL unit of a Subset SPS containing
"profile idc" (148 in the example shown in Fig. 61)
representing the profile of the 0 image among the
profiles defined as profiles for images by the 3DV
technique. The NAL unit of the Subset SPS for parallax
images (Subset SPS3) is a NAL unit of a Subset SPS
containing "profile idc" (138 in the example shown in Fig.
61) representing the profiles of the L parallax image,
the R parallax image, and the 0 parallax image among the
profiles defined as the profiles for parallax images by
the 3DV technique.
[0455]
[Example Description of the Subset SPS of the 3DV
Technique for Images]
Fig. 62 is a diagram showing an example description
of the Subset SPS of the 3DV technique for images shown
in Fig. 6l.
[0456]
As shown in Fig. 62, the Subset SPS of the 3DV
technique for images includes aSPS
(seq_parameter_set_data) containing "profile_idc" (148 in
the example shown in Fig. 62) representing the profile of
the 0 parallax image, and information about each
"profile idc" defined by the 3DV technique.
[0457]
Specifically, the Subset SPS of the 3DV technique
for images includes, as the information shown when
"profile_idc" is 148, extension information of the MVC
technique (seq_parameter set_mvc_extension) and a MVCtechnique
VUI information flag indicating whether VUI
145
,
5
10
SP308971WOOO
extension information of the MVC technique is contained
(mvc_vui_parameters_present_flag), like the Subset SPS of
the MVC technique. In a case where the MVC-technique VUI
information flag indicates that the VUI extension
information of the MVC technique is contained, the VUI
extension information of the MVC technique
(mvc_vui_parameters__extension) is also written. As the
information shown when "profile idc" is 138, the same
information as that in Fig. 55 is written.
[0458]
25
15
20
The 3DV technique may be a technique for encoding
images for display by a multi-view technique compliant to
the HEVC technique, instead of the AVC technique or the
MVC technique. The bit stream in this case will be
described below. In this specification, the HEVC
technique is based on the description in HEVC Working
Draft (Thomas Wiegand, Woo-jin Han, Benjamin Bross, JensRainer
Ohm, and Gary J. Sullivian, "WD3: Working Draft 3
of High-Efficiency Video Coding", JCTVc-E603 d5 (version
5), May 20, 2011) of JCTVC (Joint Collaborative Team on
Video Coding) .
[0459]
<7. Bit Stream in a Case Where the 3DV Technique is
Compliant with the HEVC Technique>
[Example Structure of a Bit Stream]
Fig. 63 is a diagram showing an example structure
of an access-unit bit stream in a case where the 3DV
technique is compliant with the HEVC technique.
[0460]
30 In the example shown in Fig. 63, an L viewpoint
image, an R viewpoint image, and an 0 viewpoint image,
146
t
5
10
15
20
25
30
SP308971WOOO
which are the same as those in the example shown in Fig.
54. The L image is encoded by the Ave technique, and the
L parallax image, the R viewpoint image, and the 0
viewpoint image are encoded by the 3DV technique.
[0461]
The bit stream shown in Fig. 63 differs from the
bit stream shown in Fig. 54 in that the Subset SPS of the
3DV technique for images (Subset SPSl) and the Subset SPS
of the 3DV technique for parallax images (Subset SPS2)
can be written in the SPS, and NAL headers containing
different NAL unit types from each other are attached to
the NAL units of the SEI of the HEve technique and the
SEI of the 3DV technique.
[0462]
In the bit stream shown in Fig. 63, the Subset SPS
of the 3DV technique for images and the Subset SPS of the
3DV technique for parallax images can be written only in
the SPS, or can be written separately from the SPS, or
can be written in the SPS and separately from the SPS.
Writing the Subset SPS of the 3DV technique for images
and the Subset SPS of the 3DV technique for parallax
images separately from the SPS is preferred in a case
where HEve encoded data and 3DV encoded data are
generated as different ESs (Elementary Streams) in
compliance with the BD standard.
[0463]
Also, in the bit stream shown in Fig. 63, the NAL
unit of the SEI of the HEve technique and the NAL unit
contained in the NAL header attached to the NAL unit of
the SEI of the 3DV technique are of different types from
each other. Accordingly, the NAL units of the SEI of the
147
, SP308971WOOO
HEVC technique and the SEI of the 3DV technique can be
easily extracted at the time of decoding.
[0464]
[Example Description of the SPS]
5 Fig. 64 is a diagram showing an example description
of the SPS shown in Fig. 63.
[0465]
The SPS description shown in Fig. 64 is the same as
the SPS description of the HEVC technique, except that a
10 Subset SPS information flag indicating whether Subset SPS
information is contained (subset seq_present_flag), and
the Subset SPS information is written in a case where the
Subset SPS information flag indicates that the Subset SPS
information is contained.
15 [0466]
As shown in Fig. 64, the Subset SPS information
contains the number of Subset SPSs (num_subset_seq) and a
Subset SPS flag indicating whether a Subset SPS is
contained (subset seq_info_present flag). In a case
20 where the Subset SPS flag indicates that a Subset SPS is
contained, the Subset SPS information also contains the
Subset SPS (subset seq_parameter_set_data).
[0467]
As described above, the number of Subset SPSs is
25 written as the Subset SPS information. Accordingly, it
is possible to recognize whether a Subset SPS exists
simply by reading the SPS description at the time of
decoding. Also, as the Subset SPS flag is written, the
Subset SPSs are not written in the SPS and can be written
30 separately from the SPS. Accordingly, overlapping of
Subset SPS descriptions can be prevented.
148
SP308971WOOO
[0468]
[Example Description of the Subset SPS]
Fig. 65 is a diagram showing an example description
of the Subset SPS shown in Fig. 64.
5 [0469]
The Subset SPS description shown in Fig. 65 is the
same as the SPS description of the HEVC technique, except
that the Subset SPS description includes the number of
view IDs of images and parallax images to which the
10 Subset SPS is applied (num_subset_seq_views), a parallax
image flag indicating whether the Subset SPS is applied
to a parallax image (depth_extension_flag), and an
override flag indicating whether the SPS is overridden at
the time of application of the Subset SPS
15 (seq_param_override flag).
[0470]
In a case where the parallax image flag indicates
that the Subset SPS is applied to a parallax image, the
Subset SPS includes the view ID of the image
20 corresponding to the subject parallax image (ref view id).
In a case where the override flag indicates that the SPS
is overridden at the time of application of the Subset
SPS, the Subset SPS includes information indicating
profiles (subset_seq_profile_idc), like the SPS.
25 [0471]
Also, in the Subset SPS description, the same items
as those in the SPS description include information
indicating whether the image and parallax image to which
the Subset SPS is applied are cropped
30 (subset seq_frame cropping_flag), and a VUI information
flag (subset seq_vui_parameters_present_flag) indicating
149
•
5
10
SP308971WOOO
whether VUI information is contained
(subset_seq_vui_parameters), for example. In a case
where the VUI information flag indicates that the VUI
information is contained, the VUI information is also
written, as in the SPS. In a case where the VUI
information flag indicates that the VUI information is
not contained, on the other hand, the VUI information is
not written as in the SPS. In this case, the VUI
information in the SPS is used as the VUI information.
[0472]
[Example Description of the VUI information in the Subset
SPS]
Fig. 66 is a diagram showing an example description
of the VUI information in the Subset SPS shown in Fig. 65.
15 [0473]
The VUI information description in the Subset SPS
shown in Fig. 66 is the same as the VUI information
description in the SPS of the HEVC technique, except that
a pre-conversion size information flag indicating whether
20 pre-conversion size information about a subject image
(video src info_present flag) is written.
[0474]
In a case where the pre-conversion size information
flag indicates that the pre-conversion size information
25 of the subject image is contained, the pre-conversion
size information is written in the VUI information shown
in Fig. 66. That is, information indicating the number
of macroblocks in the transverse direction of the subject
image prior to the resolution conversion
30 (src_pic_width_in_mbs_minus1) and the information
indicating the number of macroblocks in the vertical
150
5
SP308971WOOO
direction (src_pic_height_in_mbs_minus1), an aspect ratio
flag indicating whether aspect information indicating the
aspect ratio is contained
(src_aspect ratio_info_present flag), and the like are
written.
[0475]
In a case where the aspect ratio flag indicates
that the aspect information is contained, the preconversion
size information also contains the aspect
10 information formed with an aspect ratio 10
(src_aspect_ratio_idc) and the like. In a case where the
aspect ratio 10 contained in the aspect information is an
aspect ratio 10 assigned to all undefined aspect ratios
(Extended_SAR), the aspect information also contains the
15 value in the transverse direction of the parallax image
(sar_width) and the value in the vertical direction of
the subject image (sar_height) at the aspect ratio prior
to the resolution conversion.
[0476]
20 Alternatively, only the different part of the VUI
information of the Subset SPS shown in Fig. 66 from the
VUI information of the SPS may be written. In this case,
the information contained in the VUI information of the
SPS is used in place of the unwritten information in the
25 VUI information of the Subset SPS.
[0477]
[Example Description of SEI]
Fig. 67 is a diagram showing an example description
of the SEI of the 3DV technique shown in Fig. 63.
30 [0478]
The SEI description of the 3DV technique shown in
151
5
SP308971WOOO
Fig. 67 is the same as the SEI message description of a
SEI message type, except that the type of SEI message
(nesting_type), an overall component flag
(all_view_components in_au_flag), and the like are
written.
[0479]
The type of SEI message may be a MVC SEI message, a
3DV SEI message, a SEI message defined by a user, or the
like. In a case where the overall component flag
10 indicates that the SEI message is applied to not all the
images and parallax images in the access unit, the number
of view IDs of the images and parallax images to which
the SEI message is applied (num_view_components_minusl),
and the view IDs (nesting_sei_view_id) of those images
15 and parallax images are also written in the SEI of the
3DV technique.
[0480]
Based on decoded 3D image data, the above described
reproducing device 301 may generate 3D information that
20 contains image information, compatibility information,
inter-viewpoint distance information, parallax image
information, information for identifying the multiplexing
pattern of an image, information for identifying the
multiplexing pattern of a parallax image, and the like.
25 [0481]
<8. Sixth Embodiment>
[Computer]
The above described series of operations can be
performed by hardware or can be performed by software.
30 In this case, the operations may be realized by a
computer as shown in Fig. 68, for example.
152
5
, SP308971WOOO
[0482]
In Fig. 68, the CPU (Central Processing Unit) 601
of a computer 600 performs various kinds of operations in
accordance with a program stored in a ROM (Read Only
Memory) 602, or a program loaded from a storage unit 613
into a RAM (Random Access Memory) 603. Necessary data
for the CPU 601 to perform various kinds of operations
and the like are also stored in the RAM 603, where
necessary.
10 [0483]
The CPU 601, the ROM 602, and the RAM 603 are
connected to one another via a bus 604. An input/output
interface 610 is also connected to the bus 604.
[0484]
15 The input/output interface 610 has the following
components connected thereto: an input unit 611 formed
with a keyboard, a mouse, or the like; an output unit 612
formed with a display such as a CRT (Cathode Ray Tube) or
a LCD (Liquid Crystal Display), and a speaker; the
20 storage unit 613 formed with a hard disk or the like; and
a communication unit 614 formed with a modem. The
communication unit 614 performs communications via
networks including the Internet.
[0485]
25 A drive 615 is also connected to the input/output
interface 610 where necessary, and a removable medium 621
such as a magnetic disk, an optical disk, a
magnetooptical disk, or a semiconductor memory is mounted
on the drive as appropriate. A computer program read
30 from such a removable disk is installed in the storage
unit 613 where necessary.
153
SP308971WOOO
[0486]
In a case where the above described series of
operations is performed by software, the program forming
the software is installed from a network or a recording
5 medium.
[0487]
As shown in Fig. 68, this recording medium is
formed with the removable medium 621 that is distributed
for delivering the program to users separately from the
10 device, such as a magnetic disk (including a flexible
disk), an optical disk (including a CD-ROM (Compact Disc
- Read Only Memory) or a DVD (Digital Versatile Disc)), a
magnetooptical disk (including an MD (Mini Disc)), or a
semiconductor memory, which has the program recorded
15 thereon. Alternatively, the recording medium may be
formed with the ROM 602 having the program recorded
therein or a hard disk included in the storage unit 613.
Such a recording medium is incorporated beforehand into
the device prior to the delivery to users.
20 [0488]
The program to be executed by the computer may be a
program for carrying out processes in chronological order
in accordance with the sequence described in this
specification, or a program for carrying out processes in
25 parallel or whenever necessary such as in response to a
call.
[0489]
In this specification, the step written in the
program to be recorded in a recording medium includes
30 operations to be performed in parallel or independently
of one another if not necessarily in chronological order,
154
SP308971WOOO
as well as operations to be performed in chronological
order in accordance with the sequence described herein.
[0490]
<9. Seventh Embodiment>
5 [Example Structure of a Television Apparatus]
Fig. 69 schematically shows an example structure of
a television apparatus to which the present technique is
applied. The television apparatus 900 includes an
antenna 901, a tuner 902, a demultiplexer 903, a decoder
10 904, a video signal processing unit 905, a display unit
906, an audio signal processing unit 907, a speaker 908,
and an external interface unit 909. The television
apparatus 900 further includes a control unit 910, a user
interface unit 911, and the like.
15 [0491]
The tuner 902 selects a desired channel from
broadcast wave signals received at the antenna 901, and
performs demodulation. The resultant encoded bit stream
is output to the demultiplexer 903.
20 [0492]
The demultiplexer 903 extracts the video and audio
packets of the show to be viewed from the encoded bit
stream, and outputs the data of the extracted packets to
the decoder 904. The demultiplexer 903 also supplies a
25 packet of data such as EPG (Electronic Program Guide) to
the control unit 910. Where scrambling is performed, the
demultiplexer or the like cancels the scrambling.
[0493]
The decoder 904 performs a packet decoding
30 operation, and outputs the video data generated through
the decoding operation to the video signal processing
155
SP308971WOOO
unit 905, and the audio data to the audio signal
processing unit 907.
[0494]
The video signal processing unit 905 subjects the
5 video data to a noise removal and video processing or the
like in accordance with user settings. The video signal
processing unit 905 generates video data of the show to
be displayed on the display unit 906, or generates image
data or the like through an operation based on an
10 application supplied via a network. The video signal
processing unit 905 also generates video data for
displaying a menu screen or the like for item selection,
and superimposes the generated video data on the video
data of the show. Based on the video data generated in
15 this manner, the video signal processing unit 905
generates a drive signal to drive the display unit 906.
[0495]
Based on the drive signal from the video signal
processing unit 905, the display unit 906 drives a
20 display device (a liquid crystal display element, for
example) to display the video of the show.
[0496]
The audio signal processing unit 907 subjects the
audio data to predetermined processing such as a noise
25 removal, and performs a D/A conversion operation and an
amplification operation on the processed audio data. The
resultant audio data is supplied as an audio output to
the speaker 908.
[0497]
30 The external interface unit 909 is an interface for
a connection with an external device or a network, and
156
• SP308971WOOO
transmits and receives data such as video data and audio
data in accordance with the HDMI standard or the like.
[0498]
The user interface unit 911 is connected to the
5 control unit 910. The user interface unit 911 is formed
with operation switches, a remote control signal
reception unit, and the like, and supplies an operating
signal according to a user operation to the control unit
910.
10 [0499]
The control unit 910 is formed with a CPU (Central
Processing Unit), a memory, and the like. The memory
stores the program to be executed by the CPU, various
kinds of data necessary for the CPU to perform operations,
15 the EPG data, data obtained via a network, and the like.
The program stored in the memory is read and executed by
the CPU at a predetermined time such as the time of
activation of the television apparatus 900. The CPU
executes the program to control the respective components
20 so that the television apparatus 900 operates in
accordance with a user operation.
[0500]
In the television apparatus 900, a bus 912 is
provided for connecting the tuner 902, the demultiplexer
25 903, the video signal processing unit 905, the audio
signal processing unit 907, the external interface unit
909, and the like, to the control unit 910.
[0501]
In the television apparatus having such a structure,
30 the function of an image processing device (an image
processing method) of the present invention is provided
157
• SP308971WOOO
in the external I/F unit 909. Thus, image data in such a
format as to generate multi-view images can be correctly
transmitted.
[0502]
5 <10. Eighth Embodiment>
[Example Structure of a Portable Telephone Device]
Fig. 70 schematically shows an example structure of
a portable telephone device to which the present
technique is applied. The portable telephone device 920
10 includes a communication unit 922, an audio codec 923, a
camera unit 926, an image processing unit 927, a
demultiplexing unit 928, a recording/reproducing unit 929,
a display unit 930, and a control unit 931. Those
components are connected to one another via a bus 933.
15 [0503]
Also, an antenna 921 is connected to the
communication unit 922, and a speaker 924 and a
microphone 925 are connected to the audio codec 923.
Further, an operation unit 932 is connected to the
20 control unit 931.
[0504]
The portable telephone device 920 performs various
kinds of operations such as transmission and reception of
audio signals, transmission and reception of electronic
25 mail and image data, image capturing, and data recording,
in various kinds of modes such as an audio communication
mode and a data communication mode.
[0505]
In the audio communication mode, an audio signal
30 generated at the microphone 925 is converted into audio
data, and the data is compressed at the audio codec 923.
158
SP308971WOOO
The compressed data is supplied to the communication unit
922. The communication unit 922 performs a modulation
operation, a frequency conversion operation, and the like
on the audio data, to generate a transmission signal.
5 The communication unit 922 also supplies the transmission
signal to the antenna 921, and the transmission signal is
transmitted to a base station (not shown). The
communication unit 922 also amplifies a signal received
at the antenna 921, and performs a frequency conversion
10 operation, a demodulation operation, and the like. The
resultant audio data is supplied to the audio codec 923.
The audio codec 923 decompresses audio data, and converts
the audio data into an analog audio signal to be output
to the speaker 924.
15 [0506]
In a case where mail transmission is performed in
the data communication mode, the control unit 931
receives text data that is input through an operation by
the operation unit 932, and the input text is displayed
20 on the display unit 930. In accordance with a user
instruction or the like through the operation unit 932,
the control unit 931 generates and supplies mail data to
the communication unit 922. The communication unit 922
performs a modulation operation, a frequency conversion
25 operation, and the like on the mail data, and transmits
the resultant transmission signal from the antenna 921.
The communication unit 922 also amplifies a signal
received at the antenna 921, and performs a frequency
conversion operation, a demodulation operation, and the
30 like, to restore the mail data. This mail data is
supplied to the display unit 930, and the mail content is
159
• SP308971WOOO
displayed.
[0507]
The portable telephone device 920 can cause the
recording/reproducing unit 929 to store received mail
5 data into a storage medium. The storage medium is a
rewritable storage medium. For example, the storage
medium may be a semiconductor memory such as a RAM or an
internal flash memory, a hard disk, or a removable medium
such as a magnetic disk, a magnetooptical disk, an
10 optical disk, a USB memory, or a memory card.
[0508]
In a case where image data is transmitted in the
data communication mode, image data generated at the
camera unit 926 is supplied to the image processing unit
15 927. The image processing unit 927 performs an encoding
operation on the image data, to generate encoded data.
[0509]
The demultiplexing unit 928 multiplexes the encoded
data generated at the image processing unit 927 and the
20 audio data supplied from the audio codec 923 by a
predetermined technique, and supplies the multiplexed
data to the communication unit 922. The communication
unit 922 performs a modulation operation, a frequency
conversion operation, and the like on the multiplexed
25 data, and transmits the resultant transmission signal
from the antenna 921. The communication unit 922 also
amplifies a signal received at the antenna 921, and
performs a frequency conversion operation, a demodulation
operation, and the like, to restore the multiplexed data.
30 This multiplexed data is supplied to the demultiplexing
unit 928. The demultiplexing unit 928 divides the
160
•
5
10
SP308971WOOO
multiplexed data, and supplies the encoded data to the
image processing unit 927, and the audio data to the
audio codec 923.
The image processing unit 927 performs a decoding
operation on the encoded data, to generate image data.
The image processing unit 927 also supplies the image
data to the display unit 930 to display the received
image, or transmits the image data in accordance with the
HDMI standard or the like. The audio codec 923 converts
the audio data into an analog audio signal, and supplies
the analog audio signal to the speaker 924, so that the
received sound is output.
[0510]
In the portable telephone device having such a
15 structure, the function of an image processing device (an
image processing method) of the present invention is
provided in the image processing unit 927. Thus, image
data in such a format as to generate multi-view images
can be correctly transmitted.
20 [0511]
<11. Ninth Embodiment>
[Example Structure of a Recording/Reproducing Device]
Fig. 71 schematically shows an example structure of
a recording/reproducing device to which the present
25 technique is applied. The recording/reproducing device
940 records the audio data and video data of a received
broadcast show on a recording medium, for example, and
provides the recorded data to a user at the time
according to an instruction from the user. The
30 recording/reproducing device 940 can also obtain audio
data and video data from another device, for example, and
161
•
5
SP308971WOOO
record the data on a recording medium. Further, the
recording/reproducing device 940 decodes and outputs
audio data and video data recorded on a recording medium,
so that a monitor device or the like can display images
and outputs sound.
[0512]
The recording/reproducing device 940 includes a
tuner 941, an external interface unit 942, an encoder 943,
a HOD (Hard Disk Drive) unit 944, a disk drive 945, a
10 selector 946, a decoder 947, an OSD (On-Screen Display)
unit 948, a control unit 949, and a user interface unit
950.
[0513]
The tuner 941 selects a desired channel from
15 broadcast signals received at an antenna (not shown).
The tuner 941 demodulates the received signal of the
desired channel, and outputs the resultant encoded bit
stream to the selector 946.
[0514]
20 The external interface unit 942 is formed with at
least one of an IEEE1394 interface, a network interface
unit, a USB interface, a flash memory interface, an HDMI,
and the like. The external interface unit 942 is an
interface for a connection with an external device, a
25 network, a memory card, or the like, and receives data
such as video data and audio data to be recorded, and the
like.
[0515]
The encoder 943 performs predetermined encoding on
30 video data and audio data that are supplied from the
external interface unit 942 and are not encoded, and
162
• SP308971WOOO
outputs an encoded bit stream to the selector 946.
[0516]
The HOD unit 944 records content data such as
videos and sound, various kinds of programs, other data,
5 and the like on an internal hard disk, and reads the data
from the hard disk at the time of reproduction or the
like.
[0517]
The disk drive 945 performs signal recording and
10 reproduction on a mounted optical disk. The optical disk
may be a DVD disk (such as a DVD-Video, a DVD-RAM, a DVDR,
a DVD-RW, a DVD+R, or a DVD+RW) or a Blu-ray disk, for
example.
[0518]
15 The selector 946 selects an encoded bit stream from
the tuner 941 or the encoder 943 at the time of video and
audio recording, and supplies the encoded bit stream to
either the HOD unit 944 or the disk drive 945. The
selector 946 also supplies an encoded bit stream output
20 from the HOD unit 944 or the disk drive 945 to the
decoder 947 at the time of video and audio reproduction.
[0519]
The decoder 947 performs a decoding operation on an
encoded bit stream. The decoder 947 supplies the video
25 data generated by performing the decoding to the OSD unit
948. The decoder 947 also outputs the audio data
generated by performing the decoding.
[0520]
The OSD unit 948 generates video data for
30 displaying a menu screen or the like for item selection,
and superimposes the video data on video data output from
163
• SP308971WOOO
the decoder 947.
[0521]
The user interface unit 950 is connected to the
control unit 949. The user interface unit 950 is formed
5 with operation switches, a remote control signal
reception unit, and the like, and supplies an operating
signal according to a user operation to the control unit
949.
[0522]
10 The control unit 949 is formed with a CPU, a memory,
and the like. The memory stores the program to be
executed by the CPU, and various kinds of data necessary
for the CPU to perform operations. The program stored in
the memory is read and executed by the CPU at a
15 predetermined time such as the time of activation of the
recording/reproducing device 940. The CPU executes the
program to control the respective components so that the
recording/reproducing device 940 operates in accordance
with a user operation.
20 [0523]
In the recording/reproducing device having such a
structure, the function of an image processing device (an
image processing method) of the present invention is
provided in the external I/F unit 942. Thus, image data
25 in such a format as to generate multi-view images can be
correctly transmitted.
[0524]
<12. Tenth Embodiment>
[Example Structure of an Imaging Device]
30 Fig. 72 schematically shows an example structure of
an imaging apparatus to which the present technique is
164
• SP308971WOOO
applied. An imaging device 960 captures an image of an
object, and causes a display unit to display the image of
the object or records the image as image data on a
recording medium.
5 [0525]
The imaging device 960 includes an optical block
961, an imaging unit 962, a camera signal processing unit
963, an image data processing unit 964, a display unit
965, an external interface unit 966, a memory unit 967, a
10 media drive 968, an OSD unit 969, and a control unit 970.
A user interface unit 971 is connected to the control
unit 970. Further, the image data processing unit 964,
the external interface unit 966, the memory unit 967, the
media drive 968, the OSD unit 969, the control unit 970,
15 and the like are connected via a bus 972.
[0526]
The optical block 961 is formed with a focus lens,
a diaphragm, and the like. The optical block 961 forms
an optical image of an object on the imaging surface of
20 the imaging unit 962. Formed with a CCD or a CMOS image
sensor, the imaging unit 962 generates an electrical
signal in accordance with the optical image through a
photoelectric conversion, and supplies the electrical
signal to the camera signal processing unit 963.
25 [0527]
The camera signal processing unit 963 performs
various kinds of camera signal processing such as a knee
correction, a gamma correction, and a color correction on
the electrical signal supplied from the imaging unit 962.
30 The camera signal processing unit 963 supplies the image
data subjected to the camera signal processing, to the
165
SP308971WOOO
image data processing unit 964.
[0528]
The image data processing unit 964 performs an
encoding operation on the image data supplied from the
5 camera signal processing unit 963. The image data
processing unit 964 supplies the encoded data generated
by performing the encoding operation, to the external
interface unit 966 and the media drive 968. The image
data processing unit 964 also performs a decoding
10 operation on the encoded data supplied from the external
interface unit 966 and the media drive 968. The image
data processing unit 964 supplies the image data
generated by performing the decoding operation to the
display unit 965. The image data processing unit 964
15 also performs an operation to supply the image data
supplied from the camera signal processing unit 963 to
the display unit 965, or superimposes display data
obtained from the OSD unit 969 on the image data and
supplies the image data to the display unit 965.
20 [0529]
The OSD unit 969 generates a menu screen formed
with sYmbols, characters, or figures, or display data
such as icons, and outputs such data to the image data
processing unit 964.
25 [0530]
The external interface unit 966 is formed with a
USB input/output terminal or a HDMI input/output terminal,
for example, and is connected to a printer when image
printing is performed. A drive is also connected to the
30 external interface unit 966 where necessary, and a
removable medium such as a magnetic disk or an optical
166
t
5
10
15
20
25
30
SP308971WOOO
disk is mounted on the drive as appropriate. A computer
program read from such a removable disk is installed
where necessary. Further, the external interface unit
966 includes a network interface connected to a
predetermined network such as a LAN or the Internet. The
control unit 970 reads encoded data from the memory unit
967 in accordance with an instruction from the user
interface unit 971, for example, and can supply the
encoded data from the external interface unit 966 to
another device connected thereto via a network. The
control unit 970 can also obtain, via the external
interface unit 966, encoded data or image data supplied
from another device via a network, and supply the encoded
data or image data to the image data processing unit 964.
[0531]
A recording medium to be driven by the media drive
968 may be a readable/rewritable removable disk such as a
magnetic disk, a magnetooptical disk, an optical disk, or
a semiconductor memory. The recording medium may be any
type of removable medium, and may be a tape device, a
disk, or a memory card. The recording medium may of
course be a non-contact IC card or the like.
[0532]
Alternatively, the media drive 968 and a recording
medium may be integrated, and may be formed with an
immobile storage medium such as an internal hard disk
drive or a SSD (Solid State Drive).
[0533]
The control unit 970 is formed with a CPU, a memory,
and the like. The memory stores the program to be
executed by the CPU, various kinds of data necessary for
167
168
In the imaging device having such a structure, the
function of an image processing device (an image
10 processing method) of the present invention is provided
in the external IfF unit 966. Thus, image data in such a
format as to generate multi-view images can be correctly
transmitted.
[0535]
15 In this specification, a system means an entire
apparatus formed with more than one device.
[0536]
Also, in the above described examples, any
structure described as one device (or one processing
20 unit) may be divided into two or more devices (or
processing units). Conversely, any structure described
as two or more devices (or processing units) may be
combined to form one device (or one processing unit) .
Also, it is of course possible to add a structure other
25 than the above described ones to the structure of any of
the devices (or any of the processing units). Further,
as long as the structure and function of the entire
system remain the same, part of the structure of a device
(or a processing unit) may be incorporated into another
30 device (or another processing unit). That is,
embodiments of the present technique are not limited to
SP308971WOOO
the CPU to perform operations, and the like. The program
stored in the memory is read and executed by the CPU at a
predetermined time such as the time of activation of the
imaging device 960. The CPU executes the program to
control the respective components so that the imaging
device 960 operates in accordance with a user operation.
[0534]
5
, SP308971WOOO
the above described embodiments, and various
modifications may be made to them without departing from
the scope of the technique.
5 REFERENCE SIGNS LIST
[0537]
300 Transmission system, 301 Reproducing device,
302 Display device, 303 HDMI cable, 311 Decoder, 312 Mode
determining unit, 313 3D information generating unit, 314
10 Communication unit, 321 Communication unit, 322 Mode
determination unit, 323 Synchronization setting unit, 324
Synchronization control unit, 325 Image generating unit,
326 Display unit, 411 Receiving-end information acquiring
unit, 412 Synchronization setting unit, 421 Receiving-end
15 information providing unit

• SP308971WOOO
CLAIMS
1. An image processing device comprising:
a mode determining unit configured to determine a
5 compatible mode indicating compatibility with an image
data format by referring to a format of stereoscopic
image data to be used for generating a multi-view image;
a generating unit configured to generate
compatibility information as 3D information, the
10 compatibility information indicating the compatible mode
determined by the mode determining unit; and
a transmitting unit configured to transmit the
stereoscopic image data and the 3D information generated
by the generating unit to a transmission destination of
15 the stereoscopic image data.
2. The image processing device according to claim 1,
wherein the generating unit generates format information
as the 3D information, the format information indicating
20 the format of the stereoscopic image data.
3. The image processing device according to claim 2,
wherein the generating unit generates image multiplexing
information and parallax image multiplexing information
25 as the 3D information, the image multiplexing information
indicating a multiplexing technique for image data in the
stereoscopic image data, the parallax image multiplexing
information indicating a multiplexing technique for
parallax image data in the stereoscopic image data.
30
4. The image processing device according to claim 2,
170
,
5
10
15
20
25
30
SP308971WOOO
wherein the generating unit generates identification
information as the 3D information, the identification
information being for identifying a multiplexing pattern
of image data in the stereoscopic image data or a
multiplexing pattern of parallax image data in the
stereoscopic image data.
5. The image processing device according to claim 2,
wherein the transmitting unit puts the 3D information in
VSIFPC (Vendor Specific InfoFrame Packet Contents) of
HDMI (High Definition Multimedia Interface) standard, and
transmits the 3D information.
6. The image processing device according to claim 5,
wherein the transmitting unit puts the 3D information in
a free space in a region of the VSIFPC, and transmits the
3D information, the region of the VSIFPC being designed
to have information about 3D image data written therein.
7. The image processing device according to claim 2,
wherein the compatible mode includes mono compatible mode
indicating compatibility with conventional twodimensional
image data, frame packing compatible mode
indicating compatibility with conventional frame packing
3D image data, and stereoscopic compatible mode
indicating compatibility with conventional stereoscopic
3D image data.
8. The image processing device according to claim 1,
wherein the transmitting unit transmits data of one frame
of the stereoscopic image data, using an active video
171
• SP308971WOOO
interval equivalent to a plurality of cycles of a
vertical synchronization signal.
9. The image processing device according to claim 8,
5 further comprising:
an interval setting unit configured to set
irregular cycles to the vertical synchronization signal,
wherein the transmitting unit transmits data of one
frame of the stereoscopic image data, using the active
10 video interval equivalent to the cycles set by the
interval setting unit.
10. The image processing device according to claim 1,
further comprising:
15 a capability information acquiring unit configured
to acquire capability information from a device at the
transmission destination of the stereoscopic image data,
the capability information indicating processing
capability of the device; and
20 a synchronization setting unit configured to set a
synchronization method for the stereoscopic image data,
based on the processing capability of the device
indicated by the capability information acquired by the
capability information acquiring unit,
25 wherein the generating unit generates
synchronization information as the 3D information, the
synchronization information indicating the
synchronization method set by the synchronization setting
unit.
30
11. An image processing method to be performed by an
172
•
5
10
15
SP308971WOOO
image processing device, comprising:
determining a compatible mode indicating
compatibility with an image data format by referring to a
format of stereoscopic image data to be used for
generating a multi-view image, the determining being
performed by a mode determining unit;
generating compatibility information as 3D
information, the compatibility information indicating the
compatible mode determined by the mode determining unit,
the generating being performed by a generating unit; and
transmitting the stereoscopic image data and the 3D
information generated by the generating unit to a
transmission destination of the stereoscopic image data,
the transmitting being performed by a transmitting unit.
12. An image processing device comprising:
a receiving unit configured to receive stereoscopic
image data to be used for generating a multi-view image,
and receive a compatible mode as 3D information, the
20 compatible mode indicating compatibility of a format of
the stereoscopic image data with an image data format;
a mode determining unit configured to determine the
compatible mode of the stereoscopic image data received
by the receiving unit, based on the 3D information
25 received by the receiving unit; and
a synchronization control unit configured to
control synchronization of processing of the stereoscopic
image data received by the receiving unit, based on the
compatible mode determined by the mode determining unit.
30
13. The image processing device according to claim 12,
173
SP308971WOOO
wherein the receiving unit receives format information as
the 3D information, the format information indicating the
format of the stereoscopic image data.
5 14. The image processing device according to claim 13,
wherein the receiving unit generates image multiplexing
information and parallax image multiplexing information
as the 3D information, the image multiplexing information
indicating a multiple~ing technique for image data in the
10 stereoscopic image data, the parallax image multiplexing
information indicating a multiplexing technique for
parallax image data in the stereoscopic image data.
15. The image processing device according to claim 13,
15 wherein the receiving unit receives identification
information as the 3D information, the identification
information being for identifying a multiplexing pattern
of image data in the stereoscopic image data or a
multiplexing pattern of parallax image data in the
20 stereoscopic image data.
16. The image processing device according to claim 13,
wherein the receiving unit receives the 3D information
that is written in VSIFPC (Vendor Specific InfoFrame
25 Packet Contents) of HDMI (High Definition Multimedia
Interface) standard and is transmitted.
17. The image processing device according to claim 16,
wherein the receiving unit receives the 3D information
30 that is written in a free space in a region of the VSIFPC
and is transmitted, the region of the VSIFPC being
174
• SP308971WOOO
designed to have information about 3D image data written
therein.
18. The image processing device according to claim 13,
5 wherein the compatible mode includes mono compatible mode
indicating compatibility with conventional twodimensional
image data, frame packing compatible mode
indicating compatibility with conventional frame packing
3D image data, and stereoscopic compatible mode
10 indicating compatibility with conventional stereoscopic
3D image data.
19. The image processing device according to claim 12,
wherein the receiving unit receives data of one frame of
15 the stereoscopic image data transmitted using an active
video interval equivalent to a plurality of cycles of a
vertical synchronization signal.
20. The image processing device according to claim 19,
20 wherein the receiving unit receives data of one frame of
the stereoscopic image data transmitted using the active
video interval equivalent to the plurality of cycles of
the vertical synchronization signal, the cycles of the
vertical synchronization signal being irregular.
25
21. The image processing device according to claim 12,
further comprising:
a capability information providing unit configured
to provide capability information indicating processing
30 capability of the image processing device to a device as
a transmitter of the stereoscopic image data.
175
10
• SP308971WOOO
-
22. An image processing method to be performed by an
image processing device, comprising:
receiving stereoscopic image data to be used for
5 generating a multi-view image, and receiving a compatible
mode as 3D information, the compatible mode indicating
compatibility of a format of the stereoscopic image data
with an image data format, the receiving being performed
by a receiving unit;
determining the compatible mode of the stereoscopic
image data received by the receiving unit, based on the
3D information received by the receiving unit, the
determining being performed by a mode determining unit;
and
15 controlling synchronization of processing of the
stereoscopic image data received by the receiving unit,
based on the compatible mode determined by the mode
determining unit, the controlling being performed by a
synchronization control unit.
20
Dated this February 25,2013
176
b~J.L1~~
DEEPAKKUMAR
OF REMFRY & SAGAR