FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENTS RULES, 2003
COMPLETE SPECIFICATION
(See section 10, rule 13)
“IMAGE DATA TRANSMISSION DEVICE, IMAGE DATA TRANSMISSION METHOD, AND IMAGE DATA RECEIVING DEVICE”
SONY CORPORATION, of 1-7-1, Konan, Minato-ku, Tokyo
1080075, Japan
The following specification particularly describes the invention and the manner in
which it is to be performed.
2
DESCRIPTION
Title of Invention: IMAGE DATA TRANSMISSION DEVICE, IMAGE
DATA TRANSMISSION METHOD, AND IMAGE DATA RECEPTION DEVICE
Technical Field
[0001]
The present technology relates to an image data
transmission device, an image data transmission method, and
an image data reception device, and particularly to an image
data transmission device and the like which transmit image
data for displaying stereoscopic images.
Background Art
[0002]
In the related art, H.264/AVC (Advanced Video Coding)
is known as a coding method of moving images (refer to NPL
1). In addition, H.264/MVC (Multi-view Video Coding) is
known as an extension method of H.264/AVC (refer to NPL 2).
The MVC employs a structure in which image data of multiviews
is collectively coded. In the MVC, image data of
multi-views is coded as image data of a single base view and
image data of one or more non-base views.
[0003]
In addition, H.264/SVC (Scalable Video Coding) is also
known as an extension method of H.264/AVC (refer to NPL 3).
The SVC is a technique of hierarchically coding an image.
3
In the SVC, a moving image is divided into a base layer (the
lowest layer) having image data which is required to decode
a moving image so as to have minimum quality and an
enhancement layer (a higher layer) having image data which
is added to the base layer so as to increase quality of a
moving image.
Citation List
Non Patent Literature
[0004]
NPL 1: "Draft Errata List with Revision-Marked
Corrections for H.264/AVC", JVT-1050, Thomas Wiegand et al.,
Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG, 2003
NPL 2: Joint Draft 4.0 on Multiview Video Coding, Joint
Video Team of ISO/IEC MPEG & ITU-T VCEG, JVT-X209, July 2007
NPL 3: Heiko Schwarz, Detlev Marpe, and Thomas Wiegand,
"Overview of the Scalable Video Coding Extension of the
H.264/AVC Standard", IEEE TRANSACTIONS ON CIRCUITS AND
SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 17, NO. 9, SEPTEMBER 2007,
pp. 1103 to 1120.
Summary of Invention
Technical Problem
[0005]
In delivery circumstances in which an AVC stream and an
MVC stream are dynamically changed, it is expected that a
receiver corresponding to the MVC determines whether a
4
stream includes only "Stream_Type=0x1B" or both of
"Stream_Type=0x1B" and "Stream_Type=0x20", and performs
switching between reception modes.
[0006]
A normal AVC (2D) video elementary stream is sent using
"Stream_Type=0x1B" of Program Map Table (PMT). In addition,
an MVC base view video elementary stream (Base view subbitstream)
is sent using "Stream_Type=0x1B" of the PMT in
some cases.
[0007]
A structure in which an AVC stream or an MVC stream can
be discriminated depending on a level of the PMT which is
Program Specific Information (PSI) is provided in a section
of a transport stream. In other words, when a video
elementary stream includes only "Stream_Type=0x1B", the
stream is recognized as a 2D AVC stream. In addition, when
a video elementary stream includes both of
"Stream_Type=0x1B" and "Stream_Type=0x20", the stream is
recognized as an MVC stream.
[0008]
However, there is a case where the PMT is not
necessarily dynamically updated depending on transmission
side equipment. In this case, the following inconvenience
is considered when delivery content is changed from a
stereoscopic (3D) image to a two-dimensional (2D) image. In
5
other words, it is considered that a receiver also
continuously receives a stream of which the stream type
(Stream_Type) is "0x20" along with an elementary stream of
which the stream type (Stream_Type) is "0x1B" and thus
continuously waits for the data.
[0009]
Although an elementary stream of "0x20" is not received
after the delivery content is changed to a two-dimensional
(2D) image, the receiver continuously waits for the
elementary stream of "0x20" to come. As a result, there is
concern that correct decoding may not be performed, and
normal display may not be performed. As such, in a case
where the receiver determines a mode thereof using only the
kind of "Stream_type" of the PMT and, there is a probability
that the mode may not be correct, and a correct stream may
not be received.
[0010]
Fig. 94 shows a configuration example of a video
elementary stream and a Program Map Table (PMT) in a
transport stream. The period of access units (AU) of "001"
to "009" of video elementary streams ES1 and ES2 is a period
when two video elementary streams are present. This period
is, for example, a body period of a 3D program, and the two
streams form a stream of stereoscopic (3D) image data.
[0011]
6
The period of access units of "010" to "014" of the
video elementary stream ES1, subsequent thereto, is a period
when only one video elementary stream is present. This
period is, for example, a CM period inserted between body
periods of a 3D program, and this single stream forms a
stream of two-dimensional image data.
[0012]
In addition, the period of access units of "015" and
"016" of video elementary streams ES1 and ES2, subsequent
thereto, is a period when two video elementary streams are
present. This period is, for example, a body period of a 3D
program, and the two streams form a stream of stereoscopic
(3D) image data.
[0013]
A cycle (for example, 100 msec) of updating
registration of a video elementary stream in the PMT cannot
track a video frame cycle (for example, 33.3 msec). In a
method of informing of a dynamic variation in an elementary
stream forming a transport stream by using the PMT, the
elementary stream is not synchronized with a configuration
of the transport stream of the PMT, and thus an accurate
operation of the receive is not secured.
[0014]
In addition, in the existing signal standard (MPEG), a
descriptor of "MVC_extension descriptor" is essentially
7
inserted into an MVC base view video elementary stream (Base
view sub-bitstream) of "Stream_Type=0x1B" as a descriptor of
the PMT. When this descriptor is present, the presence of a
non-base view video elementary stream (Non-Base view subbitstream)
can be recognized.
[0015]
However, it cannot be said that a video elementary
stream of "Elementary PID" indicated by "Stream_Type=0x1B"
is the above-described MVC base view video elementary stream
(Base view sub-bitstream). There is a case where the stream
may be an AVC (in this case, broadly a high profile) stream
in the related art. Particularly, in order to secure
compatibility with an existing 2D receiver, there is a case
where it is recommended that, even in stereoscopic (3D)
image data, a base view video elementary stream maintains an
AVC (2D) video elementary stream in the related art.
[0016]
In this case, a stream of stereoscopic image data is
formed by an AVC (2D) video elementary stream and a non-base
view video elementary stream (Non-Base view sub-bitstream).
In that case, a descriptor of "MVC_extension descriptor" is
not correlated with a video elementary stream of
"Stream_Type=0x1B". For this reason, the presence of the
non-base view video elementary stream (Non-Base view subbitstream)
other than the AVC (2D) video elementary stream
8
corresponding to a base view video elementary stream is not
recognized.
[0017]
In addition, in the above description, a description
has been made that it is difficult to determine whether or
not an elementary stream included in the transport stream
forms stereoscopic (3D) image data. Detailed description is
omitted, and this inconvenience also occurs in a case where
an AVC stream and the above-described SVC stream are
transmitted in a time division manner.
[0018]
An object of the present technology is to enable a
reception side to appropriately and accurately handle a
dynamic variation in delivery content so as to receive a
correct stream.
Solution to Problem
[0019]
A concept of the present technology lies in an image
data transmission device including a transmission unit that
transmits one or a plurality of video streams including a
predetermined number of image data items; and an information
inserting unit that inserts auxiliary information for
identifying a first transmission mode in which a plurality
of image data items are transmitted and a second
transmission mode in which a single image data item is
9
transmitted, into the video stream.
[0020]
In the present technology, one or a plurality of video
streams including image data of a predetermined number of
views are transmitted by the transmission unit. In addition,
an information inserting unit inserts auxiliary information
for identifying the first transmission mode in which a
plurality of image data items are transmitted and the second
transmission mode in which a single image data item is
transmitted into the video stream. For example, the
information inserting unit may insert the auxiliary
information, at least with the program unit, the scene unit,
the picture group unit, or the picture unit.
[0021]
For example, the first transmission mode may be a
stereoscopic image transmission mode in which base view
image data and non-base view image data used along with the
base view image data are transmitted so as to display a
stereoscopic image, and the second transmission mode may be
a two-dimensional image transmission mode in which twodimensional
image data is transmitted.
[0022]
In addition, in this case, for example, the first
transmission mode may be a stereoscopic image transmission
mode in which image data of a left eye view and image data
10
of a right eye view for displaying a stereo stereoscopic
image are transmitted. Further, in this case, for example,
the auxiliary information indicating the stereoscopic image
transmission mode may include information indicating a
relative positional relationship of each view.
[0023]
Furthermore, for example, the first transmission mode
may be an extension image transmission mode in which image
data of the lowest layer forming scalable coded image data
and image data of layers other than the lowest layer are
transmitted, and the second transmission mode may be a base
image transmission mode in which base image data is
transmitted.
[0024]
In the present technology, for example, the information
inserting unit may insert auxiliary information indicating
the first transmission mode into the video stream in the
first transmission mode and inserts auxiliary information
indicating the second transmission mode into the video
stream in the second transmission mode.
[0025]
In addition, in the present technology, for example,
the information inserting unit may insert auxiliary
information indicating the first transmission mode into the
video stream in the first transmission mode and may not
11
insert the auxiliary information into the video stream in
the second transmission mode.
[0026]
Further, the information inserting unit may not insert
the auxiliary information into the video stream in the first
transmission mode and may insert auxiliary information
indicating the second transmission mode into the video
stream in the second transmission mode.
[0027]
In addition, in the present technology, for example,
the transmission unit may transmit a base video stream
including first image data and a predetermined number of
additional video streams including second image data used
along with the first image data in the first transmission
mode, and transmit a single video stream including the first
image data in the second transmission mode.
[0028]
In addition, in the present technology, for example,
the transmission unit may transmit a base video stream
including first image data and a predetermined number of
additional video streams including second image data used
along with the first image data in the first transmission
mode, and transmit a base video stream including first image
data and a predetermined number of additional video streams
substantially including image data which is the same as the
12
first image data in the second transmission mode.
[0029]
As above, in the present technology, when one or a
plurality of video streams including a predetermined number
of image data items are transmitted, auxiliary information
for identifying the first transmission mode in which a
plurality of image data items are transmitted and the second
transmission mode in which a single image data item is
transmitted is inserted into the video stream. For this
reason, a reception side can easily understand the first
transmission mode or the second transmission mode on the
basis of this auxiliary information, so as to appropriately
and accurately handle a variation in a stream configuration,
that is, a dynamic variation in delivery content, thereby
receiving a correct stream.
[0030]
In addition, in the present technology, the
transmission unit may transmit a container of a
predetermined format including the video stream, and the
image data transmission device may further include
identification information inserting unit that inserts
identification information for identifying whether to be in
the first transmission mode or in the second transmission
mode, into a layer of the container. As such,
identification information is inserted into the layer of the
13
container, and thereby a flexible operation can be performed
in a reception side.
[0031]
Another concept of the present technology lies in an
image data reception device including a reception unit that
receives one or a plurality of video streams including a
predetermined number of image data items; a transmission
mode identifying unit that identifies a first transmission
mode in which a plurality of image data items are
transmitted and a second transmission mode in which a single
image data item is transmitted on the basis of auxiliary
information which is inserted into the received video
stream; and a processing unit that performs a process
corresponding to each mode on the received video stream on
the basis of the mode identification result, so as to
acquire the predetermined number of image data items.
[0032]
In the present technology, one or a plurality of video
streams including a predetermined number of image data items
are received by the reception unit. The first transmission
mode in which a plurality of image data items are
transmitted or the second transmission mode in which a
single image data item is transmitted are identified by the
transmission mode identifying unit on the basis of auxiliary
information which is inserted into the received video stream.
14
[0033]
For example, the first transmission mode may be a
stereoscopic image transmission mode in which base view
image data and non-base view image data used along with the
base view image data are transmitted so as to display a
stereoscopic image, and the second transmission mode may be
a two-dimensional image transmission mode in which twodimensional
image data is transmitted. In addition, for
example, the first transmission mode may be an extension
image transmission mode in which image data of the lowest
layer forming scalable coded image data and image data of
layers other than the lowest layer are transmitted, and the
second transmission mode may be a base image transmission
mode in which base image data is transmitted.
[0034]
In the present technology, for example, the
transmission mode identifying unit may identify the first
transmission mode when auxiliary information indicating the
first transmission mode is inserted into the received video
stream, and identify the second transmission mode when
auxiliary information indicating the second transmission
mode is inserted into the received video stream.
[0035]
In addition, in the present technology, for example,
the transmission mode identifying unit may identify the
15
first transmission mode when auxiliary information
indicating the first transmission mode is inserted into the
received video stream, and identify the second transmission
mode when the auxiliary information is not inserted into the
received video stream.
[0036]
Further, in the present technology, for example, the
transmission mode identifying unit may identify the first
transmission mode when the auxiliary information is not
inserted into the received video stream, and identify the
second transmission mode when auxiliary information
indicating the second transmission mode is inserted into the
received video stream.
[0037]
In addition, in the present technology, for example,
the reception unit may receive a base video stream including
first image data and a predetermined number of additional
video streams including second image data used along with
the first image data in the first transmission mode, and
receive a single video stream including the first image data
in the second transmission mode. In this case, the
processing unit may process the base video stream and the
predetermined number of additional video streams so as to
acquire the first image data and the second image data in
the first transmission mode, and process the single video
16
stream so as to acquire the first image data in the second
transmission mode.
[0038]
Further, in the present technology, for example, the
reception unit may receive a base video stream including
first image data and a predetermined number of additional
video streams including second image data used along with
the first image data in the first transmission mode, and
receive a base video stream including first image data and a
predetermined number of additional video streams
substantially including image data which is the same as the
first image data in the second transmission mode. In this
case, the processing unit may process the base video stream
and the predetermined number of additional video streams so
as to acquire the first image data and the second image data
in the first transmission mode, and process the base video
stream so as to acquire the first image data without
performing a process of acquiring the second image data from
the predetermined number of additional video streams in the
second transmission mode.
[0039]
As above, in the present technology, the first
transmission mode in which a plurality of image data items
are transmitted or the second transmission mode in which a
single image data item is transmitted are identified based
17
on auxiliary information which is inserted into the received
video stream. In addition, a process corresponding to the
identified mode is performed on the received video stream so
as to acquire a predetermined number of image data items.
It is possible to easily understand the first transmission
mode or the second transmission mode so as to appropriately
and accurately handle a variation in a stream configuration,
that is, a dynamic variation in delivery content, thereby
receiving a correct stream.
[0040]
In addition, in the present technology, for example,
the reception unit may receive a container of a
predetermined format including the video stream, and
identification information for identifying whether to be in
the first transmission mode or in the second transmission
mode may be inserted into the container. In this case, the
transmission mode identifying unit may identify the first
transmission mode in which a plurality of image data items
are transmitted or the second transmission mode in which a
single image data item is transmitted on the basis of
auxiliary information which is inserted into the received
video stream and identification information which is
inserted into the layer of the container.
Advantageous Effects of Invention
[0041]
18
According to the present technology, a reception side
can appropriately and accurately handle a configuration
variation of an elementary stream, that is, a dynamic
variation in delivery content, so as to favorably receive a
stream.
Brief Description of Drawings
[0042]
[Fig. 1] Fig. 1 is a block diagram illustrating a
configuration example of an image transmission and reception
system as an embodiment.
[Fig. 2] Fig. 2 is a diagram illustrating an example in
which image data of each of center, left end and right end
views is coded as data of a single picture.
[Fig. 3] Fig. 3 is a diagram illustrating an example in
which image data of a center view is coded as data of a
single picture, and image data of two left end and right end
views undergoes an interleaving process so as to be coded as
data of a single picture.
[Fig. 4] Fig. 4 is a diagram illustrating an example of
a video stream including coded data of a plurality of
pictures.
[Fig. 5] Fig. 5 is a diagram illustrating an example of
a case where coded data items of three pictures are present
together in a single video stream.
[Fig. 6] Fig. 6 is a diagram schematically illustrating
19
a display unit of a receiver in a case where the number of
views is 5 in a method of transmitting image data of left
end and right end views and a center view located
therebetween among N views.
[Fig. 7] Fig. 7 is a block diagram illustrating a
configuration example of a transmission data generation unit
which generates a transport stream.
[Fig. 8] Fig. 8 is a diagram illustrating a view
selection state in a view selector of the transmission data
generation unit.
[Fig. 9] Fig. 9 is a diagram illustrating an example of
disparity data (disparity vector) of each block.
[Fig. 10] Fig. 10 is a diagram illustrating an example
of a method of generating disparity data of the block unit.
[Fig. 11] Fig. 11 is a diagram illustrating a method of
generating disparity data of the pixel unit through a
conversion process from the block unit to the pixel unit.
[Fig. 12] Fig. 12 is a diagram illustrating a
structural example of a multi-view stream configuration
descriptor as identification information.
[Fig. 13] Fig. 13 is a diagram illustrating content of
principal information in the structural example of the
multi-view stream configuration descriptor.
[Fig. 14] Fig. 14 is a diagram illustrating a
structural example of multi-view stream configuration
20
information as view configuration information.
[Fig. 15] Fig. 15 is a diagram illustrating content of
principal information in the structural example of the
multi-view stream configuration information.
[Fig. 16] Fig. 16 is a diagram illustrating content of
principal information in the structural example of the
multi-view stream configuration information.
[Fig. 17] Fig. 17 is a diagram illustrating content of
principal information in the structural example of the
multi-view stream configuration information.
[Fig. 18] Fig. 18 is a diagram illustrating an example
of a relationship between the number of views indicated by
"view_count" and positions of two views indicated by
"view_pair_position_id".
[Fig. 19] Fig. 19 is a diagram illustrating an example
in which a transmission side or a reception side generates
disparity data in a case of transmitting image data of a
pair of two views located further inward than both ends
along with image data of a pair of two views located at both
ends.
[Fig. 20] Fig. 20 is a diagram illustrating an example
in which the reception side interpolates and generates image
data of a view located between the respective views on the
basis of disparity data.
[Fig. 21] Fig. 21 is a diagram illustrating that multi21
view stream configuration SEI is inserted into a "SELs" part
of an access unit.
[Fig. 22] Fig. 22 is a diagram illustrating a
structural example of "multiview stream configuration SEI
message" and "userdata_for_multiview_stream_configuration()".
[Fig. 23] Fig. 23 is a diagram illustrating a
structural example of "user_data()".
[Fig. 24] Fig. 24 is a diagram illustrating a
configuration example of a case where three video streams
are included in a transport stream TS.
[Fig. 25] Fig. 25 is a diagram illustrating a
configuration example of a case where two video streams are
included in a transport stream TS.
[Fig. 26] Fig. 26 is a diagram illustrating a
configuration example of a case where a single video stream
is included in a transport stream TS.
[Fig. 27] Fig. 27 is a block diagram illustrating a
configuration example of a receiver forming the image
transmission and reception system.
[Fig. 28] Fig. 28 is a diagram illustrating a
calculation example of a scaling ratio.
[Fig. 29] Fig. 29 is a diagram schematically
illustrating an example of an interpolation and generation
process in a view interpolation unit.
[Fig. 30] Fig. 30 is a diagram illustrating an example
22
of a reception stream in a case where a 3D period (when a
stereoscopic image is received) and a 2D period (when a twodimensional
image is received) are alternately continued.
[Fig. 31] Fig. 31 is a diagram illustrating an example
of a reception stream in a case where a 3D period (when a
stereoscopic image is received) and a 2D period (when a twodimensional
image is received) are alternately continued.
[Fig. 32] Fig. 32 is a flowchart illustrating an
example of process procedures of operation mode switching
control in a CPU.
[Fig. 33] Fig. 33 is a diagram illustrating an example
of a video stream included in a transport stream.
[Fig. 34] Fig. 34 is a diagram illustrating a case
where a 3D period (a stereoscopic image transmission mode)
and a 2D period (a two-dimensional image transmission mode)
are alternately continued and there is no auxiliary
information (multi-view stream configuration SEI message)
for identifying a mode.
[Fig. 35] Fig. 35 is a diagram illustrating a case
where a 3D period and a 2D period are alternately continued
and there is auxiliary information (multi-view stream
configuration SEI message) for identifying a mode.
[Fig. 36] Fig. 36 is a block diagram illustrating
another configuration example of a receiver forming the
image transmission and reception system.
23
[Fig. 37] Fig. 37 is a diagram illustrating a
structural example (Syntax) of a multi-view view position
(Multiview view position()) included in a multi-view stream
configuration SEI message.
[Fig. 38] Fig. 38 is a diagram illustrating that multiview
position SEI is inserted into a "SEIs" part of an
access unit.
[Fig. 39] Fig. 39 is a diagram illustrating an example
of a reception stream in a case where a 3D period (when a
stereoscopic image is received) and a 2D period (when a twodimensional
image is received) are alternately continued.
[Fig. 40] Fig. 40 is a diagram illustrating an example
of a reception stream in a case where a 3D period (when a
stereoscopic image is received) and a 2D period (when a twodimensional
image is received) are alternately continued.
[Fig. 41] Fig. 41 is a flowchart illustrating an
example of process procedures of operation mode switching
control in the CPU.
[Fig. 42] Fig. 42 is a diagram illustrating an example
of a video stream included in a transport stream.
[Fig. 43] Fig. 43 is a diagram illustrating a case
where a 3D period and a 2D period are alternately continued
and there is auxiliary information (multi-view view position
SEI message) for identifying a mode.
[Fig. 44] Fig. 44 is a flowchart illustrating an
24
example of process procedures of operation mode switching
control in the CPU.
[Fig. 45] Fig. 45 is a diagram illustrating a
structural example (Syntax) of frame packing arrangement
data (frame_packing_arrangement_data()).
[Fig. 46] Fig. 46 is a diagram illustrating a value of
"arrangement_type" and the meaning thereof.
[Fig. 47] Fig. 47 is a diagram illustrating a
structural example (Syntax) of "user_data()".
[Fig. 48] Fig. 48 is a diagram illustrating an example
of a reception stream in a case where a 3D period (when a
stereoscopic image is received) and a 2D period (when a twodimensional
image is received) are alternately continued.
[Fig. 49] Fig. 49 is a diagram illustrating a case
where auxiliary information indicating a 2D mode is inserted
with the scene unit or the picture group unit (GOP unit)
during a 2D period.
[Fig. 50] Fig. 50 is a flowchart illustrating an
example of process procedures of operation mode switching
control in the CPU.
[Fig. 51] Fig. 51 is a diagram illustrating an example
of a reception stream in a case where a 3D period (when a
stereoscopic image is received) and a 2D period (when a twodimensional
image is received) are alternately continued.
[Fig. 52] Fig. 52 is a diagram illustrating a case
25
where a 3D period and a 2D period are alternately continued
and there is auxiliary information (an SEI message
indicating a newly defined 2D mode) for identifying a mode.
[Fig. 53] Fig. 53 is a diagram illustrating an example
in which image data of each view of the left eye and the
right eye is coded as data of a single picture.
[Fig. 54] Fig. 54 is a block diagram illustrating
another configuration example of the transmission data
generation unit which generates a transport stream.
[Fig. 55] Fig. 55 is a block diagram illustrating
another configuration example of the receiver forming the
image transmission and reception system.
[Fig. 56] Fig. 56 is a diagram illustrating an example
of a reception stream in a case where a 3D period (when a
stereoscopic image is received) and a 2D period (when a twodimensional
image is received) are alternately continued.
[Fig. 57] Fig. 57 is a diagram illustrating an example
of a reception stream in a case where a 3D period (when a
stereoscopic image is received) and a 2D period (when a twodimensional
image is received) are alternately continued.
[Fig. 58] Fig. 58 is a diagram illustrating an example
of a video stream included in a transport stream.
[Fig. 59] Fig. 59 is a diagram collectively
illustrating methods of a case A, a case B and a case C for
identifying a 3D period and a 2D period when a base stream
26
and an additional stream are present in the 3D period and
only a base stream is present in the 2D period.
[Fig. 60] Fig. 60 is a diagram illustrating an example
of a reception stream in a case where a 3D period (when a
stereoscopic image is received) and a 2D period (when a twodimensional
image is received) are alternately continued.
[Fig. 61] Fig. 61 is a diagram illustrating an example
of a reception stream in a case where a 3D period (when a
stereoscopic image is received) and a 2D period (when a twodimensional
image is received) are alternately continued.
[Fig. 62] Fig. 62 is a flowchart illustrating an
example of process procedures of operation mode switching
control in the CPU.
[Fig. 63] Fig. 63 is a diagram illustrating an example
of a reception packet process when the receiver receives a
stereoscopic (3D) image.
[Fig. 64] Fig. 64 is a diagram illustrating a
configuration example (Syntax) of a NAL unit header (NAL
unit header MVC extension).
[Fig. 65] Fig. 65 is a diagram illustrating an example
of a reception packet process when the receiver receives a
two-dimensional (2D) image.
[Fig. 66] Fig. 66 is a diagram illustrating an example
of a video stream included in a transport stream.
[Fig. 67] Fig. 67 is a diagram illustrating a case
27
where a 3D period (a 3D mode period) and a 2D period (a 2D
mode period) are alternately continued and there is
auxiliary information (multi-view view position SEI message)
for identifying a mode.
[Fig. 68] Fig. 68 is a diagram illustrating an example
of a reception stream in a case where a 3D period (when a
stereoscopic image is received) and a 2D period (when a twodimensional
image is received) are alternately continued.
[Fig. 69] Fig. 69 is a diagram illustrating an example
of a reception stream in a case where a 3D period (when a
stereoscopic image is received) and a 2D period (when a twodimensional
image is received) are alternately continued.
[Fig. 70] Fig. 70 is a diagram illustrating an example
of a video stream included in a transport stream.
[Fig. 71] Fig. 71 is a diagram illustrating a case
where a 3D period (a 3D mode period) and a 2D period (a 2D
mode period) are alternately continued and there is
auxiliary information (multi-view view position SEI message)
for identifying a mode.
[Fig. 72] Fig. 72 is a diagram illustrating an example
of a reception stream in a case where a 3D period (when a
stereoscopic image is received) and a 2D period (when a twodimensional
image is received) are alternately continued.
[Fig. 73] Fig. 73 is a diagram illustrating an example
of a reception stream in a case where a 3D period (when a
28
stereoscopic image is received) and a 2D period (when a twodimensional
image is received) are alternately continued.
[Fig. 74] Fig. 74 is a diagram illustrating an example
of a video stream included in a transport stream.
[Fig. 75] Fig. 75 is a diagram illustrating a case
where a 3D period and a 2D period are alternately continued
and there is auxiliary information (an SEI message
indicating a newly defined 2D mode) for identifying a mode.
[Fig. 76] Fig. 76 is a diagram illustrating an example
of a reception stream in a case where a 3D period (when a
stereoscopic image is received) and a 2D period (when a twodimensional
image is received) are alternately continued.
[Fig. 77] Fig. 77 is a diagram illustrating an example
of a reception stream in a case where a 3D period (when a
stereoscopic image is received) and a 2D period (when a twodimensional
image is received) are alternately continued.
[Fig. 78] Fig. 78 is a diagram illustrating an example
of a video stream included in a transport stream.
[Fig. 79] Fig. 79 is a diagram collectively
illustrating methods of a case D, a case E and a case F for
identifying a 3D period and a 2D period when a base stream
and an additional stream are present in both of the 3D
period and the 2D period.
[Fig. 80] Fig. 80 is a diagram illustrating a stream
configuration example 1 in which a base video stream and an
29
additional video stream are transmitted in a 3D period (3D
image transmission mode) and a single video stream (only a
base video stream) is transmitted in a 2D period (2D image
transmission mode).
[Fig. 81] Fig. 81 is a diagram illustrating a stream
configuration example 2 in which a base video stream and an
additional video stream are transmitted in both a 3D period
(3D image transmission mode) and a 2D period (2D image
transmission mode).
[Fig. 82] Fig. 82 is a diagram illustrating an example
in which a base video stream and an additional video stream
are present in both a 3D period and a 2D period, and
signaling is performed using both a program loop and a video
ES loop of a PMT.
[Fig. 83] Fig. 83 is a diagram illustrating a
structural example (Syntax) of a stereoscopic program
information descriptor
(Stereoscopic_program_info_descriptor).
[Fig. 84] Fig. 84 is a diagram illustrating a
structural example (Syntax) of an MPEG2 stereoscopic video
descriptor.
[Fig. 85] Fig. 85 is a diagram illustrating a
configuration example of a transport stream TS.
[Fig. 86] Fig. 86 is a diagram illustrating an example
in which a base video stream and an additional video stream
30
are present in both of a 3D period and a 2D period, and
signaling is performed using a video ES loop of the PMT.
[Fig. 87] Fig. 87 is a diagram illustrating an example
in which a base video stream and an additional video stream
are present in both of a 3D period and a 2D period, and
signaling is performed using a program loop of the PMT.
[Fig. 88] Fig. 88 is a diagram illustrating an example
in which a base video stream and an additional video stream
are present in a 3D period and only a base video stream is
present in a 2D period, and signaling is performed using
both a program loop and a video ES loop of the PMT.
[Fig. 89] Fig. 89 is a diagram illustrating an example
in which a base video stream and an additional video stream
are present in a 3D period and only a base video stream is
present in a 2D period, and signaling is performed using a
video ES loop.
[Fig. 90] Fig. 90 is a diagram illustrating an example
in which a base video stream and an additional video stream
are present in a 3D period and only a base video stream is
present in a 2D period, and signaling is performed using a
program loop of the PMT.
[Fig. 91] Fig. 91 is a diagram illustrating an example
of a reception packet process when an extended image is
received.
[Fig. 92] Fig. 92 is a diagram illustrating a
31
configuration example (Syntax) of a NAL unit header (NAL
unit header SVC extension).
[Fig. 93] Fig. 93 is a diagram illustrating an example
of a reception packet process in a base image transmission
mode.
[Fig. 94] Fig. 94 is a diagram illustrating a
configuration example of a video elementary stream and a
Program Map Table (PMT) in a transport stream.
Description of Embodiments
[0043]
Hereinafter, embodiments of the present invention will
be described. Further, the description will be made in the
following order.
1. Embodiments
2. Modification examples
[0044]
<1. Embodiments>
[Image transmission and reception system]
Fig. 1 shows a configuration example of an image
transmission and reception system 10 as an embodiment. The
image transmission and reception system 10 includes a
broadcast station 100 and a receiver 200. The broadcast
station 100 carries a transport stream TS which is a
container on a broadcast wave so as to be transmitted.
[0045]
32
When a stereoscopic (3D) image is transmitted, the
transport stream TS includes one or a plurality of video
streams which include image data of a predetermined number
of, for example, three views for stereoscopic image display
in this embodiment. In this case, the video streams are
transmitted as, for example, an MVC base view video
elementary stream (Base view sub-bitstream) and an MVC nonbase
view video elementary stream (Non-Base view subbitstream).
[0046]
In addition, when a two-dimensional (2D) image is
displayed, a video stream including a two-dimensional image
data is included in the transport stream TS. In this case,
the video stream is transmitted as, for example, an AVC (2D)
video elementary stream.
[0047]
The transport stream TS which is transmitted when a
stereoscopic (3D) image is transmitted includes one or a
plurality of video streams which are obtained by coding
image data of at least a center view, a left end view, and a
right end view among a plurality of views for stereoscopic
image display. In this case, the center view forms an
intermediate view located between the left end view and the
right end view.
[0048]
33
In the video stream included in the transport stream TS
transmitted when the stereoscopic (3D) image is transmitted,
as shown in Fig. 2, each of image data items of the center
(Center) view, the left end (Left) view, and the right end
(Right) view is coded as data of a single picture. In the
shown example, data of each picture has a full HD size of
1920*1080.
[0049]
Alternatively, in the video stream included in the
transport stream TS transmitted when the stereoscopic (3D)
image is transmitted, as shown in Fig. 3(a), image data of
the center (Center) view is coded as data of a single
picture, and image data items of the left end (Left) view
and the right end (Right) view undergo an interleaving
process and are coded as data of a single picture. In the
shown example, data of each picture has a full HD size of
1920*1080.
[0050]
In addition, in a case where image data items of the
left end view and the right end view undergo an interleaving
process and are coded as data of a single picture, the image
data of each view is decimated by 1/2 in a horizontal
direction or a vertical direction. In the shown example,
the interleaving type is a side-by-side type, and the size
of each view is 960*1080. Although not shown, a top-and34
bottom type may be considered as an interleaving type, and,
in this case, the size of each view is 1920*540.
[0051]
As above, in a case where image data items of the left
end view and the right end view undergo an interleaving
process and are coded as data of a single picture, in a
reception side, as shown in Fig. 3(b), a scaling process is
performed, and thereby the size of image data of the left
end view and the right end view is returned to a full HD
size of 1920*1080.
[0052]
A video stream included in the transport stream TS
transmitted when a stereoscopic (3D) image is transmitted
includes data of one or a plurality of pictures. For
example, the transport stream TS includes the following
three video streams (video elementary streams). In other
words, the video streams are video streams obtained by
coding each of image data items of the center view, the left
end view, and the right end view as a single picture.
[0053]
In this case, for example, a video stream obtained by
coding image data of the center view as a single picture is
an MVC base view video elementary stream (base video stream).
In addition, the other two video streams obtained by coding
each of image data items of the left end view and the right
35
end view as a single picture are MVC non-base view video
elementary stream (additional video stream).
[0054]
In addition, for example, the transport stream TS
includes the following two video streams (video elementary
streams). In other words, the video streams are a video
stream which is obtained by coding image data of the center
view as a single picture and a video stream which is
obtained by performing an interleaving process on image data
items of the left end view and the right end view so as to
be coded as a single picture.
[0055]
In this case, for example, the video stream obtained by
coding image data of the center view as a single picture is
an MVC base view video elementary stream (base video stream).
Further, the other video stream obtained by performing an
interleaving process on image data items of the left end
view and the right end view so as to be coded as a single
picture is an MVC non-base view video elementary stream
(additional video stream).
[0056]
In addition, for example, the transport stream TS
includes the following single video stream (video elementary
stream). In other words, this single video stream includes
data obtained by coding each of image data items of the
36
center view, the left end view, and the right end view as
data of a single picture. In this case, the single video
stream is an MVC base view video elementary stream (base
video stream).
[0057]
Figs. 4(a) and 4(b) show an example of the video stream
including coded data of a plurality of pictures. Coded data
of each picture is sequentially disposed in each access unit.
In this case, coded data of the initial picture is
constituted by "SPS to Coded Slice", and coded data of the
second picture and thereafter is constituted by "Subset SPS
to Coded Slice". Further, this example shows an example of
performing coding of MPEG4-AVC but is also applicable to
other coding methods. In addition, the hexadecimal digit in
the figures indicates "NAL unit type".
[0058]
In a case where coded data items of the respective
pictures are present together in a single video stream, a
boundary between the respective pictures is required to be
instantly identified. However, Access Unit Delimeter (AUD)
can be appended only to a head of each access unit.
Therefore, as shown in Fig. 4(b), it is considered that new
"NAL unit" which indicates a boundary such as "View
Separation Marker" is defined and is disposed between the
coded data times of the respective pictures. Thereby, it is
37
possible to instantly access leading data of each picture.
In addition, Fig. 4(a) shows an example in which "View
Separation Marker" is not disposed between data items of two
views.
[0059]
Figs. 5(a) and 5(b) show an example in which coded data
items of three pictures are present together in a single
video stream. Here, the coded data of each picture is
indicated by a substream. Fig. 5(a) shows a leading access
unit of Group of Pictures (GOP), and Fig. 5(b) shows an
access unit other than the leading access unit of the GOP.
[0060]
View configuration information regarding image data of
a video stream is inserted into a layer (a picture layer, a
sequence layer, or the like) of the video stream. The view
configuration information forms auxiliary information which
presents an element of stereoscopic information. The view
configuration information includes information indicating
whether or not image data included in a corresponding video
stream is image data of a portion of views forming 3D,
information (information indicating a relative positional
relationship of each view) indicating image data of which
view is image data included in the video stream in a case
where the image data is image data of a portion of views
forming 3D, information indicating whether data of a
38
plurality of pictures is coded in a single access unit of
the corresponding video stream, and the like.
[0061]
This view configuration information is inserted into,
for example, a user data region or the like of a picture
header or a sequence header of a video stream. The view
configuration information is inserted at least with the
program unit, the scene unit, the picture group unit, or the
picture unit. A reception side performs a 3D display
process or a 2D display process on the basis of the view
configuration information. In addition, in a case where the
reception side performs a 3D display process on the basis of
the view configuration information, an appropriate and
efficient process for observing three-dimensional images
(stereoscopic images) with the naked eye by using image data
of a plurality of views is performed. Details of the view
configuration information will be described later.
[0062]
In addition, identification information for identifying
whether or not view configuration information is inserted
into a layer of a video stream is inserted into the layer of
the transport stream TS. This identification information is
inserted, for example, under a video elementary loop (Video
ES loop) of a Program Map Table (PMT) included in the
transport stream TS, an Event Information Table (EIT), or
39
the like. A reception side can easily identify whether or
not the view configuration information is inserted into a
layer of a video stream on the basis of this identification
information. Details of the identification information will
be described later.
[0063]
The receiver 200 receives the transport stream TS which
is carried on a broadcast wave sent from the broadcast
station 100. In addition, the receiver 200 decodes video
streams included in the transport stream TS so as to acquire
image data of a center view, a left end view, and a right
end view when a stereoscopic (3D) image is transmitted. At
this time, the receiver 200 can understand image data of
which view position is image data included in each video
stream on the basis of view configuration information
included in the layer of the video stream.
[0064]
The receiver 200 acquires image data of a predetermined
number of views located between a center view and a left end
view and between the center view and a right end view
through an interpolation process on the basis of disparity
data between the center view and the left end view and
disparity data between the center view and the right end
view. At this time, the receiver 200 can recognize the
number of views on the basis of view configuration
40
information included in the layer of the video stream, and
thus can easily understand a view of which position is not
transmitted.
[0065]
In addition, the receiver 200 decodes a disparity data
stream which is sent along with the video stream from the
broadcast station 100 so as to acquire the above-described
disparity data. Alternatively, the receiver 200 generates
the above-described disparity data on the basis of the
acquired disparity data of the center view, the left end
view, and the right end view.
[0066]
The receiver 200 combines and displays images of the
respective views on a display unit such that threedimensional
images (stereoscopic images) are observed with
the naked eye, on the basis of the image data of each of the
center, left end and right end views sent from the broadcast
station 100 and the image data of each view acquired from
the above-described interpolation process.
[0067]
Fig. 6 schematically shows the display unit of the
receiver 200 when the number of views is five. Here,
"View_0" indicates a center view, "View_1" indicates a first
right view next to the center, "View_2" indicates a first
left view next to the center, "View_3" indicates a second
41
right view next to the center, that is, a right end view,
and "View_4" indicates a second left view next to the center,
that is, a left end view. In this case, only image data of
the views of "View_0", "View_3", and "View_4" is transmitted
from the broadcast station 100, the receiver 200 receives
the image data of the views of "View_0", "View_3", and
"View_4", and the remaining image data of the views of
"View_1" and "View_2" is obtained through an interpolation
process. In addition, the receiver 200 combines and
displays images of the five views on the display unit such
that three-dimensional images (stereoscopic images) are
observed with the naked eye. Further, Fig. 6 shows a
lenticular lens, but, a parallax barrier may be used instead
of it.
[0068]
The receiver 200 decodes a video stream included in the
transport stream TS so as to acquire two-dimensional image
data when a two-dimensional (2D) image is transmitted. In
addition, the receiver 200 displays a two-dimensional image
on the display unit on the basis of the two-dimensional
image data.
[0069]
(Configuration example of transmission data generation
unit)
Fig. 7 shows a configuration example of a transmission
42
data generation unit 110 which generates the above-described
transport stream TS in the broadcast station 100. The
transmission data generation unit 110 includes N image data
output portions 111-1 to 111-N, a view selector 112, scalers
113-1, 113-2 and 113-3, video encoders 114-1, 114-2 and 114-
3, and a multiplexer 115. In addition, the transmission
data generation unit 110 includes a disparity data
generation portion 116, a disparity encoder 117, a graphics
data output portion 118, a graphics encoder 119, an audio
data output portion 120, and an audio encoder 121.
[0070]
First, a description will be made of a case where a
stereoscopic (3D) image is transmitted. The image data
output portions 111-1 to 111-N output image data of N views
(View 1,…, and View N) for stereoscopic image display. The
image data output portions are formed by, for example, a
camera which images a subject and outputs image data, an
image data reading portion which reads image data from a
storage medium so as to be output, or the like. In addition,
image data of a view which is not transmitted may not be
present actually.
[0071]
In addition, the view selector 112 extracts at least
image data of a left end view and a right end view and
selectively extracts image data of an intermediate view (one
43
or two or more) located between the left end and the right
end from image data of the N views (View 1,…, and View N).
In this embodiment, the view selector 112 extracts image
data VL of the left end view and image data VR of the right
end view and extracts image data VC of the center view. Fig.
8 shows a view selection state in the view selector 112.
[0072]
In addition, the scalers 113-1, 113-2 and 113-3
respectively perform a scaling process on the image data
items VC, VL and VR, so as to obtain, for example, image
data items VC', VL' and VR' of a full HD size of 1920*1080.
In this case, when the image data items VC, VL and VR have
the full HD size of 1920*1080, the image data items are
output as they are. Further, when the image data items VC,
VL and VR are greater than the size of 1920*1080, the image
data items are scaled down and are then output.
[0073]
The video encoder 114-1 performs coding such as, for
example, MPEG4-AVC (MVC) or MPEG2video on the image data VC'
of the center view so as to obtain coded video data. In
addition, the video encoder 114-1 generates a video stream
which includes the coded data as a substream (sub stream 1)
by using a stream formatter (not shown) which is provided in
the subsequent stage.
[0074]
44
In addition, the video encoder 114-2 performs coding
such as, for example, MPEG4-AVC (MVC) or MPEG2video on the
image data VL' of the left end view so as to obtain coded
video data. In addition, the video encoder 114-2 generates
a video stream which includes the coded data as a substream
(sub stream 2) by using a stream formatter (not shown) which
is provided in the subsequent stage.
[0075]
Further, the video encoder 114-3 performs coding such
as, for example, MPEG4-AVC (MVC) or MPEG2video on the image
data VR' of the right end view so as to obtain coded video
data. In addition, the video encoder 114-3 generates a
video stream which includes the coded data as a substream
(sub stream 3) by using a stream formatter (not shown) which
is provided in the subsequent stage.
[0076]
The video encoders 114-1, 114-2 and 114-3 insert the
above-described view configuration information into the
layer of the video stream. The view configuration
information includes, as described above, information
indicating whether or not image data included in a
corresponding video stream is image data of a portion of
views forming 3D. Here, this information indicates that
image data included in a corresponding video stream is image
data of a portion of views forming 3D.
45
[0077]
Further, this view configuration information includes
information indicating image data of which view is image
data included in a corresponding video stream, information
indicating whether data of a plurality of pictures is coded
in a single access unit of the corresponding video stream,
and the like. This view configuration information is
inserted into, for example, a user data region of a picture
header or a sequence header of a video stream.
[0078]
The disparity data generation portion 116 generates
disparity data on the basis of the image data of each of the
center, left end and right end views output from the view
selector 112. The disparity data includes, for example,
disparity data between the center view and the left end view
and disparity data between the center view and the right end
view. In this case, disparity data is generated with the
pixel unit or the block unit. Fig. 9 shows an example of
disparity data (disparity vector) for each block.
[0079]
Fig. 10 shows an example of a method of generating
disparity data of the block unit. This example is an
example in which disparity data indicating a j-th view is
obtained from an i-th view. In this case, pixel blocks
(disparity detection blocks) such as, for example, 4*4, 8*8,
46
or 16*16 are set in a picture of the i-th view.
[0080]
As shown in the figure, the picture of the i-th view is
a detection image, the picture of the j-th view is a
reference image, and a block of the picture of the j-th view
is searched such that a sum of absolute values of a
difference between pixels becomes the minimum, for each
block of the picture of the i-th view, thereby obtaining
disparity data.
[0081]
In other words, disparity data DPn of the N-th block is
obtained through block search such that a sum of difference
absolute values in the N-th block becomes the minimum as
represented in the following Equation (1). In addition, in
Equation (1), Dj indicates a pixel value in the picture of
the j-th view, and Di indicates a pixel value in the picture
of the i-th view.
DPn=min(abs(differ(Dj-Di))) (1)
[0082]
Fig. 11 shows an example of a method of generating
disparity data of the pixel unit. This example corresponds
to a method of generating disparity data of the pixel unit
by replacing the block unit with the pixel unit. "A", "B",
"C", "D", and "X" in Fig. 11(a) respectively indicate block
regions.
47
[0083]
From disparity data of the blocks, disparity data of
each of four regions into which the block "X" is divided is
obtained using the following Equation (2), as shown in Fig.
11(b). For example, disparity data X(A, B) of the divided
region adjacent to "A" and "B" is a median of disparity data
of the blocks "A", "B" and "X". This is also the same for
the other divided regions and thus disparity data is
obtained.
[0084]
X(A, B)=median(X, A, B)
X(A, C)=median(X, A, C)
X(B, D)=median(X, B, D)
X(C, D)=median(X, C, D) (2)
[0085]
Through the above-described one conversion, a region
occupied by the disparity data is reduced to a size of 1/2
of the original width and height size. By repeatedly
performing the conversion a predetermined number of times,
disparity data of the pixel unit is obtained based on the
block size. In addition, in a case where an edge is
included in a texture, complexity of an object in a screen
is higher than other portions, or the like, it is possible
to improve texture followability of disparity data itself of
the initial block unit by appropriately setting a block size
48
to be small.
[0086]
The disparity encoder 117 performs coding on the
disparity data generated by the disparity data generation
portion 116 so as to generate a disparity stream (disparity
data elementary stream). This disparity stream includes
disparity data of the pixel unit or the block unit. In a
case where the disparity data is the pixel unit, the
disparity data can be compression-coded and be transmitted
in the same as pixel data.
[0087]
In addition, in a case where disparity data of the
block unit is included in this disparity stream, a reception
side performs the above-described conversion process so as
to be converted into the pixel unit. Further, in a case
where this disparity stream is not transmitted, as described
above, the reception side may obtain disparity data of the
block unit between the respective views and further perform
conversion into the pixel unit.
[0088]
The graphics data output portion 118 outputs data of
graphics (also including subtitles as a caption)
superimposed on an image. The graphics encoder 119
generates a graphics stream (graphics elementary stream)
including the graphics data output from the graphics data
49
output portion 118. Here, the graphics form superimposition
information, and are, for example, a logo, a caption, and
the like.
[0089]
In addition, the graphics data output from the graphics
data output portion 118 is, for example, data of graphics
superimposed on an image of the center view. The graphics
encoder 119 may create data of graphics superimposed on the
left end and right end views on the basis of the disparity
data generated by the disparity data generation portion 116,
and may generate a graphics stream including the graphics
data. In this case, it is not necessary for the reception
side to create data of graphics superimposed on the left end
and right end views.
[0090]
The graphics data is mainly bitmap data. Offset
information indicating a superimposed position on an image
is added to the graphics data. The offset information
indicates, for example, an offset value in a vertical
direction and a horizontal direction from the origin on an
upper left of an image to a pixel on an upper left of a
superimposed position of graphics. In addition, a standard
in which caption data is transmitted as bitmap data is
operated, for example, through standardization as
"DVB_Subtitling" with DVB which is a European digital
50
broadcast standard.
[0091]
The audio data output portion 120 outputs audio data
corresponding to image data. The audio data output portion
120 is constituted by, for example, an audio data reading
portion which reads audio data from a microphone or a
storage medium so as to be output. The audio encoder 121
performs coding such as MPEG-2Audio or AAC on the audio data
output from the audio data output portion 120 so as to
generate an audio stream (audio elementary stream).
[0092]
The multiplexer 115 packetizes and multiplexes the
respective elementary streams generated by the video
encoders 114-1, 114-2 and 114-3, the disparity encoder 117,
the graphics encoder 119, and the audio encoder 121 so as to
generate a transport stream TS. In this case, Presentation
Time Stamp (PTS) is inserted into a header of each
Packetized Elementary Stream (PES) such that synchronous
reproduction is performed in the reception side.
[0093]
The multiplexer 115 inserts the above-described
identification information into a layer of the transport
stream TS. This identification information is information
for identifying whether or not view configuration
information is inserted into a layer of a video stream.
51
This identification information is inserted, for example,
under a video elementary loop (Video ES loop) of a Program
Map Table (PMT) included in the transport stream TS, an
Event Information Table (EIT), or the like.
[0094]
Next, a description will be made of a case where a twodimensional
(2D) image is transmitted. Any one of the image
data output portions 111-1 to 111-N outputs two-dimensional
image data. The view selector 112 extracts the twodimensional
image data. The scaler 113-1 performs a scaling
process on the two-dimensional image data extracted by the
view selector 112, so as to obtain, for example, twodimensional
image data of a full HD size of 1920*1080. In
this case, the scalers 113-1 and 113-2 are in a nonoperation
state.
[0095]
The video encoder 114-1 performs coding such as, for
example, MPEG4-AVC (MVC) or MPEG2video on the twodimensional
image data so as to obtain coded video data. In
addition, the video encoder 114-1 generates a video stream
which includes the coded data as a substream (sub stream 1)
by using a stream formatter (not shown) which is provided in
the subsequent stage. In this case, the video encoders 114-
1 and 114-2 are in a non-operation state.
[0096]
52
The video encoder 114-1 inserts the above-described
view configuration information into the layer of the video
stream. The view configuration information includes, as
described above, information indicating whether or not image
data included in a corresponding video stream is image data
of a portion of views forming 3D. Here, this information
indicates that image data included in a corresponding video
stream is not image data of a portion of views forming 3D.
For this reason, the view configuration information does not
include other information. In addition, when a twodimensional
(2D) image is transmitted, it is considered that
the above-described view configuration information is not
inserted into the layer of the video stream.
[0097]
Although detailed description is omitted, the graphics
data output portion 118, the graphics encoder 119, the audio
data output portion 120, and the audio encoder 121 are the
same as in a case of transmitting a stereoscopic (3D) image.
In addition, the disparity data generation portion 116 and
the disparity encoder 117 are also in a non-operation state.
[0098]
The multiplexer 115 packetizes and multiplexes the
respective elementary streams generated by the video encoder
114-1, the graphics encoder 119, and the audio encoder 121
so as to generate a transport stream TS. In this case, a
53
Presentation Time Stamp (PTS) is inserted into a header of
each Packetized Elementary Stream (PES) such that
synchronous reproduction is performed in the reception side.
[0099]
An operation of the transmission data generation unit
110 shown in Fig. 7 will be described briefly. First, a
description will be made of an operation when a stereoscopic
(3D) image is transmitted. Image data of N views (View 1,…,
and View N) for stereoscopic image display, output from the
N image data output portions 111-1 to 111-N, is supplied to
the view selector 112. The view selector 112 extracts image
data VC of the center view, image data VL of the left end
view, and image data VR of the right end view from the image
data of the N views.
[0100]
The image data VC of the center view extracted from the
view selector 112 is supplied to the scaler 113-1 and
undergoes, for example, a scaling process to a full HD size
of 1920*1080. Image data VC' having undergone the scaling
process is supplied to the video encoder 114-1.
[0101]
The video encoder 114-1 performs coding on the image
data VC' so as to obtain coded video data, and generates a
video stream including the coded data as a substream (sub
stream 1). In addition, the video encoder 114-1 inserts
54
view configuration information into a user data region or
the like of a picture header or a sequence header of the
video stream. The video stream is supplied to the
multiplexer 115.
[0102]
In addition, the image data VL of the left end view
extracted from the view selector 112 is supplied to the
scaler 113-2 and undergoes, for example, a scaling process
to a full HD size of 1920*1080. Image data VL' having
undergone the scaling process is supplied to the video
encoder 114-2.
[0103]
The video encoder 114-2 performs coding on the image
data VL' so as to obtain coded video data, and generates a
video stream including the coded data as a substream (sub
stream 2). In addition, the video encoder 114-2 inserts
view configuration information into a user data region of a
picture header or a sequence header of the video stream.
The video stream is supplied to the multiplexer 115.
[0104]
In addition, the image data VR of the left end view
extracted from the view selector 112 is supplied to the
scaler 113-3 and undergoes, for example, a scaling process
to a full HD size of 1920*1080. Image data VR' having
undergone the scaling process is supplied to the video
55
encoder 114-3.
[0105]
The video encoder 114-3 performs coding on the image
data VR' so as to obtain coded video data, and generates a
video stream including the coded data as a substream (sub
stream 3). In addition, the video encoder 114-3 inserts
view configuration information into a user data region of a
picture header or a sequence header of the video stream.
The video stream is supplied to the multiplexer 115.
[0106]
Further, the image data of each of the center, left end
and right end views output from the view selector 112 is
supplied to the disparity data generation portion 116. The
disparity data generation portion 116 generates disparity
data on the basis of the image data of each view. The
disparity data includes disparity data between the center
view and the left end view and disparity data between the
center view and the right end view. In this case, disparity
data is generated with the pixel unit or the block unit.
[0107]
The disparity data generated by the disparity data
generation portion 116 is supplied to the disparity encoder
117. The disparity encoder 117 performs a coding process on
the disparity data so as to generate a disparity stream.
The disparity stream is supplied to the multiplexer 115.
56
[0108]
In addition, graphics data (also including subtitle
data) output from the graphics data output portion 118 is
supplied to the graphics encoder 119. The graphics encoder
119 generates a graphics stream including the graphics data.
The graphics stream is supplied to the multiplexer 115.
[0109]
In addition, audio data output from the audio data
output portion 120 is supplied to the audio encoder 121.
The audio encoder 121 performs coding such as MPEG-2Audio or
AAC on the audio data so as to generate an audio stream.
This audio stream is supplied to the multiplexer 115.
[0110]
The multiplexer 115 packetizes and multiplexes the
elementary streams supplied from the respective encoders so
as to generate a transport stream TS. In this case, a PTS
is inserted into each PES header such that synchronous
reproduction is performed in the reception side. Further,
the multiplexer 115 inserts identification information for
identifying whether or not view configuration information is
inserted into the layer of the video stream, under the PMT,
the EIT, or the like.
[0111]
In addition, in the transmission data generation unit
110 shown in Fig. 7, a case where three video streams are
57
included in the transport stream TS is shown. In other
words, the transport stream TS includes three video streams
obtained by coding each of image data items of the center,
left end and right end views as a single picture.
[0112]
Although detailed description is omitted, as described
above, a case where two or one video stream is included in
the transport stream TS can be configured in the same manner.
In a case where two video streams are included in the
transport stream TS, for example, the following video
streams are included. In other words, the video streams are
a video stream which is obtained by coding image data of the
center view as a single picture and a video stream which is
obtained by performing an interleaving process on image data
items of the left end view and the right end view so as to
be coded as a single picture.
[0113]
Further, in a case where a single video stream is
included in the transport stream TS, for example, the
following video stream is included. In other words, the
video stream includes a video stream including data obtained
by coding each of image data items of the center, left end
and right end views as data of a single picture.
[0114]
Next, a description will be made of an operation when a
58
two-dimensional (2D) image is transmitted. Two-dimensional
image data is output from any one of the image data output
portions 111-1 to 111-N. The view selector 112 extracts the
two-dimensional image data which is supplied to the scaler
113-1. The scaler 113-1 performs a scaling process on the
two-dimensional image data extracted from the view selector
112, so as to obtain, for example, two-dimensional image
data of a full HD size of 1920*1080. The two-dimensional
image data having undergone the scaling is supplied to the
video encoder 114-1.
[0115]
The video encoder 114-1 performs coding such as, for
example, MPEG4-AVC (MVC) or MPEG2video on the twodimensional
image data so as to obtain coded video data. In
addition, the video encoder 114-1 generates a video stream
which includes the coded data as a substream (sub stream 1)
by using a stream formatter (not shown) which is provided in
the subsequent stage.
[0116]
The video encoder 114-1 inserts the above-described
view configuration information into the layer of the video
stream. The view configuration information includes, as
described above, information indicating whether or not image
data included in a corresponding video stream is image data
of a portion of views forming 3D. Here, this information
59
indicates that image data included in a corresponding video
stream is not image data of a portion of views forming 3D,
that is, two-dimensional image data. The multiplexer 115
packetizes and multiplexes the respective elementary streams
generated by the video encoder 114-1, the graphics encoder
119, and the audio encoder 121 so as to generate a transport
stream TS.
[0117]
[Structures of identification information and view
configuration information and TS configuration]
As described above, identification information for
identifying whether or not view configuration information is
inserted into a layer of a video stream is inserted into a
layer of the transport stream TS. Fig. 12 shows a
structural example (Syntax) of a multi-view stream
configuration descriptor
(multiview_stream_configuration_descriptor) which is
identification information. In addition, Fig. 13 shows
content (Semantics) of principal information in the
structural example shown in Fig. 12.
[0118]
"multiview_stream_configuration_tag" is 8-bit data
indicating a descriptor type, and, here, indicates a multiview
stream configuration descriptor.
"multiview_stream_configuration_length" is 8-bit data
60
indicating a length (size) of a descriptor. This data is a
length of the descriptor, and indicates the number of
subsequent bytes.
[0119]
The 1-bit field of "multiview_stream_checkflag"
indicates whether or not view configuration information is
inserted into a layer of a video stream. "1" indicates that
view configuration information is inserted into a layer of a
video stream, and "0" indicates that view configuration
information is not inserted into a layer of a video stream.
If "1", a reception side (decoder) checks view configuration
information which is present in a user data region.
[0120]
In addition, as described above, view configuration
information including information and the like indicating
whether or not image data included in a corresponding video
stream is image data of a portion of views forming 3D is
inserted into the layer of the video stream. As described
above, the view configuration information is necessarily
inserted when a stereoscopic (3D) image is transmitted, and
may not be inserted when a two-dimensional (2D) image is
transmitted. Fig. 14 shows a structural example (Syntax) of
multi-view stream configuration information
(multiview_stream_configuration_info()) which is the view
configuration information. In addition, Figs. 15, 16 and 17
61
show content (Semantics) of principal information in the
structural example shown in Fig. 14.
[0121]
The 1-bit field of "3D_flag" indicates whether or not
image data included in a coded video stream is image data of
a portion of views forming 3D. "1" indicates that image
data is image data of a portion of views, and "0" indicates
that image data is not image data of a portion of views.
[0122]
If "3D_flag=1", each piece of information of
"view_count", "single_view_es_flag", and
"view_interleaving_flag" is present. The 4-bit field of
"view_count" indicates the number of views forming a 3D
service. The minimum value thereof is 1, and the maximum
value thereof is 15. The 1-bit field of
"single_view_es_flag" indicates whether or not data of a
plurality of pictures is coded in a single access unit of a
corresponding video stream. "1" indicates that data of only
a single picture is coded, and "0" indicates that data of
two or more pictures is coded.
[0123]
The 1-bit field of "view_interleaving_flag" indicates
whether or not image data of two views undergoes an
interleaving process and is coded as data of a single
picture in a corresponding video stream. "1" indicates that
62
image data undergoes an interleaving process and forms a
screen split, and "0" indicates that an interleaving process
is not performed.
[0124]
If "view_interleaving_flag=0", information of
"view_allocation" is present. The 4-bit field of
"view_allocation" indicates image data of which view is
image data included in a corresponding video stream, that is,
view allocation. For example, "0000" indicates a center
view. In addition, for example, "0001" indicates a first
left view next to the center. Further, for example, "0010"
indicates a first right view next to the center. This
"view_allocation" forms information indicating a relative
positional relationship of each view.
[0125]
If "view_interleaving_flag=1", information of
"view_pair_position_id" and "view_interleaving_type" is
present. The 3-bit field of "view_pair_position_id"
indicates relative view positions of two views in the
overall views. In this case, for example, a earlier
position in scanning order is set to left, and a later
position is set to right. For example, "000" indicates a
pair of two views located at both ends. In addition, for
example, "001" indicates a pair of two views located inward
by one from both ends. Further, for example, "010"
63
indicates a pair of two views located inward by one from
both ends.
[0126]
The 1-bit field of "view_interleaving_type" indicates
an interleaving type. "1" indicates that an interleaving
type is a side-by-side type, and "0" indicates that an
interleaving type is a top-and-bottom type.
[0127]
In addition, if "3D_flag=1", each piece of information
of "display_flag",
"indication_of_picture_size_scaling_horizontal", and
"indication_of_picture_size_scaling_vertical" is present.
The 1-bit field of "display_flag" indicates whether or not a
corresponding view is essentially displayed when an image is
displayed. "1" indicates that a view is essentially
displayed. On the other hand, "0" indicates that a view is
not essentially displayed.
[0128]
The 4-bit field of
"indication_of_picture_size_scaling_horizontal" indicates a
horizontal pixel ratio of a decoded images relative to full
HD (1920). "0000" indicates 100%, "0001" indicates 80%,
"0010" indicates 75%, "0011" indicates 66%, "0100" indicates
50%, "0101" indicates 33%, "0110" indicates 25%, and "0111"
indicates 20%.
64
[0129]
The 4-bit field of
"indication_of_picture_size_scaling_vertical" indicates a
vertical pixel ratio of a decoded images relative to full HD
(1080). "0000" indicates 100%, "0001" indicates 80%, "0010"
indicates 75%, "0011" indicates 66%, "0100" indicates 50%,
"0101" indicates 33%, "0110" indicates 25%, and "0111"
indicates 20%.
[0130]
Fig. 18 shows an example of a relationship between the
number of views indicated by "view_count" and positions of
two views (here, "View 1" and "View 2") indicated by
"view_pair_position_id". An example of (1) is a case where
the number of views indicated by "view_count" is 2, and
"view_pair_position_id=000" indicates two views located at
both ends. In addition, an example of (2) is a case where
the number of views indicated by "view_count" is 4, and
"view_pair_position_id=000" indicates two views located at
both ends.
[0131]
Further, an example of (3) is a case where the number
of views indicated by "view_count" is 4, and
"view_pair_position_id=001" indicates two views located
inward by one from both ends. Furthermore, an example of
(4) is a case where the number of views indicated by
65
"view_count" is 5, and "view_pair_position_id=000" indicates
two views located at both ends.
[0132]
In addition, an example of (5) is a case where the
number of views indicated by "view_count" is 9, and
"view_pair_position_id=000" indicates two views located at
both ends. Further, an example of (6) is a case where the
number of views indicated by "view_count" is 9, and
"view_pair_position_id=010" indicates two views located
inward by two from both ends.
[0133]
A pair of views located further inward than both ends
can be transmitted additionally to a pair of views at both
ends in order to improve a performance of interpolation and
generation in a case where two views at both ends are
unlikely to satisfy sufficient image quality when a
reception side combines views. At this time, coded video
data of a pair of views which is additionally transmitted
may be coded so as to share an access unit in a stream of a
pair of views at both ends, or may be coded as another
stream.
[0134]
Fig. 19 shows an example in which a transmission side
or a reception side generates disparity data in a case where
image data of a pair of two views located further inward
66
than both ends is transmitted along with image data of two
views located at both ends as described above. In the shown
example, the number of views indicated by "view_count" is 9.
In addition, a substream (sub stream 1) including image data
of two views (View 1 and View 2) at both ends and a
substream (sub stream 2) including image data of two views
(View 3 and View 4) located further inward than those are
present.
[0135]
In this case, first, disparity data of "View 1" and
"View 3" is calculated. Next, disparity data of "View 2"
and "View 4" is calculated. Finally, disparity data of
"View 3" and "View 4" is calculated. In addition, in a case
where resolutions of views are different between substreams,
a resolution is unified to either one, and then disparity
data is calculated.
[0136]
Fig. 20 shows an example in which the reception side
interpolates and generates image data of a view located
between the respective views on the basis of the disparity
data calculated as described above. In this case, first,
"View_A" located between "View 1" and "View 3" is
interpolated and generated using the disparity data between
"View 1" and "View 3".
[0137]
67
Next, "View_B" located between "View 2" and "View 4" is
interpolated and generated using the disparity data between
"View 2" and "View 4". Finally, "View_C", "View_D", and
"View_E" located between "View 3" and "View 4" are
interpolated and generated using the disparity data between
"View 3" and "View 4".
[0138]
Next, a description will be made of a case where the
multi-view stream configuration information
(multiview_stream_configuration_info()) which is the view
configuration information is inserted into a user data
region of the video stream (video elementary stream). In
this case, the multi-view stream configuration information
is inserted, for example, with the picture unit or the GOP
unit by using the user data region.
[0139]
For example, in a case where a coding type is AVC or
MVC, or even in a case of a coding type in which a coding
structure of an NAL packet or the like is similar such as
HEVC, the multi-view stream configuration information is
inserted into the "SEIs" part of the access unit as "Multiview
stream configuration SEI message". Fig. 21(a) shows a
leading access unit of Group Of Pictures (GOP), and Fig.
21(b) shows access units other than the leading access unit
of the GOP. In a case where the multi-view stream
68
configuration information is inserted with the GOP unit,
"Multi-view stream configuration SEI message" is inserted
only into the leading access unit of the GOP.
[0140]
Fig. 22(a) shows a structural example (Syntax) of
"Multi-view stream configuration SEI message".
"uuid_iso_iec_11578" has a UUID value indicated by "ISO/IEC
11578:1996 Annex A.".
"userdata_for_multiview_stream_configuration()" is inserted
into the field of "user_data_payload_byte". Fig. 22(b)
shows a structural example (Syntax) of
"userdata_for_multiview_stream_configuration()". The multiview
stream configuration information
(multiview_stream_configuration_info()) is inserted
thereinto (refer to Fig. 14). "userdata_id" is an
identifier of the multi-view stream configuration
information, represented by unsigned 16 bits.
[0141]
In addition, for example, in a case where a coding type
is MPEG2video, the multi-view stream configuration
information is inserted into a user data region of a picture
header part as user data "user_data()". Fig. 23(a) shows a
structural example (Syntax) of "user_data()". The 32-bit
filed of "user_data_start_code" is a start code of user data
(user_data) and is a fixed value of "0x000001B2".
69
[0142]
The 32-bit field subsequent to the start code is an
identifier for identifying content of user data. Here, the
identifier is "Stereo_Video_Format_Signaling_identifier" and
enables user data to be identified as multi-view stream
configuration information.
"Multiview_stream_configuration()" which is stream
correlation information is inserted subsequent to the
identifier as a data body. Fig. 23(b) shows a structural
example (Syntax) of "Multiview_stream_configuration()". The
multi-view stream configuration information
(multiview_stream_configuration_info()) is inserted
thereinto (refer to Fig. 14).
[0143]
The multi-view stream configuration descriptor
(multiview_stream_configuration_descriptor) which is
identification information shown in Fig. 12 described above
is inserted into a layer of the transport stream TS, for
example, under the PMT, under the EIT, or the like. In
other words, the descriptor is disposed at an optimal
position with the event unit or in a use case which is
static or dynamic.
[0144]
Fig. 24 shows a configuration example of the transport
stream TS when a stereoscopic (3D) image is transmitted. In
70
addition, in this configuration example, for simplification
of the figure, disparity data, audio, graphics, and the like
are not shown. This configuration example shows a case
where three video streams are included in the transport
stream TS. In other words, the transport stream TS includes
three video streams which are obtained by coding each of
image data items of center, left end and right end views as
a single picture. In addition, this configuration example
shows a case where the number of views is 5.
[0145]
The configuration example of Fig. 24 includes a PES
packet "video PES1" of a video stream in which the image
data VC' of the center view is coded as a single picture.
The multi-view stream configuration information inserted
into the user data region of the video stream indicates that
the number of views indicated by "View_count" is 5.
[0146]
In addition, in this information, there is
"single_view_es_flag=1" which indicates that data of only a
single picture is coded in a single access unit in the video
stream. Further, in this information, there is
"View_interleaving_flag=0" which indicates that image data
of two views does not undergo an interleaving process and is
not coded as data of a single picture in the video stream.
In addition, there is "view_allocation=0000" which indicates
71
that the image data included in the video stream is image
data of the center view.
[0147]
Further, the configuration example of Fig. 24 includes
a PES packet "video PES2" of a video stream in which the
image data VL' of the left end view is coded as a single
picture. The multi-view stream configuration information
inserted into the user data region of the video stream
indicates that the number of views indicated by "View_count"
is 5.
[0148]
In addition, in this information, there is
"single_view_es_flag=1" which indicates that data of only a
single picture is coded in a single access unit in the video
stream. Further, in this information, there is
"View_interleaving_flag=0" which indicates that image data
of two views does not undergo an interleaving process and is
not coded as data of a single picture in the video stream.
In addition, there is "view_allocation=0011" which indicates
that the image data included in the video stream is image
data of a second left view next to the center, that is, the
left end view.
[0149]
Further, the configuration example of Fig. 24 includes
a PES packet "video PES3" of a video stream in which the
72
image data VR' of the left end view is coded as a single
picture. The multi-view stream configuration information
inserted into the user data region of the video stream
indicates that the number of views indicated by "View_count"
is 5.
[0150]
In addition, in this information, there is
"single_view_es_flag=1" which indicates that data of only a
single picture is coded in a single access unit in the video
stream. Further, in this information, there is
"View_interleaving_flag=0" which indicates that image data
of two views does not undergo an interleaving process and is
not coded as data of a single picture in the video stream.
In addition, there is "view_allocation=0100" which indicates
that the image data included in the video stream is image
data of a second right view next to the center, that is, the
right end view.
[0151]
In addition, the transport stream TS includes a Program
Map Table (PMT) which is Program Specific Information (PSI).
The PSI is information describing to which program each
elementary stream included in the transport stream belongs.
In addition, the transport stream includes Event Information
Table (EIT) which is Serviced Information (SI) for
performing management of the event unit.
73
[0152]
An elementary loop which has information related to
each elementary stream is present in the PMT. In this
configuration example, a video elementary loop (Video ES
loop) is present. In the elementary loop, information such
as a packet identifier (PID) is disposed, and a descriptor
describing information related to the elementary stream is
also disposed for each stream.
[0153]
In this configuration example, a multi-view stream
configuration descriptor
(multiview_stream_configuration_descriptor) is inserted
under the video elementary loop (Video ES loop) of the PMT
in relation to each video stream. In this descriptor, there
is "multiview_stream_checkflag=1" which indicates the
presence of the multi-view stream configuration information
which is view configuration information in the user region
of the video stream. In addition, the descriptor may be
inserted under the EIT as indicated by the broken line.
[0154]
In addition, Fig. 25 also shows a configuration example
of the transport stream TS when a stereoscopic (3D) image is
transmitted. Further, also in this configuration example,
for simplification of the figure, disparity data, audio,
graphics, and the like are not shown. This configuration
74
example shows a case where two video streams are included in
the transport stream TS. In other words, the transport
stream TS includes a video stream which is obtained by
coding each of image data items of a center view as a single
picture. In addition, the transport stream TS includes a
video stream which is obtained by coding image data of a
left end view and a right end view undergoes an interleaving
process and is coded as a single picture. In addition, this
configuration example also shows a case where the number of
views is 5.
[0155]
The configuration example of Fig. 25 includes a PES
packet "video PES1" of a video stream in which the image
data VC' of the center view is coded as a single picture.
The multi-view stream configuration information inserted
into the user data region of the video stream indicates that
the number of views indicated by "View_count" is 5.
[0156]
In addition, in this information, there is
"single_view_es_flag=1" which indicates that data of only a
single picture is coded in a single access unit in the video
stream. Further, in this information, there is
"View_interleaving_flag=0" which indicates that image data
of two views does not undergo an interleaving process and is
not coded as data of a single picture in the video stream.
75
In addition, there is "view_allocation=0000" which indicates
that the image data included in the video stream is image
data of the center view.
[0157]
The configuration example of Fig. 25 includes a PES
packet "video PES2" of a video stream in which the image
data VL' of the left end view and the image data VR' of the
right end view is coded as a single picture. The multi-view
stream configuration information inserted into the user data
region of the video stream indicates that the number of
views indicated by "View_count" is 5.
[0158]
In addition, in this information, there is
"single_view_es_flag=1" which indicates that data of only a
single picture is coded in a single access unit in the video
stream. Further, in this information, there is
"View_interleaving_flag=1" which indicates that image data
of two views undergoes an interleaving process and is coded
as data of a single picture in the video stream. In
addition, there is "view_pair_position_id=000" which
indicates a pair of two views at both ends. Further, there
is "view_interleaving_type=1" which indicates that an
interleaving type is a side-by-side type.
[0159]
Further, in this configuration example, a multi-view
76
stream configuration descriptor
(multiview_stream_configuration_descriptor) is inserted
under the video elementary loop (Video ES loop) of the PMT
in relation to each video stream. In this descriptor, there
is "multiview_stream_checkflag=1" which indicates the
presence of the multi-view stream configuration information
which is view configuration information in the user region
of the video stream. In addition, the descriptor may be
inserted under the EIT as indicated by the broken line.
[0160]
In addition, Fig. 26 also shows a configuration example
of the transport stream TS when a stereoscopic (3D) image is
transmitted. Further, also in this configuration example,
for simplification of the figure, disparity data, audio,
graphics, and the like are not shown. This configuration
example shows a case where a single video stream is included
in the transport stream TS. In other words, the transport
stream TS includes a video stream including data which is
obtained by coding each of image data items of center, left
end and right end views as a single picture. In addition,
this configuration example also shows a case where the
number of views is 5.
[0161]
The configuration example of Fig. 26 includes a PES
packet "video PES1" of a single video stream. The video
77
stream includes data in which image data of each of the
center, left end and right end views is coded as data of a
single picture in a single access unit, and a user data
region is present so as to correspond to each picture. In
addition, multi-view stream configuration information is
inserted into each user data region.
[0162]
The information corresponding to the picture data
obtained by coding image data of the center view indicates
that the number of views indicated by "View_count" is 5. In
addition, in this information, there is
"single_view_es_flag=0" which indicates that data of a
plurality of pictures is coded in a single access unit in
the video stream. Further, in this information, there is
"View_interleaving_flag=0" which indicates that the picture
data is not image data of two views which undergoes an
interleaving process and is coded. In addition, there is
"view_allocation=0000" which indicates that the image data
included in the picture data is image data of the center
view.
[0163]
Further, the information corresponding to the picture
data obtained by coding image data of the left end view
indicates that the number of views indicated by "View_count"
is 5. In addition, in this information, there is
78
"single_view_es_flag=0" which indicates that data of a
plurality of pictures is coded in a single access unit in
the video stream. Further, in this information, there is
"View_interleaving_flag=0" which indicates that the picture
data is not image data of two views which undergoes an
interleaving process and is coded. In addition, there is
"view_allocation=0011" which indicates that the image data
included in the picture data is image data of a second left
view next to the center, that is, the left end view.
[0164]
In addition, the information corresponding to the
picture data obtained by coding image data of the right end
view indicates that the number of views indicated by
"View_count" is 5. In addition, in this information, there
is "single_view_es_flag=0" which indicates that data of a
plurality of pictures is coded in a single access unit in
the video stream. Further, in this information, there is
"View_interleaving_flag=0" which indicates that the picture
data is not image data of two views which undergoes an
interleaving process and is coded. In addition, there is
"view_allocation=0100" which indicates that the image data
included in the picture data is image data of a second right
view next to the center, that is, the right end view.
[0165]
Further, in this configuration example, a multi-view
79
stream configuration descriptor
(multiview_stream_configuration_descriptor) is inserted
under the video elementary loop (Video ES loop) of the PMT
in relation to a single video stream. In this descriptor,
there is "multiview_stream_checkflag=1" which indicates the
presence of the multi-view stream configuration information
which is view configuration information in the user region
of the video stream. In addition, the descriptor may be
inserted under the EIT as indicated by the broken line.
[0166]
As described above, the transmission data generation
unit 110 shown in Fig. 7 generates a transport stream TS
including a video stream which is obtained by coding at
least image data of a left end view and a right end view and
image data of an intermediate view located between the left
end and the right end among a plurality of views for
stereoscopic image display when a stereoscopic (3D) image is
transmitted. For this reason, it is possible to effectively
transmit image data for observing a stereoscopic image
formed by multi-views with the naked eye.
[0167]
In other words, since not only image data of the left
end view and the right end view but also image data of the
intermediate view is transmitted, a relative disparity
between views is small, a periphery of occlusion according
80
to processing of a fine part when image data of other views
is interpolated is easily interpolated, and thereby it is
possible to improve quality of a reproduced image. In
addition, since image data of the left end view and the
right end view is transmitted, image data of a view which is
not transmitted can be generated through an interpolation
process, and thus it is possible to easily maintain high
image quality with regard to processing of an end point of
occlusion or the like.
[0168]
In addition, in the transmission data generation unit
110 shown in Fig. 7, when a stereoscopic (3D) image is
transmitted, the multi-view stream configuration information
(multiview_stream_configuration_info()) which is view
configuration information is necessarily inserted into a
layer of a video stream. For this reason, a reception side
can perform an appropriate and efficient process for
observing a three-dimensional image (stereoscopic image)
formed by image data of a plurality of views with the naked
eye on the basis of this view configuration information.
[0169]
In addition, in the transmission data generation unit
110 shown in Fig. 7, the multi-view stream configuration
descriptor (multiview_stream_configuration_descriptor) is
inserted into a layer of the transport stream TS. This
81
descriptor forms identification information for identifying
whether or not view configuration information is inserted
into a layer of a video stream. A reception side can easily
identify whether or not view configuration information is
inserted into a layer of a video stream on the basis of this
identification information. For this reason, it is possible
to efficiently extract the view configuration information
from a user data region of the video stream.
[0170]
In addition, in the transmission data generation unit
110 shown in Fig. 7, the disparity data generation portion
116 generates disparity data between respective views, and a
disparity stream obtained by coding the disparity data is
included in the transport stream TS along with a video
stream. For this reason, a reception side can easily
interpolate and generate image data of each view which is
not transmitted, on the basis of the sent disparity data,
without performing a process of generating disparity data
from the received image data of each view.
[0171]
(Configuration example of receiver)
Fig. 27 shows a configuration example of the receiver
200. The receiver 200 includes a CPU 201, a flash ROM 202,
a DRAM 203, an internal bus 204, a remote control reception
unit (RC reception unit) 205, a remote control transmitter
82
(RC transmitter) 206. In addition, the receiver 200
includes an antenna terminal 211, a digital tuner 212, a
transport stream buffer (TS buffer) 213, and a demultiplexer
214.
[0172]
Further, the receiver 200 includes coded buffers 215-1,
215-2 and 215-3, video decoders 216-1, 216-2 and 216-3,
decoded buffers 217-1, 217-2 and 217-3, and scalers 218-1,
218-2 and 218-3. In addition, the receiver 200 includes a
view interpolation unit 219 and a pixel
interleaving/superimposing unit 220. Furthermore, the
receiver 200 includes a coded buffer 221, a disparity
decoder 222, a disparity buffer 223, and a disparity data
conversion unit 224.
[0173]
In addition, the receiver 200 includes a coded buffer
225, a graphics decoder 226, a pixel buffer 227, a scaler
228, and a graphics shifter 229. Further, the receiver 200
includes a coded buffer 230, an audio decoder 231, and a
channel mixing unit 232.
[0174]
The CPU 201 controls an operation of each unit of the
receiver 200. The flash ROM 202 stores control software and
preserves data. The DRAM 203 forms a work area of the CPU
201. The CPU 201 develops software or data read from the
83
flash ROM 202 on the DRAM 203, and activates the software so
as to control each unit of the receiver 200. The RC
reception unit 205 receives a remote control signal (remote
control code) transmitted from the RC transmitter 206 so as
to be supplied to the CPU 201. The CPU 201 controls each
unit of the receiver 200 on the basis of this remote control
code. The CPU 201, the flash ROM 202, and the DRAM 203 are
connected to the internal bus 204.
[0175]
Hereinafter, first, a description will be made of a
case where a stereoscopic (3D) image is received. The
antenna terminal 211 is a terminal to which a television
broadcast signal received by a reception antenna (not shown)
is input. The digital tuner 212 processes the television
broadcast signal input to the antenna terminal 211, and
outputs a predetermined transport stream (bitstream data) TS
corresponding to a channel selected by a user. The
transport stream buffer (TS buffer) 213 temporarily
accumulates the transport stream TS output from the digital
tuner 212.
[0176]
The transport stream TS includes video streams obtained
by coding image data of a left end view and a right end view
and image data of a center view which is an intermediate
view located between the left end and the right end among a
84
plurality of views for stereoscopic image display.
[0177]
In this case, the transport stream TS may include three,
two or one video stream (refer to Figs. 24, 25 and 26).
Here, for convenience of description, the description will
be made assuming that the transport stream TS includes three
video streams obtained by coding image data of each of the
center, left end and right end views as a single picture.
[0178]
In the transport stream TS, as described above, the
multi-view stream configuration descriptor
(multiview_stream_configuration_descriptor) is inserted
under the PMT, under the EIT, or the like. The descriptor
is identification information for identifying whether or not
view configuration information, that is, the multi-view
stream configuration information
(multiview_stream_configuration_info()) is inserted into a
layer of a video stream.
[0179]
The demultiplexer 214 extracts each of elementary
streams of video, disparity, graphics, and audio from the
transport stream TS which is temporarily accumulated in the
TS buffer 213. In addition, the demultiplexer 214 extracts
the above-described multi-view stream configuration
descriptor from the transport stream TS so as to be sent to
85
the CPU 201. The CPU 201 can easily determine whether or
not view configuration information is inserted into the
layer of the video stream on the basis of the 1-bit field of
"multiview_stream_checkflag" of the descriptor.
[0180]
The coded buffers 215-1, 215-2 and 215-3 respectively
temporarily accumulate the video streams which are obtained
by coding image data of each of the center, left end and
right end views as a single picture and are extracted by the
demultiplexer 214. The video decoders 216-1, 216-2 and 216-
3 respectively perform a decoding process on the video
streams stored in the coded buffers 215-1, 215-2 and 215-3
under the control of the CPU 201 so as to acquire image data
of each of the center, left end and right end views.
[0181]
Here, the video decoder 216-1 performs a decoding
process using a compressed data buffer so as to acquire
image data of the center view (center view). In addition,
the video decoder 216-2 performs a decoding process using a
compressed data buffer so as to acquire image data of the
left end view (left view). Further, the video decoder 216-3
performs a decoding process using a compressed data buffer
so as to acquire image data of the right end view (right
view). Furthermore, in a case where two or more views are
interleaved and are coded, the coded buffers, the video
86
decoders, decoded buffers, and the scalers are allocated
with the stream unit.
[0182]
Each video decoder extracts the multi-view stream
configuration information
(multiview_stream_configuration_info()) which is view
configuration information and is inserted into the user data
region or the like of the picture header or the sequence
header of the video stream so as to be sent to the CPU 201.
The CPU 201 performs an appropriate and efficient process
for observing a three-dimensional image (stereoscopic image)
formed by image data of a plurality of views with the naked
eye on the basis of this view configuration information.
[0183]
In other words, the CPU 201 controls operations of the
demultiplexer 214, the video decoders 216-1, 216-2 and 216-3,
the scalers 218-1, 218-2 and 218-3, the view interpolation
unit 219, and the like, with the program unit, the scene
unit, the picture group unit, or the picture unit, on the
basis of the view configuration information. For example,
the CPU 201 can recognize the number of views forming a 3D
service on the basis of the 4-bit field of "view_count".
[0184]
In addition, for example, the CPU 201 can identify
whether or not data of a plurality of pictures is coded in a
87
single access unit of the video stream on the basis of the
1-bit field of "single_view_es_flag". Further, for example,
the CPU 201 can identify whether or not image data of two
views undergoes an interleaving process and is coded as data
of a single picture in the video stream on the basis of the
1-bit field of "view_interleaving_flag".
[0185]
In addition, for example, the CPU 201 can reconginze
image data of which view is image data included in the video
stream on the basis of the 4-bit field of "view_allocation"
when image data of two views does not undergo an
interleaving process and is not coded as data of a single
picture in the video stream.
[0186]
In addition, for example, the CPU 201 can recognize
relative view positions of two views in the overall views on
the basis of the 3-bit field of "view_pair_position_id" when
image data of two views undergoes an interleaving process
and is coded as data of a single picture in the video stream.
Further, at this time, the CPU 201 can understand an
interleaving type on the basis of the 1-bit field of
"view_interleaving_type".
[0187]
In addition, for example, the CPU 201 can recognize a
horizontal pixel ratio and a vertical pixel ratio of a
88
decoded image relative to the full HD on the basis of the 4-
bit field of "indication_of_picture_size_scaling_horizontal"
and the 4-bit field of
"indication_of_picture_size_scaling_vertical".
[0188]
The decoded buffers 217-1, 217-2 and 217-3 respectively
temporarily accumulate the image data items of the
respective views acquired by the video decoders 216-1, 216-2
and 216-3. The scalers 218-1, 218-2 and 218-3 respectively
adjust output resolutions of the image data items of the
respective views output from the decoded buffers 217-1, 217-
2 and 217-3 so as to be predetermined resolutions.
[0189]
In the multi-view stream configuration information, the
4-bit field of
"indication_of_picture_size_scaling_horizontal" which
indicates a horizontal pixel ratio of a decoded image and
the 4-bit field of
"indication_of_picture_size_scaling_vertical" which
indicates a vertical pixel ratio of a decoded image are
present. The CPU 201 controls scaling ratios in the scalers
218-1, 218-2 and 218-3 so as to obtain a predetermined
resolution on the basis of this pixel ratio information.
[0190]
In this case, the CPU 201 calculates scaling ratios for
89
the image data accumulated in the decoded buffers so as to
instruct the scalers 218-1, 218-2 and 218-3 on the basis of
a resolution of decoded image data, a resolution of a
monitor, and the number of views. Fig. 28 shows a
calculation example of a scaling ratio.
[0191]
For example, when a resolution of decoded image data is
960*1080, a resolution of a monitor is 1920*1080, and the
number of views to be displayed is 4, a scaling ratio is set
to 1/2. In addition, for example, when a resolution of
decoded image data is 1920*1080, a resolution of a monitor
is 1920*1080, and the number of views to be displayed is 4,
a scaling ratio is set to 1/4. Further, for example, when a
resolution of decoded image data is 1920*2160, a resolution
of a monitor is 3840*2160, and the number of views to be
displayed is 8, a scaling ratio is set to 1/4.
[0192]
The coded buffer 221 temporarily accumulates the
disparity stream extracted by the demultiplexer 214. The
disparity decoder 222 performs an inverse process to the
disparity encoder 117 (refer to Fig. 7) of the abovedescribed
transmission data generation unit 110. In other
words, the disparity decoder 222 performs a decoding process
on the disparity stream stored in the coded buffer 221 so as
to obtain disparity data. The disparity data includes
90
disparity data between the center view and the left end view
and disparity data between the center view and the right end
view. In addition, this disparity data is disparity data of
the pixel unit or the block unit. The disparity buffer 223
temporarily accumulates the disparity data acquired by the
disparity decoder 222.
[0193]
The disparity data conversion unit 224 generates
disparity data of the pixel unit, conforming to the size of
the scaled image data on the basis of the disparity data
accumulated in the disparity buffer 223. For example, in a
case where disparity data of the block unit is transmitted,
the data is converted into disparity data of the pixel unit
(refer to Fig. 11). In addition, for example, in a case
where disparity data of the pixel unit is transmitted but
does not conform to the size of scaled image data, the data
is appropriately scaled.
[0194]
The view interpolation unit 219 interpolates and
generates image data of a predetermined number of views
which are not transmitted, from the image data of each of
the center, left end and right end views after being scaled,
on the basis of the disparity data between the respective
views obtained by the disparity data conversion unit 224.
In other words, the view interpolation unit 219 interpolates
91
and generates image data of each view located between the
center view and the left end view so as to be output.
Further, the view interpolation unit 219 interpolates and
generates image data of each view located between the center
view and the right end view so as to be output.
[0195]
Fig. 29 schematically shows an example of an
interpolation and generation process in the view
interpolation unit 219. In the shown example, for example,
a current view corresponds to the above-described center
view, a target view 1 corresponds to the above-described
left end view, and a target view 2 corresponds to the abovedescribed
right end view.
[0196]
Interpolation and generation of a view located between
the current view and the target view 1 and interpolation and
generation of a view located between the current view and
the target view 2 are performed in the same manner.
Hereinafter, a description will be made of interpolation and
generation of a view located between the current view and
the target view 1.
[0197]
A pixel of a view which is located between the current
view and the target view 1 and is interpolated and generated
is allocated as follows. In this case, two-way disparity
92
data including disparity data which indicates the target
view 1 from the current view and disparity data which
indicates the current view from the target view 1 is used.
First, a pixel of the current view is allocated as a pixel
of a view which is interpolated and generated, by shifting
disparity data as a vector (refer to the solid line arrows
and the broken line arrows directed to the target view 1
from the current view and the black circles).
[0198]
At this time, a pixel is allocated as follows in a part
where a target is occluded in the target view 1. In other
words, a pixel of the target view 1 is allocated as a pixel
of the view which is interpolated and generated, by shifting
disparity data as a vector (refer to the dot chain line
arrows directed to the current view from the target view 1
and the white circles).
[0199]
As such, since the two-way disparity data is provided,
a pixel from a view which is regarded as a background can be
allotted to a pixel of the interpolated and generated view
in the part where a target is occluded. In addition, in an
occlusion region which cannot be handled in a two-way manner,
a value is allotted through a post-process.
[0200]
In addition, the target overlapped part where the tip
93
ends of the shown arrows are overlapped is a part where
shifts due to disparity are overlapped in the target view 1.
In this part, which one of the two disparities corresponds
to a foreground of the current view is determined from a
value of the disparity data and is selected. In this case,
a smaller value is mainly selected.
[0201]
Referring to Fig. 27 again, the coded buffer 225
temporarily accumulates the graphics stream extracted by the
demultiplexer 214. The graphics decoder 226 performs an
inverse process to the graphics encoder 119 (refer to Fig.
7) of the above-described transmission data generation unit
110. In other words, the graphics decoder 226 performs a
decoding process on the graphics stream stored in the coded
buffer 225 so as to obtain decoded graphics data (including
subtitle data). In addition, the graphics decoder 226
generates bitmap data of graphics superimposed on a view
(image) on the basis of the graphics data.
[0202]
The pixel buffer 227 temporarily accumulates the bitmap
data of graphics generated by the graphics decoder 226. The
scaler 228 adjusts the size of the bitmap data of graphics
accumulated in the pixel buffer 227 so as to correspond to
the size of the scaled image data. The graphics shifter 229
performs a shift process on the bitmap data of graphics of
94
which the size has been adjusted on the basis of the
disparity data obtained by the disparity data conversion
unit 224. In addition, the graphics shifter 229 generates N
bitmap data items of graphics which are respectively
superimposed on image data items of N views (View 1, View
2,…, and View N) output from the view interpolation unit 219.
[0203]
The pixel interleaving/superimposing unit 220
superimposes the respectively corresponding bitmap data
items of graphics on the image data items of the N views
(View 1, View 2,…, and View N) which are output from the
view interpolation unit 219. In addition, the pixel
interleaving/superimposing unit 220 performs a pixel
interleaving process on image data of the N views (View 1,
View 2,…, and View N) so as to generate display image data
for observing a three-dimensional image (stereoscopic image)
with the naked eye.
[0204]
The coded buffer 230 temporarily accumulates the audio
stream extracted by the demultiplexer 214. The audio
decoder 231 performs an inverse process to the audio encoder
121 (refer to Fig. 7) of the above-described transmission
data generation unit 110. In other words, the audio decoder
231 performs a decoding process on the audio stream stored
in the coded buffer 230 so as to obtain decoded audio data.
95
The channel mixing unit 232 generates and outputs audio data
of each channel in order to realize, for example, 5.1-
channel surround, in relation to the audio data obtained by
the audio decoder 231.
[0205]
In addition, reading of the image data of each view
from the decoded buffers 217-1, 217-2 and 217-2, reading of
the disparity data from the disparity buffer 223, and
reading of the bitmap data of graphics from the pixel buffer
227 are performed based on the PTS, and thus synchronous
transmission is performed.
[0206]
Next, a description will be made of a case where a twodimensional
(2D) image is received. In addition, in a case
of being the same as the above-described case where a
stereoscopic (3D) image is received, description thereof
will be appropriately omitted. The transport stream buffer
(TS buffer) 213 temporarily accumulates the transport stream
TS output from the digital tuner 212. The transport stream
TS includes a video stream obtained by coding twodimensional
image data.
[0207]
When view configuration information, that is, the
multi-view stream configuration information
(multiview_stream_configuration_info()) is inserted into a
96
layer of a video stream, in the transport stream buffer (the
TS buffer) 213, as described above, the multi-view stream
configuration descriptor
(multiview_stream_configuration_descriptor) is inserted
under the PMT, under the EIT, or the like.
[0208]
The demultiplexer 214 extracts each of elementary
streams of video, graphics, and audio from the transport
stream TS which is temporarily accumulated in the TS buffer
213. In addition, the demultiplexer 214 extracts the abovedescribed
multi-view stream configuration descriptor from
the transport stream TS so as to be sent to the CPU 201.
The CPU 201 can easily determine whether or not view
configuration information is inserted into the layer of the
video stream on the basis of the 1-bit field of
"multiview_stream_check flag" of the descriptor.
[0209]
The coded buffer 215-1 temporarily accumulates the
video stream which is obtained by coding the two-dimensional
image data and is extracted by the demultiplexer 214. The
video decoder 216-1 performs a decoding process on the video
stream stored in the coded buffer 215-1 under the control of
the CPU 201 so as to acquire two-dimensional image data.
The decoded buffer 217-1 temporarily accumulates the twodimensional
image data acquired by the video decoder 216-1.
97
[0210]
The scaler 218-1 adjusts an output resolution of the
two-dimensional image data output from the decoded buffer
217-1 so as to be predetermined resolutions. The view
interpolation unit 219 outputs the scaled two-dimensional
image data obtained by the scaler 218-1 as it is, for
example, as image data of View 1. In this case, the view
interpolation unit 219 outputs only the two-dimensional
image data.
[0211]
In this case, the coded buffers 215-2 and 215-3, the
video decoders 216-2 and 216-3, the decoded buffers 217-2
and 217-3, and the scalers 218-2 and 218-3 are in a nonoperation
state. In addition, the demultiplexer 214 does
not extract a disparity elementary stream, and the coded
buffer 221, the disparity decoder 222, the disparity buffer
223, and the disparity data conversion unit 224 are in a
non-operation state.
[0212]
The graphics shifter 229 outputs the bitmap data of
graphics of which the size has been adjusted, obtained by
the scaler 228, as it is. The pixel
interleaving/superimposing unit 220 superimposes the bitmap
data of graphics output from the graphics shifter 229 on the
two-dimensional image data output from the view
98
interpolation unit 219 so as to generate image data for
displaying a two-dimensional image.
[0213]
A detailed description is omitted, and an audio system
is the same as in a case of transmitting a stereoscopic (3D)
image.
[0214]
An operation of the receiver 200 will be described
briefly. First, a description will be made of an operation
when a stereoscopic (3D) image is received. A television
broadcast signal input to the antenna terminal 211 is
supplied to the digital tuner 212. The digital tuner 212
processes the television broadcast signal so as to output a
predetermined transport stream TS corresponding to a channel
selected by a user. The transport stream TS is temporarily
accumulated in the TS buffer 213.
[0215]
The transport stream TS includes video streams obtained
by coding image data of a left end view and a right end view
and image data of a center view which is an intermediate
view located between the left end and the right end among a
plurality of views for stereoscopic image display.
[0216]
The demultiplexer 214 extracts each of elementary
streams of video, disparity, graphics, and audio from the
99
transport stream TS which is temporarily accumulated in the
TS buffer 213. In addition, the demultiplexer 214 extracts
multi-view stream configuration descriptor which is
identification information from the transport stream TS so
as to be sent to the CPU 201. The CPU 201 can easily
determine whether or not view configuration information is
inserted into the layer of the video stream on the basis of
the 1-bit field of "multiview_stream_checkflag" of the
descriptor.
[0217]
The video streams which are obtained by coding image
data of each of the center, left end and right end views and
are extracted by the demultiplexer 214 are supplied to the
coded buffers 215-1, 215-2 and 215-3 so as to be temporarily
accumulated. In addition, the video decoders 216-1, 216-2
and 216-3 respectively perform a decoding process on the
video streams stored in the coded buffers 215-1, 215-2 and
215-3 under the control of the CPU 201 so as to acquire
image data of each of the center, left end and right end
views.
[0218]
In addition, each video decoder extracts the multi-view
stream configuration information
(multiview_stream_configuration_info()) which is view
configuration information and is inserted into the user data
100
region or the like of the picture header or the sequence
header of the video stream so as to be sent to the CPU 201.
The CPU 201 controls an operation of each unit so as to
perform an operation when a stereoscopic (3D) image is
received, that is, when a stereoscopic (3D) display process
is performed, on the basis of this view configuration
information.
[0219]
The image data items of the respective views acquired
by the video decoders 216-1, 216-2 and 216-3 are supplied to
the decoded buffers 217-1, 217-2 and 217-3 so as to be
temporarily accumulated. The scalers 218-1, 218-2 and 218-3
respectively adjust output resolutions of the image data
items of the respective views output from the decoded
buffers 217-1, 217-2 and 217-3 so as to be predetermined
resolutions.
[0220]
In addition, the disparity stream extracted by the
demultiplexer 214 is supplied to the coded buffer 221 so as
to be temporarily accumulated. The disparity decoder 222
performs a decoding process on the disparity stream stored
in the coded buffer 221 so as to obtain disparity data. The
disparity data includes disparity data between the center
view and the left end view and disparity data between the
center view and the right end view. In addition, this
101
disparity data is disparity data of the pixel unit or the
block unit.
[0221]
The disparity data acquired by the disparity decoder
222 is supplied to the disparity buffer 223 so as to be
temporarily accumulated. The disparity data conversion unit
224 generates disparity data of the pixel unit, conforming
to the size of the scaled image data on the basis of the
disparity data accumulated in the disparity buffer 223. In
this case, when disparity data of the block unit is
transmitted, the data is converted into disparity data of
the pixel unit. In addition, in this case, when disparity
data of the pixel unit is transmitted but does not conform
to the size of scaled image data, the data is appropriately
scaled.
[0222]
The view interpolation unit 219 interpolates and
generates image data of a predetermined number of views
which are not transmitted, from the image data of each of
the center, left end and right end views after being scaled,
on the basis of the disparity data between the respective
views obtained by the disparity data conversion unit 224.
From the view interpolation unit 219, image data of N views
(View 1, View 2,…, and View N) for observing a threedimensional
image (stereoscopic image) with the naked eye
102
are obtained. In addition, image data of each of the center,
left end and right end views is also included.
[0223]
The graphics stream extracted by the demultiplexer 214
are supplied to the coded buffer 225 so as to be temporarily
accumulated. The graphics decoder 226 performs a decoding
process on the graphics stream stored in the coded buffer
225 so as to obtain decoded graphics data (including
subtitle data). In addition, the graphics decoder 226
generates bitmap data of graphics superimposed on a view
(image) on the basis of the graphics data.
[0224]
The bitmap data of graphics generated by the graphics
decoder 226 is supplied to the pixel buffer 227 so as to be
temporarily accumulated. The scaler 228 adjusts the size of
the bitmap data of graphics accumulated in the pixel buffer
227 so as to correspond to the size of the scaled image data.
[0225]
The graphics shifter 229 performs a shift process on
the bitmap data of graphics of which the size has been
adjusted on the basis of the disparity data obtained by the
disparity data conversion unit 224. In addition, the
graphics shifter 229 generates N bitmap data items of
graphics which are respectively superimposed on image data
items of N views (View 1, View 2,…, and View N) output from
103
the view interpolation unit 219, so as to be supplied to the
pixel interleaving/superimposing unit 220.
[0226]
The pixel interleaving/superimposing unit 220
superimposes the respectively corresponding bitmap data
items of graphics on the image data items of the N views
(View 1, View 2,…, and View N). In addition, the pixel
interleaving/superimposing unit 220 performs a pixel
interleaving process on image data of the N views (View 1,
View 2,…, and View N) so as to generate display image data
for observing a three-dimensional image (stereoscopic image)
with the naked eye. The display image data is supplied to a
display, and thereby an image is displayed so as to observe
a three-dimensional image (stereoscopic image) with the
naked eye.
[0227]
In addition, the audio stream extracted by the
demultiplexer 214 is supplied to the coded buffer 230 so as
to be temporarily accumulated. The audio decoder 231
performs a decoding process on the audio stream stored in
the coded buffer 230 so as to obtain decoded audio data.
The audio data is supplied to the channel mixing unit 232.
The channel mixing unit 232 generates audio data of each
channel in order to realize, for example, 5.1-channel
surround, in relation to the audio data. The audio data is
104
supplied to, for example, a speaker, and a sound is output
conforming with image display.
[0228]
Next, a description will be made of an operation when a
two-dimensional (2D) image is received. A television
broadcast signal input to the antenna terminal 211 is
supplied to the digital tuner 212. The digital tuner 212
processes the television broadcast signal so as to output a
predetermined transport stream TS corresponding to a channel
selected by a user. The transport stream TS is temporarily
accumulated in the TS buffer 213. The transport stream TS
includes a video stream obtained by coding two-dimensional
image data.
[0229]
The demultiplexer 214 extracts each of elementary
streams of video, graphics, and audio from the transport
stream TS which is temporarily accumulated in the TS buffer
213. In addition, the demultiplexer 214 extracts multi-view
stream configuration descriptor which is identification
information, if inserted, from the transport stream TS so as
to be sent to the CPU 201. The CPU 201 can easily determine
whether or not view configuration information is inserted
into the layer of the video stream on the basis of the 1-bit
field of "multiview_stream_check flag" of the descriptor.
[0230]
105
The video stream which is obtained by coding twodimensional
image data and is extracted by the demultiplexer
214 is supplied to the coded buffer 215-1 so as to be
temporarily accumulated. In addition, the video decoder
216-1 performs a decoding process on the video stream stored
in the coded buffer 215-1 under the control of the CPU 201
so as to acquire two-dimensional image data.
[0231]
In addition, in the video decoder 216-1, if inserted,
the multi-view stream configuration information
(multiview_stream_configuration_info()) which is view
configuration information and is inserted into the user data
region or the like of the picture header or the sequence
header of the video stream is extracted and is sent to the
CPU 201. The CPU 201 controls an operation of each unit so
as to perform an operation when a two-dimensional (2D) image
is received, that is, when a two-dimensional (2D) display
process is performed, on the basis of the extracted view
configuration information or on the basis of the fact that
the view configuration information is not extracted.
[0232]
The two-dimensional image data acquired by the video
decoder 216-1 is supplied to the decoded buffer 217-1 so as
to be temporarily accumulated. The scaler 218-1 adjusts an
output resolution of the two-dimensional image data output
106
from the decoded buffer 217-1 so as to be predetermined
resolutions. The scaled two-dimensional image data is
output from the view interpolation unit 219 as it is, for
example, as image data of View 1.
[0233]
The graphics stream extracted by the demultiplexer 214
are supplied to the coded buffer 225 so as to be temporarily
accumulated. The graphics decoder 226 performs a decoding
process on the graphics stream stored in the coded buffer
225 so as to obtain decoded graphics data (including
subtitle data). In addition, the graphics decoder 226
generates bitmap data of graphics superimposed on a view
(image) on the basis of the graphics data.
[0234]
The bitmap data of graphics generated by the graphics
decoder 226 is supplied to the pixel buffer 227 so as to be
temporarily accumulated. The scaler 228 adjusts the size of
the bitmap data of graphics accumulated in the pixel buffer
227 so as to correspond to the size of the scaled image data.
The bitmap data of graphics of which the size has been
adjusted, obtained by the scaler 228, is output from the
graphics shifter 229 as it is.
[0235]
The pixel interleaving/superimposing unit 220
superimposes the bitmap data of graphics output from the
107
graphics shifter 229 on the two-dimensional image data
output from the view interpolation unit 219 so as to
generate display image data of a two-dimensional image. The
display image data is supplied to a display, and thereby a
two-dimensional image is displayed.
[0236]
[Signaling in 3D period and 2D period]
Next, a description will be made of an operation mode
switching control between a stereoscopic (3D) display
process and a two-dimensional (2D) display process in the
receiver 200 shown in Fig. 27. This switching is performed
by the CPU 201. When a stereoscopic (3D) image is received,
multi-view stream configuration information extracted by
each of the video decoders 216-1, 216-2 and 216-3 is
supplied to the CPU 201. In addition, when a twodimensional
(2D) image is received, multi-view stream
configuration information, if inserted, extracted by the
video decoder 216-1 is supplied to the CPU 201. The CPU 201
controls switching between a stereoscopic (3D) display
process and a two-dimensional (2D) display process on the
basis of the presence or the absence of the information or
content thereof.
[0237]
Figs. 30 and 31 show an example of a received stream in
a case where a 3D period (when a stereoscopic image is
108
received) and a 2D period (when a two-dimensional image is
received) are alternately continued, and each period is, for
example, the program unit or the scene unit. In the 3D
period, a video stream ES1 of an intermediate view which is
a base video stream is present, and two video streams ES2
and ES3 of a left end view and a right end view which are
additional video streams are also present. In the 2D period,
only the video stream ES1 which is a base video stream is
present.
[0238]
The example of Fig. 30 shows a case where a SEI message
including the multi-view stream configuration information is
inserted with the picture unit in both of the 3D period and
the 2D period. In addition, the example of Fig. 31 shows a
case where the SEI message including the multi-view stream
configuration information is inserted with the scene unit or
the picture group unit (the GOP unit) in each period.
[0239]
In the SEI message inserted in the 3D period, there is
"3D_flag=1" which indicates a 3D mode (stereoscopic image
transmission mode). In addition, in the SEI message
inserted in the 2D period, there is "3D_flag=0" which
indicates a non-3D mode, that is, 2D mode (two-dimensional
image transmission mode). In addition, the SEI message is
not only inserted into the video stream ES1 but is also
109
inserted into the video streams ES2 and ES3, but is not
shown for simplification the drawings.
[0240]
A flowchart of Fig. 32 shows an example of process
procedures of the operation mode switching control in the
CPU 201. This example is an example of a case where a
coding method is AVC or MVC. As described above, the multiview
stream configuration information is inserted into the
"SEIs" part of the access unit as "Multi-view stream
configuration SEI message" (refer to Figs. 21 and 14). In
this case, when a stereoscopic (3D) image is received, MVC
base view stream (base video stream) and non-base view
stream (additional video stream) are received, and when a
two-dimensional (2D) image is received, an AVC (2D) stream
(base video stream) is received.
[0241]
The CPU 201 performs control according to the flowchart
for each picture frame. However, in a case where the SEI
message is not inserted with the picture unit, for example,
the SEI message is inserted with the GOP unit (refer to Fig.
31), the CPU 201 maintains the current SEI information until
the SEI information of the current GOP is replaced with the
SEI information of the next GOP.
[0242]
First, the CPU 201 starts a process in step ST1, and
110
then proceeds to a process in step ST2. In step ST2, the
CPU 201 determines whether or not SEI ("Multiview stream
configuration SEI message") is inserted into the base video
stream. When the SEI is inserted, the CPU 201 determines
whether or not information in the SEI indicates a 3D mode,
that is, "3D_flag=1" in step ST3.
[0243]
When the information in the SEI indicates a 3D mode,
that is, a stereoscopic (3D) image is received, the CPU 201
proceeds to a process in step ST4. The CPU 201 manages the
respective input buffers (coded buffers) of the base video
stream and the additional video stream in step ST4, and
decodes the base video stream and the additional video
stream, respectively, by using the decoders (video decoders)
in step ST5. Further, the CPU 201 performs control such
that the receiver 200 performs other stereoscopic (3D)
display processes in step ST6.
[0244]
In addition, the CPU 201 proceeds to a process in step
ST7 when the SEI is not inserted in step ST2, or when the
information in the SEI does not indicate a 3D mode, that is,
a two-dimensional (2D) image is received in step ST3. The
CPU 201 proceeds to the process in step ST7. The CPU 201
manages an input buffer (coded buffer) of the base video
stream in step ST7, and decodes the base video stream by
111
using the decoder (video decoder) in step ST8. Further, the
CPU 201 performs control such that the receiver 200 performs
other two-dimensional (2D) display processes in step ST9.
[0245]
As described above, in the receiver 200 shown in Fig.
27, switching between a stereoscopic (3D) display process
and a two-dimensional (2D) display process is controlled
based on the presence or the absence of the SEI message
including the multi-view stream configuration information or
content thereof. For this reason, it is possible to
appropriately and accurately handle a dynamic variation in
delivery content and to thereby receive a correct stream.
[0246]
Fig. 33 shows an example of a case where a base video
stream ES1 of an AVC base view of "Stream_Type=0x1B" and
"PID=01" is continuously included in the transport stream TS,
and MVC additional video streams ES2 and ES3 of
"Stream_Type=0x20", "PID=10" and "PID=11" are intermittently
included therein. In this case, the multi-view stream
configuration SEI message is inserted into the stream ES1.
[0247]
The SEI message is present in periods tn-1 and tn+1,
and there is "3D_flag=1" which indicates a 3D mode. For
this reason, in these periods, the receiver 200 performs a
stereoscopic (3D) display process. In other words, the
112
streams ES2 and ES3 as well as the stream ES1 are also
extracted and are decoded such that stereoscopic (3D)
display is performed. On the other hand, in the period tn,
the SEI message is present, but there is "3D_flag=0" which
indicates a 2D mode. For this reason, in this period, the
receiver 200 performs a two-dimensional (2D) display process.
In other words, only the stream ES1 is extracted and is
decoded such that two-dimensional (2D) display is performed.
[0248]
Fig. 34 shows an example of a case where a 3D period
(3D mode period) and a 2D period (2D mode period) are
alternately continued, and there is no auxiliary information
(multi-view stream configuration SEI message) for
identifying a mode. The periods T1 and T3 indicate a 3D
period, and the period T2 indicates a 2D period. Each
period represents, for example, the program unit or the
scene unit.
[0249]
In the 3D period, a base video stream of an MVC base
view of "Stream_Type=0x1B" is present, and an additional
video stream of an MVC non-base view of "Stream_Type=0x20"
is also present. In addition, in the 2D period, an AVC
stream of "Stream_Type=0x1B" is present. Further, the base
video stream has a configuration in which SPS is a head and
a predetermined number of access units (AU) are continuously
113
located. Furthermore, the additional video stream has a
configuration in which subset SPS (SSSPS) is a head and a
predetermined number of access units (AU) are continuously
located. In addition, the access units (AU) are constituted
by "PPS, Substream SEIs, and Coded Slice".
[0250]
In a case where there is no auxiliary information for
identifying a mode, the receiver recognizes that the 3D
period is switched to the 2D period when data is not input
to the input buffers of the receiver during a predetermined
period. However, it cannot be recognized at the time point
T1 that the reason why data of an additional video stream is
not input to the input buffer is that errors occur during
transmission or coding, or switching to the 2D period is
performed. Therefore, a temporal delay is required for the
receiver to be switched to a 2D processing mode.
[0251]
Fig. 35 shows an example of a case where a 3D period
and a 2D period are alternately continued, and there is
auxiliary information (multi-view stream configuration SEI
message) for identifying a mode. The periods T1 and T3
indicate a 3D period, and the period T2 indicates a 2D
period. Each period represents, for example, the program
unit or the scene unit.
[0252]
114
In the 3D period, a base video stream of an MVC base
view of "Stream_Type=0x1B" is present, and an additional
video stream of an MVC non-base view of "Stream_Type=0x20"
is also present. In addition, in the 2D period, an AVC
stream of "Stream_Type=0x1B" is present. Further, the base
video stream has a configuration in which "SPS" is a head
and a predetermined number of access units (AU) are
continuously located. Furthermore, the additional video
stream has a configuration in which "SSSPS" is a head and a
predetermined number of access units (AU) are continuously
located. In addition, the access units (AU) are constituted
by "PPS, Substream SEIs, and Coded Slice".
[0253]
The auxiliary information (multi-view stream
configuration SEI message) for identifying a mode is
inserted for each access unit (AU). The auxiliary
information inserted into the access unit in the 3D period
is indicated by "3D" which is regarded as "3D_flag=1" and
indicates a 3D mode (stereoscopic image transmission mode).
On the other hand, the auxiliary information inserted into
the access unit in the 2D period is indicated by "2D" which
is regarded as "3D_flag=0" and indicates a 2D mode (twodimensional
image transmission mode).
[0254]
As above, in a case where there is auxiliary
115
information (multi-view stream configuration SEI message)
for identifying a mode, the receiver checks the element
"3D_flag" of the auxiliary information, and can immediately
discriminate whether the element indicates a 3D mode or a 2D
mode, and thus it is possible to rapidly perform decoding
and switching between display processes. In a case where
the 3D period is switched to the 2D period, the receiver can
determine that the 3D period is switched to the 2D period at
the discrimination timing T2 when the element "3D_flag" of
the auxiliary information inserted into the first access
unit indicates a 2D mode, and thus can rapidly perform mode
switching from 3D to 2D.
[0255]
In addition, in the receiver 200 shown in Fig. 27, when
a stereoscopic (3D) image is received, at least image data
of a left end view and a right end view and image data of an
intermediate view located between the left end and the right
end are received among a plurality of views for stereoscopic
image display. In addition, in this receiver 200, other
views are obtained through an interpolation process on the
basis of disparity data. For this reason, it is possible to
favorably observe a stereoscopic image formed by multi-views
with the naked eye.
[0256]
In other words, not only image data of the left end
116
view and the right end view but also image data of the
center view is received. For this reason, a relative
disparity between views is small, a periphery of occlusion
according to processing of a fine part when image data of a
view which is not transmitted is interpolated is easily
interpolated, and thereby it is possible to improve quality
of a reproduced image. In addition, since image data of the
left end view and the right end view is received, image data
of a view which is not transmitted can be generated through
an interpolation process, and thus it is possible to easily
maintain high image quality with regard to processing of an
end point of occlusion or the like.
[0257]
In addition, the receiver 200 shown in Fig. 27 shows a
configuration example of a case where a disparity stream
obtained by coding disparity data is included in the
transport stream TS. In a case where a disparity stream is
not included in the transport stream TS, disparity data is
generated from received image data of each view and is used.
[0258]
Fig. 36 shows a configuration example of a receiver
200A in this case. In Fig. 36, a part corresponding to Fig.
27 is given the same reference numeral and detailed
description thereof will be omitted. The receiver 200A
includes a disparity data generation unit 233. The
117
disparity data generation unit 233 generates disparity data
on the basis of scaled image data of each of the center,
left end and right end views.
[0259]
Although detailed description is omitted, a method of
generating disparity data in this case is the same as the
method of generating disparity data in the disparity data
generation portion 116 of the above-described transmission
data generation unit 110. In addition, the disparity data
generation unit 233 generates and outputs the same disparity
data as disparity data of the pixel unit generated by the
disparity data conversion unit 224 of the receiver 200 shown
in Fig. 27. Disparity data generated by the disparity data
generate unit 233 is supplied to the view interpolation unit
219 and is also supplied to the graphics shifter 229 so as
to be used.
[0260]
In addition, in the receiver 200A shown in Fig. 36, the
coded buffer 221, the disparity decoder 222, the disparity
buffer 223, and the disparity data conversion unit 224 of
the receiver 200 shown in Fig. 27 are omitted. The other
configurations of the receiver 200A shown in Fig. 36 are the
same as the configurations of the receiver 200 shown in Fig.
27.
[0261]
118
[Another example of auxiliary information for
identifying mode]
In the above description, a description has been made
of an example in which the multi-view stream configuration
SEI message is used as auxiliary information for identifying
a mode, and the receiver discriminates a 3D period or a 2D
period with frame accuracy on the basis of set content
thereof. As auxiliary information for identifying a mode,
the existing multi-view view position SEI message
(multiview_view_position SEI message) may be used. If this
multi-view view position SEI message is to be inserted, a
transmission side is required to insert the message into an
intra-picture in which intra-refresh (making a compression
buffer vacant) is performed over an entire video sequence.
[0262]
Fig. 37 shows a structural example (Syntax) of a multiview
view position (Multiview view position()) included in
the SEI message. The field of "num_views_minus1" indicates
a value (0 to 1023) which is withdrawn from the number of
views. The field of "view_position[i]" indicates a relative
positional relationship when each view is displayed. In
other words, the field indicates a sequential relative
position from a left view to a right view when each view is
displayed, using a value which sequentially increases from 0.
[0263]
119
The transmission data generation unit 110 shown in Fig.
7 described above inserts the multi-view view position SEI
message into a video stream (base video stream) which is
obtained by coding image data of an intermediate view in a
3D mode (stereoscopic image transmission mode). The multiview
view position SEI message forms identification
information indicating a 3D mode. In this case, the message
is inserted at least with the program unit, the scene unit,
the picture group unit, or the picture unit.
[0264]
Fig. 38(a) shows a leading access unit of Group Of
Pictures (GOP), and Fig. 38(b) shows an access unit other
than the leading access unit of the GOP. In a case where
multi-view view position SEI is inserted with the GOP unit,
"multiview_view_position SEI message" is inserted into only
the leading access unit of the GOP.
[0265]
If this is applied to three views including a left end
(Left), center (Center) and right end (Right) views, in the
multi-view view position (Multiview view position()) (refer
to Fig. 37) included in the multi-view view position SEI
message, there is "view_position[0]=1" which indicates that
a base view video stream which is a base video stream is a
video stream obtained by coding image data of the center
view.
120
[0266]
In addition, there is "view_position[1]=0" which
indicates that a non-base view first video stream which is
an additional video stream is a video stream obtained by
coding image data of the left end view. Further, there is
"view_position[2]=2" which indicates that a non-base view
second video stream which is an additional video stream is a
video stream obtained by coding image data of the right end
view.
[0267]
A description will be made of operation mode switching
control between a stereoscopic (3D) display process and a
two-dimensional (2D) display process in the receiver 200
shown in Fig. 27 in a case of using the multi-view view
position SEI message (multiview_view_position message),.
This switching is performed by the CPU 201. When a
stereoscopic (3D) image is received, the multi-view view
position SEI message is extracted by the video decoder 216-1
and is supplied to the CPU 201. However, when a twodimensional
(2D) image is received, the SEI message is not
extracted by the video decoder 216-1 and thus is not
supplied to the CPU 201. The CPU 201 controls switching
between a stereoscopic (3D) display process and a twodimensional
(2D) display process on the basis of the
presence or the absence of the SEI message.
121
[0268]
Figs. 39 and 40 show an example of a received stream in
a case where a 3D period (when a stereoscopic image is
received) and a 2D period (when a two-dimensional image is
received) are alternately continued. Each period is, for
example, the program unit or the scene unit. In the 3D
period, a video stream ES1 of an intermediate view which is
a base video stream is present, and two video streams ES2
and ES3 of a left end view and a right end view which are
additional video streams are also present. In the 2D period,
only the video stream ES1 which is a base video stream is
present.
[0269]
The example of Fig. 39 shows a case where the multiview
view position SEI message is inserted with the picture
unit in the 3D period. In addition, the example of Fig. 40
shows a case where the multi-view view position SEI is
inserted with the scene unit or the picture group unit (the
GOP unit) in 3D period.
[0270]
A flowchart of Fig. 41 shows an example of process
procedures of the operation mode switching control in the
CPU 201. The CPU 201 performs control according to the
flowchart for each picture frame. However, in a case where
the SEI message is not inserted with the picture unit, for
122
example, the SEI message is inserted with the GOP unit
(refer to Fig. 40), the CPU 201 maintains the current SEI
information until the SEI information of the current GOP is
replaced with the SEI information of the next GOP.
[0271]
First, the CPU 201 starts a process in step ST11, and
then proceeds to a process in step ST12. In step ST12, the
CPU 201 determines whether or not SEI ("Multiview Position
SEI message") is inserted into the base video stream. When
the SEI is inserted, the CPU 201 proceeds to a process in
step ST13. In other words, when a stereoscopic (3D) image
is received, the SEI is inserted into the base video stream,
the CPU 201 proceeds to a process in step ST13.
[0272]
The CPU 201 manages the respective input buffers (coded
buffers) of the base video stream and the additional video
stream in step ST13, and decodes the base video stream and
the additional video stream, respectively, by using the
decoders (video decoders) in step ST14. Further, the CPU
201 performs control such that the receiver 200 performs
other stereoscopic (3D) display processes in step ST15.
[0273]
In this case, a video stream (additional video stream)
into which the multi-view view position SEI is not inserted
is processed according to a definition designated by the
123
element of the SEI. In other words, in this example, each
additional video stream is processed according to a relative
positional relationship designated by "view_position[i]"
when each view is displayed, and thereby image data of each
view is appropriately acquired.
[0274]
In addition, when the SEI ("multiview_view_position SEI
message") is not inserted in step ST12, the CPU 201 proceeds
to a process in step ST16. In other words, since the SEI is
not inserted into the base video stream when a twodimensional
(2D) image is received, the CPU 201 proceeds to
a process in step ST16. The CPU 201 manages an input buffer
(coded buffer) of the base video stream in step ST16, and
decodes the base video stream by using the decoder (video
decoder) in step ST17. Further, the CPU 201 performs
control such that the receiver 200 performs other twodimensional
(2D) display processes in step ST18.
[0275]
As described above, also by using the multi-view view
position SEI message, a reception side can favorably perform
switching between a stereoscopic (3D) display process and a
two-dimensional (2D) display process. For this reason, it
is possible to appropriately and accurately handle a dynamic
variation in delivery content and to thereby receive a
correct stream.
124
[0276]
Fig. 42 shows an example of a case where a base video
stream ES1 of an AVC base view of "Stream_Type=0x1B" and
"PID=01" is continuously included in the transport stream TS,
and MVC additional video streams ES2 and ES3 of
"Stream_Type=0x20", "PID=10" and "PID=11" are intermittently
included therein. In this case, the multi-view view
position SEI is inserted into the stream ES1 in the 3D
period.
[0277]
The multi-view view position SEI is present in periods
tn-1 and tn+1. For this reason, in these periods, the
receiver 200 performs a stereoscopic (3D) display process.
In other words, the streams ES2 and ES3 as well as the
stream ES1 are also extracted and are decoded such that
stereoscopic (3D) display is performed. On the other hand,
in the period tn, the multi-view view position SEI is not
present. For this reason, in this period, the receiver 200
performs a two-dimensional (2D) display process. In other
words, only the stream ES1 is extracted and is decoded such
that two-dimensional (2D) display is performed.
[0278]
In addition, at least one of the above-described multiview
stream configuration SEI and the multi-view view
position SEI may be inserted into a video stream which is
125
transmitted by a transmission side. In this case, a
reception side may control switching between a stereoscopic
(3D) display process and a two-dimensional (2D) display
process by using at least several pieces of SEI.
[0279]
Fig. 43 shows an example of a case where a 3D period
and a 2D period are alternately continued, and there is
auxiliary information (multi-view view position SEI message)
for identifying a mode. The periods T1 and T3 indicate a 3D
period, and the period T2 indicates a 2D period. Each
period represents, for example, the program unit or the
scene unit.
[0280]
In the 3D period, a base video stream of an MVC base
view of "Stream_Type=0x1B" is present, and an additional
video stream of an MVC non-base view of "Stream_Type=0x20"
is also present. In addition, in the 2D period, an AVC
stream of "Stream_Type=0x1B" is present. Further, the base
video stream has a configuration in which "SPS" is a head
and a predetermined number of access units (AU) are
continuously located. Furthermore, the additional video
stream has a configuration in which "SSSPS" is a head and a
predetermined number of access units (AU) are continuously
located. In addition, the access units (AU) are constituted
by "PPS, Substream SEIs, and Coded Slice".
126
[0281]
The auxiliary information (multi-view view position SEI
message) for identifying a mode is inserted for each access
unit (AU) in the 3D period. The auxiliary information
indicates the 3D mode which is denoted by "3D". In addition,
the auxiliary information is not inserted into each access
unit (AU) in the 2D period.
[0282]
As above, in a case where there is auxiliary
information for identifying a mode as described above, the
receiver can immediately discriminate whether a period is a
3D period or a 2D period on the basis of the presence or the
absence of the auxiliary information, and thus it is
possible to rapidly perform decoding and switching between
display processes. In a case where the 3D period is
switched to the 2D period, the receiver can determine that
the 3D period is switched to the 2D period at the
discrimination timing T2 when there is no auxiliary
information in the first access unit and thus can rapidly
perform mode switching from 3D to 2D.
[0283]
A flowchart of Fig. 44 shows an example of process
procedures of the operation mode switching control in the
CPU 201. The CPU 201 performs control according to the
flowchart for each picture frame. However, in a case where
127
the SEI message is not inserted with the picture unit, for
example, the SEI message is inserted with the GOP unit, the
CPU 201 maintains the current SEI information until the SEI
information of the current GOP is replaced with the SEI
information of the next GOP. Hereinafter, a description
will be made assuming the multi-view stream configuration
SEI as A type SEI and the multi-view view position SEI as B
type SEI.
[0284]
First, the CPU 201 starts a process in step ST21, and
then proceeds to a process in step ST22. In step ST22, the
CPU 201 determines whether or not the A type SEI is inserted
into the base video stream. When the A type SEI is inserted,
the CPU 201 determines whether or not information in the A
type SEI indicates a 3D mode, that is, "3D_flag=1" in step
ST23.
[0285]
When the information in the SEI indicates a 3D mode,
that is, a stereoscopic (3D) image is received, the CPU 201
proceeds to a process in step ST24. The CPU 201 manages the
respective input buffers (coded buffers) of the base video
stream and the additional video stream in step ST24, and
decodes the base video stream and the additional video
stream, respectively, by using the decoders (video decoders)
in step ST25. Further, the CPU 201 performs control such
128
that the receiver 200 performs other stereoscopic (3D)
display processes in step ST6.
[0286]
In addition, the CPU 201 proceeds to a process in step
ST28 when the information in the A type SEI does not
indicate a 3D mode, that is, a two-dimensional (2D) image is
received in step ST23. The CPU 201 manages an input buffer
(coded buffer) of the base video stream in step ST28, and
decodes the base video stream by using the decoder (video
decoder) in step ST29. Further, the CPU 201 performs
control such that the receiver 200 performs other twodimensional
(2D) display processes in step ST30.
[0287]
In addition, when the A type SEI is not inserted in
step ST22, the CPU 201 determines whether or not the B type
SEI is inserted into the base video stream in step ST27.
When the B type SEI is inserted, the CPU 201 proceeds to a
process in step ST24, and performs control such that the
receiver 200 performs a stereoscopic (3D) display process as
described above. On the other hand, when the B type SEI is
not inserted into the base video stream, the CPU 201
proceeds to a process in step ST28, and performs control
such that the receiver 200 performs a two-dimensional (2D)
display process.
[0288]
129
As described above, in a case where at least one of the
multi-view stream configuration SEI and the multi-view view
position SEI is inserted into a transmitted video stream, a
reception side can use at least one. Thereby, it is
possible to favorably perform switching between a
stereoscopic (3D) display process and a two-dimensional (2D)
display process. For this reason, it is possible to
appropriately and accurately handle a dynamic variation in
delivery content and to thereby receive a correct stream.
[0289]
[Still another example of auxiliary information for
identifying mode]
In the above description, a description has been made
of an example in which the multi-view stream configuration
SEI message or the multi-view view position SEI message is
used as auxiliary information for identifying a mode, and
the receiver discriminates a 3D period or a 2D period with
frame accuracy on the basis of set content thereof or the
presence or the absence thereof. As auxiliary information
for identifying a mode, still another auxiliary information
may be used. That is, auxiliary information indicating a 2D
mode is used.
[0290]
As identification information indicating a 2D mode, a
SEI message which is newly defined may be used. In addition,
130
in a case of an MPEG2 stream, existing frame packing
arrangement data (frame_packing_arrangement_data()) may be
used.
[0291]
Fig. 45 shows a structural example (Syntax) of frame
packing arrangement data (frame_packing_arrangement_data()).
The 32-bit field of "frame_packing_user_data_identifier"
enables this user data to be identified as frame packing
arrangement data. The 7-bit field of "arrangement_type"
indicates a stereo video format type
(stereo_video_format_type). As shown in Fig. 46, "0000011"
indicates stereo side-by-side, "0000100" indicates stereo
top-and-bottom, and "0001000" indicates 2D video.
[0292]
The transmission data generation unit 110 shown in Fig.
7 described above inserts auxiliary information indicating a
2D mode into a video stream (base video stream) which is
obtained by coding image data of an intermediate view in a
2D mode (stereoscopic image transmission mode). For example,
in a case where this stream is an MPEG2 stream, the frame
packing arrangement data (arrangement_type=0001000) is
inserted into the user data region. In this case, the data
is inserted at least with the program unit, the scene unit,
the picture group unit, or the picture unit.
[0293]
131
The frame packing arrangement data
(frame_packing_arrangement_data()) is inserted into the user
data region of the picture header part as user data
"user_data()". Fig. 47 shows a structural example of
"user_data()". The 32-bit field of "user_data_start_code"
is a start code of the user data (user_data) and is a fixed
value of "0x000001B2". "frame_packing_arrangement_data()"
is inserted subsequent to the start code as a data body.
[0294]
In a case of using the auxiliary information indicating
a 2D mode, a description will be made of operation mode
switching control between a stereoscopic (3D) display
process and a two-dimensional (2D) display process in the
receiver 200 shown in Fig. 27. This switching is performed
by the CPU 201. When a two-dimensional (2D) image is
received, the auxiliary information indicating a 2D mode is
extracted by the video decoder 216-1 and is supplied to the
CPU 201. However, when a stereoscopic (3D) image is
received, the auxiliary information is not extracted by the
video decoder 216-1 and thus is not supplied to the CPU 201.
The CPU 201 controls switching between a stereoscopic (3D)
display process and a two-dimensional (2D) display process
on the basis of the presence or the absence of the auxiliary
information.
[0295]
132
Figs. 48 and 49 show an example of a received stream in
a case where a 3D period (when a stereoscopic image is
received) and a 2D period (when a two-dimensional image is
received) are alternately continued. Each period is, for
example, the program unit or the scene unit. In the 3D
period, a video stream ES1 of an intermediate view which is
a base video stream is present, and two video streams ES2
and ES3 of a left end view and a right end view which are
additional video streams are also present. In the 2D period,
only the video stream ES1 which is a base video stream is
present. The example of Fig. 48 shows a case where the
auxiliary information indicating a 2D mode is inserted with
the picture unit in the 2D period. In addition, the example
of Fig. 49 shows a case where the auxiliary information
indicating a 2D mode is inserted with the scene unit or the
picture group unit (the GOP unit) in the 2D period.
[0296]
A flowchart of Fig. 50 shows an example of process
procedures of the operation mode switching control in the
CPU 201. The CPU 201 performs control according to the
flowchart for each picture frame. However, in a case where
the auxiliary information is not inserted with the picture
unit, for example, the auxiliary information is inserted
with the GOP unit (refer to Fig. 49), the CPU 201 maintains
the current auxiliary information until the auxiliary
133
information of the current GOP is replaced with the
auxiliary information of the next GOP.
[0297]
First, the CPU 201 starts a process in step ST31, and
then proceeds to a process in step ST32. In step ST32, the
CPU 201 determines whether or not the auxiliary information
indicating a 2D mode is inserted into the base video stream.
When the auxiliary information is not inserted, the CPU 201
proceeds to a process in step ST33. In other words, when a
stereoscopic (3D) image is received, the auxiliary
information is not inserted into the base video stream, the
CPU 201 proceeds to a process in step ST33.
[0298]
The CPU 201 manages the respective input buffers (coded
buffers) of the base video stream and the additional video
stream in step ST33, and decodes the base video stream and
the additional video stream, respectively, by using the
decoders (video decoders) in step ST34. Further, the CPU
201 performs control such that the receiver 200 performs
other stereoscopic (3D) display processes in step ST35.
[0299]
In addition, when the auxiliary information is inserted
in step ST32, the CPU 201 proceeds to a process in step ST36.
In other words, since the auxiliary information inserted
into the base video stream when a two-dimensional (2D) image
134
is received, the CPU 201 proceeds to a process in step ST36.
The CPU 201 manages an input buffer (coded buffer) of the
base video stream in step ST36, and decodes the base video
stream by using the decoder (video decoder) in step ST37.
Further, the CPU 201 performs control such that the receiver
200 performs other two-dimensional (2D) display processes in
step ST38.
[0300]
As described above, also by using the auxiliary
information indicating a 2D mode, a reception side can
favorably perform switching between a stereoscopic (3D)
display process and a two-dimensional (2D) display process.
For this reason, it is possible to appropriately and
accurately handle a dynamic variation in delivery content
and to thereby receive a correct stream.
[0301]
Fig. 51 shows an example of a case where a base video
stream ES1 of an MPEG2 base view of "Stream_Type=0x20" and
"PID=01" is continuously included in the transport stream TS,
and AVC additional video streams ES2 and ES3 of
"Stream_Type=0x23", "PID=10" and "PID=11" are intermittently
included therein. In this case, the frame packing
arrangement data (arrangement_type="2D") is inserted into
the stream ES1 in the 2D period.
[0302]]
135
The frame packing arrangement data
(arrangement_type="2D") is not present in periods tn-1 and
tn+1. For this reason, in these periods, the receiver 200
performs a stereoscopic (3D) display process. In other
words, the streams ES2 and ES3 as well as the stream ES1 are
also extracted and are decoded such that stereoscopic (3D)
display is performed. On the other hand, in the period tn,
frame packing arrangement data (arrangement_type="2D") is
present. For this reason, in this period, the receiver 200
performs a two-dimensional (2D) display process. In other
words, only the stream ES1 is extracted and is decoded such
that two-dimensional (2D) display is performed.
[0303]
Fig. 52 shows an example of a case where a 3D period
and a 2D period are alternately continued, and there is
auxiliary information (a newly defined SEI message
indicating a 2D mode) for identifying a mode. The periods
T1 and T3 indicate a 3D period, and the period T2 indicates
a 2D period. Each period represents, for example, the
program unit or the scene unit.
[0304]
In the 3D period, a base video stream of an MVC base
view of "Stream_Type=0x1B" is present, and an additional
video stream of an MVC non-base view of "Stream_Type=0x20"
is also present. In addition, in the 2D period, an AVC
136
stream of "Stream_Type=0x1B" is present. Further, the base
video stream has a configuration in which "SPS" is a head
and a predetermined number of access units (AU) are
continuously located. Furthermore, the additional video
stream has a configuration in which "SSSPS" is a head and a
predetermined number of access units (AU) are continuously
located. In addition, the access units (AU) are constituted
by "PPS, Substream SEIs, and Coded Slice".
[0305]
The auxiliary information for identifying a mode is
inserted into each access unit (AU) in the 2D period. The
auxiliary information indicates the 2D mode which is denoted
by "2D". In addition, the auxiliary information is not
inserted into each access unit (AU) in the 3D period.
[0306]
As above, in a case where there is auxiliary
information for identifying a mode as described above, the
receiver can immediately discriminate whether a period is a
3D period or a 2D period on the basis of the present or the
absence of the auxiliary information, and thus it is
possible to rapidly perform decoding and switching between
display processes. In a case where the 3D period is
switched to the 2D period, the receiver can determine that
the 3D period is switched to the 2D period at the
discrimination timing T2 when there is auxiliary information
137
in the first access unit and thus can rapidly perform mode
switching from 3D to 2D.
[0307]
[Case of stereo stereoscopic image]
In addition, in the above description, a description
has been made of an example in which image data of a center
view, a left end view, and a right end view for display a
multi-view stereoscopic image is transmitted from the
broadcast station 100 to the receiver 200 when a
stereoscopic (3D) image is transmitted. The present
technology is applicable in the same manner even to a case
where image data of a left eye view and a right eye view for
displaying a stereo stereoscopic image is transmitted from
the broadcast station 100 to the receiver 200 when a
stereoscopic (3D) image is transmitted.
[0308]
In this case, in a video stream included in the
transport stream TS, as shown in Fig. 53, each of image data
items of a left eye (Left) view and a right eye (Right) view
is coded as data of a single picture. In the shown example,
the data of each picture has a full HD size of 1920*1080.
In this case, for example, of a base video stream and an
additional video stream obtained by coding each of image
data items of a left eye view and a right eye view, the
multi-view view position SEI is inserted into the base video
138
stream.
[0309]
Fig. 54 shows a configuration example of a transmission
data generation unit 110B which transmits image data of a
left eye view and a right eye view for displaying a stereo
stereoscopic image in the broadcast station 100. In Fig. 54,
a part corresponding to Fig. 7 is given the same reference
numeral, and detailed description thereof will be
appropriately omitted.
[0310]
Image data (left eye image data) VL of a left eye view
output from the image data output portion 111-1 is scaled to
a full HD size of 1920*1080 by the scaler 113-1. In
addition, the scaled image data VL' is supplied to the video
encoder 114-1. The video encoder 114-1 performs coding on
the image data VL' so as to obtain coded video data, and
generates a video stream (base video stream) which includes
the coded data as a substream (sub stream 1).
[0311]
In addition, in this case, the video encoder 114-1
inserts a multi-view view position SEI message into the
video stream (base video stream) at least with the program
unit, the scene unit, the picture group unit, or the picture
unit. In a multi-view view position (Multiview view
position()) (refer to Fig. 37) included in this multi-view
139
view position SEI message, there are "view_position[0]=0"
and "view_position[1]=1".
[0312]
This indicates that a base view video stream which is a
base video stream is a video stream obtained by coding image
data of a left end view. In addition, it is indicated that
a non-base view video stream which is an additional video
stream is a video stream obtained by coding image data of a
right end view.
[0313]
Further, image data (right eye image data) VR of a
right eye view output from the image data output portion
111-2 is scaled to a full HD size of 1920*1080 by the scaler
113-2. In addition, the scaled image data VR' is supplied
to the video encoder 114-2. The video encoder 114-2
performs coding on the image data VR' so as to obtain coded
video data, and generates a video stream (additional video
stream) which includes the coded data as a substream (sub
stream 2).
[0314]
The multiplexer 115 packetizes and multiplexes the
elementary streams supplied from the respective encoders so
as to generate a transport stream TS. In this case, the
video stream (base video stream) obtained by coding the left
eye image data is transmitted as, for example, an MVC base
140
view video elementary stream (Base view sub-bitstream). In
addition, the video stream (additional video stream)
obtained by coding the right eye image data is transmitted
as, for example, an MVC non-base view video elementary
stream (Non-Base view sub-bitstream). Further, in this case,
a PTS is inserted into each PES header such that synchronous
reproduction is performed in the reception side. Detailed
description is omitted, and the remaining parts of the
transmission data generation unit 110B shown in Fig. 54 are
configured in the same manner as the transmission data
generation unit 110 shown in Fig. 7.
[0315]
Fig. 55 shows a configuration example of a receiver
200B of a stereo stereoscopic image. In Fig. 55, a part
corresponding to Fig. 27 is given the same reference numeral,
and detailed description thereof will be appropriately
omitted. The demultiplexer 214 extracts each of elementary
streams of video, disparity, graphics, and audio from the
transport stream TS which is temporarily accumulated in the
TS buffer 213.
[0316]
The video streams which are obtained by coding each of
the left eye image data and the right eye image data and are
extracted by the demultiplexer 214 are supplied to the coded
buffers 215-1 and 215-2 so as to be temporarily accumulated.
141
In addition, the video decoders 216-1 and 216-2 respectively
perform a decoding process on the video streams stored in
the coded buffers 215-1 and 215-2 under the control of the
CPU 201 so as to acquire left eye image data and right eye
image data.
[0317]
In this case, the video decoder 216-1 extracts the
multi-view view position SEI message (refer to Figs. 38 and
37) which is inserted into the video stream (base video
stream) as described above, so as to be sent to the CPU 201.
The CPU 201 controls an operation of each unit so as to
perform an operation when a stereoscopic (3D) image is
received, that is, when a stereoscopic (3D) display process
is performed, on the basis of this SEI information.
[0318]
The image data items of the respective views acquired
by the video decoders 216-1 and 216-2 are supplied to the
decoded buffers 217-1 and 217-2 so as to be temporarily
accumulated. The scalers 218-1 and 218-2 respectively
adjust output resolutions of the image data items of the
respective views output from the decoded buffers 217-1 and
217-2 so as to be predetermined resolutions.
[0319]
A superimposing unit 220B superimposes respectively
corresponding graphics bitmap data items on the left eye
142
image data and the right eye image data so as to generate
display image data for displaying a stereo stereoscopic
image. The display image data is supplied to a display, and
thereby a stereo stereoscopic (3D) image is displayed.
Detailed description is omitted, and the remaining parts of
the transmission data generation unit 200B shown in Fig. 55
are configured in the same manner as the transmission data
generation unit 200 shown in Fig. 27.
[0320]
As such, even in a case of transmitting a stereo
stereoscopic (3D) image as a stereoscopic image, it is
possible to favorably perform switching between a
stereoscopic (3D) display process and a two-dimensional (2D)
display process by using auxiliary information which
presents an element of the stereoscopic image, for example,
the above-described multi-view view position SEI in the
receiver 200B. For this reason, it is possible to
appropriately and accurately handle a dynamic variation in
delivery content and to thereby receive a correct stream.
[0321]
Figs. 56 and 57 show an example of a received stream in
a case where a 3D period (when a stereoscopic image is
received) and a 2D period (when a two-dimensional image is
received) are alternately continued. Each period is, for
example, the program unit or the scene unit. In the 3D
143
period, a video stream ES1 which is a base video stream and
includes image data of a left eye view is present, and a
video stream ES2 which is an additional video stream and
includes image data of a right eye view is also present. In
the 2D period, only video stream ES1 which is a base video
stream and includes two-dimensional image data is present.
[0322]
The example of Fig. 56 shows a case where the multiview
view position SEI message is inserted with the picture
unit in the 3D period. In addition, the example of Fig. 57
shows a case where the multi-view view position SEI is
inserted with the scene unit or the picture group unit (the
GOP unit) in 3D period.
[0323]
Fig. 58 shows an example of a case where a base video
stream ES1 of an AVC base view of "Stream_Type=0x1B" and
"PID=01" is continuously included in the transport stream TS,
and an MVC additional video stream ES2 of "Stream_Type=0x20"
and "PID=11" is intermittently included therein. In this
case, the multi-view view position SEI is inserted into the
stream ES1 in the 3D period.
[0324]
The multi-view view position SEI is present in periods
tn-1 and tn+1. For this reason, in these periods, the
receiver 200B performs a stereo stereoscopic (3D) display
144
process. In other words, the stream ES2 as well as the
stream ES1 is also extracted and is decoded so as to display
a stereo stereoscopic (3D) image.
[0325]
On the other hand, in the period tn, the multi-view
view position SEI is not present. For this reason, in this
period, the receiver 200B performs a two-dimensional (2D)
display process. In other words, only the stream ES1 is
extracted and is decoded such that two-dimensional (2D)
display is performed. At this time, in order to rapidly
transfer from a 3D processing mode to a 2D processing mode,
a processing method is also possible in which only a base
video stream is decoded, a disply process is performed for
2D display in a state in which a buffer management mode is
maintained as a 3D mode.
[0326]
In the above-described example of displaying a stereo
stereoscopic image, the multi-view view position SEI is used
as auxiliary information for identifying a mode. However,
detailed description is omitted, and there may be a
configuration in which multi-view stream configuration SEI
is used, or auxiliary information (frame packing arrangement
data or the like) indicating a 2D mode is used in the same
manner as an example of an multi-view stereoscopic image.
[0327]
145
Fig. 59 collectively shows methods of a case A, a case
B and a case C for identifying a 3D period and a 2D period
in a case where a base stream and an additional stream are
present in the 3D period and only a base stream is present
in the 2D period as described above.
[0328]
The method of the case A shown in Fig. 59(a) is a
method in which auxiliary information for identifying a mode
is inserted into a base stream in both of the 3D period and
the 2D period, and the 3D period and the 2D period can be
identified based on set content of the auxiliary information.
The method of the case A corresponds to the above-described
example of using the multi-view stream configuration SEI.
[0329]
The method of the case B shown in Fig. 59(b) is a
method in which auxiliary information indicating a 3D mode
is inserted into a base stream only in the 3D period, and
the 3D period and the 2D period can be identified based on
the presence or the absence of the auxiliary information.
The method of the case B corresponds to the above-described
example of using the multi-view view position SEI.
[0330]
The method of the case C shown in Fig. 59(c) is a
method in which auxiliary information indicating a 2D mode
is inserted into a base stream only in the 2D period, and
146
the 3D period and the 2D period can be identified based on
the presence or the absence of the auxiliary information.
The method of the case C corresponds to the above-described
example of using the auxiliary information (newly defined
SEI, frame packing arrangement data, or the like) indicating
a 2D mode.
[0331]
[Case where additional stream is present even in 2D
period]
In the above description, a description has been made
of an example in which only a base stream is present in a 2D
period. However, a configuration in a 2D period may be the
same stream configuration as in a 3D period. In other words,
for example, a base stream and an additional stream are
present in both a 3D period and a 2D period.
[0332]
In the above-described transmission data generation
unit 110 shown in Fig. 7, when a stereoscopic (3D) image is
transmitted, a base video stream of an MVC base view and two
additional video streams of an MVC non-base view are
generated as transmission video streams. In other words,
scaled image data VC' of a center (Center) view is coded so
as to obtain a base video stream of an MVC base view. In
addition, scaled image data items VL' and VR' of two views
of left end (Left) and right end (Right) are respectively
147
coded so as to obtain additional video streams of an MVC
non-base view.
[0333]
In addition, in the above-described transmission data
generation unit 110 shown in Fig. 7, for example, even when
a two-dimensional (2D) image is transmitted, a base video
stream of an MVC base view and two additional video streams
of an MVC non-base view are generated as transmission video
streams. In other words, scaled two-dimensional image data
is coded so as to obtain a base video stream of an MVC base
view. In addition, coding is performed in a coding mode
(Skipped Macro Block) in which a difference between views is
zero as a result of referring to the base video stream,
thereby obtaining two additional video streams substantially
including image data which is the same as two-dimensional
image data.
[0334]
As above, also when a two-dimensional (2D) image is
transmitted, in the same manner as when a stereoscopic (3D)
image is transmitted, a stream is configured to include a
base video stream of an MVC base view and two additional
video streams of an MVC non-base view, and thereby the
encoder can continuously operates the MVC. For this reason,
a stable operation of the transmission data generation unit
110 is expected.
148
[0335]
Here, the above-described multi-view view position SEI
message (multiview_view_position SEI message) is used as
auxiliary information for identifying a mode. The abovedescribed
transmission data generation unit 110 shown in Fig.
7 inserts the multi-view view position SEI message into a
base video stream when a stereoscopic (3D) image is
transmitted and when a two-dimensional (2D) image is
transmitted, at least with the program unit, the scene unit,
the picture group unit, or the picture unit.
[0336]
In the multi-view view position SEI message inserted
when a stereoscopic (3D) image is transmitted,
"view_position[i]" is set as follows. In other words, there
is "view_position[0]=1" which indicates a base view video
stream which is a base video stream is a video stream
obtained by coding image data of a center view.
[0337]
In addition, there is "view_position[1]=0" which
indicates that a non-base view first video stream which is
an additional video stream is a video stream obtained by
coding image data of the left end view. Further, there is
"view_position[2]=2" which indicates that a non-base view
second video stream which is an additional video stream is a
video stream obtained by coding image data of the right end
149
view.
[0338]
On the other hand, in the multi-view view position SEI
message inserted when a two-dimensional (2D) image is
transmitted, "view_position[i]" is set as follows. In other
words, all of "view_position[0]", "view_position[1]" and
"view_position[2]" are "0", "1", or "2".
[0339]
When "view_position[i]" is set in this way, a reception
side recognizes that a difference between an additional
video stream and a base video stream is zero even in a case
where the base video stream and two additional video streams
are transmitted. In other words, the reception side can
detect that a two-dimensional (2D) image is transmitted even
if a plurality of streams are transmitted, on the basis of
the setting of "view_position[i]".
[0340]
A description will be made of operation mode switching
control between a stereoscopic (3D) display process and a
two-dimensional (2D) display process in the receiver 200
shown in Fig. 27. This switching is performed by the CPU
201. When a stereoscopic (3D) image is received, the multiview
view position SEI message is extracted by the video
decoder 216-1 and is supplied to the CPU 201. The CPU 201
identifies either of a stereoscopic image transmission mode
150
and a two-dimensional image transmission mode on the basis
of set content of "view_position[i]" of the SEI message, and
controls switching between a stereoscopic (3D) display
process and a two-dimensional (2D) display process.
[0341]
Figs. 60 and 61 show an example of a received stream in
a case where a 3D period (when a stereoscopic image is
received) and a 2D period (when a two-dimensional image is
received) are alternately continued. Each period is, for
example, the program unit or the scene unit. In both the 3D
period and the 2D period, a video stream ES1 of a center
view which is a base video stream is present, and two video
streams ES2 and ES3 of a left end view and a right end view
which are additional video streams are also present.
[0342]
The example of Fig. 60 shows a case where the multiview
view position SEI message is inserted with the picture
unit in the 3D period and the 2D period. In addition, the
example of Fig. 61 shows a case where the multi-view view
position SEI is inserted with the scene unit or the picture
group unit (the GOP unit) in 3D period and 2D period.
[0343]
A flowchart of Fig. 62 shows an example of process
procedures of the operation mode switching control in the
CPU 201. The CPU 201 performs control according to the
151
flowchart for each picture frame. However, in a case where
the SEI is not inserted with the picture unit, for example,
the SEI is inserted with the GOP unit (refer to Fig. 61),
the CPU 201 maintains the current SEI information until the
SEI information of the current GOP is replaced with the SEI
information of the next GOP.
[0344]
First, the CPU 201 starts a process in step ST41, and
then proceeds to a process in step ST42. In step ST42, the
CPU 201 determines whether or not SEI
("multiview_view_position SEI message") is inserted into the
base video stream. When the SEI is inserted, the CPU 201
determines whether or not information in the SEI, that is,
set content of "view_position[i]" indicates a 3D mode in
step ST43.
[0345]
When the set content of "view_position[i]" in the SEI
indicates a 3D mode, that is, when a stereoscopic (3D) image
is received, the CPU 201 proceeds to a process in step ST44.
The CPU 201 manages the respective input buffers (coded
buffers) of the base video stream and the additional video
stream in step ST44, and decodes the base video stream and
the additional video stream, respectively, by using the
decoders (video decoders) in step ST45. Further, the CPU
201 performs control such that the receiver 200 performs
152
other stereoscopic (3D) display processes in step ST46.
[0346]
In addition, when the SEI is not inserted in step ST42,
or when the set content of "view_position[i]" in the SEI
does not indicate a 3D mode in step ST43, that is, when a
two-dimensional (2D) image is received, the CPU 201 proceeds
to a process in step ST47. The CPU 201 manages an input
buffer (coded buffer) of the base video stream in step ST47,
and decodes the base video stream by using the decoder
(video decoder) in step ST48. Further, the CPU 201 performs
control such that the receiver 200 performs other twodimensional
(2D) display processes in step ST49.
[0347]
Fig. 63 shows an example of a reception packet process
when a stereoscopic (3D) image is received in the receiver
200 shown in Fig. 27. NAL packets of a base video stream
and an additional video stream are mixed and are transmitted.
Fig. 64 shows a configuration example (Syntax) of a NAL unit
header and MVC extension of the NAL unit header (NAL unit
header MVC extension). The field of "view_id" indicates
what number view is a corresponding view. As shown in Fig.
63, the receiver 200 assigns the NAL packets which are mixed
and are transmitted to each stream and decodes each stream
on the basis of a combination of a value of the NAL unit
type and a view ID (view_id) of NAL unit header MVC
153
extension (Headermvc extension).
[0348]
Fig. 65 shows an example of a reception packet process
when a two-dimensional (2D) image is received in the
receiver 200 shown in Fig. 27. NAL packets of a base video
stream and an additional video stream are mixed and are
transmitted. As shown in Fig. 65, the receiver 200 assigns
the NAL packets which are mixed and are transmitted to each
stream and decodes only the base video stream on the basis
of a combination of a value of the NAL unit type and a view
ID (view_id) of NAL unit header MVC extension (Headermvc
extension).
[0349]
In other words, also when a two-dimensional (2D) image
is received, in the same manner as when a stereoscopic (3D)
image is received, the receiver 200 receives a base video
stream and an additional video stream but performs a twodimensional
(2D) image process without decoding a slice of
the overall picture subsequent to the SEI unlike in the
related art, on the basis of set content of
"view_position[i]" of the multi-view view position SEI
message.
[0350]
As above, since identification can be performed at a
packet (NAL packet) level without decoding coded data of an
154
additional video stream, it is possible to perform a rapid
transfer to a 2D display mode in the receiver 200. In
addition, since layers equal to or lower than the slice
layer are not decoded and can be discarded, memory
consumption can be suppressed to that extent so as to save
power or allocate a CPU budget of a system, a memory space
bandwidth, or the like to other features (for example, high
performance graphics), thereby achieving multiple functions.
[0351]
In addition, when a two-dimensional (2D) image is
received, in the same manner as when a stereoscopic (3D)
image is received, the receiver 200 receives a base video
stream and an additional video stream, but performs a twodimensional
(2D) image process without performing a
stereoscopic (3D) image process. For this reason, it is
possible to obtain display image quality equivalent to the
related art type 2D display.
[0352]
In other words, in a case of performing a stereoscopic
(3D) image process when a two-dimensional (2D) image is
received, image data obtained by decoding a base video
stream is the same as image data obtained by decoding an
additional video stream. For this reason, if display is
performed in a 3D mode, the display is flat, that is, the
display without a disparity is performed, and thereby there
155
is a possibility that image quality may deteriorate as
compared with performing the related art type 2D display.
For example, if stereo stereoscopic image display is
considered, this may occur in both passive type (using
polarization glasses) and active type (using shutter
glasses) 3D monitors.
[0353]
In 3D display performed by many passive type monitors,
data items of a left eye view (Left view) and a right eye
view (Right view) are alternately displayed with the display
line unit in a vertical direction, so as to realize 3D, but,
in a case where image data items of two views are the same,
a vertical resolution is just a half of 2D display in the
related art. On the other hand, in 3D display performed by
active type monitors, frames are alternately switched to a
left eye view and a right eye view in a temporal direction
and are displayed, but, in a case where image data items of
two views are the same, a resolution in the temporal
direction is a half of 2D display in the related art.
[0354]
Fig. 66 shows an example of a case where a base video
stream ES1 of an MVC base view of "Stream_Type=0x1B" and
"PID=01" is continuously included in the transport stream TS,
and MVC additional video streams ES2 and ES3 of
"Stream_Type=0x20", "PID=10" and "PID=11" are also
156
continuously included therein. In this case, the multi-view
view position SEI is inserted into the stream ES1 in the 3D
period and 2D period.
[0355]
In periods tn-1 and tn+1, for example, there are
"view_position[0]=1", "view_position[1]=0", and
"view_position[2]=2", which indicate a 3D mode. For this
reason, in these periods, the receiver 200 performs a
stereoscopic (3D) display process. In other words, the
streams ES2 and ES3 as well as the stream ES1 are also
extracted and are decoded such that stereoscopic (3D)
display is performed.
[0356]
On the other hand, in the period tn, for example, there
are "view_position[0]=0", "view_position[1]=0", and
"view_position[2]=0", which indicate a 2D mode. For this
reason, in this period, the receiver 200 performs a twodimensional
(2D) display process. In other words, only the
stream ES1 is extracted and is decoded such that twodimensional
(2D) display is performed.
[0357]
Fig. 67 shows an example of a case where a 3D period
(3D mode period) and a 2D period (2D mode period) are
alternately continued, and there is auxiliary information
(multi-view view position SEI message) for identifying a
157
mode. The periods T1 and T3 indicate a 3D period, and the
period T2 indicates a 2D period. Each period represents,
for example, the program unit or the scene unit.
[0358]
In both the 3D period and the 2D period, a base video
stream of an MVC base view of "Stream_Type=0x1B" is present,
and an additional video stream of an MVC non-base view of
"Stream_Type=0x20" is also present. Further, the base video
stream has a configuration in which "SPS" is a head and a
predetermined number of access units (AU) are continuously
located.
[0359]
Furthermore, the additional video stream has a
configuration in which "SSSPS" is a head and a predetermined
number of access units (AU) are continuously located. In
addition, the access units (AU) are constituted by "PPS,
Substream SEIs, and Coded Slice". However, the additional
video stream in the 2D period is coded in a coding mode
(Skipped Macro Block) in which a difference between views is
zero as a result of referring to the base video stream. The
additional video stream in this period has a configuration
in which "SSSPS" is a head and a predetermined number of
access units (AV) are continuously located. The access
units (AV) are constituted by "PPS, Substream SEIs, and
Slice Skipped MB".
158
[0360]
The auxiliary information (multi-view view position SEI
message) for identifying a mode is inserted for each access
unit (AU). The auxiliary information inserted into the
access unit in the 3D period is indicated by "3D", and
"view_position[i]" is a value indicating a relative
positional relationship of each view and indicates a 3D mode
(stereoscopic image transmission mode). On the other hand,
the auxiliary information inserted into the access unit in
the 2D period is indicated by "2D", and "view_position[i]"
is the same value in each view and indicates a 2D mode (twodimensional
image transmission mode). In other words, this
case indicates that flat 3D display is performed when a
reception side performs a 3D display process.
[0361]
As above, in a case where there is auxiliary
information (multi-view view position SEI message) for
identifying a mode, the receiver checks the element
"view_position[i]" of the auxiliary information, and can
immediately discriminate whether the element indicates a 3D
mode or a 2D mode, and thus it is possible to rapidly
perform decoding and switching between display processes.
In a case where the 3D period is switched to the 2D period,
the receiver can determine that the 3D period is switched to
the 2D period at the discrimination timing T2 when the
159
element "view_position[i]" of the auxiliary information
inserted into the first access unit indicates a 2D mode, and
thus can rapidly perform mode switching from 3D to 2D.
[0362]
In addition, in the above description, a description
has been made of an example of using the multi-view view
position SEI message as auxiliary information for
identifying a mode. Detailed description is omitted, and,
other auxiliary information, for example, a multi-view
stream configuration SEI message (refer to Figs. 21 and 14)
may be used.
[0363]
[Another example of auxiliary information for
identifying mode]
In the above description, a description has been made
of an example in which auxiliary information for identifying
a mode, for example, the multi-view view position SEI
message is inserted in both a 3D period and a 2D period, and
the receiver discriminates a 3D period or a 2D period with
frame accuracy on the basis of set content thereof. However,
auxiliary information indicating a 3D mode may be inserted
only in a 3D period, and a 3D period or a 2D period may be
discriminated with frame accuracy on the basis of the
presence or the absence thereof. Also in this case, for
example, the multi-view view position SEI message may be
160
used as auxiliary information.
[0364]
The transmission data generation unit 110 shown in Fig.
7 described above inserts the multi-view view position SEI
message into a video stream (base video stream) which is
obtained by coding image data of an intermediate view in a
3D mode (stereoscopic image transmission mode). The multiview
view position SEI message forms identification
information indicating a 3D mode. In this case, the message
is inserted at least with the program unit, the scene unit,
the picture group unit, or the picture unit.
[0365]
Figs. 68 and 69 show an example of a received stream in
a case where a 3D period (when a stereoscopic image is
received) and a 2D period (when a two-dimensional image is
received) are alternately continued. Each period is, for
example, the program unit or the scene unit. In both the 3D
period and the 2D period, a video stream ES1 of acenter view
which is a base video stream is present, and two video
streams ES2 and ES3 of a left end view and a right end view
which are additional video streams are also present.
[0366]
The example of Fig. 68 shows a case where the multiview
view position SEI message is inserted with the picture
unit in the 3D period. In addition, the example of Fig. 69
161
shows a case where the multi-view view position SEI is
inserted with the scene unit or the picture group unit (the
GOP unit) in 3D period.
[0367]
Detailed description is omitted, and process procedures
of the operation mode switching control in the CPU 201 are
also shown by the above-described flowchart of Fig. 41. The
CPU 201 performs control according to the flowchart for each
picture frame. However, in a case where the SEI is not
inserted with the picture unit, for example, the SEI is
inserted with the GOP unit (refer to Fig. 69), the CPU 201
maintains the current SEI information until the SEI
information of the current GOP is replaced with information
of the presence or the absence of the SEI of the next GOP.
[0368]
As described above, also by inserting the multi-view
view position SEI message only in a 3D period, a reception
side can favorably perform switching between a stereoscopic
(3D) display process and a two-dimensional (2D) display
process on the basis of the presence of the absence of the
SEI message. For this reason, it is possible to
appropriately and accurately handle a dynamic variation in
delivery content and to thereby receive a correct stream.
[0369]
Fig. 70 shows an example of a case where a base video
162
stream ES1 of an MVC base view of "Stream_Type=0x1B" and
"PID=01" is continuously included in the transport stream TS,
and MVC additional video streams ES2 and ES3 of
"Stream_Type=0x20", "PID=10" and "PID=11" are continuously
included therein. In this case, the multi-view view
position SEI is inserted into the stream ES1 in the 3D
period.
[0370]
The multi-view view position SEI is present in periods
tn-1 and tn+1. For this reason, in these periods, the
receiver 200 performs a stereoscopic (3D) display process.
In other words, the streams ES2 and ES3 as well as the
stream ES1 are also extracted and are decoded such that
stereoscopic (3D) display is performed. On the other hand,
in the period tn, the multi-view view position SEI is not
present. For this reason, in this period, the receiver 200
performs a two-dimensional (2D) display process. In other
words, only the stream ES1 is extracted and is decoded such
that two-dimensional (2D) display is performed.
[0371]
Fig. 71 shows an example of a case where a 3D period
(3D mode period) and a 2D period (2D mode period) are
alternately continued, and there is auxiliary information
(multi-view view position SEI message) for identifying a
mode. The periods T1 and T3 indicate a 3D period, and the
163
period T2 indicates a 2D period. Each period represents,
for example, the program unit or the scene unit. In the
same manner as the above-described example of Fig. 67, in
both the 3D period and the 2D period, a base video stream of
an MVC base view of "Stream_Type=0x1B" is present, and an
additional video stream of an MVC non-base view of
"Stream_Type=0x20" is also present.
[0372]
The auxiliary information (multi-view view position SEI
message) for identifying a mode is inserted for each access
unit (AU) in the 3D period. The auxiliary information
indicates the 3D mode which is denoted by "3D". In addition,
the auxiliary information is not inserted into each access
unit (AU) in the 2D period.
[0373]
As above, in a case where there is auxiliary
information for identifying a mode as described above, the
receiver can immediately discriminate whether a period is a
3D period or a 2D period on the basis of the presence or the
absence of the auxiliary information, and thus it is
possible to rapidly perform decoding and switching between
display processes. In a case where the 3D period is
switched to the 2D period, the receiver can determine that
the 3D period is switched to the 2D period at the
discrimination timing T2 when there is no auxiliary
164
information in the first access unit and thus can rapidly
perform mode switching from 3D to 2D.
[0374]
[Still another example of auxiliary information for
identifying mode]
In the above description, a description has been made
of an example in which the multi-view view position SEI
message is used as auxiliary information for identifying a
mode, and the receiver discriminates a 3D period or a 2D
period with frame accuracy on the basis of set content
thereof or the presence or the absence thereof. As
auxiliary information for identifying a mode, still another
auxiliary information may be used. That is, auxiliary
information indicating a 2D mode is used.
[0375]
As identification information indicating a 2D mode, a
SEI message which is newly defined may be used. In addition,
in a case of an MPEG2 stream, existing frame packing
arrangement data (frame_packing_arrangement_data()) may be
used (refer to Figs. 45 and 46).
[0376]
The transmission data generation unit 110 shown in Fig.
7 described above inserts auxiliary information indicating a
2D mode into a video stream (base video stream) which is
obtained by coding image data of an intermediate view in a
165
2D mode (stereoscopic image transmission mode). For example,
in a case where this stream is an MPEG2 stream, the abovedescribed
frame packing arrangement data
(arrangement_type=0001000) is inserted into the user data
region. In this case, the data is inserted at least with
the program unit, the scene unit, the picture group unit, or
the picture unit.
[0377]
A description will be made of operation mode switching
control between a stereoscopic (3D) display process and a
two-dimensional (2D) display process in the receiver 200
shown in Fig. 27 in a case of using the auxiliary
information indicating a 2D mode. This switching is
performed by the CPU 201. When a two-dimensional (2D) image
is received, the auxiliary information indicating a 2D mode
is extracted by the video decoder 216-1 and is supplied to
the CPU 201. However, when a stereoscopic (3D) image is
received, the auxiliary information is not extracted by the
video decoder 216-1 and thus is not supplied to the CPU 201.
The CPU 201 controls switching between a stereoscopic (3D)
display process and a two-dimensional (2D) display process
on the basis of the presence or the absence of the auxiliary
information.
[0378]
Figs. 72 and 73 show an example of a received stream in
166
a case where a 3D period (when a stereoscopic image is
received) and a 2D period (when a two-dimensional image is
received) are alternately continued. Each period is, for
example, the program unit or the scene unit. In both the 3D
period and the 2D period, a video stream ES1 of an center
view which is a base video stream is present, and two video
streams ES2 and ES3 of a left end view and a right end view
which are additional video streams are also present. The
example of Fig. 72 shows a case where the auxiliary
information indicating a 2D mode is inserted with the
picture unit in the 2D period. In addition, the example of
Fig. 73 shows a case where the auxiliary information
indicating a 2D mode is inserted with the scene unit or the
picture group unit (the GOP unit) in the 2D period.
[0379]
Detailed description is omitted, and process procedures
of the operation mode switching control in the CPU 201 in
this case are also shown, for example, by the flowchart of
Fig. 50. The CPU 201 performs control according to the
flowchart for each picture frame. However, in a case where
the SEI is not inserted with the picture unit, for example,
the SEI is inserted with the GOP unit (refer to Fig. 73),
the CPU 201 maintains the current SEI information until the
SEI information of the current GOP is replaced with
information of the presence or the absence of the SEI of the
167
next GOP.
[0380]
As described above, also by inserting the auxiliary
information indicating a 2D mode only in a 2D period, it is
possible to favorably perform switching between a
stereoscopic (3D) display process and a two-dimensional (2D)
display process on the basis of the presence or the absence
of identification information thereof. For this reason, it
is possible to appropriately and accurately handle a dynamic
variation in delivery content and to thereby receive a
correct stream.
[0381]
Fig. 74 shows an example of a case where a base video
stream ES1 of an MPEG2 base view of "Stream_Type=0x02" and
"PID=01" is continuously included in the transport stream TS,
and AVC additional video streams ES2 and ES3 of
"Stream_Type=0x23", "PID=10" and "PID=11" are continuously
included therein.
[0382]
The fram packing arrangement data
(arrangement_type="2D") is not present in periods tn-1 and
tn+1. For this reason, in these periods, the receiver 200
performs a stereoscopic (3D) display process. In other
words, the streams ES2 and ES3 as well as the stream ES1 are
also extracted and are decoded such that stereoscopic (3D)
168
display is performed. On the other hand, in the period tn,
the fram packing arrangement data (arrangement_type="2D") is
present. For this reason, in this period, the receiver 200
performs a two-dimensional (2D) display process. In other
words, only the stream ES1 is extracted and is decoded such
that two-dimensional (2D) display is performed.
[0383]
Fig. 75 shows an example of a case where a 3D period
(3D mode period) and a 2D period (2D mode period) are
alternately continued, and there is auxiliary information (a
newly defined SEI message indicating a 2D mode) for
identifying a mode. The periods T1 and T3 indicate a 3D
period, and the period T2 indicates a 2D period. Each
period represents, for example, the program unit or the
scene unit. In the same manner as the above-described
example of Fig. 67, in both the 3D period and the 2D period,
a base video stream of an MVC base view of
"Stream_Type=0x1B" is present, and an additional video
stream of an MVC non-base view of "Stream_Type=0x20" is also
present.
[0384]
The auxiliary information for identifying a mode is
inserted for each access unit (AU) in the 2D period. The
auxiliary information indicates the 2D mode which is denoted
by "2D". In addition, the auxiliary information is not
169
inserted into each access unit (AU) in the 3D period.
[0385]
As above, in a case where there is auxiliary
information for identifying a mode as described above, the
receiver can immediately discriminate whether a period is a
3D period or a 2D period on the basis of the presence or the
absence of the auxiliary information, and thus it is
possible to rapidly perform decoding and switching between
display processes. In a case where the 3D period is
switched to the 2D period, the receiver can determine that
the 3D period is switched to the 2D period at the
discrimination timing T2 when there is auxiliary information
in the first access unit and thus can rapidly perform mode
switching from 3D to 2D.
[0386]
[Case of stereo stereoscopic image]
Figs. 76 and 77 show an example of a received stream in
a case where a 3D period (when a stereoscopic image is
received) and a 2D period (when a two-dimensional image is
received) are alternately continued. However, this example
is an example of a case where stereoscopic (3D) image
display is stereo stereoscopic image display (refer to Figs.
54 and 55). Each period is, for example, the program unit
or the scene unit. In both the 3D period and the 2D period,
a video stream ES1 which is a base video stream and includes
170
image data of a left eye view is present, and a video stream
ES2 which is an additional video stream and includes image
data of a right eye view is also present.
[0387]
The example of Fig. 76 shows a case where the multiview
view position SEI message is inserted with the picture
unit in the 3D period and the 2D period. In addition, the
example of Fig. 77 shows a case where the multi-view view
position SEI is inserted with the scene unit or the picture
group unit (the GOP unit) in 3D period and the 2D period.
[0388]
Fig. 78 shows an example of a case where a base video
stream ES1 of an MVC base view of "Stream_Type=0x1B" and
"PID=01" is continuously included in the transport stream TS,
and an MVC additional video stream ES2 of "Stream_Type=0x20"
and "PID=10" is also continuously included therein. In this
case, the multi-view view position SEI is inserted into the
stream ES1 in the 3D period and 2D period.
[0389]
In periods tn-1 and tn+1, for example, there are
"view_position[0]=0" and "view_position[1]=1" which indicate
a 3D mode. For this reason, in these periods, the receiver
200 performs a stereoscopic (3D) display process. In other
words, the stream ES2 as well as the stream ES1 is extracted
and is decoded such that stereoscopic (3D) display is
171
performed.
[0390]
On the other hand, in the period tn, for example, there
are "view_position[0]=0" and "view_position[1]=0" which
indicate a 2D mode. For this reason, in this period, the
receiver 200 performs a two-dimensional (2D) display process.
In other words, only the stream ES1 is extracted and is
decoded such that two-dimensional (2D) display is performed.
[0391]
In the above-described example of stereo stereoscopic
image display, the multi-view view position SEI is inserted
in both a 3D period and a 2D period as auxiliary information
for identifying a mode, and the receiver identifies the 3D
period or the 2D period on the basis of set content thereof.
Detailed description is omitted, and an example of inserting
auxiliary information indicating a 3D mode only in the 3D
period or an example of inserting auxiliary information
indicating a 2D mode only in the 2D period can be treated in
the same manner.
[0392]
Fig. 79 collectively shows methods of a case D, a case
E and a case F for identifying a 3D period and a 2D period
in a case where a base stream and an additional stream are
present in both the 3D period and the 2D period as described
above.
172
[0393]
The method of the case D shown in Fig. 79(a) is a
method in which auxiliary information for identifying a mode
is inserted into a base stream in both of the 3D period and
the 2D period, and the 3D period and the 2D period can be
identified based on set content of the auxiliary information.
In the above description, a description has been made of an
example of using, for example, multi-view view position SEI
as auxiliary information.
[0394]
The method of the case E shown in Fig. 79(b) is a
method in which auxiliary information indicating a 3D mode
is inserted into a base stream only in the 3D period, and
the 3D period or the 2D period can be identified based on
the presence or the absence of the auxiliary information.
In the above description, a description has been made of an
example of using, for example, multi-view view position SEI
as auxiliary information.
[0395]
The method of the case F shown in Fig. 79(c) is a
method in which auxiliary information indicating a 2D mode
is inserted into a base stream only in the 2D period, and
the 3D period and the 2D period can be identified based on
the presence or the absence of the auxiliary information.
In the above description, a description has been made of an
173
example of using, for example, newly defined SEI, frame
packing arrangement data, or the like, as auxiliary
information.
[0396]
As described above, in the present technology, it is
possible to rapidly identify whether a mode is a 3D image
transmission mode or a 2D image transmission mode in a
reception side in stream configurations as shown in Figs. 80
and 81.
[0397]
Fig. 80 shows a stream configuration example 1 in which
a base video stream and an additional video stream are
transmitted in a 3D period (3D image transmission mode) and
a single video stream (only a base video stream) is
transmitted in a 2D period (2D image transmission mode). In
addition, Fig. 81 shows a stream configuration example 2 in
which a base video stream and an additional video stream are
transmitted in both a 3D period (3D image transmission mode)
and a 2D period (2D image transmission mode). However, the
additional video stream in the 2D period is coded in a
coding mode (Skipped Macro Block) in which a difference
between views is zero as a result of referring to the base
video stream. In the configuration examples 1 and 2, as
described above, it is possible to identify the 3D period
and the 2D period with frame accuracy by using the present
174
technology.
[0398]
[Signaling information of video layer and 3D and 2D
identification information of system layer]
In the above description, a description has been made
of an example in which a 3D period or a 2D period is
determined with frame accuracy on the basis of auxiliary
information inserted into a video stream, that is, auxiliary
information (signaling information) of a video layer. In
this case, the receiver is required to check a part
corresponding to associated auxiliary information at all
times.
[0399]
It is considered that a 3D period or a 2D period is
determined based on a combination of auxiliary information
(signaling information) of the video layer and 3D and 2D
identification information (signaling information) of the
system layer. In this case, the receiver first detects
identification information of the system layer and can check
a part corresponding to auxiliary information of an
associated video layer.
[0400]
(Configuration Example 1)
Fig. 82 shows an example in which a base video stream
and an additional video stream are present in both a 3D
175
period and a 2D period, and signaling is performed using
both a program loop (Program_loop) and a video ES loop
(video ES_loop) of a Program Map Table (PMT).
[0401]
In this example, in both a 3D period (event 1) and a 3D
period (event 2), there are present a base video stream of
an MPEFG2 base view of "Stream_Type=0x02" and an AVC
additional video stream of "Stream_Type=0x23. In this
example, "L" indicates left eye image data, and "R"
indicates right eye image data. When a base video stream is
"L" and an additional video stream is "R", normal 3D display
can be performed, and, when a base video stream is "L" and
an additional video stream is "L", flat 3D display is
performed.
[0402]
In a case of this example, the transmission data
generation unit 110B shown in Fig. 54 inserts frame packing
arrangement data (arrangement_type="2D") indicating a 2D
mode into the user data region of the base video stream with
the picture unit in the 2D period. Thereby, the receiver
can determine a 2D period or a 3D period with frame accuracy
in the video layer.
[0403]
In addition, in a case of this example, signaling is
performed using both a program loop (Program_loop) and a
176
video ES loop (Video ES_loop) of a Program Map Table (PMT).
A stereoscopic program information descriptor
(Stereoscopic_program_info_descriptor) is disposed in the
program loop.
[0404]
Fig. 83(a) shows a structural example (Syntax) of a
stereoscopic program information descriptor.
"descriptor_tag" is 8-bit data indicating a descriptor type,
and, here, indicates a stereoscopic program information
descriptor. "descriptor_length" is 8-bit data indicating a
length (size) of the descriptor. This data is a length of
the descriptor and indicates the number of subsequent bytes.
[0405]
The 3-bit field of "stereoscopic_service_type"
designates a service type. Fig. 83(b) shows a relationship
a value of "stereoscopic_service_type" and a service type.
For example, "011" indicates a service-compatible
stereoscopic 3D service, and "001" indicates a 2D service.
[0406]
Returning to the example of Fig. 82, a value of
"stereoscopic_service_type" of the stereoscopic program
information descriptor disposed in the program loop of the
Program Map Table (PMT) is "011" in a 3D period and is "001"
in a 2D period.
[0407]
177
In addition, in a 2D period, an MPEG2 stereoscopic
video descriptor (MPEG2_stereoscopic_video_format
descriptor) is disposed in the video ES loop. Fig. 84 shows
a structural example (Syntax) of the MPEG2 stereoscopic
video descriptor. "descriptor_tag" is 8-bit data indicating
a descriptor type, and, here, indicates an MPEG2
stereoscopic video descriptor. "descriptor_length" is 8-bit
data indicating a length (size) of the descriptor. This
data is a length of the descriptor and indicates the number
of subsequent bytes.
[0408]
If "Stereo_video_arrangement_type_present" is "1", this
indicates that 7-bit "arrangement_type" subsequent thereto
is "stereo_video_format_type". This is defined in the same
manner as "arrangement_type" of frame packing arrangement
data (frame_packing_arrangement_data()) which is inserted
into the user region as described above (refer to Fig. 46).
On the other hand, if
"Stereo_video_arrangement_type_present" is "0", this
indicates a reserved region in which there is no information
in 7 bits subsequent thereto.
[0409]
As described above, in the MPEG2 stereoscopic video
descriptor disposed in the video ES loop in a 2D period,
"Stereo_video_arrangement_type_present" is "1", and
178
"arrangement_type" indicates "2D".
[0410]
A description will be made of operation mode switching
control between a stereoscopic (3D) display process and a
two-dimensional (2D) display process in the receiver 200B
shown in Fig. 55 In a case where signaling is performed
using the video layer and the system layer as shown in Fig.
82,. This switching is performed by the CPU 201.
[0411]
When a two-dimensional (2D) image is received, the
stereoscopic program information descriptor
(stereoscopic_service_type="001") and the MPEG2 stereoscopic
video descriptor (arrangement_type="2D") are extracted by
the demultiplexer 215 and are supplied to the CPU 201.
In addition, when the two-dimensional (2D) image is
received, the frame packing arrangement data
(arrangement_type="2D") is extracted by the video decoder
216-1 and is supplied to the CPU 201. On the other hand,
when a stereoscopic (3D) image is received, the stereoscopic
program information descriptor
(stereoscopic_service_type="011") is extracted by the
demultiplexer 215 and is supplied to the CPU 201.
[0412]
The CPU 201 performs control for switching from a twodimensional
(2D) display process to a stereoscopic (3D)
179
display process at the frame (picture) timing (indicated by
"Ta") when the frame packing arrangement data
(arrangement_type="2D") is not extracted after only the
stereoscopic program information descriptor
(stereoscopic_service_type="011") is extracted.
[0413]
In addition, the CPU 201 performs control for switching
from a stereoscopic (3D) display process to a twodimensional
(2D) display process at the frame (picture)
timing (indicated by "Tb") when the frame packing
arrangement data (arrangement_type="2D") is extracted after
the stereoscopic program information descriptor
(stereoscopic_service_type="001") and the MPEG2 stereoscopic
video descriptor (arrangement_type="2D") is extracted.
[0414]
Fig. 85 shows a configuration example of the transport
stream TS. In addition, in this configuration example, for
simplification of the figure, disparity data, audio,
graphics, and the like are not shown. The transport stream
TS includes a PES packet "video PES1" of a base video stream
(MPEG2 stream) of "PID1" and also includes a PES packet
"video PES1" of an additional video stream (AVC stream) of
"PID2". Only in the 2D period, frame packing arrangement
data (arrangement_type="2D") indicating a 2D mode is
inserted into the user data region of the base video stream
180
with the picture unit.
[0415]
In addition, stereoscopic program information
descriptor (Stereoscopic_program_info_descriptor) is
disposed in the program loop under the PMT.
"stereoscopic_service_type" of the descriptor is "011" in
the 3D period, which indicates a 3D service, and is "001" in
the 2D period, which indicates a 2D service.
[0416]
In addition, the MPEG2 stereoscopic video descriptor
(MPEG2_stereoscopic_video_format descriptor) is disposed in
the video ES loop under the PMT as information regarding a
base video stream only in a case of the 2D period.
"arrangement_type" of the descriptor is "2D". This
indicates a 2D service. Conversely, if the descriptor is
not present, this indicates a 3D service.
[0417]
(Configuration Example 2)
Fig. 86 shows an example in which a base video stream
and an additional video stream are present in both a 3D
period and a 2D period, and signaling is performed using a
video ES loop (video ES_loop) of the PMT. In addition, in
Fig. 86, description of a part corresponding to Fig. 82 will
be appropriately omitted.
[0418]
181
In a case of this example, the transmission data
generation unit 110B shown in Fig. 54 inserts frame packing
arrangement data (arrangement_type="2D") indicating a 2D
mode into the user data region of the base video stream with
the picture unit in the 2D period. Thereby, the receiver
can determine a 2D period or a 3D period with frame accuracy
in the video layer.
[0419]
In addition, in a case of this example, a stereoscopic
program information descriptor
(Stereoscopic_program_info_descriptor) is disposed in the
program loop of the PMT. A value of
"stereoscopic_service_type" of the descriptor is "011" in
both a 3D period and a 2D period. In addition, in a case of
this example, in the 2D period, an MPEG2 stereoscopic video
descriptor (MPEG2_stereoscopic_video_format descriptor) is
disposed in the video ES loop. In this descriptor,
"arrangement_type" indicates "2D".
[0420]
A description will be made of operation mode switching
control between a stereoscopic (3D) display process and a
two-dimensional (2D) display process in the receiver 200B
shown in Fig. 55 in a case where signaling is performed
using the video layer and the system layer as shown in Fig.
86. This switching is performed by the CPU 201.
182
[0421]
When a two-dimensional (2D) image is received, the
stereoscopic program information descriptor
(stereoscopic_service_type="011") and the MPEG2 stereoscopic
video descriptor (arrangement_type="2D") are extracted by
the demultiplexer 215 and are supplied to the CPU 201. In
addition, when the two-dimensional (2D) image is received,
the frame packing arrangement data (arrangement_type="2D")
is extracted by the video decoder 216-1 and is supplied to
the CPU 201. On the other hand, when a stereoscopic (3D)
image is received, only the stereoscopic program information
descriptor (stereoscopic_service_type="011") is extracted by
the demultiplexer 215 and is supplied to the CPU 201.
[0422]
The CPU 201 performs control for switching from a twodimensional
(2D) display process to a stereoscopic (3D)
display process at the frame (picture) timing (indicated by
"Ta") when the frame packing arrangement data
(arrangement_type="2D") is not extracted after only the
stereoscopic program information descriptor
(stereoscopic_service_type="011") is extracted.
[0423]
In addition, the CPU 201 performs control for switching
from a stereoscopic (3D) display process to a twodimensional
(2D) display process at the frame (picture)
183
timing (indicated by "Tb") when the frame packing
arrangement data (arrangement_type="2D") is extracted after
the stereoscopic program information descriptor
(stereoscopic_service_type="001") and the MPEG2 stereoscopic
video descriptor (arrangement_type="2D") are extracted.
[0424]
(Configuration Example 3)
Fig. 87 shows an example in which a base video stream
and an additional video stream are present in both a 3D
period and a 2D period, and signaling is performed using a
program loop (Program_loop) of the PMT. In addition, in Fig.
87, description of a part corresponding to Fig. 82 will be
appropriately omitted.
[0425]
In a case of this example, the transmission data
generation unit 110B shown in Fig. 54 inserts frame packing
arrangement data (arrangement_type="2D") indicating a 2D
mode into the user data region of the base video stream with
the picture unit in the 2D period. Thereby, the receiver
can determine a 2D period or a 3D period with frame accuracy
in the video layer.
[0426]
In addition, in a case of this example, a stereoscopic
program information descriptor
(Stereoscopic_program_info_descriptor) is disposed in the
184
program loop of the PMT. A value of the descriptor is "011"
in a 3D period and is "001" in a 2D period.
[0427]
A description will be made of operation mode switching
control between a stereoscopic (3D) display process and a
two-dimensional (2D) display process in the receiver 200B
shown in Fig. 55 in a case where signaling is performed
using the video layer and the system layer as shown in Fig.
87. This switching is performed by the CPU 201.
[0428]
When a two-dimensional (2D) image is received, the
stereoscopic program information descriptor
(stereoscopic_service_type="001") is extracted by the
demultiplexer 215 and is supplied to the CPU 201. In
addition, when the two-dimensional (2D) image is received,
the frame packing arrangement data (arrangement_type="2D")
is extracted by the video decoder 216-1 and is supplied to
the CPU 201. On the other hand, when a stereoscopic (3D)
image is received, the stereoscopic program information
descriptor (stereoscopic_service_type="011") is extracted by
the demultiplexer 215 and is supplied to the CPU 201.
[0429]
The CPU 201 performs control for switching from a twodimensional
(2D) display process to a stereoscopic (3D)
display process at the frame (picture) timing (indicated by
185
"Ta") when the frame packing arrangement data
(arrangement_type="2D") is not extracted after the
stereoscopic program information descriptor
(stereoscopic_service_type="011") is extracted.
[0430]
In addition, the CPU 201 performs control for switching
from a stereoscopic (3D) display process to a twodimensional
(2D) display process at the frame (picture)
timing (indicated by "Tb") when the frame packing
arrangement data (arrangement_type="2D") is extracted after
the stereoscopic program information descriptor
(stereoscopic_service_type="001") is extracted.
[0431]
(Configuration Example 4)
Fig. 88 shows an example in which a base video stream
and an additional video stream are present in a 3D period,
only a base video stream is present in a 2D period, and
signaling is performed using both a program loop
(Program_loop) and a video ES loop (video ES_loop) of the
PMT. In addition, in Fig. 88, description of a part
corresponding to Fig. 82 will be appropriately omitted.
[0432]
In a case of this example, the transmission data
generation unit 110B shown in Fig. 54 inserts frame packing
arrangement data (arrangement_type="2D") indicating a 2D
186
mode into the user data region of the base video stream with
the picture unit in the 2D period. Thereby, the receiver
can determine a 2D period or a 3D period with frame accuracy
in the video layer.
[0433]
In addition, in a case of this example, a stereoscopic
program information descriptor
(Stereoscopic_program_info_descriptor) is disposed in the
program loop of the PMT. A value of
"stereoscopic_service_type" of the descriptor is "011" in a
3D period and is "001" in a 2D period. Further, in a case
of this example, the MPEG2 stereoscopic video descriptor
(MPEG2_stereoscopic_video_format descriptor) is disposed in
the video ES loop in the 2D period. In the descriptor,
"arrangement_type" indicates "2D".
[0434]
A description will be made of operation mode switching
control between a stereoscopic (3D) display process and a
two-dimensional (2D) display process in the receiver 200B
shown in Fig. 55 in a case where signaling is performed
using the video layer and the system layer as shown in Fig.
88,. This switching is performed by the CPU 201.
[0435]
When a two-dimensional (2D) image is received, the
stereoscopic program information descriptor
187
(stereoscopic_service_type="001") and the MPEG2 stereoscopic
video descriptor (arrangement_type="2D") are extracted by
the demultiplexer 215 and are supplied to the CPU 201. In
addition, when the two-dimensional (2D) image is received,
the frame packing arrangement data (arrangement_type="2D")
is extracted by the video decoder 216-1 and is supplied to
the CPU 201. On the other hand, when a stereoscopic (3D)
image is received, the stereoscopic program information
descriptor (stereoscopic_service_type="011") is extracted by
the demultiplexer 215 and is supplied to the CPU 201.
[0436]
The CPU 201 performs control for switching from a twodimensional
(2D) display process to a stereoscopic (3D)
display process at the frame (picture) timing (indicated by
"Ta") when the frame packing arrangement data
(arrangement_type="2D") is not extracted after only the
stereoscopic program information descriptor
(stereoscopic_service_type="011") is extracted.
[0437]
In addition, the CPU 201 performs control for switching
from a stereoscopic (3D) display process to a twodimensional
(2D) display process at the frame (picture)
timing (indicated by "Tb") when the frame packing
arrangement data (arrangement_type="2D") is extracted after
the stereoscopic program information descriptor
188
(stereoscopic_service_type="001") and the MPEG2 stereoscopic
video descriptor (arrangement_type="2D") are extracted.
[0438]
(Configuration Example 5)
Fig. 89 shows an example in which a base video stream
and an additional video stream are present in a 3D period,
only a base video stream is present in a 2D period, and
signaling is performed using a video ES loop (video ES_loop).
In addition, in Fig. 89, description of a part corresponding
to Fig. 82 will be appropriately omitted.
[0439]
In a case of this example, the transmission data
generation unit 110B shown in Fig. 54 inserts frame packing
arrangement data (arrangement_type="2D") indicating a 2D
mode into the user data region of the base video stream with
the picture unit in the 2D period. Thereby, the receiver
can determine a 2D period or a 3D period with frame accuracy
in the video layer.
[0440]
In addition, in a case of this example, a stereoscopic
program information descriptor
(Stereoscopic_program_info_descriptor) is disposed in the
program loop of the PMT. A value of
"stereoscopic_service_type" of the descriptor is "011" in
both a 3D period and a 2D period. Further, in a case of
189
this example, the MPEG2 stereoscopic video descriptor
(MPEG2_stereoscopic_video_format descriptor) is disposed in
the video ES loop in the 2D period. In the descriptor,
"arrangement_type" indicates "2D".
[0441]
A description will be made of operation mode switching
control between a stereoscopic (3D) display process and a
two-dimensional (2D) display process in the receiver 200B
shown in Fig. 55 In a case where signaling is performed
using the video layer and the system layer as shown in Fig.
89. This switching is performed by the CPU 201.
[0442]
When a two-dimensional (2D) image is received, the
stereoscopic program information descriptor
(stereoscopic_service_type="001") and the MPEG2 stereoscopic
video descriptor (arrangement_type="2D") are extracted by
the demultiplexer 215 and are supplied to the CPU 201. In
addition, when the two-dimensional (2D) image is received,
the frame packing arrangement data (arrangement_type="2D")
is extracted by the video decoder 216-1 and is supplied to
the CPU 201. On the other hand, when a stereoscopic (3D)
image is received, only the stereoscopic program information
descriptor (stereoscopic_service_type="011") is extracted by
the demultiplexer 215 and is supplied to the CPU 201.
[0443]
190
The CPU 201 performs control for switching from a twodimensional
(2D) display process to a stereoscopic (3D)
display process at the frame (picture) timing (indicated by
"Ta") when the frame packing arrangement data
(arrangement_type="2D") is not extracted after only the
stereoscopic program information descriptor
(stereoscopic_service_type="011") is extracted.
[0444]
In addition, the CPU 201 performs control for switching
from a stereoscopic (3D) display process to a twodimensional
(2D) display process at the frame (picture)
timing (indicated by "Tb") when the frame packing
arrangement data (arrangement_type="2D") is extracted after
the stereoscopic program information descriptor
(stereoscopic_service_type="011") and the MPEG2 stereoscopic
video descriptor (arrangement_type="2D") are extracted.
[0445]
(Configuration Example 6)
Fig. 90 shows an example in which a base video stream
and an additional video stream are present in a 3D period,
only a base video stream is present in a 2D period, and
signaling is performed using a program loop (Program_loop)
of the PMT. In addition, in Fig. 90, description of a part
corresponding to Fig. 82 will be appropriately omitted.
[0446]
191
In a case of this example, the transmission data
generation unit 110B shown in Fig. 54 inserts frame packing
arrangement data (arrangement_type="2D") indicating a 2D
mode into the user data region of the base video stream with
the picture unit in the 2D period. Thereby, the receiver
can determine a 2D period or a 3D period with frame accuracy
in the video layer.
[0447]
In addition, in a case of this example, a stereoscopic
program information descriptor
(Stereoscopic_program_info_descriptor) is disposed in the
program loop of the PMT. A value of the descriptor is "011"
in a 3D period and is "001" in a 2D period.
[0448]
In a case where signaling is performed using the video
layer and the system layer as shown in Fig. 90, a
description will be made of operation mode switching control
between a stereoscopic (3D) display process and a twodimensional
(2D) display process in the receiver 200B shown
in Fig. 55. This switching is performed by the CPU 201.
[0449]
When a two-dimensional (2D) image is received, the
stereoscopic program information descriptor
(stereoscopic_service_type="001") is extracted by the
demultiplexer 215 and is supplied to the CPU 201. In
192
addition, when the two-dimensional (2D) image is received,
the frame packing arrangement data (arrangement_type="2D")
is extracted by the video encoder 216-1 and is supplied to
the CPU 201. On the other hand, when a stereoscopic (3D)
image is received, the stereoscopic program information
descriptor (stereoscopic_service_type="011") is extracted by
the demultiplexer 215 and is supplied to the CPU 201.
[0450]
The CPU 201 performs control for switching from a twodimensional
(2D) display process to a stereoscopic (3D)
display process at the frame (picture) timing (indicated by
"Ta") when the frame packing arrangement data
(arrangement_type="2D") is not extracted after the
stereoscopic program information descriptor
(stereoscopic_service_type="011") is extracted.
[0451]
In addition, the CPU 201 performs control for switching
from a stereoscopic (3D) display process to a twodimensional
(2D) display process at the frame (picture)
timing (indicated by "Tb") when the frame packing
arrangement data (arrangement_type="2D") is extracted after
the stereoscopic program information descriptor
(stereoscopic_service_type="001") is extracted.
[0452]
(Other configuration examples)
193
In the above-described Configuration Examples 1 to 6, a
description has been made of an example in which auxiliary
information (for example, frame packing arrangement data)
indicating a 2D mode is inserted into each picture of a
video stream in a 2D period. Detailed description is
omitted, and, in a case where auxiliary information for
identifying a mode is inserted into each picture of video
streams in a 2D period and a 3D period, and in a case where
auxiliary information indicating a 3D mode is inserted into
each picture of a video stream in a 3D period, the same
configuration may be employed.
[0453]
<2. Modification example>
[SVC stream]
In addition, in the above-described embodiment, a
description has been made of an example in which the present
technology is applied to an MVC stream. In other words, the
example is an example in which a first transmission mode is
the stereoscopic image transmission mode for transmitting
base view image data and non-base view image data used along
with the base view image data in order to display a
stereoscopic image, and a second transmission mode is the
two-dimensional image transmission mode for transmitting
two-dimensional image data.
[0454]
194
However, the present technology may be applied to an
SVC stream in the same manner. The SVC stream includes a
video elementary stream of image data of the lowest layer
forming scalable coded image data. In addition, the SVC
stream includes a predetermined number of video elementary
streams of image data of the higher layers other than the
lowest layer forming the scalable coded image data.
[0455]
In a case of the SVC stream, a first transmission mode
is an extension image transmission mode for transmitting
image data of the lowest layer forming scalable coded image
data and image data of layers other than the lowest layer,
and a second transmission mode is a base image transmission
mode for transmitting base image data. Also in a case of
the SVC stream, a reception side can rapidly identify a mode
in the same manner as in the above-described MVC stream.
[0456]
In a case of the SVC stream, a stream configuration
example 1 is considered in which a base video stream and an
additional video stream are transmitted in the extension
image transmission mode and a single video stream (only a
base video stream) is transmitted in the base image
transmission mode (refer to Fig. 80). In this case, it is
possible to identify a mode in the same manner as in a case
of the above-described MVC stream.
195
[0457]
In addition, in a case of the SVC stream, a stream
configuration example 2 is considered in which a base video
stream and an additional video stream are transmitted in
both the extension image transmission mode and the base
image transmission mode (refer to Fig. 81). However, in the
base image transmission mode, the additional video stream is
coded in a coding mode (Skipped Macro Block) in which a
difference between views is zero as a result of referring to
the base video stream. Also in this case, it is possible to
identify a mode in the same manner as in a case of the
above-described MVC stream.
[0458]
Fig. 91 shows an example of a reception packet process
when an extension image is received. NAL packets of a base
video stream and an additional video stream are mixed and
are transmitted. Fig. 92 shows a configuration example
(Syntax) of a NAL unit header and SVC extension of the NAL
unit header (NAL unit header SVC extension). The field of
"dependency_id" indicates what number layer is a
corresponding layer. As shown in Fig. 91, a receiver
assigns the NAL packets which are mixed and are transmitted
to each stream and decodes each stream on the basis of a
combination of a value of the NAL unit type and a dependency
ID (dependency_id) of NAL unit header SVC extension (Header
196
svc extension).
[0459]
Fig. 93 shows an example of a reception packet process
in the base image transmission mode. NAL packets of a base
video stream and an additional video stream are mixed and
are transmitted. As shown in Fig. 93, the receiver assigns
the NAL packets which are mixed and are transmitted to each
stream and decodes only the base video stream on the basis
of a combination of a value of the NAL unit type and a
dependency ID (dependency_id) of NAL unit header SVC
extension (Header svc extension).
[0460]
In other words, also in the base image transmission
mode, in the same manner as in the extension image
transmission mode, the receiver receives a base video stream
and an additional video stream but performs a base image
reception process without performing an extension image
reception process, on the basis of information of an ID
value of the same type as "view_position[i]" of the multiview
view position SEI message, that is, set content in
which dependencies of a plurality of streams have the same
value.
[0461]
As above, since identification can be performed at a
packet (NAL packet) level without decoding coded data of an
197
additional video stream, it is possible to perform rapid
transfer from an extension image transmission mode to a base
image transmission mode in the receiver. In addition, since
layers equal to or lower than the slice layer are not
decoded and can be discarded, memory consumption can be
suppressed to that extent so as to save power or allocate a
CPU budget of a system, a memory space bandwidth, or the
like to other features (for example, high performance
graphics), thereby achieving multiple functions.
[0462]
[Others]
In addition, although the image transmission and
reception system 10 including the broadcast station 100 and
the receiver 200 has been described in the above-described
embodiment, a configuration of an image transmission and
reception system to which the present technology is
applicable is not limited thereto. For example, the
receiver 200 part may be configured to include a set-top box
and a monitor which are connected via a digital interface
such as, for example, High-Definition Multimedia Interface
(HDMI).
[0463]
In addition, in the above-described embodiment, a
description has been made of an example in which a container
is a transport stream (MPEG-2 TS). However, the present
198
technology is similarly applicable to a system with a
configuration in which image data delivery to a reception
terminal is performed using a network such as the Internet.
In the Internet delivery, the delivery is frequently
performed using MP4 or containers of other formats. In
other words, the containers correspond to containers of
various formats such as a transport stream (MPEG-2 TS)
employed in the digital broadcast standards and MP4 used in
the Internet delivery.
[0464]
In addition, the present technology may have the
following configuration.
(1) An image data transmission device including a
transmission unit that transmits one or a plurality of video
streams including a predetermined number of image data
items; and an information inserting unit that inserts
auxiliary information for identifying a first transmission
mode in which a plurality of image data items are
transmitted and a second transmission mode in which a single
image data item is transmitted, into the video stream.
(2) The image data transmission device set forth in (1),
wherein the information inserting unit inserts auxiliary
information indicating the first transmission mode into the
video stream in the first transmission mode and inserts
auxiliary information indicating the second transmission
199
mode into the video stream in the second transmission mode.
(3) The image data transmission device set forth in (1),
wherein the information inserting unit inserts auxiliary
information indicating the first transmission mode into the
video stream in the first transmission mode and does not
insert the auxiliary information into the video stream in
the second transmission mode.
(4) The image data transmission device set forth in (1),
wherein the information inserting unit does not insert the
auxiliary information into the video stream in the first
transmission mode and inserts auxiliary information
indicating the second transmission mode into the video
stream in the second transmission mode.
(5) The image data transmission device set forth in any
one of (1) to (4), wherein the information inserting unit
inserts the auxiliary information into the video stream, at
least with the program unit, the scene unit, the picture
group unit, or the picture unit.
(6) The image data transmission device set forth in any
one of (1) to (5), wherein the transmission unit transmits a
base video stream including first image data and a
predetermined number of additional video streams including
second image data used along with the first image data in
the first transmission mode, and transmits a single video
stream including the first image data in the second
200
transmission mode.
(7) The image data transmission device set forth in any
one of (1) to (5), wherein the transmission unit transmits a
base video stream including first image data and a
predetermined number of additional video streams including
second image data used along with the first image data in
the first transmission mode, and transmits a base video
stream including first image data and a predetermined number
of additional video streams substantially including image
data which is the same as the first image data in the second
transmission mode.
(8) The image data transmission device set forth in any
one of (1) to (7), wherein the first transmission mode is a
stereoscopic image transmission mode in which base view
image data and non-base view image data used along with the
base view image data are transmitted so as to display a
stereoscopic image, and the second transmission mode is a
two-dimensional image transmission mode in which twodimensional
image data is transmitted.
(9) The image data transmission device set forth in (8),
wherein the auxiliary information indicating the
stereoscopic image transmission mode includes information
indicating a relative positional relationship of each view.
(10) The image data transmission device set forth in
any one of (1) to (7), wherein the first transmission mode
201
is an extension image transmission mode in which image data
of the lowest layer forming scalable coded image data and
image data of layers other than the lowest layer are
transmitted, and the second transmission mode is a base
image transmission mode in which base image data is
transmitted.
(11) The image data transmission device set forth in
any one of (1) to (10), wherein the transmission unit
transmits a container of a predetermined format including
the video stream, and wherein the image data transmission
device further includes identification information inserting
unit that inserts identification information for identifying
whether to be in the first transmission mode or in the
second transmission mode, into a layer of the container.
(12) An image data transmission method including a
transmission step of transmitting one or a plurality of
video streams including a predetermined number of image data
items; and an information inserting step of inserting
auxiliary information for identifying a first transmission
mode in which a plurality of image data items are
transmitted and a second transmission mode in which a single
image data item is transmitted, into the video stream.
(13) An image data reception device including a
reception unit that receives one or a plurality of video
streams including a predetermined number of image data
202
items; a transmission mode identifying unit that identifies
a first transmission mode in which a plurality of image data
items are transmitted and a second transmission mode in
which a single image data item is transmitted on the basis
of auxiliary information which is inserted into the received
video stream; and a processing unit that performs a process
corresponding to each mode on the received video stream on
the basis of the mode identification result, so as to
acquire the predetermined number of image data items.
(14) The image data reception device set forth in (13),
wherein the transmission mode identifying unit identifies
the first transmission mode when auxiliary information
indicating the first transmission mode is inserted into the
received video stream, and identifies the second
transmission mode when auxiliary information indicating the
second transmission mode is inserted into the received video
stream.
(15) The image data reception device set forth in (13),
wherein the transmission mode identifying unit identifies
the first transmission mode when auxiliary information
indicating the first transmission mode is inserted into the
received video stream, and identifies the second
transmission mode when the auxiliary information is not
inserted into the received video stream.
(16) The image data reception device set forth in (13),
203
wherein the transmission mode identifying unit identifies
the first transmission mode when the auxiliary information
is not inserted into the received video stream, and
identifies the second transmission mode when auxiliary
information indicating the second transmission mode is
inserted into the received video stream.
(17) The image data reception device set forth in any
one of (13) to (16), wherein the reception unit receives a
base video stream including first image data and a
predetermined number of additional video streams including
second image data used along with the first image data in
the first transmission mode, and receives a single video
stream including the first image data in the second
transmission mode, and wherein the processing unit processes
the base video stream and the predetermined number of
additional video streams so as to acquire the first image
data and the second image data in the first transmission
mode, and processes the single video stream so as to acquire
the first image data in the second transmission mode.
(18) The image data reception device set forth in any
one of (13) to (16), wherein the reception unit receives a
base video stream including first image data and a
predetermined number of additional video streams including
second image data used along with the first image data in
the first transmission mode, and receives a base video
204
stream including first image data and a predetermined number
of additional video streams substantially including image
data which is the same as the first image data in the second
transmission mode, and wherein the processing unit processes
the base video stream and the predetermined number of
additional video streams so as to acquire the first image
data and the second image data in the first transmission
mode, and processes the base video stream so as to acquire
the first image data without performing a process of
acquiring the second image data from the predetermined
number of additional video streams in the second
transmission mode.
(19) The image data reception device set forth in any
one of (13) to (18), wherein the reception unit receives a
container of a predetermined format including the video
stream, wherein identification information for identifying
whether to be in the first transmission mode or in the
second transmission mode is inserted into a layer of the
container in the container, and wherein the transmission
mode identifying unit identifies the first transmission mode
in which a plurality of image data items are transmitted and
the second transmission mode in which a single image data
item is transmitted on the basis of auxiliary information
which is inserted into the received video stream and
identification information which is inserted into the layer
205
of the container.
(20) The image data reception device set forth in any
one of (13) to (19), wherein the first transmission mode is
a stereoscopic image transmission mode in which base view
image data and non-base view image data used along with the
base view image data are transmitted so as to display a
stereoscopic image, and the second transmission mode is a
two-dimensional image transmission mode in which twodimensional
image data is transmitted.
[0465]
A main feature of the present technology is that a
reception side can identify a 3D period or a 2D period with
frame accuracy on the basis of auxiliary information (a SEI
message, user data, or the like) which is inserted into a
transmission video stream in the 3D period and the 2D period,
only in the 3D period, or only in the 2D period, and thus it
is possible to appropriately and accurately handle a dynamic
variation in delivery content and to thereby receive a
correct stream (refer to Figs. 59 and 79).
Reference Signs List
[0466]
10 IMAGE TRANSMISSION AND RECEPTION SYSTEM
100 BROADCAST STATION
110 TRANSMISSION DATA GENERATION UNIT
111-1 TO 111-N IMAGE DATA OUTPUT PORTION
206
112 VIEW SELECTOR
113-1, 113-2, AND 113-3 SCALER
114-1, 114-2, AND 114-3 VIDEO ENCODER
115 MULTIPLEXER
116 DISPARITY DATA GENERATION PORTION
117 DISPARITY ENCODER
118 GRAPHICS DATA OUTPUT PORTION
119 GRAPHICS ENCODER
120 AUDIO DATA OUTPUT PORTION
121 AUDIO ENCODER
200 AND 200A RECEIVER
201 CPU
211 ANTENNA TERMINAL
212 DIGITAL TUNER
213 TRANSPORT STREAM BUFFER (TS BUFFER)
214 DEMULTIPLEXER
215-1, 215-2, 215-3, 221, 225, AND 230 CODED BUFFER
216-1, 216-2, AND 216-3 VIDEO DECODER
217-1, 217-2, AND 217-3 VIEW BUFFER
218-1, 218-2, 218-3 AND 228 SCALER
219 VIEW INTERPOLATION UNIT
220 PIXEL INTERLEAVING/SUPERIMPOSING UNIT
222 DISPARITY DECODER
223 DISPARITY BUFFER
224 DISPARITY DATA CONVERSION UNIT
207
226 GRAPHICS DECODER
227 PIXEL BUFFER
229 GRAPHICS SHIFTER
231 AUDIO DECODER
232 CHANNEL MIXING UNIT
233 DISPARITY DATA GENERATION UNIT
208
CLAIMS
[Claim 1]
An image data transmission device comprising:
a transmission unit that transmits one or a plurality
of video streams including a predetermined number of image
data items; and
an information inserting unit that inserts auxiliary
information for identifying a first transmission mode in
which a plurality of image data items are transmitted and a
second transmission mode in which a single image data item
is transmitted, into the video stream.
[Claim 2]
The image data transmission device according to claim 1,
wherein the information inserting unit inserts
auxiliary information indicating the first transmission mode
into the video stream in the first transmission mode and
inserts auxiliary information indicating the second
transmission mode into the video stream in the second
transmission mode.
[Claim 3]
The image data transmission device according to claim 1,
wherein the information inserting unit inserts
auxiliary information indicating the first transmission mode
into the video stream in the first transmission mode and
209
does not insert the auxiliary information into the video
stream in the second transmission mode.
[Claim 4]
The image data transmission device according to claim 1,
wherein the information inserting unit does not insert
the auxiliary information into the video stream in the first
transmission mode and inserts auxiliary information
indicating the second transmission mode into the video
stream in the second transmission mode.
[Claim 5]
The image data transmission device according to claim 1,
wherein the information inserting unit inserts the
auxiliary information into the video stream, at least with
the program unit, the scene unit, the picture group unit, or
the picture unit.
[Claim 6]
The image data transmission device according to claim 1,
wherein the transmission unit transmits a base video
stream including first image data and a predetermined number
of additional video streams including second image data used
along with the first image data in the first transmission
mode, and transmits a single video stream including the
first image data in the second transmission mode.
[Claim 7]
The image data transmission device according to claim 1,
210
wherein the transmission unit transmits a base video
stream including first image data and a predetermined number
of additional video streams including second image data used
along with the first image data in the first transmission
mode, and transmits a base video stream including first
image data and a predetermined number of additional video
streams substantially including image data which is the same
as the first image data in the second transmission mode.
[Claim 8]
The image data transmission device according to claim 1,
wherein the first transmission mode is a stereoscopic
image transmission mode in which base view image data and
non-base view image data used along with the base view image
data are transmitted so as to display a stereoscopic image,
and the second transmission mode is a two-dimensional image
transmission mode in which two-dimensional image data is
transmitted.
[Claim 9]
The image data transmission device according to claim 8,
wherein the auxiliary information indicating the
stereoscopic image transmission mode includes information
indicating a relative positional relationship of each view.
[Claim 10]
The image data transmission device according to claim 1,
wherein the first transmission mode is an extension
211
image transmission mode in which image data of the lowest
layer forming scalable coded image data and image data of
layers other than the lowest layer are transmitted, and the
second transmission mode is a base image transmission mode
in which base image data is transmitted.
[Claim 11]
The image data transmission device according to claim 1,
wherein the transmission unit transmits a container of
a predetermined format including the video stream, and
wherein the image data transmission device further
includes identification information inserting unit that
inserts identification information for identifying whether
to be in the first transmission mode or in the second
transmission mode, into a layer of the container.
[Claim 12]
An image data transmission method comprising:
a transmission step of transmitting one or a plurality
of video streams including a predetermined number of image
data items; and
an information inserting step of inserting auxiliary
information for identifying a first transmission mode in
which a plurality of image data items are transmitted and a
second transmission mode in which a single image data item
is transmitted, into the video stream.
[Claim 13]
212
An image data reception device comprising:
a reception unit that receives one or a plurality of
video streams including a predetermined number of image data
items;
a transmission mode identifying unit that identifies a
first transmission mode in which a plurality of image data
items are transmitted and a second transmission mode in
which a single image data item is transmitted on the basis
of auxiliary information which is inserted into the received
video stream; and
a processing unit that performs a process corresponding
to each mode on the received video stream on the basis of
the mode identification result, so as to acquire the
predetermined number of image data items.
[Claim 14]
The image data reception device according to claim 13,
wherein the transmission mode identifying unit
identifies the first transmission mode when auxiliary
information indicating the first transmission mode is
inserted into the received video stream, and identifies the
second transmission mode when auxiliary information
indicating the second transmission mode is inserted into the
received video stream.
[Claim 15]
The image data reception device according to claim 13,
213
wherein the transmission mode identifying unit
identifies the first transmission mode when auxiliary
information indicating the first transmission mode is
inserted into the received video stream, and identifies the
second transmission mode when the auxiliary information is
not inserted into the received video stream.
[Claim 16]
The image data reception device according to claim 13,
wherein the transmission mode identifying unit
identifies the first transmission mode when the auxiliary
information is not inserted into the received video stream,
and identifies the second transmission mode when auxiliary
information indicating the second transmission mode is
inserted into the received video stream.
[Claim 17]
The image data reception device according to claim 13,
wherein the reception unit receives a base video stream
including first image data and a predetermined number of
additional video streams including second image data used
along with the first image data in the first transmission
mode, and receives a single video stream including the first
image data in the second transmission mode, and
wherein the processing unit processes the base video
stream and the predetermined number of additional video
streams so as to acquire the first image data and the second
214
image data in the first transmission mode, and processes the
single video stream so as to acquire the first image data in
the second transmission mode.
[Claim 18]
The image data reception device according to claim 13,
wherein the reception unit receives a base video stream
including first image data and a predetermined number of
additional video streams including second image data used
along with the first image data in the first transmission
mode, and receives a base video stream including first image
data and a predetermined number of additional video streams
substantially including image data which is the same as the
first image data in the second transmission mode, and
wherein the processing unit processes the base video
stream and the predetermined number of additional video
streams so as to acquire the first image data and the second
image data in the first transmission mode, and processes the
base video stream so as to acquire the first image data
without performing a process of acquiring the second image
data from the predetermined number of additional video
streams in the second transmission mode.
[Claim 19]
The image data reception device according to claim 13,
wherein the reception unit receives a container of a
predetermined format including the video stream,
215
wherein identification information for identifying
whether to be in the first transmission mode or in the
second transmission mode is inserted into a layer of the
container in the container, and
wherein the transmission mode identifying unit
identifies the first transmission mode in which a plurality
of image data items are transmitted and the second
transmission mode in which a single image data item is
transmitted on the basis of auxiliary information which is
inserted into the received video stream and identification
information which is inserted into the layer of the
container.
[Claim 20]
The image data reception device according to claim 13,
wherein the first transmission mode is a stereoscopic
image transmission mode in which base view image data and
non-base view image data used along with the base view image
data are transmitted so as to display a stereoscopic image,
and the second transmission mode is a two-dimensional image
transmission mode in which two-dimensional image data is
transmitted.