DESCRIPTION
CODING APPARATUS, CODING METHOD, TRANSMISSION APPARATUS, AND RECEPTION APPARATUS
Technical Field 5
[0001] The present technology relates to a coding apparatus, a coding method, a transmission apparatus, and a reception apparatus. Specifically, the present technology relates to a coding apparatus and the like that hierarchically code image data of each picture 10 constituting moving image data.
Background Art
[0002] If service of providing compressed moving images is provided by broadcasting or through the Internet, for example, the upper limit of frame 15 frequencies that can be reproduced is limited depending on the decoding capability of a receiver. Therefore, the service provider needs to take into account the reproduction capability of a widely used receiver to limit the service to a service with a low frame 20 frequency, or to simultaneously provide services with a plurality of frame frequencies, i.e., a high frame frequency and a low frame frequency.
[0003] The receiver costs a lot to support the service with a high frame frequency, which is a 25 inhibiting factor for early widespread use of the
3
receiver. If inexpensive receivers dedicated to the
service with a low frame frequency only are widely used
at the initial stage and the service provider starts
the service with a high frame frequency in the future,
5 viewing is not possible in the absence of a new
receiver, which is an inhibiting factor for widespread
use of the new service.
[0004] For example, the time direction scalability
in which image data of each picture constituting moving
10 image data is hierarchically coded in h.265/HEVC (High
Efficiency Video Coding) has been proposed (see, nonpatent
document 1). On the receiver side, it is
possible to identify the hierarchy of each picture
based on a temporal ID (temporal_id) inserted in the
15 header of a NAL (Network Abstraction Layer) unit, and
thus to perform selective decoding up to the hierarchy
corresponding to the decoding capability.
[0005] Non-Patent Document 1: Gary J. Sullivan,
Jens-Rainer Ohm, Woo-Jin Han, Thomas Wiegand, "Overview
20 of the High Efficiency Video Coding (HEVC) Standard"
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO
TECNOROGY, VOL. 22, NO. 12, pp. 1649-1668, DECEMBER
2012
Summary of Invention
25 Problem to be solved by the Invention
[0006] It is an object of the present technology to
4
perform favorable decoding processing on the receiver
side.
Means for solving the Problem
[0007] The concept of the present technology is a
5 transmission apparatus including an image coding unit
configured to classify image data of each picture
constituting moving image data into a plurality of
hierarchies, to code the image data of the picture in
each of the classified hierarchies, to divide the
10 plurality of hierarchies into a predetermined number of
hierarchy sets, and to generate a predetermined number
of video streams having the coded image data of the
picture in each of the divided hierarchy sets, and a
transmission unit configured to transmit a container
15 including the generated predetermined number of video
streams in a predetermined format, the image coding
unit performing coding so that a decoding interval of
the coded image data of the picture in at least the
lowest hierarchy set is a regural interval.
20 [0008] In the present technology, the image coding
unit codes the image data of each picture constituting
moving image data, and thus, the predetermined number
of video streams are generated. In this case, the image
data of each picture constituting moving image data is
25 classified into a plurality of hierarchies and coded.
Then, the plurality of hierarchies are divided into a
5
predetermined number of hierarchy sets, and a
predetermined number of video streams, each of which
has the coded image data of the picture in the divided
hierarchy set, are generated.
5 [0009] In the image coding unit, coding is performed
so that a decoding interval of at least the coded iamge
data of the picture in the lowest hierarchy set is a
regular interval. For example, the image coding unit
may perform coding so that a decoding timing for the
10 coded image data of the picture in a hierarchy set
higher than the lowest hierarchy set is an average
timing of decoding timings for the coded image data of
the picture in all of the hierarchy sets lower than the
hierarchy set. Accordingly, for example, it allows the
15 receiver to sequentially and smoothly perform decoding
processing on each picture in the case where the
receiver has the decoding capability to decode the
coded image data of the picture not only in the lowest
hierarchy set but also in a hierarchy set higher than
20 that.
[0010] Moreover, for example, the image coding unit
may divide the plurality of hierarchies into the
predetermined number of hierarchy sets so that the
lowest hierarchy set includes a plurality of
25 hierarchies and a hierarchy set higher than the lowerst
hierarchy set includes one hierarchy. Accordingly, for
6
example, in the case where the receiver has the
decoding capability to perform processing on the coded
image data of the pictures in the plurality of
hierarchies of the lowerst hierarchy set, it only needs
5 the configuration that selects only a video stream
having the coded image data of the picture in the
lowest hierarchy set, takes it in a buffer, and
performs decoding processing on it, and does not need a
complex configuration that performs combination
10 processing on a plurality of video streams, for example.
[0011] By the transmission unit, a container
including the predetermined number of video streams in
a predetermined format is transmitted. For example, the
container may be a transport stream (MPEG-2 TS) adopted
15 by the digital broadcasting standards. Moreover, for
example, the container may be a container in a format
such as MP4 used for delivery on the Internet.
[0012] As described above, in the present technology,
coding is performed so that a decoding interval of at
20 least the coded image data of the picture in the lowest
hierarchy set is a regular interval. Therefore, in the
case where the receiver has the decoding capability to
perform processing on the coded image data of the
pictures in the plurality of hierarchies of the lowest
25 hierarchy set, for example, it is possible to
continuously and effortlessly perform decoding
7
processing on the coded image data of each picture.
[0013] In addition, another concept of the present
technology is a transmission apparatus including an
image coding unit configured to classify image data of
5 each picture constituting moving image data into a
plurality of hierarchies, to code the image data of the
picture in each of the classified hierarchies, to
divide the plurality of hierarchies into a
predetermined number of hierarchy sets, and to generate
10 a predetermined number of video streams having the
coded image data of the picture in each of the divided
hierarchy sets, a transmission unit configured to
transmit a contenner including the generated
predetermined number of video streams in a
15 predetermined format, and an identification information
insertion unit configured to insert, in a layer of the
contenna, identification information for identifying
whether each of the predetermined number of video
streams is a base stream having the coded image data of
20 the picture in the lowerst hierarchy set or an enhanced
stream having the coded image data of the picture in a
hierarchy set higher than the lowerst hierarchy set.
[0014] In the present technology, the image coding
unit codes the image data of each picture constituting
25 moving image data to generate a predetermined number of
video streams. In this case, the image data of each
8
picture constituting moving image data is classified
into a plurality of hierarchies and coded. Then, the
plurality of hierarchies are divided into a
predetermined number of hierarchy sets, and a
5 predetermined number of video streams, each of which
has the coded image data of the picture in the divided
hierarchy set, are generated.
[0015] For example, the image coding unit may
perform coding so that a decoding interval of at least
10 the coded image data of the picture in the lowest
hierarchy set is a regular interval. In this case, for
example, the image coding unit may perfom coding so
that a decoding timing for the coded image data of the
picture in a hierarchy set higher than the lowest
15 hierarchy set is an average timing of decoding timings
for the coded image data of the pictures in all
hierarchy sets lower than the hierarchy set.
[0016] By the transmission unit, a container
including the predetermined number of video streams in
20 a predetermined format is transmitted. For example, the
container may be a transport stream (MPEG-2 TS) adopted
by the digital broadcasting standards. Moreover, for
example, the container may be a container in a format
such as MP4 used for delivery on the Internet.
25 [0017] By the identification information insertion
unit, identification information is inserted in a layer
9
of the container. This identification information is
identification information for identifying whether each
of the predetermined number of video streams is a base
stream having the coded image data of the picture in
5 the lowest hierarchy set or an enhanced stream having
the coded image data of the picture in a hierarchy set
higher than the lowest hierarchy set.
[0018] For example, there are a plurality of
enhanced streams, the identification information may be
10 capable of identifying each of the enhanced streams.
Moreover, for example, the container may be a transport
stream and the identification information insertion
unit may insert, as a stream type, the identification
information in a video elementary stream loop arranged
15 corresponding to the predetermined number of video
streams under a program map table.
[0019] As described above, in the present technology,
the indentification information for identitying whether
each of the predetermined numbe of video streams is the
20 base stream or the enhanced stream is inserted in the
layer of the container. Therefore, on the receiver side,
it is possible to select the base stream only and to
easily and selectively decode the coded image data of
the picture in a lower hierarchy set, for example, by
25 using the identification information.
[0020] In addition, another concept of the present
10
technology is a transmission apparatus including an
image coding unit configured to classify image data of
each picture constituting moving image data into a
plurality of hierarchies, to code the image data of the
5 picture in each of the classified hierarchies, to
divide the plurality of hierarchies into a
predetermined number of hierarchy sets, and to generate
a predetermined number of video streams having the
coded image data of the picture in each of the divided
10 hierarchy sets, a transmission unit configured to
transmit a contenner including the generated
predetermined number of video streams in a
predetermined format, and a configuration information
insertion unit configured to insert, in a layer of the
15 container, configuration information of the video
stream corresponding to each of the predetermined numbe
of video streams included in the container.
[0021] In the present technology, by the image
coding unit, the image data of each pictuture
20 constituting moving image data is coded, and a
predetermined number of video streams are generated. In
this case, the coded image data of each picture
constituting moving image data is classified into a
plurality of hierarchies and coded. Then, the plurality
25 of hierarchies are divided into a predetermined number
of hierarchy sets, and a predetermined number of video
11
streams, each of which has the coded image data of the
picture in the divided hierarchy set, are generated.
Then, by the transmission unit, a container including
the predetermined number of video streams in a
5 predetermined format is transmitted.
[0022] By the configuration information insertion
unit, configuration information of the video stream is
inserted in a layer of the container corresponding to
each of the predetermined number of video streams
10 included in the container. For example, the container
may be a transport stream, and the configuration
information insertion unit may insert, as a descriptor,
the configuration information in a video elementary
stream loop arranged corresponding to each of the
15 predetermined number of video streams under a program
map table.
[0023] For example, the configuration information
may include information representing a service group to
which the video stream belongs. Moreover, for example,
20 the configuration information may include information
representing a dependence relationship between streams
starting from the base stream having the coded image
data of the picture in the lowest hierarchy set.
Moreover, for example, the configuration information
25 may include information representing the number of
hierarchies of the plurality of hierarchies classified
12
by the image coding unit.
[0024] As described above, in the present technology,
the configuration information of the video stream is
inserted in the layer of the container corresponding to
5 each of the predetermined number of video streams
included in the container. Therefore, for example, on
the receiver side, it is possible to easily know which
group the video stream included in the container
belongs to, what stream-dependent relationship is there,
10 what is the number of hierarchies in the hierarchical
coding, and the like.
[0025] In addition, another concept of the present
technology is a reception apparatus including a
reception unit configured to receive a predetermined
15 number of video streams having coded image data of a
picture in a hierarchy set of a predetermined number of
hierarchy sets, the coded image data being obtained by
classifying image data of pictures constituting moving
image data into a plurality of hierarchies and coding
20 the image data, the plurality of hierarchies being
divided into the predetermined number of hierarchy sets,
and a processing unit configured to process the
received predetermined number of video streams, in
which at least a video stream of the predetermined
25 number of video streams, which has coded image data of
a picture in the lowest hierarchy set, is coded so that
13
decoding interval for each picture is a regular
interval.
[0026] In the present technology, by the receiver,
the image data of each picture constituting moving
5 image data is classified into a plurality of
hierarchies and coded, and a pretermined number of
video streams, each of which has the coded image data
of the picture in a hierarchy set obtained by dividing
the plurality of hierarchies into a predetermined
10 number of hierarchy sets, are received. Then, by the
processing unit, the received predetermined number of
video streams are processed.
[0027] In this case, at least the video stream of
the predetermined number of video streams, which has
15 the coded image data of the picture in the lowest
hierarchy set, is coded so that a decoding interval of
each picture is a regular interval. Therefore, for
example, with the decoding capability to process the
coded image data of the pictures in the plurality of
20 hierarchies of the lowerst hierarchy set, it is
possible to continuously and effortlessly perform
decoding processing on the coded image data of each
picture.
[0028] It should be noted that in the present
25 technology, for example, the predetermined number of
video streams may be coded so that a decoding timing
14
for the coded image data of the picture in a hierarchy
set higher than the lowest hierarchy set is an average
timing of decoding timings for the coded image data of
the pictures in all hierarchy sets lower than the
5 hierarchy set. Accordingly, for example, with the
decoding capabity to decode the coded image data of the
picture not only in the lowest hierarchy set but also
in a hierarchy set higher than that, it is possible to
smoothly and sequentially perform decoding processing
10 on each picture.
[0029] In addition, another concept of the present
technology is a reception apparatus including a
reception unit configured to receive a predetermined
number of video streams having coded image data of a
15 picture in a hierarchy set of a predetermined number of
hierarchy sets, the coded image data being obtained by
classifying image data of pictures constituting moving
image data into a plurality of hierarchies and coding
the image data, the plurality of hierarchies being
20 divided into the predetermined number of hierarchy sets,
and an image decoding unit configured to selectively
take coded image data of a picture in a hierarchy that
is equal to or lower than a predetermined hierarchy
depending on a decoding capability in a buffer from the
25 predetermined number of video streams included in the
received container, to decode the coded image data of
15
the picture taken in the buffer, and to acquire image
data of the picture in the hierarchy that is equal to
or lower than the predetermined hierarchy, in which at
least a video stream of the predetermined number of
5 video streams, which has coded image data of a picture
in the lowest hierarchy set, is coded so that decoding
interval for each picture is a regular interval.
[0030] In the present technology, by the receiver, a
container in a predetermined format is received. This
10 container includes a predetermined number of video
streams having the coded image data of picture in one
of more hierarchies, which is obtained by classifying
the image data of each picture constituting moving
image data into a plurality of hierarchies, coding it,
15 and dividing the plurality of hierarchies into a
predetermined number of hierarchy sets.
[0031] By the image decoding unit, coded image data
of a picture in a hierarchy that is equal to or lower
than a predetermined hierarchy set depending on the
20 decoding capability is selectively taken in a buffer
from the predetermined number of video streams included
in the received container, and the coded image data of
each picture taken in the buffer is decoded to acquire
image data of a picture in a hierarchy that is equal to
25 or lower than the predetermined hierarchy. For example,
in the case where the coded image data of the picture
16
in the predetermined hierarchy set is included in a
plurality of video streams, the image decoding unit may
decode the coded image data of each picture as one
stream based on decoding timing information.
5 [0032] In this case, at least the video stream of
the predetermined number of video streams, which has
the coded image data of the picture in the lowest
hierarchy set, is coded so that a decoding interval of
each picture is a regular interval. Therefore, for
10 example, with the decoding capability to process the
coded image data of the pictures in the plurality of
hierarchies of the lowerst hierarchy set, it is
possible to continuously and effortlessly perform
decoding processing on the coded image data of each
15 picture.
[0033] It should be noted that in the present
technology, for example, identification information for
identifying whether each of the predetermined number of
video streams is the base stream including the coded
20 image data of the picture in the lowest hierarchy set
or the enhanced stream including the coded image data
of the picture in a hierarchy set higher than the
lowest hierarchy set may be inserted in a layer of the
container, and the image decoding unit may take the
25 coded image data in a predetermined hierarchy set
depending on the decoding capability in a buffer from
17
the predetermined number of video streams including the
base stream based on the identification information and
decode it. In this case, it is possible to select the
base stream only and to easily and selectively decode
5 the coded image data of the picture in a lower
hierarchy set, for example, by using the identification
information.
[0034] In addition, in the present technology, for
example, a post processing unit configured to match a
10 frame rate of the image data of each picture acquired
in the image decoding unit with a display capability
may be further provided. In this case, even in the case
where the decoding capability is low, it is possible to
acquire the image data at a frame rate that matches a
15 high display capability.
Effects of the Invention
[0035] According to the present technology, it is
possible to perform favorable decoding processing on
the receiver side. Effects described herein are not
20 necessarily limited, and may be any one of the effects
described herein.
Brief Description of Drawings
[0036]
[Fig. 1] A block diagram showing a configuration
25 example of a transmission/reception system according to
an embodiment.
18
[Fig. 2] A block diagram showing a configuration
example of a transmission apparatus.
[Fig. 3] A diagram showing an example of hierarchical
coding performed in an encoder.
5 [Fig. 4] A diagram showing a structural example of an
NAL unit header and the content of main parameters in
the structural example.
[Fig. 5] A diagram for explaining the configuration of
the coded image data of each picture in HEVC.
10 [Fig. 6] A diagram showing an example of encoding,
decoding, a display order, and delay during the
hierarchical coding.
[Fig. 7] A diagram showing a coded stream in the
hierarchical coding and the expected display (display
15 order) in a designated hierarchy.
[Fig. 8] A diagram for explaining the coding timing
(decoding timing) for pictures of a predetermined
number of video streams (two streams).
[Fig. 9] A diagram showing an example of the coding
20 timing (decoding timing) for each picture in the case
where two video streams of a base stream and an
enhanced stream are generated.
[Fig. 10] A diagram showing another example of the
coding timing (decoding timing) for each picture in the
25 case where two video streams of a base stream and an
enhanced stream are generated.
19
[Fig. 11] A diagram for explaining the coding timing
(decoding timing) of the pictures of a predetermined
number of video streams (3 streams).
[Fig. 12] A diagram showing an example of the coding
5 timing (decoding timing) for each picture in the case
where three video streams of a base stream and two
enhanced streams are generated.
[Fig. 13] A diagram showing another example of the
coding timing (decoding timing) for each picture in the
10 case where three video streams of a base stream and two
enhanced streams are generated.
[Fig. 14] A diagram showing an example of HRD
(Hypothetical Reference Decoder) control of an encoder.
[Fig. 15] A block diagram showing a configuration
15 example of the encoder.
[Fig. 16] A diagram showing an example of processing
flow of the encoder.
[Fig. 17] A diagram showing a structural example of
HEVC descriptor (HEVC_descriptor).
20 [Fig. 18] A diagram showing a structural example of a
multistream descriptor (multistream_descriptor).
[Fig. 19] A diagram showing the content of main
information in the structural example of the
multistream descriptor (multistream_descriptor).
25 [Fig. 20] A diagram showing an example of
"Stream_type," "Group_id", "max/min layer",
20
"max_layer_in_group", and "Stream_dependency_ordering"
in the case where a transport stream TS includes, for
example, a video stream group of services 1 and 2.
[Fig. 21] A block diagram showing a configuration
5 example of a multiplexer.
[Fig. 22] A diagram showing an example of processing
flow of the multiplexer.
[Fig. 23] A diagram showing a configuration example of
the transport stream TS in 2 stream delivery.
10 [Fig. 24] A diagram showing a configuration example of
the transport stream TS in 3 stream delivery.
[Fig. 25] A block diagram showing a configuration
example of a reception apparatus.
[Fig. 26] A block diagram showing a configuration
15 example of a demultiplexer.
[Fig. 27] A diagram showing an example of stream
selection.
[Fig. 28] A diagram showing an example of processing
flow of the demultiplexer.
20 [Fig. 29] A block diagram showing a configuration
example of a decoder.
[Fig. 30] A diagram showing an example of stream
combination.
[Fig. 31] A diagram showing a configuration example of
25 a post processing unit.
[Fig. 32] A diagram showing an example of processing
21
flow of the post processing unit.
Modes for Performing the Invention
[0037] Hereinafter, an embodiment for performing the
present invention (hereinafter, referred to as
5 embodiment) will be described. It should be noted that
a description will be made in the following order.
1. Embodiment
2. Modified Example
[0038] <1. Embodiment>
10 [Transmission/reception system]
Fig. 1 shows a configuration example of a
transmission/reception system 10 according to an
embodiment. The transmission/reception system 10
includes a transmission apparatus 100 and a reception
15 apparatus 200.
[0039] the transmission apparatus 100 causes a
transport stream TS serving as a container to be
carried on a broadcast wave for transmission. The
transport stream TS includes a predetermined number of
20 video streams, each of which has image data of a
picture in a hierarchy set obtained by classifying
image data of each picture constituting moving image
data into a plurality of hierarchies, coding it, and
dividing the plurality of hierarchies into a
25 predetermined number of hierarchy sets. In this case,
coding such as H.264/AVC and H.265/HEVC is performed so
22
that the referred picture belongs to a self-hierarchy
and/or a hierarchy lower than the self-hierarchy.
[0040] In this embodiment, in the case where the
plurality of hierarchies are divided into a
5 predetermined number of hierarchy sets, the division is
made so that the lowest hierarchy set includes a
plurality of hierarchies and a hierarchy set higher
than the lowest hierarchy set includes one hierarchy.
Such division allows the receiver to select only a
10 video stream having the coded image data of the picture
in the lowest hierarchy set, to take it in a buffer,
and to perform decoding processing on it, if the
receiver has the decoding capability to process coded
image data of pictures of the plurality of hierarchies
15 included in the lowest hierarchy set, for example.
[0041] To the coded image data of the picture in
each hierarchy, hierarchy identification information
for identifying the belonging hierarchy is added for
each picture. In this embodiment, at the header portion
20 of a NAL unit (nal_unit) of each picture, the hierarchy
identification information ("nuh_temporal_id_plus1"
representing temporal_id) is arranged. By adding the
hierarchy identification information as described above,
the receiver can identify the hierarchy of each picture
25 in the layer of the NAL unit, and selectively take out
the coded image data in a hierarchy that equals to or
23
lower than a predetermined hierarchy to perform
decoding processing on it.
[0042] In this embodiment, at least the video stream
having the coded image data of the picture of the
5 lowest hierarchy set out of the predetermined number of
video streams is coded so that the decoding interval
between the pictures is a regular interval. This coding
allows the receiver to continuously and effortlessly
perform decoding processing on the coded image data of
10 each picture if the receiver has the decoding
capability to process coded image data of pictures of
the plurality of hierarchies included in the lowest
hierarchy set.
[0043] In this embodiment, coding is performs so
15 that the decoding timing of the coded image data of the
picture in a hierarchy set higher than the lowest
hierarchy set is set to an average timing of coding
timings of the pictures of all the hierarchy sets lower
than this hierarchy set. This coding allows the
20 receiver to successively and smoothly perform decoding
processing on each picture if the receiver has
capabilities to decode the coded image data of the
picture not only in the lowest hierarchy set but also
in a hierarchy set higher than the lowest hierarchy set.
25 [0044] In this embodiment, in the layer of the
transport stream TS, identification information for
24
identifying whether each of the predetermined number of
video streams is a base stream having the coded image
data of the picture in the lowest hierarchy set or an
enhanced stream having the coded image data of the
5 picture in a hierarchy set higher than the lowest
hierarchy set is inserted. This identification
information is inserted as a stream type in a video
elementary stream loop arranged corresponding to each
of the predetermined number of video streams under a
10 program map table. The identification information
allows the receiver to easily select only the base
stream to selectively decode the coded image data of
the picture in a lower hierarchy set.
[0045] In this embodiment, in the layer of the
15 transport stream TS, configuration information of the
video stream corresponding to each of the predetermined
number of video streams included therein is inserted.
This configuration information is inserted as a
descriptor in the video elementary stream loop arranged
20 corresponding to the predetermined number of video
streams under the program map table. This configuration
information allows the receiver to easily know which
group the video stream included in the container
belongs to, what stream-dependent relationship is there,
25 what is the number of hierarchies in the hierarchical
coding, and the like.
25
[0046] The reception apparatus 200 receives the
above-mentioned transport stream TS that is carried on
a broadcast wave and is transmitted from the
transmission apparatus 100. The reception apparatus 200
5 selectively takes the coded image data of the picture
in a hierarchy that is equal to or lower than the
predetermined hierarchy, which is selected depending on
the decoding capability from the predetermined number
of video streams included in the transport stream TS,
10 in a buffer, decodes it, acquires the image data of
each picture, and performs image reproduction.
[0047] As described above, the layer of the
transport stream TS includes the identification
information for identifying whether the predetermined
15 number of video streams is the base stream or the
enhanced stream. Based on the identification
information, the coded image data in the predetermined
hierarchy set depending on the decoding capability is
taken, from the predetermined number of video streams
20 including the base stream, in the buffer for processing.
[0048] In addition, the reception apparatus 200
performs post processing in which the frame rate of the
image data of each picture, which is obtained by the
above-mentioned decoding, is matched with the display
25 capability. According to this post processing, it is
possible to acquire the image data at the frame rate
26
that matches a high display capability even if the
decoding capability is low, for example.
[0049] "Configuration of Transmission Apparatus"
Fig. 2 shows a configuration example of the
5 transmission apparatus 100. This transmission apparatus
100 includes a CPU (Central Processing Unit) 101, an
encoder 102, a compressed data buffer (cpb: coded
picture buffer) 103, a multiplexer 104, and a
transmission unit 105. The CPU 101 is a controller, and
10 controls the operation of the respective units of the
transmission apparatus 100.
[0050] The encoder 102 inputs non-compressed moving
image data, and hierarchically codes it. The encoder
102 classifies the image data of each picture
15 constituting the moving image data into a plurality of
hierarchies. Then, the encoder 102 codes the image data
of the picture in each of the classified hierarchies to
generate a video stream having the coded image data of
the picture in each hierarchy. The encoder 102 performs
20 coding such as H.264/AVC and H.265/HEVC. At this time,
the encoder 102 performs coding so that the picture to
be referred to (referred picture) belongs to the selfhierarchy
and/or a hierarchy lower than the selfhierarchy.
25 [0051] Fig. 3 shows an example of hierarchical
coding performed in the encoder 102. This is an example
27
in which the image data is divided into 5 hierarchies
from 0 to 4, and coding is performed on the image data
of the picture in each hierarchy.
[0052] The vertical axis represents the hierarchy.
5 As temporal_id (hierarchy identification information)
arranged at the header portion of the NAL unit
(nal_unit) constituting the coded image data of the
picture in the hierarchies from 0 to 4, the numbers
from 0 to 4 are set, respectively. On the other hand,
10 the horizontal axis represents the display order (poc:
picture order of composition). The left side is earlier
display time, and the right side is later display time.
[0053] Fig. 4(a) shows a structural example (Syntax)
of a NAL unit header, and Fig. 4(b) shows the content
15 (Semantics) of main parameters in the structural
example. In the 1 bit field of "Forbidden_zero_bit," 0
is essential. The 6 bit field of "Nal_unit_type"
represents the NAL unit type. In the 6 bit field of
"Nuh_layer_id," 0 is assumed. The 3 bit field of
20 "Nuh_temporal_id_plus1" represents the temporal_id and
takes values (from 1 to 7) obtained by adding 1 thereto.
[0054] Now, return to Fig. 3. Each of the
rectangular frames represents a picture, and the number
represents the order of pictures being coded, i.e.,
25 encoding order (decoding order on the receiver side).
For example, a sub-picture group (Sub group of
28
pictures) includes 16 pictures from "2" to "17," and
the "2" is the top picture of the sub-picture group.
The "1" is a picture of a previous sub-picture group.
Some sub-picture groups are collected to form GOP
5 (Group of Pictures).
[0055] The coded image data of the top picture of
the GOP includes the NAL unit of AUD, VPS, SPS, PPS,
PSEI, SLICE, SSEI, and EOS, as shown in Fig. 5. On the
other hand, the picture other than the top picture of
10 the GOP includes the NAL unit of AUD, PPS, PSEI, SLICE,
SSEI, and EOS. The VPS and SPS can be transmitted once
for each sequence (GOP), and the PPS can be transmitted
for each picture.
[0056] Now, return to Fig. 3. The arrows shown by
15 solid lines represent the picture-reference
relationship in coding. For example, the picture of "2"
is a P picture, and is coded with reference to the
picture of "1." Moreover, the picture of "3" is a B
picture, and is coded with reference to the pictures of
20 "1" and "3." Similarly, other pictures are coded with
reference to a near picture in the display order. It
should be noted that the picture in the hierarchy 4 is
not referred to from other pictures.
[0057] The encoder 102 divides the plurality of
25 hierarchies into a predetermined number of hierarchy
sets to generate a predetermined number of video
29
streams having the coded image data of the picture in
the hierarchy set. For example, the encoder 102
performs the division so that the lowest hierarchy set
includes a plurality of hierarchies and a hierarchy set
5 higher than the lowest hierarchy set includes one
hierarchy.
[0058] For example, in the example of the
hierarchical coding shown in Fig. 3, the encoder 102
divides the plurality of hierarchies into 2 hierarchy
10 sets including the hierarchies from 0 to 3 as the
lowest hierarchy set and the hierarchy 4 as a hierarchy
set higher than the lowest hierarchy set, as shown by
the alternate long and short dash line. In this case,
the encoder 102 generates two video streams (coded
15 streams) having the coded image data of the picture in
the hierarchy set.
[0059] Moreover, for example, in the example of the
hierarchical coding shown in Fig. 3, the encoder 102
divides the plurality of hierarchies into 3 hierarchy
20 sets including the hierarchies from 0 to 2 as the
lowerst hierarchy set, the hierarchy 3 as a hierarchy
set higher than the lowerst hierarchy set, and the
hierarchy 4 as a hierarchy higher than the hierarchy,
as shown by the alternate long and short dash line and
25 the long dashed double-short dashed line. In this case,
the encoder 102 generates three video streams (coded
30
video streams) having the coded image data of the
picture of the hierarchy set.
[0060] In this case, the video stream having the
coded image data of the picture in the lowest hierarchy
5 set is the base stream, and the stream type is "0x24."
Moreover, the video stream including the coded image
data of the picture in a hierarchy set higher than the
lowest hierarchy set is the enhanced stream, and the
stream type is newly defined as "0x25."
10 [0061] It should be noted that in the case where
there are a plurality of enhanced streams, it is
possible to newly define the stream type so that the
stream types of all of the enhanced streams are not
"0x25" but each enhance stream can be identified. For
15 example, in the case where there are two enhanced
streams, the stream type of a first enhanced stream is
"0x25," and the stream type of a second enhanced stream
is "0x26."
[0062] This stream type constitutes identification
20 information for identifying whether each of the
predetermined number of video streams is the base
stream or the enhanced stream. This stream type is
inserted in the layer of the transport stream TS.
Specifically, this stream type is inserted in the video
25 elementary stream loop arranged corresponding to each
of the predetermined number of video streams under the
31
program map table (PMT: Program Map Table).
[0063] Fig. 6 shows an example of encoding, decoding,
a display order, and delay during the hierarchical
coding. This example corresponds to the above-mentioned
5 example of the hierarchical coding shown in Fig. 3.
This example shows the case where all of the
hierarchies (all layers) are hierarchically coded with
a full time resolution. Fig. 6(a) represents the
encoder input. As shown in Fig. 6(b), with the delay in
10 the amount corresponding to 16 pictures, each picture
is encoded in the encoding order, and thus, a coded
stream is acquired. Moreover, Fig. 6(b) represents the
decoder input, and each picture is decoded in the
decoding order. Then, as shown in Fig. 6(c), with the
15 delay in the amount corresponding to 4 pictures, the
image data of each picture is acquired in the display
order.
[0064] Fig. 7(a) shows the coded stream that is
similar to that shown in Fig. 6(b) described above with
20 three divided stages of hierarchies, i.e., the
hierarchies 0 to 2, the hierarchy 3, and the hierarchy
4. Here, the "Tid" represents the temporal_id. Fig.
7(b) represents the expected display (display order) in
the case where each picture in the hierarchies of 0 to
25 2, i.e., the partial hierarchies having Tid of 0 to 2,
is selectively decoded. Moreover, Fig. 7(c) represents
32
the expected display (display order) in the case where
each picture in the hierarchies of 0 to 3, i.e., the
partial hierarchies having Tid of 0 to 3, is
selectively decoded. Furthermore, Fig. 7(d) represents
5 the expected display (display order) in the case where
each picture in the hierarchies of 0 to 4, i.e., all of
the hierarchies having Tid of 0 to 4, is selectively
decoded.
[0065] In the case where decoding processing is
10 performed on the coded stream shown in Fig. 7(a)
depending on the decoding capability, the decoding
capability having a time resolution at the full rate is
needed. However, in the case where decoding with Tid of
0 to 2 is performed, a decoder having a decoding
15 capability of 1/4 of the coded full time resolution
should be capable of performing the decoding. Moreover,
in the case where decoding with Tid of 0 to 3 is
performed, a decoder having a decoding capability of
1/2 of the coded full time resolution should be capable
20 of performing the decoding.
[0066] However, if pictures in a low hierarchy
referred to in the hierarchical coding are sequentially
coded at a timing with a full time resolution, it
outstrips the capability of the decoder that performs
25 the partial decoding. The period of time shown by A in
Fig. 7(a) corresponds to that. Because the decoder that
33
decodes the partial hierarchies having Tid of 0 to 2 or
Tid of 0 to 3 performs decoding and display with a
capability of 1/4 or 1/2 of the time axis as shown in
the example of display, it cannot sequentially decode
5 the pictures with the coded full time resolution during
the period of time A.
[0067] The Ta represents the time required for the
decoding processing for each picture in the decoder
that decodes Tid of 0 to 2. The Tb represents the time
10 required for the decoding processing for each picture
in the decoder that decodes Tid of 0 to 3. The Tc
represents the time required for the decoding
processing for each picture in the decoder that decodes
Tid of 0 to 4 (all hierarchies). In the relationship
15 between these time periods, the following equation is
established: Ta>Tb>Tc.
[0068] In this embodiment, the encoder 102 performs
decoding so that a decoding interval of at least the
coded image data of the picture in the lowest hierarchy
20 set is a regular interval. Fig. 8(a) shows the case
where each picture is coded at a timing of 120 Hz and
the full time resolution in the example of the
hierarchical coding shown in Fig. 3, and is divided
into two hierarchy sets in which the hierarchies 0 to 3
25 are the lowest hierarchy set constituting the base
stream (B stream) and the hierarchy 4 is a hierarchy
34
set higher than that constituting the enhanced stream
(E stream).
[0069] In this case, the time resolution of the
picture in the lowest hierarchy set is 60 fps, and
5 there exist pictures sequentially coded at timing of
120 Hz. The decoder having a decoding capability of 60
fps cannot sequentially and reliably perform the
decoding processing. Therefore, as shown in Fig. 8(b),
the coding timing for the picture in the lowest
10 hierarchy set constituting the base stream is adjusted
to be 60 Hz, and the coding is performed so that the
decoding interval of the coded image data of the
picture in the lowest hierarchy set is a regular
interval. Accordingly, the decoder having a decoding
15 capability of 60 fps can sequentially and reliably
perform decoding processing on the coded image data of
the picture in the lowest hierarchy set constituting
the base stream.
[0070] Moreover, as shown in Fig. 8(b), the decoding
20 is performed so that the coding timing for the picture
in the hierarchy set constituting the enhanced stream
(E stream) and therefore, the decoding timing of the
coded image data of the picture are average timing of
the decoding timings of the coded image data of the
25 picture in the lowerst hierarchy set constituting the
base stream (B stream). Accordingly, in the case where
35
the receiver has the capability to decode the coded
image data of the picture in the hierarchy set
constituting not only the base stream but also the
enhanced stream, it is possible to successively and
5 smoothly perform decoding processing on each picture.
[0071] Fig. 9 shows an example of coding timing
(decoding timing) of each picture in the case where two
video streams of the base stream (B stream) and the
enhanced stream (E stream) are generated in the example
10 of the hierarchical coding shown in Fig. 3. This is an
example in which the decoding delay of the enhanced
stream is decreased against the base stream. The
decoding delay in this case corresponds to 8 pictures
at the coding interval of the full time resolution (1/2
15 of the coding interval of the base stream).
[0072] In this example, the coding timing of the
picture of the base stream (B stream) is an evennumbered
timing, and the coding timing of the enhanced
stream (E stream) is an odd-numbered timing. Then, in
20 this example, the enhanced stream (E stream) is coded
immediately after the coding order of the highest layer
of the base stream (B stream). Specifically, the
picture of "9" of the enhanced stream (E stream) is
coded right after the picture of "8" of the base stream
25 (B stream).
[0073] Fig. 10 shows another example of coding
36
timing (decoding timing) for each picture in the case
where two video streams of the base stream (B stream)
and the enhanced stream (E stream) are generated in the
example of the hierarchical coding shown in Fig. 3.
5 This is an example in which the decoding delay of the
enhanced stream is increased against the base stream.
The decoding delay in this case corresponds to 16
pictures at the coding interval of the full time
resolution (1/2 of the coding interval of the base
10 stream). In the case where the decoding delay is
increased as described above, it needs a reference
memory having a large capacity in the non-compressed
data buffer (dpb: decoded picture buffer).
[0074] In this example, the coding timing for the
15 picture of the base stream (B stream) is an evennumbered
timing, and the coding timing of the enhanced
stream (E stream) is an odd-numbered timing. Then, in
this example, the enhanced stream (E stream) is coded
after the coding of the highest layer of the base
20 stream (B stream) is finished. Specifically, the
picture of "17" of the enhanced stream (E stream) is
coded right after the picture of "16" of the base
stream (B stream).
As described above, it is possible to decrease or
25 increase the delay amount of the enhanced stream
against the base stream. This delay amount can be
37
adjusted by limiting the destination picture with
coding efficiency priority or achievement of low delay.
[0075] Fig. 11(a) shows the case where each picture
is coded at the timing of 120 Hz and the full time
5 resolution in the example of the hierarchical coding
shown in Fig. 3, and is divided into three hierarchy
sets in which the hierarchies 0 to 2 are the lowest
hierarchy set constituting the base stream (B stream),
the hierarchy 3 is a hierarchy set higher than the
10 hierarchy set constituting the enhances stream (E
stream 1), and the hierarchy 4 is a hierarchy set
higher than the hierarchy set constituting the enhanced
stream (E stream 2).
[0076] In this case, the time resolution of the
15 picture in the lowest hierarchy set is 30 fps, and
there are pictures sequentially coded at timing of 120
Hz. The decoder having a decoding capability of 30 fps
cannot sequentially and reliably perform the decoding
processing. Therefore, as shown in Fig. 11(b), the
20 coding timing for the picture in the lowest hierarchy
set constituting the base stream is adjusted to be 30
Hz, and the coding is performed so that the decoding
interval of the coded image data of the picture in the
lowest hierarchy set is a regular interval. Accordingly,
25 the decoder having a decoding capability of 30 fps can
sequentially and reliably perform decoding processing
38
on the coded image data of the picture in the lowest
hierarchy set constituting the base stream.
[0077] Moreover, as shown in Fig. 11(b), the
decoding is performed so that the coding timing for the
5 picture in the hierarchy set constituting the enhanced
stream (E stream 1) and therefore, the decoding timing
of the coded image data of the picture are average
timing of the decoding timings for the coded image data
of the pictures in the lowerst hierarchy set
10 constituting the base stream (B stream). Furthermore,
as shown in Fig. 11(b), the decoding is performed so
that the coding timing for the picture in the hierarchy
set constituting the enhanced stream (E stream 2) and
therefore, the decoding timing of the coded image data
15 of the picture are average timing of the decoding
timings for the coded image data of the pictures in the
hierarchy set constituting the base stream (B stream)
and the enhanced stream (E stream 1). Accordingly, in
the case where the receiver has the capability to
20 decode the coded image data of the picture in the
hierarchy set constituting not only the base stream but
also the two enhanced streams, it is possible to
successively and smoothly perform decoding processing
on each picture.
25 [0078] Fig. 12 shows an example of coding timing
(decoding timing) for each picture in the case where
39
three video streams of the base stream (B stream), the
enhanced stream (E stream 1), and the enhanced stream
(E stream 2) are generated in the example of the
hierarchical coding shown in Fig. 3. This is an example
5 in which the decoding delay of the enhanced stream is
decreased against the base stream. The decoding delay
in this case corresponds to 12 pictures at the coding
interval of the full time resolution (1/4 of the coding
interval of the base stream).
10 [0079] In this example, the coding timing for the
picture of the base stream (B stream) is a timing of
multiples of four, and the coding timing of the
enhanced stream (E stream 1) is an average timing of
the coding timings for the pictures of the base stream
15 (B stream) in multiples of four. Moreover, the coding
timing of the enhanced stream (E stream 1) is an oddnumbered
timing.
[0080] Then, in this example, the enhanced stream (E
stream 1) is coded immediately after the coding order
20 of the highest layer of the base stream (B stream).
Specifically, the picture of "10" of the enhanced
stream (E stream 1) is coded right after the picture of
"8" of the base stream (B stream) is coded. Moreover,
in this example, the enhanced stream (E stream 2) is
25 coded immediately after the coding order of the
enhanced stream (E stream 1). Specifically, the picture
40
of "11" of the enhanced stream (E stream 2) is coded
immediately after the picture of "10" of the enhanced
stream (E stream 1).
[0081] Fig. 13 shows another example of the coding
5 timing (decoding timing) of each picture in the case
where three video streams of the base stream (B stream),
the enhanced stream (E stream 1), and the enhanced
stream (E stream 2) are generated in the example of the
hierarchical coding shown in Fig. 3. This is an example
10 in which the decoding delay of the enhanced stream is
increased against the base stream. The decoding delay
in this case corresponds to 27 pictures at the coding
interval of the full time resolution (1/4 of the coding
interval of the base stream). In the case where the
15 decoding delay is increased as described above, it
needs a reference memory having a large capacity in the
non-compressed data buffer (dpb: decoded picture
buffer).
[0082] In this example, the coding timing of the
20 picture of the base stream (B stream) is a timing of
multiples of four, and the coding timing of the
enhanced stream (E stream 1) is an average timing of
the coding timing of the picture of the base stream (B
stream) in multiples of four. Moreover, the coding
25 timing of the enhanced stream (E stream 1) is an oddnumbered
timing.
41
[0083] Then, in this example, the enhanced stream (E
stream 1) is coded after the highest layer of the base
stream (B stream) is coded. Specifically, the picture
of "14" of the enhanced stream (E stream 1) is coded
5 right after the picture of "12" of the base stream (B
stream). Moreover, in this example, the enhanced stream
(E stream 2) is coded after the enhanced stream (E
stream 1) is coded. Specifically, the picture of "27"
of the enhanced stream (E stream 2) is coded right
10 after the picture of "26" of the enhanced stream (E
stream 1).
[0084] Fig. 14 shows an example of HRD (Hypothetical
Reference Decoder) control of the encoder 102. This is
an example in which two video streams of the base
15 stream (B stream) and the enhanced stream (E stream)
are generated. Here, a description will be made with
the base stream as a substream 1 (Substream 1) and the
enhanced stream as a substream 2 (Substream 2).
[0085] A solid line a1 having a step-like shape
20 represents the shift of the data amount of the
substream 1 generated by encoding (coding), and each
step corresponds to the unit of one picture. The height
of the step represents the data amount generated by the
encoding.
25 [0086] A timing P01 represents the timing when the
first byte of the coded image data of the first picture
42
enters cpb1 (coded picture buffer 1: compressed data
buffer). R1 represents the input bit rate to the cpb1
of the coded image data of the first picture. Here, if
the amount of coded data input to the cpb1 at the time
5 of T1 is Q1, the following equation is established:
R1=Q1/T1. It should be noted that in the example shown
in the figure, the case where the input bit rate to the
cpb1 of the coded image data of another picture is also
R1 is shown.
10 [0087] A solid line b1 having a step-like shape
represents the shift of the data amount consumed by
decoding in the cpb1, and each step corresponds to the
unit of one picture. The height of the step represents
the data amount consumed by the decoding. Qcpb1
15 represents the occupation amount of the cpd1. The
encoding is performed so that this occupation amount
does not exceed the size of cpb1 (memory capacity) at
any timing.
[0088] Moreover, a solid line a2 having a step-like
20 shape represents the shift of the data amount of the
substream 2, which is generated by the encoding
(coding), and each step corresponds to the unit of one
picture. The height of the step represents the data
amount generated by the encoding.
25 [0089] A timing P02 represents a timing when the
first byte of the coded image data of the first picture
43
enters cpb2 (coded picture buffer 2: compressed data
buffer). R2 represents the input bit rate to the cpb2
of the coded image data of the first picture. Here, if
the amount of coded data input to the cpb2 at the time
5 of T2 is Q2, the following equation is established:
R2=Q2/T2. It should be noted that in the example shown
in the figure, the case where the input bit rate to the
cpb2 of the coded image data of another picture is also
R2 is shown.
10 [0090] A solid line b2 having a step-like shape
represents the shift of the data amount consumed by
decoding in the cpb2, and each step corresponds to the
unit of one picture. The height of the step represents
the data amount consumed by the decoding. Qcpb2
15 represents the occupation amount of the cpd2. The
encoding is performed so that this occupation amount
does not exceed the size of the cpb2 (memory capacity)
at any timing.
[0091] In the example shown in the figure, decoding
20 is performed for the substream 1 in the picture order
of "1-0," "1-1," "1-2," "1-3," …, for the substream 2
in the picture order of "2-0," "2-1," "2-2," "2-3" …,
as described above, the pictures of the substream 1 and
the pictures of the substream 2 are alternately decoded.
25 The decoded image data of each picture is input to the
dpb (decoded picture buffer: non-compressed data). In
44
this example, the number of delay pictures from when
the decoding is performed to when display is started is
4 pictures.
[0092] It should be noted that in the above, R1 and
5 R2 represent the example of the fixed bit rate
(constant_bit_rate). However, it is not limited thereto,
and the same idea can be applied to a variable bit rate
(variable_bit_rate).
[0093] Fig. 15 shows a configuration example of the
10 encoder 102. The encoder 102 includes a temporal ID
generation unit 121, a buffer delay controller 122, an
HRD (Hypothetical Reference Decoder) setting unit 123,
a parameter set/SEI encoding unit 124, a slice encoding
unit 125, and a NAL packetizing unit 126.
15 [0094] To the temporal ID generation unit 121,
information on the number of hierarchies (Number of
layers) is supplied from the CPU 101. The temporal ID
generation unit 121 generates a temporal_id depending
on the number of hierarchies based on the information
20 on the number of hierarchies. For example, in the
example of the hierarchical coding shown in Fig. 3,
temporal_ids of 0 to 4 are generated.
[0095] To the buffer delay controller 122,
information of minimum decoding capability
25 (minimum_target_decoder_level_idc) is supplied from the
CPU 101, and the temporal_id generated in the temporal
45
ID generation unit 121 is supplied. The buffer delay
controller 122 calculates "initial_cpb_removal_delay"
being an initial value of cpb buffering for each video
stream, and "cpb_removal_delay" and "dpb_output_delay"
5 for each picture.
[0096] The buffer delay controller 122 controls
"Cpb_removal_delay" in the cpb buffer for each
substream (Sub-stream). The buffer delay controller 122
performs the control so that a buffer failure does not
10 occur between the decoding timing of the decoder and
the display timing in the dpb buffer. In this case,
"cpb_removal_delay" is controlled so that the decoding
timing of the picture of the lowest hierarchy set is a
regular interval. Moreover, in this case,
15 "cpb_removal_delay" is controlled so that the encoding
timing of the coded image data of the picture in the
hierarchy set higher than the lowest hierarchy set is
an average timing of the encoding timings of the coded
image data of the pictures of all hierarchy sets lower
20 than the hierarchy set. Moreover, "dpb_output_delay" is
controlled so that a cpb buffer failure does not occur.
It should be noted that the encoding timing represents
the same meaning as the decoding timing read from the
compressed data buffer (cpb: coded picture buffer) on
25 the receiver side.
[0097] To the HRD (Hypothetical Reference Decoder)
46
setting unit 123, "cpb_removal_delay" and
"dpb_output_delay" of the picture of each video stream
calculated in the buffer delay controller 122 is
supplied, and the information on the number of streams
5 (Number of streams) is supplied from the CPU 101. The
HRD setting unit 123 performs HRD setting based on the
information.
[0098] To the parameter set/SEI encoding unit 124,
HRD setting information and a temporal_id are supplied.
10 The parameter set/SEI encoding unit 124 generates
parameter sets of each hierarchy such as VPS, SPS, and
PPS, and SEI, depending on the number of streams to be
coded.
[0099] For example, a picture timing SEI including
15 "cpb_removal_delay" and "dpb_output_delay" (Picture
timing SEI) is generated. In addition, for example, a
buffering period SEI including
"initial_cpb_removal_time" (Buffering Period SEI) is
generated. The buffering period SEI is generated
20 corresponding to the top picture of GOP (access unit).
[0100] The "initial cpb removal time" represents
time (initial time) for taking the coded image data of
the top picture of GOP (Group Of Pictures) from the
compressed data buffer (cpb) for decoding. The
25 "cpb_removal_delay" represents time for taking the
coded image data of each picture from the compressed
47
data buffer (cpb), and the time is determined together
with "initial_cpb_removal_time." Moreover, the
"dpb_output_delay" represents time for taking after
decoding and entering the compressed data buffer (dpb).
5 [0101] The slice encoding unit 125 encodes the image
data of the picture of each hierarchy to acquire slice
data (slice segment header, slice segment data). The
slice encoding unit 125 uses a frame buffer to insert
"ref_idx_l0_active (ref_idx_l1_active)" representing
10 the index of the picture predicted by the "Prediction
Unit" in "slice segment header" as information
representing the prediction state of the time direction.
Accordingly, when decoding is performed, the hierarchy
level shown by the temporal_id as well as the reference
15 picture is determined. Moreover, the slice encoding
unit 125 inserts the index of the current slice in the
"slice segment header" as "short_term_ref_pic_set_idx"
or "it_idx_sps."
[0102] The NAL packetizing unit 126 generates the
20 coded image data of each hierarchy based on the
parameter set and SEI generated by the parameter
set/SEI encoding unit 124 and the slice data amount
generated by the slice encoding unit 125 to output
video streams (coded streams) depending on the number
25 of streams.
[0103] At this time, a temporal_id representing the
48
hierarchy is added to the NAL unit header for each
picture (see, Fig. 4). Moreover, the picture that
belongs to the hierarchy shown by the temporal_id is
tied up as a sublayer (sub_layer), and the level
5 designation value "level_idc" of the bit rate for each
sublayer is regarded as "sublayer_level_idc" and is
inserted in VPS or SPS.
[0104] Fig. 16 shows processing flow of the encoder
102. The encoder 102 starts processing in step ST1, and
10 after that, performs processing of step ST2. In step
ST2, the encoder 102 sets the hierarchy number N in the
hierarchical coding. Next, the encoder 102 sets the
temporal_id of the picture of each hierarchy to 0 to
(N-1) in step ST3.
15 [0105] Next, the encoder 102 sets a hierarchy level
K at which the decoder having the smallest capability
out of target decoders can perform decoding to 0 to N-1
in step ST4. Then, the encoder 102 sets the picture
encoding interval and the encoding timing in each
20 hierarchy set in the buffer delay controller 122 in
step ST5.
[0106] Next, the encoder 102 reflects the picture
encoding interval and the encoding timing obtained in
step ST5 to "cpb_removal_delay" and "dpb_output_delay,"
25 performs HRD setting, encoding of the parameter set/SEI,
and slice encoding, and transfers them to a
49
multiplexing block as a NAL unit in step ST6. After
that, the encoder 102 finishes the processing in step
ST7.
[0107] Now, return to Fig. 2. The compressed data
5 buffer (cpb) 103 temporarily stores the video stream
including the coded data of the picture of each
hierarchy generated by the encoder 102. The multiplexer
104 reads the video stream stored in the compressed
data buffer 103, and PES-packetizes, transport10
packetizes, and multiplexes it to acquire the transport
stream TS serving as a multiplexed stream.
[0108] The transport stream TS includes a
predetermined number of video streams having the coded
image data of the picture of each hierarchy set
15 obtained by dividing a plurality of hierarchies, as
described above. The multiplexer 104 inserts the
identification information for identifying each of the
predetermined number of video streams is the base
stream or the enhanced stream in the transport stream
20 TS. In this case, the identification information is
inserted as the stream type in the video elementary
stream loop (Video ES loop) arranged corresponding to
each of the predetermined number of video streams under
the program map table.
25 [0109] In this case, the stream type of the base
stream is "0x24." Moreover, the stream type of the
50
enhanced stream is newly defined. For example, it is
"0x25." It should be noted that in the case where there
are a plurality of enhanced streams, the stream types
of all of the enhanced streams do not have to be the
5 same and a plurality of stream types may be newly
defined as the stream types of the enhanced streams so
that each enhanced stream can be identified. For
example, in the case where there are two enhanced
streams, the stream type of a first enhanced stream is
10 "0x25," and the stream type of a second enhanced stream
is "0x26."
[0110] The multiplexer 104 inserts the configuration
information of the video stream in the layer of the
transport stream TS so as to correspond to the
15 predetermined number of streams. The multiplexer 104
inserts the configuration information as a descriptor
in the video elementary stream loop arranged
corresponding to each of the predetermined number of
video streams under the program map table.
20 [0111] The multiplexer 104 inserts a multistream
descriptor (multistream_descriptor) to be newly defined
as well as a HEVC descriptor (HEVC_descriptor). Fig. 17
shows a structural example (Syntax) of the HEVC
descriptor (HEVC_descriptor). The 8 bit field of
25 "descriptor_tag" represents the descriptor type, and
represents HEVC descriptor, here. The 8 bit field of
51
"descriptor_length" represents the length of the
descriptor (size), and represents the number of
subsequent bytes as the length of the descriptor.
[0112] The 8 bit field of "level_idc" represents the
5 level designation value of a bit rate. Moreover, in the
case where "temporal_layer_subset_flag = 1," there are
the 5 bit field of "temporal_id_min" and the 5 bit
field of "temporal_id_max." The "temporal_id_min"
represents the value of the temporal_id in the lowest
10 hierarchy of the hierarchically coded data included in
the corresponding video stream. The "temporal_id_max"
represents the value of the temporal_id in the highest
hierarchy of the hierarchically coded data of the
corresponding video stream.
15 [0113] Fig. 18 shows a structure example (Syntax) of
the multistream descriptor (multistream_descriptor).
Moreover, Fig. 19 shows the content (Semantics) of the
main information in the structural example.
[0114] The 8 bit field of the
20 "multistream_descriptor_tag" represents the descriptor
type, and represents the multistream descriptor, here.
The 8 bit field of the "multistream_descriptor_length"
represents the length (size) of the descriptor, and
represents the number of subsequent bytes as the length
25 of the descriptor. Here, it represents 2 bytes. The 4
bit field of the "group_id" represents the ID of the
52
group associated in a series of services. In this case,
the base stream (base stream) and all of the non-base
stream (non-base stream=enhanced stream) based thereon
have the same id.
5 [0115] The 4 bit field of the
"stream_dependency_ordering" defines the dependence
relationship between streams starting from the base
stream (base stream) in ascending order. The "0001"
represents a basic stream. The "0010" represents a
10 second stream from the basic stream (enhanced stream).
The "0011" represents a third stream from the basic
stream. The "max_layer_in_group" represents the maximum
value in the hierarchy coded in the group.
[0116] Fig. 20 shows an example of "Stream type,"
15 "Group_id," "max/min layer," "max_layer_in_group," and
"Stream_dependency ordering" in the case where the
transport stream TS includes a video stream group of a
service 1 (SERVICE 1) and a service 2 (SERVICE 2), for
example.
20 [0117] In this example, as the video stream of the
service 1, three video streams of the base stream (Base
stream), the enhanced stream (Enhanced stream 1), and
the enhanced stream (Enhanced stream 2) are included.
In the service 1, the value of the "Group_id" is "0."
25 Moreover, in the service 1, the number of hierarchies
is 5 and the value of the "max/min layer" is "4,"
53
similarly to the example of the hierarchical coding
shown in Fig. 3.
[0118] Moreover, the service 1 is divided into three
hierarchy sets. The value of the "Stream_type" of the
5 base stream is set to "0x24," and the "max/min layer"
of the HEVC descriptor represents to include pictures
in the hierarchies 0 to 2. Moreover, the value of the
"Stream_type" of the enhanced stream (Enhanced stream
1) is set to "0x25," and the "max/min layer" of the
10 HEVC descriptor represents to include the picture in
the hierarchy 3. Furthermore, the value of the
"Stream_type" of the enhanced stream (Enhanced stream
2) is set to "0x25," and the "max/min layer" of the
HEVC descriptor represents to include the picture in
15 the hierarchy 4.
[0119] Moreover, in this example, as the video
stream of the service 2, three video streams of the
base stream (Base stream), the enhanced stream
(Enhanced stream 1), and the enhanced stream (Enhanced
20 stream 2) are included. In the service 2, the value of
the "Group_id" is "1." Moreover, in the service 2, the
number of hierarchies is 4, and the value of the
"max/min layer" is "3," for example.
[0120] Moreover, the service 2 is divided into three
25 hierarchy sets. The value of the "Stream_type" of the
base stream is set to "0x24," and the "max/min layer"
54
of the HEVC descriptor represents to include pictures
of the hierarchies 0 to 1. Moreover, the value of
"Stream_type" of the enhanced stream (Enhanced stream
1) is set to "0x25," and the "max/min layer" of the
5 HEVC descriptor represents to include the picture in
the hierarchy 2. Furthermore, the value of the
"Stream_type" of the enhanced stream (Enhanced stream
2) is set to "0x25," and the "max/min layer" of the
HEVC descriptor represents to include the picture in
10 the hierarchy 3.
[0121] Fig. 21 shows a configuration example of the
multiplexer 104. The multiplexer 104 includes a section
coding unit 142, PES packetizing units 143-1 to 143-N,
a switching unit 144, and a transport packetizing unit
15 145.
[0122] The PES packetizing units 143-1 to 143-N read
video streams 1 to N stored in the compressed data
buffer 103, respectively, and generate PES packets.
Here, the video streams 1 to N each includes one base
20 stream and at least one enhanced stream.
[0123] At this time, the PES packetizing units 143-1
to 143-N add the time stamp of DTS (Decoding Time
Stamp) and PTS (Presentation Time Stamp) to the PES
header based on the HRD information of the video
25 streams 1 to N, respectively. In this case,
"cpu_removal_delay" and "dpb_output_delay" of each
55
picture are referred to, DTS and PTS are generated with
the precision in sync with the STC (System Time Clock)
time, and arranged at a predetermined position in the
PES header.
5 [0124] The switching unit 144 selectively takes PES
packets generated in the PES packetizing units 143-1 to
143-N based on the packet identifier (PID), and
transmits it to the transport packetizing unit 145. The
transport packetizing unit 145 generates a TS packet
10 including the PES packet in the payload to obtain the
transport stream TS.
[0125] The section coding unit 142 generates various
kinds of section data to be inserted in the transport
stream TS. to the section coding unit 142, information
15 such as the number of hierarchies (Number of layers)
and the number of streams (Number of streams) is
supplied from the CPU 101. The section coding unit 142
generates the above-mentioned HEVC descriptor
(HEVC_descriptor) and multistream descriptor
20 (multistream_descriptor) based on the information.
[0126] The section coding unit 142 transmits the
various kinds of section data to the transport
packetizing unit 145. The transport packetizing unit
145 generates a TS packet including the section data,
25 and inserts it in the transport stream TS. It should be
noted that at this time, also the stream type is
56
inserted in the video elementary stream loop (Video ES
loop) arranged corresponding to each video stream. In
this case, the stream type of the base stream is
"0x24," and the stream type of the enhanced stream is
5 newly defined as "0x25," for example.
[0127] Fig. 22 shows the processing flow of the
multiplexer 104. The multiplexer 104 starts the
processing in step ST11, and thereafter, performs the
processing of step ST12. in step ST12, the multiplexer
10 104 refers to the HRD information (cpu_removal_delay,
dpb_output_delay) to determine DTS and PTS, and inserts
them in a predetermined position in the PES header.
[0128] Next, the multiplexer 104 determines whether
or not it is multistream, i.e., N is plural number in
15 step ST13. If it is multistream, the multiplexer 104
performs multiplex processing with a plurality of PIDs
in step ST14. Then, the multiplexer 104 determines
whether or not it is the base stream in step ST15.
[0129] If it is the base stream, the multiplexer 104
20 sets the stream type to "0x27" in step ST16, and
thereafter performs the processing of step ST10. On the
other hand, if it is the enhanced stream, the
multiplexer 104 sets the stream type to a value that
represents the enhanced stream, e.g., newly defined as
25 "0x28" in step ST17, and thereafter performs the
processing of step ST18.
57
[0130] It should be noted that if it is not the
multistream in step ST13, the multiplexer 104 performs
multiplex processing with one PID in step ST19 and
thereafter performs the processing of step ST18.
5 [0131] In step ST18, the multiplexer 104 performs
section coding on the HEVC descriptor, multistream
descriptor, and the like, inserts the coded stream
(video elementary stream) in the PES payload for PES
packetization. Then, the multiplexer 104 performs
10 transport-packetization in step ST20 to obtain the
transport stream TS. After that, the multiplexer 104
finishes the processing in step ST21.
[0132] Fig. 23 shows a configuration example of the
transport stream TS in the case where a service is
15 delivered with two streams. The transport stream TS
includes two video streams of the base stream and the
enhanced stream. Specifically, in this configuration
example, there are a PES packet "video PES1" of the
base stream and a PES packet "video PES2" of the
20 enhanced stream.
[0133] Moreover, the transport stream TS includes a
PMT (Program Map Table) as one of PSI (Program Specific
Information). This PSI is information that represents
which program each elementary stream included in the
25 transport stream belongs to.
[0134] In the PMT, there is a program loop (Program
58
loop) to which information related to the entire
program is written. Moreover, in the PMT, there is an
elementary stream loop having information related to
each video stream. In this configuration example, there
5 are a video elementary stream loop "video ES1 loop"
corresponding to the base stream and a video elementary
stream loop "video ES2 loop" corresponding to the
enhanced stream.
[0135] On the "video ES1 loop," information such as
10 the stream type and the packet identifier (PID) is
arranged corresponding to the base stream (video PES1),
and also a descriptor to which information related to
the video stream is written is arranged. This stream
type is "0x24" that represents the base stream.
15 Moreover, as one of the descriptor, the above-mentioned
HEVC descriptor and multistream descriptor are inserted.
[0136] Moreover, on the "video ES2 loop,"
information such as the stream type and the packet
identifier (PID) is arranged corresponding to the
20 enhanced stream (video PES2), and also a descriptor to
which information related to the video stream is
written is arranged. This stream type is, for example,
newly defined as "0x25" that represents enhanced stream.
Moreover, as one of the descriptor, the above-mentioned
25 HEVC descriptor and multistream descriptor are inserted.
[0137] Fig. 24 shows a configuration example of the
59
transport stream TS in the case where a service is
delivered with three streams. The transport stream TS
includes three streams of the base stream and two
enhanced streams. Specifically, in this configuration
5 example, there are a PES packet "video PES1" of the
base stream and PES packets "video PES2" and "video
PES3" of the enhanced streams.
[0138] Moreover, in the PMT, there is an elementary
stream loop having information related to each video
10 stream. In this configuration example, there are a
video elementary stream loop "video ES1 loop"
corresponding to the base stream and a video elementary
stream loop "video ES2 loop" and a video elementary
stream loop "video ES3 loop" corresponding to the two
15 enhanced streams.
[0139] On the "video ES1 loop," information such as
the stream type and the packet identifier (PID) is
arranged corresponding to the base stream (video PES1),
and a descriptor to which information related to the
20 video stream is written is arranged. This stream type
is "0x24" that represents the base stream. Moreover, as
one of the descriptor, the above-mentioned HEVC
descriptor and multistream descriptor are inserted.
[0140] Moreover, on the "video ES2 loop,"
25 information such as the stream type and the packet
identifier (PID) is arranged corresponding to the
60
enhanced stream (video PES2), and also a descriptor to
which information related to the video stream is
written is arranged. This stream type is, for example,
newly defined as "0x25" that represents the enhanced
5 stream. Moreover, as one of the descriptor, the abovementioned
HEVC descriptor and multistream descriptor
are inserted.
[0141] Moreover, on the "video ES3 loop",
information such as the stream type and the packet
10 identifier (PID) is arranged corresponding to the
enhanced stream (video PES3), and also a descriptor to
which information related to the video stream is
written is arranged. This stream type is, for example,
newly defined as "0x25" or "0x26" that represents the
15 enhanced stream. Moreover, as one of the descriptor,
the above-mentioned HEVC descriptor and multistream
descriptor are inserted.
[0142] Now, return to Fig. 2. The transmission unit
105 modulates the transport stream TS in a modulation
20 form suitable for broadcasting such as QPSK/OFDM, and
transmits an RF modulation signal from a transmission
antenna.
[0143] The operation of the transmission apparatus
100 shown in Fig. 2 will be simply described. To the
25 encoder 102, non-compressed moving image data is input.
The encoder 102 performs hierarchical coding on the
61
moving image data. Specifically, in the encoder 102,
the image data of each picture constituting the moving
image data is classified into a plurality of
hierarchies and is coded to generate a video stream
5 having the coded image data of the picture in each
hierarchy. At this time, the coding is performed so
that the referred picture belongs to a self-hierarchy
and/or a hierarchy lower than the self-hierarchy.
[0144] In the encoder 102, a plurality of
10 hierarchies are divided into a predetermined number of
hierarchy sets, and a predetermined number of video
streams having the coded image data of the picture in
each hierarchy set is generated. In this case, the base
stream having the coded image data of the picture in
15 the lowest hierarchy set is generated, and a
predetermined number of enhanced streams having the
coded image data of the picture in the hierarchy set
higher than the lowest hierarchy set is generated.
[0145] The predetermined number of video streams
20 generated in the encoder 102 is supplied to the
compressed data buffer (cpb) 103, and is temporarily
stored. In the multiplexer 104, each video stream
stored in the compressed data buffer 103 is read, PESpacketized,
transport-packetized, and multiplexed to
25 obtain the transport stream TS as the multiplexed
stream.
62
[0146] Moreover, in the multiplexer 104,
identification information for identifying whether the
predetermined number of video streams is the base
stream or the enhanced stream is inserted in the layer
5 of the transport stream TS. The identification
information is the stream type to be inserted in the
video elementary stream loop (Video ES loop) arranged
corresponding to each video stream. In this case, the
stream type of the base stream is "0x24," and the
10 stream type of the enhanced stream is, for example,
newly defined as "0x25."
[0147] Moreover, in the multiplexer 104, the
configuration information of the video stream is
inserted in the layer of the transport stream TS
15 corresponding to the predetermined number of video
streams. Specifically, in the multiplexer 104, the HEVC
descriptor and the multistream descriptor are inserted
in the video elementary stream loop corresponding to
each video stream.
20 [0148] The transport stream TS generated in the
multiplexer 104 is transmitted to the transmission unit
105. In the transmission unit 105, the transport stream
TS is modulated in the modulation method suitable for
broadcasting such as QPSK/OFDM, and an RF modulation
25 signal is transmitted from the transmission antenna.
[0149]
63
[Configuration of Reception Apparatus]
Fig. 25 shows a configuration example of the
reception apparatus 200. the reception apparatus 200
includes a CPU (central processing unit) 201, a
5 reception unit 202, a demultiplexer 203, and a
compressed data buffer (cpb: coded picture buffer) 204.
Moreover, the reception apparatus 200 includes a
decoder 205, a non-compressed data buffer (dpb: decoded
picture buffer) 206, and a post processing unit 207.
10 The CPU 201 constitutes a controller, and controls the
operation of the respective units of the reception
apparatus 200.
[0150] The reception unit 202 demodulates an RF
modulation signal received by a reception antenna to
15 acquire the transport stream TS. The demultiplexer 203
selectively takes the coded image data of the picture
in the hierarchy set depending on the decoding
capability (Decoder temporal layer capability) from the
transport stream TS, and transmit it to the compressed
20 data buffer (cpb: coded picture buffer) 204.
[0151] Fig. 26 shows a configuration example of the
demultiplexer 203. The demultiplexer 203 includes a TS
adaptation field extraction unit 231, a clock
information extraction unit 232, a TS payload
25 extraction unit 233, a section extraction unit 234, a
PSI table/descriptor extraction unit 235, and a PES
64
packet extraction unit 236. Moreover, the demultiplexer
203 includes a PES header extraction unit 237, a time
stamp extraction unit 238, a PES payload extraction
unit 240, and a stream configuration unit (stream
5 composer) 241.
[0152] The TS adaptation field extraction unit 231
extracts an adaptation field from the TS packet having
the adaptation field of the transport stream TS. The
clock information extraction unit 232 extracts a PCR
10 (Program Clock Reference) from the adaptation field
including the PCR, and transmits it to the CPU 201.
[0153] The TS payload extraction unit 233 extracts a
TS payload of the transport stream TS from the TS
packet including the TS payload. The section extraction
15 unit 234 extracts section data from the TS payload
including the section data. The PSI table/descriptor
extraction unit 235 analyzes the section data extracted
by the section extraction unit 234, and extracts a PSI
table or a descriptor. Then, the PSI table/descriptor
20 extraction unit 235 transmits the minimum value (min)
and the maximum value (max) of the temporal_id, the
maximum hierarchy number, the stream dependence
relationship, the group id, and the like, to the CPU
201 and to the stream configuration unit 241.
25 [0154] The PES packet extraction unit 236 extracts a
PES packet from the TS payload including the PES packet.
65
The PES header extraction unit 237 extracts a PES
header from the PES packet extracted by the PES packet
extraction unit 236. The time stamp extraction unit 238
extracts a time stamp (DTS, PTS) inserted in the PES
5 header for each picture, and transmits it to the CPU
201 and the stream configuration unit 241.
[0155] The PES payload extraction unit 240 extracts
the PES payload, i.e., coded image data of the picture
in each hierarchy, from the PES packet extracted by the
10 PES packet extraction unit 236. The stream
configuration unit 241 selectively takes, from the
coded image data of the picture in each hierarchy to be
taken out by the PES payload extraction unit 240, only
the base stream or the coded image data of the picture
15 of the predetermined number of enhanced streams, and
transmits it to the compressed data buffer (cpb: coded
picture buffer) 204. In this case, the stream
configuration unit 241 refers to hierarchy information
acquired by the PSI table/descriptor extraction unit
20 235, for example.
[0156] For example, the case where the frame rate of
the predetermined number of video streams (coded
streams) included in the transport stream TS is 120 fps
will be discussed. For example, a plurality of
25 hierarchies are divided into two hierarchy sets of a
lower hierarchy set and a higher hierarchy set, and the
66
frame rates of the pictures in the hierarchy sets are
60 fps. For example, in the above-mentioned example of
the hierarchical coding shown in Fig. 3, the
hierarchies 0 to 3 are the lower hierarchy set, and the
5 decoder that supports the level_idc of 60 fps can
perform the decoding. Moreover, the hierarchy 4 is the
higher hierarchy set, and the decoder that supports the
level_idc of 120 fps can perform the decoding.
[0157] In the case where the decoding capability
10 supports 120 fps, the stream configuration unit 241
takes the coded image data of the base stream and the
enhanced stream based on the packet ID (PID), and
transmits it to the compressed data buffer (cpb: coded
picture buffer) 204. On the other hand, in the case
15 where the decoding capability supports not 120 fps but
60 fps, the stream configuration unit 241 takes only
the coded image data of the picture of the base stream
based on the packet ID (PID), and transmits it to the
compressed data buffer (cpb: coded picture buffer) 204.
20 [0158] Fig. 27 shows an example of picture (slice)
selection by the stream configuration unit 241 in the
case where the transport stream TS includes two video
streams (coded streams) of the base stream and the
enhanced stream. The packet identifier (PID) of the
25 base stream is a PID A and the packet identifier (PID)
of the enhanced stream is a PID B. The example shown in
67
the figure corresponds to the above-mentioned example
shown in Fig. 9 and shows only the portion of the n-th
sub-picture group (Sub group of pictures), and the
number added to each picture shown by a rectangular
5 frame represents the decoding order (encoding order on
the transmission side).
[0159] In the case where the decoding capability
supports 120 fps, the stream configuration unit 241
performs filtering based on the packet identifier (PID),
10 takes the coded image data of the pictures of the base
stream being the PID A and the enhanced stream being
the PID B, and transmits it to the compressed data
buffer (cpb: coded picture buffer) 204. In this case,
the coded image data of the picture of the base stream
15 is stored in an area 1 (cpb_1), and the coded image
data of the picture of the enhanced stream is stored in
an area 2 (cpb_2).
[0160] on the other hand, in the case where the
decoding capability supports not 120 fps but 60 fps,
20 the stream configuration unit 241 performs filtering
based on the packet identifier (PID), takes only the
coded image data of the picture of the base stream
being PID A, transmits it to the compressed data buffer
(cpb: coded picture buffer) 204, and stores it in the
25 area 1(cpb_1).
[0161] Fig. 28 shows an example of processing flow
68
of the demultiplexer 203. This processing flow shows
the case where the transport stream TS includes the
base stream having the coded image data of the picture
in the lowest hierarchy set and the predetermined
5 number of enhanced streams having the coded image data
of the picture in the predetermined number of hierarchy
sets higher than the lowest hierarchy set.
[0162] The demultiplexer 203 starts processing in
step ST41, and thereafter performs the processing of
10 step ST42. In step ST42, the decoding capability
(Decoder temporal layer capability) is set from the CPU
201. Next, the demultiplexer 203 determines whether or
not it is possible to perform decoding on the all
hierarchies (layers) in step ST43.
15 [0163] In the case where it is possible to perform
decoding on all of the hierarchies, the demultiplexer
203 selects all streams constituting all of the
hierarchies by a PID filter, and performs section
parsing (Section parsing)in step ST44. After that, the
20 demultiplexer 203 performs the processing of step ST45.
[0164] In the case where it is not possible to
perform decoding on all of the hierarchies in step ST43,
the demultiplexer 203 selects a predetermined number of
streams including the base stream constituting the
25 lower hierarchy that can be decoded, and performs
relevant section parsing (Section parsing) in step ST46.
69
After that, the demultiplexer 203 performs the
processing of step ST45.
[0165] In step ST45, the demultiplexer 203 reads the
HEVC descriptor and the multistream descriptor in the
5 target PID section to acquire the dependence
relationship of the streams in the group, the maximum
hierarchy number, and the maximum value and the minimum
value of the temporal_id.
[0166] Next, the demultiplexer 203 transfers the
10 coded image data of the picture of the stream selected
in step ST44 or step ST46 to the compressed data buffer
(cpb: coded picture buffer) 204 in step ST47. After the
processing of the step ST47, the demultiplexer 203
finishes the processing in step ST48.
15 [0167] Now, return to Fig. 25. The compressed data
buffer (cpb) 204 temporarily stores the video stream
(coded stream) taken out by the demultiplexer 203. The
decoder 205 takes, from the video stream stored in the
compressed data buffer 204, the coded image data of the
20 picture in the hierarchy designated as hierarchy to be
decoded. Then, the decoder 205 decodes the coded image
data of each picture thus taken out at the decoding
timing of the picture, and transmits it to the noncompressed
data buffer (dpb) 206.
25 [0168] Here, in the decoder 205, the hierarchy to be
decoded is designated with the temporal_id from the CPU
70
201. The designated hierarchy is regarded as all
hierarchies or a part of lower hierarchies included in
the video stream (coded stream) taken out by the
demultiplexer 203, and is automatically set by the CPU
5 201 or set according to a user' s operation. Moreover,
to the decoder 205, a decoding timing is given from the
CPU 201 based on the DTS (Decoding Time Stamp). It
should be noted that the decoder 205 reads and use the
image data of the referred picture from the non10
compressed data buffer 206 as necessary when the coded
image data of each picture is decoded.
[0169] Fig. 29 shows a configuration example of the
decoder 205. The decoder 205 includes a temporal ID
analysis unit 251, a target hierarchy selection unit
15 252, a stream combination unit 253, and a decoding unit
254. The temporal ID analysis unit 251 reads the video
stream (coded stream) stored in the compressed data
buffer 204, and analyzes the temporal_id inserted in
the NAL unit header of the coded image data of each
20 picture.
[0170] The target hierarchy selection unit 252 takes,
from each video stream read from the compressed data
buffer 204, the coded image data of the picture in the
hierarchy designated as the hierarchy to be decoded
25 based on the analysis results obtained from the
temporal ID analysis unit 251. In this case, from the
71
target hierarchy selection unit 252, one or more video
stream (coded stream) is output depending on the number
of video streams read from the compressed data buffer
204 and the designated hierarchy.
5 [0171] The stream combination unit 253 combines the
video streams (coded streams) output from the target
hierarchy selection unit 252 together. It should be
noted that although it differs from that shown in the
figure, the stream combination unit 253 may combine the
10 video streams (coded streams) output from the cpb
buffer 204 together. In this case, the stream
combination unit 253 simultaneously performs the target
hierarchy selection and the temporal ID analysis. The
stream combination unit 253 combines the coded image
15 data of the pictures together based on the decoding
timing information. Fig. 30 shows an example of stream
combination.
[0172] This example corresponds to the abovementioned
example shown in Fig. 9, and is an example in
20 which the pictures of the base stream with intervals of
60 Hz and the pictures of the enhanced stream with
intervals of 60 Hz are combined together. In this case,
each picture is regarded as one stream of a time stamp
of 120 Hz.
25 [0173] It should be noted that the one stream is
transmitted to the decoding unit 254, and each of the
72
coded image data of the picture is decoded at the
decoding timing and is stored in the dpb (noncompressed
data buffer) 206, as will be described later.
After that, the image data of each picture is
5 sequentially read from the non-compressed data buffer
206 after reordering of the pictures. In the example
shown in the figure, the picture in a sub-picture group
(shown by the downward-sloping hatching) is read first,
and then, the picture in the next sub-picture group
10 (shown by the upward-sloping hatching) is read. After
the decoding, the picture in the next sub-picture group
is stored in the non-compressed data buffer 206 during
the picture in a sub-picture group is displayed, and
becomes a subsequent referred picture.
15 [0174] It should be noted that the combination
processing itself of the pictures of the plurality of
streams may be performed for the plurality of selected
streams in the above-mentioned demultiplexer 203, and
may be transferred to the compressed data buffer (cpb:
20 coded picture buffer) 204 as one stream. Also the
combination processing at this time is performed based
on the decoding timing information similarly. In this
case, the combination processing does not need to be
performed in the decoder.
25 [0175] The decoding unit 254 sequentially decodes
the coded image data of each picture of the video
73
stream (coded stream) combined by the stream
combination unit 253 at the decoding timing, and
transmits it to the non-compressed data buffer (dpb)
206.
5 [0176] In this case, the decoding unit 254 analyzes
the VPS and SPS, grasps the level designation value
"sublayer_level_idc" of the bit rate for each sublayer,
for example, to check whether or not it can be decoded
with the decoding capability. Moreover, in this case,
10 the decoding unit 254 performs SEI analysis, grasps
"initial_cpb_removal_time" and "cpb_removal_delay," for
example, and checks whether or not the decoding timing
from the CPU 201 is appropriate.
[0177] The decoding unit 254 acquires
15 "ref_idx_l0_active (ref_idx_l1_active)" from the slice
header (Slice header) as information representing the
prediction destination of the time direction when
decoding the slice (Slice) to predict the time
direction. It should be noted that the picture after
20 decoding is processed as reference by another picture
with the "short_term_ref_pic_set_idx " or "it_idx_sps"
obtained from the slice header (slice header) as an
index.
[0178] Now, return to Fig. 25. The con-compressed
25 data buffer (dpb) 206 temporarily stores the image data
of each picture decoded by the decoder 205. The post
74
processing unit 207 performs processing to match the
frame rate of the image data of each picture
sequentially read from the non-compressed data buffer
(dpb) 206 at the display timing with the display
5 capability. In this case, the display timing is given
from the CPU 201 based on the PTS (Presentation Time
Stamp).
[0179] For example, if the frame rate of the image
data of each picture after decoding is 120 fps and the
10 display capability is 120 fps, the post processing unit
207 transmits the image data of each picture after
decoding to the display as it is. Moreover, for example,
if the frame rate of the image data of each picture
after decoding is 120 fps and the display capability is
15 60 fps, the post processing unit 207 performs subsampling
processing so that the time direction
resolution is reduced by half against the image data of
each picture after decoding, and transmits it to the
display as the image data of 60 fps.
20 [0180] Moreover, for example, if the frame rate of
the image data of each picture after decoding is 60 fps
and the display capability is 120 fps, the post
processing unit 207 performs interpolation processing
so that the time direction resolution is doubled
25 against the image data of each picture after decoding,
and transmits it to the display as the image data of
75
120 fps. Moreover, for example, if the frame rate of
the image data of each picture after decoding is 60 fps
and the display capability is 60 fps, the post
processing unit 207 transmits the image data of each
5 picture after decoding to the display as it is.
[0181] Fig. 31 shows a configuration example of the
post processing unit 207. This is an example in which
it is possible to process the case where the frame rate
of the image data of each picture after decoding is 120
10 fps or 60 fps as described above, and the display
capability is 120 fps or 60 fps.
[0182] The post processing unit 207 includes an
interpolation unit 271, a sub-sampling unit 272, and a
switching unit 273. The image data of each picture
15 after decoding from the non-compressed data buffer 206
is directly input to the switching unit 273, input to
the switching unit 273 after the frame rate is doubled
by the interpolation unit 271, or input to the
switching unit 273 after the frame rate is reduced by
20 half by the sub-sampling unit 272.
[0183] To the switching unit 273, selection
information is supplied from the CPU 201. This
selection information is automatically generated by the
CPU 201 referring to the display capability, or is
25 generated according to a user's operation. The
switching unit 273 selectively outputs any of the input
76
based on the selection information. Accordingly, the
frame rate of the image data of each picture
sequentially read from the non-compressed data buffer
(dpb) 206 at the display timing is regarded as one that
5 matches the display capability.
[0184] Fig. 32 shows an example of processing flow
of the decoder 205 and the post processing unit 207.
The decoder 205 and the post processing unit 207 start
processing in step ST51, and thereafter perform the
10 processing of step ST52. In this step ST52, the decoder
205 reads the video stream being the decoding target,
which is stored in the compressed data buffer (cpb:
coded picture buffer) 204, and selects the picture in
the hierarchy designated as a target from the CPU 201
15 based on the temporal_id.
[0185] Next, the decoder 205 sequentially decodes
the coded image data of each of the selected pictures
at the decoding timing, transfers the image data of
each picture after decoding to the non-compressed data
20 buffer (dpb) 206, and temporarily stores it in step
ST53. Next, the post processing unit 207 reads the
image data of each picture at the display timing from
the non-compressed data buffer (dpb) 206 in step ST54.
[0186] Next, the post processing unit 207 determines
25 whether or not the frame rate of the read image data of
each picture matches the display capability. If the
77
frame rate does not match the display capability, the
post processing unit 207 causes the frame rate to match
the display capability and transmits it to the display
in step ST56, and finishes the processing in step ST57.
5 On the other hand, if the frame rate matches the
display capability, the post processing unit 207
transmits the frame rate to the display as it is in
step ST58, and thereafter finishes the processing in
step ST57.
10 [0187] The operation of the reception apparatus 200
shown in Fig. 25 will be simply described. In the
reception unit 202, an RF modulation signal received by
a reception antenna is demodulated to acquire the
transport stream TS. This transport stream TS is
15 transmitted to the demultiplexer 203. In the
demultiplexer 203, all or a part of video streams are
PID-filtered from the transport stream TS depending on
the decoding capability (Decoder temporal layer
capability).
20 [0188] For example, if the decoding capability is
high, all of the video streams of the base stream and
the enhanced stream are selected. Moreover, for example,
if the decoding capability is low, a predetermined
number of video streams that include a decodable
25 hierarchy and the base stream are selected. Then, the
coded image data of the picture of the selected video
78
stream is transmitted from the demultiplexer 203 to the
compressed data buffer (cpb: coded picture buffer) 204,
and is temporarily stored.
[0189] In the decoder 205, the coded image data in
5 the hierarchy designated as the hierarchy to be decoded
is taken out from the video stream stored in the
compressed data buffer 204. Then, in the decoder 205,
the coded image data of each picture thus-taken out is
decoded at the decoding timing, is transmitted to the
10 non-compressed data buffer (dpb) 206, and is
temporarily stored. In this case, the image data of the
referred picture is read from the non-compressed data
buffer 206 as necessary when the coded image data of
each picture is decoded, and is used.
15 [0190] The image data of each picture sequentially
read from the non-compressed data buffer (dpb) 206 at
the display timing is transmitted to the post
processing unit 207. In the post processing unit 207,
interpolation or sub-sampling for matching the frame
20 rate of the image data of each picture with the display
capability is performed. The image data of each picture
processed in the post processing unit 207 is supplied
to the display, and moving images of the image data of
each picture are displayed.
25 [0191] As described above, in the
transmission/reception system 10 shown in Fig. 1,
79
coding is performs so that at least the decoding
interval of the coded image data of the picture in the
lowest hierarchy set is a regular interval on the
transmission side. Therefore, in the case where the
5 receiver has the decoding capability to perform
processing on the coded image data of the pictures in
the plurality of hierarchies of the lowest hierarchy
set, for example, it is possible to continuously and
effortlessly perform decoding processing on the coded
10 image data of each picture.
[0192] Moreover, in the transmission/reception
system 10 shown in Fig. 1, coding is performed so that
the decoding timing of the coded image data of the
picture in the hierarchy set higher than the lowest
15 hierarchy set is an average decoding timing for the
coded image data of the pictures in all of the
hierarchy sets lower than the hierarchy set on the
transmission side. Therefore, in the case where the
receiver has decoding capability to decode the coded
20 image data of the picture not only in the lowest
hierarchy set but also in the hierarchy set higher than
that, it is possible to sequentially and smoothly
perform the decoding processing on each picture.
[0193] Moreover, in the transmission/reception
25 system 10 shown in Fig. 1, when the plurality of
hierarchies are divided into a predetermined number of
80
hierarchy sets on the transmission side, the division
is performed so that the lowerst hierarchy set includes
a plurality of hierarchies and a hierarchy set higher
than the lowerst hierarchy set includes one hierarchy.
5 Therefore, in the case where the receiver has the
decoding capability to perform processing on the coded
image data of the pictures in the plurality of
hierarchies of the lowerst hierarchy set, it only needs
the configuration that selects only a video stream
10 having the coded image data of the picture in the
lowest hierarchy set, takes it in a buffer, and
performs decoding processing on it, and does not need a
complex configuration that performs combination
processing on a plurality of video streams, for example.
15 [0194] Moreover, in the transmission/reception
system 10 shown in Fig. 1, on the transmission side,
identification information for identifying whether each
of the predetermined number of video streams is the
base stream or the enhanced stream is inserted in the
20 layer of the transport stream TS. Therefore, the
receiver can easily select the base stream and
selectively decode the coded image data of the picture
in the lower hierarchy set by using the identification
information, for example.
25 [0195] Moreover, in the transmission/reception
system 10 shown in Fig. 1, on the transmission side,
81
the configuration information of the video stream is
inserted in the layer of the transport stream TS
corresponding to the predetermined number of video
streams included in the transport stream TS. Therefore,
5 the receiver can easily grasp, for example, which group
each video stream included in the transport stream TS
belongs to, what stream dependence relationship there
is between the video streams, and what the number of
hierarchies.
10 [0196] Moreover, in the transmission/reception
system 10 shown in Fig. 1, on the receiver side, the
coded image data of the picture in a hierarchy that is
equal to or lower than a predetermined hierarchy is
selectively taken in the compressed data buffer 204
15 depending on the decoding capability (Decoder temporal
layer capability) from the received video stream, and
is decoded. Therefore, it is possible to perform
appropriate decoding processing depending on the
decoding capability, for example.
20 [0197] Moreover, in the transmission/reception
system 10 shown in Fig. 1, on the receiver side, the
frame rate of the image data of each picture after
decoding is caused to match the display capability in
the post processing unit 207. Therefore, even if the
25 decoding capability is low, for example, it is possible
to acquire the image data of the frame rate matching
82
the display capability.
[0198]
<2. Modified Example>
It should be noted that in the above-mentioned
5 embodiment, the "max_layer_in_group" being information
of the hierarchy maximum value is written to the
multistream descriptor (see Fig. 18), and is
transmitted to the receiver. However, instead of
writing the information of the hierarchy maximum value
10 to the descriptor and supplying it to the receiver, it
is possible to describe that "designate the maximum
layer" on the face of a file or paper written standards
to designate or set the hierarchy maximum value in the
reception apparatus in advance. Also in this case, the
15 receiver refers to the information of the hierarchy
maximum value, performs filtering on the stream
including the hierarchy depending on the own decoding
capability, and performs decoding processing, similarly
to the case of supplying it with a descriptor from the
20 transmitter side as described above,
[0199] Moreover, in the above-mentioned embodiment,
an example of the time scalability (temporal
scalability) is described, and the multistream
descriptor is used to transmit the configuration
25 information of each stream to the receiver. However,
although a detailed description will be omitted, it is
83
possible to apply the above-mentioned multistream
descriptor in the service in which a basic stream (base
stream) and an extended stream (enhanced stream)
coexist in relation to scalability such as spatial
5 scalability and bit rate scalability. Specifically, the
above-mentioned multistream descriptor represents a
useful signaling method when multistream coding is
performed.
[0200] Moreover, in the above-mentioned embodiment,
10 the transmission/reception system 10 including the
transmission apparatus 100 and the reception apparatus
200 has been described. However, the configuration of
the transmission/reception system to which the present
technology can be applied is not limited thereto. For
15 example, the part of the reception apparatus 200 may
have the configuration of a set top box and a monitor
connected by a digital interface such as HDMI (High-
Definition Multimedia Interface). It should be noted
that "HDMI" is a registered trademark.
20 [0201] Moreover, in the above-mentioned embodiment,
an example in which the contenna is a transport stream
(MPEG-2 TS) has been described. However, the present
technology can be applied also to the system having the
configuration that uses a network such as the Internet
25 to perform delivery to the reception terminal similarly.
In the case of delivery using the Internet, delivery is
84
performed by a contenna in the format of MP4 or the
like. In other words, the contenna in the various
formats such as the transport stream (MPEG-2 TS)
adopted by the digital broadcasting standards and MP4
5 used by the Internet delivery corresponds to the
contenna.
[0202] Moreover, the present technology may also
take the following configurations.
(1) A coding apparatus, comprising
10 an image coding unit configured
to classify image data of each picture
constituting moving image data into a plurality of
hierarchies,
to code the image data of the picture in each
15 of the classified hierarchies,
to divide the plurality of hierarchies into a
predetermined number of hierarchy sets, and
to generate a predetermined number of video
streams having the coded image data of the picture in
20 each of the divided hierarchy sets, the image coding
unit performing coding so that a decoding interval of
the coded image data of the picture in at least the
lowest hierarchy set is a regural interval.
(2) The coding apparatus according to (1) above,
25 in which
the image coding unit performs coding so that a
85
decoding timing for the coded image data of the picture
in a hierarchy set higher than the lowest hierarchy set
is an average timing of decoding timings for the coded
image data of the pictures in all hierarchy sets lower
5 than the hierarchy set.
(3) The coding apparatus according to (1) or (2)
above, in which
the image coding unit divides the plurality of
hierarchies into the predetermined number of hierarchy
10 sets so that the lowest hierarchy set includes a
plurality of hierarchies and a hierarchy set higher
than the lowerst hierarchy set includes one hierarchy.
(4) A coding method, including:
classifying, by an image coding unit, image data
15 of each picture constituting moving image data into a
plurality of hierarchies;
coding the image data of the picture in each of
the classified hierarchies;
dividing the plurality of hierarchies into a
20 predetermined number of hierarchy sets; and
generating a predetermined number of video streams
having the coded image data of the picture in each of
the divided hierarchy sets, the image coding unit
performing coding so that a decoding interval of the
25 coded image data of the picture in at least the lowest
hierarchy set is a regural interval.
86
(5) A transmission apparatus, including:
an image coding unit configured
to classify image data of each picture
constituting moving image data into a plurality of
5 hierarchies,
to code the image data of the picture in each
of the classified hierarchies,
to divide the plurality of hierarchies into a
predetermined number of hierarchy sets, and
10 to generate a predetermined number of video
streams having the coded image data of the picture in
each of the divided hierarchy sets; and
a transmission unit configured to transmit a
contenner including the generated predetermined number
15 of video streams in a predetermined format, the image
coding unit performing coding so that a decoding
interval of the coded image data of the picture in at
least the lowest hierarchy set is a regural interval.
(6) A transmission apparatus, including:
20 an image coding unit configured
to classify image data of each picture
constituting moving image data into a plurality of
hierarchies,
to code the image data of the picture in each
25 of the classified hierarchies,
to divide the plurality of hierarchies into a
87
predetermined number of hierarchy sets, and
to generate a predetermined number of video
streams having the coded image data of the picture in
each of the divided hierarchy sets;
5 a transmission unit configured to transmit a
contenner including the generated predetermined number
of video streams in a predetermined format; and
an identification information insertion unit
configured to insert, in a layer of the contenna,
10 identification information for identifying whether each
of the predetermined number of video streams is a base
stream having the coded image data of the picture in
the lowerst hierarchy set or an enhanced stream having
the coded image data of the picture in a hierarchy set
15 higher than the lowerst hierarchy set.
(7) The transmission apparatus according to (6)
above, in which
the contenna is a transport stream, and
the identification information insertion unit
20 inserts, as a stream type, the identification
information in a video elementary stream loop arranged
corresponding to each of the predetermined number of
video streams under a program map table.
(8) The transmission apparatus according to (6) or
25 (7) above, in which
the image coding unit performs coding so that a
88
decoding interval of the coded image data of the
picture in at least the lowerst hierarchy set is a
regular interval.
(9) The transmission apparatus according to (8)
5 above, in which
the image coding unit performs coding so that a
decoding timing for the coded image data of the picture
in a hierarchy set higher than the lowest hierarchy set
is an average timing of decoding timings for the coded
10 image data of the pictures in all hierarchy sets lower
than the hierarchy set.
(10) A transmission apparatus, including:
an image coding unit configured
to classify image data of each picture
15 constituting moving image data into a plurality of
hierarchies,
to code the image data of the picture in each
of the classified hierarchies,
to divide the plurality of hierarchies into a
20 predetermined number of hierarchy sets, and
to generate a predetermined number of video
streams having the coded image data of the picture in
each of the divided hierarchy sets;
a transmission unit configured to transmit a
25 contenner including the generated predetermined number
of video streams in a predetermined format; and
89
a configuration information insertion unit
configured to insert, in a layer of the container,
configuration information of the video stream
corresponding to each of the predetermined numbe of
5 video streams included in the container.
(11) The transmission apparatus according to (10)
above, in which
the configuration information includes information
representing a service group to which the video stream
10 belongs.
(12) The transmission apparatus according to (10)
or (11) above, in which
the configuration information includes information
representing a dependence relationship between streams
15 starting from the base stream having the coded image
data of the picture in the lowest hierarchy set.
(13) The transmission apparatus according to any
one of (10) to (12) above, in which
the configuration information includes information
20 representing the number of hierarchies of the plurality
of hierarchies classified by the image coding unit.
(14) The transmission apparatus according to any
one of (10) to (13) above, in which
the container is a transport stream, and
25 the configuration information insertion unit
inserts, as a descriptor, the configuration information
90
in a video elementary stream loop arranged
corresponding to each of the predetermined number of
video streams under a program map table.
(15) A reception apparatus, including:
5 a reception unit configured to receive a
predetermined number of video streams having coded
image data of a picture in a hierarchy set of a
predetermined number of hierarchy sets, the coded image
data being obtained by classifying image data of
10 pictures constituting moving image data into a
plurality of hierarchies and coding the image data, the
plurality of hierarchies being divided into the
predetermined number of hierarchy sets; and
a processing unit configured to process the
15 received predetermined number of video streams, at
least a video stream of the predetermined number of
video streams, which has coded image data of a picture
in the lowest hierarchy set, being coded so that
decoding interval for each picture is a regular
20 interval.
(16) The reception apparatus according to (15)
above, in which
the predetermined number of video streams is coded
so that a decoding timing for the coded image data of
25 the picture in a hierarchy set higher than the lowest
hierarchy set is an average timing of decoding timings
91
for the coded image data of the pictures in all
hierarchy sets lower than the hierarchy set.
(17) A reception apparatus, including:
a reception unit configured to receive a
5 predetermined number of video streams having coded
image data of a picture in a hierarchy set of a
predetermined number of hierarchy sets, the coded image
data being obtained by classifying image data of
pictures constituting moving image data into a
10 plurality of hierarchies and coding the image data, the
plurality of hierarchies being divided into the
predetermined number of hierarchy sets; and
an image decoding unit configured
to selectively take coded image data of a
15 picture in a hierarchy that is equal to or lower than a
predetermined hierarchy depending on a decoding
capability in a buffer from the predetermined number of
video streams included in the received container;
to decode the coded image data of the picture
20 taken in the buffer; and
to acquire image data of the picture in the
hierarchy that is equal to or lower than the
predetermined hierarchy, at least a video stream of the
predetermined number of video streams, which has coded
25 image data of a picture in the lowest hierarchy set,
being coded so that decoding interval for each picture
92
is a regular interval.
(18) The reception apparatus according to (17)
above, in which
identification information for identifying whether
5 each of the predetermined number of video streams is a
base stream having the coded image data of the picture
in the lowerst hierarchy set or an enhanced stream
having the coded image data of the picture in a
hierarchy set higher than the lowerst hierarchy set is
10 inserted in a layer of the container, and
the image decoding unit takes coded image data of
a picture in a predetermined hierarchy set depending on
the decoding capability in the buffer from the
predetermined number of video streams including the
15 base stream based on the identification information,
and decodes it.
(19) The reception apparatus according to (17) or
(18) above, in which
in the case where the coded image data of the
20 picture in the predetermined hierarchy set is included
in a plurality of video streams, the image decoding
unit decodes the coded image data of each picture as
one stream based on decoding timing information.
(20) The reception apparatus according to any one
25 of (17) to (19) above, further including
a post processing unit configured to match a frame
93
rate of the image data of each picture acquired in the
image decoding unit with a display capability.
[0203] The main features of the present technology
are to allow a receiver to continuously and
5 effortlessly perform decoding processing on coded image
data of each picture if the receiver has the decoding
capability to process coded image data of pictures of a
plurality of hierarchies included in the lowest
hierarchy set by performing decoding so that at least a
10 decoding interval for the coded image data of the
picture in the lowest hierarchy set is a regular
interval (see Fig. 8, Fig. 11).
[0204] In addition, the main features of the present
technology are to selectively and easily decode a base
15 stream only, for example, by a receiver using
identification information for identifying whether each
of a predetermined number of video streams is the base
stream or an enhanced stream, by inserting the
identification information in a layer of the transport
20 stream TS (see Fig. 20, Fig. 23).
[0205] In addition, the main features of the present
technology are to allow it to easily know which group
the video stream included in the transport stream TS
belongs to, what stream-dependent relationship is there,
25 what is the number of hierarchies in the hierarchical
coding, and the like, by inserting configuration
94
information of the video stream corresponding to each
of the predetermined number of video streams included
in the transport stream TS in a layer of the transport
stream TS (see Fig. 20, Fig. 23).
5 Description of Reference Numerals
[0206]
10 transmission/reception system
100 transmission apparatus
101 CPU
10 102 encoder
103 compressed data buffer (cpb)
104 multiplexer
105 transmission unit
121 temporal ID generation unit
15 122 buffer delay controller
123 HRD setting unit
124 parameter set/SEI encoding unit
125 slice encoding unit
126 NAL packetizing unit
20 142 section coding unit
143-1 to 143-N PES packetizing unit
144 switching unit
145 transport packetizing unit
200 reception apparatus
25 201 CPU
202 reception unit
95
203 demultiplexer
204 compressed data buffer (cpb)
205 decoder
206 non-compressed data buffer (dpb)
5 207 post processing unit
231 TS adaptation field extraction unit
232 clock information extraction unit
233 TS payload extraction unit
234 section extraction unit
10 235 PSI table/descriptor extraction unit
236 PES packet extraction unit
237 PES header extraction unit
238 time stamp extraction unit
240 PES payload extraction unit
15 241 stream configuration unit
251 temporal ID analysis unit
252 target hierarchy selection unit
253 stream combination unit
254 decoding unit
20 271 interpolation unit
272 sub-sampling unit
273 switching unit
96
Claims
[1] A coding apparatus, comprising
an image coding unit configured
to classify image data of each picture
5 constituting moving image data into a plurality of
hierarchies,
to code the image data of the picture in each
of the classified hierarchies,
to divide the plurality of hierarchies into a
10 predetermined number of hierarchy sets, and
to generate a predetermined number of video
streams having the coded image data of the picture in
each of the divided hierarchy sets, the image coding
unit performing coding so that a decoding interval of
15 the coded image data of the picture in at least the
lowest hierarchy set is a regural interval.
[2] The coding apparatus according to claim 1, wherein
the image coding unit performs coding so that a
decoding timing for the coded image data of the picture
20 in a hierarchy set higher than the lowest hierarchy set
is an average timing of decoding timings for the coded
image data of the pictures in all hierarchy sets lower
than the hierarchy set.
[3] The coding apparatus according to claim 1, wherein
25 the image coding unit divides the plurality of
hierarchies into the predetermined number of hierarchy
97
sets so that the lowest hierarchy set includes a
plurality of hierarchies and a hierarchy set higher
than the lowerst hierarchy set includes one hierarchy.
[4] A coding method, comprising:
5 classifying, by an image coding unit, image data
of each picture constituting moving image data into a
plurality of hierarchies;
coding the image data of the picture in each of
the classified hierarchies;
10 dividing the plurality of hierarchies into a
predetermined number of hierarchy sets; and
generating a predetermined number of video streams
having the coded image data of the picture in each of
the divided hierarchy sets, the image coding unit
15 performing coding so that a decoding interval of the
coded image data of the picture in at least the lowest
hierarchy set is a regural interval.
[5] A transmission apparatus, comprising:
an image coding unit configured
20 to classify image data of each picture
constituting moving image data into a plurality of
hierarchies,
to code the image data of the picture in each
of the classified hierarchies,
25 to divide the plurality of hierarchies into a
predetermined number of hierarchy sets, and
98
to generate a predetermined number of video
streams having the coded image data of the picture in
each of the divided hierarchy sets; and
a transmission unit configured to transmit a
5 contenner including the generated predetermined number
of video streams in a predetermined format, the image
coding unit performing coding so that a decoding
interval of the coded image data of the picture in at
least the lowest hierarchy set is a regural interval.
10 [6] A transmission apparatus, comprising:
an image coding unit configured
to classify image data of each picture
constituting moving image data into a plurality of
hierarchies,
15 to code the image data of the picture in each
of the classified hierarchies,
to divide the plurality of hierarchies into a
predetermined number of hierarchy sets, and
to generate a predetermined number of video
20 streams having the coded image data of the picture in
each of the divided hierarchy sets;
a transmission unit configured to transmit a
contenner including the generated predetermined number
of video streams in a predetermined format; and
25 an identification information insertion unit
configured to insert, in a layer of the contenna,
99
identification information for identifying whether each
of the predetermined number of video streams is a base
stream having the coded image data of the picture in
the lowerst hierarchy set or an enhanced stream having
5 the coded image data of the picture in a hierarchy set
higher than the lowerst hierarchy set.
[7] The transmission apparatus according to claim 6,
wherein
the contenna is a transport stream, and
10 the identification information insertion unit
inserts, as a stream type, the identification
information in a video elementary stream loop arranged
corresponding to each of the predetermined number of
video streams under a program map table.
15 [8] The transmission apparatus according to claim 6,
wherein
the image coding unit performs coding so that a
decoding interval of the coded image data of the
picture in at least the lowerst hierarchy set is a
20 regular interval.
[9] The transmission apparatus according to claim 8,
wherein
the image coding unit performs coding so that a
decoding timing for the coded image data of the picture
25 in a hierarchy set higher than the lowest hierarchy set
is an average timing of decoding timings for the coded
100
image data of the pictures in all hierarchy sets lower
than the hierarchy set.
[10] A transmission apparatus, comprising:
an image coding unit configured
5 to classify image data of each picture
constituting moving image data into a plurality of
hierarchies,
to code the image data of the picture in each
of the classified hierarchies,
10 to divide the plurality of hierarchies into a
predetermined number of hierarchy sets, and
to generate a predetermined number of video
streams having the coded image data of the picture in
each of the divided hierarchy sets;
15 a transmission unit configured to transmit a
contenner including the generated predetermined number
of video streams in a predetermined format; and
a configuration information insertion unit
configured to insert, in a layer of the container,
20 configuration information of the video stream
corresponding to each of the predetermined numbe of
video streams included in the container.
[11] The transmission apparatus according to claim 10,
wherein
25 the configuration information includes information
representing a service group to which the video stream
101
belongs.
[12] The transmission apparatus according to claim 10,
wherein
the configuration information includes information
5 representing a dependence relationship between streams
starting from the base stream having the coded image
data of the picture in the lowest hierarchy set.
[13] The transmission apparatus according to claim 10,
wherein
10 the configuration information includes information
representing the number of hierarchies of the plurality
of hierarchies classified by the image coding unit.
[14] The transmission apparatus according to claim 10,
wherein
15 the container is a transport stream, and
the configuration information insertion unit
inserts, as a descriptor, the configuration information
in a video elementary stream loop arranged
corresponding to each of the predetermined number of
20 video streams under a program map table.
[15] A reception apparatus, comprising:
a reception unit configured to receive a
predetermined number of video streams having coded
image data of a picture in a hierarchy set of a
25 predetermined number of hierarchy sets, the coded image
data being obtained by classifying image data of
102
pictures constituting moving image data into a
plurality of hierarchies and coding the image data, the
plurality of hierarchies being divided into the
predetermined number of hierarchy sets; and
5 a processing unit configured to process the
received predetermined number of video streams, wherein
at least a video stream of the predetermined
number of video streams, which has coded image data of
a picture in the lowest hierarchy set, is coded so that
10 decoding interval for each picture is a regular
interval.
[16] The reception apparatus according to claim 15,
wherein
the predetermined number of video streams is coded
15 so that a decoding timing for the coded image data of
the picture in a hierarchy set higher than the lowest
hierarchy set is an average timing of decoding timings
for the coded image data of the pictures in all
hierarchy sets lower than the hierarchy set.
20 [17] A reception apparatus, comprising:
a reception unit configured to receive a
predetermined number of video streams having coded
image data of a picture in a hierarchy set of a
predetermined number of hierarchy sets, the coded image
25 data being obtained by classifying image data of
pictures constituting moving image data into a
103
plurality of hierarchies and coding the image data, the
plurality of hierarchies being divided into the
predetermined number of hierarchy sets; and
an image decoding unit configured
5 to selectively take coded image data of a
picture in a hierarchy that is equal to or lower than a
predetermined hierarchy depending on a decoding
capability in a buffer from the predetermined number of
video streams included in the received container;
10 to decode the coded image data of the picture
taken in the buffer; and
to acquire image data of the picture in the
hierarchy that is equal to or lower than the
predetermined hierarchy, wherein
15 at least a video stream of the predetermined
number of video streams, which has coded image data of
a picture in the lowest hierarchy set, is coded so that
decoding interval for each picture is a regular
interval.
20 [18] The reception apparatus according to claim 17,
wherein
identification information for identifying whether
each of the predetermined number of video streams is a
base stream having the coded image data of the picture
25 in the lowerst hierarchy set or an enhanced stream
having the coded image data of the picture in a
104
hierarchy set higher than the lowerst hierarchy set is
inserted in a layer of the container, and
the image decoding unit takes coded image data of
a picture in a predetermined hierarchy set depending on
5 the decoding capability in the buffer from the
predetermined number of video streams including the
base stream based on the identification information,
and decodes it.
[19] The reception apparatus according to claim 17,
10 wherein
in the case where the coded image data of the
picture in the predetermined hierarchy set is included
in a plurality of video streams, the image decoding
unit decodes the coded image data of each picture as
15 one stream based on decoding timing information.
[20] The reception apparatus according to claim 17,
further comprising
a post processing unit configured to match a frame
rate of the image data of each picture acquired in the
20 image decoding unit with a display capability.