FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENTS RULES, 2003
COMPLETE SPECIFICATION
(See section 10, rule 13)
“ENCODING DEVICE, TRANSMITTING DEVICE, AND RECEIVING
DEVICE”
SONY CORPORATION, of 1-7-1, Konan, Minato-ku, Tokyo 108-0075, Japan
The following specification particularly describes the invention and the manner in
which it is to be performed.
2
Description
Title of Invention
ENCODING DEVICE, TRANSMISSION DEVICE, AND RECEPTION DEVICE
5
Technical Field
[0001]
The present technology relates to an encoding device, a transmission device,
and a reception device, and particularly to an encoding device which generates video
10 data having a frame rate switched part, and the like.
Background Art
[0002]
The frame rate of the current broadcasting services is set to, for example,
15 29.97 Hz. There is, however, a possibility that a broadcasting service adopting
another frame rate will also be provided in the future (for example, refer to Patent
Literature 1). In that case, a situation in which video data having a frame rate
switched part in the middle of data is distributed is also considered.
20 Citation List
Patent Literature
[0003]
Patent Literature 1: JP 2011-172164A
25 Summary of Invention
Technical Problem
[0004]
An objective of the present technology is, when video data having a frame
rate switched part is distributed, to enable a reception side to easily secure a display
30 continuity.
3
Solution to Problem
[0005]
A concept of the present technology is a transmission device including: an
image encoding unit configured to generate video data having a switched part from
encoded image data of a first sequence to encoded image data of 5 a second sequence
having a different frame rate from the first sequence; and a transmission unit
configured to transmit a container in a predetermined format which includes the
generated video data. The image encoding unit performs encoding in a manner that
a display end timing of a last picture of the encoded image data of the first sequence
10 is set to a same timing as a display start timing of a first picture of the encoded image
data of the second sequence.
[0006]
In the present technology, video data having a switched part from encoded
image data of a first sequence to encoded image data of a second sequence having a
15 different frame rate from the first sequence is generated. In this case, in the image
encoding unit, encoding is performed in a manner that a display end timing of a last
picture of the encoded image data of the first sequence is set to a same timing as a
display start timing of a first picture of the encoded image data of the second
sequence.
20 [0007]
The transmission unit transmits a container in a predetermined format which
includes the above-described video data. The container may be, for example, a
transport stream (MPEG-2 TS) that has been employed in digital broadcasting
standards. In addition, the container may be, for example, MP4 that has been used
25 in distribution on the Internet and the like, or a container in a format other than that.
[0008]
In the present technology as above, encoding is performed in a manner that
the display end timing of the last picture of the encoded image data of the first
sequence is set to the same timing as the display start timing of the first picture of the
30 encoded image data of the second sequence. Thus, for example, a reception side
can easily secure a display continuity between the first sequence and the second
4
sequence.
[0009]
In the present technology, for example, the image encoding unit may
perform encoding in a manner that the encoded image data of the first sequence and
the encoded image data of the second sequence have the same 5 number of delayed
pictures from decoding until display. By performing encoding in this manner, for
example, the reception side can avoid a buffer failure.
[0010]
In the present technology, for example, the image encoding unit may
10 perform encoding in a manner that a decoding timing of the first picture of the
encoded image data of the second sequence is set at a timing obtained by adding a
time interval of one picture of the encoded image data of the first sequence to a
decoding timing of the last picture of the encoded image data of the first sequence.
By performing encoding in this manner, for example, the reception side can easily
15 secure a continuity in decoding timings of the first sequence and the second sequence,
and can avoid late reading of the first picture of the second sequence well.
[0011]
In the present technology, for example, an information insertion unit
configured to insert information indicating whether or not there is a change in a
20 frame rate into each piece of encoded image data of each picture of the video data
may be further included. In this case, for example, when the information indicates
there is a change in the frame rate, information indicating a value of the changed
frame rate may be added to the foregoing information. By inserting the information
in this manner, the reception side can easily ascertain the point of the change from
25 the first sequence to the second sequence.
[0012]
In the present technology, for example, an information insertion unit
configured to insert information notifying of a change from the encoded image data
of the first sequence to the encoded image data of the second sequence into a packet
30 containing the video data may be further included. In this case, for example, the
information insertion unit may insert the notification information into an extension
5
field of a PES packet which includes encoded image data of each picture in its
payload. By inserting the notification information in this manner, the reception side
can easily ascertain the change from the first sequence to the second sequence.
[0013]
In this case, for example, the notification information 5 may include
information indicating whether or not there is a discontinuity in a change of a value
of a decoding time. Then, when the information indicates that there is a
discontinuity, for example, the count-down value of the number of pictures until the
picture at which the discontinuity has occurred may be added to the information.
10 [0014]
In addition, in this case, for example, the notification information may
include information indicating whether or not a display time interval of each picture
is switched. Then, when the information indicates switching, for example, the
count-down value of the number of pictures until the switching may be added to the
15 foregoing information.
[0015]
In the present technology, for example, an information insertion unit
configured to insert information notifying of a change from the encoded image data
of the first sequence to the encoded image data of the second sequence into a layer of
20 the container may be further included. In this case, for example, the notification
information may include information on a frame rate of the first sequence and a
frame rate of the second sequence. By inserting the notification information in that
manner, the reception side can easily ascertain the change from the first sequence to
the second sequence.
25 [0016]
Another concept of the present technology is a reception device including: a
reception unit configured to receive a container in a predetermined format which
includes video data having a switched part from encoded image data of a first
sequence to encoded image data of a second sequence having a different frame rate
30 from the first sequence. The video data is encoded in a manner that a display end
timing of a last picture of the encoded image data of the first sequence is set to a
6
same timing as a display start timing of a first picture of the encoded image data of
the second sequence. The reception device further includes an image decoding unit
configured to decode the video data included in the received container to obtain
image data with a display continuity between the first sequence and the second
5 sequence maintained.
[0017]
In the present technology, the reception unit receives the container in the
predetermined format. This container includes the video data having the switched
part from the encoded image data of the first sequence to the encoded image data of
10 the second sequence having the different frame rate from the first sequence. In
addition, the image decoding unit decodes the video data included in the received
container and obtains the image data with the display continuity between the first
sequence and the second sequence maintained.
[0018]
15 Here, the video data is encoded in a manner that the display end timing of
the last picture of the encoded image data of the first sequence is set to the same
timing as the display start timing of the first picture of the encoded image data of the
second sequence. Thus, the display continuity between the first sequence and the
second sequence can be easily secured.
20 [0019]
In the present technology, for example, before the same timing, the image
decoding unit may decode encoded image data of each picture of the second
sequence at a timing synchronized with a decoding timing of encoded image data of
each picture of the first sequence. In this case, implementation in which two
25 asynchronous vertical synchronization signals are simultaneously generated can be
avoided.
[0020]
In the present technology, for example, information notifying of a change
from the encoded image data of the first sequence to the encoded image data of the
30 second sequence may be inserted into at least one of a packet containing the video
data and a layer of the container. A process of the image decoding unit may be
7
controlled based on the notification information.
[0021]
Another concept of the present technology is a reception device including: a
reception unit configured to receive video data having a switched part from encoded
image data of a first sequence to encoded image data of a second 5 sequence having a
different frame rate from the first sequence; and a processing unit configured to
process the received video data. The video data is encoded in a manner that a
display end timing of a last picture of the encoded image data of the first sequence is
set to a same timing as a display start timing of a first picture of the encoded image
10 data of the second sequence.
[0022]
In the present technology, the reception unit receives the video data having
the switched part from the encoded image data of the first sequence to the encoded
image data of the second sequence having the different frame rate from the first
15 sequence. In addition, the processing unit processes the received video data.
[0023]
In this case, the video data is encoded in a manner that the display end
timing of the last picture of the encoded image data of the first sequence is set to the
same timing as the display start timing of the first picture of the encoded image data
20 of the second sequence. Accordingly, for example, a display continuity between the
first sequence and the second sequence can be easily secured.
[0024]
This video data may be encoded in a manner that, for example, the encoded
image data of the first sequence and the encoded image data of the second sequence
25 have the same number of delayed pictures from decoding until display. In addition,
for example, the video data may be encoded in a manner that a decoding timing of
the first picture of the encoded image data of the second sequence is set at the timing
obtained by adding a time interval of one picture of the encoded image data of the
first sequence to a decoding timing of the last picture of the encoded image data of
30 the first sequence.
8
Advantageous Effects of Invention
[0025]
According to the present technology, a reception side can easily secure a
display continuity. It should be noted that the effects described herein are not
necessarily limitative, and any effect described in the present 5 disclosure may be
exhibited.
Brief Description of Drawings
[0026]
10 [FIG. 1] FIG. 1 is a block diagram showing a configuration example of a transmission
and reception system of an embodiment.
[FIG. 2] FIG. 2 is a block diagram showing a configuration example of a transmission
device.
[FIG. 3] FIG. 3 is a diagram showing an example of HRD control of an encoder for
15 securing a display continuity.
[FIG. 4] FIG. 4 is a diagram showing an example of HRD control of the encoder.
[FIG. 5] FIG. 5 is a diagram showing a structure example of an interface for inserting
temporal refresh information SEI and a structure example of
“temporal_refresh_information().”
20 [FIG. 6] FIG. 6 is a diagram showing content of main information of the structure
example of “temporal_refresh_information().”
[FIG. 7] FIG. 7 is a block diagram showing a configuration example of an encoder.
[FIG. 8] FIG. 8 is a diagram showing an example of a process flow of the encoder.
[FIG. 9] FIG. 9 is a diagram showing a structure example of PES extension field data
25 (pes_extension_field_data) and content of main information thereof.
[FIG. 10] FIG. 10 is a diagram showing a structure example of temporal discontinuity
information “temporal_discontinuity_information().”
[FIG. 11] FIG. 11 is a diagram showing content of main information of the structure
example of temporal discontinuity information
30 “temporal_discontinuity_information().”
[FIG. 12] FIG. 12 is a diagram showing an example of conversion of information
9
regarding a display continuity at the time of switching from a first sequence
(sequence A) to a second sequence (sequence B).
[FIG. 13] FIG. 13 is a diagram showing a structure example of a video parameter
descriptor “Video_parameter_descriptor.”
[FIG. 14] FIG. 14 is a diagram showing content of main information 5 of the structure
example of the video parameter descriptor “Video_parameter_descriptor.”
[FIG. 15] FIG. 15 is a block diagram showing a configuration example of a
multiplexer.
[FIG. 16] FIG. 16 is a diagram showing an example of a process flow of a multiplexer.
10 [FIG. 17] FIG. 17 is a diagram showing a configuration example of a transport stream
TS.
[FIG. 18] FIG. 18 is a block diagram showing a configuration example of a reception
device.
[FIG. 19] FIG. 19 is a block diagram showing a configuration example of a
15 demultiplexer.
[FIG. 20] FIG. 20 is a diagram showing an example of a process flow of the
demultiplexer.
[FIG. 21] FIG. 21 is a block diagram showing a configuration example of a decoder.
[FIG. 22] FIG. 22 is a diagram for describing an example of a switching timing of a
20 vertical synchronization signal Vsync.
[FIG. 23] FIG. 23 is a diagram for describing another example of the switching
timing of the vertical synchronization signal Vsync.
[FIG. 24] FIG. 24 is a diagram showing an example of HRD control of an encoder for
securing a display continuity and decoding continuity.
25
Description of Embodiments
[0027]
Hereinafter, embodiments for implementing this technology (hereinafter
30 referred to as “embodiments”) will be described. Note that description will be
provided in the following order.
10
1. Embodiment
2. Modified example
[0028]
<1. Embodiment>
[Transmission and 5 reception system]
FIG. 1 shows a configuration example of a transmission and reception
system 10 as an embodiment. This transmission and reception system 10 is
configured to have a transmission device 100 and a reception device 200.
[0029]
10 The transmission device 100 transmits a transport stream TS that is a
container by causing the stream to be carried on broadcast waves. This transport
stream TS includes a video stream with a frame rate switched part, i.e., a part which
is switched from encoded image data of a first sequence to encoded image data of a
second sequence which has a different frame rate from the first sequence.
15 [0030]
Encoding of, for example, H.264/AVC, H.265/HEVC, or the like is assumed
to be performed. Here, the encoding is performed such that the display end timing
of the last picture of the encoded image data of the first sequence is set to the same
timing as the display start timing of the first picture of the encoded image data of the
20 second sequence. Accordingly, a reception side, for example, can easily secure a
display continuity between the first sequence and the second sequence.
[0031]
Information indicating whether or not there is a change in a frame rate is
inserted into encoded image data of each picture of a video stream. Accordingly,
25 the reception side can easily ascertain, for example, the point of the change from the
first sequence to the second sequence.
[0032]
In addition, information notifying of a change from the encoded image data
of the first sequence to the encoded image data of the second sequence is inserted
30 into a packet containing the video stream, for example, into a PES packet. In
addition, the information notifying of the change from the encoded image data of the
11
first sequence to the encoded image data of the second sequence is inserted into a
layer of the container, i.e., a layer of the transport stream. Accordingly, the
reception side can easily ascertain, for example, the change from the first sequence to
the second sequence.
5 [0033]
The reception device 200 receives the above-described transport stream TS
sent by being carried on the broadcast waves from the transmission device 100.
The reception device 200 decodes the video stream included in the transport stream
TS to obtain image data. As described above, the video stream is encoded such that
10 the display end timing of the last picture of the encoded image data of the first
sequence is set to the same timing as the display start timing of the first picture of the
encoded image data of the second sequence. Thus, image data with display
continuity between the first sequence and the second sequence maintained is
obtained.
15 [0034]
Here, the information notifying of the change from the encoded image data
of the first sequence to the encoded image data of the second sequence is inserted
into at least one of the packet containing the video stream and the layer of the
container. An image decoding process is controlled based on this notification
20 information. For example, the switched part from the first sequence to the second
sequence is ascertained based on this notification information, and switching from a
vertical synchronization signal synchronized with the encoded image data of the first
sequence to a vertical synchronization signal synchronized with the encoded image
data of the second sequence is performed.
25 [0035]
In addition, encoded image data of each picture of the second sequence is
decoded at a timing synchronized with a decoding timing of encoded image data of
each picture of the first sequence before the above-described same timing.
Accordingly, for example, implementation in which two asynchronous vertical
30 synchronization signals are simultaneously generated can be avoided.
[0036]
12
[Configuration of transmission device]
FIG. 2 shows a configuration example of the transmission device 100.
This transmission device 100 has a central processing unit (CPU) 101, an encoder
102, a compressed data buffer (coded picture buffer or cpb) 103, a multiplexer 104,
and a transmission unit 105. The CPU 101 is a control 5 unit, which controls
operations of each unit of the transmission device 100.
[0037]
The encoder 102 receives an input of uncompressed image data and
performs encoding thereon. The encoder 102 performs encoding in, for example,
10 H.264/AVC, H.265/HEVC, or the like. In this embodiment, uncompressed image
data with a different frame rate is input to the encoder 102 in a switching manner.
Thus, the encoder 102 generates video streams (video data) with a frame rate
switched part i.e., a part which is switched from the encoded image data of the first
sequence to the encoded image data of the second sequence having a different frame
15 rate.
[0038]
Here, the encoder 102 performs the encoding such that the display end
timing of the last picture of the encoded image data of the first sequence is set to the
same timing as the display start timing of the first picture of the encoded image data
20 of the second sequence.
[0039]
FIG. 3 shows an example of Hypothetical Reference Decoder (HRD) control
of the encoder 102. A staircase-like solid line a represents progress of the amount
of data generated in encoding, and each step corresponds to the unit of one picture.
25 The height of each step represents an amount of data generated in encoding.
[0040]
A timing P0 indicates a timing at which the first byte of encoded image data
of the first picture of the first sequence (sequence A) enters a coded picture buffer
(cpd or compressed data buffer). A timing P1 indicates a timing at which the first
30 byte of the encoded image data of the last picture of the first sequence enters the cpd.
A timing P2 indicates a timing at which the last byte of the encoded image data of the
13
last picture of the first sequence enters the cpd and the first byte of the encoded
image data of the first picture of the second sequence (sequence B) enters the cpd.
[0041]
In addition, Ra indicates an input bit rate of the encoded image data of the
first picture of the first sequence to the cpd. Here, when the 5 amount of data of the
encoded image data of the first picture of the first sequence is set to Qa and the data
is assumed to be input into the cpd for a time Ta, Ra = Qa/Ta. In the illustrated
example, a case in which an input bit rate of encoded image data of other pictures of
the first sequence to the cpd is also set to Ra is shown.
10 [0042]
In addition, Rb indicates an input bit rate of the encoded image data of the
first picture of the second sequence to the cpd. Here, when the amount of data of
the encoded image data of the first picture of the second sequence is set to Qb and
the data is assumed to be input into the cpd for a time Tb, Rb = Qb/Tb. In the
15 illustrated example, a case in which an input bit rate of encoded image data of other
pictures of the second sequence to the cpd is also set to Rb is shown. It should be
noted that, although Ra and Rb show an example of a constant bit rate
(constant_bit_rate), they are not limited thereto, and the same approach is applied to
the case of a variable bit rate (variable_bit_rate).
20 [0043]
A staircase-like solid line b represents progress of the amount of data
consumed in decoding, and each step corresponds to the unit of one picture. The
height of each step represents the amount of data consumed in decoding. Qcpb
represents an occupation amount of the cpd. Encoding is performed such that the
25 occupation amount fits within a size of the cpb (memory capacity) at any timing.
[0044]
In the illustrated example, the first sequence is composed of 4 pictures,
which are decoded in the order of a0, a1, a2, and a3 and displayed in the order of a0,
a2, a3, and a1. In this case, image data of each of the decoded pictures is input to a
30 decoded picture buffer (dpb or uncompressed data buffer). In this example, the
number of delayed pictures after the first sequence is decoded until display thereof is
14
started is set to 2.
[0045]
Each of the pictures of the first sequence is decoded and displayed at a
timing of a vertical synchronization signal (Vsync) with a time interval
corresponding to a frame rate (frame frequency) thereof. For example, 5 the pictures
a0, a1, a2, and a3 are decoded at the timings Ta(0), Ta(1), Ta(2), and Ta(3), and
display of the pictures a0, a2, a3, and a1 is started at the timings Ta(2), Ta(3), Ta(4),
and Ta(5).
[0046]
10 The second sequence is composed of 6 pictures, which are decoded in the
order of b0, b1, b2, b3, b4, and b5, and displayed in the order of b1, b2, b0, b4, b5,
and b3. In this case, encoded data of each of the decoded pictures is input to the
dpb. In this example, the number of delayed pictures after the second sequence is
decoded until display thereof is started is set to 2, the same as the first sequence.
15 Accordingly, a buffer failure is avoided.
[0047]
Each of the pictures of the second sequence is decoded and displayed at the
timing of the vertical synchronization signal (Vsync) with a time interval
corresponding to a frame rate (frame frequency) thereof. For example, the pictures
20 b0, b1, b2, b3, b4, and b5 are decoded at the timings Tb(0), Tb(1), Tb(2), Tb(3),
Tb(4), and Tb(5) and display of the pictures b1, b2, b0, b4, b5, and b3 is started at
the timings Tb(2), Tb(3), Tb(4), Tb(5), Tb(6), and Tb(7).
[0048]
In this example, the display end timing of the last picture of the first
25 sequence becomes the same as the display start timing of the first picture of the
second sequence. That is, the timing Ta(6) is set to the same timing as the timing
Tb(2). Accordingly, a display continuity between the first sequence and the second
sequence is ensured.
[0049]
30 At this time, the timing (Tb(0)) at which the first picture of the second
sequence is decoded is a timing delayed “Initial_cpb_removal_delay B” from a
15
timing (P2) at which the leading byte of the first picture of the second sequence starts
accumulating in the compressed data buffer (cpb). If the timing (Tb(0)) is set to be
later than the timing (Ta(4)), late reading of the first picture of the second sequence is
avoided.
5 [0050]
FIG. 4 also shows an example of HRD control. This example is a
comparison example to the above-described HRD control shown in FIG. 3. In this
example, the timing (Tb(0)) at which the first picture of the second sequence is
decoded is set to the same timing as the timing (Ta(4)) obtained by adding the period
10 of one frame of the second sequence to the timing (Ta(3)) at which the last picture of
the first sequence is decoded.
[0051]
In this case, although a continuity in decoding timings of the first sequence
and the second sequence is ensured, a continuity in display is not. In other words,
15 the display end timing (Ta(6)) of the last picture of the first sequence deviates from
the display start timing (Tb(4)) of the first picture of the second sequence. In this
case, it is necessary to delay display of the last picture of the first sequence to
immediately before the timing (Tb(4)) as shown by P4.
[0052]
20 In addition, the encoder 102 inserts information indicating whether or not
there is a change in the frame rate of encoded image data of each picture of the video
stream. The encoder 102 inserts temporal refresh information SEI
(temporal_refresh_information SEI) which newly defines that information as one of
prefix SEI (Prefix_SEI).
25 [0053]
FIG. 5(a) shows a structure example (syntax) of an interface (I/F) for
inserting the temporal refresh information SEI. The field of “uuid_iso_iec_11578”
has a UUID value indicated by “ISO/IEC 11578:1996 Annex A.”
“temporal_refresh_information()” is inserted into the field of
30 “user_data_payload_byte.”
[0054]
16
FIG. 5(b) shows a structure example (syntax) of
“temporal_refresh_information().” FIG. 6 shows content (semantics) of main
information of the structure example. The ID of predetermined user data is given in
the 16-bit field of “userdata_id.” The 8-bit field of
“temporal_refresh_information_length” indicates the 5 number of bytes of
““temporal_refresh_information” (counted from an element next to this element).
[0055]
The 1-bit field of “temporal_timing_discontinuity_flag” is a flag indicating
a discontinuity occurring in display time information and a time stamp. “1”
10 indicates a discontinuity occurred. “0” indicates no discontinuity occurred. When
the flag is “1,” the 8-bit field of “num_unit_in_tick” and the 8-bit field of
“times_scale” are presented.
[0056]
The 8-bit field of “num_unit_in_tick” is the number of blocks indicating a
15 display period of a corresponding slice or picture. The 8-bit field of “times_scale”
indicates a time information scaling value. Here, by calculating “times_scale” and
“num_unit_in_tick,” a frame rate (frame frequency) can be obtained.
[0057]
FIG. 7 shows a configuration example of the encoder 102. This encoder
20 102 has a buffer delay control unit 122, a hypothetical reference decoder (HRD)
setting unit 123, a parameter set/SEI encoding unit 124, a slice encoding unit 125,
and an NAL packetizing unit 126.
[0058]
The buffer delay control unit 122 calculates “initial_cpb_removal_delay”
25 which is an initial value of buffering of a sequence and “cpb_removal_delay” and
“dpb_output_delay” of each picture for sequences disposed chronologically before
and after switching. In this case, in order to assure a continuous connection relation
of display between the two sequences, “dpb_output_delay” and
“cpb_removal_delay” of the two sequences are controlled such that a dpb buffer
30 output of the last picture of the previous sequence becomes time-serially continuous
with a dpb buffer output of the first picture of the after sequence (see FIG. 3).
17
[0059]
“Initial cpb removal time” indicates a time (initial time) taken when
encoded image data of the leading picture of a Group of Pictures (GOP) from the cpb
(compressed data buffer) is decoded. “cpb_removal_delay” is a time at which
encoded image data of each picture is taken out from the cpb, 5 and a time at which
each picture is decoded is decided in accordance with “initial_cpb_removal_time.”
In addition, “dpb_output_delay” indicates a time from entering the dpb
(uncompressed data buffer) to taking-out of data after the decoding.
[0060]
10 The HRD setting unit 123 performs HRD setting based on the information
of “initial cpb removal time,” “cpb_removal_delay,” and “dpb_output_delay”
calculated by the buffer delay control unit 122. The parameter set/SEI encoding
unit 124 generates SEI, in addition to parameter sets such as a video parameter set
(VPS) and a sequence parameter set (SPS) based on the HRD setting or the like.
15 [0061]
For example, picture timing SEI that includes “cpb_removal_delay” and
“dpb_output_delay” is generated. In addition, for example, buffereing perifod SEI
including “initial_cpb_removal_time” is generated. In addition, for example,
temporal refresh information SEI (temporal_refresh_information SEI) which
20 elucidates a switching timing as described above is generated.
[0062]
The slice encoding unit 125 obtains slice data (slice segment header and
slice segment data) by encoding image data of each picture. The slice decoding unit
125 inserts “ref_idx_l0_active (ref_idx_l1_active), which indicates an index of a
25 picture that is a prediction destination of a “prediction unit,” into the “slice segment
header” as information indicating a state of time direction prediction using a frame
buffer. Accordingly, during decoding, a reference source picture is decided. In
addition, the slice decoding unit 125 inserts the index of a current slice into the “slice
segment header” as “short_term_ref_pic_set_idx” or “it_idx_sps.”
30 [0063]
The NAL packetizing unit 126 generates encoded image data of each picture
18
based on the parameter sets and SEI generated by the parameter set/SEI encoding
unit 124 and the slice data generated by the slice encoding unit 125, and outputs
video streams (encoded streams).
[0064]
FIG. 8 shows an example of the process flow of the 5 encoder 102. The
encoder 102 starts the process in Step ST1, and then moves to the process of Step
ST2. In this Step ST2, the encoder 102 checks a parameter of a dynamic sequence
to be encoded. This parameter also includes the frame rate (frame frequency).
[0065]
10 Then, the encoder 102 determines whether or not there is a change in the
frame rate in the sequence in Step ST3. When there is a change in the frame rate,
the encoder 102 proceeds to the process or Step ST4. In Step ST4, the encoder
manages buffer input and output timings before and after the point of the change.
[0066]
15 In this case, the following expression (1) is satisfied.
T_2(first_presentation) = T_1(last_presentation) + 1*(T_1 temporal
disctance) … (1)
[0067]
Here, “T_2(first_presentation)” indicates a display start timing of the first
20 picture of the second sequence. “T_1(last_presentation)” indicates the display start
timing of the last picture of the first sequence. “(T_1 temporal disctance) indicates
the time interval between vertical synchronization signals, i.e., the frame period of
the first sequence.
[0068]
25 After the process of Step ST4, the encoder 102 proceeds to the process of
Step ST5. When there is no change in the frame rate in Step ST3 described above,
the encoder 102 directly moves to the process of Step 5. In this Step ST5, the
encoder 102 performs HRD management in all pictures (slices), performs encoding
of the slices, the parameter sets, SEI, and the like, and then configures a stream with
30 NAL packets. Thereafter, the encoder 102 finishes the process in Step ST6.
[0069]
19
Returning to FIG. 2, the compressed data buffer (cpb) 103 temporarily
accumulates a video stream including the encoded data of each picture generated by
the encoder 102. The multiplexer 104 reads the video stream accumulated in the
compressed data buffer 103, makes them into PES packets, further makes them into
transport packets and multiplexes them, and thereby obtains a transport 5 stream TS as
a multiplexed stream.
[0070]
The multiplexer 104 inserts the information notifying of a change from the
encoded image data of the first sequence to the encoded image data of the second
10 sequence into the packet containing the video stream, for example, the PES packet.
In this embodiment, in the field of a PES extension of the PES packet, temporal
discontinuity information (temporal_discontinuity_information) is defined.
[0071]
This temporal discontinuity information includes information indicating
15 whether or not there is a discontinuity in a change of a value of a decoding time, and
when the information indicates that there is a discontinuity, the count-down value of
the number of pictures until a picture at which the discontinuity has occurred is
added to that information. In addition, this temporal discontinuity information
includes information indicating whether or not a display time interval of each of
20 pictures is switched, and when the information indicates switching, the count-down
value of the number of pictures until the switching is added to the information.
[0072]
FIG. 9(a) shows a structure example (syntax) of the PES extension field data
(pes_extension_field_data). FIG. 9(b) shows the content (semantics) of main
25 information in the structure information. It should be noted that “PES_extension
field length” is assumed to be given outside of the syntax structure. The 8-bit field
of “start_sync_byte” indicates a code value expressing a start of an extension field.
[0073]
The 8-bit field of “extension_field_type” indicates the type of the extension
30 field. “0  01” indicates the type of “temporal_discontinuity_information,” i.e., a
structure in which information of a temporal discontinuity of access units (pictures)
20
is supplied. When the type of the extension field is “0  01,” temporal discontinuity
information (temporal_discontinuity_information) is inserted into the field of
“data_byte.”
[0074]
FIG. 10 shows a structure example (syntax) of temporal 5 discontinuity
information “temporal_discontinuity_information().” FIG. 11 shows content
(semantics) of main information of the structure example.
[0075]
The 1-bit field of “decode_time_discontinuity_flag” is a flag indicating
10 whether or not there is a discontinuity in a change of the value of the decoding time
(or decoding time stamp). “1” indicates that there is a discontinuity and “0”
indicates that there is no discontinuity. The 1-bit field of
“presentation_frequency_change_flag” is flag information indicating whether or not
the display time interval of access units is switched. “1” indicates that it is switched
15 and “0” indicates that it is not switched.
[0076]
When “decode_time_discontinuity_flag” is “1,” the 8-bit field of
“au_count_down_to_decode_time_switching” is presented. This field indicates a
count-down value in units of access units (pictures) until the discontinuity has
20 occurred in the variation of the decoding time. For example, “0  00” indicates a
corresponding access unit is the last access unit before the point of discontinuity, and
“0  01 to 0  FE” indicate the values of corresponding access units counted to the
last access unit before the point of discontinuity.
[0077]
25 In addition, when “presentation_frequency_change_flag” is “1,” the 8-bit
field of “au_count_down_to_presentation_frequency_change” is presented. This
field indicates a count-down value in units of access units until the display time
interval of each of the access units is switched. For example, “0  00” indicates
that a corresponding access unit is the last access unit before switching of the display
30 time interval, and “0  01 to 0  FE” indicate the values of corresponding access
units counted to the last access unit before switching of the display time interval.
21
[0078]
FIG. 12 shows an example of conversion of information regarding a display
continuity at the time of switching from the first sequence (sequence A) to the second
sequence (sequence B). Here, one frame period of the first sequence is assumed to
be “Time Distance A,” and one frame period of the second sequence 5 is assumed to be
“Time Distance B.” In addition, flag information of
“temporal_timing_discontinuity_flag” included in temporal refresh information SEI
(temporal_refresh_information SEI) and count-down values of
“au_count_down_to_presentation_frequency_change” included in temporal
10 discontinuity information (temporal_discontinuity_information) are shown here.
[0079]
In addition, the multiplexer 104 inserts the information notifying of the
change from the encoded image data of the first sequence to the encoded image data
of the second sequence into the transport stream TS.
15 [0080]
As program specific information (PSI), the transport stream TS includes a
program map table (PMT). This PMT has a video elementary loop (video ES1
loop) with information relating to each video stream. In this video elementary loop,
information of a stream type, a packet identifier (PID), and the like as well as a
20 descriptor describing the information relating to the video stream are disposed to
correspond to each video stream. In this embodiment, the multiplexer 104 inserts a
video parameter descriptor (Video_parameter_descriptor) to be newly defined as one
of the descriptors.
[0081]
25 FIG. 13 shows a structure example (syntax) of a video parameter descriptor
(Video_parameter_descriptor). In addition, FIG. 14 shows content (semantics) of
main information of the structure example.
[0082]
The 8-bit field of “descriptor_tag” indicates the type of the descriptor,
30 indicating video parameter descriptor here. The 8-bit field of “descriptor_length”
indicates the length (size) of the descriptor, showing the number of succeeding bytes
22
as the length of the descriptor.
[0083]
The 1-bit field of “resolution_change_flag” is flag information indicating
whether or not there is a change in horizontal and vertical resolution. The “1-bit
field of “video_frame_rate_change_flag” is a 1-bit field indicating 5 whether or not
there is a change in the frame frequency (frame rate) of a picture. The 1-bit field of
“video_bit_depth_change_flag” is flag information indicating whether or not there is
a change in an encoded bit width. The 1-bit field of “color_gamut_change_flag” is
flag information indicating whether or not there is a change in a color gamut of an
10 encoded picture. The 1-bit field of “dynamic_range_change_flag” is flag
information indicating whether or not there is a change in the range of brightness and
darkness of luminance.
[0084]
The 4-bit field of “video_resolution1” indicates current horizontal and
15 vertical resolution. The 4-bit field of “video_frame_rate1” indicates the frame
frequency of a current picture. “video_bit_depth1” indicates a current encoded bit
width. “color_gamut1” indicates a gamut of a currently encoded picture.
“luma_dynamic_range1” indicates the range of brightness and darkness of current
luminance.
20 [0085]
When “resolution_change_flag = 1” is set indicating that there is a change
in horizontal and vertical resolution, the 4-bit field of “video_resolution2” is
presented. This field indicates the changed horizontal and vertical resolution. In
addition, when “video_frame_rate_change_flag = 1” is set indicating that there is a
25 change in the frame frequency of the picture, the 4-bit field of “video_frame_rate2”
is presented. This field indicates the changed frame frequency of the picture.
[0086]
In addition, when “video_bit_depth_change_flag = 1” is set indicating that
there is a change in an encoded bit width, the 4-bit field of “video_bit_depth2” is
30 presented. This field indicates the changed encoded bit width. Furthermore, when
“color_gamut_change_flag =1” is set indicating that there is a change in a gamut of
23
an encoded picture, the 4-bit field of “color_gamut2” is presented. This field
indicates the changed gamut of the encoded picture. Furthermore, when
“dynamic_range_change_flag = 1” is set indicating that there is a change in the range
of brightness and darkness of luminance, the 4-bit field of “luma_dynamic_range2”
is presented. This field indicates the changed range of brightness 5 and darkness of
luminance.
[0087]
FIG. 15 shows a configuration example of the multiplexer 104. The
multiplexer has a section coding unit 142, a PES packetizing unit 143, and a
10 transport packetizing unit 145.
[0088]
The PES packetizing unit 143 reads a video stream (elementary stream)
accumulated in the compressed data buffer 103 to generate a PES packet. At this
time, the PES packetizing unit 143 places a time stamp such as a decoding time
15 stamp (DTS) and a presentation time stamp (PTS) to a PES header based on HRD
information and the like of the video stream. In this case, “cpu_removal_delay”
and “dpb_output_delay” of each picture are referred to, time stamps are converted
into each DTS and PTS in synchronized precision with the time of a system time
clock (STC), and disposed in predetermined positions of the PES headers.
20 [0089]
In addition, at this time, the PES packetizing unit 143 creates the abovedescribed
temporal discontinuity information (temporal_discontinuity_information)
(see FIG. 10) based on information on a temporal discontinuity of pictures (access
units) included in a video stream. Then, the PES packetizing unit 143 inserts this
25 information into the field of a PES extension of a PES packet.
[0090]
The section coding unit 142 generates various kinds of section data to be
inserted into the transport stream TS. At this time, the section coding unit 142 also
creates the above-described video parameter descriptor
30 (Video_parameter_descriptor) (see FIG. 13) based on, for example, the information
on the temporal discontinuity of the pictures (access units) included in the video
24
stream.
[0091]
The PES packet generated by the PES packetizing unit 143 is sent to the
transport packetizing unit 145. In addition, the various kinds of section data
generated by the section coding unit 142 are also sent to the 5 transport packetizing
unit 145. The transport packetizing unit 145 generates a TS packet which includes
the PES packet and section data in its payload, and then generates a transport stream
TS.
[0092]
10 FIG. 16 shows an example of the process flow of the multiplexer 104. The
multiplexer 104 starts the process in Step ST11, and then moves to the process of
Step ST12. In this Step ST12, the multiplexer 104 computes a time stamp from
temporal refresh information SEI, picture timing SEI, buffering period SEI, or HRD
information of the video stream (elementary stream).
15 [0093]
Then, the multiplexer 104 creates temporal discontinuity information in Step
ST13 as discontinuity information, and inserts the information into the field of the
PES extension. Then, the multiplexer 104 inserts the video stream (elementary
stream) into the PES payload in Step ST14.
20 [0094]
Then, the multiplexer 104 generates various kinds of section data in Step
ST15. As one kind of this section data, a video parameter descriptor which
indicates presence of a change in a parameter of the video stream (elementary
stream) is also generated. Then, the multiplexer 104 generates and outputs a TS
25 packet which includes a PES packet and section data in its payload in Step ST16.
After the process of Step ST16, the multiplexer 104 finishes the process in Step ST17.
[0095]
FIG. 17 shows a configuration example of the transport stream TS. This
transport stream TS includes one video stream. In other words, in this configuration
30 example, there is a PES packet “video PES1” of the video stream.
[0096]
25
Encoded image data of each picture is disclosed in the PES payload. In the
case of HEVC encoding, the encoded image data is composed of NAL units such as a
VPS, an SPS, SEI, a PPS, P-SEI, SLICE, S-SEI, and EOS. The above-described
temporal refresh information SEI (temporal_refresh_information SEI) is inserted as
5 one of the P-SEI.
[0097]
In addition, the above-described temporal discontinuity information
(temporal_continuity_information) is inserted into the field of the PES extension of a
PES header. It should be noted that a DTS and a PTS are also disposed in the PES
10 header.
[0098]
In addition, the transport stream TS includes a program map table (PMT) as
program specific information (PSI). This PSI is information describing to which
program each elementary stream included in the transport stream is affiliated.
15 [0099]
In the PMT, there is a program loop describing information relating to an
entire program. In addition, in the PMT, there is an elementary loop having
information relating to each elementary stream. In this configuration example,
there is a video elementary loop (video ES1 loop).
20 [0100]
In the video elementary loop, information of the type of stream, a packet
identifier (PID), and the like are disposed, and descriptors describing information
relating to the video stream are also described corresponding to the video stream
(video PES1). As one of the descriptors, the above-described video parameter
25 descriptor (video_parameter_descriptor) is inserted.
[0101]
Returning to FIG. 2, the transmission unit 105 modulates the transport
stream TS in a modulation scheme proper for broadcasting, for example, QPSKOFDM,
and transmits an RF modulation signal from a transmission antenna.
30 [0102]
An operation of the transmission device 100 shown in FIG. 2 will be briefly
26
described. The encoder 102 receives an input of uncompressed image data. In the
encoder 102, encoding in, for example, H.264/AVC, H.265/HEVC, or the like is
performed on the image data. In this case, image data of different frame rates is
switched to be input to the encoder 102. Thus, the encoder 102 generates a video
stream having a frame rate switched part, i.e., a part which is switched 5 from encoded
image data of a first sequence to encoded image data of a second sequence which has
a different frame rate from the first sequence.
[0103]
The encoder 102 performs encoding such that a display end timing of the
10 last picture of the encoded image data of the first sequence is set to the same timing
as a display start timing of the first picture of the encoded image data of the second
sequence. In addition, the encoder 102 performs encoding such that the encoded
image data of the first sequence and the encoded image data of the second sequence
have the same number of delayed pictures for the time from decoding to display.
15 Furthermore, the encoder 102 performs encoding such that the decoding timing of
the first picture of the encoded image data of the second sequence is set at a timing
later than a timing obtained by adding one frame period of the encoded image data of
the first sequence to the decoding timing of the last picture of the encoded image data
of the first sequence.
20 [0104]
In addition, the encoder 102 inserts information indicating whether or not
there is a change in a frame rate into each piece of encoded image data of each
picture of the video stream. In other words, the encoder 102 inserts temporal
refresh information SEI (temporal_refresh_information SEI) (see FIG. 5(b)) into the
25 encoded image data of each picture as one kind of prefix SEI (Prefix_SEI).
[0105]
The video stream which is generated by the encoder 102, including the
encoded data of the pictures of each layer, is supplied to the compressed data buffer
(cpb) 103 and temporarily stored therein. In the multiplexer 104, the video stream
30 accumulated in the compressed data buffer 103 is read, PES packetized, and further
transport packetized and multiplexed, and thereby a transport stream TS is obtained
27
as a multiplexed stream.
[0106]
The multiplexer 104 inserts information notifying of a change from the
encoded image data of the first sequence to the encoded image data of the second
sequence into the packet which contains the video stream, for example, 5 a PES packet.
In other words, temporal discontinuity information
(temporal_discontinuity_information) (see FIG. 10) is inserted into the field of the
PES extension of the PES packet.
[0107]
10 In addition, the multiplexer 104 inserts the information notifying of the
change from the encoded image data of the first sequence to the encoded image data
of the second sequence into the transport stream TS. In other words, a video
parameter descriptor (Video_parameter_descriptor) (see FIG. 13) is inserted into a
video elementary loop.
15 [0108]
The transport stream TS generated by the multiplexer 104 is sent to the
transmission unit 105. In the transmission unit 105, the transport stream TS is
modulated in a modulation scheme proper for broadcasting, for example, QPSKOFDM,
and an RF modulation signal is transmitted from a transmission antenna.
20 [0109]

FIG. 18 shows a configuration example of the reception device 200. This
reception device 200 has a central processing unit (CPU) 201, a reception unit 202, a
demultiplexer 203, and a compressed data buffer (coded picture buffer or cpb) 204.
25 Further, the reception device 200 has a decoder 205, an uncompressed data buffer
(decoded picture buffer or dpb) 206, a post-processing unit 207, a clock generation
unit 208, and a frequency divider 209. The CPU 201 constitutes a control unit
which controls operations of each unit of the reception device 200.
30 [0110]
The reception unit 202 demodulates an RF-modulated signal received by a
28
reception antenna to acquire the transport stream TS. The transport stream TS
includes the video stream (video data) with the switched part from the encoded
image data of the first sequence to the encoded image data of the second sequence
having the different frame rate from the first sequence. The demultiplexer 203
takes out the video stream and sends the stream to the compressed 5 data buffer (coded
picture buffer or cpb) 204.
[0111]
FIG. 19 shows a configuration example of the demultiplexer 203. The
demultiplexer 203 has a TS adaptation field extraction unit 231, a clock information
10 extraction unit 232, a TS payload extraction unit 233, a section extraction unit 234, a
PSI table/descriptor extraction unit 235, and a PES packet extraction unit 236.
Furthermore, the demultiplexer 203 has a PES header extraction unit 237, a time
stamp extraction unit 238, a discontinuity/count-down information extraction unit
239, and a PES payload extraction unit 240.
15 [0112]
The TS adaptation field extraction unit 231 extracts an adaptation field from
a TS packet having the adaptation field of the transport stream TS. The clock
information extraction unit 232 extracts a program clock reference (PCR) from the
adaptation field in which the PCR is included, and sends the PCR to the clock
20 generation unit 208.
[0113]
The TS payload extraction unit 233 extracts the TS payload from the TS
packet having the TS payload of the transport stream TS. The section extraction
unit 234 extracts section data from the TS payload which includes the section data.
25 The PSI table/descriptor extraction unit 235 analyzes the section data extracted by
the section extraction unit 234 and extracts a PSI table or a descriptor. Then, the
PSI table/descriptor extraction unit 235 sends descriptor information to the CPU 201.
[0114]
This descriptor information also includes the above-described video
30 parameter descriptor (Video_parameter_descriptor). The CPU 201 can recognize a
change in the frame frequency (frame rate) in advance from the information of the
29
video parameter descriptor and further can also recognize the changed frame
frequency.
[0115]
The PES packet extraction unit 236 extracts a PES packet from the TS
payload which includes the PES packet. The PES header 5 extraction unit 237
extracts the PES header from the PES packet extracted by the PES packet extraction
unit 236. The time stamp extraction unit 238 extracts time stamps (DTS and PTS)
inserted into the PES header in units of pictures and sends the time stamps to the
CPU 201.
10 [0116]
In addition, the discontinuity/count-down information extraction unit 139
extracts the above-described temporal discontinuity information
(temporal_discontinuity_information) inserted into the area of the PES extension of
the PES header in units of pictures, and sends discontinuity information and count15
down information to the CPU 201.
[0117]
In this case, when there is a discontinuity in a change of the value of a
decoding time (or decoding time stamp), the CPU 201 can recognize the
discontinuity in advance and further can clearly ascertain progress until a
20 discontinuity occurs in units of access units (pictures) from the count-down
information (count-down value). In addition, when a display time interval of each
access unit (picture) is switched, the CPU 201 can recognize the switching in
advance and further can clearly ascertain progress of units of access units (pictures)
until the switching from the count-down information (count-down value).
25 [0118]
The PES payload extraction unit 240 extracts the PES payload, i.e., encoded
image data of each picture, from the PES packet extracted by the PES packet
extraction unit 236 and sends the data to the compressed data buffer (coded picture
buffer or cpb) 204.
30 [0119]
FIG. 20 shows an example of the process flow of the demultiplexer 203.
30
The demultiplexer 203 starts the process in Step ST31 and then extracts the TS
payload in Step ST32. Then, the demultiplexer 203 performs analysis of a PID in
Step ST33 to determine whether or not it is a section. When it is determined to be a
section, the demultiplexer 203 demultiplexes a packet passing through a
corresponding PID filter in Step ST34 to perform 5 section parsing.
[0120]
Then, the demultiplexer 203 analyzes the video parameter descriptor
(Video_parameter_descriptor) in Step ST35. Then, the demultiplexer 203
determines whether or not there is a change in the frame rate (frame frequency) in
10 Step ST36. When there is a change in the frame rate, the demultiplexer 203 notifies
the system, i.e., the CPU 201 of the changed frame rate in Step ST37.
[0121]
Then, the demultiplexer 203 moves to Step ST38 and finishes the process.
It should be noted that, when there is no change in the frame rate in Step ST36, the
15 demultiplexer 203 directly moves to Step ST38 to finish the process.
[0122]
In addition, when it is not a section in Step ST33, the demultiplexer 203
extracts the PES packet in Step ST39. Then, the demultiplexer 203 determines
whether or not it is the PES header in Step ST40. When it is the PES header, the
20 demultiplexer 203 determines whether or not there is a change in the frame rate in
the analysis of the video parameter descriptor in Step ST41.
[0123]
When there is a change in the frame rate, the demultiplexer 203 determines
whether or not there is temporal discontinuity information
25 (temporal_discontinuity_information) in the field of the PES extension in Step ST42.
The determination of Step ST41 is assumed to coincide with the result of Step ST36.
[0124]
When there is temporal discontinuity information, the demultiplexer 203
detects a discontinuity of the time stamp with a discontinuity flag of the temporal
30 discontinuity information and the count-down value, and notifies the system, i.e., the
CPU 201 of the detection in Step ST43. Then, the demultiplexer 203 moves to the
31
process of Step ST44. It should be noted that, when there is no change in the frame
rate Step ST41, or when there is no temporal discontinuity information in Step ST42,
the demultiplexer 203 directly moves to the process of Step ST44.
[0125]
In Step ST44, the demultiplexer 203 determines whether 5 or not there are a
DTS and a PTS. When there are a DTS and a PTS, the demultiplexer 203 moves to
the process of Step ST46. On the other hand, when there is neither DTS nor PTS,
the demultiplexer 203 generates a DTS and a PTS through interpolation in Step ST45,
and then moves to the process of Step ST46. In Step ST46, the demultiplexer 203
10 notifies the system, i.e., the CPU 201 of the DTS and the PTS. Then, the
demultiplexer 203 moves to Step ST38 to finish the process.
[0126]
In addition, when it is not the PES header in Step ST40, the demultiplexer
203 extracts the PES payload in Step ST47. Then, the demultiplexer 203 transfers
15 an encoded stream for the PID to the compressed data buffer (cpb) 204 in Step ST48.
Then, the demultiplexer 203 moves to Step ST38 to finish the process.
[0127]
Returning to FIG. 18, the compressed data buffer (cpb) 204 temporarily
accumulates the video stream (encoded stream) taken by the demultiplexer 203.
20 The decoder 205 decodes encoded image data of each of pictures of the video stream
accumulated in the compressed data buffer 204 at decoding timings of the pictures,
and sends the data to the uncompressed data buffer (dpb) 206.
[0128]
Here, the CPU 201 gives the decoder 205 decoding timings based on the
25 decoding time stamp (DTS). It should be noted that, when the decoder 205 decodes
the encoded image data of each picture, the decoder reads and uses image data of a
reference source picture from the uncompressed data buffer 206.
[0129]
FIG. 21 shows a configuration example of the decoder 205. This decoder
30 205 has an NAL packet analysis unit 251 and a decoding unit 252. The NAL packet
analysis unit 251 analyzes each NAL packet composing encoded image data to
32
determine a parameter set such as a VPS, an SPS, or a PPS, SEI, and a slice. The
NAL packet analysis unit 251 sends each NAL packet along with the determination
result to the decoding unit 252.
[0130]
The decoding unit 252 sequentially decodes encoded image 5 data of each of
pictures included in the video stream (encoded stream) sent via the NAL packet
analysis unit 251 at decoding timings, and sends the data to the uncompressed data
buffer (dpb) 206. In this case, the decoding unit 252 performs analysis of timing
information of a VPS, an SPS, and a PPS to ascertain “general_level_idc” which is
10 the level of the bit rate of an entire stream or “sublayer_level_idc” which is the level
of the bit rate of each sublayer, and to check whether or not decoding can be
performed within its decoding capability. In addition, in this case, the decoding unit
252 analyzes SEI to ascertain, for example, “initial_cpb_removal_time” and
“cpb_removal_delay” and to check whether a decoding timing sent from the CPU
15 201 is proper.
[0131]
In addition, in this case, the decoding unit 252 obtains a frame rate (frame
frequency) from information of “vps_num_units_in_tick” and “vps_times_scale”
included in the VPS and sends the frame rate to the CPU 201. Accordingly, the
20 CPU 201 can check the frame rate of a picture that is currently undergoing decoding.
It should be noted that the decoding unit 252 can determine whether or not a picture
is one with a discontinuity in time information and a time stamp by analyzing
temporal refresh information SEI, and when the picture is one with a discontinuity, a
frame rate (frame frequency) can also be obtained from information of
25 “num_units_in_tick” and “times_scale” included in that SEI.
[0132]
When decoding a slice, the decoding unit 252 acquires “ref_idx_l0_active
(ref_idx_l1_active) from the slice header as information indicating a prediction
destination in the time direction to perform prediction in the time direction. It
30 should be noted that decoded pictures are processed to be referred to by other
pictures using “short_term_ref_pic_set_idx” or “it_idx_sps” obtained from the slice
33
header as an index.
[0133]
Returning to FIG. 18, the uncompressed data buffer (dpb) 206 temporarily
stores the image data of each picture decoded by the decoder 205. The postprocessing
unit 207 performs a process on the image data of 5 each of the pictures
sequentially read from the uncompressed data buffer (dpb) 206 at display timings to
cause the frame rate thereof to match a display capability. In this case, the display
timings are given to the decoder 205 from the CPU 201 based on a presentation time
stamp (PTS).
10 [0134]
For example, when the frame rate of the decoded image data of the pictures
is 120 fps and the display capability is 120 fps, the post-processing unit 207 sends
the decoded image data of the pictures to a display as it is. In addition, when, for
example, the frame rate of the decoded image data of the pictures is 120 fps and the
15 display capability is 60 fps, the post-processing unit 207 performs a subsampling
process so that time-direction resolution is half that of the decoded image data of the
pictures, and sends the data to the display as image data of 60 fps.
[0135]
In addition, when the frame rate of the decoded image data of the pictures is
20 60 fps and the display capability is 120 fps, the post-processing unit 207 performs an
interpolation process so that time-direction resolution is twice the decoded image
data of the pictures, and sends the data to the display as image data of 120 fps. In
addition, when, for example, the frame rate of the decoded image data of the pictures
is 60 fps and the display capability is 60 fps, the post-processing unit 207 sends the
25 decoded image data of the pictures to the display as it is.
[0136]
The clock generation unit CPU 201 generates a system clock STC which is
synchronized with the PCR extracted by the demultiplexer 203. The frequency
divider 209 divides the frequency of this system clock STC and outputs a vertical
30 synchronization signal Vsync. This vertical synchronization signal Vsync is
supplied to the CPU 201 and also supplied to the demultiplexer 203, the decoder 205,
34
and the post-processing unit 207, and further other necessary spots.
[0137]
The frequency of this vertical synchronization signal Vsync is controlled
according to supply of a frequency division value (divider value) from the CPU 201.
In this embodiment, the frequency of the vertical synchronization 5 signal Vsync is
switched to match the frame rate of the first sequence until the display start timing of
the first picture of the second sequence, and to match the frame rate of the second
sequence from the display start timing.
[0138]
10 Thus, in the present embodiment, before the display start timing of the first
picture of the second sequence, encoded image data of each picture of the second
sequence is decoded at the decoding timing of encoded image data of each picture of
the first sequence.
[0139]
15 FIG. 22 shows an example of a switching timing of the vertical
synchronization signal Vsync. In this illustrated example, the first sequence is
composed of 4 pictures, which are decoded in the order of a0, a1, a2, and a3 and
displayed in the order of a0, a2, a3, and a1. In this case, the number of delayed
pictures after the first sequence is decoded until display thereof is started is set to 2.
20 [0140]
Each of the pictures of the first sequence is decoded and displayed at a
timing of the vertical synchronization signal (Vsync) having a time interval Va
corresponding to a frame rate (frame frequency) thereof. For example, the pictures
a0, a1, a2, and a3 are decoded at timings Ta(0), Ta(1), Ta(2), and Ta(3), and display
25 of the pictures a0, a2, a3, and a1 is started at timings Ta(2), Ta(3), Ta(4), and Ta(5).
[0141]
The second sequence is composed of 7 pictures, which are decoded in the
order of b0, b1, b2, b3, b4, b5, and b6 and displayed in the order of b0, b2, b3, b1, b5,
b6, and b3. In this case, the number of delayed pictures after the second sequence is
30 decoded until display thereof is started is set to 2, the same as the first sequence.
[0142]
35
Timings of decoding and display are set in encoding such that each of the
pictures of the second sequence is decoded and displayed at timings of the vertical
synchronization signal (Vsync) having a time interval Vb (Vb).
5 [0147]
In the illustrated example, the first sequence is composed of 4 pictures,
which are decoded in the order of b0, b1, b2, and b3, and displayed in the order of b0,
b2, b3, and b1. In this case, the number of delayed pictures after the sequence is
decoded until display thereof is started is set to 2.
10 [0148]
Each of the pictures of the first sequence is decoded and displayed at a
timing of the vertical synchronization signal (Vsync) having the time interval Vb
corresponding to a frame rate (frame frequency) thereof. For example, the pictures
b0, b1, b2, and b3 are decoded at the timings Tb(0), Tb(1), Tb(2), and Tb(3), and
15 display of the pictures b0, b2, b3, and b1 is started at the timings Tb(2), Tb(3), Tb(4),
and Tb(5).
[0149]
The second sequence is composed of 4 pictures, which are decoded in the
order of a0, a1, a2, and a3, and displayed in the order of a0, a2, a3, and a1. In this
20 case, the number of delayed pictures after the sequence is decoded until display
thereof is started is set to 2, the same as the first sequence.
[0150]
Timings of decoding and display are set in encoding such that each of the
pictures of the second sequence is decoded and displayed at a timing of the vertical
25 synchronization signal (Vsync) having the time interval Va (>Vb) corresponding to a
frame rate (frame frequency) thereof. In this embodiment, however, the frequency
is switched to match the frame rate of the first sequence until the display start timing
(Ta(2)) of the first picture of the second sequence, and to match the frame rate of the
second sequence from that timing.
30 [0151]
Thus, before the display start timing (Ta(2)) of the first picture of the second
37
sequence, the decoder 205 decodes the encoded image data of each of the pictures
(the pictures a0 and a1 in the illustrated example) of the second sequence at the
decoding timing of the encoded image data of each picture of the first sequence with
continuity in decoding timings with the first sequence maintained.
5 [0152]
In other words, the pictures a0 and a1 are decoded at the timings Tb(4) and
Tb(5). The picture a0 can be decoded even after the timing (Ta(0)) because the
demultiplexer 203 extracts temporal discontinuity information
(temporal_discontinuity_information) from the area of the PES extension at the
10 timing at which the leading byte of the last picture of the first sequence enters the
compressed data buffer (cpb) 204 at the latest (the timing P1 in FIG. 3) and detects
discontinuity information.
[0153]
In addition, switching of the vertical synchronization signal Vsync is
15 performed at the display start timing (Ta(2)) of the first picture of the second
sequence , i.e., the display end timing (Tb(6)) of the last picture of the first sequence
as described above. The CPU 201 can detect the necessity of the switching of the
vertical synchronization signal Vsync at this timing from the discontinuity
information that is the temporal discontinuity information and the count-down
20 information.
[0154]
An operation of the reception device 200 shown in FIG. 18 will be briefly
described. The reception unit 202 demodulates an RF-modulated signal received by
the reception antenna to acquire the transport stream TS. The transport stream TS
25 includes a video stream with a switched part from encoded image data of the first
sequence to encoded image data of the second sequence having a different frame rate
from the first sequence. This transport stream TS is sent to the demultiplexer 203.
The demultiplexer 203 takes out the video stream from the transport stream TS and
sends the stream to the compressed data buffer (cpb) 204 to cause the stream to be
30 temporarily accumulated.
[0155]
38
In addition, the demultiplexer 203 extracts the program clock reference
(PCR) from an adaptation field of a TS packet. This PCR is supplied to the clock
generation unit 208. The clock generation unit 208 generates the system clock STC
which is synchronized with the PCR. Then, this system clock STC undergoes
frequency division by the frequency divider 209 to 5 obtain the vertical
synchronization signal Vsync. This vertical synchronization signal Vsync is
supplied to the CPU 201 and also to necessary spots such as the demultiplexer 203,
the decoder 205, and the post-processing unit 207.
[0156]
10 In addition, the demultiplexer 203 extracts a descriptor such as the video
parameter descriptor (Video_parameter_descriptor) from the transport stream TS and
supplies the descriptor to the CPU 201. The CPU 201 can recognize a change in the
frame frequency (frame rate) in advance from the information of the video parameter
descriptor, and can further recognize the changed frame frequency.
15 [0157]
In addition, the demultiplexer 203 extracts the temporal discontinuity
information (temporal_discontinuity_information) inserted in the area of the PES
extension of the PES header for each picture, and the discontinuity information and
the count-down information are supplied to the CPU 201.
20 [0158]
Accordingly, when there is a discontinuity in a change of the value of a
decoding time (or decoding time stamp), the CPU 201 can recognize the
discontinuity in advance and further can clearly ascertain progress of units of access
units (pictures) until the discontinuity occurs from the count-down information
25 (count-down value). In addition, when a display time interval of each access unit
(picture) is switched, the CPU 201 can recognize the switching in advance and
further can clearly ascertain progress in units of access units (pictures) until the
switching from the count-down information (count-down value).
[0159]
30 In addition, the demultiplexer 203 extracts the time stamps of the DTS and
PTS disposed in the PES header for each picture and supplies them to the CPU 201.
39
The CPU 201 gives decoding timings and display timings to the decoder 205 based
on the time stamps.
[0160]
The decoder 205 decodes encoded image data of each of the pictures of the
video stream accumulated in the compressed data buffer 204 at the 5 decoding timings
of the pictures and sends the data to the uncompressed data buffer (dpb) 206 to cause
the data to be temporarily accumulated. In this case, when the encoded image data
of each of the pictures is decoded, image data of a reference target picture is read
from the uncompressed data buffer 206 and used if necessary.
10 [0161]
The image data of each of the pictures sequentially read from the
uncompressed data buffer (dpb) 206 at the display timings is sent to the postprocessing
unit 207. In this case, the display end timing of the last picture of the
first sequence is set to the same timing as the display start timing of the first picture
15 of the second sequence.
[0162]
The post-processing unit 207 performs interpolation or subsampling to
cause the frame rate of the image data of the pictures to match the display capability.
The image data of the pictures processed by the post-processing unit 207 is supplied
20 to the display and a dynamic image of the image data of the pictures is displayed.
[0163]
It should be noted that the frequency of the vertical synchronization signal
Vsync is switched as a frequency division value (divider value) supplied from the
CPU 201 is changed. The frequency of the vertical synchronization signal Vsync is
25 switched to match the frame rate of the first sequence until the display start timing of
the first picture of the second sequence and to match the frame rate of the second
sequence from that timing. The CPU 201 recognizes this timing of switching based
on the discontinuity information that is the temporal discontinuity information and
the count-down information.
30 [0164]
In addition, the frequency of the vertical synchronization signal Vsync is set
40
to match the frame rate of the first sequence before the display start timing of the
first picture of the second sequence as above. For this reason, the encoded image
data of each of the pictures of the second sequence is decoded at the decoding
timings of the encoded image data of each of the pictures of the first sequence before
the display start timing of the first picture of the second sequence 5 according to
control of the CPU 201. This control is possible since the discontinuity information
of the temporal discontinuity information is detected.
[0165]
As described above, in the transmission and reception system 10 shown in
10 FIG. 1, the transmission side performed encoding such that the display end timing of
the last picture of the encoded image data of the first sequence is set to the same
timing as the display start timing of the first picture of the encoded image data of the
second sequence. Thus, for example, the reception side can easily secure a display
continuity between the first sequence and the second sequence.
15 [0166]
In addition, in the transmission and reception system 10 shown in FIG. 1,
the transmission side performs encoding such that the encoded image data of the first
sequence and the encoded image data of the second sequence have the same number
of delayed pictures after decoding until displaying. Thus, for example, the
20 reception side can avoid a buffer failure that could occur in the switched part from
the first sequence to the second sequence.
[0167]
In addition, in the transmission and reception system 10 shown in FIG. 1,
the transmission side inserts information indicating whether or not there is a change
25 in a frame rate, i.e., temporal refresh information SEI (temporal_refresh_information
SEI) into the encoded image data of each picture of the video stream. Thus, for
example, the reception side can easily ascertain the point of the change from the first
sequence to the second sequence.
[0168]
30 In addition, in the transmission and reception system 10 shown in FIG. 1,
the transmission side inserts information notifying of a change from the encoded
41
image data of the first sequence to the encoded image data of the second sequence,
i.e., temporal discontinuity information (temporal_discontinuity_information) into a
packet containing the video stream, for example, a PES packet. Thus, for example,
the reception side can easily ascertain the change from the first sequence to the
second sequence, time information, discontinuity information 5 of a time stamp, and
count-down information until the change.
[0169]
In addition, in the transmission and reception system 10 shown in FIG. 1,
the transmission side inserts information notifying of a change from the encoded
10 image data of the first sequence to the encoded image data of the second sequence,
i.e., a video parameter descriptor (Video_parameter_descriptor) into a layer of a
container, i.e., a layer of the transport stream TS. Thus, for example, the reception
side can easily ascertain the change from the first sequence to the second sequence
and the changed frame rate.
15 [0170]
In addition, in the transmission and reception system 10 shown in FIG. 1,
the reception side decodes the encoded image data of each of the pictures of the
second sequence at the decoding timing of the encoded image data of each of the
pictures of the first sequence before the display start timing of the first picture of the
20 second sequence. Thus, for example, implementation in which two asynchronous
vertical synchronization signals are simultaneously generated can be avoided.
[0171]
<2. Modified example>
In the above-described embodiment, the example in which the reception
25 side can easily secure a display continuity between the first sequence and the second
sequence is shown. To this end, the transmission side performs encoding such that
the display end timing of the last picture of the encoded image data of the first
sequence is set to the same timing as the display start timing of the first picture of the
encoded image data of the second sequence.
30 [0172]
Here, the reception side is considered to easily secure a display continuity
42
between the first sequence and the second sequence and further easily secure
continuity in decoding timings of the first sequence and the second sequence. To
this end, the transmission side performs encoding such that the decoding timing of
the first picture of the encoded image data of the second sequence is set at the timing
obtained by adding the time interval of one picture of the encoded 5 image data of the
first sequence to the decoding timing of the last picture of the encoded image data of
the first sequence.
[0173]
FIG. 24 shows an example of hypothetical reference decoder (HRD) control
10 of the encoder 102 in that case. In FIG. 24, detailed descriptions of portions
corresponding to those of FIG. 3 described above are appropriately omitted.
[0174]
In the illustrated example, the first sequence is composed of 4 pictures,
which are decoded in the order of a0, a1, a2, and a3, and displayed in the order of a0,
15 a2, a3, and a1. In this case, image data of each decoded picture is input to the
uncompressed data buffer (decoded picture buffer or dpb). In this example, the
number of delayed pictures after the first sequence is decoded until display thereof is
started is set to 2.
[0175]
20 Each of the pictures of the first sequence is decoded and displayed at a
timing of the vertical synchronization signal (Vsync) having a time interval
corresponding to a frame rate (frame frequency) thereof. For example, the pictures
a0, a1, a2, and a3 are decoded at the timings Ta(0), Ta(1), Ta(2), and Ta(3), and
display of the pictures a0, a2, a3, and a1 is started at the timings Ta(2), Ta(3), Ta(4),
25 and Ta(5).
[0176]
The second sequence is composed of 6 pictures, which are decoded in the
order of b0, b1, b2, b3, b4, and b5, and displayed in the order of b1, b2, b0, b4, b5,
and b3. In this case, encoded data of each of the decoded pictures is input to the
30 dpb. In this example, the number of delayed pictures after the second sequence is
decoded until display thereof is started is set to 2, the same as the first sequence.
43
Accordingly, a buffer failure is avoided.
[0177]
Like in the example of FIG. 3, the display end timing of the last picture of
the encoded image data of the first sequence is set to the same timing as the display
start timing of the first picture of the encoded image data of the 5 second sequence.
In other words, the timing Ta(6) is set to the same timing as the timing Tb(2).
Accordingly, a display continuity between the first sequence and the second
sequence is ensured.
[0178]
10 From the display start timing (Tb(2)) of the first picture of the second
sequence, the sequence is decoded and displayed at timings of the vertical
synchronization signal (Vsync) having a time interval corresponding to the frame
rate (frame frequency) thereof. For example, the pictures b2, b3, b4, and b5 are
decoded at the timings Tb(2), Tb(3), Tb(4), and Tb(5), and display of the pictures b1,
15 b2, b0, b4, b5, and b3 is started at the timings Tb(2), Tb(3), Tb(4), Tb(5), Tb(6), and
Tb(7).
[0179]
In addition, before the display start timing (Tb(2)) of the first picture thereof,
the second sequence is decoded at the timings of the vertical synchronization signal
20 (Vsync) having a time interval corresponding to the first frame rate (frame
frequency). In this example, the pictures b0 and b1 are decoded at Tb(0) and Tb(1),
under the setting of Tb(0) = Ta(4) and Tb(1) = Ta(5).
[0180]
In this case, the decoding timing of the first picture of the encoded image
25 data of the second sequence is set at the timing obtained by adding the time interval
of one picture of the encoded image data of the first sequence to the decoding timing
of the last picture of the encoded image data of the first sequence. In other words,
the timing Ta(4) is set to the same timing as the timing Tb(0). Therefore,
“Initial_cpb_removal_delayB” is set so that the timing Tb(0) is set to the same
30 timing as the timing Ta(4).
[0181]
44
By performing encoding such that the timing Tb(0) is set to the same timing
as the timing Ta(4), the reception side easily secures continuity in the decoding
timings of the first sequence and the second sequence, and late reading of the first
picture of the second sequence is avoided. In this case, the reception side does not
necessitate a process of adjusting the decoding timings of the pictures 5 of the second
sequence positioned before the display start timing of the first picture of the second
sequence as described in FIGS. 22 and 23 above. Therefore, a burden of the
reception side can also be reduced.
[0182]
10 <2. Modified example>
It should be noted that, although the above-described embodiment shows the
transmission and reception system 10 constituted by the transmission device 100 and
the reception device 200, a configuration of a transmission and reception system to
which the present technology can be applied is not limited thereto. For example,
15 the reception device 200 part may be configured with, for example, a set-top box
connected with a digital interface such as High-Definition Multimedia Interface
(HDMI) and a monitor, or the like. Note that “HDMI” is a registered trademark.
[0183]
In addition, the above-described embodiment shows the example in which
20 the container is a transport stream (MPEG-2 TS). The present technology, however,
can also be similarly applied to a system configured to distribute data to a reception
terminal using a network such as the Internet. In distribution on the Internet, there
are many cases of distribution using a container in MP4 or other formats. In other
words, as containers, various formats including transport streams (MPEG-2 TS)
25 employed in digital broadcasting standards, MP4 used in Internet distribution, and
the like are equivalent thereto.
[0184]
Additionally, the present technology may also be configured as below.
(1)
30 An encoding device including:
an image encoding unit configured to generate video data having a switched
45
part from encoded image data of a first sequence to encoded image data of a second
sequence having a different frame rate from the first sequence,
wherein the image encoding unit performs encoding in a manner that a
display end timing of a last picture of the encoded image data of the first sequence is
set to a same timing as a display start timing of a first picture of 5 the encoded image
data of the second sequence.
(2)
The encoding device according to (1), wherein the image encoding unit
performs encoding in a manner that the encoded image data of the first sequence and
10 the encoded image data of the second sequence have the same number of delayed
pictures from decoding until display.
(3)
The encoding device according to (1) or (2), wherein the image encoding
unit performs encoding in a manner that a decoding timing of the first picture of the
15 encoded image data of the second sequence is set at a timing obtained by adding a
time interval of one picture of the encoded image data of the first sequence to a
decoding timing of the last picture of the encoded image data of the first sequence.
(4)
A transmission device including:
20 an image encoding unit configured to generate video data having a switched
part from encoded image data of a first sequence to encoded image data of a second
sequence having a different frame rate from the first sequence; and
a transmission unit configured to transmit a container in a predetermined
format which includes the generated video data,
25 wherein the image encoding unit performs encoding in a manner that a
display end timing of a last picture of the encoded image data of the first sequence is
set to a same timing as a display start timing of a first picture of the encoded image
data of the second sequence.
(5)
30 The transmission device according to (4), further including:
an information insertion unit configured to insert information indicating
46
whether or not there is a change in a frame rate into each piece of encoded image
data of each picture of the video data.
(6)
The transmission device according to (5), wherein, when the information
indicates that there is a change in the frame rate, information indicating 5 a value of a
changed frame rate is added to the information.
(7)
The transmission device according to any of (4) to (6), further including:
an information insertion unit configured to insert information notifying of a
10 change from the encoded image data of the first sequence to the encoded image data
of the second sequence into a packet containing the video data.
(8)
The transmission device according to (7), wherein the notification
information includes information indicating whether or not there is a discontinuity in
15 a change of a value of a decoding time.
(9)
The transmission device according to (8), wherein, when the information
indicates that there is a discontinuity, a count-down value of the number of pictures
until the discontinuity occurs is added to the information.
20 (10)
The transmission device according to any of (7) to (9), wherein the
notification information includes information indicating whether or not a display
time interval of each picture is switched.
(11)
25 The transmission device according to (10), wherein, when the information
indicates switching, a count-down value of the number of pictures until the switching
is added to the information.
(12)
The transmission device according to any of (7) to (11), wherein the
30 information insertion unit inserts the notification information into an extension field
of a PES packet which includes encoded image data of each picture in a payload.
47
(13)
The transmission device according to any of (4) to (12), further including:
an information insertion unit configured to insert information notifying of a
change from the encoded image data of the first sequence to the encoded image data
of the second sequence into a layer 5 of the container.
(14)
The transmission device according to (13), wherein the notification
information includes information on a frame rate of the first sequence and a frame
rate of the second sequence.
10 (15)
A reception device including:
a reception unit configured to receive video data having a switched part
from encoded image data of a first sequence to encoded image data of a second
sequence having a different frame rate from the first sequence; and
15 a processing unit configured to process the received video data,
wherein the video data is encoded in a manner that a display end timing of a
last picture of the encoded image data of the first sequence is set to a same timing as
a display start timing of a first picture of the encoded image data of the second
sequence.
20 (16)
The reception device according to (15), wherein the video data is encoded in
a manner that the encoded image data of the first sequence and the encoded image
data of the second sequence have the same number of delayed pictures from
decoding until display.
25 (17)
The reception device according to (15) or (16), wherein the video data is
encoded in a manner that a decoding timing of the first picture of the encoded image
data of the second sequence is set at a timing obtained by adding a time interval of
one picture of the encoded image data of the first sequence to a decoding timing of
30 the last picture of the encoded image data of the first sequence.
(18)
48
A reception device including:
a reception unit configured to receive a container in a predetermined format
which includes video data having a switched part from encoded image data of a first
sequence to encoded image data of a second sequence having a different frame rate
from 5 the first sequence,
wherein the video data is encoded in a manner that a display end timing of a
last picture of the encoded image data of the first sequence is set to a same timing as
a display start timing of a first picture of the encoded image data of the second
sequence, and
10 wherein the reception device further includes an image decoding unit
configured to decode the video data included in the received container to obtain
image data with a display continuity between the first sequence and the second
sequence maintained.
(19)
15 The reception device according to (18), wherein, before the same timing, the
image decoding unit decodes encoded image data of each picture of the second
sequence at a timing synchronized with a decoding timing of encoded image data of
each picture of the first sequence.
(20)
20 The reception device according to (18) or (19),
wherein information notifying of a change from the encoded image data of
the first sequence to the encoded image data of the second sequence is inserted into
at least one of a packet containing the video data and a layer of the container, and
wherein a process of the image decoding unit is controlled based on the
25 notification information.
[0185]
A main characteristic of the present technology is that, when video data
having a frame rate switched part is distributed, a reception side can easily secure a
display continuity between a first sequence and a second sequence as encoding is
30 performed such that the display end timing of the last picture of encoded image data
of the first sequence is set to the same timing as the display start timing of the first
49
picture of encoded image data of the second sequence (see FIG. 3).
Reference Signs List
[0186]
10 transmission 5 and reception system
100 transmission device
101 CPU
102 encoder
103 compressed data buffer (cpb)
10 104 multiplexer
105 transmission unit
122 buffer delay control unit
123 HRD setting unit
124 parameter set/SEI encoding unit
15 125 slice encoding unit
126 NAL packetizing unit
142 section coding unit
143 PES packetizing unit
144 transport packetizing unit
20 200 reception device
201 CPU
202 reception unit
203 demultiplexer
204 compressed data buffer (cpb)
25 205 decoder
206 uncompressed data buffer (dpb)
207 post-processing unit
208 clock generation unit
209 frequency divider
30 231 TS adaptation field extraction unit
232 clock information extraction unit
50
233 TS payload extraction unit
234 section extraction unit
235 PSI table/descriptor extraction unit
236 PES packet extraction unit
237 PES header 5 extraction unit
238 time stamp extraction unit
239 discontinuity/count-down information extraction unit
240 PES payload extraction unit
251 NAL packet analysis unit
10 252 decoding unit
51
CLAIMS
Claim 1
An encoding device comprising:
an image encoding unit configured to generate video data having a switched
part from encoded image data of a first sequence to encoded image 5 data of a second
sequence having a different frame rate from the first sequence,
wherein the image encoding unit performs encoding in a manner that a
display end timing of a last picture of the encoded image data of the first sequence is
set to a same timing as a display start timing of a first picture of the encoded image
10 data of the second sequence.
Claim 2
The encoding device according to claim 1, wherein the image encoding unit
performs encoding in a manner that the encoded image data of the first sequence and
15 the encoded image data of the second sequence have the same number of delayed
pictures from decoding until display.
Claim 3
The encoding device according to claim 1, wherein the image encoding unit
20 performs encoding in a manner that a decoding timing of the first picture of the
encoded image data of the second sequence is set at a timing obtained by adding a
time interval of one picture of the encoded image data of the first sequence to a
decoding timing of the last picture of the encoded image data of the first sequence.
25 Claim 4
A transmission device comprising:
an image encoding unit configured to generate video data having a switched
part from encoded image data of a first sequence to encoded image data of a second
sequence having a different frame rate from the first sequence; and
30 a transmission unit configured to transmit a container in a predetermined
format which includes the generated video data,
52
wherein the image encoding unit performs encoding in a manner that a
display end timing of a last picture of the encoded image data of the first sequence is
set to a same timing as a display start timing of a first picture of the encoded image
data of the second sequence.
5
Claim 5
The transmission device according to claim 4, further comprising:
an information insertion unit configured to insert information indicating
whether or not there is a change in a frame rate into each piece of encoded image
10 data of each picture of the video data.
Claim 6
The transmission device according to claim 5, wherein, when the
information indicates that there is a change in the frame rate, information indicating a
15 value of a changed frame rate is added to the information.
Claim 7
The transmission device according to claim 4, further comprising:
an information insertion unit configured to insert information notifying of a
20 change from the encoded image data of the first sequence to the encoded image data
of the second sequence into a packet containing the video data.
Claim 8
The transmission device according to claim 7, wherein the notification
25 information includes information indicating whether or not there is a discontinuity in
a change of a value of a decoding time.
Claim 9
The transmission device according to claim 8, wherein, when the
30 information indicates that there is a discontinuity, a count-down value of the number
of pictures until the discontinuity occurs is added to the information.
53
Claim 10
The transmission device according to claim 7, wherein the notification
information includes information indicating whether or not a display time interval of
each 5 picture is switched.
Claim 11
The transmission device according to claim 10, wherein, when the
information indicates switching, a count-down value of the number of pictures until
10 the switching is added to the information.
Claim 12
The transmission device according to claim 7, wherein the information
insertion unit inserts the notification information into an extension field of a PES
15 packet which includes encoded image data of each picture in a payload.
Claim 13
The transmission device according to claim 4, further comprising:
an information insertion unit configured to insert information notifying of a
20 change from the encoded image data of the first sequence to the encoded image data
of the second sequence into a layer of the container.
Claim 14
The transmission device according to claim 13, wherein the notification
25 information includes information on a frame rate of the first sequence and a frame
rate of the second sequence.
Claim 15
A reception device comprising:
30 a reception unit configured to receive video data having a switched part
from encoded image data of a first sequence to encoded image data of a second
54
sequence having a different frame rate from the first sequence; and
a processing unit configured to process the received video data,
wherein the video data is encoded in a manner that a display end timing of a
last picture of the encoded image data of the first sequence is set to a same timing as
a display start timing of a first picture of the encoded image 5 data of the second
sequence.
Claim 16
The reception device according to claim 15, wherein the video data is
10 encoded in a manner that the encoded image data of the first sequence and the
encoded image data of the second sequence have the same number of delayed
pictures from decoding until display.
Claim 17
15 The reception device according to claim 15, wherein the video data is
encoded in a manner that a decoding timing of the first picture of the encoded image
data of the second sequence is set at a timing obtained by adding a time interval of
one picture of the encoded image data of the first sequence to a decoding timing of
the last picture of the encoded image data of the first sequence.
20
Claim 18
A reception device comprising:
a reception unit configured to receive a container in a predetermined format
which includes video data having a switched part from encoded image data of a first
25 sequence to encoded image data of a second sequence having a different frame rate
from the first sequence,
wherein the video data is encoded in a manner that a display end timing of a
last picture of the encoded image data of the first sequence is set to a same timing as
a display start timing of a first picture of the encoded image data of the second
30 sequence, and
wherein the reception device further includes an image decoding unit
55
configured to decode the video data included in the received container to obtain
image data with a display continuity between the first sequence and the second
sequence maintained.
5 Claim 19
The reception device according to claim 18, wherein, before the same
timing, the image decoding unit decodes encoded image data of each picture of the
second sequence at a timing synchronized with a decoding timing of encoded image
data of each picture of the first sequence.
10
Claim 20
The reception device according to claim 18,
wherein information notifying of a change from the encoded image data of
the first sequence to the encoded image data of the second sequence is inserted into
15 at least one of a packet containing the video data and a layer of the container, and
wherein a process of the image decoding unit is controlled based on the
notification information.
56
ABSTRACT
“ENCODING DEVICE, TRANSMITTING DEVICE, AND RECEIVING
DEVICE”
The present invention makes it possible to easily ensure display continuity on the
receiving side when video data having a frame rate switching portion is delivered.
Video data having a portion of switching from encoded image data in a first sequence
to encoded image data in a second sequence having a frame rate different from that of
the first sequence is generated. This video data is encoded such that the display end
timing of the last picture of the encoded image data in the first sequence and the
display start timing of the first picture of the encoded image data in the second
sequence are the same timing.
FIG. 3