FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENTS RULES, 2003
COMPLETE SPECIFICATION
(See section 10, rule 13)
“TRANSMISSION APPARATUS, TRANSMISSION METHOD, RECEPTION
APPARATUS AND RECEPTION METHOD”
SONY CORPORATION, of 1-7-1, Konan, Minato-ku, Tokyo 108-0075, Japan
The following specification particularly describes the invention and the manner in which it is to
be performed.
2
DESCRIPTION
TRANSMISSION DEVICE, TRANSMISSION METHOD, RECEPTION DEVICE,
AND RECEPTION METHOD
5 TECHNICAL FIELD
[0001]
The present technology relates to a transmission device,
a transmission method, a reception device, and a reception
method. In detail, the present technology relates, for
10 example to a transmission device that transmits the moving
image data at a high frame rate (High Frame Rate).
BACKGROUND ART
[0002]
15 Cameras that take an image at a high frame rate with
a high-speed frame shutter have been known in recent years
(see, for example, Patent Document 1). For example, a high
frame rate is several times, dozens of times, or hundreds of
times as fast as a normal frame rate, for example, of 60 or
20 50 fps.
[0003]
In a service with a high frame rate, the moving image
data taken with a camera with a high-speed frame shutter may
be transmitted by being converted into a moving image sequence
25 at a frequency lower than the frequency of the data before
the moving image data is transmitted. However, the images
taken with a high-speed frame shutter have a factor causing
a problem on the image quality of the conventional frame
interpolation technique used in the receiving and replaying
30 end although the images have an effect of improving motion
blurs and achieving an image quality with a high degree of
3
sharpness.
[0004]
The frame interpolation with images with a high degree
of sharpness, which are taken with a high-speed frame shutter,
brings a larger difference between the frames 5 to which motion
vector search is applicable and the frames to which motion
vector search is not applicable. Thus, the difference is
displayed as noticeable image degradation. High-load
calculation is required in order to improve the accuracy of
10 motion vector search in frame interpolation. However, the
high-load calculation adversely affects the cost of the
receiver.
CITATION LIST
15 PATENT DOCUMENT
[0005]
Patent Document 1: Japanese Patent Application Laid-Open No.
2010-178124
20 SUMMARY OF THE INVENTION
PROBLEMS TO BE SOLVED BY THE INVENTION
[0006]
In order to allow the compatibility with the conventional
receiver at a normal frame rate, the compatibility is a service
25 with a high frame rate, it is expected to convert the data
of the image taken with a high-speed frame shutter in order
to display the image with a predetermined or higher quality
on a conventional receiver that decodes the data at a normal
frame rate.
30 [0007]
An objective of the present technology is to transmit
4
the moving image data at a high frame rate in good condition.
SOLUTIONS TO PROBLEMS
[0008]
A concept of the present technology 5 lies in a
transmission device including:
an image processing unit that provides second moving
image data at a predetermined frame rate by processing first
moving image data at the predetermined frame rate in units
10 of consecutive N pictures, using an image data item provided
by averaging image data items of the N pictures in an averaging
process as an image data item of a first picture, and using
image data items of second to Nth pictures of the N pictures
as image data items of second to Nth pictures without any change,
15 the N being an integer larger than or equal to two;
an image encoding unit that generates a video stream
by encoding the image data items of the pictures in the second
moving image data provided in the image processing unit; and
a transmission unit that transmits the video stream
20 generated in the image encoding unit.
[0009]
According to the present technology, the image
processing unit can provide the second moving image data at
a predetermined frame rate from the first moving image data
25 at the predetermined frame rate. At that time, the image
processing unit provides the second moving image data by
processing the first moving image data in units of consecutive
N pictures (N is an integer larger than or equal to two), using
the image data item provided by averaging the image data items
30 of the N pictures in an averaging process as the image data
item of the first picture, and using the image data items of
5
the second to Nth pictures of the N pictures as the image data
items of the second to Nth pictures without any change. For
example, the averaging process is a process for calculating
a linear average or a process for calculating a non-linear
5 average.
[0010]
The image encoding unit generates a video stream by
encoding the image data item of each picture in the second
moving image data provided with the image processing unit.
10 For example, the image encoding unit may generate a first video
stream that includes the encoded image data item of the first
picture in each of the units of N pictures, and a predetermined
number of second video streams that include the encoded image
data items of the second to Nth pictures in each of the units
15 of N pictures. This allows, for example, a receiving end,
which has a decoding capacity to decode the moving image data
at a frame rate of 1/N of the predetermined frame rate, to
select and decode only the first video stream in a decoding
process. The receiving end does not need having a complicated
20 configuration in order to perform, for example, a process for
combining several video streams.
[0011]
Note that, in such case, the image encoding unit may
classify the image data items of the pictures in the second
25 moving image data provided in the image processing unit into
a plurality of hierarchal layers, encode the classified image
data items of the pictures in each of the hierarchal layers,
divide the hierarchal layers into a predetermined number of
hierarchal groups, and generate a predetermined number of video
30 streams, the video streams may include the encoded image data
items of the pictures in each of the divided hierarchal groups,
6
and a lowest hierarchal group includes the encoded image data
item of the first picture in each of the units of N pictures.
[0012]
In such case, for example, the image encoding unit may
insert first identification information into 5 the encoded image
data items in the first video stream, and the first
identification information indicates that the first video
stream includes a temporally varying element of the second
video stream. For example, the image encoding unit may further
10 insert information about the averaging process into the encoded
image data items in the first video stream. This enables the
receiving end, for example, to recognize that the image data
item corresponding to the encoded image data item of the first
picture in each of the units of N pictures in the first video
15 stream is provided from the averaging process, and which
averaging process the performed averaging process is.
[0013]
In such case, for example, the transmission unit may
transmit the container being in a predetermined format and
20 including the video streams, and the transmission device may
further include: an identification information inserting unit
that inserts second identification information into a layer
in a container, the second identification information
indicating that the first identification information is
25 inserted in the encoded image data items in the first video
stream. This enables the receiving end, for example, to
recognize that the first identification information is
inserted in the encoded image data from the second
identification information without processing the encoded
30 image data.
[0014]
7
Additionally, the image encoding unit can be configured
to insert the information about the shutter aperture time ratio
of the image data item of each picture in the first moving
image data at the predetermined frame rate into the encoded
image data item of each picture in the video 5 stream. This
insertion enables the receiving end, for example, to adaptively
switch the processes for frame interpolation in accordance
with the information about the shutter aperture time ratio.
[0015]
10 According to the present technology as described above,
the moving image data at a predetermined frame rate is not
transmitted without any change. The image data in which the
image data item provided by averaging the image data items
of N pictures in an averaging process is used as the image
15 data item of the first picture of each of the units of N pictures
is transmitted. This allows, for example, a receiving end,
which has a decoding capacity to decode the moving image data
at a frame rate of 1/N of the predetermined frame rate, to
select and decode only the encoded image data item of the first
20 picture in each of the units of N pictures in a decoding process
in order to provide the moving image data. This enables the
receiving end to display a smooth image, and to prevent the
frame interpolation process with low-load calculation from
causing a problem on the image quality.
25 [0016]
Further, another concept of the present technology lies
in a reception device including:
a reception unit that receives a video stream, the video
stream being generated by encoding an image data item of each
30 picture in moving image data at a predetermined frame rate;
and
8
a processing unit that processes the video stream
received with the reception unit,
wherein the moving image data at the predetermined frame
rate is second moving image data at the predetermined frame
rate, and the second moving image data is provided 5 by processing
first moving image data at the predetermined frame rate in
units of consecutive N pictures, using an image data item
provided by averaging image data items of the N pictures in
an averaging process as an image data item of a first picture,
10 and using image data items of second to Nth pictures of the
N pictures as image data items of second to Nth pictures without
any change, and the N is an integer larger than or equal to
two.
[0017]
15 According to the present technology, the reception unit
receives a video stream generated by encoding the image data
item of each picture in the moving image data at a predetermined
frame rate. The moving image data at the predetermined frame
rate is the second moving image data at the predetermined frame
20 rate, which is provided by processing the first moving image
data at the predetermined frame rate in units of consecutive
N pictures (N is an integer larger than or equal to two), using
the image data item provided by averaging the image data items
of the N pictures in an averaging process as the image data
25 item of the first picture, and using the image data items of
the second to Nth pictures of the N pictures as the image data
items of the second to Nth pictures without any change. Then,
the processing unit processes the received video stream.
[0018]
30 For example, the video stream received with the reception
unit may include a first video stream that includes the encoded
9
image data item of the first picture in each of the units of
N pictures, and a predetermined number of second video streams
that include the encoded image data items of the second to
Nth pictures in each of the units of N pictures.
5 [0019]
Further, another concept of the present technology lies
in a reception device including:
a reception unit that receives a video stream generated
by encoding an image data item of each picture in second moving
10 image data at a predetermined frame rate, the second moving
image data being provided by processing first moving image
data at the predetermined frame rate in units of consecutive
N pictures, using an image data item provided by averaging
image data items of the N pictures in an averaging process
15 as an image data item of a first picture, and using image data
items of second to Nth pictures of the N pictures as image
data items of second to Nth pictures without any change, the
N being an integer larger than or equal to two;
an image decoding unit that provides the second moving
20 image data at the predetermined frame rate by decoding the
video stream received with the reception unit; and
an image processing unit that provides the first moving
image data by processing the second moving image data provided
in the image decoding unit.
25 [0020]
According to the present technology, the reception unit
receives a video stream. The video stream is generated by
encoding the image data item of each picture in the second
moving image data at a predetermined frame rate. The second
30 moving image data is provided by processing the first moving
image data at the predetermined frame rate in units of
10
consecutive N pictures (N is an integer larger than or equal
to two), using the image data item provided by averaging the
image data items of the N pictures in an averaging process
as the image data item of the first picture, and using the
image data items of the second to Nth pictures 5 of the N pictures
as the image data items of the second to Nth pictures without
any change.
[0021]
For example, the video stream received with the reception
10 unit may include a first video stream that includes the encoded
image data item of the first picture in each of the units of
N pictures, and a predetermined number of second video streams
that include the encoded image data items of the second to
Nth pictures in each of the units of N pictures.
15 [0022]
The image decoding unit provides the second moving image
data at the predetermined frame rate by decoding the video
stream received with the reception unit. The image processing
unit provides the first moving image data by processing the
20 second moving image data provided with the image decoding unit.
In the process, the image data item of the first picture in
each of the units of N pictures is restored to the image data
item of the first picture before the averaging process from
the image data item provided by averaging the image data items
25 of the N pictures.
[0023]
According to the present technology as described above,
the image data item of the first picture in each of the units
of N pictures is restored to the image data item of the first
30 picture before the averaging process. This enables, for
example, a receiving end, which has a decoding capacity to
11
decode the moving image data at the predetermined frame rate,
to display a smooth image with a high degree of sharpness at
a high frame rate.
[0024]
Note that, in the present technology, 5 for example, a
display processing unit that provides third moving image data
at a frame rate higher than the predetermined frame rate by
interpolating the first moving image data provided in the image
processing unit in a temporal direction may further be included.
10 In such case, for example, information about a shutter aperture
time ratio of an image data item of each picture in the first
moving image data may be inserted in an encoded image data
item of each picture in the video stream received with the
reception unit, and the display processing unit may switch
15 processes for frame interpolation in accordance with the
shutter aperture time ratio information.
[0025]
Further, the concept of the present technology lies in
a transmission device including:
20 an image encoding unit that generates a video stream
by encoding an image data item of each picture in moving image
data at a predetermined frame rate; the image encoding unit
that inserts information about a shutter aperture time ratio
of an image data item of each picture in the first moving image
25 data at the predetermined frame rate into the encoded image
data item of each picture in the video stream; and
a transmission unit that transmits the video stream
generated in the image encoding unit.
[0026]
30 Further, the concept of the present technology lies in
a reception device including:
12
a reception unit that receives a video stream provided
by encoding an image data item of each picture in moving image
data at a predetermined frame rate;
an image decoding unit that provides the moving image
data at the predetermined frame rate by 5 decoding the video
stream received with the reception unit; and
a display processing unit that provides third moving
image data at a frame rate higher than the predetermined frame
rate by interpolating the first image data provided in the
10 image decoding unit in a temporal direction,
wherein information about a shutter aperture time ratio
of an image data item of each picture in the moving image data
at the predetermined frame rate is inserted in an encoded image
data item of each picture in the video stream received with
15 the reception unit, and
the display processing unit switches processes for frame
interpolation in accordance with the shutter aperture time
ratio information.
20 EFFECTS OF THE INVENTION
[0027]
According to the present technology, the moving image
data at a high frame rate can be transmitted in good condition.
Note that the effects of the present technology are not
25 necessarily limited to the effects described herein, and can
include any one of the effects described herein.
BRIEF DESCRIPTION OF DRAWINGS
[0028]
30 Fig. 1 is a block diagram of an exemplary configuration
of a transmission and reception system that is an embodiment.
13
Fig. 2 is a diagram of exemplary processes for changing
a frame rate.
Fig. 3 is a block diagram of an exemplary configuration
of a transmission device.
Fig. 4 is a block diagram of an exemplary 5 configuration
of a preprocessor included in the transmission device.
Fig. 5 is a diagram of an exemplary relationship among
data items that are input to and output from the preprocessor
included in the transmission device, and are input to and output
10 from a postprocessor included in a reception device.
Fig. 6 is a diagram of an exemplary hierarchical encoding
performed in an encoder.
Fig. 7 is a diagram of an exemplary structure of an NAL
unit header and the contents of main parameters in the exemplary
15 structure.
Fig. 8 is an explanatory diagram of the formation of
encoded image data item of each picture in HEVC.
Fig. 9 is a diagram of an exemplary structure of an
interface for inserting capturing speed information SEI, and
20 an exemplary structure of “capturing_speed_information()”.
Fig. 10 is a diagram of the contents of main information
in the exemplary structure of
“capturing_speed_information()”.
Fig. 11 is a diagram of an exemplary structure of a shutter
25 speed information descriptor, and the contents of main
information in the exemplary structure.
Fig. 12 is a diagram of an exemplary structure of an
HEVC descriptor.
Fig. 13 is a diagram of an exemplary formation of a
30 transport stream TS.
Fig. 14 is a block diagram of an exemplary configuration
14
of a (high frame rate compatible) reception device.
Fig. 15 is an explanatory diagram of stream selection
in a demultiplexer in the reception device.
Fig. 16 is a block diagram of an exemplary configuration
of the postprocessor included in the 5 reception device.
Fig. 17 is a block diagram of an exemplary configuration
of a (normal frame rate compatible) reception device.
Fig. 18 is a diagram of an exemplary relationship among
data items that are input to and output from the preprocessor
10 included in the transmission device, and are input to and output
from the postprocessor included in the reception device.
MODE FOR CARRYING OUT THE INVENTION
[0029]
15 Modes for carrying out the invention (hereinafter
referred to as “embodiments”) will be described hereinafter.
Note that the description will be given in the following order.
1. Embodiment
2. Exemplary variation
20 [0030]
<1. Embodiment>
[Transmission and Reception System]
Fig. 1 illustrates an exemplary configuration of a
transmission and reception system 10 that is an embodiment.
25 The transmission and reception system 10 includes a
transmission device 100 and a reception device 200.
[0031]
The transmission device 100 transmits a transport stream
TS that is a container over broadcasting waves. The transport
30 stream TS includes a video stream, for example, provided by
classifying image data item of each picture forming moving
15
image data at a high frame rate, which is 120 fps in this
embodiment, into a plurality of hierarchical layers and
encoding the image data items. For example, the image data
items are encoded in H.264/AVC or H.265/HEVC in such a case.
5 [0032]
The moving image data at 120 fps that is to be encoded
is the second moving image data at 120 fps provided by processing
the first moving image data at 120 fps in units of consecutive
two pictures, using the image data item provided by averaging
10 the image data items of the two pictures in a linear or non-linear
averaging process as the image data item of the first picture,
and using the image data item of the second picture of the
two pictures as the image data item of the second picture without
any change.
15 [0033]
Alternatively, the moving image data at 120 fps that
is to be encoded can be, for example, the output from a camera
that outputs the moving image data at 120 fps, or can be the
data provided by changing the frame rate of the output from
20 a camera that outputs the moving image data at a higher frame
rate in a changing process.
[0034]
Fig. 2 illustrates exemplary processes for changing the
frame rate. Fig. 2(a) illustrates the original moving image
25 data that is the output from a camera, the shutter aperture
time ratio is 100%, and the frame rate is 360 fps. Fig. 2
(b) illustrates exemplary moving image data at the frame rate
of 120 fps after the frame rate is changed in a changing process.
In this changing process, the original moving image data is
30 processed in units of three pictures. The image data items
of the three pictures are averaged so that the image data item
16
of a picture in the moving image data at the changed frame
rate is generated. All of the image data items of the three
pictures are used in this changing process, and thus the shutter
aperture time ratio is 100% to the time covered by the original
5 image sequence.
[0035]
Fig. 2(c) illustrates exemplary moving image data at
the frame rate of 120 fps after the frame rate is changed.
In this changing process, the original moving image data is
10 processed in units of three pictures. The image data items
of two pictures among the three pictures are averaged so that
the image data item of a picture in the moving image data at
the changed frame rate is generated. The image data items
of two pictures among the three pictures are used in this
15 changing process, and thus the shutter aperture time ratio
is 66.7% to the time covered by the original image sequence.
[0036]
Fig. 2(d) illustrates exemplary moving image data at
the frame rate of 120 fps after the frame rate is changed.
20 In this changing process, the original moving image data is
processed in units of three pictures. The image data item
of a picture among the three pictures is used as the image
data item of a picture in the moving image data at the changed
frame rate. The image data item of a picture among the three
25 pictures is used in this changing process, and thus the shutter
aperture time ratio is 33.3% to the time covered by the original
image sequence.
[0037]
Fig. 2(e) illustrates exemplary moving image data at
30 the frame rate of 180 fps after the frame rate is changed.
In this changing process, the original moving image data is
17
processed in units of two pictures. The image data item of
a picture of the two pictures is used as the image data item
of a picture in the moving image data at the changed frame
rate. The image data item of a picture of the two pictures
is used in this changing process, and thus the 5 shutter aperture
time ratio is 50% to the time covered by the original image
sequence.
[0038]
Fig. 2(f) illustrates exemplary moving image data at
10 the frame rate of 60 fps after the frame rate is changed. In
this changing process, the original moving image data is
processed in units of six pictures. The image data items of
the first three pictures among the six pictures are averaged
so that the image data item of a picture in the moving image
15 data at the changed frame rate is generated. The image data
items of three pictures among the six pictures are used in
this changing process, and thus the shutter aperture time ratio
is 50% to the time covered by the original image sequence.
[0039]
20 The transport stream TS includes a predetermined number
of video streams. In this embodiment, the transport stream
TS includes a base stream (first video stream) that includes
the encoded image data item of the first picture in each of
the units of two pictures, and an enhancement stream (second
25 video stream) that includes the encoded image data item of
the second picture in each of the units of two pictures.
[0040]
In this embodiment, a plurality of hierarchal layers
is divided into two hierarchal groups. A base stream including
30 the encoded image data items of the pictures in the lowest
hierarchal group, and an enhancement stream including the
18
encoded image data items of the pictures in a hierarchal group
upper than the lowest hierarchal group are generated. In such
a case, the encoded image data item of the first picture in
each of the units of two pictures corresponds to the encoded
image data item of a picture in the lowest 5 hierarchal group.
[0041]
The hierarchy identification information that is used
to identify the hierarchal layer to which each picture belongs
is added to the encoded image data item of each picture in
10 each hierarchal layer. In this embodiment, the hierarchy
identification information (“nuh_temporal_id_plus1” that
means temporal_id) is placed at the header part of an NAL unit
(nal_unit) in each picture. Adding the hierarchy
identification information as described above enables the
15 receiving end to identify the hierarchal layer of each picture
in the NAL unit layer.
[0042]
In this embodiment, the identification information
indicating that the base stream includes a temporally varying
20 element of the enhancement stream, in other words, that the
image data item corresponding to the encoded image data item
of each picture in the base stream is provided from an averaging
process (the first identification information), and the
averaging process information indicating which averaging
25 process is performed are inserted in the encoded image data
items included in the base stream and the enhancement stream.
The information enables the receiving end to easily recognize
that the image data item corresponding to the encoded image
data item of the first picture of each of the units of two
30 pictures in the base stream is provided from an averaging
process, and which averaging process the performed averaging
19
process is.
[0043]
In this embodiment, the identification information (the
second identification information) indicating that the
above-mentioned identification information 5 (the first
identification information) is inserted in the encoded image
data item is inserted in the layers of the transport stream
TS. This identification information is inserted as a
descriptor in a video elementary stream loop corresponding
10 to each video stream under a program map table. This
information enables, for example, the receiving end to identify
that the identification information is inserted in the encoded
image data item from this identification information without
processing the encoded image data item.
15 [0044]
In this embodiment, the information about the shutter
aperture time ratio of the image data item of each picture
in the first moving image data is inserted in the encoded image
data items in the base stream and the enhancement stream. This
20 information enables, for example, the receiving end to
adaptively switch the processes for frame interpolation in
accordance with the information about the shutter aperture
time ratio.
[0045]
25 The reception device 200 receives the transport stream
TS transmitted from the transmission device 100 over the
broadcasting waves. When having a decoding capacity to decode
the moving image data of 60 fps, the reception device 200 decodes
only the base stream included in the transport stream TS to
30 provide the moving image data of 60 fps, and replays the image.
[0046]
20
Alternatively, when having a decoding capacity to decode
the moving image data of 120 fps, the reception device 200
decodes both of the base stream and enhancement stream included
in the transport stream TS and provides the moving image data
of 120 fps (the second moving image data). Then, 5 the reception
device 200 generates the first image data of 120 fps from the
second moving image data of 120 fps in accordance with, for
example, the averaging process information, and replays the
image.
10 [0047]
Note that the reception device 200 interpolates the
moving image data in a temporal direction in an interpolation
process as necessary to provide the moving image data at a
higher frame rate before displaying the image. Then, the
15 reception device 200 displays the image. In such a case, the
reception device 200 switches the processes for frame
interpolation in accordance with the information about the
shutter aperture time ratio inserted in the encoded image data
item of each picture.
20 [0048]
[Configuration of Transmission Device]
Fig. 3 illustrates an exemplary configuration of the
transmission device 100. The transmission device 100
includes a preprocessor 101, an encoder 102, a multiplexer
25 103, and a transmission unit 104. The preprocessor 101 process
moving image data P(n) (the first moving image data) of 120
fps to provide the moving image data Q(n) (the second moving
image data) of 120 fps.
[0049]
30 In this configuration, the preprocessor 101 provides
the moving image data Q(n) at 120 fps by processing the moving
21
image data P(n) of 120 fps in units of consecutive two pictures,
using the image data item provided by averaging the image data
items of the two pictures in a linear or non-linear averaging
process as the image data item of the first picture, and using
the image data item of the second picture of 5 the two pictures
as the image data item of the second picture without any change.
[0050]
Fig. 4 illustrates an exemplary configuration of the
preprocessor 101. The preprocessor 101 includes switch
10 circuits 110 and 114, delay elements 111 and 113 for a frame
period, and an addition circuit 112. The moving image data
P(n) of 120 fps is provided to a movable terminal of the switch
circuit 110. The switch circuit 110 is connected to a fixed
terminal on the “Even” side while an even frame is provided,
15 and connected to a fixed terminal on the “Odd” side while an
odd frame is provided.
[0051]
The image data item of the even frame provided to the
fixed terminal on the “Even” side of the switch circuit 110
20 is delayed by a frame period in the delay element 111 and is
input with a gain of “1/2” to the addition circuit 112. The
image data item of the odd frame provided to the fixed terminal
on the “Odd” side of the switch circuit 110 is input with a
gain of “1/2” to the addition circuit 112.
25 [0052]
The addition circuit 112 calculates a linear average
of consecutive two image data items (the image data items of
an even frame and an odd frame). The image data item output
from the addition circuit 112 is input to the fixed terminal
30 on the “Odd” side of the switch circuit 114. The image data
item of the odd frame provided to the fixed terminal on a side
22
of the switch circuit 110 is delayed by a frame period in the
delay element 113, and is input to the fixed terminal of the
“Even” side of the switch circuit 114.
[0053]
The switch circuit 114 is connected to 5 the fixed terminal
of the “Even” side while an even frame is provided, and is
connected to the fixed terminal of the “Odd” side while an
odd frame is provided. The moving image data Q(n) at 120 fps
in which the image data items of the even frames and image
10 data items of the odd frames provided from the averaging process
are alternately placed is provided from the movable terminal
of the switch circuit 114.
[0054]
Figs. 5(a) and 5(b) schematically illustrate an
15 exemplary relationship between the data (the moving image data
P(n)) input to the preprocessor 101 and the data (the moving
image data Q(n)) output from the preprocessor 101. The image
data items Q0#, Q1, Q2#, Q3, … of pictures in the moving image
data Q(n) are provided from the image data items P0, P1, P2,
20 P3, … of pictures in the moving image data P(n), respectively.
[0055]
Note that, in Figs. 5(a) and 5(b), the numerical value
in the double quotation marks “” is an exemplary pixel value
in the image data item in each frame. For example, the image
25 data item Q0# has a pixel value of “7”, which is the average
of the pixel value “6” of the image data item P0 and the pixel
value “8” of the image data item P1. This value indicates
that the image data Q0# is provided by averaging the image
data item P0 and image data item P1 in a linear averaging process.
30 For example, the image data item Q1 has a pixel value of “8”,
which is the same as the pixel value “8” of the image data
23
item P1. This value indicates that the image data item Q1
is identical to the image data item P1.
[0056]
With reference to Fig. 3 again, the encoder 102
hierarchically encodes the moving image data 5 Q(n) at 120 fps
provided in the preprocessor 101. The encoder 102 classifies
the image data items of the pictures in the moving image data
Q(n) into a plurality of hierarchical layers. The encoder
102 encodes the classified image data items in each of the
10 hierarchical layers and generates a video stream including
the encoded image data items of the pictures in each of the
hierarchical layers. The encoder 102 performs encoding, for
example, in H.264/AVC or H.265/HEVC. In the encoding, the
encoder 102 performs the encoding so that the picture to be
15 referenced (the referenced picture) belongs to the
self-hierarchical layer and/or a layer lower than the
self-hierarchical layer.
[0057]
Fig. 6 illustrates exemplary hierarchical encoding
20 performed in the encoder 102. In this example, the data times
are classified into five hierarchical layers from zero to four.
The image data items of the pictures in each of the hierarchical
layers are encoded. The hierarchical layers are shown on the
vertical axis. Zero to four are set as temporal_id (the
25 hierarchy identification information) placed in the header
part of the NAL unit (nal _unit) included in the encoded image
data items of the pictures in the hierarchical layers from
zero to four. On the other hand, the picture order of
composition (POC) is shown on the horizontal axis. The earlier
30 display times are shown on the left side of the horizontal
axis, and the later display times are shown on the right side.
24
[0058]
Fig. 7(a) illustrates an exemplary structure (Syntax)
of an NAL unit header. Fig. 7(b) illustrates the contents
(Semantics) of the main parameters in the exemplary structure.
The one-bit field of “Forbidden_zero_bit” requires 5 zero. The
six-bit field of “NaL_unit_type” indicates the type of the
NAL unit. The six-bit field of “Nuh_layer_id” is assumed as
zero herein. The three-bit field of “Nuh_temporal_id_plus1”
indicates temporal_id and takes the value increased by one
10 (one to seven).
[0059]
With reference to Fig. 6 again, each rectangular frame
is a picture and the number is the order in which the pictures
are encoded, in other words, the order of encoding (the order
15 of decoding on the receiving end). In the example of Fig.
6, 16 pictures “2” to “17” form a sub picture group (Sub group
of pictures). The picture “2” is the first picture of the
sub picture group. The picture “1” is a picture of the previous
sub picture group. Some sub picture groups gather and form
20 a group of pictures (GOP).
[0060]
The encoded image data items of the first pictures of
the GOP includes NAL units AUD, VPS, SPS, PPS, PSEI, SLICE,
SSEI, and EOS as illustrated in Fig. 8. On the other hand,
25 the pictures other than the first pictures of GOP include the
NAL units AUD, PPS, PSEI, SLICE, SSEI, and EOS. VPS can be
transmitted once in a sequence (GOP) together with SPS while
PPS can be transmitted in every picture. Note that EOS does
not necessarily exist.
30 [0061]
With reference to Fig. 6 again, the solid arrows show
25
the reference relationships among the pictures when the
pictures are encoded. For example, the picture “2” is a P
picture and is encoded with reference to the picture “1”. The
picture “3” is a B picture that can be referenced and is encoded
with reference to the pictures “1” and “5 2”. Similarly, the
other pictures are encoded with reference to a near picture
in the display order. Note that the pictures on the top
hierarchal layer are not referenced by the other pictures.
[0062]
10 With reference to Fig. 3 again, the encoder 102 generates
a predetermined number of video streams. In this embodiment,
the encoder 102 generates a base stream that includes the
encoded image data items of even frames, and an enhancement
streams that include the encoded image data items of odd frames.
15 Then, a plurality of hierarchal layers is divided into two
groups of hierarchal layers. A base stream that includes the
encoded image data items of the pictures on the lowest
hierarchal group and an enhancement stream that includes the
encoded image data items of the pictures on a hierarchal group
20 upper than the lowest hierarchal group are generated. The
encoded image data items of even frames described above
correspond to the encoded image data items of the pictures
on the lowest hierarchal group.
[0063]
25 In the exemplary hierarchal encoding in Fig. 6, the
encoder 102 classifies the hierarchal layers into two
hierarchal groups, for example, by classifying the hierarchal
layers 0 to 3 into the lowest hierarchal group and the hierarchal
layer 4 into the hierarchal group positioned on the lowest
30 hierarchal group. The hierarchal groups are divided with a
dashed line in Fig. 6. In this division, the video stream
26
including the encoded image data items of the pictures on the
lowest hierarchal group is determined as the base stream, of
which stream type is “0x24”. The video stream including the
encoded image data of the pictures on the hierarchal group
positioned on the lowest hierarchal group 5 is determined as
the enhancement stream, of which stream type is “0x25” that
is newly defined.
[0064]
The stream types are included in the identification
10 information used to identify whether each of the predetermined
number of video streams is a base stream or an enhancement
stream. The stream type is inserted in a layer in the transport
stream TS. In other words, the stream type is inserted in
a video elementary stream loop (Video ES loop) placed under
15 the program map table (PMT), corresponding to each of the
predetermined number of video streams.
[0065]
As described above, the encoder 102 inserts the
identification information (the first identification
20 information) and the averaging process information into the
encoded image data item of each picture included in the base
stream and the enhancement stream. The first identification
information indicates that the image data item corresponding
to the encoded image data item of each picture included in
25 the base stream is provided from an averaging process. The
averaging process information indicates which averaging
process is performed. As described above, the encoder 102
further inserts the information about the shutter aperture
time ratio of the image data item of each picture in the moving
30 image data P(n) into the encoded image data item of each picture
included in the base stream and the enhancement stream (see
27
Figs. 2(b) to 2(d)).
[0066]
The encoder 102 inserts the capturing speed information
SEI (Capturing_speed_information SEI) that is newly defined
as one of the prefixes 5 SEI (Prefix_SEI).
[0067]
Fig. 9(a) illustrates an exemplary structure (Syntax)
of the interface (I/F) used to insert the capturing speed
information SEI. The field of “uuid_iso_iec_11578” has an
10 UUID value indicated with “ISO/IEC 11578:1996 Annex A.”. The
“capturing_speed_information()” is inserted into the field
of “user_data_payload_byte”.
[0068]
Fig. 9(b) illustrates an exemplary structure (Syntax)
15 of the “capturing_speed_information()”. Fig. 10 illustrates
the contents (Semantics) of the main information in the
exemplary structure. The ID of predetermined user data is
attached to the 16-bit field of “userdata_id”. The eight-bit
field of “capturing_speed_information_length” indicates the
20 number of bytes of “capturing_speed_information” (the number
is counted from the next element of the element).
[0069]
The five-bit field of
“shutter_ratio_denominator_minus1” indicates the value of
25 the denominator of the shutter aperture time ratio when an
image to be input to the encoder is generated from the source
data that the camera takes. The five-bit field of
“shutter_ratio_numerator_minus1” indicates the value of the
numerator of the shutter aperture time ratio when an image
30 to be input to the encoder is generated from the source data
that the camera takes. In this example, the shutter aperture
28
time ratio is indicated with (SRN + 1)/(SRD + 1).
[0070]
The one-bit field of
“cross_stream_smoothing_base_flag” indicates whether the
stream is encoded so that a plurality of streams 5 included in
a service has temporal scalability and the base stream (base
stream) on the lowest hierarchal layer includes a temporally
varying element of a stream on a layer other than the lowest
hierarchal layer. In other words, the one-bit field indicates
10 whether the image data item corresponding to the encoded image
data item of each picture included in the base stream is provided
from an averaging process. The “1” means that the base stream
includes a temporally varying element of a stream on a layer
other than the lowest hierarchal layer. The “0 ” means that
15 the base stream does not include a temporally varying element
of a stream on a layer other than the lowest hierarchal layer.
[0071]
When the one-bit field of
“cross_stream_smoothing_base_flag” is “1”, the eight-bit
20 field of “smoothing_type” exists. The field indicates a
method for smoothing, namely, for an averaging process. The
“0x00” indicates that a linear average is calculated while
the other values indicate that a non-linear average is
calculated. The procedures of the averaging process are
25 designated depending on the type. Note that Figs. 5(a) and
5(b) are exemplary calculation of a linear average. When a
non-linear average is calculated, for example, the gain of
the picture P0 differs from the gain of the picture P1.
[0072]
30 With reference to Fig. 3 again, the multiplexer 103
packetizes the video stream generated in the encoder 102 as
29
a Packetized Elementary Stream (PES) packet. The multiplexer
103 multiplexes the video stream by packetizing the video
stream as a transport packet. Then, the multiplexer 103
provides the transport stream TS as a multiplexed stream. In
this embodiment, the transport stream TS includes 5 a base stream
and an enhancement stream.
[0073]
The multiplexer 103 inserts the identification
information (the second identification information) into a
10 layer in the transport stream TS. The second identification
information indicates that the identification information
(the first identification information) is inserted in the
encoded image data items included in the base stream and the
enhancement stream. The identification information is
15 inserted as a descriptor in a video elementary stream loop
placed under the program map table, corresponding to each video
stream.
[0074]
The multiplexer 103 inserts a shutter speed information
20 descriptor (Shutter_speed_information_descriptor) that is
newly defined together with an HEVC descriptor
(HEVC_descriptor). Fig. 11(a) illustrates an exemplary
structure (Syntax) of the shutter speed information descriptor.
Fig. 11(b) illustrates the contents (Semantics) of the main
25 information in the exemplary structure.
[0075]
The eight-bit field of
“Shutter_speed_information_descriptor_tag” indicates the
type of the descriptor. In this example, the eight-bit field
30 indicates that the descriptor is a shutter speed information
descriptor. The eight-bit field of
30
“Shutter_speed_information_descriptor_length” indicates
the length (size) of the descriptor. The eight-bit field
indicates the number of subsequent bytes as the length of the
descriptor. In this example, the eight-bit field indicates
that the 5 length is a byte.
[0076]
The one-bit field of
“Shutter_speed_information_SEI_existed” indicates whether
the capturing speed information SEI
10 (Capturing_speed_information SEI) is encoded in the video
stream. The “1” indicates that it is ensured that the SEI
is encoded in the video stream. The “0” indicates that it
is not ensured that the SEI is encoded in the video stream.
[0077]
15 Fig. 12 illustrates an exemplary structure (Syntax) of
an HEVC descriptor (HEVC_descriptor). The eight-bit field
of “descriptor_tag” indicates the type of the descriptor. In
this example, the eight-bit field indicates that the descriptor
is an HEVC descriptor. The eight-bit field of
20 “descriptor_length” indicates the length (size) of the
descriptor. The eight-bit field indicates the number of
subsequent bytes as the length of the descriptor.
[0078]
The eight-bit field of “level_idc” indicates a level
25 designation value of the bit rate. When
“temporal_layer_subset_flag = 1” holds, the five-bit field
of “temporal_id_min” and the five-bit field of
“temporal_id_max” exist. The “temporal_id_min” indicates
the value of temporal_id on the lowest hierarchal layer of
30 the hierarchically encoded data items included in the
corresponding video stream. The “temporal_id_max” indicates
31
the value of temporal_id on the top hierarchal layer of the
hierarchically encoded data items included in the
corresponding video stream.
[0079]
Fig. 13 illustrates an exemplary 5 formation of the
transport stream TS. The transport stream TS includes two
video streams, a base stream and an enhancement stream. In
other words, a PES packet “video PES1” that is the base stream
exists and a PES packet “video PES2” that is the enhancement
10 stream exists in the exemplary formation. The capturing speed
information SEI is inserted in the encoded image data item
of each picture contained in the PES packets “video PES1” and
“video PES2”.
[0080]
15 A Program Map Table (PMT) is included as a type of Program
Specific Information (PSI) in the transport stream TS. The
PSI is the information that indicates which program each
elementary stream included in the transport stream belongs
to.
20 [0081]
A program loop (Program loop) that describes the
information about the entire program exists in the PMT. An
elementary stream loop including the information about each
video stream also exists in the PMT. In the exemplary formation,
25 the video elementary stream loop “video ES1 loop” corresponding
to the base stream exists while the video elementary stream
loop “video ES2 loop” corresponding to the enhancement stream
exists.
[0082]
30 The information about the base stream (video PES1), for
example, the type of the stream, and the packet identifier
32
(PID) is placed in the “video ES1 loop”, together with a
descriptor that describes the information about the video
stream. The type of stream is “0x24” that indicates the base
stream. The HEVC descriptor and shutter speed information
descriptor described above are inserted 5 as a descriptor.
[0083]
The information about the enhancement stream (video
PES2), for example, the type of the stream, and the packet
identifier (PID) is placed in the “video ES2 loop”, together
10 with a descriptor that describes the information about the
video stream. The type of stream indicates the enhancement
stream and, for example, is “0x25” that is newly defined. The
HEVC descriptor and shutter speed information descriptor
described above are inserted as a descriptor.
15 [0084]
With reference to Fig. 3 again, the transmission unit
104 modulates the transport stream TS, for example, with a
modulation method appropriate for the broadcasting, for
example, with QPSK/OFDM modulation. The transmission unit
20 104 transmits an RF modulation signal from a transmission
antenna.
[0085]
The operation of the transmission device 100 illustrated
in Fig. 3 will briefly be described. The moving image data
25 P(n) (the first moving image data) at 120 fps is input as the
moving image data at a high frame rate to the preprocessor
101. The preprocessor 101 processes the moving image data
P(n) at 120 fps and provides the moving image data Q(n) (the
second moving image data) at 120 fps.
30 [0086]
In this process, the preprocessor 101 provides the moving
33
image data Q(n) at 120 fps by processing the moving image data
P(n) in units of consecutive two pictures, using the image
data item provided by averaging the image data items of the
two pictures in a linear or non-linear averaging process as
the image data item of the first picture, and 5 using the image
data item of the second picture of the two pictures as the
image data item of the second picture without any change (see
Fig. 5(a) and 5(b)).
[0087]
10 The moving image data Q(n) at 120 fps provided in the
preprocessor 101 is provided to the encoder 102. The encoder
102 hierarchically encodes the moving image data Q(n). In
other words, the encoder 102 classifies the image data items
of the pictures in the moving image data Q(n) into a plurality
15 of hierarchal layers to encode the image data items, and
generates a video stream including the encoded image data items
of the pictures in each of the hierarchal layers. The
hierarchical encoding is performed so that the picture to be
referenced belongs to the self-hierarchal layer and/or a
20 hierarchal layer lower than the self-hierarchal layer.
[0088]
The encoder 102 generates a base stream that includes
the encoded image data items of even frames, and an enhancement
stream that includes the encoded image data items of odd frames.
25 In the generation, the hierarchal layers are divided into two
hierarchal groups. Thus, the encoder 102 generates the base
stream, which includes the encoded image data items of the
pictures in the lowest hierarchal group, and the enhancement
stream, which includes the encoded image data items of the
30 pictures in the hierarchal group upper than the lowest group.
The encoded image data items of even frames described above
34
correspond to the encoded image data items of the pictures
in the lowest hierarchal group.
[0089]
The encoder 102 inserts the identification information
(the first identification information) 5 and the averaging
process information into the encoded image data item of each
picture included in the base stream and the enhancement stream.
The first identification information indicates that the image
data item corresponding to the encoded image data item of each
10 picture included in the base stream is provided from an
averaging process. The averaging process information
indicates which averaging process is performed. The encoder
102 further inserts the information about the shutter aperture
time ratio of the image data item of each picture included
15 in the moving image data P(n) into the encoded image data item
of each picture included in the base stream and the enhancement
stream. Specifically, the encoder 102 inserts the capturing
speed information SEI, which is newly defined, as one of the
prefixes SEI.
20 [0090]
The video stream generated in the encoder 102 is provided
to the multiplexer 103. The multiplexer 103 packetizes the
video stream generated in the encoder 102 as a PES packet,
and multiplexes the video stream as a transport packet to
25 provide the transport stream TS as a multiplexed stream. The
transport stream TS includes a base stream and an enhancement
stream.
[0091]
The multiplexer 103 inserts the identification
30 information (the second identification information) into a
layer in the transport stream TS. The second identification
35
information indicates that the identification information
(the first identification information) is inserted in the
encoded image data items included in the base stream and the
enhancement stream. The identification information is
inserted as a descriptor into a video elementary 5 stream loop
placed under the program map table, corresponding to each video
stream. Specifically, the multiplexer 103 inserts a shutter
speed information descriptor, which is newly defined, together
with an HEVC descriptor.
10 [0092]
The transport stream TS generated in the multiplexer
103 is transmitted to the transmission unit 104. The
transmission unit 104 modulates the transport stream TS in
a modulation method appropriate for broadcasting, for example,
15 in QPSK/OFDM modulation, and transmits an RF modulation signal
from a transmission antenna.
[0093]
[Configuration of Reception Device]
Fig. 14 illustrates an exemplary configuration of a
20 reception device 200A that has a decoding capacity to decode
the moving image data at 120 fps. The reception device 200A
includes a reception unit 201, a demultiplexer 202, a decoder
203, a postprocessor 204, and a display processor 205.
[0094]
25 The reception unit 201 demodulates the RF modulation
signal received at a reception antenna to obtain the transport
stream TS. As illustrated in Fig. 15(b), the demultiplexer
202 extracts the base stream and the enhancement stream from
the transport stream TS by filtering PID. Then, the
30 demultiplexer 202 integrates the base stream and the
enhancement stream into a video stream in accordance with the
36
decoding timing information, and provides the video stream
to the decoder 203.
[0095]
The demultiplexer 202 extracts the section information
from a layer in the transport stream TS, 5 and transmits the
section information to a control unit (not illustrated). At
that time, the shutter speed information descriptor is also
extracted. From the information, the control unit recognizes
that the capturing speed information SEI is inserted in the
10 encoded image data items included in the base stream and the
enhancement stream.
[0096]
The decoder 203 decodes the video stream provided from
the demultiplexer 202 to provide the moving image data Q’(n)
15 at 120 fps that corresponds to the moving image data Q(n) in
the transmission device 100. The decoder 203 extracts the
capturing speed information SEI inserted in the encoded image
data item of each picture included in the video stream, and
transmits the capturing speed information SEI to the control
20 unit (not illustrated).
[0097]
From the information, the control unit recognizes that
the image data item corresponding to the encoded image data
item of each picture included in the base stream is provided
25 from an averaging process. The control unit further
recognizes which averaging process is performed. The control
unit further recognizes the shutter aperture time ratio of
the image data item of each picture included in the moving
image data Z(n) at 120 fps provided in the postprocessor 204
30 to be described below.
[0098]
37
The postprocessor 204 processes the moving image data
Q’(n) at 120 fps in an inverse process to the process with
the preprocessor 101 in the transmission device 100 to provide
the moving image data Z(n) at 120 fps that corresponds to the
moving image data P(n). The postprocessor 5 204 performs the
process with reference to the averaging process information
under the control by the control unit when the image data item
corresponding to the encoded image data item of each picture
included in the base stream is provided from the averaging
10 process.
[0099]
Fig. 16 illustrates an exemplary configuration of the
postprocessor 204. The postprocessor 204 includes switch
circuits 240 and 244, delay elements 241 and 243 for a frame
15 period, and an addition circuit 242. The moving image data
Q’(n) at 120 fps is provided to a movable terminal of the switch
circuit 240. The switch circuit 240 is connected to a fixed
terminal on the “Even” side while an even frame is provided,
and is connected to a fixed terminal on the “Odd” side while
20 an odd frame is provided.
[0100]
The image data item of an even frame provided at the
fixed terminal on the “Even” side of the switch circuit 240
is delayed by a frame period in the delay element 241, and
25 is input with a gain of “1” to the addition circuit 242. The
image data item of an odd frame provided at the fixed terminal
on the “Odd” side of the switch circuit 110 is input with a
gain of “- 1/2” to the addition circuit 242.
[0101]
30 The addition circuit 242 subtracts the halved image data
item of the odd frame from the image data item of the even
38
frame in each of the units of consecutive two image data items
(the image data items of an even frame and an odd frame). The
image data item output from the addition circuit 242 is input
to the fixed terminal on the “Odd” side of the switch circuit
244 with a gain of “2”. The image data item 5 of an odd frame
provided at the fixed terminal on the side of the switch circuit
240 is delayed by a frame period in the delay element 243,
and is input to the fixed terminal on the “Even” side of the
switch circuit 244.
10 [0102]
The switch circuit 244 is connected to the fixed terminal
of the “Even” side while an even frame is provided, and is
connected to the fixed terminal of the “Odd” side while an
odd frame is provided. The moving image data Z(n) at 120 fps
15 in which the image data items of the even frames and the image
data items of the odd frames, which are restored to the state
before the averaging process, are alternately placed is
provided from the movable terminal of the switch circuit 244.
[0103]
20 Figs. 5(b) and 5(c) schematically illustrate the
relationship between the data input to the postprocessor 204
(the moving image data Q’(n)) and the data output from the
postprocessor 204 (the moving image data Z(n)). The image
data items Z0, Z1, Z2, Z3, … of the pictures in the moving
25 image data Q(n) are provided from the image data items Q0#,
Q1, Q2#, Q3, … of the pictures in the moving image data Q’(n).
[0104]
Note that, in Figs. 5(b) and 5(c), the numerical value
in the double quotation marks “” is an exemplary pixel value
30 in the image data item in each frame. For example, the image
data items Z0, Z2, and Z4 have pixel values of “6”, “12”, and
39
“8”, respectively, which are identical to the pixel values
of “6”, “12”, and “8” of the image data items P0, P2, and P4.
The values indicate that the image data items of the frames
are restored to the state before the averaging process.
5 [0105]
The display processor 205 interpolates the moving image
data Z(n) at 120 fps provided in the postprocessor 204 in a
temporal direction, namely, in a frame interpolation process
to provide the moving image data at a frame rate higher than
10 120 fps. The display processor 205 switches the processes
for frame interpolation in accordance with the shutter aperture
time ratio of the image data item of each pictures in the moving
image data Z(n) under the control by the control unit.
[0106]
15 This switching decreases the block size to be processed
in a motion predicting process, for example, when the ratio
of the shutter speed to the frame frequency decreases (when
the shutter speed relatively increases), in other words, as
the shutter aperture time ratio decreases. This switches the
20 prediction algorithm to a prediction algorithm with a higher
degree of accuracy. This enables the display with a
high-definition stable quality.
[0107]
The operation of the reception device 200 illustrated
25 in Fig. 14 will briefly be described. The reception unit 201
demodulates the RF modulation signal received at a reception
antenna to provide the transport stream TS. The transport
stream TS is transmitted to the demultiplexer 202. The
demultiplexer 202 extracts the base stream and the enhancement
30 stream from the transport stream TS by filtering PID. Then,
the demultiplexer 202 integrates the base stream and the
40
enhancement stream into a video stream in accordance with the
decoding timing information, and provides the video stream
to the decoder 203.
[0108]
The demultiplexer 202 extracts the section 5 information
from a layer in the transport stream TS, and transmits the
section information to a control unit (not illustrated). At
that time, the shutter speed information descriptor is also
extracted. From this information, the control unit
10 recognizes that the capturing speed information SEI is inserted
in the encoded image data items included in the base stream
and the enhancement stream.
[0109]
The decoder 203 decodes the video stream provided from
15 the demultiplexer 202 to provide the moving image data Q’(n)
at 120 fps that corresponds to the moving image data Q(n) in
the transmission device 100. The decoder 203 extracts the
capturing speed information SEI inserted in the encoded image
data item of each picture included in the video stream, and
20 transmits the capturing speed information SEI to the control
unit (not illustrated).
[0110]
From this information, the control unit recognizes that
the image data item corresponding to the encoded image data
25 item of each picture included in the base stream is provided
from an averaging process. The control unit further
recognizes which averaging process is performed. The control
unit further recognizes the shutter aperture time ratio of
the image data item of each picture included in the moving
30 image data Z(n) at 120 fps provided in the postprocessor 204
to be described below.
41
[0111]
The moving image data of 120 fps provided in the decoder
203 is provided to the postprocessor 204. The postprocessor
204 processes the moving image data Q’(n) at 120 fps in an
inverse process to the process with the preprocessor 5 101 in
the transmission device 100 under the control by the control
unit to provide the moving image data Z(n) at 120 fps that
corresponds to the moving image data P(n). At that time, the
control unit recognizes, in accordance with the capturing speed
10 information SEI, that the image data item corresponding to
the encoded image data item of each picture included in the
base stream is provided from the averaging process. The
postprocessor 204 performs a process with reference to the
averaging process information.
15 [0112]
The moving image data Z(n) at 120 Hz provided in the
postprocessor 204 is provided to the display processor 205.
The display processor 205 interpolates the moving image data
Z(n) at 120 fps provided in the postprocessor 204 in a temporal
20 direction, namely, in a frame interpolation process to provide
the moving image data at a frame rate higher than 120 fps.
The image data is provided to the display unit so that the
image is displayed.
[0113]
25 Fig. 17 illustrates an exemplary configuration of a
reception device 200B that has a decoding capacity to decode
the moving image data at 60 fps. In Fig. 17, the components
corresponding to the components in Fig. 14 are put with the
same reference signs, and the descriptions of the details will
30 properly be omitted. The reception device 200B includes a
reception unit 201, a demultiplexer 202, a decoder 203, and
42
a display processor 205.
[0114]
The reception unit 201 demodulates the RF modulation
signal received at a reception antenna to obtain the transport
stream TS. As illustrated in Fig. 15(a), 5 the demultiplexer
202 extracts only the base stream from the transport stream
TS by filtering PID to provide the base stream to the decoder
203.
[0115]
10 The decoder 203 decodes the base stream provided from
the demultiplexer 202 to provide the moving image data at 60
fps. The display processor 205 interpolates the moving image
data at 60 fps provided in the decoder 203 in a temporal direction,
namely, in a frame interpolation process to provide the moving
15 image data at a frame rate higher than 60 fps. The image data
is provided to the display unit so that the image is displayed.
[0116]
As described above, the transmitting end in the
transmission and reception system 10 illustrated in Fig. 1
20 does not transmit the moving image data P(n) at 120 fps that
is the moving image data at a high frame rate as it is. The
transmitting end transmits the image data in which the image
data item provided by averaging the image data items in each
of the units of two pictures in an averaging process is used
25 as the image data item of the first picture in each of the
units of two pictures.
[0117]
Thus, for example, when having a decoding capacity to
decode the moving image data at 60 fps (see Fig. 17), the
30 receiving end selects only the encoded image data item of the
first picture in each of the units of two pictures, and decodes
43
the image data items to provide the moving image data at 60
fps. This enables the receiving end to display a smooth image.
This can prevent a frame interpolation process with low load
calculation from causing a problem on the image quality.
5 [0118]
The receiving end in the transmission and reception
system 10 illustrated in Fig. 1 restores the image data item
of the first picture in each of the units of two pictures to
the image data item of the first picture before the averaging
10 process. This enables the receiving end to display a smooth
image with a high degree of sharpness at a high frame rate
when the receiving end has a decoding capacity to decode, for
example, the moving image data at 120 fps (see Fig. 14).
[0119]
15 <2. Exemplary Variation>
Note that, although the frame rate of the entire data
is 120 fps while the frame rate of the base stream is 60 fps
in the embodiment, the combination of the frame rates is not
limited to the embodiment. For example, the combination of
20 the frame rates of 100 fps and 50 fps can be processed in a
similar manner.
[0120]
In the embodiment, the ratio of the frame rate of the
entire data to the frame rate of the base stream is 2 : 1,
25 in other words, N = 2 holds. However, the present technology
is applicable not only to the ratio 2 : 1, but also to another
ratio in a similar manner.
[0121]
Figs. 18(a) and 18(b) schematically illustrate an
30 exemplary relationship between the data input to the
preprocessor 101 (the moving image data P(n)) and the data
44
output from the preprocessor 101 (the moving image data Q(n))
when the ratio is 4 : 1, in other words, N = 4 holds. In this
example, a linear average is calculated. The image data items
Q0#, Q1, Q2, Q3, Q4#, Q5, Q26#, Q7 … of pictures in the moving
image data Q(n) are provided from the image 5 data items P0,
P1, P2, P3, P4, P5, P6, P7… of pictures in the moving image
data P(n), respectively.
[0122]
Note that, in Figs. 18(a) and 18(b), the numerical value
10 in the double quotation marks “” is an exemplary pixel value
in the image data item in each frame. For example, the image
data item Q0# has a pixel value of “9”, which is the average
of the pixel values “6”, “8”, “12”, and “10” of the image data
items P0, P1, P2, and P3. This value indicates that the image
15 data Q0# is provided by linearly averaging the image data items
P0, P1, P2, and P3. For example, the image data items Q1,
Q2, and Q3 have pixel values of “8”, “12”, and “10”, which
are the same as the pixel values “8”, “12”, and “10” of the
image data items P1, P2, and P3. The values indicate that
20 the image data items Q1, Q2, and Q3 are identical to the image
data items P1, P2, and P3.
[0123]
Figs. 18(b) and 18(c) schematically illustrate an
exemplary relationship between the data input to the
25 postprocessor 204 (the moving image data Q’(n)) and the data
output from the postprocessor 204 (the moving image data Z(n))
when the ratio is 4 : 1, in other words, N = 4 holds. The
image data items Z0, Z1, Z2, Z3, Z4, Z5, Z6, Z7 … of pictures
in the moving image data Q(n) are provided from the image data
30 items Q0#, Q1, Q2, Q3, Q4#, Q5, Q26#, Q7… of pictures in the
moving image data Q’(n), respectively.
45
[0124]
Note that, in Figs. 18(b) and 18(c), the numerical value
in the double quotation marks “” is an exemplary pixel value
in the image data item in each frame. For example, the image
data items Z0 and Z4 have pixel values 5 of “6” and “8”,
respectively, which are identical to the pixel values of “6”
and “8” of the image data items P0 and P4. The values indicate
that the image data items of the frames are restored to the
state before the averaging process.
10 [0125]
In the embodiment, the transmission and reception system
10 includes the transmission device 100 and the reception
device 200. However, the configuration of the transmission
and reception system to which the present technology is
15 applicable is not limited to the embodiment. For example,
the reception device 200 can include a set-top box and a monitor
that are connected via a digital interface such as a
High-Definition Multimedia Interface (HDMI). Note that
“HDMI” is a registered trademark.
20 [0126]
The container is a transport stream (MPEG-2 TS) in the
embodiment. However, the present technology is similarly
applicable to a system configured to use a network such as
the Internet to distribute data to the receiving terminals.
25 A container in an MP4 format or another format is often used
for distribution via the Internet. In other words, various
containers such as a transport stream (MPEG-2 TS) used in
digital broadcasting standards, or a container in an MP4 format
used for distribution via the Internet can be used as the
30 container.
[0127]
46
Alternatively, the present technology can have the
following configuration.
(1)
A transmission device including:
an image processing unit that provides 5 second moving
image data at a predetermined frame rate by processing first
moving image data at the predetermined frame rate in units
of consecutive N pictures, using an image data item provided
by averaging image data items of the N pictures in an averaging
10 process as an image data item of a first picture, and using
image data items of second to Nth pictures of the N pictures
as image data items of second to Nth pictures without any change,
the N being an integer larger than or equal to two;
an image encoding unit that generates a video stream
15 by encoding the image data items of the pictures in the second
moving image data provided in the image processing unit; and
a transmission unit that transmits the video stream
generated in the image encoding unit.
(2)
20 The transmission device according to (1), wherein the
image encoding unit generates a first video stream that
includes the encoded image data item of the first picture in
each of the units of N pictures, and a predetermined number
of second video streams that include the encoded image data
25 items of the second to Nth pictures in each of the units of
N pictures.
(3)
The transmission device according to (2), wherein
the image encoding unit classifies the image data items
30 of the pictures in the second moving image data provided in
the image processing unit into a plurality of hierarchal layers,
47
encodes the classified image data items of the pictures in
each of the hierarchal layers, divides the hierarchal layers
into a predetermined number of hierarchal groups, and generates
a predetermined number of video streams, the video streams
include the encoded image data items of the 5 pictures in each
of the divided hierarchal groups, and
a lowest hierarchal group includes the encoded image
data item of the first picture in each of the units of N pictures.
(4)
10 The transmission device according to (2) or (3), wherein
the image encoding unit inserts first identification
information into the encoded image data items in the first
video stream, and the first identification information
indicates that the first video stream includes a temporally
15 varying element of the second video stream.
(5)
The transmission device according to any of (2) to (4),
wherein
the image encoding unit further inserts information
20 about the averaging process into the encoded image data items
in the first video stream.
(6)
The transmission device according to (4) or (5), wherein
the transmission unit transmits the container being in
25 a predetermined format and including the video streams,
the transmission device further including:
an identification information inserting unit that
inserts second identification information into a layer in a
container, the second identification information indicating
30 that the first identification information is inserted in the
encoded image data items in the first video stream.
48
(7)
The transmission device according to any of (1) to (6),
wherein
the image encoding unit inserts information about a
shutter aperture time ratio of the image 5 data item of each
of the pictures in the first moving image data at the
predetermined frame rate into the encoded image data items
included in the video stream.
(8)
10 A transmission method including:
an image processing step of providing second moving image
data at a predetermined frame rate by processing first moving
image data at the predetermined frame rate in units of
consecutive N pictures, using an image data item provided by
15 averaging image data items of the N pictures in an averaging
process as an image data item of a first picture, and using
image data items of second to Nth pictures of the N pictures
as image data items of second to Nth pictures without any change,
the N being an integer larger than or equal to two;
20 an image encoding step of generating a video stream by
encoding the image data items of the pictures in the second
moving image data provided in the image processing step; and
a transmission step of transmitting the video stream
generated in the image encoding step.
25 (9)
A reception device including:
a reception unit that receives a video stream, the video
stream being generated by encoding an image data item of each
picture in moving image data at a predetermined frame rate;
30 and
a processing unit that processes the video stream
49
received with the reception unit,
wherein the moving image data at the predetermined frame
rate is second moving image data at the predetermined frame
rate, and the second moving image data is provided by processing
first moving image data at the predetermined 5 frame rate in
units of consecutive N pictures, using an image data item
provided by averaging image data items of the N pictures in
an averaging process as an image data item of a first picture,
and using image data items of second to Nth pictures of the
10 N pictures as image data items of second to Nth pictures without
any change, and the N is an integer larger than or equal to
two.
(10)
The reception device according to (9), wherein
15 the video stream received with the reception unit
includes a first video stream that includes the encoded image
data item of the first picture in each of the units of N pictures,
and a predetermined number of second video streams that include
the encoded image data items of the second to Nth pictures
20 in each of the units of N pictures.
(11)
A reception device including:
a reception unit that receives a video stream generated
by encoding an image data item of each picture in second moving
25 image data at a predetermined frame rate, the second moving
image data being provided by processing first moving image
data at the predetermined frame rate in units of consecutive
N pictures, using an image data item provided by averaging
image data items of the N pictures in an averaging process
30 as an image data item of a first picture, and using image data
items of second to Nth pictures of the N pictures as image
50
data items of second to Nth pictures without any change, the
N being an integer larger than or equal to two;
an image decoding unit that provides the second moving
image data at the predetermined frame rate by decoding the
video stream received with the reception 5 unit; and
an image processing unit that provides the first moving
image data by processing the second moving image data provided
in the image decoding unit.
(12)
10 The reception device according to (11), wherein
the video stream received with the reception unit
includes a first video stream that includes the encoded image
data item of the first picture in each of the units of N pictures,
and a predetermined number of second video streams that include
15 the encoded image data items of the second to Nth pictures
in each of the units of N pictures.
(13)
The reception device according to (11) or (12), further
including:
20 a display processing unit that provides third moving
image data at a frame rate higher than the predetermined frame
rate by interpolating the first moving image data provided
in the image processing unit in a temporal direction.
(14)
25 The reception device according to (13), wherein
information about a shutter aperture time ratio of an
image data item of each picture in the first moving image data
is inserted in an encoded image data item of each picture in
the video stream received with the reception unit, and
30 the display processing unit switches processes for frame
interpolation in accordance with the shutter aperture time
51
ratio information.
(15)
A reception method including:
a reception step of receiving a video stream generated
by encoding an image data item of each picture 5 in second moving
image data at a predetermined frame rate, the second moving
image data being provided by processing first moving image
data at the predetermined frame rate in units of consecutive
N pictures, using an image data item provided by averaging
10 image data items of the N pictures in an averaging process
as an image data item of a first picture, and using image data
items of second to Nth pictures of the N pictures as image
data items of second to Nth pictures without any change, the
N being an integer larger than or equal to two;
15 an image decoding step of providing the second moving
image data at the predetermined frame rate by decoding the
video stream received with the reception step; and
an image processing step of providing the first moving
image data by processing the second moving image data provided
20 in the image decoding step.
(16)
A transmission device including:
an image encoding unit that generates a video stream
by encoding an image data item of each picture in moving image
25 data at a predetermined frame rate; the image encoding unit
that inserts information about a shutter aperture time ratio
of an image data item of each picture in the first moving image
data at the predetermined frame rate into the encoded image
data item of each picture in the video stream; and
30 a transmission unit that transmits the video stream
generated in the image encoding unit.
52
(17)
A reception device including:
a reception unit that receives a video stream provided
by encoding an image data item of each picture in moving image
data at a predetermined 5 frame rate;
an image decoding unit that provides the moving image
data at the predetermined frame rate by decoding the video
stream received with the reception unit; and
a display processing unit that provides third moving
10 image data at a frame rate higher than the predetermined frame
rate by interpolating the first image data provided in the
image decoding unit in a temporal direction,
wherein information about a shutter aperture time ratio
of an image data item of each picture in the moving image data
15 at the predetermined frame rate is inserted in an encoded image
data item of each picture in the video stream received with
the reception unit, and
the display processing unit switches processes for frame
interpolation in accordance with the shutter aperture time
20 ratio information.
[0128]
A main feature of the present technology is to transmit
the image data item provided by averaging the image data items
of N pictures as the image data item of the first picture in
25 each of the units of N pictures in order to transmit the moving
image data at a high frame rate. This can secure the
compatibility with a conventional receiver that is compatible
with a normal frame rate (see Figs. 3 to 5).
30 REFERENCE SIGNS LIST
[0129]
53
10 Transmission and reception system
100 Transmission device
101 Preprocessor
102 Encoder
5 103 Multiplexer
104 Transmission unit
110, 114 Switch circuit
111, 113 Delay element
112 Addition circuit
10 200, 200A, 200B Reception device
201 Reception unit
202 Demultiplexer
203 Decoder
204 Postprocessor
15 205 Display processor
240, 244 Switch circuit
241, 243 Delay element
242 Addition circuit
54
CLAIMS
1. A transmission device comprising:
an image processing unit that provides second moving
image data at a predetermined frame rate by 5 processing first
moving image data at the predetermined frame rate in units
of consecutive N pictures, using an image data item provided
by averaging image data items of the N pictures in an averaging
process as an image data item of a first picture, and using
10 image data items of second to Nth pictures of the N pictures
as image data items of second to Nth pictures without any change,
the N being an integer larger than or equal to two;
an image encoding unit that generates a video stream
by encoding the image data items of the pictures in the second
15 moving image data provided in the image processing unit; and
a transmission unit that transmits the video stream
generated in the image encoding unit.
2. The transmission device according to claim 1, wherein
20 the image encoding unit generates a first video stream that
includes the encoded image data item of the first picture in
each of the units of N pictures, and a predetermined number
of second video streams that include the encoded image data
items of the second to Nth pictures in each of the units of
25 N pictures.
3. The transmission device according to claim 2, wherein
the image encoding unit classifies the image data items
of the pictures in the second moving image data provided in
30 the image processing unit into a plurality of hierarchal layers,
encodes the classified image data items of the pictures in
55
each of the hierarchal layers, divides the hierarchal layers
into a predetermined number of hierarchal groups, and generates
a predetermined number of video streams, the video streams
include the encoded image data items of the pictures in each
of the divided hierarchal 5 groups, and
a lowest hierarchal group includes the encoded image
data item of the first picture in each of the units of N pictures.
4. The transmission device according to claim 2, wherein
10 the image encoding unit inserts first identification
information into the encoded image data items in the first
video stream, and the first identification information
indicates that the first video stream includes a temporally
varying element of the second video stream.
15
5. The transmission device according to claim 4, wherein
the image encoding unit further inserts information
about the averaging process into the encoded image data items
in the first video stream.
20
6. The transmission device according to claim 4, wherein
the transmission unit transmits the container being in
a predetermined format and including the video streams,
the transmission device further comprising:
25 an identification information inserting unit that
inserts second identification information into a layer in a
container, the second identification information indicating
that the first identification information is inserted in the
encoded image data items in the first video stream.
30
7. The transmission device according to claim 1, wherein
56
the image encoding unit inserts information about a
shutter aperture time ratio of the image data item of each
of the pictures in the first moving image data at the
predetermined frame rate into the encoded image data items
included in 5 the video stream.
8. A transmission method comprising:
an image processing step of providing second moving image
data at a predetermined frame rate by processing first moving
10 image data at the predetermined frame rate in units of
consecutive N pictures, using an image data item provided by
averaging image data items of the N pictures in an averaging
process as an image data item of a first picture, and using
image data items of second to Nth pictures of the N pictures
15 as image data items of second to Nth pictures without any change,
the N being an integer larger than or equal to two;
an image encoding step of generating a video stream by
encoding the image data items of the pictures in the second
moving image data provided in the image processing step; and
20 a transmission step of transmitting the video stream
generated in the image encoding step.
9. A reception device comprising:
a reception unit that receives a video stream, the video
25 stream being generated by encoding an image data item of each
picture in moving image data at a predetermined frame rate;
and
a processing unit that processes the video stream
received with the reception unit,
30 wherein the moving image data at the predetermined frame
rate is second moving image data at the predetermined frame
57
rate, and the second moving image data is provided by processing
first moving image data at the predetermined frame rate in
units of consecutive N pictures, using an image data item
provided by averaging image data items of the N pictures in
an averaging process as an image data item of 5 a first picture,
and using image data items of second to Nth pictures of the
N pictures as image data items of second to Nth pictures without
any change, and the N is an integer larger than or equal to
two.
10
10. The reception device according to claim 9, wherein
the video stream received with the reception unit
includes a first video stream that includes the encoded image
data item of the first picture in each of the units of N pictures,
15 and a predetermined number of second video streams that include
the encoded image data items of the second to Nth pictures
in each of the units of N pictures.
11. A reception device comprising:
20 a reception unit that receives a video stream generated
by encoding an image data item of each picture in second moving
image data at a predetermined frame rate, the second moving
image data being provided by processing first moving image
data at the predetermined frame rate in units of consecutive
25 N pictures, using an image data item provided by averaging
image data items of the N pictures in an averaging process
as an image data item of a first picture, and using image data
items of second to Nth pictures of the N pictures as image
data items of second to Nth pictures without any change, the
30 N being an integer larger than or equal to two;
an image decoding unit that provides the second moving
58
image data at the predetermined frame rate by decoding the
video stream received with the reception unit; and
an image processing unit that provides the first moving
image data by processing the second moving image data provided
in the image 5 decoding unit.
12. The reception device according to claim 11, wherein
the video stream received with the reception unit
includes a first video stream that includes the encoded image
10 data item of the first picture in each of the units of N pictures,
and a predetermined number of second video streams that include
the encoded image data items of the second to Nth pictures
in each of the units of N pictures.
15 13. The reception device according to claim 11, further
comprising:
a display processing unit that provides third moving
image data at a frame rate higher than the predetermined frame
rate by interpolating the first moving image data provided
20 in the image processing unit in a temporal direction.
14. The reception device according to claim 13, wherein
information about a shutter aperture time ratio of an
image data item of each picture in the first moving image data
25 is inserted in an encoded image data item of each picture in
the video stream received with the reception unit, and
the display processing unit switches processes for frame
interpolation in accordance with the shutter aperture time
ratio information.
30
15. A reception method comprising:
59
a reception step of receiving a video stream generated
by encoding an image data item of each picture in second moving
image data at a predetermined frame rate, the second moving
image data being provided by processing first moving image
data at the predetermined frame rate in units 5 of consecutive
N pictures, using an image data item provided by averaging
image data items of the N pictures in an averaging process
as an image data item of a first picture, and using image data
items of second to Nth pictures of the N pictures as image
10 data items of second to Nth pictures without any change, the
N being an integer larger than or equal to two;
an image decoding step of providing the second moving
image data at the predetermined frame rate by decoding the
video stream received with the reception step; and
15 an image processing step of providing the first moving
image data by processing the second moving image data provided
in the image decoding step.
16. A transmission device comprising:
20 an image encoding unit that generates a video stream
by encoding an image data item of each picture in moving image
data at a predetermined frame rate; the image encoding unit
that inserts information about a shutter aperture time ratio
of an image data item of each picture in the first moving image
25 data at the predetermined frame rate into the encoded image
data item of each picture in the video stream; and
a transmission unit that transmits the video stream
generated in the image encoding unit.
30 17. A reception device comprising:
a reception unit that receives a video stream provided
60
by encoding an image data item of each picture in moving image
data at a predetermined frame rate;
an image decoding unit that provides the moving image
data at the predetermined frame rate by decoding the video
stream received with the reception 5 unit; and
a display processing unit that provides third moving
image data at a frame rate higher than the predetermined frame
rate by interpolating the first image data provided in the
image decoding unit in a temporal direction,
10 wherein information about a shutter aperture time ratio
of an image data item of each picture in the moving image data
at the predetermined frame rate is inserted in an encoded image
data item of each picture in the video stream received with
the reception unit, and
15 the display processing unit switches processes for frame
interpolation in accordance with the shutter aperture time
ratio information.