Description
Title of Invention
IMAGE PROCESSING DEVICE AND IMAGE PROCESSING METHOD
5
Technical Field
[0001]
The present disclosure relates to an image processing device and an image
processing method.
10
Background Art
[0002]
The standardization of an image coding scheme called High Efficiency
Video Coding (HEVC) by Joint Collaboration Team-Video Coding (JCTVC), which
15 is a joint standardization oiganization of ITU-T and ISO/IEC, is currently under way
for the purpose of improving encoding efficiency more than H. 264/AVC (see, for
example, Non-Patent Literature 1 below).
[0003]
HEVC provides not only coding of a single layer but also scalable video
20 coding, as in known image coding schemes such as MPEG2 and Advanced Video
Coding (AVC) (for example, see Non-Patent Literature 2 below). An HEVC
scalable video coding technology is also called Scalable HEVC (SHVC). In SHVC,
while an enhancement layer is encoded in the HEVC scheme, a base layer may be
encoded in the HEVC scheme or encoded in an image coding scheme other than the
25 HEVC scheme (for example, the AVC scheme).
[0004]
Generally, scalable video coding refers to a technology for hierarchically
encoding a layer transmitting a rough image signal and a layer transmitting a fine
image signal. Typical attributes hierarchized in the scalable video coding mainly
30 include the following three:
- Space scalability: Spatial resolutions or image sizes are hierarchized.
SP351446WO00
2/104
- Time scalability: Frame rates are hierarchized.
- Signal-to-noise ratio (SNR) scalability: SN ratios are hierarchized.
Further, though not yet adopted in the standard, the bit depth scalability and
chroma format scalability are also discussed.
5 [0005]
In the scalable video coding, coding efficiency can be improved by
encoding a parameter, which can be re-used in layers, in only one layer (for example,
see Non-Patent Literature 3).
Citation List
10 Non-Patent Literature
[0006]
Non-Patent Literature 1: "High Efficiency Video Coding (HEVC) text
specification draft 9" by Benjamin Bross, Woo-Jin Han, Jens-Rainer Ohm, Gary J.
Sullivan, Thomas Wiegand (JCTVC-K1003_v9, October 10th to 19th, 2012)
15 Non-Patent Literature 2: "NAL unit header and parameter set designs for
HEVC extensions" by Jill Boyce, Ye-Kui Wang (JCTVC-K1007, October 10th to 19th,
2012)
Non-Patent Literature 3: "TE6: Inter-layer syntax prediction from AVC
base layer" by Jill Boyce, Kawamura Kei, Haricharan Lakshman (JCTVC-K! 106v2,
20 October 10th to 19th, 2012)
Summary of Invention
Technical Problem
[0007]
25 According to techniques of scalable video coding proposed so far, however,
many parameters relating to quantization of transform coefficient data after an
orthogonal transform are not reused in layers. In order to optimize coding
efficiency, it is desirable to cause parameters relating to quantization to be reused in
layers if possible. This point applies not only to scalable video coding but also to a
30 general multi-layer codec that supports inter-layer prediction. Another example of a
multi-layer codec is a multi-view codec.
SP351446WO00
3/104
Solution to Problem
[0008]
According to an embodiment of the present disclosure, there is provided an
5 image processing device including a control section configured to set, based on a
first quantization parameter offset set for a chroma component of a first layer, a
second quantization parameter offset for a chroma component of a second layer
decoded with reference to the first layer, and an inverse quantization section
configured to inversely quantize transform coefficient data of the chroma component
10 of the second layer using a quantization parameter computed using the second
quantization parameter offset set by the control section.
[0009]
The image processing device mentioned above may be typically realized as
an image decoding device that decodes an image.
15 [0010]
According to another embodiment of the present disclosure, there is
provided an image processing method including setting, based on a first quantization
parameter offset set for a chroma component of a first layer, a second quantization
parameter offset for a chroma component of a second layer decoded with reference to
20 the first layer, and inversely quantizing transform coefficient data of the chroma
component of the second layer using a quantization parameter computed using the
set second quantization parameter offset.
[0011]
According to another embodiment of the present disclosure, there is
25 provided an image processing device including a quantization section configured to
quantize transform coefficient data of a chroma component of a second layer
encoded with reference to a first layer using a given quantization parameter, and an
encoding section configured to encode a second quantization parameter offset of a
chroma component of the second layer computed based on a first quantization
30 parameter offset set for a chroma component of the first layer and the given
quantization parameter.
4/104
SP351446WO00
[0012]
The image processing device mentioned above may be typically realized as
an image encoding device that encodes an image.
[0013]
5 According to another embodiment of the present disclosure, there is
provided an image processing method including quantizing transform coefficient
data of a chroma component of a second layer encoded with reference to a first layer
using a given quantization parameter, and encoding a second quantization parameter
offset of the chroma component of the second layer computed based on a first
10 quantization parameter offset set for a chroma component of the first layer and the
given quantization parameter.
Advantageous Effects of Invention
[0014]
15 According to the technology relating to the present disclosure, in a multilayer
codec, coding efficiency can be enhanced by reusing a parameter relating to
quantization in layers.
Brief Description of Drawings
20 [0015]
[FIG 1] FIG 1 is an illustrative diagram for describing scalable video coding.
[FIG. 2] FIG. 2 is an illustrative diagram for describing granularity of rate control.
[FIG 3] FIG 3 is an illustrative diagram showing an example of a relation between
quantization parameters and quantization steps.
25 [FIG. 4] FIG. 4 is an illustrative diagram showing an exampie of a referential relation
of blocks in in-CTB prediction and inter-CTB prediction.
[FIG 5] FIG. 5 is a block diagram showing a schematic configuration of an image
encoding device according to an embodiment.
[FIG 6] FIG. 6 is a block diagram showing a schematic configuration of an image
30 decoding device according to an embodiment.
[FIG. 7] FIG. 7 is a block diagram showing an example of a configuration of an EL
SP351446WO00
5/104
encoding section illustrated in FIG 5
[FIG 8] FIG 8 is a block diagram showing an example of a configuration of a
quantization control section illustrated in FIG. 7.
[FIG 9A] FIG 9A is a first illustrative diagram for describing reuse of quantization
5 matrixes.
[FIG 9B] FIG. 9B is a second illustrative diagram for describing reuse of
quantization matrixes.
[FIG 9C] FIG 9C is a third illustrative diagram for describing reuse of quantization
matrixes.
10 [FIG. 9D] FIG 9D is a fourth illustrative diagram for describing reuse of quantization
matrixes.
[FIG. 10] FIG. 10 is a flow chart showing an example of the flow of a schematic
process for encoding according to an embodiment.
[FIG 11] FIG. 11 is a flow chart showing an example of the flow of a process relating
15 to quantization in an encoding process of an enhancement layer.
[FIG 12] FIG 12 is a block diagram showing an example of a configuration of an EL
decoding section illustrated in FIG 6.
[FIG 13] FIG. 13 is a block diagram showing an example of a configuration of an
inverse quantization control section illustrated in FIG 12.
20 [FIG 14] FIG. 14 is a flow chart showing an example of the flow of a schematic
process for decoding according to an embodiment.
[FIG. 15] FIG 15 is a flow chart showing an example of the flow of a process
relating to inverse quantization in a decoding process on an enhancement layer.
[FIG. 16A] FIG 16A is a first illustrative diagram for describing an example of
25 syntax of an enhancement layer that can be employed in an embodiment.
[FIG 16B] FIG. 16B is a second illustrative diagram for describing an example of
syntax of the enhancement layer that can be employed in an embodiment.
[FIG 16C] FIG 16C is a third illustrative diagram for describing an example of
syntax of the enhancement layer that can be employed in an embodiment.
30 [FIG. 16D] FIG 16D is an illustrative diagram for describing a first modified
example of syntax of the enhancement layer.
SP361446WO00
6/104
[FIG. 16E] FIG. 16E is an illustrative diagram for describing a second modified
example of syntax of the enhancement layer.
[FIG. 17A] FIG. 17A is a first illustrative diagram for describing a combination of
codecs which can be permitted or prohibited.
5 [FIG. 17B] FIG. 17B is a second illustrative diagram for describing a combination of
codecs which can be permitted or prohibited.
[FIG. 18] FIG. 18 is a block diagram showing an example of a schematic
configuration of a television.
[FIG. 19] FIG. 19 is a block diagram showing an example of a schematic
10 configuration of a mobile phone.
[FIG. 20] FIG. 20 is a block diagram showing an example of a schematic
configuration of a recording and reproduction device.
[FIG. 21] FIG. 21 is a block diagram showing an example of a schematic
configuration of an imaging device.
15 [FIG. 22] FIG. 22 is an illustrative diagram for describing a first example of use of
the scalable video coding.
[FIG. 23] FIG. 23 is an illustrative diagram for describing a second example of use of
the scalable video coding.
[FIG. 24] FIG. 24 is an illustrative diagram for describing a third example of use of
20 the scalable video coding.
[FIG. 25] FIG. 25 is an illustrative diagram for describing a multi-view codec.
[FIG. 26] FIG. 26 is a block diagram showing a schematic configuration of an image
encoding device for the multi-view codec.
[FIG 27] FIG. 27 is a block diagram showing a schematic configuration of an image
25 decoding device for the multi-view codec.
[FIG. 28] FIG. 28 is a block diagram showing an example of a schematic
configuration of a video set.
[FIG. 29] FIG. 29 is a block diagram showing an example of a schematic
configuration of a video processor,
30 [FIG. 30] FIG. 30 is a block diagram showing another example of the schematic
configuration of the video processor.
SP351446WO00
7/104
[FIG. 31] FIG. 31 is an illustrative diagram showing an overview of a content
reproduction system.
[FIG. 32] FIG. 32 is an illustrative diagram showing an example of the flow of data in
the content reproduction system.
5 [FIG. 33] FIG. 33 is an illustrative diagram showing a specific example of an MPD.
[FIG. 34] FIG. 34 is a block diagram showing an example of a configuration of a
content server.
[FIG. 35] FIG. 35 is a block diagram showing an example of a configuration of a
content reproduction device.
10 [FIG. 36] FIG. 36 is a block diagram showing another example of a configuration of a
content server.
[FIG. 37] FIG. 37 is the former half of a sequence diagram showing a basic operation
sequence in a wireless communication system that is formed in a P2P mode of Wi-Fi.
[FIG. 38] FIG. 38 is the latter half of the sequence diagram showing the basic
15 operation sequence in the wireless communication system that is formed in the P2P
mode of Wi-Fi.
[FIG. 39] FIG. 39 is an illustrative diagram showing an example of a frame format of
a MAC frame for an extended operation sequence.
[FIG. 40] FIG. 40 is a sequence diagram showing an extended operation sequence.
20
Description of Embodiments
[0016]
Hereinafter, preferred embodiments of the present disclosure will be
described in detail with reference to the appended drawings. Note that, in this
25 specification and the drawings, elements that have substantially the same function
and structure are denoted with the same reference signs, and repeated explanation is
omitted.
[0017]
In addition, description will be provided in the following order.
30 1. Overview
1-1. Scalable video coding
SP351446WO00
8/104
1-2. General rate control
1-3. Quantization parameter of a chroma component
1-4. Quantization matrix
1-5. Basic configuration example of an encoder
5 1-6. Basic configuration example of a decoder
2. Configuration example of an EL encoding section according to an
embodiment
2-1. Overall configuration
2-2. Detailed configuration of a quantization control section
10 3. Flow of a process for encoding according to an embodiment
3-1. Schematic flow
3-2. Process relating to quantization
4. Configuration example of an EL decoding section according to an
embodiment
15 4-1. Overall configuration
4-2. Detailed configuration of an inverse quantization control section
5. Flow of a process of decoding according to an embodiment
5-1. Schematic flow
5-2. Process relating to inverse quantization
20 5-3. Example of syntax
6. Example of a combination of codecs
7. Application example
7-1. Application to various products
7-2. Various uses of scalable video coding
25 7-3. Application to other codec
7-4. Various mounting levels
7-5. System that uses MPEG-DASH
7-6. System that uses a P2P mode of Wi-Fi
8. Conclusion
30 [0018]
<1. Overview>
SP351446WO00
9/104
[1-1. Scalable video coding]
In scalable video coding, a plurality of layers, each containing a series of
images, are encoded. A base layer is a layer encoded first to represent a roughest
image. An encoded stream of the base layer may be independently decoded without
5 decoding encoded streams of other layers. Layers other than the base layer are
layers called enhancement layers representing finer images. Encoded streams of the
enhancement layers are encoded by using information contained in the encoded
stream of the base layer. Therefore, to reproduce an image of an enhancement layer,
encoded streams of both the base layer and the enhancement layer are decoded.
10 The number of layers handled in the scalable video coding may be any number equal
to or greater than 2. When three layers or more are encoded, the lowest layer is the
base layer and the remaining plural layers are enhancement layers. For an encoded
stream of a higher enhancement layer, information contained in encoded streams of a
lower enhancement layer and the base layer may be used for encoding and decoding.
15 [0019]
FIG. 1 shows three layers LI, L2, and L3 subjected to scalable video coding.
The layer LI is a base layer and the layers L2 and L3 are enhancement layers. Note
that, among various kinds of scalabilities, space scalability is taken as an example
herein. A space resolution ratio of the layer L2 to the layer LI is 2:1. A space
20 resolution ratio of the layer L3 to the layer LI is 4:1. The resolution ratios herein
are merely examples, and for example, a resolution ratio of a non-integer such as
1.5:1 may be used. A block Bl of the layer LI is a processing unit of an encoding
process in a picture of the base layer. A block B2 of the layer L2 is a processing
unit of an encoding process in a picture of the enhancement layer to which a common
25 scene to the block Bl is projected. The block B2 corresponds to the block Bl of the
layer LI. A block B3 of the layer L3 is a processing unit of an encoding process in
a picture of the enhancement layer higher than the layers to which the common scene
to the blocks Bl and B2 is projected. The block B3 corresponds to the block Bl of
the layer LI and the block B2 of the layer L2.
30 [0020]
In such a layer structure, layers in which a common scene is projected have
SP351446WO00
10/104
a similar image frequency characteristic. For example, when an image of the block
Bl of the layer LI is flat and a high frequency component thereof is small, there is a
high possibility of an image of the block B2 of the layer L2 also having a small high
frequency component. The same applies to the block B2 of the layer L2 and the
5 block B3 of the layer L3.
[0021]
[1-3. General rate control]
A frequency characteristic of an image affects the number of bits of
transform coefficient data that is generated as a result of an orthogonal transform.
10 In order to maintain a bit rate of an encoded stream, transform coefficient data is
generally quantized using a greater quantization step if the number of bits of the
transform coefficient data is great. An example of a rate control scheme for
realizing an expected bit rate is expressed in an MPEG2 test model. In the MPEG2
test model, an allocated code amount of each picture is first decided based on an
15 amount of bits allocated to a GOP, a picture type of each picture in the GOP, and
complexity (Global Complexity Measure). In addition, a quantization parameter of
each macroblock in each picture (quantization scale code) is computed based on an
activity computed for each macroblock and the allocated code amount of the picture.
The activity referred to herein is a type of index expressing complexity of an image.
20 Note that details of the rate control scheme in the MPEG2 test model are disclosed at
the following web page.
- Reference URL http://www.mpeg.org/MPEG/MSSG/tm5/Chl0/Chl0.htmI
[0022]
In the MPEG2 scheme, however, the above-described quantization scale
25 code is decided for each macroblock having the size of 16x16 pixels. The
maximum size of a largest coding unit (LCU) corresponding to a macroblock in the
HEVC scheme is 64x64 pixels. As granularity of rate control becomes excessively
large, however, local regions in which a quantization step is improper for complexity
of an image can be generated. For example, in FIG. 2, the result obtained by
30 computing a normalization activity of the image shown in the upper part with
granularity of 64x64 pixels is shown in the middle part, and the result obtained by
11/104
SP351446WO00
computing the normalization activity with granularity of 16x16 pixels is shown in
the lower part. Herein, higher activities are shown in white. As understood from
FIG. 2, if rate control is performed with granularity of 64x64 pixels, many regions in
which, for example, quantization is performed only using a large quantization step
5 even though complexity is low, or quantization is performed using a small
quantization step even though complexity is high can be generated.
[0023]
In order to appropriately execute rate control while avoiding such
impropriety resulting from the above-described block sizes, the HEVC scheme
10 employs a technique of controlling quantization steps in units of blocks that arc
smaller than an LCU. To be more specific, referring to syntax of a picture
parameter set (PPS) described in Non-Patent Literature 1 described above, when
cu_qp_delta_enabled_flag is true, the size of a block that is the unit for designating a
quantization step is designated based on size information of diffcuqpdeltadepth.
15 In reality, since a logarithm of a quantization step is in a proportional relation with a
signal-to-noise (SN) ratio, a quantization parameter is designated for each block,
rather than direct designation of a quantization step. If a quantization step
corresponding to a quantization parameter qp is set to S(qp), a relation between a
quantization parameter qp and a quantization step S(qp) is defined so that the
20 quantization step S(qp) satisfies the following expression.
[0024]
[Math 1]
S(qp + a) =b
S(qp) (x}
[0025]
25 In the HEVC scheme, values of a variable a=6 and a variable b=2 are
adopted so that ranges of a quantization parameter and a quantization step are
optimized. A relation between quantization parameters qp and quantization steps
S(qp) of this case is illustrated in FIG. 3. As illustrated in FIG. 3, the value of a
SP351446WO00
12/104
quantization step S(qp) doubles each time the value of a quantization parameter qp
increases by six.
[0026]
A quantization parameter is actually designated using a combination of a
5 plurality of quantization relevant parameters. Referring to the syntax described in
Non-Patent Literature 1 described above, quantization parameters are initialized
using parameters of pic_init_qp_minus26 in a PPS and sliceqpdelta in a slice
header (SliceQPy = 26 + pic init_qp_minus26 + slice_qp_delta). Quantization
parameters of individual coding units (CU) are predicted from the quantization
10 parameter of one of an upper adjacent CU and a left adjacent CU or the quantization
parameters of both of them (in-CTB prediction), or predicted from a quantization
parameter of the previous CU in a decoding order (inter-CTB prediction). In FIG. 4,
a referential relation in prediction of quantization parameters in coding tree blocks
(CTBs) and between CTBs is exemplified. In in-CTB prediction in which both of
15 an upper adjacent CU and a left adjacent CU can be referred to, a predicted value of
a quantization parameter is equal to the average value ((TopQP + LeftQP + I) » 1)
of the quantization parameters of the upper adjacent CU and the left adjacent CU
(TopQP and LeftQP). In addition, by adding residuals indicated by
cuqpdelta abs and cu_qp_delta sign in a transform unit (TU) to the predicted
20 value, the values of individual quantization parameters are computed.
[0027]
[1-3. Quantization parameter of a chroma component]
The quantization parameter computation technique described in the above
section is mainly a technique regarding a quantization parameter of a luma
25 component. With regard to a chroma component, by adding an offset to a
quantization parameter of a luma component, a quantization parameter that is
adjusted for the chroma component can be used. Referring to the syntax described
in Non-Patent Literature 1 described above, an offset of a quantization parameter of a
chroma component is designated based on pic_cb_qp offset and piccrqpoffset in
30 a PPS and slice_cb_qp_offset and slicecrqpofTset in a slice header. The sum of
pic_cb_qp_offset and sliceebqpoffset is the offset of the quantization parameter
i
SP351446WO00
13/104
of a Cb component, and the sum of piccrqpoffsct and slice_cr_qp_offset is the
offset of the quantization parameter of a Cr component.
[0028]
Note that a specific expression of a relation between a quantization
5 parameter of a luma component and a quantization parameter of a chroma
component in the HEVC scheme is described in Non-Patent Literature 1 described
above. As a difference from the AVC scheme, when offsets of a Cb component and
a Cr component are equal to each other in the AVC scheme, only one
chromaqpindexoffset is encoded in a PPS. On the other hand, in the HEVC
10 scheme, respective offsets of a Cb component and a Cr component are encoded in a
PPS and a slice header as described above. Thus, a quantization parameter of a
chroma component can be flexibly adjusted in units of slices in the HEVC scheme.
In addition, while the upper value of a quantization parameter of a chroma
component is 39 in the AVC scheme, the upper value of a quantization parameter of a
15 chroma component is 51, which is the same as a luma component in the HEVC
scheme. Thus, in the HEVC scheme, overflow of a hypothetical reference decoder
(HRD) buffer is easily avoided by reducing an information amount of transform
coefficient data of a chroma component.
[0029]
20 [1-4. Quantization matrix]
A quantization matrix (which is also referred to as a scaling list) is a
technology introduced to quantize high frequency components more coarsely than
low frequency components using characteristics of vision of human beings, who
have difficulty perceiving high frequency components of an image. When a
25 quantization matrix is used, a quantization step scaled with values of respective
elements of the quantization matrix is used to perform quantization and inverse
quantization on transform coefficient data, instead of the quantization step itself
corresponding to the quantization parameter described in the above section. In the
AVC scheme, quantization matrixes each having the sizes of 4x4 pixels and 8x8
30 pixels can be used. On the otlier hand, in the HEVC scheme, quantization matrixes
each having the sizes of 4x4 pixels, 8x8 pixels, 16x16 pixels, and 32x32 pixels can
SP351446WO00
14/104
be used. In the HEVC scheme, however, quantization matrixes of 16x16 pixels and
32x32 pixels are each encoded in the size of 8x8 pixels, excluding DC components
in order to reduce a code amount, and are up-sampled by zero-order hold from the
size of 8x8 pixels upon use. The DC components are separately encoded.
5 [0030]
As described above, various parameters relating to quantization are encoded
in the current specification of the HEVC scheme. Here, frequency characteristics of
an image are similar in layers as described above. In addition, such a frequency
characteristic of an image affects the number of bits of transform coefficient data,
10 and the number of bits is controlled by quantization. Thus, by controlling
quantization of a base layer and an enhancement layer in scalable video coding using
common parameters to layers, appropriate rate control is expected to be able to be
realized and a code amount of a parameter relating to quantization to be reduced.
Thus, embodiments of an image processing device that enables a parameter relating
15 to quantization to be reused in layers will be described in detail in the following
sections.
[0031]
[1-5. Basic configuration example of an encoder]
FIG. 5 is a block diagram showing a schematic configuration of an image
20 encoding device 10 according to an embodiment supporting scalable video coding.
Referring to FIG. 5, the image encoding device 10 includes a base layer (BL)
encoding section la, an enhancement layer (EL) encoding section lb, a common
1116111017 2, and a multiplexing section 3.
[0032]
25 The BL encoding section la encodes a base layer image to generate an
encoded stream of the base layer. Hie EL encoding section lb encodes an
enhancement layer image to generate an encoded stream of an enhancement layer.
The common memory 2 stores information commonly used between layers. The
multiplexing section 3 multiplexes an encoded stream of the base layer generated by
30 the BL encoding section la and an encoded stream of one or more enhancement
layers generated by the EL encoding section lb to generate a multilayer multiplexed
SP351446WO00
15/104
stream.
[0033]
[1-6. Basic configuration example of a decoder]
FIG. 6 is a block diagram showing a schematic configuration of an image
5 decoding device 60 according to an embodiment supporting scalable video coding.
Referring to FIG. 6, the image decoding device 60 includes a demultiplexing section
5, a base layer (BL) decoding section 6a, an enhancement layer (EL) decoding
section 6b, and a common memory 7.
[0034]
10 The demultiplexing section 5 demultiplexes a multilayer multiplexed stream
into an encoded stream of the base layer and an encoded stream of one or more
enhancement layers. The BL decoding section 6a decodes a base layer image from
an encoded stream of the base layer. The EL decoding section 6b decodes an
enhancement layer image from an encoded stream of an enhancement layer. The
15 common memory 7 stores information commonly used between layers.
[0035]
In the image encoding device 10 illustrated in FIG. 5, the configuration of
the BL encoding section la to encode the base layer and that of the EL encoding
section lb to encode an enhancement layer are similar to each other. Some
20 parameters generated or acquired by the BL encoding section la may be buffered by
using the common memory 2 and reused by the EL encoding section 1 b. In the next
section, such a configuration of the EL encoding section lb will be described in
detail. ,
[0036]
25 Similarly, in the image decoding device 60 illustrated in FIG. 6, the
configuration of the BL decoding section 6a to decode the base layer and that of the
EL decoding section 6b to decode an enhancement layer are similar to each other.
Some parameters generated or acquired by the BL decoding section 6a may be
buffered by using the common memory 7 and reused by the EL decoding section 6b.
30 Further in the next section, such a configuration of the EL decoding section 6b will
be described in detail.
16/104
SP351446WO00
[0037]
<2. Configuration example of an EL encoding section according to an embodiment
[2-1. Overall configuration]
FIG. 7 is a block diagram showing an example of the configuration of the
5 EL encoding section lb illustrated in FIG. 5. Referring to FIG. 7, the EL encoding
section lb includes a sorting buffer 11, a subtraction section 13, an orthogonal
transform section 14, a quantization section 15, a lossless encoding section 16, an
accumulation buffer 17, a rate control section 18, an inverse quantization section 21,
an inverse orthogonal transform section 22, an addition section 23, a deblock filter 24,
10 a frame memory 25, selectors 26 and 27, an intra prediction section 30, an inter
prediction section 35, and a quantization control section 40.
[0038]
The sorting buffer 11 sorts the images included in the series of image data.
After sorting the images according to a GOP (Group of Pictures) structure according
15 to the encoding process, the sorting buffer 11 outputs the image data which has been
sorted to the subtraction section 13, the intra prediction section 30, and the inter
prediction section 35.
[0039]
The image data input from the sorting buffer 11 and predicted image data
20 input by the intra prediction section 30 or the inter prediction section 35 to be
described later are supplied to the subtraction section 13. The subtraction section
13 computes predicted error data which is a difference between the image data input
from the sorting buffer 11 and the predicted image data and outputs the computed
predicted error data to the orthogonal transform section 14.
25 [0040]
The orthogonal transform section 14 performs orthogonal transform on the
predicted error data input from the subtraction section 13. The orthogonal
transform to be performed by the orthogonal transform section 14 may be discrete
cosine transform (DCT) or Karhunen-Loeve transform, for example. The
30 orthogonal transform section 14 outputs transform coefficient data acquired by the
orthogonal transform process to the quantization section 15.
SP351446WO00
17/104
[0041]
The quantization section 15 receives the transform coefficient data from the
orthogonal transform section 14 and receives a quantization parameter from the
quantization control section 40 to be described later. The quantization section 15
5 can also receive a quantization matrix from the quantization control section 40.
Using a quantization step decided from the input quantization parameter (and the
quantization matrix), the quantization section 15 quantizes the transform coefficient
data of an enhancement layer and thereby generates quantized data (quantized
transform coefficient data). Then, the quantization section 15 outputs the quantized
10 data to the lossless encoding section 16 and the inverse quantization section 21.
[0042]
The lossless encoding section 16 performs a lossless encoding process on
the quantized data input from the quantization section 15 to generate an encoded
stream of the enhancement layer. The lossless encoding section 16 encodes various
15 parameters referred to when the encoded stream is decoded and inserts the encoded
parameters into a header region of the encoded stream. The parameters encoded by
the lossless encoding section 16 can include information regarding intra prediction to
be described later, information regarding inter prediction, and quantization relevant
parameters. Then, the lossless encoding section 16 outputs the generated encoded
20 stream to the accumulation buffer 17.
[0043]
The accumulation buffer 17 temporarily accumulates the encoded stream
input from the lossless encoding section 16 using a storage medium such as a
semiconductor memory. Then, the accumulation buffer 17 outputs the accumulated
25 encoded stream to a transmission section that is not shown (for example, a
communication interface or a connection interface to peripheral devices) at a rate in
accordance with the band of a transmission patii.
[0044]
The rate control section 18 monitors vacant capacity of the accumulation
30 buffer 17. Then the rate control section 18 generates a rate control signal according
to the vacant capacity of the accumulation buffer 17 and outputs the generated rate
SP351446WO00
18/104
control signal to the quantization control section 40. The rate control section 18
may control the rate of the encoded stream according to the same rate control scheme
as the above-described MPEG2 test model and other rate control schemes. The rate
control signal output to the quantization control section 40 can include the
5 quantization parameter and the quantization matrix.
[0045]
The inverse quantization section 21, the inverse orthogonal transform
section 22, and the addition section 23 constitute a local decoder. The inverse
quantization section 21 receives the quantized data from the quantization section 15
10 and receives the quantization parameter from the quantization control section 40 to
be described later. The inverse quantization section 21 can also receive the
quantization matrix from the quantization control section 40. The inverse
quantization section 21 performs inverse quantization on the quantized data of the
enhancement layer to restore the transform coefficient data using the quantization
15 step decided from the input quantization parameter (and the quantization matrix).
Then the inverse quantization section 21 outputs the restored transform coefficient
data to the inverse orthogonal transform section 22.
[0046]
The inverse orthogonal transform section 22 restores predicted error data by
20 performing an inverse orthogonal transform process on the transform coefficient data
input from the inverse quantization section 21. Then, the inverse orthogonal
transform section 22 outputs the restored predicted error data to the addition section
23.
[0047]
25 The addition section 23 adds the restored predicted error data input from the
inverse orthogonal transform section 22 and the predicted image data input from the
intra prediction section 30 or the inter prediction section 35 to thereby generate
decoded image data (reconstructed image of the enhancement layer). Then, the
addition section 23 outputs the generated decoded image data to the deblock filter 24
30 and the frame memory 25.
[0048]
a
SP351446WO00
19/104
The deblock filter 24 performs a filtering process for reducing block
distortion that occurs during encoding of an image. The deblock filter 24 performs
filtering on the decoded image data input from the addition section 23 to remove
block distortion, and then outputs the filtered decoded image data to the frame
5 memory 25.
[0049]
The frame memory 25 stores the decoded image data input from the
addition section 23 and the filtered decoded image data input from the deblock filter
24 using a storage medium.
10 [0050]
The selector 26 reads the decoded image data before the filtering used for
the intra prediction from the frame memory 25 and supplies the read decoded image
data as reference image data to the intra prediction section 30. Further, the selector
26 reads the filtered decoded image data used for the inter prediction from the frame
15 memory 25 and supplies the read decoded image data as reference image data to the
inter prediction section 35.
[0051]
In the intra prediction mode, the selector 27 outputs predicted image data as
a result of intra prediction output from the intra prediction section 30 to the
20 subtraction section 13 and also outputs information about the intra prediction to the
lossless encoding section 16. Further, in the inter prediction mode, the selector 27
outputs predicted image data as a result of inter prediction output from the inter
prediction section 35 to the subtraction section 13 and also outputs information about
the inter prediction to the lossless encoding section 16. The selector 27 switches the
25 inter prediction mode and the intra prediction mode in accordance with the
magnitude of a cost function value.
[0052]
The intra prediction section 30 performs an intra prediction process on each
prediction unit (PU) of the HEVC scheme based on the original image data and the
30 decoded image data of the enhancement layer. For example, the intra prediction
section 30 evaluates a prediction result according to each candidate mode in a
SP351446WO00
20/104
prediction mode set using a predetermined cost function. Next, the intra prediction
section 30 selects a prediction mode in which a cost function value is the minimum,
i.e., a prediction mode in which a compression ratio is the highest, as an optimum
prediction mode. In addition, the intra prediction section 30 generates predicted
5 image data of the enhancement layer according to the optimum prediction mode.
Further, the intra prediction section 30 outputs information regarding the intra
prediction including prediction mode information indicating the selected optimum
prediction mode, the cost function value, and the predicted image data to the selector
27.
10 [0053]
The inter prediction section 35 performs an inter prediction process on each
prediction unit of the HEVC scheme based on the original image data and the
decoded image data of the enhancement layer. For example, the inter prediction
section 35 evaluates a prediction result according to each candidate mode in a
15 prediction mode set using a predetermined cost function. Next, the inter prediction
section 35 selects a prediction mode in which a cost function value is the minimum,
i.e., a prediction mode in which a compression ratio is the highest, as an optimum
prediction mode. In addition, the inter prediction section 35 generates predicted
image data of the enhancement layer according to the optimum prediction mode.
20 Further, the inter prediction section 35 outputs information regarding the inter
prediction including prediction mode information and motion information indicating
the selected optimum prediction mode, the cost function value, and the predicted
image data to the selector 27.
[0054]
25 The quantization control section 40 controls a quantization process and an
inverse quantization process performed in the EL encoding section lb using
information buffered by the common memory 1. In addition, the quantization
control section 40 generates a quantization relevant parameter. In the present
embodiment, the quantization relevant parameter generated by the quantization
30 control section 40 can include a parameter used when deciding a quantization
parameter and a parameter used when deciding a quantization matrix. In the
SP351446WO00
21/104
present specification, the parameter used when deciding a quantization parameter
will be referred to as a quantization parameter (QP) parameter, and the parameter
used when deciding a quantization matrix will be referred to as a scaling list (SL)
parameter.
5 [0055]
A QP parameter mainly includes a parameter group relating to luma
components and a parameter group relating to chroma components. As an example,
the parameter group relating to luma components can include pic_init_qp_minus26
that is a parameter of a picture unit, sliceqpdelta that is a parameter of a slice unit,
10 and cu_qp_delta_abs and cu_qp_delta_sign that are parameters of a CU unit. The
parameter group relating to chroma components can include slice_cbE qpoffset and
sIice_crE_qp_offset that are parameters of the slice unit, pic cbqpoffset and
pic_cr_qp_offset that are parameters of the picture unit relating to chroma
components generated in a known technique are not generated in the present
15 embodiment.
[0056]
When a quantization matrix set in a base layer is not reused in an
enhancement layer, an SL parameter which designates quantization matrixes having
one or more sizes to be set for the enhancement layer is generated. When a
20 quantization matrix is reused in layers, if the base layer is encoded in the HEVC
scheme, the SL parameter may not be generated. When quantization matrixes are
reused in layers, if the base layer is encoded in the AVC scheme, the SL parameter
which designates quantization matrixes having (he sizes of 16xi6 pixels and 32x32
pixels to be set for the enhancement layer can be generated. When quantization
25 matrixes are reused in layers, a flag that indicates reuse of the quantization matrix
and information that indicates a layer whose quantization matrix should be reused
may be generated as the SL parameter.
[0057]
Note that a quantization step (or a quantization parameter) that is actually
30 used for quantizing and inversely quantizing transform coefTicient data may be
designated based on, for example, a rate control signal input from the rate control
SP361446WO00
22/104
section 18. Instead, a quantization step (or a quantization parameter) that is actually
used may be designated in advance by a user.
[0058]
[2-2. Detailed configuration of a quantization control section]
5 FIG. 8 is a block diagram showing an example of a configuration of the
quantization control section 40 illustrated in FIG. 7. Referring to FIG. 8, the
quantization control section 40 has a QP setting section 41, a QP parameter
generation section 42, an SL setting section 43, and an SL parameter generation
section 44.
10 [0059]
(1) QP setting section
The QP setting section 41 sets quantization parameters that are actually used
by the quantization section 15 when quantizing transform coefficient data of a luma
component and chroma components of an enhancement layer for respective blocks of
15 the enhancement layer. The quantization parameters set by the QP setting section
41 are also used when the inverse quantization section 21 inversely quantizes
quantized data. Here, a block can have a size that is equal to or smaller than that of
an LCU. For example, the QP setting section 41 may set a quantization parameter
of the luma component (Y) and quantization parameters of chroma components (Cb
20 and Cr) according to a rate control signal input from the rate control section 18.
Instead, the QP setting section 41 may set a quantization parameter of the luma
component and quantization parameters of the chroma components designated in
advance by a user, Then, the QP setting section 41 outputs the quantization
parameters (QP (EL)) set for the respective blocks of the enhancement layer to the
25 quantization section 15, the inverse quantization section 21, and the QP parameter
generation section 42.
[0060]
(2) QP parameter generation section
The QP parameter generation section 42 generates a QP parameter of an
30 enhancement layer based on the quantization parameters of the enhancement layer
input from the QP setting section 41 and a QP parameter of the base layer buffered
SP351446WO00
23/104
by the common memory 2. A QP parameter of the Iuma component may be
generated according to the specification described in Non-Patent Literature 1
described above.
[0061]
5 For example, when the base layer is encoded in the HEVC scheme, the QP
parameter of the Cb component of the base layer can include pic cbqpoffset and
slicecbqpoffset. When a quantization parameter offset corresponding to a given
quantization parameter that is actually used for the Cb component of the
enhancement layer is set to cbE_qp_offset, the QP parameter slice_cbE_qp_offset of
10 the Cb component of the enhancement layer may be generated according to, for
example, the following expression:
[0062]
[Math 2]
slice _ cbE qp _ offset
~ cbE qp _ offset - cbB qp _ offset
= cbE_qp _offset - (pic cb qp_offset + slice cb_qp_offset) (2)
15 [0063]
Likewise, when the base layer is encoded in the HEVC scheme, the QP
parameter of the Cr component of the base layer can include piccrqpoffset and
slicecrqpoffset. When a quantization parameter offset corresponding to a given
quantization parameter that is actually used for the Cr component of the
20 enhancement layer is set to crEqpoffset, the QP parameter siice_crE_qp_offset of
the Cr component of the enhancement layer may be generated according to, for
example, the following expression:
[0064]
[Math 3]
SP351446WO00
24/104
slice __ crE _qp _ offset
= crE_qp __q^fre* - cri? _qp_offset
= c/*£ _ /c _ cr _qp _ offset + sZ/ce _cr _qp _ offset (12)
[0122]
In a second technique, the quantization parameter offsets cbBqpoffset and
crBqp offset of the base layer are each equal to the offsets of the picture unit as
5 shown in the following expressions (13) and (14).
[0123]
[Math 9]
cbB _qp_ offset = pic _cb _qp _ offset (13)
crB_qp^offset - pic_cr qp_offset (14)
[0124]
10 In both techniques, the quantization parameter offsets cbB_qp_offset and
crB_qp_offset of the base layer are decided from the QP parameters of the base layer
buffered by the common memory 7.
[0125]
The QP setting section 92 computes the quantization parameters of the
15 chroma components of the enhancement layer by inputting the sum of the
quantization parameter offsets computed according to one of the above-described
techniques and the quantization parameter of the luma component into a
predetermined relational expression. The relational expression used here is
described in Non-Patent Literature i described above. Then, the QP setting section
20 92 outputs the computed quantization parameter to the inverse quantization section
63.
[0126]
Note that, when the QP parameter reuse flag acquired by the QP parameter
acquisition section 91 indicates that the quantization parameter offsets should be
25 reused for the chroma components (i.e., the quantization parameter offsets of the
s
SP351446WO00
. 39/104
enhancement layer should be set based on the quantization parameter offsets of the
base layer), the QP setting section 92 may execute the above-described process with
respect to the chroma components. When the QP parameter reuse flag does not
indicate that the quantization parameter offsets should be reused, the QP setting
5 section 92 can set the quantization parameter offsets of the chroma components of
the enhancement layer without referring to the quantization parameter offsets of the
base layer.
[0127]
(3) SL parameter acquisition section
10 The SL parameter acquisition section 93 acquires an SL parameter decoded
by the lossless decoding section 62 from an encoded stream of an enhancement layer
when a base layer is encoded in an image coding scheme other than the HEVC
scheme (for example, the AVC scheme). For example, when the base layer is
encoded in the AVC scheme, the SL parameter acquisition section 93 acquires SL
15 parameters which designate quantization matrixes having the sizes of 16x16 pixels
and 32x32 pixels. In addition, even when the quantization matrixes are not reused
in the layers, the SL parameter acquisition section 93 acquires the SL parameters
decoded by the lossless decoding section 62 from the encoded stream of the
enhancement layer. In this case, the SL parameters which designate the
20 quantization matrixes having the sizes of 4x4 pixels to 32x32 pixels are acquired.
Note that the SL parameters of the quantization matrixes having the sizes equal to or
greater than 16x16 pixels arc typically encoded after down-sampling each of the
quantization matrixes to have the size of 8x8 pixels as described using FIG. 9B.
The SL parameter acquisition section 93 outputs the acquired SL parameters of the
25 enhancement layer to the SL setting section 94. Note that, when the quantization
matrixes are reused in the layers and the base layer is encoded in the HEVC scheme,
the SL parameter acquisition section 93 may not acquire the SL parameters of the
enhancement layer. The SL parameter acquisition section 93 can determine whether
or not the base layer has been encoded in the HEVC scheme with reference to a flag
30 avc_base__layer_flag decoded from a VPS.
[0128]
SP351446WO00
40/104
Furthermore, when a quantization matrix reuse flag and a setting technique
flag have been decoded by the lossless decoding section 62, the SL parameter
acquisition section 93 may output the decoded quantization matrix reuse flag and
setting technique flag to the SL setting section 94.
5 [0129]
(4) SL setting section
The SL setting section 94 sets quantization matrixes to be used by the
inverse quantization section 63 when the transform coefficient data of the luma
component and the chroma components of the enhancement layer is inversely
10 quantized for the enhancement layer,
[0130]
For example, the SL setting section 94 acquires the quantization matrixes of
the base layer having the sizes of 4x4 pixels and 8x8 pixels from the common
memory 7 when the base layer is encoded in the HEVC scheme or the AVC scheme.
15 For example, the SL setting section 94 may duplicate the corresponding quantization
matrixes of the enhancement layer from the acquired quantization matrixes of the
base layer, instead, the SL setting section 94 may predict the corresponding
quantization matrixes of the enhancement layer from the acquired quantization
matrixes of the base layer. In the latter case, a residual of the prediction of the
20 quantization niatrixes is additionally decoded from the encoded stream of the
enhancement layer, and the residual can be added to the predicted quantization
matrixes. The SL setting section 94 may choose whether the quantization matrixes
of the enhancement layer should be duplicated or predicted according to the setting
technique flag acquired by the SL parameter acquisition section 93.
25 [0131]
When the base layer has been encoded in the HEVC scheme, the SL setting
section 94 also acquires the quantization matrixes of the base layer having the sizes
of 16x16 pixels and 32x32 pixels from the common memory 7. Then, the SL
setting section 94 duplicates or predicts the corresponding quantization matrixes of
30 the enhancement layer from the acquired quantization matrixes of the base layer.
When the base layer has been encoded in the AVC scheme, the SL setting section 94
I
SP351446WO00
41/104
generates the respective quantization matrixes of the enhancement layer having the
sizes of 16x16 pixels and 32x32 pixels according to the SL parameters input from
the SL parameter acquisition section 93, not based on the quantization matrixes of
the base layer.
5 [0132]
Note that, when the quantization matrix reuse flag that can be acquired by
the SL parameter acquisition section 93 indicates that the quantization matrixes are
not reused (i.e., the quantization matrixes should be decoded from the encoded
stream of the enhancement layer), the SL setting section 94 can generate all
10 necessary quantization matrixes according to the SL parameters of the enhancement
layer, not based on the quantization matrixes of the base layer. The quantization
matrix reuse flag and the setting technique flag may be decoded from the encoded
stream of the enhancement layer separately for different quantization matrix sizes,
different prediction modes, or different color components.
15 [0133]
<5. Flow of a process of decoding according to an embodiment
[5-1. Schematic flow]
FIG. 14 is a flow chart showing an example of the flow of a schematic
process for decoding according to an embodiment. For the sake of brevity of
20 description, process steps not directly relevant to the technology in the present
disclosure are omitted from the drawing.
[0134]
Referring to FIG. 14, the demultiplexing section 5 first demultiplexes a
multilayer multiplexed stream into an encoded stream of the base layer and an
25 encoded stream of the enhancement layer (Step S60).
[0135]
Next, the BL decoding section 6a performs a decoding process on the base
layer to reconstruct a base layer image from the encoded steam of the base layer
(StepS61).
30 [0136]
The common memory 7 buffers quantization relevant parameters decoded in
SP351446WO00
42/104
the decoding process of the base layer (Step S62). The quantization relevant
parameters buffered here can include, for example, the quantization parameter offsets
of the chroma components set for each block of the base layer and quantization
matrixes.
5 [0137]
Next, the EL decoding section 6b executes a decoding process of the
enhancement layer using the information buffered by the common memory 7 to
reconstruct an enhancement layer image (Step S63).
[0138]
10 [5-2. Process relating to inverse quantization]
FTG. 15 is a flow chart showing an example of the flow of a process relating
to inverse quantization in the decoding process (Step S63 of FIG. 14) on the
enhancement layer. The process shown in FIG 15 can be repeated for, for example,
eacii slice of the enhancement layer image.
15 [0139]
Referring to FIG. 15, first, the QP parameter acquisition section 91 acquires
QP parameters decoded by the lossless decoding section 62 from the encoded stream
of the enhancement layer (Step S71). Then, the QP parameter acquisition section
91 outputs the acquired QP parameters of the enhancement layer to the QP setting
20 section 92.
[0140]
In addition, the SL parameter acquisition section 93 determines whether or
not the quantization matrixes are reused in the layers by, for example, referring to the
quantization matrix reuse flag (Step S72). When the quantization matrixes are
25 reused in the layers, the SL parameter acquisition section 93 ftnlher determines
whether or not the encoding scheme of the base layer is the HEVC scheme (Step
S73). When the quantization matrixes are not reused in the layers or when the
encoding scheme of the base layer is not the HEVC scheme, the SL parameter
acquisition section 93 acquires SL parameters of the enhancement layer decoded by
30 the lossless decoding section 62 from the encoded stream of the enhancement layer
(Step S74). Then, the SL parameter acquisition section 93 outputs the acquired SL
SP351446WO00
43/104
parameters of the enhancement layer to the SL setting section 94.
[0141]
Next, the QP setting section 92 sets quantization parameters of the
enhancement layer for each block of the enhancement layer (Step S75). For
5 example, the QP setting section 92 computes quantization parameter offsets of the
enhancement layer for the chroma components based on the quantization parameter
offset differences indicated by the quantization parameter offsets and the QP
parameters of the base layer. In addition, the QP setting section 92 inputs the sums
of the computed quantization parameter offsets and the quantization parameter of the
10 luma component into a predetermined relational expression to compute the
quantization parameters of the enhancement layer. Then, the QP setting section 92
outputs the computed quantization parameters to the inverse quantization section 63.
[0142]
Note that, when a QP parameter reuse flag does not indicate that the
15 quantization parameter offsets of the base layer should be reused, the QP setting
section 92 can set a quantization parameter offset indicated by the QP parameters for
the enhancement layer without referring to the quantization parameter offsets of the
base layer for the chroma components.
[0143]
20 In addition, the SL setting section 94 sets quantization matrixes of the
enhancement layer for each block of the enhancement layer (Step S76). For
example, when the base layer has been encoded in the HEVC scheme or the AVC
scheme, the SL setting section 94 can acquire the quantization matrixes of the base
layer having the sizes of 4x4 pixels and 8x8 pixels from the common memory 7. In
25 addition, when the base layer has been encoded in the HEVC scheme, the SL setting
section 94 also acquires the quantization matrixes of the base layer having the sizes
of 16x16 pixels and 32x32 pixels from the common memory 7. Then, the SL
setting section 94 duplicates or predicts the corresponding quantization matrixes of
the enhancement layer from the acquired quantization matrixes of the base layer to
30 set the quantization matrixes of the enhancement layer. When the base layer has
been encoded in the AVC scheme, the SL setting section 94 sets the quantization
SP351446WO00
44/104
matrixes of the enhancement layer having the sizes of 16x16 pixels and 32x32 pixels
according to the SL parameters input from the SL parameter acquisition section 93.
Then, the SL setting section 94 outputs the set quantization matrixes to the inverse
quantization section 63.
5 [0144]
Note that, when the quantization matrix reuse flag indicates that the
quantization matrixes of the base layer are not reused, the SL setting section 94 can
set all necessary quantization matrixes of the enhancement layer according to the SL
parameters input from the SL parameter acquisition section 93.
10 [0145]
Next, the lossless decoding section 62 decodes the quantized data of the
enhancement layer from the encoded stream of the enhancement layer (Step S77).
Then, the lossless decoding section 62 outputs the decoded quantized data to the
inverse quantization section 63.
15 [0146]
Next, the inverse quantization section 63 inversely quantizes the quantized
data of the enhancement layer using a quantization step decided from the
quantization parameters (and the quantization matrixes) input from the inverse
quantization control section 90 to restore the transform coefficient data (Step S78).
20 Then, the inverse quantization section 63 outputs the restored transform coefficient
data to the inverse orthogonal transform section 64.
[0147]
Then, successive processes such as an inverse orthogonal transform,
addition of a predicted image and a predicted error, and filtering are executed.
25 [0148]
[5-3. Example of syntax]
(1) Basic example
FIGS. 16A to 16C are illustrative diagrams for describing examples of
syntax of an enhancement layer that can be employed for the technology according to
30 the present disclosure. Herein, a QP parameter reuse flag and a quantization matrix
reuse flag are encoded in a PPS of the enhancement layer.
45/104
SP351446WO00
[0149]
FIGS. 16A and 16B illustrate syntax of a PPS as an example. In FIG. 16A,
a QP parameter reuse flag "BL_chroma_qp_offset_flag is in the 15Ih row. When the
QP parameter reuse flag indicates "false," quantization parameter offsets are not
5 reused in the layers, and the quantization parameter offsets "pic cb_qp_offset" and
"piccr qp_offset" of the picture unit are encoded in the 18n and 19n rows. When
the QP parameter reuse flag indicates "true," encoding of the quantization parameter
offsets " p i c c b q p offset" and "pic c r q p offset" of the picture unit is skipped.
[0150]
10 In FIG. 16B, a function "scaling_list_data()" for SL parameters of the
enhancement layer is present in the 52n row. In FIG. 16C, specific syntax of the
function "scaling listdataO" is shown. In FIG. 16C, a quantization matrix reuse
flag "BL scaling_list flag" is present in the 2" row When the quantization matrix
reuse flag indicates "true," the quantization matrixes are reused in the layers. When
15 the quantization matrix reuse flag indicates "true" and the base layer is encoded in
the AVC scheme, however, only the quantization matrixes having the sizes of 4x4
pixels and 8x8 pixels are reused (see the 7th row; "sizeID=0" means the size of 4x4
pixels and "sizeID=l" means the size of 8x8 pixels). When the quantization
matrixes are not reused, the SL parameter which designates a quantization matrix of
20 the enhancement layer is encoded from the 8U row.
[0151]
Note that the syntax described herein is merely an example. The QP
parameter reuse flag and the quantization matrix reuse flag, for example, may each
be encoded in the header region other than the PPS (for example, an SPS or the like).
25 In addition, when the quantization matrixes are reused, the PPS may include the
above-described setting technique flag or residual data obtained when the
quantization matrixes are predicted.
[0152]
(2) Modified examples
30 FIGS. 16D and 16E each illustrate modified examples of syntax of the
enhancement layer. In these modified examples, SL parameters encoded by the EL
i
SP35144GWO00
46/104
encoding section lb and decoded by the EL decoding section 6b include reference
layer information as described using FIG. 9D.
[0153]
In the first modified example shown in FIG. 16D, a single common piece of
5 the reference layer information is encoded in a plurality of sizes and a plurality of
types of the quantization matrixes. The flag copyscalinglist from_refJayer_flag
in the 3r row of the extension of the SPS (sps_extension()) of FIG. 16D is a
quantization matrix reuse flag indicating whether or not a quantization matrix of the
enhancement layer should be generated based on a quantization matrix of a
10 subordinate layer designated by the reference layer information. When the
quantization matrix reuse flag indicates "true," the reference layer information
scaling listref layer in the 511 row is further encoded. As an example, the
reference layer information scalinglistref layer may designate a subordinate layer
which serves as a basis of duplication or prediction of a quantization matrix using a
15 layer number (for example, a 6-bit integer with no symbol or the like) given to a
reference target layer. Extension of a PPS (pps extension()) can also include the
same reference layer information of the syntax as the extension of the SPS.
[0154]
In the second modified example shown in FIG. 16E, different kinds of
20 reference layer information are encoded for each size and type of the quantization
matrixes. The functions scaling_list_extension() in the 3r row of the extension of
the SPS and in the 3rd row of the extension of the PPS of FIG, 16E define syntax for
reference layer information. The 2" row of the function scaling_list_extension()
means iteration with respect to about 4 sizes of quantization matrixes specified by a
25 variable sizeld. The following 3rd row means iteration with respect to about 6 types
of quantization matrixes specified by a variable matrix Id (since a maximum size of a
chroma component is 16x16 pixels, there are only about two types thereof with
respect to 32x32 pixels). The flag
copyscalinglistfrom_ref_layer_flag[sizeld][rnatrixld] in the 4th row in each
30 iteration is a quantization matrix reuse flag indicating that a quantization matrix
should be reused in the layers. When the quantization matrix reuse flag indicates
47/104
SP351446WO00
"true," the reference layer information scalinglisMef layer[sizeld][matrixfd] in the
6th row is further encoded. Here, the reference layer information may also
designate a subordinate layer which serves as a basis of duplication or prediction of
the quantization matrix using, for example, a layer number given to the reference
5 target layer.
[0155]
Note that when the quantization matrixes are reused only in layers which
share an image coding scheme as described using FIG. 9C, and the lowermost base
layer is encoded in the AVC scheme, the reference layer information can indicate a
10 layer number other than the number of the lowermost base layer (for example, zero)
(generally in SHVC, only the lowermost layer is a layer that can be encoded in the
AVC scheme).
[0156]
If the reference layer information as described above is introduced here, by
15 flexibly selecting a quantization matrix to be reused from a plurality of subordinate
layers, an optimum quantization matrix can be used in enhancement layers and thus
coding efficiency can be effectively enhanced.
[0157]
<6. Example of a combination of codecs>
20 The technology according to the present disclosure can be applied to
scalable video coding with various combinations of codecs. The number of layers
to be multiplexed may be any number equal to or greater than 2. When a
combination of codecs which are not standardized is used, however, there can be a
possibility of the decoder not operating normally. Thus, only combinations of
25 codecs which arc defined in advance may be permitted to be hierarchically encoded.
Instead, several combinations of codecs may be prohibited from being hierarchically
encoded.
[0158]
FIGS. 17A and 17B are illustrative diagrams for describing combinations of
30 codecs which can be permitted or prohibited.
[0159]
SP351446WO00
48/104
Referring to FIG. 17A, five layers from a lowermost first layer L10 to an
uppermost fifth layer L14 are shown. The first layer L10 and the second layer Lll
are encoded in the MPEG2 scheme. The third layer LI2 and the fourth layer LI3
are encoded in the AVC scheme. The fifth layer LI 4 is encoded in the HEVC
5 scheme. Among these layers, hierarchical encoding of the first layer L10 and the
second layer Lll is permitted due to their shared codec. Hierarchical encoding of
the second layer LI 1 and the third layer L12 can be prohibited due to the fact that the
combination of the MPEG2 scheme and the AVC scheme is not standardized.
Hierarchical encoding of the third layer L12 and the fourth layer LI 3 is permitted
10 due to their shared codec. Hierarchical encoding of the fourth layer LI 3 and the
fifth layer L14 is permitted due to the fact that the combination of the AVC scheme
and the HEVC scheme is standardized. In the example of FIG. I7A, a multiplexed
stream of the first layer L10 and the second layer Lll and a multiplexed stream of
the third layer L12, the fourth layer L13 and the fifth layer L14 can be multiplexed
15 using simulcast coding, rather than scalable video coding.
[0160]
Referring to FIG. I7B, five layers from a lowermost first layer L20 to an
uppermost fifth layer L24 are shown. The first layer L20 and the second layer L21
are encoded in the AVC scheme. The third layer L22 and the fourth layer L23 are
20 encoded in the MPEG2 scheme. The fifth layer L24 is encoded in the HEVC
scheme. Among these layers, hierarchical encoding of the first layer L20 and the
second layer L21 is permitted due to their shared codec. Hierarchical encoding of
the second layer L21 and the third layer L22 can be prohibited due to the fact that the
combination of the AVC scheme and the MPEG2 scheme is not standardized.
25 Hierarchical encoding of the third layer L22 and the fourth layer L23 is permitted
due to their shared codec. Hierarchical encoding of the fourth layer L23 and the
fifth layer L24 is permitted due to the fact that the combination of the MPEG2
scheme and the HEVC scheme is standardized. In the example of FIG. 17B, a
multiplexed stream of the first layer L20 and the second layer L21 and a multiplexed
30 stream of the third layer L22, the fourth layer L23 and the fifth layer L24 can be
multiplexed using simulcast coding rather than scalable video coding.
f
49/104
SP351446WO00
[0161]
<7. Application example>
[7-1. Application to various products]
The image encoding device 10 and the image decoding device 60 according
5 to the embodiment described above can be applied to various electronic appliances
such as a transmitter and a receiver for satellite broadcasting, cable broadcasting of a
cable TV, distribution on the Internet, distribution to terminals via cellular
communication, and the like, a recording device that records images in a medium
such as an optical disc, a magnetic disk or a flash memory, a reproduction device that
10 reproduces images from such storage media, and the like. Four application
examples will be described below.
[0162]
(1) First application example
FIG. 18 illustrates an example of a schematic configuration of a television
15 device to which the aforementioned embodiment is applied. A television device
900 includes an antenna 901, a tuner 902, a demultiplexer 903, a decoder 904, a
video signal processing section 905, a display 906, an audio signal processing section
907, a speaker 908, an external interface 909, a control section 910, a user interface
911, and a bus 912.
20 [0163]
The tuner 902 extracts a signal of a desired channel from a broadcast signal
received through the antenna 901 and demodulates the extracted signal. The tuner
902 then outputs an encoded bit stream obtained by the demodulation to the
demultiplexer 903. That is, the tuner 902 has a role as a transmission means
25 receiving the encoded stream in which an image is encoded, in the television device
900.
[0164]
The demultiplexer 903 separates a video stream and an audio stream of a
program to be viewed from the encoded bit stream and outputs each of the separated
30 streams to the decoder 904. The demultiplexer 903 also extracts auxiliary data such
as an electronic program guide (GEP) from the encoded bit stream and supplies the
SP351446WO00
50/104
extracted data to the control section 910. Here, the demultiplexer 903 may
descramble the encoded bit stream when it is scrambled.
[0165]
The decoder 904 decodes the video stream and the audio stream that are
5 input from the demultiplexer 903. The decoder 904 then outputs video data
generated by the decoding process to the video signal processing section 905.
Furthermore, the decoder 904 outputs audio data generated in the decoding process
to the audio signal processing section 907.
[0166]
10 The video signal processing section 905 reproduces the video data input
from the decoder 904 and displays the video on the display 906. The video signal
processing section 905 may also display an application screen supplied through the
network on the display 906. The video signal processing section 905 may further
perform an additional process, for example, noise reduction on the video data
15 according to the setting. Furthermore, the video signal processing section 905 may
generate an image of a graphical user interface (GUI) such as a menu, a button, or a
cursor and superpose the generated image onto the output image.
[0167]
The display 906 is driven by a drive signal supplied from the video signal
20 processing section 905 and displays video or an image on a video screen of a display
device (such as a liquid crystal display, a plasma display, or an OELD).
[0168]
The audio signal processing section 907 performs a reproduction process
such as D-A conversion and amplification on the audio data input from the decoder
25 904 and outputs the audio from the speaker 908. The audio signal processing
section 907 may also perform an additional process such as noise reduction on the
audio data.
[0169]
The external interface 909 is an interface for connecting the television
30 device 900 with an external device or a network. For example, the decoder 904
may decode a video stream or an audio stream received through, for example, the
SP351446WO00
51/104
external interface 909. In other words, the external interface 909 also has a role as
the transmission means receiving the encoded stream in which an image is encoded,
in the television device 900.
[0170]
5 The control section 910 includes a processor such as a central processing
unit (CPU) and a memory such as a random access memory (RAM) and a read only
memory (ROM). The memory stores a program executed by the CPU, program
data, EPG data, and data acquired through the network. The program stored in the
memory is read by the CPU at the start-up of the television device 900 and executed,
10 for example. By executing the program, the CPU controls operations of the
television device 900 in accordance with an operation signal that is input from the
user interface 911, for example.
[0171]
The user interface 911 is connected to the control section 910. The user
15 interface 911 includes a button and a switch for a user to operate the television
device 900 as well as a reception part of a remote control signal, for example. The
user interface 911 detects a user operation through these components, generates the
operation signal, and outputs the generated operation signal to the control section 910.
[0172]
20 The bus 912 connects the tuner 902, the demultiplexer 903, the decoder 904,
the video signal processing section 905, the audio signal processing section 907, the
external interface 909, and the control section 910 to each other.
[0173]
The decoder 904 in the television device 900 configured in the
25 aforementioned manner has a function of the image decoding device 60 according to
the aforementioned embodiment. Thus, when the television device 900 decodes an
image with a multi-layer codec, coding efficiency can be enhanced by reusing
quantization relevant parameters in layers.
[0174]
30 (2) Second application example
FIG. 19 illustrates an example of a schematic configuration of a mobile
SP351446WO00
52/104
telephone to which the aforementioned embodiment is applied. A mobile telephone
920 includes an antenna 921, a communication section 922, an audio codec 923, a
speaker 924, a microphone 925, a camera section 926, an image processing section
927, a multiplexing and separation section 928, a recording and reproduction section
5 929, a display 930, a control section 931, an operation section 932, and a bus 933.
[0175]
The antenna 921 is connected to the communication section 922. The
speaker 924 and the microphone 925 are connected to the audio codec 923. The
operation section 932 is connected to the control section 931. The bus 933 connects
10 the communication section 922, the audio codec 923, the camera section 926, the
image processing section 927, the multiplexing and separation section 928, the
recording and reproduction section 929, the display 930, and the control section 931
to each other.
[0176]
15 The mobile telephone 920 performs operations such as
transmitting/receiving an audio signal, transmitting/receiving an electronic mail or
image data, imaging an image, and recording data in various operation modes
including an audio call mode, a data communication mode, a photography mode, and
a videophone mode.
20 [0177]
In the audio call mode, an analog audio signal generated by the microphone
925 is supplied to the audio codec 923. The audio codec 923 then converts the
analog audio signal into audio data, performs A-D conversion on the converted audio
data, and compresses the data. The audio codec 923 thereafter outputs the
25 compressed audio data to the communication section 922. The communication
section 922 encodes and modulates the audio data to generate a transmission signal.
The communication section 922 then transmits the generated transmission signal to a
base station (not illustrated) through the antenna 921. Furthermore, the
communication section 922 amplifies a radio signal received through the antenna 921,
30 converts a frequency of the signal, and acquires a reception signal. The
communication section 922 thereafter demodulates and decodes the reception signal
SP351446WO00
53/104
to generate the audio data and output the generated audio data to the audio codec 923.
The audio codec 923 decompresses the audio data, performs D-A conversion on the
data, and generates the analog audio signal. The audio codec 923 then outputs the
audio by supplying the generated audio signal to the speaker 924.
5 [0178]
In addition, in the data communication mode, for example, the control
section 931 generates character data constituting an electronic mail, in accordance
with a user operation through the operation section 932. The control section 931
further causes characters to be displayed on the display 930. Moreover, the control
10 section 931 generates electronic mail data in accordance with a transmission
instruction from a user through the operation section 932 and outputs the generated
electronic mail data to the communication section 922. The communication section
922 encodes and modulates the electronic mail data to generate a transmission signal.
Then, the communication section 922 transmits the generated transmission signal to
15 the base station (not illustrated) through the antenna 921. The communication
section 922 further amplifies a radio signal received through the antenna 921,
converts a frequency of the signal, and acquires a reception signal. The
communication section 922 thereafter demodulates and decodes the reception signal,
restores the electronic mail data, and outputs the restored electronic mail data to the
20 control section 931. The control section 931 causes the content of the electronic
mail to be displayed on the display 930 as well as the electronic mail data to be
stored in a storage medium of the recording and reproduction section 929.
[0179]
The recording and reproduction section 929 includes an arbitrary readable
25 and writable storage medium. For example, the storage medium may be a built-in
storage medium such as a RAM or a flash memory, or may be an externally-mounted
storage medium such as a hard disk, a magnetic disk, a magneto-optical disc, an
optical disc, a USB memory, or a memory card.
[0180]
30 In the photography mode, for example, the camera section 926 images an
object, generates image data, and outputs the generated image data to the image
54/104
SP351446WO00
processing section 927. The image processing section 927 encodes the image data
input from the camera section 926 and stores an encoded stream in the storage
medium of the recording and reproduction section 929.
[0181]
5 In addition, in the videophone mode, for example, the multiplexing and
separation section 928 multiplexes a video stream encoded by the image processing
section 927 and an audio stream input from the audio codec 923, and outputs the
multiplexed streams to the communication section 922. The communication section
922 encodes and modulates the streams to generate a transmission signal. The
10 communication section 922 then transmits the generated transmission signal to the
base station (not illustrated) through the antenna 921. Moreover, the
communication section 922 amplifies a radio signal received through the antenna 921,
converts a frequency of the signal, and acquires a reception signal. The
transmission signal and the reception signal can include an encoded bit stream.
15 Then, the communication section 922 demodulates and decodes the reception signal
to restore the stream, and outputs the restored stream to the multiplexing and
separation section 928. The multiplexing and separation section 928 separates the
video stream and the audio stream from the input stream and outputs the video
stream and the audio stream to the image processing section 927 and the audio codec
20 923, respectively. The image processing section 927 decodes the video stream to
generate video data. The video data is then supplied to the display 930, and thereby
the display 930 displays a series of images. The audio codec 923 decompresses and
performs D-A conversion on the audio stream to generate an analog audio signal.
The audio codec 923 then supplies the generated audio signal to the speaker 924 to
25 output the audio.
[0182]
The image processing section 927 in the mobile telephone 920 configured in
the aforementioned manner has a function of the image encoding device 10 and the
image decoding device 60 according to the aforementioned embodiment. Thus,
30 when the mobile telephone 920 encodes or decodes an image with a multi-layer
codec, coding efficiency can be enhanced by reusing quantization relevant
SP351446WO00
55/104
parameters in layers.
[0183]
(3) Third application example
FIG. 20 illustrates an example of a schematic configuration of a recording
5 and reproduction device to which the aforementioned embodiment is applied. The
recording and reproduction device 940 encodes audio data and video data of a
received broadcast program and records the data into a recording medium, for
example. The recording and reproduction device 940 may also encode audio data
and video data acquired from another device and record the data into the recording
10 medium, for example. In addition, in response to a user instruction, for example,
the recording and reproduction device 940 reproduces the data recorded in the
recording medium on a monitor and from a speaker. The recording and
reproduction device 940 at this time decodes the audio data and the video data.
[0184]
15 The recording and reproduction device 940 includes a tuner 941, an external
interface 942, an encoder 943, a hard disk drive (HDD) 944, a disk drive 945, a
selector 946, a decoder 947, an on-screen display (OSD) 948, a control section 949,
and a user interface 950.
[0185]
20 The tuner 941 extracts a signal of a desired channel from a broadcast signal
received through an antenna (not illustrated) and demodulates the extracted signal.
The tuner 941 then outputs an encoded bit stream obtained from the demodulation to
the selector 946. That is, the tuner 941 has a role as a transmission means in the
recording and reproduction device 940.
25 [0186]
The external interface 942 is an interface for connecting the recording and
reproduction device 940 with an external device or a network. The external
interface 942 may be, for example, an IEEE 1394 interface, a network interface, a
USB interface, or a flash memory interface. The video data and the audio data
30 received through the external interface 942 are input to the encoder 943, for example.
That is, the external interface 942 has a role as a transmission means in the recording
SP351446WO00
56/104
and reproduction device 940.
[0187]
The encoder 943 encodes the video data and the audio data when the video
data and the audio data input from the external interface 942 are not encoded. The
5 encoder 943 thereafter outputs an encoded bit stream to the selector 946.
[0188]
The HDD 944 records the encoded bit stream in which content data such as
video and audio is compressed, various programs, and other data into an internal hard
disk. In addition, the HDD 944 reads these data from the hard disk when
10 reproducing the video and the audio.
[0189]
The disk drive 945 records and reads data into and from a recording medium
which is mounted to the disk drive. The recording medium mounted to the disk
drive 945 may be, for example, a DVD disk (such as DVD-Video, DVD-RAM,
15 DVD-R, DVD-RW, DVD+R, or DVD+RW) or a Blu-ray (registered trademark) disk.
[0190]
The selector 946 selects the encoded bit stream input from the tuner 941 or
the encoder 943 when recording the video and audio, and outputs the selected
encoded bit stream to the HDD 944 or the disk drive 945. In addition, when
20 reproducing the video and audio, the selector 946 outputs the encoded bit stream
input from the HDD 944 or the disk drive 945 to the decoder 947.
[0191]
The decoder 947 decodes the encoded bit stream to generate the video data
and the audio data. Then, the decoder 947 outputs the generated video data to the
25 OSD 948. In addition, the decoder 904 outputs the generated audio data to an
external speaker.
[0192]
The OSD 948 reproduces the video data input from the decoder 947 and
displays the video. The OSD 948 may also superpose an image of a GUI, for
30 example, a menu, a button, or a cursor on the displayed video.
[0193]
SP351446WO00
57/104
The control section 949 includes a processor such as a CPU and a memory
such as a RAM and a ROM. The memory stores a program executed by the CPU as
well as program data. The program stored in the memory is read by the CPU at the
start-up of the recording and reproduction device 940 and executed, for example.
5 By executing the program, the CPU controls operations of the recording and
reproduction device 940 in accordance with an operation signal that is input from the
user interface 950, for example.
[0194]
The user interface 950 is connected to the control section 949. The user
10 interface 950 includes a button and a switch for a user to operate the recording and
reproduction device 940 as well as a reception part of a remote control signal, for
example. The user interface 950 detects a user operation through these components,
generates the operation signal, and outputs the generated operation signal to the
control section 949.
15 [0195]
The encoder 943 in the recording and reproduction device 940 configured in
the aforementioned manner has a function of the image encoding device 10
according to the aforementioned embodiment. In addition, the decoder 947 has a
function of the image decoding device 60 according to the aforementioned
20 embodiment. Thus, when the recording and reproduction device 940 encodes or
decodes an image with a multi-layer codec, coding efficiency can be enhanced by
reusing quantization relevant parameters in layers.
[0196]
(4) Fourth application example
25 FIG. 21 illustrates an example of a schematic configuration of an imaging
device to which the aforementioned embodiment is applied. The imaging device
960 images an object, generates an image, encodes image data, and records the data
into a recording medium.
[0197]
30 The imaging device 960 includes an optica! block 961, an imaging section
962, a signal processing section 963, an image processing section 964, a display 965,
3
SP351446WO00
73/104
device 1005, the tablet device 1006 or the mobile phone 1007 in the data
transmission system 1000 described using FIG. 22, the broadcasting station 1101 or
the terminal device 1102 of the data transmission system 1100 described using FIG.
23, or the imaging device 1201 or the stream storage device 1202 of the data
5 transmission system 1200 described using FIG. 24. Further, the video set 1300 may
correspond to a content reproduction system to be exemplified in FIG. 31 or a device
included in a wireless communication system to be exemplified in FIG. 40.
[0256]
(2) Video processor
10 FIG. 29 is a block diagram showing an example of a schematic
configuration of the video processor 1332. The video processor 1332 has a function
of encoding an input video signal and an input audio signal and thereby generating
video data and audio data, and a function of decoding the encoded video data and
audio data and thereby generating an output video signal and an output audio signal.
15 [0257]
Referring to FIG. 29, the video processor 1332 has a video input processing
section 1401, a first scaling section 1402, a second scaling section 1403, a video
output processing section 1404, a frame memory 1405, a memory control section
1406, an encoder and decoder engine 1407, video elementary stream (ES) buffers
20 1408A and 1408B, audio ES buffers 1409A and 1409B, an audio encoder 1410, an
audio decoder 1411, a multiplexing section (MUX) 1412, a demultiplexing section
(DEMUX) 1413, and a stream buffer 1414.
[0258]
The video input processing section 1401 converts, for example, a video
25 signal input from the connectivity module 1321 into digital image data. The first
scaling section 1402 performs format conversion and scaling (enlargement or
reduction) on the image data input from the video input processing section 1401.
The second scaling section 1403 performs format conversion and scaling
(enlargement or reduction) on the image data to be output to the video output
30 processing section 1404. The format conversion performed by the first scaling
section 1402 and the second scaling section 1403 may be conversion between, for
SP351446WO00
74/104
example, the format of 4:2:2/Y-Cb-Cr and the format of 4:2:0/Y-Cb-Cr. The video
output processing section 1404 converts digital image data into an output video
signal, and outputs the output video signal to, for example, the connectivity module
1321.
5 [0259]
The frame memory 1405 is a memory device for storing image data, and is
shared by the video input processing section 1401, the first scaling section 1402, the
second scaling section 1403, the video output processing section 1404, and the
encoder and decoder engine 1407. The frame memory 1405 may be realized using,
10 for example, a semiconductor memory such as a DRAM.
[0260]
The memory control section 1406 controls access to the frame memory
1405 based on a synchronization signal input from the encoder and decoder engine
1407 according to an access schedule for the frame memory 1405 stored in an access
15 management table I406A. The access management table 1406A depends on
processes executed by the encoder and decoder engine 1407, the first scaling section
1402, the second scaling section 1403, and the like, and is updated by the memory
control section 1406.
[0261]
20 The encoder and decoder engine 1407 performs an encoding process of
encoding on image data to generate an encoded video stream and a decoding process
for decoding image data from an encoded video stream. For example, the encoder
and decoder engine 1407 encodes image data read from the frame memory 1405 and
sequentially writes an encoded video stream on the video ES buffer 1408A. Tn
25 addition, for example, the encoder and decoder engine sequentially reads an encoded
video stream from the video ES buffer 1408B and writes the decoded image data on
the frame memory 1405. The encoder and decoder engine 1407 can use the frame
memory 1405 as a work area for the processes. The encoder and decoder engine
1407 outputs a synchronization signal to the memory control section 1406 at, for
30 example, a timing at which a process of each largest coding unit (LCU) is started.
[0262]
SP351446WO00
75/104
The video ES buffer 1408A buffers the encoded video stream generated by
the encoder and decoder engine 1407. The encoded video stream buffered by the
video ES buffer 1408A is output to the multiplexing section 1412. The video ES
buffer 1408B buffers the encoded video stream input from the demultiplexing section
5 1413. The encoded video stream buffered by the video ES buffer 1408B is output
to the encoder and decoder engine 1407.
[0263]
The audio ES buffer 1409A buffers an encoded audio stream generated by
the audio encoder 1410. The encoded audio stream buffered by the audio ES buffer
10 1409A is output to the multiplexing section 1412. The audio ES buffer 1409B
buffers an encoded audio stream input from the demultiplexing section 1413. The
encoded audio stream buffered by the audio ES buffer I409B is output to the audio
decoder 1411.
[0264]
15 The audio encoder 1410 performs digital conversion on, for example, an
input audio signal input from the connectivity module 1321, and encodes the input
audio signal according to an audio encoding scheme, for example, an MPEG audio
scheme or an audio code number 3 (AC3) scheme. The audio encoder 1410
sequentially writes an encoded audio stream on the audio ES buffer 1409A. The
20 audio decoder 1411 decodes audio data from the encoded audio stream input from
the audio ES buffer 1409B and converts the data into an analog signal. The audio
decoder 1411 outputs, for example, an audio signal to the connectivity module 1321
as a reproduced analog audio signal.
[0265]
25 The multiplexing section 1412 multiplexes an encoded video stream and an
encoded audio stream to generate a multiplexed bit stream. A format of the
multiplexed bit stream may be any format. The multiplexing section 1412 may add
predetermined header information to the bit stream. In addition, the multiplexing
section 1412 may convert the format of the stream. For example, the multiplexing
30 section 1412 can generate a transport stream (a bit stream in a format for transport)
obtained by multiplexing the encoded video stream and the encoded audio stream.
i
SP35144GWO00
76/104
In addition, the multiplexing section 1412 can generate file data (data in a format for
recording) obtained by multiplexing the encoded video stream and the encoded audio
stream.
[0266]
5 The demultiplexing section 1413 demultiplexes an encoded video stream
and an encoded audio stream from a multiplexed bit stream using a method opposite
to multiplexing performed by the multiplexing section 1412. In other words, the
demultiplexing section 1413 extracts (or separates) a video stream and an audio
stream from a bit stream read from the stream buffer 1414. The demultiplexing
10 section 1413 may convert (inversely convert) the format of a stream. For example,
the demultiplexing section 1413 may acquire a transport stream which can be input
from the connectivity module 1321 or the broadband modem 1333 via the stream
buffer 1414, and convert the transport stream into a video stream and an audio stream.
In addition, the demultiplexing section 1413 may acquire file data read from a
15 storage medium by the connectivity module 1321 via the stream buffer 1414 and
convert the file data into a video stream and an audio stream.
[0267]
The stream buffer 1414 buffers a bit stream. For example, the stream
buffer 1414 buffers a transport stream input from the multiplexing section 1412, and
20 outputs the transport stream to, for example, the connectivity module 1321 or the
broadband modem 1333 at a predetermined timing or according to a request from the
outside. In addition, the stream buffer 1414, for example, buffers file data input
from the multiplexing section 1412, and outputs the file data to, for example, the
connectivity module 1321 for recording of the data at a predetermined timing or
25 according to a request from the outside. Furthermore, the stream buffer 1414
buffers a transport stream acquired via, for example, the connectivity module 1321 or
the broadband modem 1333, and outputs the transport stream to the demultiplexing
section 1413 at a predetermined timing or according to a request from the outside.
In addition, the stream buffer 1414 buffers file data read from a storage medium by,
30 for example, the connectivity module 1321, and outputs the file data to the
demultiplexing section 1413 at a predetermined timing or according to a request
SP351446WO00
77/104
from the outside.
[0268]
In the video processor 1332 configured as described above, the technology
according to the present disclosure can be used in, for example, the encoder and
5 decoder engine 1407. In this case, the video processor 1332 is a chip or a module to
which the technology according to the present disclosure is applied.
[0269]
FIG. 30 is a block diagram showing another example of the schematic
configuration of the video processor 1332. In the example of FIG. 30, the video
10 processor 1332 has functions of encoding and decoding video data using a
predetermined scheme.
[0270]
Referring to FIG 30, the video processor 1332 has a control section 1511, a
display interface 1512, a display engine 1513, an image processing engine 1514, an
15 internal memory 1515, a codec engine 1516, a memory interface 1517, a
multiplexing and demultiplexing section 1518, a network interface 1519, and a video
interface 1520.
[0271]
The control section 1511 controls operations of various processing sections
20 inside the video processor 1332 such as the display interface 1512, the display engine
1513, the image processing engine 1514, the codec engine 1516, and the like. The
control section 1511 has, for example, a main CPU 1531, a sub CPU 1532, and a
system controller 1533. The main CPU 153! executes programs for controlling
operations of each processing sections of the video processor 1332. The main CPU
25 1531 supplies generated control signals to each processing section through execution
of the programs. The sub CPU 1532 plays an auxiliary role of the main CPU 1531.
For example, the sub CPU 1532 executes child processes and subroutines of the
programs executed by the main CPU 1531. The system controller 1533 manages
execution of the programs by the main CPU 1531 and the sub CPU 1532.
30 [0272]
The display interface 1512 outputs image data to, for example, the
SP351446WO00
78/104
connectivity module 1321 under control of the control section 1511. For example,
the display interface 1512 outputs an analog image signal converted from digital
image data or the digital image data itself to a display that is connected to the
connectivity module 1321. The display engine 1513 executes format conversion,
5 size conversion, and color space conversion on image data so that attributes of the
image data fit specifications of the display that is an output destination under control
of the control section 1511. The image processing engine 1514 executes image
processing which can include a filtering process for the purpose of image quality
i
improvement or the like on the image data under control of the control section 1511.
10 [0273]
The internal memory 1515 is a memory device which is provided inside the
video processor 1332 and shared by the display engine 1513, the image processing
engine 1514, and the codec engine 1516. The internal memory 1515 is used when
image data is input and output between, for example, the display engine 1513, the
15 image processing engine 1514, and the codec engine 1516. The internal memory
1515 may be any type of memory device. For example, the internal memory 1515
may have a relatively small memory size for storing image data of a block unit and
relevant parameters. The internal memory 1515 may be a memory which has a
small capacity but a high response speed (for example, relatively to the external
20 memory 1312), for example, a static random access memory (SRAM).
[0274]
The codec engine 1516 performs an encoding process of encoding image
data to generate an encoded video stream and a decoding process of decoding image
data from an encoded video stream. An image coding scheme supported by the
25 codec engine 1516 may be one or more arbitrary schemes. In the example
illustrated in FIG 30, the codec engine 1516 has an MPEG-2 video block 1541, an
AVC/H.264 block 1542, an HEVC/H.265 block 1543, an HEVC/H.265 (scalable)
block 1544, an HEVC/H.265 (multi-view) block 1545, and an MPEG-DASH block
1551. These respective functional blocks encode and decode image data according
30 to a corresponding image coding scheme.

Claim I
An image processing device comprising:
a control section configured to set, based on a first quantization parameter
5 offset set for a chroma component of a first layer, a second quantization parameter
offset for a chroma component of a second layer decoded with reference to the first
layer; and
an inverse quantization section configured to inversely quantize transform
coefficient data of the chroma component of the second layer using a quantization
10 parameter computed using the second quantization parameter offset set by the control
section.
Claim 2
The image processing device according to claim 1, wherein the second
15 quantization parameter offset is equal to the sum of the first quantization parameter
offset and a quantization parameter offset difference.
Claim 3
The image processing device according to claim 2, further comprising:
20 a decoding section configured to decode the quantization parameter offset
difference from an encoded stream.
Claim 4
The image processing device according to claim 1, wherein the first
25 quantization parameter offset is equal to the sum of an offset set for the first layer in
a picture unit and an offset set for the fust layer in a slice unit.
Claim 5
The image processing device according to claim 1, wherein the first
30 quantization parameter offset is equal to an offset set for the first layer in a picture
unit.
SP351446WO00
100/104
Claim 6
The image processing device according to claim 1, wherein the control
section sets the second quantization parameter offset separately for a Cb component
5 and a Cr component of the second layer.
Claim 7
The image processing device according to claim 1, wherein, when a first
flag decoded from an encoded stream indicates that the second quantization
10 parameter offset should be set based on the first quantization parameter offset, the
control section sets the second quantization parameter offset based on the first
quantization parameter offset.
Claim 8
15 The image processing device according to claim I,
wherein, based on a first quantization matrix set for the first layer, the
control section sets a second quantization matrix for the second layer, and
wherein the inverse quantization section inversely quantizes transform
coefficient data of the second layer using the second quantization matrix set by the
20 control section.
Claim 9
The image processing device according to claim 8, wherein, when reference
layer information designates the first layer as a reference layer, the control section
25 sets the second quantization matrix for the second layer based on the first
quantization matrix set for the first layer.
Claim 10
The image processing device according to claim 8, wherein the control
30 section duplicates or predicts the second quantization matrix from the first
quantization matrix.
Claim 11
The image processing device according to claim 8, wherein, when a second
flag decoded from an encoded stream indicates that the second quantization matrix
5 should be set based on the first quantization matrix, the control section sets the
second quantization matrix based on the first quantization matrix.
Claim 12
The image processing device according to claim 11, wherein the second flag
10 is decoded from the encoded stream separately for different quantization matrix sizes,
different prediction modes, or different color components.
Claim 13
The image processing device according to claim 10, wherein the control
15 section selects a setting technique for setting the second quantization matrix
according to a third flag decoded from an encoded stream.
Claim 14
The image processing device according to claim 13, wherein the third flag is
20 decoded from the encoded stream separately for different quantization matrix sizes,
different prediction modes, or different color components.
Claim 15
The image processing device according to claim 8,
25 wherein the second layer is encoded in a High Efficiency Video Coding
(HEVC) scheme, and
wherein, when the first layer is encoded in an Advanced Video Coding
(AVC) scheme, the control section sets the second quantization matrix having a size
of 8x8 pixels or smaller for the second layer based on the first quantization matrix,
30 and sets a third quantization matrix having a size of 16x16 pixels or greater for the
second layer without being based on tiie first quantization matrix.
Claim 16
An image processing method comprising:
setting, based on a first quantization parameter offset set for a chroma
5 component of a first layer, a second quantization parameter offset for a chroma
component of a second layer decoded with reference to the first layer; and
inversely quantizing transform coefficient data of the chroma component of
the second layer using a quantization parameter computed using the set second
quantization parameter offset.
10
Claim 17
An image processing device comprising:
a quantization section configured to quantize transform coefficient data of a
chroma component of a second layer encoded with reference to a first layer using a
15 given quantization parameter; and
an encoding section configured to encode a second quantization parameter
offset of a chroma component of the second layer computed based on a first
quantization parameter offset set for a chroma component of the first layer and the
given quantization parameter.
20
Claim 18
The image processing device according to claim 17,
wherein the quantization section quantizes transform coefficient data of the
second layer using a given quantization matrix, and
25 wherein, when a decoder should set the given quantization matrix for the
second layer based on a quantization matrix set for the first layer, the encoding
section refrains from encoding the given quantization matrix.
Claim 19
30 The image processing device according to claim 18, wherein the encoding
section encodes reference layer information which designates the first layer as a
reference layer to be referred to when the given quantization matrix is set.
Claim 20
An image processing method comprising:
quantizing transform coefficient data of a chroma component of a second
layer encoded with reference to a first layer using a given quantization parameter;
and
encoding a second quantization parameter offset of the chroma component
of the second layer computed based on a first quantization parameter offset set for a
chroma component of the first layer and the given quantization parameter.