Title of the invention: Image processing apparatus and image processing method
Technical field
[0001]
　The present disclosure relates to an image processing apparatus and an image processing method.
Background technology
[0002]
　Conventionally, in order to efficiently transmit, store or record digital images, it is said to achieve about twice the coding (compression) efficiency of H.264 / AVC or H.264 / AVC. Many video encoding methods such as H.265 / HEVC have been standardized. JVET (Joint Video Experts Team), a joint standardization organization of ITU-T and ISO / IEC, is a next-generation video coding method with the aim of further improving coding efficiency compared to H.265 / HEVC. We have started studying for standardization of FVC (Future Video Coding). The FVC reference software under development based on the HEVC model is called JEM (Joint Exploration Model), and various technical elements incorporated into JEM are described in Non-Patent Document 1.
[0003]
　Existing video coding schemes use various techniques such as prediction (intra / inter-prediction), orthogonal transformation, quantization and entropy coding. In particular, in the quantization process, the high-frequency component of the conversion coefficient is quantized coarser than the low-frequency component in the frequency domain after orthogonal conversion to achieve a desired data rate while suppressing subjective deterioration of image quality. be able to. According to H.265 / HEVC (hereinafter simply referred to as HEVC), orthogonal transformation and quantization are performed for each block called TU (Transform Unit). Candidates for TU size are 4x4, 8x8, 16x16 or 32x32, and the quantization matrix corresponding to some TU sizes can be signaled from the encoder to the decoder. The quantization matrix influences the quantization step for each frequency component of the conversion coefficient of each block. In FVC, the maximum size of TUs has been extended to 128x128, and non-square TUs are also allowed.
[0004]
　The more diverse the TU size, the more diverse the quantization matrix used, but signaling all of these quantization matrices increases the overhead in the encoded bitstream and reduces the coding efficiency. Therefore, Patent Document 1 and Patent Document 2 signal not all but only a part of the quantization matrix used, and generate the remaining quantization matrix from the signaled quantization matrix to increase the overhead. We are proposing techniques to avoid it.
Prior art literature
Non-patent literature
[0005]
Non-Patent Document 1: J. Chen, E. Alshina, GJ Sullivan, JR Ohm and J. Boyce, "Algorithm Description of Joint Exploration Test Model (JEM7)", JVET-G1001, Joint Video Exploration Team (JVET) of ITU- T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29/WG 11 7th Meeting: Torino, IT, 13-21 July 2017
Patent documents
[0006]
Patent Document 1: International Publication No. 2012/077408
Patent Document 2: International Publication No. 2012/160890
Outline of the invention
Problems to be solved by the invention
[0007]
　There is a trade-off between the decrease in coding efficiency due to the signaling of the quantization matrix and the effect of generating another quantization matrix from one quantization matrix on the performance of the device. In particular, when the combination of block size and shape is highly diversified as in FVC, not only the coding efficiency but also the processing cost required to generate the quantization matrix (for example, occupancy of hardware resources, processing delay, or processing delay) Increased power consumption, etc.) cannot be ignored either.
[0008]
　Therefore, it is desirable to provide an improved mechanism for efficiently generating or signaling a quantization matrix.
Means to solve problems
[0009]
　According to the present disclosure, for a decoding unit that decodes scaling list data to generate a first quantization matrix of first size, and for a second size conversion block to which zeroing of high frequency components is applied. A generation unit that generates the second quantization matrix of the above by referring only to a submatrix of the first quantization matrix generated by the decoding unit, and the second quantum generated by the generation unit. An image processing apparatus is provided that includes an inverse quantization unit that inversely quantizes the quantization conversion coefficient of the conversion block of the second size using a conversion matrix.
[0010]
　Further, according to the present disclosure, it is an image processing method executed by an image processing apparatus, in which scaling list data is decoded to generate a first quantization matrix of a first size, and a high-frequency component. Generating and generating a second quantization matrix for a second size conversion block to which zeroization is applied by referring only to the submatrix of the first quantization matrix generated. An image processing method including inverse quantization of the quantization conversion coefficient of the conversion block of the second size by using the second quantization matrix is ​​provided.
[0011]
　Further, according to the present disclosure, the second quantization matrix for the conversion block of the second size to which the zeroing of the high frequency component is applied is limited to the submatrix of the first quantization matrix of the first size. The conversion coefficient of the generation unit generated by referring to and the conversion block of the second size in the image to be encoded is quantized by using the second quantization matrix generated by the generation unit. An image including a quantization unit that generates a quantization conversion coefficient, and a coding unit that encodes a scaling list representing the quantization conversion coefficient and the first quantization matrix to generate a coded stream. A processing device is provided.
[0012]
　Further, according to the present disclosure, a second quantization matrix for a second size conversion block to which zeroing of high frequency components is applied, which is an image processing method executed by an image processing apparatus, is described. The second size is generated by referring only to a submatrix of the first quantization matrix of size 1, and the conversion coefficient of the second size conversion block in the image to be encoded is generated. Quantization using a quantization matrix to generate a quantization conversion coefficient, and encoding the scaling list representing the quantization conversion coefficient and the first quantization matrix to generate a coded stream. An image processing method including the above is provided.
Effect of the invention
[0013]
　According to the technique according to the present disclosure, a quantization matrix can be efficiently generated or signaled.
　It should be noted that the above effects are not necessarily limited, and either in combination with or in place of the above effects, any of the effects shown herein, or any other effect that can be grasped from this specification. May be played.
A brief description of the drawing
[0014]
[Fig. 1] Fig. 1 is an explanatory diagram for explaining the types of quantization matrices that can be used in HEVC.
[Fig. 2] Fig. 2 is an explanatory diagram showing an example of QTBT block division in FVC.
[Fig. 3A] Fig. 3A is an explanatory diagram for explaining zero conversion coefficient of a square conversion block in FVC.
[Fig. 3B] Fig. 3B is an explanatory diagram for explaining zero conversion coefficient of a non-square conversion block in FVC.
FIG. 4 is an explanatory diagram for explaining an example of a basic implementation of the technology according to the present disclosure on the decoder side.
FIG. 5 is an explanatory diagram for explaining the generation of a quantization matrix for a transformation block to which zeroization is not applied.
FIG. 6A is a first explanatory diagram for explaining the generation of a quantization matrix for a transformation block to which zeroization is applied.
FIG. 6B is a second explanatory diagram for explaining the generation of a quantization matrix for a transformation block to which zeroization is applied.
FIG. 6C is a third explanatory diagram for explaining the generation of a quantization matrix for a transformation block to which zeroization is applied.
FIG. 7 is an explanatory diagram for explaining an example of a basic implementation of the technology according to the present disclosure on the encoder side.
FIG. 8 is a block diagram showing an example of an encoder configuration according to the first embodiment.
FIG. 9 is a flowchart showing an example of a flow of quantization-related processing executed in the encoder shown in FIG.
FIG. 10 is a flowchart showing an example of the flow of the quantization matrix generation process.
FIG. 11 is a flowchart showing an example of the flow of scaling list coding processing.
FIG. 12 is a block diagram showing an example of a configuration of a decoder according to the first embodiment.
FIG. 13 is a flowchart showing an example of a flow of inverse quantization-related processing executed in the decoder shown in FIG.
FIG. 14 is a flowchart showing an example of a flow of scaling list data decoding processing.
FIG. 15 is a block diagram showing an example of an encoder configuration according to a second embodiment.
FIG. 16 is a flowchart showing an example of a flow of quantization-related processing executed in the encoder shown in FIG.
FIG. 17 is a block diagram showing an example of a configuration of a decoder according to a second embodiment.
FIG. 18 is a flowchart showing an example of a flow of inverse quantization-related processing executed in the decoder shown in FIG.
[Fig. 19] Fig. 19 is a block diagram showing an example of a hardware configuration.
Mode for carrying out the invention
[0015]
　Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals to omit duplicate description.
[0016]
　The scope of the present disclosure is not limited to the content of the following detailed description, and the contents of the following references REF1 to REF3 known at the time of filing are also incorporated herein by reference. That is, the contents of these references also serve as a basis for determining the sufficiency of support requirements. For example, the quad-tree block structure described in Reference REF2 and the QTBT (Quad Tree Plus Binary Tree) block structure described in Reference REF3 are not explicitly described. To the extent, it is included in the scope of this disclosure. Similarly, technical terms such as Parsing, Syntax and Semantics are also included in the scope of the present disclosure, whether directly referred to in the detailed description below. Support requirements are met for those terms that may be included in the claims.
　・ REF1: Recommendation ITU-T H.264 (04/2017) “Advanced video coding for generic audiovisual services”, April 2017
　・ REF2: Recommendation ITU-T H.265 (12/2016) “High efficiency video coding”, December 2016
　・ REF3: J. Chen, E. Alshina, GJ Sullivan, JR Ohm and J. Boyce, “Algorithm Description of Joint Exploration Test Model (JEM7)”, JVET-G1001, Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29/WG 11 7th Meeting: Torino, IT, 13-21 July 2017
[0017]
　Hereinafter, the explanations will be given in the following order.
　　1. 1. Outline
　　　1-1. Definition of terms
　　　1-2. Existing technology and problems
　　　1-3. Principles of Technology Related to the Disclosure
　　　1-4. Zero control
　　　1-5. Controlling the size of the basic quantization matrix
　　2. First Embodiment
　　　2-1. Encoder configuration
　　　2-2. Examples of syntax and semantics
　　　2-3. Selection of reference quantization matrix
　　　2-4. Resampling method
　　　2-5. Flow of quantization-related processing at the time of coding
　　　2-6. Decoder configuration
　　　2-7. Flow of dequantization related processing at the time of decoding
　　3. Second Embodiment
　　　3-1. Encoder configuration
　　　3-2. Flow of quantization-related processing at the time of coding
　　　3-3. Decoder configuration
　　　3-4. Flow of dequantization related processing at the time of decoding
　　4. Hardware configuration example
　　5. Summary
[0018]
　<1. Overview>
　　[1-1. Definitions of Terms] Definitions of
　some terms used herein are as follows:
　• A transformation block is treated as a processing unit for orthogonal and inverse orthogonal transformations set in an image. Refers to a block. The conversion block has a size of M × N, where M is the number of horizontal components and N is the number of vertical components (M and N are integers greater than or equal to 2). When M = N, the conversion block is square, and when M ≠ N, the conversion block is non-square.
　-The conversion coefficient means a coefficient for each frequency component derived by converting a signal sample in the spatial domain into a frequency domain. The transformation between the spatial domain and the frequency domain can be an orthogonal transform (or vice versa), such as the Discrete Cosine Transform (DCT) or the Discrete Cosine Transform (DST). Normally, when the signal sample of the two-dimensional M × N conversion block is orthogonally converted, a two-dimensional array of the same size, that is, the conversion coefficient of M × N is derived. Typically, the upper left component of the derived two-dimensional array is the direct current (DC) component, with components farther from the upper left corresponding to higher frequencies.
　-The quantization matrix is ​​a matrix that expresses the quantization steps used when the two-dimensional array of conversion coefficients is quantized in different quantization steps for each component in the frequency domain. The element of the quantization matrix does not necessarily have to be the quantization step itself, and may be, for example, a parameter input to the relational expression for deriving the quantization step.
　-A scaling list is a list of a series of values ​​derived by making the elements of the quantization matrix one-dimensional in some scan order. That is, the scaling list is a kind of information representing the quantization matrix. Usually, when trying to explicitly signal a two-dimensional quantization matrix in a coded stream, the two-dimensional quantization matrix is ​​encoded after being converted into a scaling list.
[0019]
　　[1-2. Existing technology and problems]
　　(1) Quantization matrix signaling in
　HEVC In HEVC, the CU (Coding Unit), which is the processing unit of coding processing, is set in the shape of a quad-tree in the image. .. Then, for the CU for which inter-prediction is selected as the prediction type, one or more TUs are set by directly dividing the CU, and for the CU for which intra-prediction is selected, each PU (Prediction Unit) in the CU. One or more TUs are set by dividing. In HEVC, this TU (Transform Unit) can correspond to a transformation block. The minimum size of the TU is 4x4 and the maximum size is 32x32. Orthogonal transformation and quantization in the encoder, and inverse quantization and inverse quantization in the decoder are performed using these TUs as processing units. In HEVC, the quantization steps may be uniform within the conversion block, or may differ from position to position within the conversion block (ie, from frequency component of conversion factor). For example, assuming that the bit rates of the coded streams are the same, the subjective deterioration of image quality can be relatively suppressed by quantizing the high frequency component of the conversion coefficient coarser than the low frequency component.
[0020]
　The different quantization steps for each position in the transformation block are represented by the elements of the quantization matrix of the same size as the transformation block. In HEVC, for each of the 4 × 4, 8 × 8 and 16 × 16 TU size candidates, a total of 6 (=) are different for each combination of the prediction type (intra or inter) and the color component (Y, Cb or Cr). 2 × 3) types of quantization matrices can be used. For a TU size of 32 x 32, a total of two types of quantization matrices can be used, which differ for each prediction type (intra or inter). The size of the quantization matrix is ​​identified by the size ID, and the combination of the predicted type and the color component is identified by the matrix ID. The types of quantization matrices available in HEVC are schematically shown in FIG.
[0021]
　In addition, HEVC defines three types of default quantization matrices:
　・ Independent of prediction type and color component ・ 4 × 4 flat default quantization matrix
　・ Independent of color component, intra 8x8 non-flat default quantization matrix
　for prediction ・8x8 non-flat default quantization matrix for inter-prediction that does not depend on color components
16 × 16 default quantization matrix is ​​8 It is generated by upsampling the elements of the default quantization matrix of × 8 by the Nearest Neighbor method. The 32x32 default quantization matrix is ​​generated by similarly upsampling the elements of the 16x16 default quantization matrix. Then, in HEVC, when it is desired to use a quantization matrix different from the default quantization matrix, it is possible to explicitly signal a unique quantization matrix defined by the user. However, 4x4 and 8x8 quantization matrices can be signaled as a whole by scanning all of their elements, while 16x16 and 32x32 quantization matrix signaling is 8x8. It is realized by signaling and upsampling of the quantization matrix of. It should be noted that the values ​​of the elements for the DC component of the 16 × 16 and 32 × 32 unique quantization matrices can be signaled separately.
[0022]
　When explicitly signaling the quantization matrix, as described above, the elements of the two-dimensional quantization matrix are mapped to the scaling list, which is a one-dimensional array according to a certain scan order, and each element of the scaling list immediately precedes. It is encoded as a difference value from the element of. The syntax of scaling list data pointing to the scaling list thus generated in HEVC is described in Section 7.3.4 of reference REF2.
[0023]
　　(2) QTBT block structure in
　FVC In FVC, CU is set in a QTBT shape in the image, and TU is shared with CU. More specifically, each of the CTUs (Coding Tree Units) arranged in a grid pattern in the image is first divided into quad-trees, and if necessary, squares of various sizes. Quadtree leaf is formed. Then, each quadtree leaf is further divided into a binary-tree shape at either a horizontal or vertical boundary, if necessary. FIG. 2 shows an example of QTBT block division in FVC. The image Im0 shown in FIG. 2 contains four CTUs, each having a size of 128 × 128. The upper left CTU contains 13 CUs formed by four recursive QT splits, of which the smallest CU size is 8x8 and the largest CU size is 64x64. The lower left CTU contains 5 CUs formed by 4 recursive BT divisions. The CTU in the upper right contains 9 CUs formed by multiple recursive QT and BT divisions. The lower right CTU is not divided and contains one CU. Although the smallest CU size in the example of FIG. 2 is 8x8, square or non-square CUs with side lengths of 2 or 4 are also acceptable. In FVC, each of these CUs also serves as a conversion block.
[0024]
　　(3) Zeroization of high-frequency components in FVC As
　can be understood from the above explanation, the upper limit of the conversion block size (that is, TU size) allowed in HEVC was 32 × 32, whereas it is allowed in FVC. The upper limit of the conversion block size to be generated is raised to 128 x 128. Such a large conversion block can be used, for example, to efficiently encode a so-called 4K high-resolution video. However, in reality, the high frequency component of the conversion coefficient obtained by performing the orthogonal conversion with a size larger than 32 × 32 does not contribute much to the subjective image quality as compared with the data size. Therefore, in FVC, a method of zeroing the frequency component on the high frequency side after the 32nd is adopted in the conversion block having a side length exceeding 32. 3A and 3B schematically show some examples of such zeroing in FVC.
[0025]
　FIG. 3A shows three square conversion blocks B01, B02 and B03. The size of the conversion block B01 is 32 × 32, and zeroization is not applied to the conversion block B01. The size of the conversion block B02 is 64 × 64, and the conversion coefficients other than the conversion coefficient of 32 × 32 on the upper left of the conversion block B02 are zeroed. The size of the conversion block B03 is 128 × 128, and the conversion coefficients other than the conversion coefficient of 32 × 32 on the upper left of the conversion block B03 are zeroed. FIG. 3B shows nine non-square conversion blocks of various sizes, in addition to the 32x32 square conversion blocks. As can be seen from FIG. 3B, in FVC, up to 32 frequency components on the low frequency side are maintained and the remaining (high frequency side) frequency components are zeroed on the side having a length of 64 or more. To.
[0026]
　　(4) Explanation of Problems
　As described above, in HEVC, in order to avoid a decrease in coding efficiency due to signaling of the quantization matrix, the signaling of the 16 × 16 and 32 × 32 quantization matrices is 8 It is realized by signaling and upsampling of the x8 quantization matrix. However, there is a trade-off between the reduced coding efficiency due to the signaling of the quantization matrix and the effect of generating another quantization matrix from one quantization matrix (eg through upsampling) on ​​the device performance. There is a relationship. In particular, when block size and shape combinations are highly diversified as in FVC, if a quantization matrix corresponding to many of these combinations is simply generated from another quantization matrix, the processing required for the generation process is required. Costs (eg, occupying hardware resources, processing delays, or increased power consumption) can be significant and can degrade the performance of devices such as encoders or decoders.
[0027]
　　[1-3. Principles of Technology Related to the Disclosure] In
　order to solve or at least alleviate the problems of the existing technology described in the previous section, the technology according to the present disclosure is an improvement for efficiently generating or signaling a quantization matrix. Provide the mechanism.
[0028]
　　(1) Implementation Example on the Decoder Side
　FIG. 4 is an explanatory diagram for explaining an example of a basic implementation on the decoder side of the technique according to the present disclosure. With reference to FIG. 4, processing steps S11-S16 related to inverse quantization that can be performed in the decoder are shown.
[0029]
　Step S11: Decoding the coded stream, parsing some control data that influences the operation of inverse quantization and inverse orthogonal transformation, and generating quantization transformation coefficients. Control parameters include, for example, block division data that defines block division and scaling list (SL) data that defines a quantization matrix of one or more sizes.
[0030]
　-Step S12: A plurality of conversion blocks are set in a QTBT shape in the image based on the block division data. The block division data includes, for example, a set of parameters indicating a recursive QT division or BT division of each CTU. The conversion block set here can have various shapes and sizes.
[0031]
　Step S13: The scaling list data is decoded to generate a scaling list representing a quantization matrix of one or more sizes, and the generated scaling list is converted into each quantization matrix. For each size, multiple types of quantization matrices with different combinations of prediction types and color components can be generated. In the present specification, the quantization matrix (that is, the explicitly signaled quantization matrix) generated based on the scaling list data is referred to as a basic quantization matrix (basic QM). The basic quantization matrix generated in step S13 is stored in the QM memory M11.
[0032]
　Step S14: Each of the quantization matrices of one or more sizes other than the basic quantization matrix is ​​generated by referring to one of the generated quantization matrices. In the present specification, the quantization matrix additionally generated by referring to other quantization matrices having different sizes is referred to as an additional quantization matrix (additional QM). Further, the quantization matrix referred to when the additional quantization matrix is ​​generated is referred to as a reference quantization matrix (reference QM). Typically, the additional quantization matrix can be generated by resampling the reference quantization matrix. If the size of one side of the additional quantization matrix is ​​larger than the size of the corresponding side of the reference quantization matrix, upsampling is performed in the direction along that side. If the size of one side of the additional quantization matrix is ​​smaller than the size of the corresponding side of the reference quantization matrix, downsampling is performed in the direction along that side. The reference memory M12 stores at least a part of the reference quantization matrix referred to in these upsampling and downsampling.
[0033]
　As described above, in FVC, the high frequency component of the conversion block having a side length exceeding a certain threshold is forcibly zeroed. If the size of the quantization matrix is ​​equal to the block size to which these quantizations are applied, it should not be necessary to calculate the elements of the quantization matrix that correspond to the transformation coefficients that are always zero. Also, if a lot of memory resources are allocated for the calculation of those elements, it is a waste of resources. Therefore, in the technique according to the present disclosure, when generating an additional quantization matrix of a size to which zeroization is applied, a range that substantially contributes to the quantization of non-zero coefficients rather than the entire reference quantization matrix. A method that refers only to the submatrix is ​​adopted. This is because the operation for calculating the element at the position corresponding to the coefficient to be zeroed is skipped, and the reference memory M12 does not hold the element of the reference quantization matrix outside the range of the submatrix. It means good. The additional quantization matrix generated in step S14 is stored in the QM memory M11.
[0034]
　Step S15: The quantization conversion coefficient is inversely quantized for each of the plurality of conversion blocks set in step S12, and the conversion coefficient is restored. The quantization matrix used in step S15 has the same size as the target transformation block corresponding to the combination of the prediction type and the color component of the target transformation block among the plurality of quantization matrices stored in the QM memory M11. It is a quantization matrix. For conversion blocks to which zeroization is applied, some frequency components of the conversion factor have been zeroed out and the inverse quantization process may be skipped for those frequency components.
[0035]
　In certain embodiments, step S14 described above is included in step S15, where the quantization matrix required for each transformation block is generated so-called "on the fly" at the time of dequantization (if unquantized). May be done. In another embodiment, step S14 described above may be performed prior to processing across a plurality of transformation blocks, and all size candidate quantization matrices may be pre-stored in the QM memory M11.
[0036]
　Step S16: For each of the plurality of conversion blocks set in step S12, the conversion coefficients in the frequency domain are inversely orthogonally converted to restore the prediction error in the spatial domain. The predicted error restored here can be combined with the predicted image to reconstruct the image in further processing (not shown).
[0037]
　　(2) Various resamplings The
　process of generating an additional quantization matrix from the reference quantization matrix in step S14 of FIG. 4 is typically perpendicular to the downsampling or upsampling of the horizontal elements of the quantization matrix. A combination of downsampling or upsampling of directional elements. Upsampling involves interpolating the elements of a matrix by interpolation methods such as, for example, the Nearest Neighbor method, the bilinear method, or the bicubic method. Downsampling involves, for example, thinning out the elements of a matrix.
[0038]
　FIG. 5 is an explanatory diagram for explaining the generation of the quantization matrix for the transformation block to which zeroization is not applied. The reference quantization matrix B10 is shown in the upper left of FIG. Here, for the sake of simplicity of explanation, the shape of the reference quantization matrix B10 is assumed to be square, and the size of the reference quantization matrix B10 is N 1 × N 1 . Further, it is assumed that zeroization of the high frequency component is not applied to the conversion block of size N 1 × N 1 .
[0039]
　The quantization matrices B11, B12, B13 and B14 shown at the bottom of FIG. 5 are additional quantization matrices to be generated. The horizontal size of the additional quantization matrix B11 is smaller than N 1 , and the vertical size is also smaller than N 1 . The additional quantization matrix B11 corresponds to a conversion block to which zeroing of high frequency components is not applied. Therefore, the additional quantization matrix B11 is generated by downsampling the entire reference quantization matrix B10 in both the horizontal and vertical directions. The horizontal size of the additional quantization matrix B12 is smaller than N 1 and the vertical size is larger than N 1 . The additional quantization matrix B12 corresponds to a conversion block to which zeroing of high frequency components is not applied. Therefore, the additional quantization matrix B12 is generated by downsampling the entire reference quantization matrix B10 in the horizontal direction and upsampling in the vertical direction. The horizontal size of the additional quantization matrix B13 is larger than N 1 and the vertical size is smaller than N 1 . The additional quantization matrix B13 corresponds to a conversion block to which zeroing of high frequency components is not applied. Therefore, the additional quantization matrix B13 is generated by upsampling the entire reference quantization matrix B10 in the horizontal direction and downsampling in the vertical direction. The horizontal size of the additional quantization matrix B14 is larger than N 1 , and the vertical size is also N 1.Greater than The additional quantization matrix B14 corresponds to a conversion block to which zeroing of high frequency components is not applied. Therefore, the additional quantization matrix B14 is generated by upsampling the entire reference quantization matrix B10 in both the horizontal and vertical directions. In FIG. 5, horizontal resampling (downsampling or upsampling) and vertical resampling are drawn as separate operations, but these resamplings are integrally executed as one operation. You may.
[0040]
　6A-6C are explanatory diagrams for explaining the generation of the quantization matrix for the transformation block to which zeroization is applied. As mentioned above, the generation of the additional quantization matrix for the transformation block to which zeroization is applied is done by referring only to the submatrix of the reference quantization matrix. Similar to the example of FIG. 5, again, for the sake of simplicity, it is assumed that the square quantization matrix B10 of size N 1 × N 1 is selected as the reference quantization matrix.
[0041]
　The quantization matrices B21 and B22 shown at the bottom of FIG. 6A are additional quantization matrices to be generated. The horizontal size of the additional quantization matrix B21 is larger than the upper limit N TH of the size to which zeroing of the high frequency components is not applied , while the vertical size is smaller than N 1 . The additional quantization matrix B21 corresponds to a conversion block to which zeroing of high frequency components is applied. Therefore, the additional quantization matrix B21 is generated by upsampling the submatrix of the reference quantization matrix B10 in the horizontal direction and downsampling in the vertical direction. In the horizontal direction, the ratio of the size N PAR of the submatrix to the size N 1 of the reference quantization matrix B10 is equal to the ratio of the size N TH of the nonzero portion to the size N 2 of the additional quantization matrix B21 . The horizontal size of the additional quantization matrix B22 is also greater than the upper bound N TH of the size to which zeroing of the high frequency components is not applied , while the vertical size is greater than N 1 but smaller than N TH . The additional quantization matrix B22 corresponds to a conversion block to which zeroing of high frequency components is applied. Therefore, the additional quantization matrix B22 is generated by upsampling the submatrix of the reference quantization matrix B10 in both the horizontal and vertical directions.
[0042]
　The quantization matrices B31 and B32 shown at the bottom of FIG. 6B are additional quantization matrices to be generated. The horizontal size of the additional quantization matrix B31 is smaller than N 1 , while the vertical size is larger than the upper limit N TH of the size to which zeroing of the high frequency components is not applied . The additional quantization matrix B31 corresponds to a conversion block to which zeroing of high frequency components is applied. Therefore, the additional quantization matrix B31 is generated by downsampling the submatrix of the reference quantization matrix B10 in the horizontal direction and upsampling in the vertical direction. In the vertical direction, the ratio of the submatrix size N PAR to the size N 1 of the reference quantization matrix B10 is equal to the ratio of the size N TH of the nonzero portion to the size N 2 of the additional quantization matrix B31 . The horizontal size of the additional quantization matrix B32 is greater than N 1 but smaller than N TH , while the vertical size is greater than the upper bound N TH of the size to which zeroing of the high frequency components is not applied . The additional quantization matrix B32 corresponds to a conversion block to which zeroing of high frequency components is applied. Therefore, the additional quantization matrix B32 is generated by upsampling the submatrix of the reference quantization matrix B10 in both the horizontal and vertical directions.
[0043]
　The quantization matrix B41 shown at the bottom of FIG. 6C is an additional quantization matrix to be generated. The horizontal size of the additional quantization matrix B41 is larger than the upper limit N TH of the size to which zeroing of the high frequency components is not applied , and the vertical size is also larger than N TH . The additional quantization matrix B41 corresponds to a conversion block to which zeroing of high frequency components is applied. Therefore, the additional quantization matrix B41 is generated by upsampling the submatrix of the reference quantization matrix B10 in both the horizontal and vertical directions. In the horizontal direction, the ratio of the size N PAR_H of the submatrix to the size N 1 of the reference quantization matrix B10 is equal to the ratio of the size N TH of the nonzero portion to the size N 2_H of the additional quantization matrix B41 . In the vertical direction, the ratio of the size N PAR_V of the submatrix to the size N 1 of the reference quantization matrix B10 is equal to the ratio of the size N TH of the nonzero portion to the size N 2_V of the additional quantization matrix B41 .
[0044]
　Resampling matrix elements by referencing only the submatrix rather than the entire reference quantization matrix when additionally generating an unexplicitly signaled quantization matrix, as in the examples of FIGS. 6A-6C. It is possible to reduce the cost of calculation for and avoid wasting memory resources.
[0045]
　　(3) Implementation Example on Encoder Side
　FIG. 7 is an explanatory diagram for explaining an example of basic implementation on the encoder side of the technology according to the present disclosure. With reference to FIG. 7, processing steps S21-S27 related to quantization that can be performed in the encoder are shown.
[0046]
　-Step S21: Set a plurality of conversion blocks in a QTBT shape in the image. The block structure can be determined, for example, as a result of prior image analysis or search for the optimal block structure. Further, a set of parameters representing the block structure set here (for example, parameters indicating recursive QT division or BT division of each CTU) is generated as block division data.
[0047]
　Step S22: Set a user-defined (ie, not the default quantization matrix) basic quantization matrix of one or more sizes. For each size, a plurality of types of quantization matrices having different combinations of prediction types and color components can be set. One kind of quantization matrix may be the same as another kind of quantization matrix. The basic quantization matrix set in step S22 is stored in the QM memory M21.
[0048]
　Step S23: Each of the quantization matrices (additional quantization matrices) of one or more sizes other than the basic quantization matrix is ​​generated by referring to one of the basic quantization matrices. The additional quantization matrix can be generated by resampling the reference quantization matrix as described in connection with FIGS. 5-6C. The reference memory M22 stores at least a part of the reference quantization matrix that is referred to when generating the additional quantization matrix. Similar to the decoder implementation described above, in the encoder implementation, when generating an additional quantization matrix of a size to which quantization is applied, the non-zero coefficients of the reference quantization matrix, rather than the entire, Only sub-matrixes in the range that substantially contribute to quantization can be referenced.
[0049]
　Step S24: For each of the plurality of conversion blocks set in step S21, the prediction error in the spatial domain is orthogonally converted to generate the conversion coefficient in the frequency domain.
[0050]
　Step S25: The conversion coefficient is quantized for each of the plurality of conversion blocks set in step S21 to generate a quantized conversion coefficient. The quantization matrix used in step S25 has the same size as the target transformation block, which corresponds to the combination of the prediction type and the color component of the target transformation block among the plurality of quantization matrices stored in the QM memory M21. It is a quantization matrix. For conversion blocks to which zeroization is applied, some frequency components of the conversion factor have been zeroed out and the quantization process may be skipped for those frequency components.
[0051]
　In certain embodiments, step S23 described above is included in step S25, where the quantization matrix required for each transformation block is generated so-called "on the fly" at the time of the quantization process (if ungenerated). You may. In another embodiment, the above-mentioned step S23 may be executed prior to the processing over a plurality of conversion blocks, and the quantization matrix of all the size candidates may be stored in the QM memory M21 in advance.
[0052]
　As will be described later, the encoder usually includes a local decoder, and dequantization is performed in the local decoder. Although FIG. 7 does not show dequantization in the local decoder, the same quantization matrix used in step S25 may be used for this dequantization.
[0053]
　Step S26: The basic quantization matrix of one or more sizes set in step S22 is converted into a scaling list, and each scaling list is encoded to generate scaling list data.
[0054]
　Step S27: A control parameter including block division data and scaling list data and a quantization conversion coefficient are encoded to generate a coded stream.
[0055]
　　[1-4. Control of zeroing] In the
　previous section, we mainly explained an example in which zeroing of high-frequency components is applied to a conversion block whose at least one side size is larger than the upper limit N TH . According to reference REF3, this upper bound N TH in FVC is equal to 32 as a static definition in the specification. In this case, it is not necessary to encode the control parameters that indicate which transformation block the zeroing applies to. However, in order to achieve more flexible zeroization control, for example the following control parameters may be additionally encoded:
[0056]
　-Zeroization flag: If this flag is True, zeroing of the high frequency component is applied to the conversion block associated with this flag. If it is False, it is associated with this flag. No zeroing of high frequency components is applied to the conversion blocks to be generated.
[0057]
　-Zeroization size information: Indicates which size conversion block the high frequency component zeroization is applied to. The zeroization size information may include, for example, a boundary value (threshold value) corresponding to at least one of an upper limit and a lower limit of the size to which zeroization is not applied (or is applied). Instead, the zeroing size information may include an index that identifies the size to which zeroing is not applied (or is applied). The zeroization size information may be encoded with or without the zeroization flag, or may be encoded only if the zeroization flag is true.
[0058]
　The control parameters for zeroing control described above may be encoded, for example, for each sequence, picture, slice, tile, CTU or conversion block. By dynamically controlling the on / off of zeroization or the target size of zeroization in this way, flexible reproduction of an image that expresses even minute high-frequency components according to user needs or system requirements or restrictions. Can be made possible.
[0059]
　　[1-5. Controlling the size of the basic quantization matrix]
　As mentioned above, in HEVC, the 16x16 and 32x32 unique quantization matrices are not directly signaled, and the smaller quantization matrix is ​​upsampled. It is generated by each. Specifically, in HEVC, it is defined in advance as a rule in the specification that a quantization matrix having a size of up to 8 × 8 is directly signaled. On the other hand, when the elements of the most efficient M × N quantization matrix (M and N are powers of 2) are calculated based on a model of human visual sensitivity characteristics (see, for example, reference REF4 below), they are smaller. It is known that the quantization matrix becomes a subset of the larger quantization matrix. Therefore, in order to obtain the optimum set of quantization matrices, instead of upsampling the smaller basic quantization matrix to generate additional quantization matrices, downsampling the larger basic quantization matrix is ​​performed. Techniques for generating additional quantization matrices are also useful.
　・ REF4: Long-Wen Chang, Ching-Yang Wang and Shiuh-Ming Lee, “Designing JPEG quantization tables based on human visual system”, International Conference on Image Processing, ICIP 99, 24-28 October, 1999
[0060]
　Therefore, in some embodiments described later, size instruction information indicating which size of the quantization matrix is ​​generated as the basic quantization matrix from the scaling list data is encoded. By variably indicating the size of the basic quantization matrix with the size instruction information, it is possible to flexibly use a wider variety of quantization matrices. As a matter of course, the technique according to the present disclosure can be applied to a method in which the size of the quantization matrix to be directly signaled is defined in advance in the specifications regardless of the size.
[0061]
　The technical principle according to the present disclosure described so far may be implemented in an image processing device (that is, an encoder) that encodes a series of images constituting an image, or decodes an encoded bit stream. It may be implemented in an image processing device (that is, a decoder) that reconstructs an image. One image processing device may have the functionality of both an encoder and a decoder. The image processing device may be any kind of device such as an image pickup device, a video recording device, a playback device, a display device, a communication device, or an information processing device. From the next section, two specific embodiments related to such an image processing apparatus will be described in more detail.
[0062]
　<2. First Embodiment>
　　[2-1. Encoder Configuration]
　　(1) Overall Configuration
　FIG. 8 is a block diagram showing an example of the configuration of an image processing device 10a having encoder functionality according to the first embodiment. Referring to FIG. 8, the image processing apparatus 10a includes a coding control unit 11, a sorting buffer 12, a subtraction unit 13, an orthogonal conversion unit 14, a quantization unit 15a, a reversible coding unit 16, a storage buffer 17, and a rate control unit. 18. Inverse quantization unit 21, inverse orthogonal conversion unit 22, addition unit 23, in-loop filter 24, frame memory 25, switch 26, intra-prediction unit 30, inter-prediction unit 35, mode setting unit 40, and QM storage unit 115a. Be prepared.
[0063]
　The coding control unit 11 controls the overall functionality of the image processing device 10a as an encoder, as described in detail below. In the present embodiment, the coding control unit 11 includes a block setting unit 111 and a basic QM setting unit 113. The block setting unit 111 is a module that executes the block setting process S21 described with reference to FIG. 7. The basic QM setting unit 113 is a module that executes the basic QM setting process S22 described with reference to FIG. 7. These modules will be described further later.
[0064]
　The sorting buffer 12 sorts a series of images constituting the video to be encoded according to a predetermined GOP (Group of Pictures) structure. The sorting buffer 12 outputs the sorted images to the subtraction unit 13, the intra prediction unit 30, and the inter prediction unit 35.
[0065]
　The subtraction unit 13 calculates a prediction error, which is the difference between the image (original image) input from the sorting buffer 12 and the prediction image, and outputs the calculated prediction error to the orthogonal conversion unit 14.
[0066]
　The orthogonal conversion unit 14 executes orthogonal conversion for each of one or more conversion blocks set in the image to be encoded. The orthogonal transform here may be, for example, a discrete cosine transform (DCT) or a discrete sine transform (DST). More specifically, the orthogonal conversion unit 14 orthogonally converts the signal sample in the spatial domain representing the prediction error input from the subtraction unit 13 for each conversion block to generate a conversion coefficient in the frequency domain. Further, the orthogonal conversion unit 14 zeros the high frequency component of the conversion block having a certain size according to the control by the coding control unit 11. For example, in a conversion block having a side length of more than 32, the frequency component on the high frequency side after the 32nd may be zeroed. Then, the orthogonal conversion unit 14 outputs the generated conversion coefficient to the quantization unit 15a.
[0067]
　The quantization unit 15a is supplied with a conversion coefficient input from the orthogonal conversion unit 14 and a rate control signal from the rate control unit 18 described later. The quantization unit 15a quantizes the conversion coefficient for each of the one or more conversion blocks in the image to be encoded using a quantization matrix of the same size as the conversion block, and the quantization conversion coefficient (hereinafter, quantum). Quantization data) is generated. The quantization unit 15a skips the quantization of the forcibly zeroed frequency component included in the conversion coefficient according to the control by the coding control unit 11. Then, the quantization unit 15a outputs the generated quantization data to the reversible coding unit 16 and the inverse quantization unit 21. Further, the quantization unit 15a changes the bit rate of the quantization data by switching the quantization step based on the rate control signal from the rate control unit 18. In the present embodiment, the quantization unit 15a includes a QM generation unit 117a. The QM generation unit 117a is a module that executes the QM generation process S23 described with reference to FIG. 7. The QM generation unit 117a also includes a reference memory M22 (not shown in FIG. 8). This module will be described further later.
[0068]
　The reversible coding unit 16 encodes the quantization data input from the quantization unit 15a to generate a coded stream. Further, the reversible coding unit 16 encodes various control parameters referred to by the decoder and inserts the coding parameters into the coding stream. The control parameters encoded here include, for example, the block division data and the scaling list (or scaling list data) described above. The reversible coding unit 16 outputs the generated coded stream to the storage buffer 17. In the present embodiment, the reversible coding unit 16 includes an SL coding unit 119. The SL coding unit 119 is a module that executes the QM conversion / SL data generation process S26 described with reference to FIG. 7. This module will be described further later.
[0069]
　The storage buffer 17 temporarily stores the coded stream input from the reversible coding unit 16 using a storage medium. Then, the storage buffer 17 outputs the stored coded stream to a transmission unit (for example, a communication interface or a connection interface with a peripheral device) (not shown) at a rate corresponding to the band of the transmission line.
[0070]
　The rate control unit 18 monitors the free space of the storage buffer 17. Then, the rate control unit 18 generates a rate control signal according to the free capacity of the storage buffer 17, and outputs the generated rate control signal to the quantization unit 15a. For example, the rate control unit 18 generates a rate control signal for lowering the bit rate of the quantized data when the free space of the storage buffer 17 is small. Further, the rate control unit 18 generates a rate control signal for increasing the bit rate of the quantized data when the free capacity of the storage buffer 17 is sufficiently large.
[0071]
　The inverse quantization unit 21, the inverse orthogonal conversion unit 22, and the addition unit 23 form a local decoder. The local decoder is responsible for decoding the encoded data and reconstructing the image.
[0072]
　The inverse quantization unit 21 dequantizes the quantization conversion coefficient for each conversion block using the same quantization matrix used by the quantization unit 15a, and restores the conversion coefficient. The inverse quantization unit 21 skips the inverse quantization of the forcibly zeroed frequency component included in the quantization conversion coefficient. Then, the inverse quantization unit 21 outputs the restored conversion coefficient to the inverse orthogonal conversion unit 22.
[0073]
　The inverse orthogonal conversion unit 22 executes the inverse orthogonal conversion for each conversion block. More specifically, the inverse orthogonal conversion unit 22 performs inverse orthogonal conversion of the conversion coefficient of the frequency domain input from the inverse quantization unit 21 for each conversion block, and restores the prediction error of the signal sample format in the spatial domain. To do. Then, the inverse orthogonal conversion unit 22 outputs the restored prediction error to the addition unit 23.
[0074]
　The addition unit 23 reconstructs the decoded image by adding the restored prediction error input from the inverse orthogonal conversion unit 22 and the prediction image input from the intra prediction unit 30 or the inter prediction unit 35. Then, the addition unit 23 outputs the reconstructed decoded image to the in-loop filter 24 and the frame memory 25.
[0075]
　The in-loop filter 24 is composed of a series of filters applied to the decoded image for the purpose of improving the image quality. The in-loop filter 24 may include, for example, one or more of the bilateral filter, the deblocking filter, the adaptive offset filter, and the adaptive loop filter described in reference REF3. The in-loop filter 24 outputs the decoded image filtered through a series of filters to the frame memory 25.
[0076]
　The frame memory 25 stores the decoded image before filtering input from the addition unit 23 and the decoded image after filtering input from the in-loop filter 24.
[0077]
　The switch 26 reads the decoded image before filtering used for the intra prediction from the frame memory 25, and supplies the read decoded image as a reference image to the intra prediction unit 30. Further, the switch 26 reads the filtered decoded image used for the inter prediction from the frame memory 25, and supplies the read decoded image as a reference image to the inter prediction unit 35.
[0078]
　The intra prediction unit 30 executes the intra prediction process based on the original image and the decoded image. For example, the intra prediction unit 30 evaluates the cost based on the prediction error and the amount of code generated for each of the prediction mode candidates included in the search range. Next, the intra prediction unit 30 selects the prediction mode that minimizes the cost as the optimum prediction mode. In addition, the intra prediction unit 30 generates a prediction image according to the selected optimum prediction mode. Then, the intra prediction unit 30 outputs the prediction image and the corresponding cost to the mode setting unit 40 together with some control parameters including the prediction mode information.
[0079]
　The inter-prediction unit 35 executes inter-prediction processing (motion compensation) based on the original image and the decoded image. For example, the inter-prediction unit 35 evaluates the cost based on the prediction error and the amount of code generated for each of the prediction mode candidates included in a certain search range. Next, the inter-prediction unit 35 selects the prediction mode that minimizes the cost as the optimum prediction mode. In addition, the inter-prediction unit 35 generates a prediction image according to the selected optimum prediction mode. Then, the inter-prediction unit 35 outputs the prediction image and the corresponding cost to the mode setting unit 40 together with some control parameters including the prediction mode information.
[0080]
　The mode setting unit 40 sets the prediction type for each block based on the comparison of the costs input from the intra prediction unit 30 and the inter prediction unit 35. The mode setting unit 40 outputs the prediction image generated by the intra prediction unit 30 to the subtraction unit 13 and the addition unit 23 for the block for which the intra prediction is set as the prediction type. Further, the mode setting unit 40 outputs the prediction image generated by the inter-prediction unit 35 to the subtraction unit 13 and the addition unit 23 for the block for which the inter-prediction is set as the prediction type. Further, the mode setting unit 40 outputs the control parameter to be encoded to the reversible coding unit 16.
[0081]
　　(2) Functionality Related to Quantization Matrix Generation
　Among the various components shown in FIG. 8, the block setting unit 111, the basic QM setting unit 113, the QM storage unit 115a, the QM generation unit 117a, and the SL coding unit 119 is mainly related to the generation of the quantization matrix performed in the encoder.
[0082]
　The block setting unit 111 sets a plurality of conversion blocks for each of the series of images by dividing each image into a plurality of conversion blocks through the QTBT block division. Further, the block setting unit 111 generates block division data that defines the block structure of the set conversion block, and outputs the generated block division data to the reversible coding unit 16. The size of the conversion block set by the block setting unit 111 may be in the range of, for example, 2 × 2 to 128 × 128. The shape of the conversion block may be square or non-square. Some examples of the shape and size of the conversion block are shown in FIG.
[0083]
　The basic QM setting unit 113 sets a basic quantization matrix of one or more sizes as a quantization matrix used by the image processing apparatus 10a. The basic quantization matrix typically has at least one element that differs in value from the default quantization matrix defined by the FVC specification. The values ​​of the elements of the basic quantization matrix can be determined, for example, as a result of prior image analysis or parameter tuning. The basic QM setting unit 113 may set a plurality of types of quantization matrices having different combinations of prediction types and color components for each size. One kind of quantization matrix may be the same as another kind of quantization matrix. As an example, the basic quantization matrix contains only a square quantization matrix. As another example, the basic quantization matrix includes both a square quantization matrix and a non-square quantization matrix. In the syntax example described later, the size of the quantization matrix is ​​identified by the size ID, and the type of the quantization matrix is ​​identified by the matrix ID.
[0084]
　The QM storage unit 115a is a storage module that stores various types of quantization matrices of various sizes used by the image processing device 10a. The quantization matrix stored by the QM storage unit 115a includes a basic quantization matrix set by the basic QM setting unit 113 and an additional quantization matrix additionally generated by the QM generation unit 117a described later. In the present embodiment, the basic quantization matrix is ​​set prior to orthogonal transformation and quantization over a plurality of transformation blocks, and is stored by the QM storage unit 115a through these processes. On the other hand, the additional quantization matrix is ​​generated as needed when the conversion coefficients of the individual conversion blocks are quantized, and is stored in the QM storage unit 115a. The QM storage unit 115a may manage matrix management information which is internal control information indicating which size of quantization matrix already exists. The matrix management information is, for example, a flag indicating whether or not a quantization matrix identified by two size IDs corresponding to horizontal and vertical sizes exists (for example, whether or not it has been generated). It consists of a set.
[0085]
　When the conversion coefficient of each conversion block is quantized in the quantization unit 15a, the QM generation unit 117a determines whether a quantization matrix having the same size as the conversion block has already been generated in the QM storage unit 115a. Judgment is made by referring to the above-mentioned matrix management information provided by. Then, when the QM generation unit 117a has already generated a quantization matrix of the relevant size, the QM generation unit 117a reads the generated quantization matrix from the QM storage unit 115a. On the other hand, when the quantization matrix having the same size as the conversion block has not been generated, the QM generation unit 117a selects and selects one of the generated basic quantization matrices as the reference quantization matrix. An additional quantization matrix is ​​generated by resampling the reference quantization matrix. The QM generation unit 117a includes a memory that temporarily holds the reference quantization matrix or its submatrix to be resampled.
[0086]
　The QM generation unit 117a generates, for example, a quantization matrix for a conversion block when the high-frequency component zeroization is not applied to a conversion block of a certain target size by referring to the entire reference quantization matrix. To do. On the other hand, the QM generator 117a refers to the quantization matrix for the conversion block when the high-frequency component zeroization is applied to the conversion block of another target size, and refers only to the submatrix of the reference quantization matrix. It is generated by doing. In the latter case, the ratio of the size of the referenced (ie, temporarily held by the memory of the QM generator 117a) submatrix to the size of the reference quantization matrix is ​​non-zero to the size of the additional quantization matrix generated. Equal to the percentage of the zero part. The quantization unit 15a quantizes the conversion coefficient of each conversion block by using one of the various quantization matrices generated while conserving resources in this way.
[0087]
　In one example, the QM generator 117a may determine whether zeroization is applied to each conversion block according to a specification rule that predefines which size conversion block the zeroization is applied to. .. In another example, the QM generation unit 117a may determine that zeroization is applied to one conversion block and no zeroing is applied to another conversion block according to the control by the coding control unit 11. In the latter example, of the zeroization flags that indicate whether zeroization is applied to each conversion block and the zeroization size information that indicates which size conversion block the zeroization is applied to, as described above. One or both can be encoded as control parameters and inserted into the encoded stream.
[0088]
　The SL coding unit 119 encodes the scaling list representing the above-mentioned basic quantization matrix set by the basic QM setting unit 113 to generate scaling list data. The scaling list data is inserted into the coded stream generated by the reversible coding unit 16.
[0089]
　In one example, the SL coding unit 119 includes size-indicating information in the scaling list data that indicates which size of quantization matrix is ​​explicitly signaled as the basic quantization matrix via the scaling list data. When it is variable how many sizes of quantization matrix are signaled through the scaling list data, the SL coding unit 119 further scales the size number information indicating the number of sizes to be signaled. It may be included in the list data. In this case, when two or more sizes are signaled, a bitstream constraint may be imposed that they are of different sizes (identified by different size IDs). Such a bitstream constraint can prevent redundant coding of size instruction information in the encoder, reduce the code amount overhead, and avoid wasting resources.
[0090]
　In another example, the size of the basic quantization matrix explicitly signaled through the scaling list data is predefined as a specification rule. In this case, the SL coding unit 119 does not include the above-mentioned size instruction information and size number information in the scaling list data.
[0091]
　　[2-2. Examples of syntax and semantics]
　Table 1 below shows the syntax of the HEVC scaling list data described in reference REF2.
[0092]
[table 1]

[0093]
　The HEVC scaling list data is one or more of the following control parameters for each combination of the four sizes identified by the size ID (sizeId) and the six types identified by the matrix ID (matrixId). Includes:
　-scaling_list_pred_mode_flag [sizeId] [matrixId]
　-scaling_list_pred_matrix_id_delta [sizeId] [matrixId]
　-scaling_list_dc_coef_minus8 [sizeId-2] [matrixId]
　-scaling_list_delta_coef
scaling_list_delta_coef scaling_list_pred_mode It is a control flag. The type of quantization matrix for which this control flag is false is simply encoded by referring to other types of quantization matrices pointed to by scaling_list_pred_matrix_id_delta [sizeId] [matrixId]. The quantization matrix for which the control flag is true is differentially encoded using scaling_list_dc_coef_minus8 [sizeId-2] [matrixId] and a plurality of scaling_list_delta_coef. The number of scaling_list_delta_coef is indicated by the intermediate variable coefNum.
[0094]
　Table 2 below shows an example of the syntax of scaling list data that can be modified in this embodiment. The example in Table 2 is shown in [1-5. Controlling the size of the basic quantization matrix] includes size instruction information indicating which size of the quantization matrix is ​​generated as the basic quantization matrix.
[0095]
[Table 2]

[0096]
　Specifically, the parameter size_id_minusX in the second row of Table 2 corresponds to the size instruction information. The value of the parameter size_id_minusX is derived by subtracting the predefined offset value X from the actual size ID.
[0097]
　Table 3 below shows an example of the size ID definition that can be modified from the definition in HEVC.
[0098]
[Table 3]

[0099]
　In the example of Table 3, indexes (0 to 6) of 0 to 6 are assigned as size IDs to the candidate values ​​(2 to 128) of the size of one side of the square quantization matrix in ascending order. The relationship between the one-sided size candidate value N and the size ID value sizeId is as follows in this example:

　　　　N = 1 << (sizeId + 1)

where the size candidate value and the size ID are The relationship is not limited to the above equation.
[0100]
　The size of the non-square quantization matrix can be identified by specifying the size ID for each of the horizontal and vertical directions. In the syntax example of Table 2, the size indication information contains only one parameter, size_id_minusX. This means that only the square quantization matrix is ​​explicitly signaled as the basic quantization matrix. Also, the quantization matrix of the size corresponding to the size ID less than the offset value X is not signaled. Not limited to this example, the size instruction information may include two parameters for identifying two size IDs for identifying a non-square basic quantization matrix. Further, the offset may be omitted, and the size ID that is not offset may be encoded as it is.
[0101]
　Table 4 below shows an example of a matrix ID definition that can be modified from the HEVC definition.
[0102]
[Table 4]

[0103]
　In the definition in HEVC described in relation to FIG. 1, the definition of matrix ID is exceptional when the size ID is equal to 3 (corresponding to size 32 × 32), but in the example of Table 4, the value of the size ID Regardless, the definition of matrix ID is common. This affects the difference between the matrix ID loop in the third row of the syntax in Table 1 and the matrix ID loop in the fourth row of the syntax in Table 2. The maximum number of difference-encoded element values ​​pointed to by the intermediate variable coefNum is 64 in both the syntaxes of Table 1 and Table 2, but in order to enable more flexible design of the quantization matrix. , This maximum number may be changed or set variably.
[0104]
　　[2-3. Selection of reference quantization matrix]
　Which quantization matrix should be referred to when generating an additional quantization matrix may be determined according to any rule. As a simple example, the largest of the available square quantization matrices may be selected as the reference quantization matrix. In this example, the size ID of the largest quantization matrix that is explicitly signaled is "maxSignaledSizeId", and whether or not there is a quantization matrix identified by a combination of a certain size ID and matrix ID is matrix management information. It shall be indicated by "QMAvailFlag [sizeId] [matrixId]" (there is a quantization matrix identified if QMAvailFlag [sizeId] [matrixId] is true, otherwise it does not exist). Then, the reference size ID “refSizeId” indicating the size of the reference quantization matrix can be determined by the following pseudo code.
[0105]
[Table 5]

[0106]
　As another example, the matrix that has the smallest size difference from the additional quantization matrix to be generated among the available quantization matrices may be selected as the reference quantization matrix. In addition, the reference quantization matrix information indicating which quantization matrix should be referred to may be additionally encoded.
[0107]
　　[2-4. Resampling method]
　Next, an example of a specific resampling method for additionally generating a quantization matrix will be described. Here, for the sake of simplicity of explanation, it is assumed that zeroing of the high frequency component is applied to the target conversion block on condition that the size of at least one side of the target conversion block is larger than the upper limit N TH . It should be noted that the following description can be applied to cases where the conditions for applying zeroization are different, with slight modifications that will be apparent to those skilled in the art.
[0108]
　First, it is assumed that the first quantization matrix of the first size is selected as the reference quantization matrix for generating the second quantization matrix of the second size. Further, the first size is W 1 × H 1 , and the second size is W 2 × H 2 . A flag “zoFlag” can be set to indicate whether high frequency component zeroing is applied to the second size conversion block according to one of the following two equations:

　　　　zoFlag = max (W 2 , H 2). )> N TH
　　　　zoFlag = W 2 > N TH　 || H 2 > N TH

According to these equations, when zeroing of the high frequency component is applied to the conversion block of the second size (W 2 × H 2 ). The flag zoFlag is set to 1 otherwise, the flag zoFlag is set to 0 otherwise.
[0109]
　Among the second quantization matrix, actually the horizontal range of the element to be generated through the resampled R Width2 , the vertical range R HEIGHT2 When these ranges may be given by the following equation:

　　　　W R2 = min (W 2 , N TH )
　　　　H R2 = min (H 2 , N TH )
　　　　R Width2 = [0, W R2 -1]
　　　　R HEIGHT2 = [0, H R2 -1]

where, W R2 and H R2 is , Represents the number of elements included in each of the horizontal and vertical ranges. Second size (W 2 x H 2)Horizontal ratio r of the non-zero part (portion where zeroing is not applied) for) Width2 and vertical ratio r HEIGHT2 may be given by the following 　　　　equation: r Width2 = W R2 / W 　　　　2 r HEIGHT2 = H R2 / H 2 Using these ratios r WIDTH2 and r HEIGHT2 , the horizontal range R WIDTH1 and the vertical range of the part referred to when generating the non-zero part of the second quantization matrix of the first quantization matrix. range R height1 a, can be derived as 　　　　follows: W R1 = W 1 · r Width2 　　　　H R1 = H 1 · r

HEIGHT2
　　　　R WIDTH1 = [0, W R1 -1]
　　　　R height1 = [0, H R1 -1]

where, W R1 and H R1 represents the number of elements in each range of the horizontal and vertical directions ..
[0110]
　As can be understood from the above description, when zeroization is applied to the target transform block, by referring only to the submatrix of the first quantization matrix of first size W 1 × H 1 . It is possible to generate a second quantization matrix for quantizing the conversion coefficient of the target conversion block (or dequantizing the quantization conversion coefficient). On the other hand, if the zeroing is not applied to the target conversion block, W R2 = W 2 and H R2 = H 2 , i.e. r Width2 = r HEIGHT2 a = 1, W R1 = W 1 and H R1 = H 1 and Become. In this case, the entire first quantization matrix is ​​referred to in order to generate the second quantization matrix. Therefore, the QM generation unit 117a has the reference ranges R WIDTH1 and R HEIGHT1 among the elements of the reference quantization matrix when the zeroization of the high frequency component is applied to the target conversion block. Only the elements contained therein are read from the QM storage unit 115a for resampling and buffered. When zeroization is not applied to the target conversion block, the QM generation unit 117a reads all the elements of the reference quantization matrix from the QM storage unit 115a and buffers them.
[0111]
　If the ratio of the first size to the second size is s WIDTH and s HEIGHT in the horizontal and vertical directions, respectively , the size ratios s WIDTH and s HEIGHT are given as follows:

　　　　s WIDTH = W 1 / W 2
　　　　s HEIGHT = H 1 / H 2 As

an example, the elements Q ADD [j] [i] of the second quantization matrix Q ADD are obtained by upsampling the first quantization matrix Q REF by the nearest neighbor method . The derivation process can be expressed as follows using these size ratios s WIDTH and s HEIGHT . Note that j and i are indexes corresponding to rows and columns, respectively. It is assumed that the elements of the second quantization matrix are all initialized to 0 prior to resampling:

　　j'= Floor (j · s WIDTH )
　　i'= Floor (i · s HEIGHT )
　　Q ADD [j] [i] = Q REF [ j'] [i' ] (for i ∈ R WIDTH2, j ∈ R HEIGHT2) )

Here, Floor (x) is a function that returns the largest integer less than or equal to the argument x.
[0112]
　The process of deriving the elements Q ADD [j] [i] of the second quantization matrix Q ADD by downsampling the first quantization matrix Q REF can be expressed as follows. It is assumed that all the elements of the second quantization matrix are initialized to 0 prior to resampling: 　　Q ADD [j] [i] = Q REF [j · s WIDTH ] [i · s HEIGHT ] (For i ∈ R WIDTH2, j ∈ R HEIGHT2 )

[0113]
　The resampling method described in this section is only an example. For example, it is possible to perform upsampling in one direction of the matrix and downsampling in the other direction. It is also of course possible not to perform resampling in one direction of the matrix. Further, in upsampling, other interpolation methods such as a bilinear method or a bicubic method may be used instead of the nearest neighbor method shown above.
[0114]
　　[2-5. Flow of quantization related process at the time of encoding]
　　(1) Overall Flow
　Figure 9 is a flowchart illustrating an example of the quantization-related processing flow executed in the image processing apparatus 10a shown in FIG. Note that, for the sake of simplicity of explanation, the description of the processing steps not related to quantization among the various processing steps included in the coding process is omitted here.
[0115]
　First, the basic QM setting unit 113 sets one or more basic quantization matrices including the first quantization matrix of the first size (step S111). These basic quantization matrices are stored in the QM storage unit 115a.
[0116]
　Subsequent processes in steps S113 to S116 are repeated for each of the plurality of conversion blocks set in the image through the QTBT block division by the block setting unit 111 (step S112). The conversion block that is the target of each repetition is referred to as the target conversion block here.
[0117]
　First, the QM generation unit 117a determines whether or not a quantization matrix corresponding to the block size of the target conversion block exists by referring to, for example, the matrix management information provided by the QM storage unit 115a (step S113). If the quantization matrix does not exist, the QM generation unit 117a executes a quantization matrix generation process described later to generate an additional quantization matrix from the reference quantization matrix (step S114). The additional quantization matrix generated here is stored by the QM storage unit 115a. On the other hand, when the required quantization matrix already exists, the QM generation unit 117a reads out the quantization matrix corresponding to the block size of the target conversion block from the QM storage unit 115a (step S115). Then, the quantization unit 15a quantizes the conversion coefficient of the target conversion block using the quantization matrix additionally generated or read (step S116).
[0118]
　The SL coding unit 119 converts each of the one or more basic quantization matrices set in step S111 into a scaling list by making them one-dimensional (step S117). Next, the SL coding unit 119 executes the scaling list coding process described later to generate the scaling list data (step S118).
[0119]
　Although not shown in FIG. 9, the quantization conversion coefficient for each conversion block generated in step S116 is encoded by the reversible coding unit 16 and becomes a part of the coding stream together with the scaling list data. Scaling list data may be updated in any unit such as sequences, pictures, slices or tiles.
[0120]
　　(2) Quantization Matrix Generation Process
　FIG. 10 is a flowchart showing an example of the flow of the quantization matrix generation process that can be executed in step S114 of FIG.
[0121]
　First, the QM generation unit 117a selects a reference quantization matrix to be referred to when generating a quantization matrix for the target transformation block (step S121). As an example, the reference quantization matrix follows the rules predefined in the specification (eg, the quantization matrix with the largest size available or the size closest to the size of the target transformation block, etc.). It may be selected. As another example, the reference quantization matrix may be dynamically selected.
[0122]
　Next, the QM generation unit 117a determines whether or not zeroization of the high frequency component is applied to the target conversion block (step S122). As an example, the QM generator 117a applies zeroing of the high frequency component to the target conversion block according to a rule defined in the specification (for example, whether the length of at least one side of the target conversion block exceeds a certain threshold value). It may be determined whether or not it is done. As another example, it may be dynamically switched whether or not zeroization of the high frequency component is applied to the target conversion block.
[0123]
　When zeroization is not applied to the target transformation block, the QM generation unit 117a reads the entire reference quantization matrix selected in step S121 from the QM storage unit 115a and buffers it in the internal memory (step S123). Next, the QM generation unit 117a resamples the read reference quantization matrix to generate an additional quantization matrix (step S124).
[0124]
　On the other hand, when zeroization is applied to the target conversion block, the QM generation unit 117a calculates the ratio of the non-zero portion to the size of the target conversion block (for example, the ratios r WIDTH2 and r HEIGHT2 described above ) (step S125). .. Next, the QM generation unit 117a reads a submatrix of the selected reference quantization matrix from the QM storage unit 115a according to the calculated ratio, and buffers it in the internal memory (step S126). Next, the QM generation unit 117a resamples the read submatrix to generate a non-zero portion of the additional quantization matrix (step S127).
[0125]
　Then, the QM generation unit 117a stores the generated additional quantization matrix in the QM storage unit 115a (step S168).
[0126]
　　(3) Scaling list coding process
　FIG. 11 is a flowchart showing an example of a flow of the scaling list coding process that can be executed in step S118 of FIG.
[0127]
　First, the SL coding unit 119 encodes a size ID that identifies the size of the basic quantization matrix to generate size instruction information (step S131). When the basic quantization matrix is ​​always a square matrix, only one size ID as illustrated in Table 3 is encoded. If the basic quantization matrix also includes a non-square matrix, two size IDs corresponding to the two directions may be encoded. The size indication information may be generated by subtracting a predefined offset from the size ID.
[0128]
　Subsequent processing in steps S133 to S136 is repeated for each combination of the prediction type and the color component, that is, the type of the quantization matrix identified by the matrix ID (step S132). The matrix ID that is the target of each repetition is referred to as the target matrix ID here.
[0129]
　The SL coding unit 119 first determines whether or not to explicitly encode the values ​​of a series of elements of the scaling list associated with the target matrix ID (step S133). That is, the SL coding unit 119 determines the coding method of the scaling list. For example, if the scaling list associated with the target matrix ID is the same as the scaling list associated with another matrix ID (of the same size ID), the SL encoding unit 119 instead of encoding the element values. You may choose to simply encode only the reference scaling information.
[0130]
　When simple coding is selected, the SL coding unit 119 determines the reference scaling list (step S134) and encodes the reference scaling list information pointing to the determined reference scaling list (step S135).
[0131]
　On the other hand, when it is selected to encode the values ​​of a series of elements, the SL coding unit 119 sets the values ​​of the series of elements of the scaling list derived in step S117 of FIG. 9 to, for example, DPCM (Differential Pulse Code Modulation). ) Scaling list data is generated by encoding in the method (step S136).
[0132]
　The flowchart shown in FIG. 11 shows an example in which the size ID of the basic quantization matrix is ​​explicitly encoded. In other examples, the coding of the size of the basic quantization matrix may be omitted. In yet another example, the number of sizes targeted for signaling may be further encoded as size number information.
[0133]
　　[2-6. Configuration of Decoder]
　　(1) Overall Configuration
　FIG. 12 is a block diagram showing an example of the configuration of the image processing device 60a having the functionality of the decoder according to the first embodiment. Referring to FIG. 12, the image processing apparatus 60a includes a decoding control unit 61, a reversible decoding unit 62, an inverse quantization unit 63a, an inverse orthogonal conversion unit 64, an addition unit 65, an in-loop filter 66, a sorting buffer 67, and a frame memory. It includes 68, selectors 70 and 71, an intra prediction unit 80, an inter prediction unit 85, and a QM storage unit 165a.
[0134]
　The decoding control unit 61 controls the overall functionality of the image processing device 60a as a decoder, as described in detail below. In the present embodiment, the decoding control unit 61 includes a block setting unit 161. The block setting unit 161 is a module that executes the block setting process S12 described with reference to FIG. This module will be described further later.
[0135]
　The reversible decoding unit 62 parses various control parameters from a coded stream input from a transmission unit (for example, a communication interface or a connection interface with a peripheral device) (not shown). The control parameters parsed by the reversible decoding unit 62 include, for example, the block division data and the scaling list data described above. The block division data is output to the decoding control unit 61. In the present embodiment, the reversible decoding unit 62 includes the SL decoding unit 163. The SL decoding unit 163 is a module that executes the scaling list decoding process S13 described with reference to FIG. This module will be described further later. The reversible decoding unit 62 further decodes the coded stream to generate the quantization data of each of the one or more conversion blocks. The reversible decoding unit 62 outputs the generated quantization data to the inverse quantization unit 63a.
[0136]
　The inverse quantization unit 63a back-quantizes the quantization data input from the reversible decoding unit 62, that is, the quantization conversion coefficient for each of the one or more conversion blocks set in the image, and restores the conversion coefficient. .. The inverse quantization unit 63a uses a quantization matrix having the same size as the conversion block among the plurality of quantization matrices stored by the QM storage unit 165a to inversely quantize the quantization conversion coefficient of the conversion block. .. The dequantization unit 63a skips the dequantization of the forcibly zeroed frequency component according to the control by the decoding control unit 61. Then, the inverse quantization unit 63a outputs the restored conversion coefficient to the inverse orthogonal conversion unit 64. In the present embodiment, the inverse quantization unit 63a includes a QM generation unit 167a. The QM generation unit 167a is a module that executes the QM generation process S14 described with reference to FIG. This module will be described further later.
[0137]
　The inverse orthogonal transform unit 64 executes inverse orthogonal transform for each of one or more conversion blocks. The inverse orthogonal transform here may be, for example, an inverse discrete cosine transform or an inverse discrete sine transform. More specifically, the inverse orthogonal conversion unit 64 performs inverse orthogonal conversion of the conversion coefficient in the frequency domain input from the inverse quantization unit 63a for each conversion block to generate a prediction error which is a signal sample in the spatial domain. .. The inverse orthogonal conversion unit 64 outputs the generated prediction error to the addition unit 65.
[0138]
　The addition unit 65 generates a decoded image by adding the prediction error input from the inverse orthogonal conversion unit 64 and the prediction image input from the selector 71. Then, the addition unit 65 outputs the generated decoded image to the in-loop filter 66 and the frame memory 68.
[0139]
　The in-loop filter 66 is composed of a series of filters applied to the decoded image for the purpose of improving the image quality. The in-loop filter 66 may include, for example, one or more of the bilateral filter, the deblocking filter, the adaptive offset filter, and the adaptive loop filter described in reference REF3. The in-loop filter 66 outputs the decoded image filtered through a series of filters to the sorting buffer 67 and the frame memory 68.
[0140]
　The sorting buffer 67 rearranges the images input from the in-loop filter 66 to generate a sequence of time-series images constituting the video. Then, the sorting buffer 67 outputs the generated sequence of images to an external device (for example, a display connected to the image processing device 60a).
[0141]
　The frame memory 68 stores the decoded image before filtering input from the addition unit 65 and the decoded image after filtering input from the in-loop filter 66.
[0142]
　The selector 70 switches the output destination of the image from the frame memory 68 between the intra prediction unit 80 and the inter prediction unit 85 according to the prediction type for each block. For example, when the intra prediction is specified as the prediction type, the selector 70 outputs the decoded image before filtering supplied from the frame memory 68 to the intra prediction unit 80 as a reference image. When inter-prediction is specified as the prediction type, the selector 70 outputs the filtered decoded image as a reference image to the inter-prediction unit 85.
[0143]
　The selector 71 switches the output source of the prediction image to be supplied to the addition unit 65 between the intra prediction unit 80 and the inter prediction unit 85 according to the prediction type for each block. For example, the selector 71 supplies the prediction image output from the intra prediction unit 80 to the addition unit 65 when the intra prediction is specified. Further, when the inter prediction is specified, the selector 71 supplies the prediction image output from the inter prediction unit 85 to the addition unit 65.
[0144]
　The intra prediction unit 80 executes the intra prediction based on the information about the intra prediction parsed from the coded stream and the reference image from the frame memory 68, and generates the prediction image. Then, the intra prediction unit 80 outputs the generated prediction image to the selector 71.
[0145]
　The inter-prediction unit 85 executes inter-prediction based on the information about the inter-prediction parsed from the coded stream and the reference image from the frame memory 68, and generates a prediction image. Then, the inter prediction unit 85 outputs the generated prediction image to the selector 71.
[0146]
　　(2) Functionality Related to Quantization Matrix Generation
　Among the various components shown in FIG. 12, the block setting unit 161, the SL decoding unit 163, the QM storage unit 165a, and the QM generation unit 167a are executed in the decoder. It is mainly related to the generation of the quantization matrix.
[0147]
　The block setting unit 161 sets a plurality of conversion blocks in each image through the QTBT block division executed according to the block division data. The size of the conversion block set by the block setting unit 161 may be in the range of, for example, 2 × 2 to 128 × 128. The shape of the conversion block may be square or non-square. Some examples of the shape and size of the conversion block are shown in FIG.
[0148]
　The SL decoding unit 163 decodes the scaling list data to generate a basic quantization matrix of one or more sizes. In one example, the SL decoding unit 163 decodes the size indication information indicating the size of the quantization matrix generated from the scaling list data. In this case, the SL decoding unit 163 recognizes which size of the quantization matrix is ​​explicitly signaled as the basic quantization matrix via the scaling list data by referring to the size instruction information. The size number information indicating the number of sizes to be signaled may be further decoded. In this case, when two or more sizes are signaled, a bitstream constraint may be imposed that they are of different sizes (identified by different size IDs). Such a bitstream constraint can prevent redundant coding of size instruction information in the encoder, reduce the code amount overhead, and avoid wasting resources. In another example, the size of the basic quantization matrix explicitly signaled through the scaling list data is predefined as a specification rule. In this case, the scaling list data does not include the above-mentioned size instruction information and size number information, and the SL decoding unit 163 decodes the scaling list data for each of one or more predefined sizes, and one of them 1 Generate a quantization matrix of one or more sizes.
[0149]
　The SL decoding unit 163 stores the basic quantization matrix generated based on the scaling list data in the QM storage unit 165a. As described above, for each size, there may be a plurality of types of basic quantization matrices having different combinations of prediction types and color components. The basic quantization matrix may be generated by decoding the values ​​of a series of differentially encoded elements, or may be generated by referring to other types of basic quantization matrices. As an example, the basic quantization matrix contains only a square quantization matrix. As another example, the basic quantization matrix includes both a square quantization matrix and a non-square quantization matrix. An example of the syntax of scaling list data is [2-2. Examples of syntax and semantics].
[0150]
　The QM storage unit 165a is a storage module that stores various types of quantization matrices of various sizes used by the image processing device 60a. The quantization matrix stored by the QM storage unit 165a includes a basic quantization matrix generated by the SL decoding unit 163 and an additional quantization matrix additionally generated by the QM generation unit 167a described later. In the present embodiment, the basic quantization matrix is ​​generated prior to the inverse quantization and the inverse orthogonal transformation over a plurality of transformation blocks, and is stored by the QM storage unit 165a through these processes. On the other hand, the additional quantization matrix is ​​generated as needed when the quantization conversion coefficient of each conversion block is inversely quantized, and is stored in the QM storage unit 165a. Similar to the QM storage unit 115a on the encoder side, the QM storage unit 165a may manage matrix management information indicating which size of the quantization matrix already exists. The matrix management information consists, for example, of a set of flags indicating whether or not there is a quantization matrix identified by two size IDs corresponding to the horizontal and vertical sizes, respectively.
[0151]
　In the QM generation unit 167a, when the quantization conversion coefficient of each conversion block is inversely quantized in the inverse quantization unit 63a, a quantization matrix having the same size as the conversion block is already generated based on the scaling list data. It is determined by referring to the matrix management information provided by the QM storage unit 165a. Then, when the QM generation unit 167a has already generated a quantization matrix of the relevant size, the QM generation unit 167a reads the generated quantization matrix from the QM storage unit 165a. On the other hand, when a quantization matrix having the same size as the conversion block has not been generated, the QM generation unit 167a resamples one of the generated basic quantization matrices or a submatrix thereof to perform additional quantum. Generate a quantization matrix. The QM generation unit 167a includes a memory that temporarily holds the reference quantization matrix or its submatrix to be resampled.
[0152]
　The QM generation unit 167a generates, for example, a quantization matrix for a conversion block when the high-frequency component zeroization is not applied to a conversion block of a certain target size by referring to the entire reference quantization matrix. To do. On the other hand, when the high-frequency component zeroization is applied to a conversion block of another target size, the QM generation unit 167a refers to the quantization matrix for the conversion block and only the submatrix of the reference quantization matrix. It is generated by doing. As mentioned above, the ratio of the size of the referenced submatrix to the size of the reference quantization matrix is ​​equal to the ratio of the nonzero portion to the size of the additional quantization matrix generated. An example of the calculation formula for calculating these ratios is described in [2-4. Resampling method]. The dequantization unit 63a dequantizes the quantization conversion coefficient of each conversion block by using any one of the various quantization matrices generated while conserving resources in this way.
[0153]
　In one example, the QM generator 167a may determine whether zeroization is applied to each conversion block according to a specification rule that predefines which size conversion block the zeroization is applied to. .. In another example, the QM generator 167a includes a zeroization flag that indicates whether zeroization is applied to each conversion block, and zeroization size information that indicates which size conversion block the zeroization is applied to. It may be determined whether zeroization is applied to each transform block based on one or more control parameters that can be additionally parsed from the coded stream. Examples of these control parameters are shown in [1-4. Control of zeroing].
[0154]
　　[2-7. Decoding the time of inverse quantization related Process Flow
　　(1) Overall Flow
　Figure 13 is a flowchart showing an example of the inverse quantization related processing flow executed in the image processing apparatus 60a shown in FIG. 12. Note that, for the sake of simplicity of explanation, the description of the processing steps not related to inverse quantization among the various processing steps included in the decoding process is omitted here.
[0155]
　First, the SL decoding unit 163 executes a scaling list data decoding process described later to generate a scaling list representing a basic quantization matrix having one or more sizes (step S161). Next, the SL decoding unit 163 converts each scaling list, which is an array of one-dimensional element values, into a basic quantization matrix by mapping it to a two-dimensional array in some scan order (step S162). The QM storage unit 165a stores the basic quantization matrix generated in this way (step S163).
[0156]
　Subsequent processes of steps S165 to S168 are repeated for each of the plurality of conversion blocks set in the image through the QTBT block division by the block setting unit 161 (step S164). The conversion block that is the target of each repetition is referred to as the target conversion block here.
[0157]
　First, the QM generation unit 167a determines whether or not a quantization matrix corresponding to the block size of the target conversion block exists by referring to, for example, the matrix management information provided by the QM storage unit 165a (step S165). If the quantization matrix does not exist, the QM generation unit 167a executes the quantization matrix generation process described above with reference to FIG. 10 to generate an additional quantization matrix from the reference quantization matrix (step S166). ). The additional quantization matrix generated here is stored by the QM storage unit 165a. On the other hand, when the required quantization matrix already exists, the QM generation unit 167a reads out the quantization matrix corresponding to the block size of the target conversion block from the QM storage unit 165a (step S167). Then, the inverse quantization unit 63a inversely quantizes the quantization conversion coefficient of the target conversion block by using the quantization matrix additionally generated or read (step S168).
[0158]
　Although not shown in FIG. 13, the conversion coefficient of the frequency domain generated as a result of the inverse quantization in step S168 is converted into a prediction error which is a signal sample of the spatial domain by the inverse orthogonal conversion unit 64. The quantization matrix may be updated in any unit such as a sequence, a picture, a slice or a tile according to the update of the scaling list data.
[0159]
　　(2) Scaling list data decoding process
　FIG. 14 is a flowchart showing an example of a flow of the scaling list data decoding process that can be executed in step S161 of FIG.
[0160]
　First, the SL decoding unit 163 decodes the size instruction information and sets a size ID that identifies the size of the quantization matrix corresponding to the scaling list to be generated (step S171). When the basic quantization matrix is ​​always a square matrix, only one size ID as illustrated in Table 3 is set. When the basic quantization matrix also includes a non-square matrix, two size IDs corresponding to the two directions may be set based on the size instruction information. The size ID may be derived by adding a predefined offset to the value indicated by the size indication information.
[0161]
　Subsequent processing in steps S173 to S177 is repeated for each combination of prediction type and color component, i.e., each type of quantization matrix identified by the matrix ID (step S172). The matrix ID that is the target of each repetition is referred to as the target matrix ID here.
[0162]
　The SL decoding unit 163 first determines whether the values ​​of a series of elements in the scaling list associated with the target matrix ID are explicitly encoded (step S173). For example, in the SL decoding unit 163, the values ​​of a series of elements are coded based on the control flag (for example, scaling_list_pred_mode_flag [sizeId] [matrixId] in Table 1) associated with the size ID and the target matrix ID set in step S171. It can be determined whether it is encoded or only the reference scaling list information is encoded.
[0163]
　When only the reference scaling list information (for example, scaling_list_pred_matrix_id_delta [sizeId] [matrixId] in Table 1) is encoded, the SL decoding unit 163 parses the reference scaling list information to refer to the basic quantization matrix. The matrix ID is derived (step S174). Then, the SL decoding unit 163 generates a scaling list of the target matrix ID based on the reference scaling list referred to by using the derived matrix ID as a key (step S175).
[0164]
　On the other hand, when the values ​​of a series of elements in the scaling list are encoded, the SL decoding unit 163 parses the difference values ​​of the series of elements that are differentially encoded in the scaling list data (step S176). Then, the SL decoding unit 163 generates a scaling list of the target matrix ID by decoding the difference values ​​by the DPCM method (step S177).
[0165]
　The flowchart shown in FIG. 14 shows an example in which the size instruction information indicating the size ID of the basic quantization matrix is ​​decoded. In another example, the size of the basic quantization matrix may be predefined in the specification and decoding of the size indication information may be omitted. Alternatively, the size number information indicating the number of sizes targeted for signaling may be further decoded.
[0166]
　In the first embodiment described in this section, the quantization matrix required for each transformation block is so-called "on the fly" if it has not been generated at the time of quantization or dequantization. According to such a configuration, among various combinations of the shapes and sizes of the quantization matrix, the quantization matrix corresponding to the combination that is not actually used is not generated. Therefore, the processing cost required for generating the quantization matrix can be reduced. In addition to the basic quantization matrix, only the additional quantization matrix that is actually used is stored, so that the consumption of memory resources can be reduced.
[0167]
　In contrast to the first embodiment, in the second embodiment described in the next section, the generation of the additional quantization matrix is ​​performed prior to processing across multiple transformation blocks.
[0168]
　<3. Second Embodiment>
　　[3-1. Encoder Configuration]
　　(1) Overall Configuration
　FIG. 15 is a block diagram showing an example of the configuration of the image processing device 10b having encoder functionality according to the second embodiment. Referring to FIG. 15, the image processing apparatus 10b includes a coding control unit 11, a sorting buffer 12, a subtraction unit 13, an orthogonal conversion unit 14, a quantization unit 15b, a reversible coding unit 16, a storage buffer 17, and a rate control unit. 18, inverse quantization unit 21, inverse orthogonal conversion unit 22, addition unit 23, in-loop filter 24, frame memory 25, switch 26, intra-prediction unit 30, inter-prediction unit 35, mode setting unit 40, QM storage unit 115b and A QM generation unit 117b is provided. The coding control unit 11 includes a block setting unit 111 and a basic QM setting unit 113 as in the first embodiment. The reversible coding unit 16 includes the SL coding unit 119 as in the first embodiment.
[0169]
　The conversion coefficient input from the orthogonal conversion unit 14 and the rate control signal from the rate control unit 18 are supplied to the quantization unit 15b. The quantization unit 15b quantizes the conversion coefficient for each of the one or more conversion blocks in the image to be encoded using a quantization matrix of the same size as the conversion block, and the quantization conversion coefficient (quantization data). ) Is generated. The quantization unit 15b skips the quantization of the forcibly zeroed frequency component included in the conversion coefficient according to the control by the coding control unit 11. Then, the quantization unit 15b outputs the generated quantization data to the reversible coding unit 16 and the inverse quantization unit 21. Further, the quantization unit 15b may change the bit rate of the quantization data by switching the quantization step based on the rate control signal.
[0170]
　　(2) Functionality Related to Quantization Matrix Generation
　Among the various components shown in FIG. 15, the block setting unit 111, the basic QM setting unit 113, the QM storage unit 115b, the QM generation unit 117b, and the SL coding unit 119 is mainly related to the generation of the quantization matrix performed in the encoder. Among the functions of these components, the parts that differ from the corresponding components in the first embodiment will be described below.
[0171]
　The QM storage unit 115b is a storage module that stores various types of quantization matrices of various sizes used by the image processing device 10b. The quantization matrix stored by the QM storage unit 115b includes a basic quantization matrix set by the basic QM setting unit 113 and an additional quantization matrix additionally generated by the QM generation unit 117b described later. In this embodiment, both the basic quantization matrix and the additional quantization matrix are generated prior to orthogonal transformation and quantization over a plurality of transformation blocks, and are stored by the QM storage unit 115b through their processing. The QM storage unit 115b may manage matrix management information which is internal control information indicating which size of quantization matrix already exists.
[0172]
　The QM generation unit 117b generates an additional quantization matrix corresponding to each of the plurality of size candidates of the conversion block prior to the quantization of the conversion coefficients of the plurality of conversion blocks. For example, the QM generation unit 117b refers to one of the generated basic quantization matrices for each of the size candidates determined to be ungenerated based on the matrix management information provided by the QM storage unit 115b. An additional quantization matrix of the size candidate is generated by selecting it as the quantization matrix and resampling the selected reference quantization matrix. Then, the QM generation unit 117b stores the generated additional quantization matrix in the QM storage unit 115b. The QM generation unit 117b includes a memory that temporarily holds the reference quantization matrix or its submatrix to be resampled.
[0173]
　Similar to the QM generation unit 117a according to the first embodiment, the QM generation unit 117b refers to the entire reference quantization matrix when generating an additional quantization matrix having a size to which zeroization of the high frequency component is not applied. .. Further, the QM generation unit 117b refers only to a submatrix of the reference quantization matrix when generating a quantization matrix of a size to which the zeroization of the high frequency component is applied. In the latter case, the ratio of the size of the referenced submatrix to the size of the reference quantization matrix is ​​equal to the ratio of the nonzero portion to the size of the generated additional quantization matrix, as described above.
[0174]
　It should be noted that the examples of syntax and semantics described in detail above in relation to the first embodiment, the examples of the method of selecting the reference quantization matrix, and the additional quantization matrix through resampling (upsampling or downsampling). The content of the example of the method of generating the above applies equally to the second embodiment.
[0175]
　　[3-2. Flow of quantization-related processing at the time of coding]
　FIG. 16 is a flowchart showing an example of the flow of quantization-related processing executed in the image processing apparatus 10b shown in FIG. Note that, for the sake of simplicity of explanation, the description of the processing steps not related to quantization among the various processing steps included in the coding process is omitted here.
[0176]
　First, the basic QM setting unit 113 sets one or more basic quantization matrices including the first quantization matrix of the first size (step S211). These basic quantization matrices are stored in the QM storage unit 115b.
[0177]
　Subsequent processing in steps S213 to S214 is repeated for each of the one or more size candidates, that is, for each combination of the horizontal size ID and the vertical size ID (step S212a). Further, in the repetition, the repetition is performed for each of the plurality of matrix IDs corresponding to the combination of the prediction type and the color component (step S212b). The size candidates and matrix IDs that are the targets of each repetition are referred to as target size candidates and target matrix IDs, respectively.
[0178]
　The QM generation unit 117b first determines whether or not a quantization matrix corresponding to the target size candidate and the target matrix ID exists by referring to, for example, the matrix management information provided by the QM storage unit 115b (step S213). .. If the quantization matrix does not exist, the QM generation unit 117b executes the quantization matrix generation process described above with reference to FIG. 10 to generate an additional quantization matrix from the reference quantization matrix (step S214). ). The additional quantization matrix generated here is stored by the QM storage unit 115b.
[0179]
　After the iteration for generating the additional quantization matrix is ​​completed, the processes of steps S216 to S217 are repeated for each of the plurality of transformation blocks set in the image by the block setting unit 111 (step S215). The conversion block that is the target of each repetition is referred to as the target conversion block here.
[0180]
　The quantization unit 15b reads out the quantization matrix corresponding to the block size of the target conversion block from the QM storage unit 115b (step S216). Then, the quantization unit 15b quantizes the conversion coefficient of the target conversion block using the read quantization matrix (step S217).
[0181]
　The SL coding unit 119 converts each of the one or more basic quantization matrices set in step S211 into a scaling list by making them one-dimensional (step S218). Next, the SL coding unit 119 executes the scaling list coding process described above with reference to FIG. 11 to generate scaling list data (step S219).
[0182]
　Although not shown in FIG. 16, the quantization conversion coefficient for each conversion block generated in step S217 is encoded by the reversible coding unit 16 and becomes a part of the coding stream together with the scaling list data. Scaling list data may be updated in any unit such as sequences, pictures, slices or tiles.
[0183]
　　[3-3. Configuration of Decoder]
　　(1) Overall Configuration
　FIG. 17 is a block diagram showing an example of the configuration of the image processing device 60b having the functionality of the decoder according to the second embodiment. Referring to FIG. 17, the image processing apparatus 60b includes a decoding control unit 61, a reversible decoding unit 62, an inverse quantization unit 63b, an inverse orthogonal conversion unit 64, an addition unit 65, an in-loop filter 66, a sorting buffer 67, and a frame memory. It includes 68, selectors 70 and 71, an intra prediction unit 80, an inter prediction unit 85, a QM storage unit 165b, and a QM generation unit 167b. The decoding control unit 61 includes a block setting unit 161 as in the first embodiment. The reversible decoding unit 62 includes the SL decoding unit 163, as in the first embodiment.
[0184]
　The inverse quantization unit 63b dequantizes the quantization data input from the reversible decoding unit 62, that is, the quantization conversion coefficient for each of the one or more conversion blocks set in the image, and restores the conversion coefficient. .. The inverse quantization unit 63b uses a quantization matrix having the same size as the conversion block among the plurality of quantization matrices stored by the QM storage unit 165b to inverse quantize the quantization conversion coefficient of the conversion block. .. The dequantization unit 63b skips the dequantization of the forcibly zeroed frequency component according to the control by the decoding control unit 61. Then, the inverse quantization unit 63b outputs the restored conversion coefficient to the inverse orthogonal conversion unit 64.
[0185]
　　(2) Functionality related to the generation of the quantization matrix
　Among the various components shown in FIG. 17, the block setting unit 161, the SL decoding unit 163, the QM storage unit 165b, and the QM generation unit 167b are executed in the decoder. It is mainly related to the generation of the quantization matrix. Among the functions of these components, the parts that differ from the corresponding components in the first embodiment will be described below.
[0186]
　The QM storage unit 165b is a storage module that stores various types of quantization matrices of various sizes used by the image processing apparatus 60b. The quantization matrix stored by the QM storage unit 165b includes a basic quantization matrix generated by the SL decoding unit 163 and an additional quantization matrix additionally generated by the QM generation unit 167b described later. In the present embodiment, both the basic quantization matrix and the additional quantization matrix are generated prior to the inverse quantization and inverse orthogonal transformation over a plurality of transformation blocks, and are stored in the QM storage unit 165b through these processes. The QM storage unit 165b may manage matrix management information which is internal control information indicating which size of quantization matrix already exists.
[0187]
　The QM generation unit 167b generates an additional quantization matrix corresponding to each of the plurality of size candidates of the conversion block prior to the inverse quantization of the quantization conversion coefficient of the plurality of conversion blocks. For example, the QM generation unit 167b refers to one of the generated basic quantization matrices for each of the size candidates determined to be ungenerated based on the matrix management information provided by the QM storage unit 165b. An additional quantization matrix of the size candidate is generated by selecting it as the quantization matrix and resampling the selected reference quantization matrix. Then, the QM generation unit 167b stores the generated additional quantization matrix in the QM storage unit 165b. The QM generator 167b includes a memory that temporarily holds the reference quantization matrix or its submatrix to be resampled.
[0188]
　Similar to the QM generation unit 167a according to the first embodiment, the QM generation unit 167b refers to the entire reference quantization matrix when generating an additional quantization matrix having a size to which zeroization of the high frequency component is not applied. .. Further, the QM generation unit 117b refers only to a submatrix of the reference quantization matrix when generating a quantization matrix of a size to which the zeroization of the high frequency component is applied. In the latter case, the ratio of the size of the referenced submatrix to the size of the reference quantization matrix is ​​equal to the ratio of the nonzero portion to the size of the generated additional quantization matrix, as described above.
[0189]
　　[3-4. Flow of dequantization-related processing at the time of decoding]
　FIG. 18 is a flowchart showing an example of the flow of dequantization-related processing executed in the image processing apparatus 60b shown in FIG. Note that, for the sake of simplicity of explanation, the description of the processing steps not related to inverse quantization among the various processing steps included in the decoding process is omitted here.
[0190]
　First, the SL decoding unit 163 executes the scaling list data decoding process described above with reference to FIG. 14 to generate a scaling list representing a basic quantization matrix of one or more sizes (step S261). .. Next, the SL decoding unit 163 converts each scaling list, which is an array of one-dimensional element values, into a basic quantization matrix by mapping it to a two-dimensional array in some scan order (step S262). The QM storage unit 165b stores the basic quantization matrix thus generated (step S263).
[0191]
　Subsequent processing in steps S265 to S266 is repeated for each of the one or more size candidates, that is, for each combination of the horizontal size ID and the vertical size ID (step S264a). Further, in the repetition, the repetition is performed for each of the plurality of matrix IDs corresponding to the combination of the prediction type and the color component (step S264b). The size candidates and matrix IDs that are the targets of each repetition are referred to as target size candidates and target matrix IDs, respectively.
[0192]
　The QM generation unit 167b first determines whether or not a quantization matrix corresponding to the target size candidate and the target matrix ID exists by referring to, for example, the matrix management information provided by the QM storage unit 165b (step S265). .. If the quantization matrix does not exist, the QM generation unit 167b executes the quantization matrix generation process described above with reference to FIG. 10 to generate an additional quantization matrix from the reference quantization matrix (step S266). ). The additional quantization matrix generated here is stored by the QM storage unit 165b.
[0193]
　After the iteration for generating the additional quantization matrix is ​​completed, the processing of steps S268 to S269 is repeated for each of the plurality of transformation blocks set in the image by the block setting unit 161 (step S267). The conversion block that is the target of each repetition is referred to as the target conversion block here.
[0194]
　The inverse quantization unit 63b reads out the quantization matrix corresponding to the block size of the target conversion block from the QM storage unit 165b (step S268). Then, the inverse quantization unit 63b inversely quantizes the quantization conversion coefficient of the target conversion block using the read quantization matrix (step S269).
[0195]
　Although not shown in FIG. 18, the conversion coefficient of the frequency domain generated as a result of the inverse quantization in step S269 is converted into a prediction error which is a signal sample of the spatial domain by the inverse orthogonal converter 64. The quantization matrix may be updated in any unit such as a sequence, a picture, a slice or a tile according to the update of the scaling list data.
[0196]
　In the second embodiment described in this section, a large number of quantization matrices that may be required for the quantization process or the dequantization process are pre-generated prior to the process over a plurality of transformation blocks. According to such a configuration, after the processing for each block in the image is started, the necessity of the calculation for additionally generating the missing quantization matrix can be eliminated, and the processing performance can be improved.
[0197]
　<4. Hardware Configuration Example> The
　above-described embodiment may be realized by using software, hardware, or a combination of software and hardware. When the image processing devices 10a, 10b, 60a, and 60b use the software, the computer program constituting the software is, for example, a computer-readable storage medium (non-temporary medium: non-) provided inside or outside the device. It is stored in advance in transitory media). Then, for example, each program is read into a RAM (Random Access Memory) at the time of execution and executed by a processor such as a CPU (Central Processing Unit).
[0198]
　FIG. 19 is a block diagram showing an example of a hardware configuration of a device to which the above-described embodiment can be applied. Referring to FIG. 19, the device 800 includes a system bus 810, an image processing chip 820 and an off-chip memory 890. The image processing chip 820 includes n processing circuits 830-1, 830-2, ..., 830-n, a reference buffer 840, a system bus interface 850, and a local bus interface 860.
[0199]
　The system bus 810 provides a communication path between the image processing chip 820 and an external module (eg, central control function, application function, communication interface, user interface, etc.). The processing circuits 830-1, 830-2, ..., 830-n are connected to the system bus 810 via the system bus interface 850 and to the off-chip memory 890 via the local bus interface 860. The processing circuits 830-1, 830-2, ..., 830-n can also access the reference buffer 840, which may correspond to an on-chip memory (eg, SRAM). The on-chip memory may include, for example, the internal memory M12 illustrated in FIG. 4 or the internal memory M22 exemplified in FIG. On the other hand, the off-chip memory 890 may include, for example, the QM memory M11 illustrated in FIG. 4 or the QM memory M21 exemplified in FIG. The off-chip memory 890 may further include a frame memory for storing image data processed by the image processing chip 820.
[0200]
　<5. Summary>
　Up to this point, embodiments of the technique according to the present disclosure have been described in detail with reference to FIGS. 3A to 19. According to the above-described embodiment, it is generated by decoding the scaling list data when generating the second quantization matrix for the second size conversion block to which the zeroing of the high frequency component is applied. Only the sub-matrix of the first quantization matrix is ​​referred to, not the entire first quantization matrix of the first size. Thereby, it is possible to at least reduce the influence on the processing cost caused by the generation of various quantization matrices, for example, the occupation of hardware resources, the processing delay, or the increased power consumption. Therefore, an improved mechanism for efficiently generating or signaling a quantization matrix is ​​provided.
[0201]
　Further, according to the above-described embodiment, when the second size is the size to which the zeroization of the high frequency component is applied, it is referred to when generating the second quantization matrix for the first size. The ratio of the size of the submatrix is ​​equal to the ratio of the non-zero part to the second size. As a result, the waste of resources associated with calculating the elements of the quantization matrix corresponding to the conversion coefficient that is always zero can be eliminated.
[0202]
　Further, according to an embodiment, size instruction information indicating the size of the basic quantization matrix generated from the scaling list data is explicitly encoded or decoded as a control parameter. This makes it possible to variably indicate the size of the basic quantization matrix that is the basis for the generation of various quantization matrices, and therefore it is possible to flexibly use the quantization matrix in the form desired by the user. Means to be. According to other embodiments, scaling list data for each of one or more predefined sizes is decoded, and the one or more sizes include the size of the basic quantization matrix. In this case, the need for encoding the size indication information is eliminated.
[0203]
　Also, according to one embodiment, rules regarding which size conversion block to which zeroing of high frequency components is applied are predefined in the specification. In this case, it is not necessary to encode the control parameters for determining whether or not the zeroing of the high frequency component is applied to each conversion block. According to another embodiment, the zeroization flag associated with each conversion block indicating whether zeroing of the high frequency component is applied is explicitly encoded or decoded as a control parameter. This makes it possible to dynamically control the on / off of zeroing. According to another embodiment, zeroization size information indicating to which size conversion block the high frequency component zeroization is applied is explicitly encoded or decoded as a control parameter. As a result, it becomes possible to dynamically control the target size for zeroization. According to these examples, a mechanism is provided that enables flexible reproduction of an image that expresses even minute high-frequency components according to the needs of the user or the requirements or restrictions of the system.
[0204]
　Further, according to the above-described embodiment, when the second quantization matrix is ​​generated, the submatrix is ​​read into the memory and temporarily held. Therefore, since only a necessary and sufficient amount of memory resources are secured, the performance of a device such as an encoder or a decoder is optimized.
[0205]
　Although the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field of the present disclosure can come up with various modifications or modifications within the scope of the technical ideas described in the claims. Of course, it is understood that the above also belongs to the technical scope of the present disclosure.
[0206]
　(Combination of Embodiments) The embodiments
　described above and various related examples may be combined in any manner, and the techniques according to the present disclosure are not limited to the combinations directly mentioned above. For example, any of the methods described in relation to the first embodiment may be combined with any of the methods described in connection with the second embodiment.
[0207]
　(Applicable subject of the proposed technology)
　The technology according to the present disclosure can be applied to any video coding method (video decoding method). That is, unless there is a clear contradiction, the specifications of processing related to coding and decoding such as conversion (reverse conversion), quantization (reverse quantization), coding (decoding), prediction and filtering are related to the present disclosure. Does not limit the scope of technology. In addition, some of these processes may be omitted as long as no obvious contradiction occurs.
[0208]
　(Block) In
　general, "block" is a term that refers to any subregion of an image (including pictures, slices, tiles, etc.) (exceptionally, it may refer to a functional block that performs some functionality. obtain). Unless otherwise stated, block size, shape and properties do not limit the scope of the technology according to the present disclosure. For example, the concept of "block" is referred to in References REF1 to REF3 described above as TB (Transform Block), TU (Transform Unit), PB (Prediction Block), PU (Prediction Unit), SCU (Smallest Coding Unit). ), CU (Coding Unit), LCU (Largest Coding Unit), CTB (Coding Tree Block), CTU (Coding Tree Unit), subblocks, macroblocks, tiles and slices, etc. ..
[0209]
　(Data unit / Processing unit)
　The range of objects to which various parameters, data or information referred to in the present specification are applied is not limited to the above-mentioned example, and may be any range. Similarly, the scope of the objects to which the various processes referred to herein are performed is not limited to the examples described above, and may be any scope. For example, those ranges are in units of at least one of TB, TU, PB, PU, ​​SCU, CU, LCU, CTB, CTU, subblock, block, tile, slice, picture, sequence, or component (data unit / processing). It may be set as a unit). Of course, the data unit / processing unit may be set for each individual parameter, data or information, or for each processing, and may or may not be common to all of them. The parameters, data or information may be stored or transmitted at any location and may be retained, for example, in the header or parameter set of the unit described above. Further, it may be stored or transmitted in a distributed manner in a plurality of places.
[0210]
　(Control Information)
　Any control information related to the technique according to the present disclosure may be transmitted from the coding side to the decoding side. For example, control information (eg, enabled_flag) may be transmitted to instruct to enable or disable any part of the various functionality described above. In addition, control information indicating the applicable range (or non-applicable range) of any part of the various functionalities described above may be transmitted. For example, control information may be transmitted that specifies the block size (upper and lower limits, or both), frames, components, or layers to which the techniques according to the present disclosure apply.
[0211]
　(Block size information) In the
　control information, the block size may be indirectly expressed via an identifier or index mapped to the size value instead of being directly expressed by the size value. Also, for example, a size value or corresponding identifier or index may be expressed as a ratio or difference to some reference value (eg, the size of an LCU or SCU). For example, the size information included in the syntax element may indirectly specify the size value by the method described here. Thereby, it may be possible to reduce the amount of information encoded, transmitted or stored, and improve the coding efficiency. Further, the method for specifying the block size described here may be used when specifying the range of the block size.
[0212]
　(Other)
　Generally, "flag" means information for identifying a plurality of states, and not only distinguishes between two states of true (1) and false (0), but also means three or more states. It may be distinguishable. That is, the value of the "flag" may be, for example, a binary value of 1 or 0, or may be a value of 3 or more. Therefore, one "flag" may be composed of any number of bits, and may be a single bit or a plurality of bits. Further, the control information that may include flags and other identification information may be included in the bitstream itself, or may be included in the bitstream as difference information with respect to the information as some reference.
[0213]
　Further, various data and metadata related to the coded data, such as a coded stream or a coded bit stream, may be transmitted or recorded in any form as long as they are associated with the coded data. Here, the term "associate" means, for example, making the other data available (eg, linking or mapping to each other) when processing one piece of data. That is, the data associated with each other may be treated as one data in an integrated manner, or may be treated as separate data. For example, the information associated with the coded data (encoded image) may be transmitted on a transmission path different from the coded data. Further, the information associated with the coded data may be recorded on a recording medium (or another recording area of ​​the same recording medium) different from the coded data. In addition, the above-mentioned "association" may be performed between arbitrary parts rather than the entire data. For example, an image and information related to the image may be associated with each other in arbitrary units such as a plurality of frames, one frame, or a part within the frame.
[0214]
　In addition, in this specification, "synthesize", "multiplex", "add", "integrate", "include", "store", "insert", "insert" and "insert". The term "" may be used to mean treating separate data in an integrated manner, or to "associate" separate data with each other.
[0215]
　The technology according to the present disclosure is a device or any component constituting the system (for example, a processor such as a system LSI (Large Scale Integration), a module including a plurality of processors, a unit including a plurality of modules, or a module or a unit. It may be implemented in a set) with additional functionality added to.
[0216]
　As used herein, a system means a collection of elements (eg, devices, units, modules, components, etc.), provided that all of these elements may or may not be present in the same enclosure. .. That is, the system is a concept including, for example, a set of a plurality of modules housed in separate housings and connected to each other via a network, and a set of a plurality of modules housed in one housing. Is.
[0217]
　Further, for example, a component described as a single component (for example, a processing unit) may be configured in a manner of being divided into a plurality of components. Conversely, the plurality of components described herein may be configured as a single component. Configurations other than those described may be added to the components described herein. Further, as long as the functionality or operation of the system as a whole is substantially the same, a part of the configuration of one device may be included in another device. Further, for example, the technique according to the present disclosure is realized by using a cloud computing technique in which one or a plurality of functions are performed cooperatively or distributedly by a plurality of devices interconnected via a network. May be good.
[0218]
　Instead of performing one or more steps described in a flowchart in one device, they may be performed in a distributed manner in multiple devices. A plurality of operations constituting one step may be performed in a plurality of devices in a distributed manner instead of being executed in one device.
[0219]
　The program instructions that make up a program executed by a computer may cause the computer to execute the two or more processing steps described herein in the order described or in parallel. Alternatively, it may be executed individually triggered by the occurrence of some event or an external call. That is, the processing steps described herein may be performed in a different order than described, as long as no apparent contradiction arises. Further, a processing step executed based on a certain program or program instruction may be executed in parallel or jointly with a processing step executed based on another program or program instruction.
[0220]
　The effects described herein are descriptive or exemplary and are not limiting. That is, the techniques according to the present disclosure may exhibit other effects apparent to those skilled in the art from the description herein, in addition to or in place of the above effects.
[0221]
　The following configurations also belong to the technical scope of the present disclosure.
(1)
　A decoding unit that decodes the scaling list data to generate a first quantization matrix of the first size, and a
　second for a second size conversion block to which zeroization of the high frequency component is applied. of the quantization matrix, a generation unit for generating by referring to only the first partial matrix of the quantization matrix generated by the decoding unit,
　the second quantization matrix generated by the generating unit An
　image processing apparatus including an inverse quantization unit for inverse quantization of the quantization conversion coefficient of the conversion block of the second size .
(2)
　The image processing apparatus according to (1), wherein the ratio of the size of the submatrix to the first size is equal to the ratio of the non-zero portion to the second size.
(3) The
　decoding unit decodes the size instruction information indicating the size of the quantization matrix generated from the scaling list data, and the size indicated by the size instruction information
　includes the first size. The
　image processing apparatus according to (1) or (2) above , which does not include a second size .
(4) The
　decoding unit decodes the scaling list data for each of the one or more predefined sizes to generate the quantization matrix of the one or more sizes.

　The image processing apparatus according to (1) or (2) 　, wherein the one or more predefined sizes include the first size and do not include the second size .
(5) The
　generation unit uses a third quantization matrix for a conversion block of a third size to which zeroization of the high-frequency component is not applied, and the entire first quantization matrix generated by the decoding unit. The image processing apparatus according to any one of (1) to (4) above, which is generated by referring to.
(6) The
　generation unit determines that the zeroization is applied to the conversion block of the second size according to a rule that defines in advance which size conversion block the zeroization is applied to. The image processing apparatus according to any one of (1) to (5) above.
(7) The
　generation unit indicates that the zeroization is applied to the conversion block of the second size based on the zeroization flag associated with each conversion block, which indicates whether the zeroization is applied. The image processing apparatus according to any one of (1) to (5) above for determining.
(8) The
　generation unit determines that the zeroization is applied to the conversion block of the second size based on the zeroization size information indicating which size conversion block the zeroization is applied to. The image processing apparatus according to any one of (1) to (5) above for determining.
(9) The
　item according to any one of (1) to (8) above, wherein the generation unit includes a memory that temporarily holds the submatrix when the second quantization matrix is ​​generated. Image processing device.
(10) The
　image processing apparatus further includes a storage unit that stores the first quantization matrix through processing over a plurality of conversion blocks, and the
　generation unit is a quantum of individual conversion blocks of the second size.
　The image processing apparatus according to any one of (1) to (9) above , which generates the second quantization matrix when the conversion coefficient is inversely quantized .
(11) The
　generation unit generates the second quantization matrix prior to the processing over the plurality of transformation blocks, and the
　image processing apparatus generates the first quantization matrix and the second quantization matrix.
　The image processing apparatus according to any one of (1) to (9) , further comprising a storage unit that stores a plurality of quantization matrices including the above through processing over the plurality of conversion blocks .
(12) The
　dequantization unit executes dequantization processing for each of a plurality of conversion blocks set in the image through QTBT (Quad-Tree plus Binary Tree) block division, described in (1) to (11). The image processing apparatus according to any one of ().
(13)
　An image processing method executed by an image processing apparatus, in which
　scaling list data is decoded to generate a first quantization matrix of a first size.
　To generate a second quantization matrix for a second size transformation block to which zeroing of high frequency components is applied by referring only to the submatrix of the first quantization matrix generated. , An 　image processing method including
　inverse quantization of the quantization conversion coefficient of the conversion block of the second size using the generated second quantization matrix
.
(14)
　By referring only to the submatrix of the first quantization matrix of the first size, the second quantization matrix for the second size conversion block to which the zeroing of the high frequency components is applied. a generation unit for generating,
　the transform coefficients of the transform blocks of the second size in the encoding target image, and quantized using the second quantization matrix generated by the generation unit, the quantized transform coefficients An 　image processing apparatus including a quantization unit for generating the
　above-mentioned, and a coding unit for encoding the quantization conversion coefficient and the scaling list expressing the first quantization matrix to generate a coded stream
.
(15)
　An image processing method executed by an image processing apparatus, wherein a
　second quantization matrix for a second size conversion block to which zeroing of high frequency components is applied is a first size first. It is generated by referring only to a sub-matrix of the quantization matrix of 1, and
　the conversion coefficient of the conversion block of the second size in the image to be encoded is obtained by using the generated second quantization matrix. To quantize and generate a quantization conversion coefficient,
　An
　image processing method comprising encoding a scaling list representing the quantization conversion coefficient and the first quantization matrix to generate a coded stream .
Description of the sign
[0222]
　10a, 10b Image processing device (encoder)
　15a, 15b Quantization unit
　16 Reversible coding unit
　111 Block setting unit
　113 QM setting unit
　115a, 115b QM storage unit
　117a, 117b QM generation unit
　119 SL coding unit
　60a, 60b Image processing Device (decoder)
　62 Reversible decoding unit
　63a, 63b Inverse quantization unit
　161 Block setting unit
　163 SL decoding unit
　165a, 165b QM storage unit
　167a, 167b QM generation unit
The scope of the claims
[Claim 1]
　A decoding unit that decodes the scaling list data to generate a first quantization matrix of first size, and a second quantization
　for a second size conversion block to which zeroing of high frequency components is applied. Using the generation unit generated by referring only to the submatrix of the first quantization matrix generated by the decoding unit and
　the second quantization matrix generated by the generation unit, the matrix is ​​used. An
　image processing apparatus including an inverse quantization unit that dequantizes the quantization conversion coefficient of the second size conversion block .
[Claim 2]
　The image processing apparatus according to claim 1, wherein the ratio of the size of the submatrix to the first size is equal to the ratio of the non-zero portion to the second size.
[Claim 3]
　The decoding unit decodes the size instruction information indicating the size of the quantization matrix generated from the scaling list data, and the size indicated by the size instruction information
　includes the first size and the second size.
　The image processing apparatus according to claim 1 , which does not include a size .
[Claim 4]
　The decoding unit decodes the scaling list data for each of the one or more
　predefined sizes to generate the quantization matrix of the one or more sizes, and the one or more sizes defined in advance.
　The image processing apparatus according to claim 1, wherein the image processing apparatus includes the first size and does not include the second size .
[Claim 5]
　The generator refers to a third quantization matrix for a third size conversion block to which zeroing of high frequency components is not applied, and the entire first quantization matrix generated by the decoding unit. The image processing apparatus according to claim 1, which is generated by the above.
[Claim 6]
　1. The generation unit determines that the zeroization is applied to the conversion block of the second size according to a rule that defines in advance which size conversion block the zeroization is applied to. The image processing apparatus according to.
[Claim 7]
　The generator determines that the zeroing is applied to the second size conversion block based on the zeroing flag associated with each conversion block, which indicates whether the zeroing is applied. The image processing apparatus according to claim 1.
[Claim 8]
　The generation unit determines that the zeroization is applied to the conversion block of the second size based on the zeroization size information indicating which size conversion block the zeroization is applied to. The image processing apparatus according to claim 1.
[Claim 9]
　The image processing apparatus according to claim 1, wherein the generation unit includes a memory that temporarily holds the submatrix when generating the second quantization matrix.
[Claim 10]
　The image processing apparatus further includes a storage unit that stores the first quantization matrix through processing over a plurality of conversion blocks, and the
　generation unit is a quantization conversion coefficient of the individual conversion blocks of the second size.
　The image processing apparatus according to claim 1 , wherein the second quantization matrix is ​​generated when the second quantization matrix is ​​dequantized .
[Claim 11]
　The generation unit generates the second quantization matrix prior to the processing over the plurality of transformation blocks, and the
　image processing apparatus includes the first quantization matrix and the second quantization matrix. The
　image processing apparatus according to claim 1 , further comprising a storage unit that stores the quantization matrix of the above through processing over the plurality of conversion blocks .
[Claim 12]
　The image processing apparatus according to claim 1, wherein the inverse quantization unit executes an inverse quantization process for each of a plurality of conversion blocks set in the image through QTBT (Quad-Tree plus Binary Tree) block division.
[Claim 13]
　A second image processing method performed by an image processing apparatus, in which
　scaling list data is decoded to generate a first quantization matrix of first size, and
　zeroing of high frequency components is applied. A second quantization matrix for a conversion block of the size of is generated by referring only to a submatrix of the first quantization matrix generated, and
　the second quantization matrix generated. An
　image processing method comprising dequantizing the quantization conversion coefficient of the second size conversion block using the above .
[Claim 14]
　Generation of generating a second quantization matrix for a second size conversion block to which zeroing of the high frequency components is applied by referring only to a submatrix of the first size first quantization matrix. The part and
　the conversion coefficient of the conversion block of the second size in the image to be encoded are quantized by using the second quantization matrix generated by the generation unit to generate a quantization conversion coefficient. An 　image processing apparatus including a
　quantization unit, a coding unit that encodes a scaling list representing the quantization conversion coefficient and the first quantization matrix, and generates a coded stream
.
[Claim 15]
　A
　second quantization matrix for a second size conversion block to which zeroing of high frequency components is applied, which is an image processing method performed by an image processing apparatus, and a first quantum of the first size. It is generated by referring only to a submatrix of the conversion matrix, and
　the conversion coefficient of the conversion block of the second size in the image to be encoded is quantized using the generated second quantization matrix. An 　image processing method including
　generating a quantization conversion coefficient and encoding a scaling list representing the quantization conversion coefficient and the first quantization matrix to generate a coded stream
.