Technical field
[0001]
　The present disclosure relates to an image processing apparatus and method, and more particularly, to an image processing apparatus and method which make it possible to suppress the reduction of the coding efficiency.
BACKGROUND
[0002]
　Conventionally, in image coding, after performing the primary transform on the prediction residual being the image and the difference of the predicted image, further increasing the energy compaction (to concentrate the conversion factor to the low frequency) for conversion block for each sub-block of the inner, it is disclosed that the application of secondary transformation (e.g., see non-Patent Document 1). Its Non-Patent Document 1 also discloses that signal the secondary conversion identifier CU unit indicating whether to apply any secondary conversion.
[0003]
　Further, in the encoder, based on RDO (Rate-Distortion Optimization), which secondary transform to decide whether to apply the CU units described in Non-Patent Document 1, computational complexity is large, secondary conversion block to signal the secondary conversion flag indicating whether to apply the conversion have been disclosed (e.g., see non-Patent Document 2). Its Non-Patent Document 2, which secondary indicating whether to apply a secondary transform conversion identifier, it is also disclosed that derived based on the primary conversion identifiers and the intra prediction mode.
[0004]
　However, even in the method according to any of the Non-Patent Documents 1 and 2, if the non-zero coefficient is sparse subblock is input to the secondary transform coefficients, the secondary transform is applied, low in subblock following coefficient is diffused reduces the energy compaction to high order components from the encoding efficiency is likely to be reduced.
[0005]
　Moreover, the Joint Exploration Test Model 1 (JEM1), for encoding efficiency in the high-resolution images such as 4K, to extend the maximum size of the CTU (Coding Tree Unit) to 256x256, accordingly, the transform block the maximum size is also disclosed that extended to 64x64 (e.g., see non-Patent Document 1). Its non-patent document 1, if the transform block size is 64x64, encoder, the band limitation as transform coefficients of the high frequency components other than the low frequency components of 32x32 of the upper left transform block is forced to 0 performed (truncate high frequency components), it is also disclosed that encode only non-zero coefficients of low frequency components.
[0006]
　In this case, the decoder decodes only the non-zero coefficients of the low frequency components, it may be performed inverse quantization and inverse transform on the non-zero coefficients. Therefore, compared to the case where the encoder does not perform a band limitation, reducing the computational complexity and implementation costs of the decoder.
[0007]
　However, the encoder for converting blocks of 64x64, transformation skip (Transform Skip), or skip transform and quantization (hereinafter referred to as transform quantization bypass) when performing, transform coefficients of a transform block of 64x64 is a prediction residual of the previous conversion. Therefore, the band limitation is performed in this case, the strain increases. As a result, there is a possibility that the coding efficiency is reduced. Moreover, despite to convert quantization bypass lossless coding purposes (Trans / Quant Bypass), it is impossible to perform the lossless coding.
CITATION
Non-patent literature
[0008]
非特許文献1 : Jianle Chen, Elena Alshina, Gary J. Sullivan, Jens-Rainer Ohm, Jill Boyce, "Algorithm Description of Joint Exploration Test Model 2", JVET-B1001_v3, Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 2nd Meeting: San Diego, USA, 20-26 February 2016
非特許文献2 : X.Zhao, A.Said, V.Seregin, M.Karczewicz, J.Chen, R.Joshi, "TU-level non-separable secondary transform", JVET-B0059, Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 2nd Meeting: San Diego, USA, 20-26 February 2016
Summary of the Invention
Problems that the Invention is to Solve
[0009]
　As described above, there is a risk that the encoding efficiency is reduced.
[0010]
　The present disclosure has been made in view of such circumstances, it is desirable to make it possible to suppress the reduction of the coding efficiency.
Means for Solving the Problems
[0011]
　The image processing apparatus of the first embodiment of the present technology, a primary transform is a transformation process for prediction residual is the difference between the predicted image of the image with the image, the prediction residual is obtained is the primary conversion primary case of skipping the secondary conversion and a conversion process for conversion coefficient, band-limited to the secondary transformation coefficients the primary transform coefficients obtained are the secondary conversion is also an image processing apparatus including a control unit to skip.
[0012]
　The image processing method of the first embodiment of the present technology, the inverse primary transform is an inverse transformation of the primary transformation is a transformation process for prediction residual is the difference between the predicted image of the image with the image, the prediction residual case of skipping the inverse secondary transform that is a conversion process in which the secondary conversion of inverse transform to the primary transform coefficients obtained are the primary conversion, secondary to the primary transform coefficients obtained by said being secondary converted by band-limited band limitation for transform coefficients is also an image processing apparatus including a control unit to skip.
[0013]
　The image processing apparatus of the second embodiment of the present technology, the inverse primary transform is an inverse transformation of the primary transformation is a transformation process for prediction residual is the difference between the predicted image of the image with the image, the prediction residual case of skipping the inverse secondary transform that is a conversion process in which the secondary conversion of inverse transform to the primary transform coefficients obtained are the primary conversion, secondary to the primary transform coefficients obtained by said being secondary converted by band-limited band limitation for transform coefficients is also an image processing apparatus including a control unit to skip.
[0014]
　The image processing method of the second embodiment of the present technology, the image processing apparatus, the inverse primary transform is an inverse transformation of the primary transformation is a transformation process for prediction residual is the difference between the predicted image of the image with the image, case of skipping the inverse secondary transform and which is the inverse of the secondary transform the prediction residual is conversion processing for the primary transform coefficients obtained are the primary conversion is band-limited the primary transform coefficients is the secondary conversion band-limited to the secondary transform coefficients obtained by also an image processing method including a control step of skipping.
[0015]
　The image processing apparatus of the third embodiment of the present technology, the skip of the inverse transform processing for transform coefficient prediction residual is the difference between the predicted image of the image with the image by reverse conversion is obtained, each sub-block based on the number of nonzero coefficients of the transform coefficients of an image processing apparatus including a control unit for controlling each said sub-block.
[0016]
　The image processing method of the third aspect of the present technology, the skip of the inverse transform processing for transform coefficient prediction residual is the difference between the predicted image of the image with the image by reverse conversion is obtained, each sub-block based on the number of nonzero coefficients of the transform coefficients, an image processing method for controlling for each of the sub-blocks.
[0017]
　The image processing apparatus of the fourth aspect of the present technique, a matrix of inverse transform processing for transform coefficient, and a setting unit that sets, based on the content and the scan method of the inverse transform process, an image by reverse conversion using a rasterizing unit for converting the transform coefficients prediction residual is obtained which is a difference between the predicted image of said image into a one-dimensional vector, the matrix set by the setting unit, the matrix with respect to the one-dimensional vector image processing comprising: a matrix calculation unit for performing an operation, and a scaling unit for scaling to the matrix operation of the one-dimensional made vector, a matrix unit for a matrix of scaled the one-dimensional vector it is a device.
[0018]
　The image processing method of the fourth aspect of the present technology, a matrix of inverse transform processing for transform coefficient, and set on the basis of the content and the scan method of the inverse transformation processing, the image and of the image by reverse conversion It converts the transform coefficient prediction residual is the difference between the predicted image is obtained on the one-dimensional vector, by using the set the matrix, performs a matrix operation on the one-dimensional vector, the matrix operation is performed wherein performs scaling with respect to one-dimensional vector, a scaling image processing method for a matrix of said one-dimensional vector.
[0019]
　In the image processing apparatus and method of the first aspect of the present technology, a primary transform is a transformation process for prediction residual is the difference between the predicted image of the image with the image, the prediction residual is the primary conversion case of skipping the secondary conversion and a conversion process for the primary transform coefficients obtained, the band for the secondary transform coefficients the primary transform coefficients obtained are the secondary conversion is also skipped.
[0020]
　In the image processing apparatus and method of the second aspect of the present technology, the inverse primary transform is an inverse transformation of the primary transformation is a transformation process for prediction residual is the difference between the predicted image of the image with the image, the prediction If the residual is to skip the inverse secondary transform that is the inverse of the secondary transform is a transformation processing on the primary transform coefficients obtained are the primary conversion by the primary transform coefficients are the secondary converted by band-limited band limitation to obtain a secondary transform coefficients is skipped.
[0021]
　In the image processing apparatus and method of the third aspect of the present technology, the inverse transform process skip to the image and the conversion coefficients prediction residual is obtained which is a difference between the predicted image of the image by reverse conversion, based on the number of nonzero coefficients of the transform coefficients for each sub-block, is controlled for each sub-block.
[0022]
　In the image processing apparatus and method of the fourth aspect of the present technique, a matrix of inverse transform processing for transform coefficient is set based on the content and the scanning method of the inverse transform process, an image by reverse conversion its conversion coefficient prediction residual is obtained which is a difference between the predicted image of the image is converted into a one-dimensional vector, by using the set matrix, is performed matrix operation with respect to the one-dimensional vector, the matrix operation scaling is performed on the one-dimensional vector is performed, one-dimensional vectors that scaling is matrixing.
Effect of the invention
[0023]
　According to the present disclosure, an image can be processed. In particular, it is possible to suppress the reduction of the coding efficiency.
BRIEF DESCRIPTION OF THE DRAWINGS
[0024]
FIG. 1 is an explanatory diagram for explaining the outline of recursive block division for CU.
It is a [2] explanatory diagram for describing the setting of the PU to the CU shown in FIG.
3 is an explanatory diagram for illustrating setting of TU to CU shown in FIG.
FIG. 4 is an explanatory diagram for explaining the scanning order of the CU / PU.
5 is a block diagram showing a main configuration example of a secondary converter section.
6 is a diagram illustrating an example of the way in which the secondary conversion.
7 is a block diagram showing a main configuration example of an image encoding apparatus.
8 is a block diagram showing a main configuration example of a conversion unit.
It is a diagram showing an example of the scanning method corresponding to FIG. 9 Scan identifier.
Is a diagram illustrating an example of FIG. 10 for the secondary transformation matrix.
11 is a flowchart illustrating an example of the image encoding processing flow.
12 is a flowchart illustrating an example of a conversion processing flow.
13 is a block diagram showing a main configuration example of an image decoding apparatus.
14 is a block diagram showing a main configuration example of the inverse transform unit.
Is a flowchart illustrating an example of the flow of FIG. 15 the image decoding processing.
Is a flowchart illustrating an example of the flow of FIG. 16 inverse transform process.
17 is a diagram for explaining an example of the relationship between the intra prediction mode and the scan method.
18 is a block diagram showing a main configuration example of a conversion unit.
19 is a block diagram showing a main configuration example of a secondary change selecting section.
FIG. 20 is a flowchart illustrating an example of a conversion processing flow.
It is a block diagram showing a main configuration example of FIG. 21 inverse transform unit.
It is a block diagram showing a main configuration example of FIG. 22 inverse secondary transform selection portion.
FIG. 23 is a flowchart for explaining an example of the inverse conversion processing flow.
It is a block diagram showing a main configuration example of FIG. 24 converter unit.
Is a diagram illustrating a FIG. 25 band limits.
FIG. 26 is a block diagram showing a main configuration example of a conversion unit.
FIG. 27 is a flowchart illustrating an example of a conversion processing flow.
It is a block diagram showing a main configuration example of FIG. 28 inverse transform unit.
FIG. 29 is a flowchart for explaining an example of the inverse conversion processing flow.
[Figure 30] CU, diagrams PU, and TU of the shape will be described.
[FIG. 31] is a block diagram showing a main configuration example of a conversion unit.
[FIG. 32] is a diagram showing an example of a band-limiting filter.
[FIG 33 is a flowchart illustrating an example of a conversion processing flow.
It is a block diagram showing a main configuration example of FIG. 34 inverse transform unit.
Is a flowchart illustrating an example of the flow of FIG. 35 inverse transform process.
[FIG. 36] is a diagram showing another example of the band limiting filter.
It is a diagram showing another example of FIG. 37 band-limiting filter.
[FIG. 38] is a diagram showing another example of the band limiting filter.
[39] is a flowchart illustrating another example of a conversion processing flow.
It is a flowchart illustrating another example of the flow of FIG. 40 inverse transform process.
[FIG. 41] is a block diagram showing a main configuration example of a computer.
[FIG. 42] is a block diagram showing an example of a schematic configuration of a television device.
[43] is a block diagram showing an example of a schematic configuration of a mobile phone.
[FIG. 44] is a block diagram showing an example of a schematic configuration of a recording and reproducing apparatus.
Is a block diagram showing an example of a schematic configuration of FIG. 45 imaging device.
[FIG. 46] is a block diagram showing an example of a schematic configuration of a video set.
[FIG. 47] is a block diagram showing an example of a schematic configuration of the video processor.
[FIG. 48] is a block diagram showing another example of a schematic configuration of the video processor.
[FIG. 49] is a block diagram showing an example of a schematic configuration of a network system.
DESCRIPTION OF THE INVENTION
[0025]
　The following describes embodiments of the present disclosure (hereinafter referred to as embodiments). The description will be made in the following order.
　1. First Embodiment (skipping the secondary conversion for each sub-block)
　2. Second Embodiment (Selection of Secondary conversion using the scanning method)
　3. Third Embodiment (skipping band limiting when the block is a square)
　4. 4 (skip band limiting when the block is a rectangle composed of square or rectangular) embodiment of
　5. Fifth Embodiment (Other)
[0026]
　<1. First Embodiment>
　　
　MPEG2 (the Moving Picture Experts Group 2 (ISO / IEC 13818-2)) or MPEG-4 Part10 (Advanced Video Coding , hereinafter referred to as AVC) older image encoding method such as in the encoding process is executed by the processing units called macroblocks. Macro block is a block having a uniform size of 16x16 pixels. In contrast, in HEVC (High Efficiency Video Coding), the encoding processing is executed by the CU (Coding Unit) and referred to the processing unit (coding unit). CU is the maximum coding unit LCU and (Largest Coding Unit) is formed by dividing recursively, a block having a variable size. The maximum size of the selectable CU is 64x64 pixels. The minimum size of the selectable CU is 8x8 pixels. The minimum size of the CU is referred to as the SCU (Smallest Coding Unit). The maximum size of the CU is not limited to 64x64 pixels, larger 128x128 pixels, or as a block size, such as 256x256 pixels.
[0027]
　Thus, the results of CU having a variable size is employed, in HEVC, it is possible to adjust the image quality and encoding efficiency adaptively depending on the content of the image. Prediction process for predictive coding is performed in PU (Prediction Unit) and referred to the processing unit (prediction unit). PU is formed by dividing the CU in one of several split pattern. Further, PU is composed of luminance (Y) and color difference (Cb, Cr) for each of PB (Prediction Block) and referred to the processing unit (prediction block). Further, orthogonal transform processing is executed by the TU (Transform Unit) and referred to the processing unit (conversion unit). TU is formed by dividing to a depth in the CU or PU. Moreover, TU consists luminance (Y) and color difference (Cb, Cr) for each of TB (Transform Block) and referred to the processing unit (transformation blocks).
[0028]
　　
　Figure 1 is an explanatory diagram for explaining the outline of recursive block division of CU in HEVC. Block division of CU is carried out by repeating a division into 4 (= 2x2) sub-blocks of a block recursively, resulting quadtree (Quad-Tree) like a tree structure is formed . The whole of one of the quad-tree is referred to as CTB (Coding Tree Block), a logical unit corresponding to the CTB that CTU.
[0029]
　At the top of FIG. 1, as an example, C01 is a CU having a size of 64x64 pixels is illustrated. Dividing the depth of the C01 is equal to zero. This means that C01 corresponds to and LCU root of CTU. LCU size may be specified by parameters that are encoded in the SPS (Sequence Parameter Set) or PPS (Picture Parameter Set). A CU C02 is one of the four CU divided from C01, having a size of 32x32 pixels. Dividing the depth of the C02 is equal to 1. A CU C03 is one of the four CU divided from C02, having a size of 16x16 pixels. Dividing the depth of the C03 is equal to 2. A CU C04 is one of the four CU divided from C03, having a size of 8x8 pixels. Dividing the depth of the C04 is equal to 3. Thus, CU is formed by recursively dividing an image to be encoded. The depth of the division is variable. For example, a flat image region such as a blue sky, the larger size (i.e., smaller depth) CU can be set. On the other hand, the steep image area containing many edges, the smaller size (i.e., greater depth) CU can be set. Each of the set CU becomes the processing unit of encoding processing.
[0030]
　　
　PU is a process unit of the prediction process including intra prediction and inter prediction. PU is formed by dividing the CU in one of several split pattern. Figure 2 is an explanatory diagram for illustrating setting of the PU to the CU shown in FIG. The right of FIG. 2, 2Nx2N, 2NxN, Nx2N, NxN, 2NxnU, 2NxnD, that nLx2N and NRx2N, are shown eight division patterns. Of these division patterns, the intra prediction, two 2Nx2N and NxN is selectable (NxN only selectable SCU). The inter prediction In contrast, when an asymmetric motion division is enabled, all eight split patterns can be selected.
[0031]
　　
　TU is a processing unit of orthogonal transform processing. TU is CU (for intra CU, each PU in the CU) is formed by dividing to a depth in the. Figure 3 is an explanatory diagram for describing TU settings to CU shown in FIG. To the right of FIG. 3, there is shown one or more TU that may be set in C02. For example, a TU T01 has a size of 32x32 pixels, the depth of the TU division is equal to zero. A TU T02 has a size of 16x16 pixels, the depth of the TU division is equal to 1. A TU T03 has a size of 8x8 pixels, the depth of the TU division is equal to 2.
[0032]
　Above CU, it is whether to what block division to set the block such PU and TU in the image, is typically determined based on a comparison of influences cost coding efficiency. Encoder, for example, a CU of one 2Mx2M pixel, compare the cost with the CU four MxM pixels, if there is better to set the CU four MxM pixels encoding efficiency is high, the 2Mx2M pixel CU It decides to split into four MxM pixels CU a.
[0033]
　　
　when encoding an image, the image (or slice, tiles) CTB set in a grid pattern in (or LCU) is scanned in a raster scan order. Within one CTB, CU is scanned to follow the quadtree from left to right, top to bottom. When processing a current block, the above information and the left neighboring block is used as input information. Figure 4 is an explanatory diagram for explaining the scanning order of the CU and PU. At the top left of FIG. 4, a four CU that may be included in one of CTB, C10, C11, C12 and C13 are shown. The number in the frame of each CU has represent the order of processing. Encoding process, a top left CU C10, which is the upper right of the CU C11, C12 is a lower left CU, are executed in the order of C13 is CU at the lower right. The right of Figure 4, one or more PU for inter prediction that can be set in C11 is CU are shown. Under 4, one or more PU for intra prediction may be set in C12 is CU are shown. As shown in the figures within the framework of these PU, PU is also scanned to follow from left to right, top to bottom.
[0034]
　In the following, there are (not a block of processing unit) When described with reference to "block" as the partial area and processing unit of an image (picture). The "block" in this case refers to any partial area in the picture, its size, shape, and characteristics, etc. are not limited. That is, the "block" in this case, for example, TB, TU, PB, PU, ​​SCU, CU, LCU (CTB), the sub-block, macroblock, tiles, or slice, etc., any partial region (processing units), It is intended to be included.
[0035]
　　
　Non-Patent Document 1 and Non-Patent Document 2, an image and the prediction residual being the difference between the predicted image after the primary conversion, further transform coefficients increase the energy compaction (low-pass for the cause) concentrate, for each sub-block in the transform block, it is described that applies a secondary transformation.
[0036]
　However, the nonzero coefficient secondary converts the sparse prediction residual coefficients from low-order to high-order component is diffused energy compaction decreased, the encoding efficiency is likely to be reduced.
[0037]
　Figure 5 is a block diagram showing a main configuration example of a secondary converter section for performing secondary conversion. Secondary transformation unit 11 shown in FIG. 5, for the primary transform coefficients prediction residual is primary conversion is a processing unit that performs secondary conversion by the method described in Non-Patent Document 1 and Non-Patent Document 2. As shown in FIG. 5, the secondary conversion unit 11 has a rasterizing unit 21, matrix operation unit 22, the scaling unit 23, and the matrix unit 24.
[0038]
　Rasterizing unit 21, a primary transform coefficients Coeff_P entered, then scanned in the scan method indicated by the scan identifier ScanIdx, 1-dimensional vectors X 1d is converted to. For example, the primary transform coefficients Coeff_P is, non-zero coefficients as shown in the following equation (1) and is sparse 4 × 4 matrix, the scan identifier scanIdx to denote the horizontal scanning (hor), rasterizer 21 , the primary transform coefficients Coeff_P 1 dimensional vector X as shown in the following equation (2) 1d is converted to.
[0039]
[Number

　1] (1)
[number

　2] (2)
[0040]
　Matrix operation unit 22, a one-dimensional vector X obtained as described above 1d respect, performing a matrix operation, such as the following equation (3) using a matrix R of the secondary conversion. For example, one-dimensional vector X represented by formula (2) described above 1d respect, by the matrix operation is performed, the one-dimensional vector Y as shown in the following equation (4) 1d is obtained .
[0041]
　The Y 1D T = R & lt · X- 1D T
　· (3)
[Number

　3] ... (4)
[0042]
　Scaling unit 23, a one-dimensional vector Y obtained as described above 1d respect, in order to normalize the norm, such N (N is a natural number) bit shift operation of the bit as shown in the following expression (5) I do. For example, one-dimensional vector Y shown in the above equation (4) 1d respect, by the bit shift operation is performed, the one-dimensional vector Z as shown in the following equation (6) 1d obtained It is.
[0043]
　Z 1d = ( Y 1d )>>N
　・・・（５）
[数4]

　・・・（６）
[0044]
　Matrixing unit 24, a one-dimensional vector Z obtained as described above 1d , and based on the scanning method is specified by the scan identifier ScanIdx, into a matrix. This matrix is the primary transform coefficients Coeff_P is supplied to the subsequent processing unit as a secondary transform coefficients Coeff which is secondary converted (e.g. quantization unit, etc.). For example, one-dimensional vector Z shown in the above equation (4) 1d respect, by the matrix of is performed, the secondary transform coefficients a 4 × 4 matrix as shown in the following equation (7) Coeff can be obtained.
[0045]
[Formula

　5] · (7)
[0046]
　As described above, when the non-zero coefficients are secondary converts the sparse primary transform coefficients Coeff_P, energy compaction decreases factor diffuses from lower order to higher order components, the coding efficiency is likely to be reduced .
[0047]
　Incidentally, Non-Patent Document 2, in order to reduce the overhead of the secondary conversion flag, if non-zero coefficients in a transform block is below a predetermined threshold value, it is decided not to apply a secondary transform to omit the signal flag There has been disclosed. For example, when the threshold TH = 2, primary transform coefficients Coeff_P is, in the case of a matrix as shown in equation (1) described above, the secondary transform is skipped (omitted).
[0048]
　However, in the case of the method described in Non-Patent Document 1 and Non-Patent Document 2, for secondary converts the converted block for each sub-block. Thus, for example, as shown in FIG. 6, the transform block of the primary transform coefficients Coeff_P is constituted by 2 × 2 sub-blocks, in 4 × 4 matrix as each sub-block is shown in equation (1) above some cases, the non-zero coefficients in a transform block is 4, the threshold TH (= 2) from large, so that the secondary transform is applied. Secondary transformation as described above, since carried out for each sub-block (4 × 4 matrix for each), the secondary transform coefficients Coeff of each sub-block is as shown in the above equation (7). That is, energy compaction decreases factor diffuses from lower order to higher order components, the coding efficiency is likely to be reduced.
[0049]
　　
　Therefore, the skip of the conversion processing for the transform coefficients obtained from the prediction residual is the difference between the predicted image of the image and the image, the non-zero coefficients of the transform coefficients for each sub-block based on the number, so as to control for each sub-block. Also, the skip of the inverse transform processing for transform coefficient prediction residual is obtained a difference image and the predicted image of the image by reverse conversion, based on the number of nonzero coefficients of the transform coefficients for each sub-block Te, so as to control for each sub-block. For example, the non-zero coefficients in the sub-block when the threshold value or less, so as to skip (omit) the secondary transform or inverse secondary transform.
[0050]
　By doing so, with respect to the transform coefficients of non-zero coefficients is sparse subblocks, it is possible to suppress the applying conversion processing (inverse conversion process), suppressing a decrease in energy compaction, code it is possible to suppress the reduction in efficiency.
[0051]
　　
　FIG. 7 is a block diagram showing an example of a configuration of an image encoding device which is an embodiment of an image processing apparatus to which the present technology is applied. The image coding apparatus 100 shown in FIG. 7, as in the AVC and HEVC, an apparatus for encoding a prediction residual of the image and its prediction picture. For example, the image coding apparatus 100, technologies and proposed in HEVC, implementing the techniques proposed by JVET (Joint Video Exploration Team).
[0052]
　Incidentally, in FIG. 7 shows the main ones, such as the flow of the processing unit and data is not all that shown in Figure 7. That is, in the image encoding apparatus 100, or there is processing unit not shown as a block in FIG. 7, may be or there is a flow of processing and data not shown as arrows or the like in FIG.
[0053]
　The image coding apparatus 100 as shown in FIG. 7, the control unit 101, arithmetic unit 111, conversion unit 112, a quantization unit 113, the coding unit 114, an inverse quantization unit 115, inverse transform unit 116, arithmetic unit 117 , a frame memory 118 and the prediction unit 119,.
[0054]
　Control unit 101, based on the external or the block size of the pre-specified processing unit, the block of a processing unit moving image input to the image coding apparatus 100 (CU, PU, ​​transformation block (TB), etc.) to divided, to supply the image I corresponding to the divided block to the calculating unit 111. The control unit 101, coding parameters supplied to each block (header information Hinfo, prediction mode information pinfo, conversion information Tinfo, etc.), for example, determined based on RDO (Rate-Distortion Optimization). Determined coding parameters are supplied to each block.
[0055]
　Header information Hinfo, for example, a video parameter set (VPS (Video Parameter Set)), sequence parameter set (SPS (Sequence Parameter Set)), picture parameter set (PPS (Picture Parameter Set)), the slice header (SH), such as including the information. For example, the header information Hinfo, the image size (width PicWidth, longitudinal width PicHeight), bit depth (brightness BitDepthY, chrominance BitDepthC), the maximum value MaxCUSize / minimum value MinCUSize, maximum MaxTBSize / minimum value of the transform block size of the CU size MinTBSize, (also referred to as maximum conversion skipped block size) maximum MaxTSSize transform skip blocks, (also referred to as valid flag) off flags of the coding tools includes information defining the like. Of course, the contents of the header information Hinfo is optional, any information other than the examples described above may be included in the header information Hinfo.
[0056]
　The prediction mode information pinfo, for example, PU size is information indicating a (prediction block size) PU size PUSize be processed PU, which is information on the intra prediction mode of the block to be processed intra prediction mode information IPinfo (e.g., JCTVC -W1005, 7.3.8.5 Coding Unit syntax in prev_intra_luma_pred_flag, mpm_idx, rem_intra_pred_mode etc.), which is information relating to motion prediction block to be processed motion prediction information MVinfo (e.g., JCTVC-W1005, merge_idx in 7.3.8.6 prediction Unit syntax , merge_flag, inter_pred_idc, ref_idx_LX, mvp_lX_flag, X = {0,1}, include mvd etc.) and the like. Of course, the content of the prediction mode information pinfo is optional, any information other than the examples described above may be included in the prediction mode information pinfo.
[0057]
　The conversion information Tinfo, for example, include the following information.
[0058]
　Block size TBSize (or logarithm of TBSize that a base of 2 Log2TBSize, also referred to as transform block size) is information indicating a block size of processed transform block.
[0059]
　Secondary transformation identifier (St_idx), in the data unit of interest, an identifier indicating whether to apply any secondary transform or inverse secondary transform ((reverse) referred secondary transform both) (e.g., JVET-B1001,2.5.2 Secondary Transforms In a reference .JEM2, nsst_idx, also referred to as rot_idx). In other words, the secondary transform identifier is information about the contents of (inverse) Secondary transformation in the data units of interest.
[0060]
　For example, secondary converter identifier st_idx, if its value is greater than 0, an identifier for specifying the matrix (inverse) Secondary transformation. In other words, in this case, the secondary transform identifier st_idx shows the execution of (inverse) Secondary transformation. Further, for example, secondary converter identifier st_idx, if its value is 0, indicating the skip (inverse) Secondary transformation.
[0061]
　Scan identifier (scanIdx) is information about the scanning method. Quantization parameter (qp) is the data unit of interest is information indicating the quantization parameter used for (inverse) quantization. Quantization matrices (Scaling_matrix), in the data unit of interest is information indicating the quantization matrix used in the (inverse) quantization (e.g., JCTVC-W1005, 7.3.4 Scaling list data syntax).
[0062]
　Of course, the content of the conversion information TINFO is optional, any information other than the examples described above may be included in the conversion information TINFO.
[0063]
　Header information Hinfo, for example, it is supplied to each block. Prediction mode information Pinfo, for example, is supplied to the coding section 114 and the prediction unit 119. Conversion information Tinfo, for example, conversion unit 112, a quantization unit 113, an encoding unit 114, are supplied to the inverse quantization unit 115, and inverse transform unit 116.
[0064]
　Computing unit 111 from the image I corresponding to the block of the input processing unit, a predicted image P supplied from the prediction unit 119 obtains a prediction residual D is subtracted as shown in equation (8), it and supplies to the converter 112.
[0065]
　Ｄ＝Ｉ－Ｐ
　・・・（８）
[0066]
　Converter 112, based on the conversion information Tinfo supplied from the control unit 101 performs conversion processing on prediction residual D supplied from the arithmetic unit 111, and the transform coefficients Coeff. Conversion unit 112 supplies the transform coefficients Coeff to the quantization unit 113.
[0067]
　Quantization unit 113, based on the conversion information Tinfo supplied from the control unit 101, a scaling (quantized) transformation coefficients Coeff supplied from the conversion unit 112. That is, the quantization unit 113 quantizes the transform coefficient Coeff the conversion process has been performed. Quantization unit 113, transform coefficients after quantization obtained through the quantization, ie supplies quantized transform coefficient level level to encoding section 114 and the inverse quantization unit 115.
[0068]
　Encoding unit 114 encodes the quantized transform coefficient levels level like supplied from the quantization unit 113 in a predetermined manner. For example, the coding unit 114, along with the definition of the syntax table, coding parameters supplied from the control unit 101 (header information Hinfo, prediction mode information pinfo, such conversion information TINFO) and, supplied from the quantization unit 113 the quantized transform coefficient level level that is, converted into the syntax values ​​of each syntax element, coding each syntax value (e.g., arithmetic coding), and generates a bit sequence (encoded data).
[0069]
　The encoding unit 114 derives a residual information rinfo from the quantized transform coefficient levels level, encodes the residual information rinfo, generates a bit sequence (encoded data).
[0070]
　The residual information rinfo, for example, the last non-zero coefficient X-coordinate (last_sig_coeff_x_pos), last non-zero coefficient Y coordinates (last_sig_coeff_y_pos), sub-block non-zero coefficient presence flag (coded_sub_block_flag), non-zero coefficients presence flag (sig_coeff_flag), non level of zero coefficients is flag information indicating whether greater than 1 GR1 flag (gr1_flag), GR2 flag level of the non-zero coefficient is flag information indicating whether greater than 2 (gr2_flag), indicating the sign of non-zero coefficients sign in a sign code (sign_flag), and the like non-zero coefficient residual level is information indicating the residual level of non-zero coefficients (coeff_abs_level_remaining) (e.g., see 7.3.8.11 residual coding syntax of JCTVC-W1005). Of course, the content of residual information rinfo is optional, any information other than the examples described above may be included in the residual information rinfo.
[0071]
　Encoding unit 114, for example, a bit string of each syntax element is encoded (encoded data) multiplexed, and outputs it as bit stream.
[0072]
　Inverse quantization unit 115, based on the conversion information Tinfo supplied from the control unit 101, the scaling values ​​of the quantized transform coefficient levels level supplied from the quantization unit 113 (inverse quantization) and, after dequantization to derive the conversion coefficient Coeff_IQ. Inverse quantization unit 115 supplies the transform coefficients Coeff_IQ inverse transformation unit 116. Inverse quantization performed by the inverse quantization unit 115, an inverse process of the quantization performed by the quantization unit 113 is the same processing as the inverse quantization performed in the image decoding apparatus described later. Therefore, this inverse quantization will be described later in description of the image decoding apparatus.
[0073]
　Inverse transform unit 116, based on the conversion information Tinfo supplied from the control unit 101 performs inverse transform on the transform coefficients Coeff_IQ supplied from the inverse quantization unit 115 derives a prediction residual D '. Inverse transform unit 116 supplies the prediction residual D 'to the arithmetic unit 117. Inverse conversion performed by the inverse transformation unit 116, an inverse process of conversion performed by the conversion unit 112, an inverse transform and the same processing performed in the image decoding apparatus described later. Therefore, this inverse transform will be described later in description of the image decoding apparatus.
[0074]
　Calculation unit 117, the prediction residual D supplied from the inverse transform unit 116 'and is supplied from the prediction unit 119, the prediction residual D' and the predicted image P (prediction signal) corresponding to the following formula (9) adding to derive the local decoded image Rec as. Operation unit 117 supplies the local decoded image Rec in the frame memory 118.
[0075]
　Rec＝Ｄ’＋Ｐ
　・・・（９）
[0076]
　Frame memory 118, reconstructs the decoded image for each picture unit with a local decoded image Rec supplied from the arithmetic unit 117 is stored in the buffer in the frame memory 118. The frame memory 118 is read out from the buffer the decoded image is designated by the prediction unit 119 as a reference image, and supplies the prediction unit 119. The frame memory 118, the header information Hinfo according to generation of the decoded image, the prediction mode information pinfo, and conversion information TINFO, may be stored in a buffer in the frame memory 118.
[0077]
　Prediction unit 119 is designated by the prediction mode information pinfo, acquires the decoded image stored in the frame memory 118 as a reference image, using the reference image, the predicted image by the prediction method specified by the prediction mode information Pinfo to generate a P. Prediction unit 119 supplies the generated prediction image P in the arithmetic unit 111 and the operation unit 117.
[0078]
　In such an image encoding apparatus 100, the skip of the conversion processing for the transform coefficients obtained from the prediction residual is the difference between the predicted image of an image and its image, the number of non-zero coefficients of the transform coefficients for each sub-block based on, so that a control unit for controlling each subblock. That is, the conversion unit 112, the image and the skip of the conversion processing for the transform coefficients obtained from the prediction residual is the difference between the predicted image of the image, based on the number of nonzero coefficients of the transform coefficients for each sub-block, so as to control for each sub-block.
[0079]
　　
　FIG. 8 is a block diagram showing a main configuration example of a conversion unit 112. 8, converter 112 includes a primary converter section 131 and a secondary converter section 132.
[0080]
　Primary conversion unit 131, the prediction residual D supplied from the arithmetic operation unit 111, for example, performs the primary conversion of the orthogonal transform or the like, transform coefficients Coeff_P (primary conversion after primary transformation corresponding to the prediction residual D to derive the coefficient also referred to). That is, the primary conversion unit 131 converts the prediction residual D to the primary transform coefficients Coeff_P. Primary conversion unit 131 supplies the derived primary transform coefficients Coeff_P the secondary converter section 132 (rasterizing unit 141 and the switch 148 to be described later).
[0081]
　Secondary conversion unit 132 converts the primary transform coefficients Coeff_P supplied from the primary conversion unit 131 into one-dimensional vector (also referred to as row vector) performs matrix operation with respect to the one-dimensional vector, the matrix operation is performed and one-dimensional scale the vector, performing secondary transform is a transformation process to the matrix of the one-dimensional vectors that scaling.
[0082]
　Secondary converter section 132, and the secondary conversion identifier st_idx is information about the contents of the secondary conversion, based on the scan identifier scanIdx and is information about the scanning method of the transform coefficients, performs secondary transform to the primary transform coefficients Coeff_P, secondary deriving the transform coefficients Coeff after conversion (also referred to as secondary transform coefficients). That is, the secondary conversion unit 132 converts the primary transform coefficients Coeff_P secondary transform coefficient Coeff. Secondary transformation unit 132, the secondary transform coefficients Coeff, supplies the quantization unit 113.
[0083]
　Incidentally, the secondary conversion unit 132, a secondary conversion skipped (omitted), the primary transform coefficients Coeff_P, as a secondary transform coefficients Coeff, can be supplied to the quantization unit 113.
[0084]
　As shown in FIG. 8, a secondary converter section 132, the rasterizing unit 141, the matrix calculating unit 142, a scaling unit 143, the matrix section 144, a secondary change selecting section 145, a quantization unit 146, the non-zero coefficient count determination unit 147 , and a switch 148.
[0085]
　Rasterizing unit 141, based on the scan method of transform coefficients that is specified by the scan identifier ScanIdx, for each sub-block (4 × 4 sub-blocks), the primary transform coefficients Coeff_P supplied from the primary conversion unit 131 one-dimensional vector X 1D to convert to. Rasterizing unit 141, a one-dimensional vector X obtained 1d supplies the matrix calculator 142.
[0086]
　A of FIG. 9 shows a scan type scanType specified by the value of the scan identifier ScanIdx. As will be shown in A of FIG. 9, when the scan identifier scanIdx is zero, it is specified oblique direction scan (up-right diagonal scan), when the scan identifier scanIdx is 1, the horizontal scan (horizontal fast scan) specified, the scan identifier scanIdx the case 2, the vertical scanning (vertical fast scan) is designated. B to D of FIG. 9 in FIG. 9 shows a scanning order of coefficients in each scan in 4 × 4 sub-blocks. In D and B to 9 in FIG. 9, number assigned to each coefficient position indicates the order in which the coefficient position is scanned. B of FIG. 9 shows a scanning order of an example of a horizontal scan (horizontal fast scan), C in FIG. 9 shows an example of a scan order vertical scanning (vertical fast scan), D in FIG. 9, an example of a scan order in an oblique direction scan (up-right diagonal scan).
[0087]
　Secondary change selecting section 145 supplies the matrix R of the secondary conversion, read from the internal memory (not shown) of the secondary change selecting section 145, the matrix calculator 142 which is specified by the secondary conversion identifier St_idx. For example, a secondary change selecting section 145, when the value of a secondary conversion identifier St_idx, as the secondary conversion, reads the matrix R of 16 × 16 as shown in FIG. 10, and supplies the matrix calculator 142.
[0088]
　Incidentally, the secondary change selecting section 145, a secondary conversion identifier st_idx and intra prediction mode information IPinfo (e.g., prediction mode number) in accordance with, may select the matrix R of the secondary conversion. Further, the secondary change selecting section 145, instead of the intra prediction mode information IPinfo, in response to the movement prediction information MVinfo and secondary conversion identifier St_idx, may be selected matrix R of the secondary conversion.
[0089]
　Matrix calculator 142, a one-dimensional vector X 1d using matrix R and secondary conversion, performs a matrix calculation shown in the following equation (10), one-dimensional vector Y which is the result 1d scaling unit and supplies to 143. In the formula (10), the operator "·" represents an operation for performing an inner product of the matrix-matrix (matrix product) operator "T" represents the operation of the transposed matrix.
[0090]
　Y 1d T＝Ｒ・X 1d T
　・・・（１０）
[0091]
　Scaling unit 143, the signal Y supplied from the matrix calculator 142 1d (i.e. one-dimensional vector Y 1d to normalize the norm of), N (N is a natural number as shown in the following equation (11) ) performs bit shift operation of the bit, the signal Z after bit shift 1d (i.e. one-dimensional vector Z 1d Request). Note that before the shift operation of N bits as shown in the following expression (12), as an offset, the value of 1 << (N-1), 1-dimensional vector Z 1d be added to each element of good.
[0092]
　Z 1d＝（Y 1d）>>Ｎ
　・・・（１１）
　Z 1d＝（Y 1d＋（（N-1）<<1）・Ｅ）>>Ｎ
　・・・（１２）
[0093]
　In the equation (12), E is the value of all the elements is 1 × 16 dimensional vector of 1. For example, the matrix R of the secondary conversion shown in FIG. 10 are the matrix is 8-bit scaling, the scaling unit 143, the value of N used for the normalization of the norm is 8. Generally, the matrix R of the secondary conversion, if it is N bits scaling, bit shift amount of the norm normalization is N bits. Scaling unit 143, a one-dimensional vector Z obtained as described above 1d supplies the matrixing unit 144.
[0094]
　Matrixing unit 144, based on the scan method specified by the scan identifier scanIdx, 1 × 16 dimensional vector Z after norm normalization 1d converts the into 4 × 4 matrix. Matrixing unit 144, the resulting transform coefficients Coeff, supplies the quantization unit 146 and a switch 148.
[0095]
　Quantization unit 146 performs quantization unit 113 are basically the same processing (quantization). However, the quantization unit 146, the primary conversion coefficients Coeff_P, secondary transform coefficient Coeff (after processing execution until rasterizing unit 141 to matrix unit 144), which is part of the conversion information Tinfo quantization parameter qp, and the quantization and enter the matrix scaling_matrix. Quantization unit 146, the primary conversion coefficients Coeff_P supplied from the primary conversion unit 131, for each to be the transform coefficient supplied from the matrix unit 144 secondary transform coefficients Coeff, the quantization parameter qp, quantization matrices scaling_matrix refers to the, for example, quantized by the following equation (13) and (14), (referred to as quantized primary transform coefficients) primary transform coefficients level_P after quantization, the secondary transform coefficients level_S after quantization ( deriving a called quantization secondary transform coefficients).
[0096]
　level_P(i,j) = sign (coeff_P(i,j)) × (( abs ( coeff_P(i,j)) × f[qp%6] × (16 / w(i,j)) + offsetQ ) >> qp / 6 ) >> shift1
　・・・（１３）
　level_S(i,j) = sign (coeff (i,j)) × (( abs ( coeff (i,j)) × f[qp%6] × (16 / w(i,j)) + offsetQ ) >> qp / 6 ) >> shift1
　・・・（１４）
[0097]
　In the formula (13) or equation (14), operator sign (X) is an operator that returns a positive or negative sign of the input value X. For example, if X> = 0, +1, if X <0, and -1. Further, f [] is a scaling factor that depends on the quantization parameter takes a value such as the following equation (15).
[0098]
　F[qp%6] = [26214, 23302, 20560, 18396, 16384, 16384, 14564]
　・・・（１５）
[0099]
　Further, in the equation (13) or equation (14), w (i, j) is, Coeff (i, j) and (Coeff_P (i, j)) coefficient position of (i, j) quantization matrix corresponding to is the value of the scaling_matrix. In other words, the w (i, j) is obtained as the following equation (16). Furthermore, shift1 and offsetQ is obtained as the following equation (17) or (18).
[0100]
　W (I, J) = Scaling_matrix
　(I, J) · · · (16)
　Shift1 = 29 - M - B
　· · · (17)
　OffsetQ = 28 - M - B
　· · · (18)
　However, the formula (17 ) and in (18), M is a counter value of the block size TBSize transform block to the base 2, B is the bit depth bitDepth input signal.
　= Log2 M (TBSize)
　B = BitDepth
[0101]
　Note that regardless of the quantization shown in equation (13) and (14), the quantization method may be changed in extent practicable.
[0102]
　Quantization unit 146 supplies the above quantization primary transform coefficients obtained as level_P and quantized secondary transform coefficients level_S nonzero coefficient number determination unit 147.
[0103]
　Non-zero coefficient count determination unit 147, for each sub-block, and inputs the quantized primary transform coefficients level_P and quantized secondary transform coefficients Level_S. Non-zero coefficient number determination section 147 refers to the quantized primary transform coefficients supplied from the quantization unit 146 Level_P and quantized secondary transform coefficients Level_S, for each, the number numSigInSBK_P of nonzero coefficients in the sub-block (Quantum reduction is also referred to as non-zero coefficients speed of the primary transform coefficients) and numSingInSBK_S a (also referred to as non-zero coefficient number of the quantized secondary transform coefficients), for example, be derived by the following equation (19) and (20). In the equation (19) and equation (20), (i, j) represents the coordinates of the sub-block is i = 0 ... 3, j = 0 ... 3. Also, operators abs (X) is an operator that returns the absolute value of the input value X. By equation (19) and (20) can level value of the transform coefficients after quantization is to derive the number of greater than zero transform coefficients (non-zero coefficient). In the equation (19) and (20), of non-zero coefficients determination condition (abs (level_X (i, j))> 0) (X = P, S) is, (level_X (i, j)! = 0) may be replaced by a determination condition.
[0104]
　numSigInSBK_P = Σ{abs(level_P(i,j))>0 ? 1 : 0}
　・・・（１９）
　numSigInSBK_S = Σ{abs(level_S(i,j))>0 ? 1 : 0}
　・・・（２０）
[0105]
　Non-zero coefficient count determination unit 147, a non-zero coefficient number numSigInSBK_P quantized primary transform coefficients, with reference to the non-zero coefficient number numSigInSBK_S with a predetermined threshold value TH of the quantized secondary transform coefficients, the following equation (21) , to derive a secondary conversion skip flag StSkipFlag is information about skipping the secondary conversion.
[0106]
　StSkipFlag = ( numSigInSBK_P <= numSigInSBK_S && numSigInSBK_P <= TH ) ? 1 : 0
　・・・（２１）
[0107]
　Threshold value TH, for example, sets of two, not limited to this and can be set to a value of from 0 to 16. The threshold TH may be notified in the header information such as VPS / SPS / PPS / slice header SH. The threshold TH is the encoding side (for example, an image coding apparatus 100) and the decoding side (e.g., the image decoding apparatus 200 to be described later) previously negotiated beforehand with the notification from the encoding side to the decoding side (threshold of TH, may be omitted transmission) from the encoding side to the decoding side.
[0108]
　In equation (21), the non-zero coefficient number numSigInSBK_P quantized primary transform coefficients in the case of the following non-zero coefficient number numSigInSBK_S quantized secondary transform coefficients and nonzero coefficient number numSigInSBK_P quantized primary transform coefficients If equal to or smaller than the threshold value TH, the value of the secondary transform skip StSkipFlag is set to 1. That is, in the secondary conversion skip StSkipFlag, to skip a secondary conversion shown. Otherwise (numSigInSBK_P> numSigInSBK_S || numSigInSBK_P> TH), the value of the secondary transform skip StSkipFlag is set to 0. That is, in the secondary conversion skip StSkipFlag, be performed secondary conversion shown.
[0109]
　Instead of the above equation (21), it may be used the following equation (22) or formula (23). When using the equation (23) may be omitted quantization processing of the secondary transform coefficients.
[0110]
　StSkipFlag = ( numSigInSBK_P <= numSigInSBK_S && numSigInSBK_S <= TH ) ? 1 : 0
　・・・（２２）
　StSkipFlag = ( numSigInSBK_P <= TH ) ? 1 : 0
　・・・（２３）
[0111]
　Non-zero coefficient count determination unit 147 supplies the derived secondary transform skip flag StSkipFlag the switch 148.
[0112]
　Switch 148, the primary conversion coefficients of the sub-block unit Coeff_P, an input secondary transform coefficients Coeff, and secondary conversion skip flag StSkipfFlag. Switch 148, in accordance with the secondary conversion skip flag StSkipFlag supplied from non-zero coefficient number determination unit 147, controls the skipping secondary conversion.
[0113]
　For example, if the value of the secondary transform skip flag StSkipFlag is 0, i.e., in the secondary conversion skip StSkipFlag, when the execution of the secondary transform is shown, switch 148, to perform the secondary transformation. That is, the switch 148 supplies the secondary transform coefficients Coeff supplied from the matrix unit 144 to the quantization unit 113. For example, when the value of the secondary transform skip flag StSkipFlag is 1, i.e., in the secondary conversion skip StSkipFlag, if skip secondary transform is shown, switch 148 is skipped secondary conversion. That is, the switch 148 supplies the quantization unit 113 to the primary transform coefficients Coeff_P supplied from the primary conversion unit 131 as a secondary transform coefficients Coeff.
[0114]
　The switch 148 is also based on secondary conversion identifier St_idx, it is possible to control the skipping secondary conversion. For example, if the secondary conversion identifier st_idx is zero (indicating a skip secondary conversion), the switch 148, regardless of the value of the secondary transform skip flag StSkipFlag, to skip the secondary conversion. That is, the switch 148 supplies the quantization unit 113 to the primary transform coefficients Coeff_P supplied from the primary conversion unit 131 as a secondary transform coefficients Coeff. For example, when the secondary conversion identifier st_idx is greater than 0 (indicating execution of a secondary conversion), the switch 148 refers to the secondary conversion skip flag StSkipFlag, controls the secondary conversion as described above.
[0115]
　As described above, the non-zero coefficient count determination unit 147 based on the number of non-zero coefficients for each sub-block, setting the secondary conversion skip flag StSkipFlag, switch 148, based on the secondary conversion skip flag StSkipFlag, secondary to control the skip of the conversion. By doing so, since the non-zero coefficients becomes possible to skip the secondary transform for sparse subblock, suppressing a decrease in energy compaction, it is possible to suppress the reduction of the coding efficiency.
[0116]
　　
　Next, an example of the flow of each process executed by the image encoding apparatus 100. First, an example of the flow of the image encoding process will be described with reference to the flowchart of FIG. 11.
[0117]
　When the image coding process is started, in step S101, the control unit 101 performs encoding control processing, performs setting of block division and the coding parameter.
[0118]
　In step S102, the prediction unit 119 performs prediction processing, and generates a prediction image or the like of the optimum prediction mode. For example, in the prediction processing, the prediction unit 119 performs intra prediction to generate a prediction image or the like of the optimum intra prediction mode by performing inter prediction generates a prediction image like the optimal inter prediction mode, their from within, to select an optimal prediction mode based on a cost function value and the like.
[0119]
　In step S103, the calculation unit 111 calculates the difference between the input image, and the prediction image of the selected optimum mode by the prediction processing in step S102. That is, the arithmetic unit 111 generates a prediction residual D between the input image and the prediction image. Prediction residual D obtained in this way, the data amount is reduced as compared to the original image data. Therefore, as compared with the case of directly coding the image, it is possible to compress the data amount.
[0120]
　In step S104, the conversion unit 112 performs conversion processing on the prediction residual D generated by the processing in step S103, and the transform coefficients Coeff. It will be described later in detail the processing in step S104.
[0121]
　In step S105, the quantization unit 113, and the like using the quantization parameters calculated by the control unit 101, the transform coefficient Coeff obtained by the process of step S104 is quantized to derive the quantized transform coefficient level level .
[0122]
　In step S106, the inverse quantization unit 115, the quantized transform coefficient level level generated by the processing in step S105, inverse quantization with characteristics corresponding to the characteristics of the quantization of the step S105, and the transform coefficients Coeff_IQ .
[0123]
　In step S107, the inverse transform unit 116, the transform coefficients Coeff_IQ obtained by the processing in step S106, the inverse transform by a method corresponding to the conversion process in step S104, to derive a prediction residual D '. Incidentally, the inverse transform process is the inverse process of the conversion process in step S104, are performed similarly to the inverse transform processing performed by the image decoding process described below. Therefore, description of the inverse conversion process is carried out in the description of the decoding side.
[0124]
　In step S108, the arithmetic unit 117 processes the prediction residual D 'derived by the step S107, by adding the predicted image obtained by the prediction processing in step S102, the decoded picture which is locally decoded generated.
[0125]
　In step S109, the frame memory 118 is obtained by the processing in step S108, and stores the decoded picture which is locally decoded.
[0126]
　In step S110, the encoding unit 114 encodes the quantized transform coefficient levels level obtained by the processing in step S105. For example, the coding unit 114, the quantized transform coefficient level level is information about the image, coded by arithmetic encoding or the like, to generate encoded data. At this time, the coding unit 114, various coding parameters (header information HINFO, prediction mode information pinfo, conversion information TINFO) for encoding. Furthermore, the coding unit 114 derives a residual information rinfo from the quantized transform coefficient levels level, encodes the residual information rinfo. Coding section 114, collectively coded data of the generated various kinds of information in this manner, and outputs to the outside of the image coding apparatus 100 as a bit stream. This bit stream, for example, is transmitted to the decoding side through a transmission path or a recording medium.
[0127]
　When the process of step S110 is completed, the image encoding process is completed.
[0128]
　These processes units of each processing is arbitrary, it may not be identical to each other. Accordingly, the process of each step, as appropriate, in parallel with the processing of the other steps, or may be executed by interchanging the processing order.
[0129]
　　
　Next, an example of a conversion processing flow to be executed in step S104 of FIG. 11 will be described with reference to the flowchart of FIG. 12.
[0130]
　When the conversion process is started, in step S121, the primary conversion unit 131, based on the primary transformation identifier Pt_idx, performs primary conversion for the prediction residual D, and deriving a primary transform coefficients Coeff_P.
[0131]
　In step S122, the secondary conversion unit 132 (the switch 148) determines whether the secondary conversion identifier St_idx applies a secondary conversion (st_idx> 0). If the secondary transform identifier st_idx is determined to be 0 (indicating the skip secondary conversion), a secondary conversion (processing of steps S123 to step S134) is skipped, the conversion process is completed, the process returns to FIG. 11. That is, the secondary conversion unit 132 (the switch 148) and supplies it to the quantization unit 113 to the primary transform coefficients Coeff_P as the transformation coefficient Coeff.
[0132]
　Further, in step S122, if it is determined that the larger the secondary conversion identifier st_idx is 0 (indicating the execution of the secondary conversion), the process proceeds to step S123.
[0133]
　In step S123, the secondary change selecting section 145 selects the matrix R of the secondary transform that is specified by the secondary conversion identifier St_idx.
[0134]
　In step S124, the secondary conversion unit 132 divides the transform block to be processed into sub-blocks, selects an unprocessed sub-block.
[0135]
　In step S125, the rasterization unit 141, based on the scan method specified by the scan identifier ScanIdx, the primary transform coefficients Coeff_P 1-dimensional vectors X 1d is converted to.
[0136]
　In step S126, the matrix calculator 142, a one-dimensional vector X 1d matrix product of the matrix R and the secondary conversion operation, a one-dimensional vector Y 1d seek.
[0137]
　In step S127, the scaling unit 143, a one-dimensional vector Y 1d the norm of normalized one-dimensional vector Z 1d seek.
[0138]
　In step S128, the matrix section 144, based on the scan method specified by the scan identifier ScanIdx, 1-dimensional vector Z 1d converts the matrix of 4 × 4, the secondary transform coefficients Coeff sub block to be processed Ask.
[0139]
　In step S129, the quantization unit 146, for each of the primary transform coefficients Coeff_P and secondary transform coefficients Coeff, the quantization parameter qp, quantized with reference to quantization matrices Scaling_matrix, the quantized primary transform coefficients Level_P, quantization secondary to derive a conversion coefficient level_S.
[0140]
　In step S130, the non-zero coefficient count determination unit 147, derives for each sub-block, the quantized primary transform coefficients Level_P, based on the quantization secondary transform coefficients Level_S, and the threshold value TH, as described above the secondary transformation skip flag StSkipFlag to.
[0141]
　In step S131, the switch 148, the secondary transform skip flag StSkipfFlag derived in step S130, it is determined whether or not indicate a skip of the secondary conversion. Showing the execution of a secondary conversion, i.e., when the value of the secondary transform skip flag StSkipFlag is 0, the process proceeds to step S132.
[0142]
　In step S132, (supplied to the quantization unit 113) switch 148, for outputting a secondary transform coefficients Coeff obtained by the processing in step S128. Processing as the steps S132 ends, the process proceeds to step S134.
[0143]
　Further, in step S131, the indicating the skip secondary conversion, i.e., when the value of the secondary transform skip flag StSkipFlag is 1, the process proceeds to step S133.
[0144]
　In step S133, the switch 148, (supplied to the quantization unit 113) to the primary transform coefficients Coeff_P obtained in the processing in step S121 is output as the secondary transform coefficients Coeff. Processing and the processing of step S133 is completed, the process proceeds to step S134.
[0145]
　In step S134, the secondary conversion unit 132 determines whether processing of all of the sub-blocks of transform blocks processed. If unprocessed sub-block is determined to exist, processing returns to step S124, the repeat the process. That is, for each sub-block of transform blocks processed, the processing in steps S124 to step S134 (secondary conversion) is executed. In step S134, if it is determined that the processing of all the sub-blocks (were executed or skipped secondary conversion of all of the sub-blocks), the conversion process is completed, the process returns to FIG. 11.
[0146]
　The conversion process is a feasible range, the processing order and replacement steps, may change the contents of processing. For example, if it is determined that the secondary conversion identifier st_idx = 0 in step S122, 16 matrix of × 16 is selected as the matrix R of the secondary conversion, as the processing in steps S124 through step S134 is executed it may be.
[0147]
　By executing the processes as described above, a sub-block unit, it is possible to control the skipping secondary conversion (run). Therefore, the non-zero coefficients with respect to sparse residual signal, it is possible to suppress the reduction of the energy compaction. That is, it is possible to suppress the reduction of the coding efficiency. In other words, while suppressing a decrease in encoding efficiency, it is possible to suppress the increase of the load of encoding (secondary transform and inverse secondary transform).
[0148]
　　
　Next, a description will be given decoding of encoded data encoded as described above. Figure 13 is a block diagram showing an example of a configuration of an image decoding apparatus which is an embodiment of an image processing apparatus to which the present technology is applied. The image decoding apparatus 200 shown in FIG. 13 is an image decoding apparatus corresponding to the image encoding apparatus 100 of FIG. 7, the encoded data by the image coding apparatus 100 has generated (bit stream), the image coding apparatus 100 decoding by the decoding method corresponding to the encoding method according. For example, the image decoding device 200, technologies and proposed in HEVC, implementing the proposed technique in JVET.
[0149]
　Incidentally, in FIG. 13 shows the main ones, such as the flow of the processing unit and data is not all that shown in Figure 13. That is, the image decoding apparatus 200, or there is processing unit not shown as a block in FIG. 13 may be or there is a flow of processing and data not shown as arrows or the like in FIG. 13.
[0150]
　The image decoding apparatus 200 as shown in FIG. 13 has a decoding unit 211, an inverse quantization unit 212, inverse transform unit 213, arithmetic unit 214, frame memory 215 and the prediction unit 216,. The image decoding apparatus 200, for example via a transmission medium or a recording medium such as the image coding apparatus 100 or the like is supplied as encoded data generated by, for example, a bit stream or the like.
[0151]
　Decoding unit 211, the encoded data supplied is decoded by a predetermined decoding method corresponding to the coding method. For example, the decoding unit 211, along with the definition of the syntax table, a bit string of the supplied encoded data (bit stream) and decodes the syntax values ​​of each syntax element. The syntax element, for example, header information Hinfo, prediction mode information pinfo, conversion information TINFO, includes information such as the residual information rinfo.
[0152]
　Decoding section 211 refers to the residual information rinfo, to derive the quantized transform coefficient levels level for each coefficient position in each transform block. Decoding unit 211 supplies the prediction mode information Pinfo obtained by decoding the quantized transform coefficient levels level, the conversion information Tinfo to each block. For example, the decoding unit 211 supplies the prediction mode information Pinfo to the prediction unit 216 supplies the quantized transform coefficient level level to the inverse quantization unit 212, a conversion information Tinfo to the inverse quantization unit 212 and inverse transform unit 213 supplies.
[0153]
　Inverse quantization unit 212, based on the conversion information Tinfo supplied from the decoding unit 211, the value of the quantized transform coefficient levels level supplied from the decoding unit 211 scales (inverse quantization) to the dequantized to derive a conversion coefficient Coeff_IQ. The inverse quantization is an inverse process of the quantization performed by the quantization unit 113 of the image coding apparatus 100 (FIG. 7). Incidentally, the inverse quantization unit 115 (FIG. 7) performs the same inverse quantization and the inverse quantization unit 212. Inverse quantization unit 212 supplies the obtained transform coefficient Coeff_IQ the inverse transform unit 213.
[0154]
　Inverse transform unit 213, based on the conversion information Tinfo supplied from the decoding unit 211, inverse transform the transform coefficients Coeff_IQ supplied from the inverse quantization unit 212 derives a prediction residual D '. This inverse transform is the inverse process of the conversion processing performed by the conversion unit 112 of the image coding apparatus 100 (FIG. 7). Incidentally, the inverse transform unit 116 performs the same inverse transform and the inverse transform unit 213. Details of the reverse transformation will be described later. Inverse transform unit 213 supplies the obtained prediction residual D 'to the arithmetic unit 214.
[0155]
　Calculation unit 214, as shown in the following equation (24), adds the predicted image P (prediction signal) corresponding prediction residual D supplied from the inverse transform unit 213 'and its prediction residual D' in and derives a local decoded image Rec. Calculation unit 214, by using the obtained local decoded image Rec reconstructs the decoded image for each picture unit, and outputs the decoded image obtained on the outside of the image decoding apparatus 200. The arithmetic unit 214 also supplies the local decoded image Rec in the frame memory 215.
[0156]
　Rec＝Ｄ’＋Ｐ
　・・・（２４）
[0157]
　Frame memory 215, reconstructs the decoded image for each picture unit with a local decoded image Rec supplied from the arithmetic unit 214 is stored in a buffer in the frame memory 215. The frame memory 215 is read out from the buffer the decoded image is designated by the prediction mode information Pinfo prediction unit 216 as a reference image, and supplies the prediction unit 216. The frame memory 215, the header information Hinfo according to the generation of the decoded image, the prediction mode information pinfo, may be stored, such as in the buffer in the frame memory 215 converts information TINFO.
[0158]
　Prediction unit 216, is designated by the prediction mode information PInfo supplied from the decoding unit 211, it acquires the decoded image stored in the frame memory 215 as a reference image, using the reference image, by the prediction mode information Pinfo the prediction method is designated to generate the predicted image P. Prediction unit 216 supplies the generated predicted image P, the arithmetic unit 214.
[0159]
　In such an image decoding apparatus 200, the skip of the inverse transform processing on the image and the conversion coefficients prediction residual is obtained which is a difference between the predicted image of the image by inverse transformation processing, the transform coefficients of each sub-block based on the number of non-zero coefficients, so that a control unit for controlling each subblock. In other words, the inverse transform unit 213, the skip of the inverse transform processing on the image and the conversion coefficients prediction residual is obtained which is a difference between the predicted image of the image by inverse transformation processing, the transform coefficients of each sub-block non based on the number of zero coefficients, so as to control for each sub-block.
[0160]
　　
　FIG. 14 is a block diagram showing a main configuration example of the inverse transform unit 213 of FIG. 13. As shown in FIG. 14, the inverse transform unit 213 includes an inverse secondary transform unit 231 and the inverse primary transform unit 232,.
[0161]
　Inverse secondary transform unit 231, transform coefficients Coeff_IQ supplied from the inverse quantization unit 212, i.e., the decrypted encoded data is inverse quantized (also referred to as secondary transform coefficients) transform coefficients Coeff_IQ obtained by a one-dimensional into a vector, it performs a matrix operation for the one-dimensional vector, scale the one-dimensional vector that matrix operation is performed, the inverse secondary which is a conversion process that matrixes the one-dimensional vectors that scaled do the conversion.
[0162]
　Inverse secondary transform unit 231, and the secondary conversion identifier st_idx is information about the contents of the secondary conversion, based on the scan identifier scanIdx and is information about the scanning method of the transform coefficients, performs an inverse secondary transform to the secondary transform coefficients Coeff_IQ , to derive a transform coefficient Coeff_IS after inverse secondary transform (also primary transform coefficients referred to). In other words, the inverse secondary transform unit 231 converts the secondary transform coefficients Coeff_IQ primary transform coefficients Coeff_IS. Inverse secondary transform unit 231, the primary conversion coefficients Coeff_IS, supplies the inverse primary transform unit 232.
[0163]
　Incidentally, the inverse secondary transform unit 231, an inverse secondary transform is skipped (omitted), the secondary transform coefficients Coeff_IQ, as the primary transform coefficients Coeff_IS, can be supplied to the inverse primary transform unit 232. For details of the inverse secondary transform unit 231 will be described later.
[0164]
　Inverse primary transform unit 232, to the primary transform coefficients Coeff_IS supplied from the inverse secondary transform unit 231, for example, performs the inverse primary transform of the inverse orthogonal transform or the like, to derive the prediction residual D '. That is, the inverse primary transform unit 232 converts the primary transform coefficients Coeff_IS the prediction residual D '. Inverse primary transform unit 232 supplies the derived prediction residual D 'to the arithmetic unit 214.
[0165]
　Next, a description will be given inverse secondary transform unit 231. As shown in FIG. 14, the inverse secondary transform unit 231, a non-zero coefficient count determination unit 241, the switch 242, the rasterizing unit 243, the matrix calculating unit 244, a scaling unit 245, the matrix section 246, and the inverse secondary transform selector with a 247.
[0166]
　Non-zero coefficient count determination unit 241, an input of secondary transform coefficients Coeff_IQ sub block. Non-zero coefficient number determination section 241 refers to the secondary transform coefficients Coeff_IQ supplied from the inverse quantization unit 212, the number numSigInSBK of nonzero coefficients in the sub-block (also referred to as non-zero coefficient number of transform coefficients), for example, it derived as the following equation (25). In the equation (25), the secondary transform coefficients Coeff_IQ (i, j) (i, j) represents the coordinates of the sub-block, i = 0 ... 3, j = 0 ... 3. Also, operators abs (X) is an operator that returns the absolute value of the input value X.
[0167]
　numSigInSBK = Σ {abs (Coeff_IQ (i, j))> 0? 1:
　0} · · · (25)
[0168]
　Incidentally, without reference to secondary transform coefficients Coeff_IQ, based on the non-zero coefficients presence flag Sig_coeff_flag, may derive the non-zero coefficients number numSigInSBK of transform coefficients.
[0169]
　Then, the non-zero coefficient count determination unit 241, as non-zero coefficient number numSigInSBK of transform coefficients is equal to or less than a predetermined threshold value TH, as shown in the following equation (26), the determination result based on, to derive the secondary conversion skip flag StSkipFlag.
[0170]
　StSkipFlag = numSigInSBK <= TH ? 1 : 0
　・・・（２６）
[0171]
　The threshold value TH, for example, may be a 2, or may be any value from 0 to 16. The threshold TH may be notified from the outside (for example, an encoding side and the control side, etc.) in the header information such as VPS / SPS / PPS / slice header SH. The threshold TH is the encoding side (for example, an image coding apparatus 100) and the decoding side (e.g., the image decoding apparatus 200 to be described later) previously negotiated beforehand with the notification from the encoding side to the decoding side (threshold of TH, may be omitted transmission) from the encoding side to the decoding side.
[0172]
　In the formula (26), for example, the value of non-zero coefficients number numSigInSBK of transform coefficients may equal to or smaller than the threshold value TH, the value of the secondary transform skip flag StSkipFlag is set to 1. Further, for example, the value of non-zero coefficients number numSigInSBK of transform coefficients is greater than the threshold value TH, the value of the secondary transform skip flag StSkipFlag is set to 0.
[0173]
　Non-zero coefficient count determination unit 241 supplies the derived StSkipFlag the switch 242.
[0174]
　Switch 242 has an input secondary transform coefficients Coeff_IQ sub block, and a secondary conversion skip flag StSkipfFlag. Switch 242, in accordance with the secondary conversion skip flag StSkipFlag supplied from non-zero coefficient number determination unit 241, controls the skipping inverse secondary transform.
[0175]
　For example, if the value of the secondary transform skip flag StSkipFlag is 0, i.e., in the secondary conversion skip StSkipFlag, when the execution of the secondary transform is shown, switch 242, to perform the secondary transformation. That is, the switch 242 supplies the secondary transform coefficients Coeff_IQ supplied from the inverse quantization unit 212 to the rasterizing unit 243. For example, when the value of the secondary transform skip flag StSkipFlag is 1, i.e., in the secondary conversion skip StSkipFlag, if skipping inverse secondary transform is shown, switch 242 is skipped inverse secondary transform. That is, the switch 242, the secondary transform coefficients Coeff_IQ supplied from the inverse quantization unit 212 as the primary transform coefficients Coeff_IS, it supplies the inverse primary transform unit 232.
[0176]
　Rasterizing unit 243, based on the scan method of transform coefficients that is specified by the scan identifier scanIdx supplied from the decoding unit 211, for each sub-block (4 × 4 sub-blocks), the transform coefficients Coeff_IQ supplied from the switch 242 1 dimensional vector X 1d is converted to. Rasterizing unit 243, a one-dimensional vector X obtained 1d supplies the matrix calculator 244.
[0177]
　Inverse secondary transform selection portion 247 is supplied from the decoding unit 211, inverse secondary transform matrix IR specified by the secondary conversion identifier st_idx is information about the contents of the inverse secondary transform (= R T a), inverse secondary transform selection read from the internal memory (not shown) of section 247, and supplies the matrix calculator 244. For example, the inverse secondary transform selection portion 247, when the value of a secondary conversion identifier St_idx, as a matrix IR inverse secondary transform, transpose matrix R matrix R 16 × 16 as shown in FIG. 10 T reads, matrix it and supplies to the computing unit 244.
[0178]
　Incidentally, the inverse secondary transform selection portion 247, for example, secondary converter identifier st_idx and intra prediction mode information IPinfo supplied from the decoding unit 211 (e.g., intra prediction mode number) in accordance with the inverse secondary transform matrix IR (= R T ) may be selected. Further, instead of the intra prediction mode information IPinfo, according to the motion prediction information MVinfo and secondary conversion identifier St_idx, may be inverse transform IR is selected.
[0179]
　Matrix calculator 244, for each sub-block (4 × 4 sub-blocks), one-dimensional vector X 1d and inverse secondary transform matrix IR (= R T shown with a matrix of), 1d to derive. Here, the operator "·" represents an operation for performing an inner product of the matrix-matrix (matrix product) operator "T" represents the operation of the transposed matrix. Matrix calculator 244, a one-dimensional vector Y derived 1d supplies the scaling unit 245.
[0180]
　Y 1d T＝IR・X 1d T＝Ｒ Ｔ・X 1d T
　・・・（２７）
[0181]
　Scaling unit 245, for each sub-block (4 × 4 sub-blocks), one-dimensional vector Y supplied from the matrix calculator 244 1d to normalize the norm of, represented by the following formula (28) a N (N is a natural number) bit shift operations of a bit one-dimensional vector Y 1d performed for all the elements of the one-dimensional vector Z after bit shifting 1d seek.
[0182]
　Z 1d＝（Y 1d）>>Ｎ
　・・・（２８）
[0183]
　Incidentally, as shown in the following equation (29), before the shift operation of N bits as an offset, the value of 1 << (N-1), 1-dimensional vector Z 1d is added to each element of it may be so. In the equation (29), the vector E is the value of all elements is one-dimensional vector of 1.
[0184]
　Z 1d＝（Y 1d＋（（N-1）<<1)・E）>>Ｎ
　・・・（２９）
[0185]
　For example, the inverse secondary transform matrix IR (= R T is a matrix of), Generally, the inverse secondary transform matrix IR (= R T ) is, if it is N bits scaling, bit shift amount of the norm normalization is N bits. Scaling unit 245, a one-dimensional vector Z after norm normalization thus obtained 1d supplies the matrixing unit 246.
[0186]
　Matrixing unit 246, for each sub-block (4 × 4 sub-blocks), one-dimensional vector Z after norm normalization 1d as inputs and scan identifier ScanIdx, designated by the scan identifier ScanIdx supplied from the decoding unit 211 that based on the scan method, one-dimensional vector Z supplied from the scaling unit 245 1d converts the primary transform coefficients Coeff_IS of 4 × 4 matrix. Matrixing unit 246 supplies the primary transform coefficients Coeff_IS obtained in inverse primary transform unit 232.
[0187]
　As described above, the non-zero coefficient count determination unit 241 based on the number of non-zero coefficients for each sub-block, setting the secondary conversion skip flag StSkipFlag, switch 242, based on the secondary conversion skip flag StSkipFlag, secondary to control the skip of the conversion. By doing so, since the non-zero coefficients becomes possible to skip the secondary transform for sparse subblock, suppressing a decrease in energy compaction, it is possible to suppress the reduction of the coding efficiency.
[0188]
　　
　Next, a flow of each process executed by the image decoding apparatus 200 will be described. First, with reference to the flowchart of FIG. 15, an example of a flow of the image decoding processing.
[0189]
　When the image decoding process is started, in step S201, the decoding unit 211 decodes the bit stream supplied to the image decoding apparatus 200 (coded data), the header information Hinfo, prediction mode information pinfo, conversion information TINFO, residual information rinfo, obtain information such as quantized transform coefficient levels level.
[0190]
　In step S202, the inverse quantization unit 212, and the transform coefficients Coeff_IQ inversely quantizes the quantized transform coefficient levels level obtained by the processing in step S201. The inverse quantization is the inverse process of the quantization performed in step of the image encoding process S105 (FIG. 11), the same processing as the inverse quantization performed in step of the image encoding process S106 (FIG. 11) is there.
[0191]
　In step S203, the inverse transform unit 213 inversely transforms the transform coefficients Coeff_IQ obtained by the processing in step S202, to derive a prediction residual D '. This inverse transform is the inverse process of the conversion process performed in the image encoding process step S104 (FIG. 11), the inverse transform processing similar to that performed in step of the image encoding process S107 (Figure 11).
[0192]
　In step S204, the prediction unit 216, based on the prediction mode information pinfo, making predictions in the prediction and the same prediction mode for encoding, to the predicted image generation.
[0193]
　In step S205, the arithmetic unit 214 processes the prediction residual D 'obtained by the step S203, it adds the predicted image obtained by the processing in step S204, obtaining a decoded image.
[0194]
　When the process of step S205 is completed, the image decoding process is terminated.
[0195]
　　
　Next, an example of the inverse conversion processing flow executed in step S203 of FIG. 15 will be described with reference to the flowchart of FIG. 16.
[0196]
　When inverse transform processing is started, in step S221, the inverse secondary transform unit 231 (the switch 242) determines whether the secondary conversion identifier St_idx applies the inverse secondary transform (st_idx> 0). If the secondary transform identifier st_idx is determined to be 0 (indicating the skip of the secondary transform identifier st_idx inverse secondary transform), inverse secondary transform (processing of steps S222 to step S230) is skipped and the process proceeds to step S231 . In other words, the inverse secondary transform unit 231 (the switch 242), the secondary transform coefficients Coeff_IQ obtained by the processing in step S202 in FIG. 15 as the primary transform coefficients Coeff_IS, it supplies the inverse primary transform unit 232.
[0197]
　Further, in step S221, if the secondary conversion identifier st_idx is determined to be greater than 0 (indicating execution of a secondary conversion identifier st_idx inverse secondary transform), the process proceeds to step S222.
[0198]
　In step S222, the inverse secondary transform selection unit 247 selects the matrix IR inverse secondary transform that is specified by the secondary conversion identifier St_idx.
[0199]
　In step S223, the inverse secondary transform unit 231 selects the unprocessed sub-blocks included in the conversion block to be processed.
[0200]
　In step S224, the non-zero coefficient count determination unit 241, as described above, we derive the non-zero coefficients number numSigInSBK of transform coefficients on the basis of the secondary transform coefficients Coeff_IQ sub block obtained by the processing in step S202 in FIG. 15 and further derives the secondary conversion skip flag StSkipFlag with non-zero coefficients number numSigInSBK and threshold TH of the conversion coefficients.
[0201]
　In step S225, the switch 242, the secondary transform skip flag StSkipFlag obtained by the processing in step S224 determines whether indicating the skip inverse secondary transform. Showing the execution of a secondary conversion, i.e., when the value of the secondary transform skip flag StSkipFlag is determined to be 0, the process proceeds to step S226.
[0202]
　In step S226, the rasterization unit 243, based on the scan method that is specified by the scan identifier ScanIdx, the secondary transform coefficients Coeff_IQ obtained by the processing in step S202 of FIG. 15 1-dimensional vectors X 1d is converted to.
[0203]
　In step S227, the matrix computing unit 244, the one-dimensional vector X 1d calculates and, the matrix product of the matrix IR inverse secondary transform obtained by the processing in step S222, one-dimensional vector Y 1d seek.
[0204]
　In step S228, the scaling unit 245, the one-dimensional vector Y 1d the norm of normalized one-dimensional vector Z 1d seek.
[0205]
　In step S229, the matrix unit 246, based on the scan method that is specified by the scan identifier ScanIdx, the one-dimensional vector Z 1d converts the matrix of 4 × 4, the primary transform coefficients to be processed subblocks Coeff_IS the seek. When the processing of step S229 is completed, the process proceeds to step S230. Further, in step S225, indicating the skip inverse secondary transform, i.e., when the value of the secondary transform skip flag StSkipFlag is 1, the process proceeds to step S230.
[0206]
　In step S230, the inverse secondary transform unit 231 determines whether or not processing of all of the sub-blocks of transform blocks processed. If unprocessed sub-block is determined to exist, processing returns to step S223, and the process thereafter is repeated. That is, for each sub-block of transform blocks processed, the processing in steps S223 to step S230 (the inverse secondary transform) is performed. In step S230, the case where it is determined that the processing of all of the sub-blocks (a Been executed or skipped inverse secondary transform of all sub blocks), the process proceeds to step S231.
[0207]
　In step S231, the inverse primary transform unit 232, based on the primary transformation identifier Pt_idx, performs inverse primary transform to the primary transform coefficients Coeff_IS, derives the prediction residual D '. The prediction residual D 'is supplied to the arithmetic operation unit 214.
[0208]
　When the process of step S231 is completed, the inverse transform process is completed, the process returns to FIG. 15.
[0209]
　Incidentally, the inverse conversion processing described above, in feasible range, the processing order and replacement steps, may change the contents of processing. For example, if the secondary conversion identifier st_idx is determined to be 0 in step S221, 16 matrix of × 16 is selected as a matrix IR inverse secondary transform, as the processing in steps S222 through step S230 is executed it may be.
[0210]
　By executing the processes as described above, a sub-block unit, it is possible to control the inverse secondary transform skip (execution). Therefore, the non-zero coefficients with respect to sparse residual signal, it is possible to suppress the reduction of the energy compaction. That is, it is possible to suppress the reduction of the coding efficiency. In other words, while suppressing a decrease in encoding efficiency, it is possible to suppress the increase of the load of decoding (inverse secondary transform).
[0211]
　In the foregoing is directed to skip (inverse) secondary transform to control for each sub-block, control skips performed for each sub-block, not only (inverse) secondary transform to any conversion process it is possible to apply.
[0212]
　<2. Second Embodiment>
　　Selection of Secondary conversion using
　Now, even in the method according to any of the Non-Patent Documents 1 and 2, the secondary transform is the number of classes intra prediction mode and the the number of secondary transformation corresponding to the class, and had a matrix of secondary conversion. Therefore, huge memory size to hold a matrix of secondary conversion was necessary. For example, in the case of the method described in Non-Patent Document 1, a number of classes intra prediction mode 12, the number of secondary conversion for each class is 3, 12 * 3 = 36 matrix were present. Further, in the case of the method described in Non-Patent Document 2, a number of classes intra prediction mode 35, the number of secondary transform for each class = is 5, was present matrix 35 * 5 = 175 .
[0213]
　Thus, for example, when holding the elements of each matrix in 9bit accuracy, in the case of the method described in Non-Patent Document 1, so that the required memory size as shown in the following equation (30). Further, in the case of the method described in Non-Patent Document 2, so that the required memory size as shown in the following equation (31).
[0214]
　Memory size 9Bit * 16 = * 16 * 36 = 829944 (Bits) = 10368 (Bytes) = 10.125 (KB)
　· · · (30)
　memory size = 9bit * 16 * 16 * 175 = 403200 (bits) = 50400 (bytes ) = 49.21875
　(KB) · · · (31)
[0215]
　Thus, the data amount of retention to (reverse) of the secondary transformation matrix is ​​increased, there is a possibility that the load of encoding and decoding increases. Also, since the increase memory size required, there is a possibility that the cost is increased.
[0216]
　Therefore, the matrix IR matrix R and the inverse secondary transform the secondary conversion, so as to set based on the secondary conversion identifier and scan identifier. That is, the direction of the intra prediction mode, focusing on the point where the scanning method correspond replaces the classification of the intra prediction mode, to scan the identifier is information about the scanning method (scanIdx). Figure 17 shows an example of correspondence between the intra prediction mode and the scan identifier (scanIdx) is shown.
[0217]
　Secondary transformation corresponding to each value of the scan identifier (scanIdx) is five. This is because assigning each secondary conversion identifier (st_idx). Therefore, in this case, the total number of secondary conversion becomes 3 × 5 = 15. In other words, it is possible to reduce the number of secondary conversion compared to the method described in Non-Patent Document 1 and Non-Patent Document 2 described above. When the 9bit precision, the memory size required to hold the entire secondary conversion is given by the following equation (32).
[0218]
　Memory size 9Bit * 16 = * 16 = 15 = 34560 (Bits) = 4320 (Bytes) = 4.21875
　(KB) · · · (32)
[0219]
　Therefore, as compared with the case of the aforementioned formula (30) or (31) (case of the method described in Non-Patent Document 1 and Non-Patent Document 2), it is possible to greatly reduce the data amount of the matrix of secondary conversion . This suppresses the increase of the load of encoding and decoding, it is possible to suppress the increase of the memory size required to hold the (inverse) of the secondary transformation matrix.
[0220]
　　
　image encoding apparatus 100 also in this case has basically the same configuration as in the first embodiment. However, the image coding apparatus 100 in this case, the matrix of the transformation process for the transform coefficients, the difference between the setting unit that sets, based on the content and the scanning method of the conversion process, the image and the predicted image of the image using a rasterizing unit for converting the transform coefficients in the prediction residuals are obtained are converted processed into a one-dimensional vector, the set matrix by the setting unit, a matrix operation unit for performing matrix operation with respect to the one-dimensional vector When provided with a scaling unit for scaling against one-dimensional vector that matrix operation is performed, a matrix unit for a matrix of the one-dimensional vectors that scaling. In other words, converter 112 in this case, the matrix of the transformation process for the conversion coefficient, and set on the basis of the content and the scanning method of the conversion process, the prediction residual being the difference image and the predicted image of the image the transform coefficients obtained are converted processed into a one-dimensional vector, by using the set matrix performs matrix operation with respect to the one-dimensional vector, with respect to a one-dimensional vector the matrix calculation is performed It performs scaling Te, so as to matrixes the one-dimensional vectors that scaling.
[0221]
　Figure 18 is a block diagram showing a main configuration example of a conversion unit 112 in this case. As shown in FIG. 18, also converting unit 112 in this case has the same configuration as that of basically the first embodiment (FIG. 8). However, the secondary conversion unit 132 in this case, it is possible to omit the quantization unit 146 to the switch 148, also has a secondary change selecting section 301 in place of the secondary change selecting section 145.
[0222]
　Secondary change selecting section 301, an input secondary conversion identifier St_idx, and a scan identifier ScanIdx. Secondary change selecting section 301, based on the secondary conversion identifier st_idx and scan identifier scanIdx input, select the matrix R of the secondary transform is supplied to a matrix operation unit 142.
[0223]
　　
　FIG. 19 is a block diagram showing a main configuration example of a secondary change selecting section 301. As shown in FIG. 19, the secondary change selecting section 301 has a secondary conversion derivation unit 311 and the secondary conversion holder 312.
[0224]
　Secondary transformation deriving unit 311 has an input of secondary conversion identifier st_idx and scan identifier ScanIdx. Secondary transformation derivation unit 311, based on the secondary conversion identifier st_idx and scan identifier scanIdx input, from the secondary transformation matrix table LIST_FwdST stored in the secondary converter holder 312 [] [], the appropriate secondary transformation matrix R reading as the following equation (33), and outputs to the outside. Here, the inverse secondary transform matrix table LIST_FwdST [] [], the matrix R of the secondary transform is stored corresponding to each scan identifier scanIdx and secondary conversion identifier St_idx.
[0225]
　R = LIST_FwdST[ scanIdx ][ st_idx ]
　・・・（３３）
[0226]
　Secondary transformation holder 312 holds the scan identifier scanIdx and secondary conversion identifier st_idx secondary transformation matrix table LIST_FwdST to matrix R of the secondary transform is stored corresponding to each [] []. Based on an instruction of the secondary converter deriving unit 311, the appropriate supplying matrix R of the secondary transform to the secondary converter deriving unit 311.
[0227]
　　
　Next, an example of the flow of each process executed by the image encoding apparatus 100. In this case the image encoding apparatus 100, the image encoding processing basically similarly performed as in the first embodiment (FIG. 11). An example of conversion processing flow in this case will be described with reference to the flowchart of FIG. 20.
[0228]
　When the conversion process is started, the processing in steps S301 and S302 are performed in the same manner as the processes of steps S121 and step S122 in FIG. 12. That is, if it is determined that the secondary conversion identifier st_idx is zero (indicating a skip secondary conversion), a secondary conversion (processing of step S303 through step S309) is skipped, the conversion process is completed, the process in FIG. 11 Return. That is, the secondary conversion unit 132 supplies the quantization unit 113 to the primary transform coefficients Coeff_P as the transformation coefficient Coeff.
[0229]
　Further, in step S302, if it is determined that the larger the secondary conversion identifier st_idx is 0 (indicating the execution of the secondary conversion), the process proceeds to step S303.
[0230]
　In step S303, the secondary change selecting section 301 selects the matrix R of the secondary transform that corresponds to the secondary conversion identifier st_idx scan identifier ScanIdx. That is, the secondary conversion derivation unit 311 selects reads the matrix R of the secondary transformation corresponding than secondary transformation matrix table stored in the secondary converter holder 312 and the secondary conversion identifier st_idx scan identifier ScanIdx.
[0231]
　The processes of steps S304 to step S309, the step S124 through step S128 in FIG. 12, as well, it is performed in the same manner as the processing in step S134. That is, by the processing of steps S304 through step S309 is performed for each sub-block, the secondary transform is performed for each sub-block. Then, in step S309, the case where it is determined that the processing of all the sub-blocks, the conversion process is completed, the process returns to FIG. 11.
[0232]
　The conversion process is a feasible range, the processing order and replacement steps, may change the contents of processing. For example, if it is determined that the secondary conversion identifier st_idx = 0 in step S302, 16 matrix of × 16 is selected as the matrix R of the secondary conversion, as the processing in steps S304 through step S309 is executed it may be.
[0233]
　By executing the processes as described above, it is possible to select the matrix R of the secondary conversion based on secondary conversion identifier st_idx and scan identifier ScanIdx. Therefore, it is possible to greatly reduce the data amount of the matrix of the secondary conversion. This suppresses the increase in the load of the encoding, it is possible to suppress the increase of the memory size required to hold the matrix of the secondary conversion.

claims
[Requested item 1]
　A primary transform is a transformation process for prediction residual is the difference between the predicted image of the image with the image, the prediction residual is skipped secondary conversion and a conversion process for the primary transform coefficients obtained are the primary conversion case, the control unit the band limited to skip to the secondary transformation coefficients the primary transform coefficients obtained are the secondary conversion
　image processing apparatus comprising a.
[Requested item 2]
　The image processing apparatus,
　a converting process with respect to difference and the primary transform is a transformation process for prediction residual is, the primary transform coefficients the prediction residual is obtained is the primary conversion of the prediction image of the image with the image secondary If to skip the conversion and control steps of band-limited to skip to the secondary transformation coefficients the primary transform coefficients obtained are the secondary conversion
　image processing method comprising.
[Requested item 3]
　And inverse primary transform is an inverse transformation of the primary transformation is a transformation process for prediction residual is the difference between the predicted image of the image and the image conversion process with respect to the primary transform coefficients the prediction residual is obtained is the primary conversion case of skipping the inverse secondary transform that is the inverse of the secondary transform is a control unit which also skipped band limits for secondary transform coefficients obtained by the primary transform coefficients are the secondary converted by band-limited
　image comprising processing apparatus.
[Requested item 4]
　Wherein, when the not Skip inverse primary transform, when the transform block size of the inverse primary transform and the inverse secondary transform is equal to or greater than a predetermined size, to perform band limitation to said secondary transform coefficients
　is adapted and
　image processing apparatus according to claim 3.
[Requested item 5]
　Wherein, when the not Skip inverse primary transform, based on the horizontal size and vertical size of the transform block of the inverse primary transform and the inverse secondary transform to perform a band limitation for the secondary transform coefficients
　configured to
　image processing apparatus according to claim 4.
[Requested item 6]
　Wherein, when the not Skip inverse primary transform, when the larger of the horizontal and vertical size of the transform block is equal to or greater than a predetermined value, to perform band limitation to said secondary transform coefficients
　as configured
　image processing apparatus according to claim 5.
[Requested item 7]
　Wherein, if not Skip the inverse primary transform, when the sum or the product of the horizontal and vertical size of the transform block is equal to or greater than a predetermined value, to perform band limitation to said secondary transform coefficients
　configured has been
　image processing apparatus according to claim 5.
[Requested item 8]
　The band limitation, the secondary transform coefficients after the band limitation of a predetermined region other than the square area of the size of the inverse primary transform and the rectangular conversion block of the inverse secondary transform is performed by the 0
　to configured
　image processing apparatus according to claim 4.
[Requested item 9]
　The band limitation is row by one of the pixels constituting the transform block of the inverse primary transform and the inverse secondary transform, the secondary transform coefficients after the band limitation of the pixel processing order is a predetermined value or more to 0 dividing
　configured to
　image processing apparatus according to claim 4.
[Requested item 10]
　The image processing apparatus,
　the inverse primary transform is an inverse transformation of the primary transformation is a transformation process for prediction residual is the difference between the predicted image of the image with the image, primary said prediction residual is obtained is the primary conversion case of skipping the inverse secondary transform that is a secondary transform of the inverse transform is a transformation process on the transform coefficients, the band also to skip to the secondary transform coefficients obtained by the primary transform coefficients are the secondary converted by band-limited control step
　image processing method comprising.
[Requested item 11]
　Skip inverse transform processing for transform coefficient prediction residual is obtained which is a difference between the predicted image of the image with the image by the inverse transformation processing, based on the number of nonzero coefficients of the transform coefficients for each sub-block, control unit for controlling each said sub-block
　image processing apparatus comprising a.
[Requested item 12]
　The control unit may skip the inverse secondary transform to the Secondary transform coefficients encoded data is obtained by being decoded, based on the number of non-zero coefficients of the transform coefficients for each of the sub-blocks is controlled for each of the sub-blocks
　the image processing apparatus according to claim 11.
[Requested item 13]
　On the basis of the secondary transform coefficients, whether to skip the inverse secondary transform, further comprising a determination unit for each sub-block,
　the control unit, according to the result of determination by the determination unit, wherein skipping inverse secondary transform configured to control for each of the sub-block
　image processing apparatus according to claim 12.
[Requested item 14]
　The inverse secondary transform may
　　convert the secondary transform coefficients into a one-dimensional vector,
　　the performs matrix operation on one-dimensional vector,
　　performs a scaling of the one-dimensional vector the matrix operation is performed,
　　the scaled matrices of the one-dimensional vector
　is converted processing
　image processing apparatus according to claim 12.
[Requested item 15]
　Skip inverse transform processing for transform coefficient prediction residual is obtained which is a difference between the predicted image of the image with the image by the inverse transformation processing, based on the number of nonzero coefficients of the transform coefficients for each sub-block, controlling for each of the sub-block
　image processing method.
[Requested item 16]
　The matrix of inverse transform processing for transform coefficient, and a setting unit that sets, based on the content and the scan method of the inverse transformation processing,
　the prediction residual being the difference between the predicted image of the image with the image by reverse conversion a rasterizing unit for converting the transform coefficients are obtained in a one-dimensional vector,
　by using the matrix set by the setting unit, and a matrix calculator for performing matrix operation with respect to the one-dimensional vector,
　the matrix calculation row a scaling unit for scaling to the one-dimensional vector cracking,
　a matrix unit for a matrix of said one-dimensional vector scaled
　image processing apparatus comprising a.
[Requested item 17]
　Further comprising a storage unit for storing the candidates of the matrix,
　the setting unit, the matrix corresponding to the content and the scan method of the inverse transformation processing, from among the candidates for the matrix stored in the storage unit by selecting is configured to set the matrix
　image processing apparatus according to claim 16.
[Requested item 18]
　The setting unit sets the matrix corresponding to the contents of the inverse transform process, and the scan method used in the rasterizing unit and the matrix section
　image processing apparatus according to claim 16.
[Requested item 19]
　The setting unit sets the matrix based conversion identifier indicating the contents of the inverse transform process, to the scan identifier is information about the scanning method
　The image processing apparatus according to claim 16.
[Requested item 20]
　The matrix of inverse transform processing for transform coefficient, and set on the basis of the content and the scan method of the inverse transformation processing,
　the prediction residual being the difference between the prediction image of the image the image is obtained by inverse transformation processing converts the transform coefficient into a one-dimensional vector,
　by using the set the matrix, performs a matrix operation on the one-dimensional vector,
　performs scaling with respect to the one-dimensional vector the matrix operation is performed,
　matrices of the scaled the one-dimensional vector
　image processing method.