Title of the invention: Image processing apparatus and image processing method
Technical field
[0001]
　The present technology relates to an image processing apparatus and an image processing method, and more particularly to an image processing apparatus and an image processing method that enable various DFs (deblock filters) to be provided.
Background technology
[0002]
　In H.265 / HEVC, which is one of the standard specifications of the image coding method, a deblock filter is applied to the block boundary of the decoded image in order to suppress the deterioration of image quality due to the block distortion that occurs during coding. Will be done. In H.265 / HEVC, there are two types of deblocking filters that can be applied to the luminance component, a weak filter and a strong filter, while only one type of deblocking filter that can be applied to the color difference component is a weak filter.
[0003]
　In addition, JVET (Joint Video Experts Team), a joint standardization organization of ITU-T and ISO / IEC, is currently working on the next generation with the aim of further improving coding efficiency over H.265 / HEVC. Standardization work of VVC (Versatile Video Coding), which is an image coding method, is underway (see, for example, Non-Patent Document 1).
[0004]
　In the standardization work of VVC, in Non-Patent Document 1 below, the deblocking filter that can be applied to the color difference component is changed to two types like the deblocking filter that can be applied to the luminance component, and the strong filter is also applied to the color difference component. Has been proposed as a method to which.
Prior art literature
Non-patent literature
[0005]
Non-Patent Document 1: Jianle Chen, Yan Ye, Seung Hwan Kim: Algorithm description for Versatile Video Coding and Test Model 2 (VTM 2), Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29/WG 11 11th Meeting, Ljubljana, SI, 10-18 July 2018.
Outline of the invention
Problems to be solved by the invention
[0006]
　Regarding DF, it is required to provide various DF.
[0007]
　This technology was made in view of such a situation, and makes it possible to provide various DFs.
Means to solve problems
[0008]
　The first image processing apparatus of the present technology includes a decoding unit that decodes a bit stream to generate a decoded image and pixels having a color difference component located near the block boundary of the decoded image decoded by the decoding unit. On the other hand, the reduced second luminance filter in which the filter characteristics of the second luminance filter having a stronger filter strength than the first luminance filter or the reduced first luminance filter in which the filter characteristics of the first luminance filter are reduced are used. This is an image processing apparatus including a filter unit that is applied as a second luminance filter having a stronger filter intensity than the one luminance filter.
[0009]
　The first image processing method of the present technology is to generate a decoded image by decoding a bit stream, and for pixels having a color difference component located near the block boundary of the decoded image decoded by the decoding unit. Therefore, the reduced first luminance filter having a reduced filter characteristic of the second luminance filter having a stronger filter strength than the first luminance filter or the reduced first luminance filter having the reduced filter characteristic of the first luminance filter is used. This is an image processing method including applying the filter as a second luminance filter having a stronger filter intensity than the luminance filter.
[0010]
　In the first image processing apparatus and image processing method of the present technology, the bit stream is decoded to generate a decoded image. Then, the reduced second luminance filter or the first luminance filter in which the filter characteristics of the second luminance filter having a stronger filter intensity than the first luminance filter are reduced with respect to the pixels of the color difference component located near the block boundary of the decoded image. The reduced first luminance filter in which the filter characteristics of the first luminance filter are reduced is applied as the second luminance filter having a stronger filter strength than the first luminance filter.
[0011]
　The second image processing apparatus of the present technology filters the pixels of the color difference component located near the block boundary of the locally decoded image that has been locally decoded when the image is encoded, rather than the first luminance filter. The reduced first luminance filter with reduced filter characteristics of the second luminance filter with strong intensity or the reduced first luminance filter with reduced filter characteristics of the first luminance filter has a stronger filter intensity than the first luminance filter. It is an image processing apparatus including a filter unit to be applied as a two-luminance filter, and a coding unit for encoding the image using the locally decoded image to which the second color difference filter is applied by the filter unit. ..
[0012]
　The second image processing method of the present technology filters the pixels of the color difference component located near the block boundary of the locally decoded image that has been locally decoded when the image is encoded, rather than the first luminance filter. The reduced first luminance filter with reduced filter characteristics of the second luminance filter with strong intensity or the reduced first luminance filter with reduced filter characteristics of the first luminance filter has a stronger filter intensity than the first luminance filter. It is an image processing method including application as a two-color difference filter and coding processing of the image using the locally decoded image to which the second color difference filter is applied.
[0013]
　In the second image processing apparatus and image processing method of the present technology, the first pixel of the color difference component located near the block boundary of the locally decoded image that has been locally decoded when the image is encoded is first. The filter characteristics of the second luminance filter, which has stronger filter strength than the luminance filter, are reduced. The reduced second luminance filter or the reduced first luminance filter, which has reduced filter characteristics of the first luminance filter, is more than the first luminance filter. It is applied as a second luminance filter with strong filter intensity. Then, the image is encoded using the locally decoded image to which the second color difference filter is applied.
[0014]
　The image processing device can be realized by causing a computer to execute a program. The program can be provided by recording on a recording medium or by transmitting through a transmission medium.
A brief description of the drawing
[0015]
FIG. 1 is a table for explaining the calculation of bS in HEVC.
FIG. 2 is a table for explaining the calculation of bS in Non-Patent Document 1.
FIG. 3 is an explanatory diagram showing an example of pixels of color difference components (U component and V component) in two blocks Bp and block Bq adjacent to each other with a vertical block boundary BB in between.
FIG. 4 is a table for explaining the calculation of bS in one embodiment of the present disclosure.
FIG. 5 is a block diagram showing an example of a configuration of an image coding device 10 which is an aspect of the image processing device according to the same embodiment.
FIG. 6 is a block diagram showing an example of a configuration of an image decoding device 60, which is an aspect of the image processing device according to the same embodiment.
FIG. 7 is a block diagram showing an example of a detailed configuration of the deblock filter 26 according to the same embodiment.
FIG. 8 is a table showing an example of bS calculated by the boundary strength calculation unit 261.
FIG. 9 is a flowchart showing an example of a processing flow by the deblock filter 26 according to the same embodiment.
FIG. 10 is a flowchart for explaining a flow of a boundary strength calculation process executed by the boundary strength calculation unit 261.
FIG. 11 is a block diagram showing a configuration example of a DF300 as a new DF.
FIG. 12 is a diagram showing a configuration example of a decoded image processed by the DF 300.
FIG. 13 is a flowchart illustrating processing of the DF300.
[Fig. 14] Fig. 14 is a diagram illustrating DF of HEVC.
[Fig. 15] Fig. 15 is a diagram illustrating the new DF.
FIG. 16 is a diagram showing an example of pixels of a color difference component at a block boundary.
FIG. 17 is a diagram showing a filter NC1 and required pixels when a filter based on the filter Y1 is adopted as the filter NC1.
FIG. 18 is a diagram showing a filter NC1 and required pixels when a filter based on the filter OF is adopted as the filter NC1.
FIG. 19 is a diagram showing a filter NC1 and required pixels when a filter based on the filter Y2 is adopted as the filter NC1.
FIG. 20 is a diagram showing an example of an application method of applying the filter NC1 to a decoded image.
FIG. 21 is a diagram showing an example of an application method of applying the filter NC1 to a decoded image.
FIG. 22 is a block diagram showing a configuration example of an embodiment of a computer.
Mode for carrying out the invention
[0016]
　Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.
[0017]
　Further, the scope disclosed in the present specification is not limited to the contents of the examples, and the contents of the following references REF1 to REF3 known at the time of filing are also incorporated in the present specification by reference. Is done. In other words, the contents described in the following references REF1 to REF3 are also the basis for judging the support requirements. For example, even if the Quad-Tree Block Structure described in Reference REF2 and the QTBT (Quad Tree Plus Binary Tree) Block Structure described in Reference REF3 are not directly defined in the detailed description of the invention. , Within the scope of this disclosure and shall meet the support requirements of the scope of the claim. Similarly, technical terms such as Parsing, Syntax, and Semantics are also within the scope of the present disclosure, even if they are not directly defined in the detailed description of the invention. Yes, and shall meet the support requirements of the claims.
　REF1: Recommendation ITU-T H.264 (04/2017) “Advanced video coding for generic audiovisual services”, April 2017
　REF2: Recommendation ITU-T H.265, (12/2016) “High efficiency video coding”, December 2016
　REF3: J. Chen, E. Alshina, GJ Sullivan, J.-R. Ohm, J. Boyce, "Algorithm Description of Joint Exploration Test Model (JEM7)", JVET-G1001, Joint Video Exploration Team (JVET) of ITU －T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29/WG 11 7th Meeting: Torino, IT, 13-21 July 2017
[0018]
　Further, in the following, unless otherwise specified, a signal in the YUV420 format will be described as an example, and the luminance component may be represented as the Y component, the color difference component may be represented as the U component, and the V component. However, the technique described below can be similarly applied to signals in other formats such as the YUV444 format and the YUV422 format. Further, the representation of the luminance component and the color difference component is different depending on the target signal. For example, the technique described below is similarly applied to a signal in which the luminance component and the color difference component are represented by YCbCr. It is possible to do.
[0019]
　In addition, the following terms used in the present specification are defined as follows.
　The color difference-related parameters mean all the parameters related to the color difference. For example, the color difference-related parameters include the conversion coefficient of the color difference component included in each TU (Transform Unit) and the conversion coefficient of the color difference component such as a flag indicating the presence or absence of the significance coefficient (non-zero conversion coefficient) of the color difference component in each TU. May contain information about. However, the color difference-related parameters are not limited to these examples, and may be various parameters related to the color difference.
　The necessity of applying the deblock filter means whether or not the deblock filter should be applied. For example, determining whether or not a deblocking filter should be applied means determining whether or not a deblocking filter should be applied. Further, the determination result of whether or not the deblock filter is applied is the result of determining whether or not the deblock filter should be applied. For example, the determination result should be applied or should not be applied. It can be information indicating.
　The large block determination means a determination as to whether or not the block to be determined is a large block. In the present specification, the block to be determined may be a block sandwiching a block boundary, as will be described later. Further, the large block determination can be performed by comparing the block size (block size) with a predetermined threshold value. The case where the large block determination is performed and the details of the large block determination will be described later.
[0020]
　<1. Overview>
　　[1-1. Existing method]
[0021]
　The processing related to the deblock filter in the existing image coding method such as HEVC includes the application necessity determination processing, the filter strength determination processing, and the filtering processing (filter application processing). In the following, the processing applied to the existing deblocking filter will be described using the HEVC deblocking filter as an example. In the following, the deblocking filter for the color difference component of the decoded image (including the image locally decoded at the time of encoding) will be mainly described, and the description of the deblocking filter for the luminance component of the decoded image will be omitted as appropriate.
[0022]
　As the process related to the deblock filter, the application necessity determination process is first performed. The application necessity determination process is a process of determining whether or not a deblock filter should be applied to the block boundary (Block Boundary) of the decoded image. In HEVC, the block boundary is specified based on the block structure of Quad-Tree Block Structure described in Reference REF2. Specifically, the condition that the edge of the 8 × 8 pixel block (sample grid), which is the smallest block unit, is at least one of the TU (Transform Unit) boundary and the PU (Prediction Unit) boundary is satisfied. Edges are identified as block boundaries in HEVC.
[0023]
　The application necessity determination process is performed based on the boundary strength of the block boundary (Boundary Strength: hereinafter may be referred to as bS). In HEVC, bS is calculated every 4 lines of the specified block boundary. When the block boundary is a vertical boundary, the above line corresponds to a line orthogonal to the vertical boundary. When the block boundary is a horizontal boundary, the line corresponds to a column orthogonal to the horizontal boundary.
[0024]
　FIG. 1 is a table for explaining the calculation of bS in HEVC. As shown in FIG. 1, in HEVC, bS is true of condition A which is a condition related to intra prediction, condition B1 which is a condition related to the significance coefficient of the Y component, and condition B2 which is a condition related to a motion vector (MV) and a reference picture. Calculated based on falsehood (satisfied or unsatisfied). Referring to FIG. 1, bS is set to 2 when condition A is true. Further, bS is set to 1 when the condition A is false and at least one of the condition B1 and the condition B2 is true. Then, when the condition A, the condition B1, and the condition B2 are all false, bS is set to 0. The conditions A, B1 and B2 shown in FIG. 1 are as follows.
[0025]
　-Condition A: At least one of the CUs (Coding Units) including the pixels of the uppermost line of the bS calculation target lines and sandwiching the block boundary is the intra prediction mode.-
　Condition B1: Block The boundary is the TU boundary, and at least one of the two TUs including the pixel of the uppermost line of the bS calculation target lines and sandwiching the block boundary has a significance coefficient of the Y component-
　condition B2: bS. The absolute value of the difference in MV is 1 pixel or more, or the reference picture of motion compensation is different, or the number of MVs between two CUs including the pixel of the top line of the calculation target lines and sandwiching the block boundary. Is different
[0026]
　Further, in HEVC, a deblocking filter for the luminance component (Y component) of the decoded image can be applied to the block boundary where the bS set as described above is 1 or more. Therefore, in HEVC, the determination result of whether or not the deblock filter should be applied to the luminance component of the decoded image may differ depending on whether or not the conditions B1 and B2 are satisfied.
[0027]
　In HEVC, as a deblocking filter for the luminance component of the decoded image, a strong filter having a large filter strength and a weak filter having a low filter strength are prepared. When bS is 1 or more, the process relating to the deblocking filter for the luminance component of the decoded image is followed by a filter intensity determination process and a filtering process after further application necessity determination process based on further conditions is performed. Details of these processes are described in the above-mentioned reference REF2, and the description thereof is omitted here.
[0028]
　On the other hand, the deblock filter for the color difference components (U component, V component) of the decoded image in HEVC is applied only to the block boundary where bS is 2. Therefore, as shown in FIG. 1, whether or not the conditions B1 and B2 are satisfied does not affect the determination of whether or not the deblock filter is applied to the color difference component of the decoded image in HEVC.
[0029]
　Further, in HEVC, the only deblocking filter that can be applied to the color difference component of the decoded image is the weak filter. Therefore, it is not necessary to determine the filter intensity for the color difference component of the decoded image, and when bS is 2, a weak filter is applied to the color difference component of the decoded image.
[0030]
　By the way, as described in the above-mentioned reference REF3, in the block division by the QTBT Block Structure in VVC, a block having a larger size than the block division by the Quad-Tree Block Structure in HEVC can be selected. When the block size is large in a flat region (a region in which the change in pixel value in the region is small), block distortion is likely to occur. Therefore, in VVC where a block of a larger size can be selected, if the deblocking filter that can be applied to the color difference component of the decoded image is only the weak filter as in HEVC, remarkable block distortion remains in the color difference component. There was a risk that it would end up. In view of such a situation, it is desired to improve the deblocking filter for the color difference component of the decoded image.
[0031]
　For example, in Non-Patent Document 1, the deblocking filter that can be applied to the color difference component can be changed to two types like the deblocking filter that can be applied to the luminance component, and the strong filter can be applied to the color difference component as well. A method has been proposed. Further, Non-Patent Document 1 describes that the deblock filter can be applied to the color difference component of the decoded image not only when the bS is 2 but also when the bS is 1.
[0032]
　FIG. 2 is a table for explaining the calculation of bS in Non-Patent Document 1. As shown in FIG. 2, in Non-Patent Document 1, bS is calculated based on the above-mentioned conditions A, B1 and B2, as in the example of HEVC shown in FIG. However, as described above, in Non-Patent Document 1, the deblock filter can be applied to the color difference component of the decoded image not only when the bS is 2 but also when the bS is 1. Therefore, as shown in FIG. 2, in Non-Patent Document 1, it is necessary to apply the deblock filter to the color difference component (U component, V component) of the decoded image depending on whether the condition B1 and the condition B2 are satisfied. Judgment result may be different.
[0033]
　Hereinafter, the application necessity determination process, the filter strength determination process, and the filtering process regarding the deblock filter that can be applied to the color difference component of the decoded image in Non-Patent Document 1 will be described with reference to FIG. FIG. 3 is an explanatory diagram showing an example of pixels of color difference components (U component and V component) in two blocks Bp and block Bq adjacent to each other with a vertical block boundary BB in between. Although the vertical boundary will be described here as an example, the matters described here can be applied to the horizontal boundary as well. Further, although FIG. 3 shows an example in which the block Bp and the block Bq are 4 × 4 in the color difference component, the matters described here can be similarly applied to blocks of other sizes.
[0034]
　In the example of FIG. 3, the pixels of the color difference component in the block Bp are indicated by the symbols p i and j . i is the column index and j is the row index. The column indexes i are numbered 0, 1, 2, and 3 in order from the column closest to the block boundary BB (from left to right in the figure). The row index j is numbered 0, 1, 2, 3 from top to bottom. On the other hand, the pixels of the color difference component in the block Bq are indicated by the symbols q k and j . k is the column index and j is the row index. The column index k is numbered 0, 1, 2, and 3 in order from the column closest to the block boundary BB (from right to left in the figure).
[0035]
　After the bS is calculated as described with reference to FIG. 2, the application necessity determination process and the filter strength determination process are performed using the following three conditions. In the case of the YUV420 format, such processing is performed every two lines in the color difference component. For example, in the example shown in FIG. 3, the determination regarding the line L11 and the line L12 and the determination regarding the line L21 and the line L22 are performed separately. The determination for each line is performed using the pixels of the line to be determined. Hereinafter, the application necessity determination process, the filter strength determination process, and the filtering process will be described by taking the line L11 and the line L12 as examples.
[0036]
　First, in the application necessity determination process, it is determined in order whether or not the following condition C91 and condition C92 are true.
[0037]

-Condition 　C91 : (bS == 2 || bS == 1 && (block_width> 16 && block_height> 16))
　-Condition C92: d 16 && block_height> 16)
[0045]
　The block_width and block_height in the above condition C93 are the horizontal size and the vertical size of the block related to the block boundary to be determined, respectively, like the block_width and the block_height in the condition C91.
[0046]
　When the condition C93 is true, a strong filter is applied to the color difference component of the decoded image at the target block boundary, and when the condition C93 is false, the color difference of the decoded image is applied at the target block boundary. A weak filter is applied to the ingredients.
[0047]
　The strong filter applied to the color difference component in Non-Patent Document 1 is the same as the strong filter applied to the luminance component in HEVC, and is represented by the following equations (8) to (13).
[0048]
　p 0 ′ = Clip3 (p 0 -2 * tc, p 0 + 2 * t C , (p 2 + 2 * p 1 + 2 * p 0 + 2 * q 0 + q 1 +4) >> 3)… (8)
　p 1 ′ = Clip3 (p 1 -2 * tc, p 1 + 2 * t C , (p 2 + p 1 + p 0 + q 0 +2) >> 2)… (9)
　p 2 ′ = Clip3 (p 2 -2 * tc, p 2 + 2 * t C, (2 * p 3 + 3 * p 2 + p 1 + p 0 + q 0 +4) >> 3)… (10)
　q 0 ′ = Clip3 (q 0 -2 * tc, q 0 + 2 * t C , (p 1 + 2p 0 + 2q 0 + 2q 1 + q 2 +4) >> 3)… (11)
　q 1 ′ = Clip3 (q 1 -2 * tc, q 1 + 2 * t C , ( p 0 + q 0 + q 1 + q 2 +2) >> 2)… (12)
　q 2 ′ = Clip3 (q 2 -2 * t c , q 2 +2 * t C , (p 0 + q 0 + q 1 + 3 * q 2 +2 * q 3 +4) >> 3)… (13)
[0049]
　In the above equations (8) to (13), p i and q k are pixel values ​​of the color difference component before applying the deblock filter. Further, p i ′ and q k ′ are pixel values ​​of the color difference component after applying the deblock filter. Here, i and k are the indexes of the columns in the blocks Bp and Bq described above, respectively, and the row indexes are omitted in the equations (8) to (13), respectively. In addition, t C is a parameter given according to the quantization parameter. Further, Clip3 (a, b, c) represents a clipping process in which the value c is clipped in the range of a ≦ c ≦ b.
[0050]
　Since the weak filter applied to the color difference component in Non-Patent Document 1 is the same as the weak filter applied to the color difference component in HEVC, the description here is omitted.
[0051]
　The process relating to the deblocking filter that can be applied to the color difference component of the decoded image has been described above in Non-Patent Document 1. According to the above-mentioned method, it is possible to apply the strong filter not only to the luminance component but also to the color difference component depending on the conditions.
[0052]
　However, as explained with reference to FIG. 2, the condition B1 used in the calculation of bS in Non-Patent Document 1 depends on the presence or absence of the significance coefficient of the luminance component (Y component) as in the case of HEVC. Even if other conditions are included, the information on the color difference component (U component, V component) is not used. However, the spatial pattern of the luminance component and the spatial pattern of each of the color difference components do not always match. Therefore, when the necessity of applying the deblock filter for the color difference component is determined according to the condition based on the information of the luminance component, the deblock filter is not properly applied even though the block distortion occurs, and the block is blocked. There was a risk that distortion would remain.
[0053]
　Further, when bS is 1, in order for the condition C91 used for the application necessity determination process in Non-Patent Document 1 to be true, the horizontal size of the block relating to the block boundary to be determined and the vertical direction Both sizes must be greater than 16. However, as described in reference REF3, the shape of a block (eg, CU) in VVC can be a non-square rectangle as well as a square. Then, block distortion tends to occur depending on the size in the direction orthogonal to the block boundary rather than the size in the same direction as the block boundary. Therefore, depending on the shape of the block, the deblock filter may not be properly applied in the application necessity determination process of Non-Patent Document 1, and the block distortion may remain.
[0054]
　Further, the strong filter in Non-Patent Document 1 is the same as the strong filter applied in HEVC. On the other hand, as described above, in VVC, a block having a size larger than that in HEVC can be selected. Therefore, even if the strong filter in Non-Patent Document 1 is applied, the block distortion may not be sufficiently reduced. there were.
[0055]
　　[1-2. Outline of one embodiment of the present disclosure]
　Therefore, one embodiment of the present disclosure has been created with the above circumstances as the first point of view. The image processing apparatus according to the embodiment of the present disclosure needs to apply a deblock filter to the color difference component of the decoded image based on the boundary intensity (bS) calculated using the color difference related parameters related to the color difference of the decoded image. Performs application necessity judgment processing. Hereinafter, an outline of one embodiment of the present disclosure will be described.
[0056]
　FIG. 4 is a table for explaining the calculation of bS in the present embodiment. As shown in FIG. 4, regarding the significance coefficients of the conditions B1-Y and U components, which are the conditions relating to the significance coefficients of the A and Y components, which are the conditions relating to the intra prediction. It is calculated based on the condition B1-V, which is a condition, and the condition B2, which is a condition related to MV and the reference picture.
[0057]
　Referring to FIG. 4, bS is set to 16 when condition A is true. Further, when the condition A is false and the condition B2 is true, bS is set to 1. Then, when the condition A and the condition B2 are false and any one of the condition B1-Y, the condition B1-U, and the condition B1-V is true, bS is a value between 2 and 14. Is set to. Then, when the condition A, the condition B1-Y, the condition B1-U, the condition B1-V, and the condition B2 are all false, bS is set to 0. The conditions A, B1-Y, and B2 shown in FIG. 4 are the same as the conditions A, B1, and B2 described with reference to FIG. 1, respectively. Further, the method of calculating bS according to the present embodiment will be described in more detail later.
[0058]
　Further, in the conditions B1-U and B1-V shown in FIG. 4, instead of the presence / absence of the significance coefficient of the Y component in the condition B1-Y, the presence / absence of the significance coefficient of the U component and the significance coefficient of the V component, respectively, It corresponds to the condition that the presence or absence is used for the judgment, and is expressed as follows. The authenticity of the following conditions B1-U and B1-V can be determined based on a flag (an example of a color difference-related parameter) indicating the presence or absence of a significance coefficient of the color difference component in each TU.
[0059]
　-Condition B1-U: The block boundary is the TU boundary, and at least one of the two TUs including the pixel of the uppermost line of the bS calculation target lines and sandwiching the block boundary has the significance coefficient of the U component. Existence
　-Condition B1-V: The block boundary is the TU boundary, and the significance of the V component in at least one of the two TUs including the pixel of the uppermost line of the bS calculation target lines and sandwiching the block boundary. There is a coefficient
[0060]
　In the present embodiment, it is determined whether or not the deblock filter is applied to the color difference component of the decoded image based on the bS calculated by using the conditions B1-U and B1-V related to the color difference as described above. .. With such a configuration, it becomes possible to apply the deblock filter more appropriately to the color difference component.
[0061]
　Further, in the present embodiment, as will be described later, it is determined whether or not the deblock filter is applied to the color difference component of the decoded image based on the size in the direction orthogonal to the block boundary. With such a configuration, the deblocking filter can be applied more appropriately even when the shape of the block is a non-square rectangle.
[0062]
　Further, in the present embodiment, as will be described later, a strong filter having a higher intensity (stronger low-pass characteristic) than the strong filter in Non-Patent Document 1 can be applied to the color difference component of the decoded image. Further, in order to apply such a strong filter more appropriately, in the present embodiment, the filter strength is determined by a method different from the filter strength determination process in Non-Patent Document 1. With such a configuration, block distortion can be further reduced.
[0063]
　The outline of one embodiment of the present disclosure has been described above. Hereinafter, the configuration and operation of the present embodiment for realizing the above-mentioned effects will be described in detail in order.
[0064]
　<2. Schematic configuration of the device>
　First, the schematic configuration of the device as an example to which the technique disclosed in the present specification can be applied will be described with reference to FIGS. 5 and 6. The techniques disclosed herein are applicable, for example, to image coding and decoding devices.
[0065]
　[2-1. Image Coding Device]
　FIG. 5 is a block diagram showing an example of the configuration of the image coding device 10 which is one aspect of the image processing device according to the embodiment of the present disclosure.
[0066]
　Referring to FIG. 5, the image coding apparatus 10 includes a sorting buffer 11, a control unit 12, a subtraction unit 13, an orthogonal transform unit 14, a quantization unit 15, a lossless coding unit 16, a storage buffer 17, and an inverse quantization unit. It includes 21, an inverse orthogonal transform unit 22, an addition unit 23, an in-loop filter 24, a frame memory 30, a switch 31, a mode setting unit 32, an intra prediction unit 40, and an inter prediction unit 50.
[0067]
　The sorting buffer 11 sorts a series of images (original images) to be encoded according to the GOP (Group of Pictures) structure to be encoded. The sorting buffer 11 outputs the sorted images to the control unit 12, the subtraction unit 13, the intra prediction unit 40, and the inter prediction unit 50.
[0068]
　The control unit 12 divides the image into blocks of processing units based on the block size of the external or predetermined processing unit. The CU of the Quad-Tree Block Structure or the QTBT (Quad Tree Plus Binary Tree) Block Structure may be formed as a processing unit by the block division by the control unit 12. Further, the control unit 12 determines the parameters related to the coding process based on, for example, RDO (Rate-Distortion Optimization). The determined parameters are supplied to each part.
[0069]
　The subtraction unit 13 calculates a prediction error, which is the difference between the image input from the sorting buffer 11 and the prediction image, and outputs the calculated prediction error to the orthogonal transform unit 14.
[0070]
　The orthogonal transform unit 14 executes the orthogonal transform process for each of the one or more conversion blocks (TUs) set in each region. The orthogonal transform here may be, for example, a discrete cosine transform or a discrete sine transform. More specifically, the orthogonal transform unit 14 converts the prediction error input from the subtraction unit 13 from the image signal in the spatial domain to the conversion coefficient in the frequency domain for each conversion block. Then, the orthogonal transform unit 14 outputs the conversion coefficient to the quantization unit 15.
[0071]
　Further, the orthogonal transform unit 14 generates a flag indicating the presence / absence of a significance coefficient in each TU for each component (for each Y component, U component, and V component) based on the conversion coefficient obtained by the orthogonal transform, and has a reversible code. It may be output to the conversion unit 16 and the in-loop filter 24. The flag indicating the presence / absence of the significance coefficient of the U component in each TU and the flag indicating the presence / absence of the significance coefficient of the V component in each TU, which are generated by the orthogonal transform unit 14, are included in the color difference-related parameters.
[0072]
　The quantization unit 15 is supplied with a conversion coefficient input from the orthogonal transform unit 14 and a rate control signal from the rate control unit 18 to be described later. The quantization unit 15 quantizes the conversion coefficient and outputs the quantized conversion coefficient (hereinafter, also referred to as quantization data) to the reversible coding unit 16 and the inverse quantization unit 21. Further, the quantization unit 15 changes the bit rate of the quantization data input to the lossless coding unit 16 by switching the quantization scale based on the rate control signal from the rate control unit 18.
[0073]
　The lossless coding unit 16 generates a coded stream by coding the quantization data input from the quantization unit 15. Further, the lossless coding unit 16 encodes various parameters referred to by the decoder and inserts the coding parameters into the coding stream. The parameters encoded by the lossless coding unit 16 may include the parameters determined by the control unit 12 described above.
[0074]
　Further, the parameters encoded by the lossless coding unit 16 may include color difference related parameters. The color difference-related parameters encoded by the reversible coding unit 16 are, for example, a flag indicating the presence or absence of the significance coefficient of the U component in each TU input from the orthogonal transform unit 14 as described above, and the significance of the V component in each TU. Includes a flag indicating the presence or absence of a coefficient. The lossless coding unit 16 outputs the generated coded stream to the storage buffer 17.
[0075]
　The storage buffer 17 temporarily stores the coded stream input from the lossless coding unit 16 using a storage medium such as a semiconductor memory. Then, the storage buffer 17 outputs the stored coded stream to a transmission unit (for example, a communication interface or a connection interface with a peripheral device) (not shown) at a rate corresponding to the band of the transmission line.
[0076]
　The rate control unit 18 monitors the free space of the storage buffer 17. Then, the rate control unit 18 generates a rate control signal according to the free capacity of the storage buffer 17, and outputs the generated rate control signal to the quantization unit 15. For example, the rate control unit 18 generates a rate control signal for lowering the bit rate of the quantized data when the free space of the storage buffer 17 is small. Further, for example, the rate control unit 18 generates a rate control signal for increasing the bit rate of the quantized data when the free capacity of the storage buffer 17 is sufficiently large.
[0077]
　The inverse quantization unit 21, the inverse orthogonal transform unit 22, and the addition unit 23 form a local decoder. The local decoder has a role of locally decoding the decoded image from the encoded data.
[0078]
　The inverse quantization unit 21 dequantizes the quantization data with the same quantization parameters as those used by the quantization unit 15 and restores the conversion coefficient. Then, the inverse quantization unit 21 outputs the restored conversion coefficient to the inverse orthogonal conversion unit 22.
[0079]
　The inverse orthogonal transform unit 22 restores the prediction error by executing the inverse orthogonal transform process on the conversion coefficient input from the inverse quantization unit 21. Then, the inverse orthogonal transform unit 22 outputs the restored prediction error to the addition unit 23.
[0080]
　The addition unit 23 adds the restored prediction error input from the inverse orthogonal transform unit 22 and the prediction image input from the intra prediction unit 40 or the inter prediction unit 50 to obtain a decoded image (reconstruct image). Generate. Then, the addition unit 23 outputs the generated decoded image to the in-loop filter 24 and the frame memory 30.
[0081]
　The in-loop filter 24 applies a series of in-loop filters for the purpose of improving the image quality of the decoded image. For example, as described in "2.5. In-loop filtering" of reference REF3, four in-loop filters are applied in the order of bilateral filter, deblock filter, adaptive offset filter, and adaptive loop filter. You may. The in-loop filter 24 shown in FIG. 5 includes, for example, a bilateral filter 25, a deblocking filter 26a, an adaptive offset filter 27, and an adaptive loop filter 28, and the above four in-loop filters can be applied in order. However, the in-loop filter 24 is not limited to such a configuration, and which of the four in-loop filters is applied and in what order may be appropriately selected. The deblock filter 26a will be described in detail later.
[0082]
　The in-loop filter 24 outputs the decoded image to which the in-loop filter is applied to the frame memory 30.
[0083]
　The frame memory 30 stores the decoded image before filtering input from the addition unit 23 and the decoded image to which the in-loop filter input from the in-loop filter 24 is applied by using the storage medium.
[0084]
　The switch 31 reads the decoded image before filtering used for the intra prediction from the frame memory 30, and supplies the read decoded image as a reference image to the intra prediction unit 40. Further, the switch 31 reads the filtered decoded image used for the inter-prediction from the frame memory 30, and supplies the read decoded image as a reference image to the inter-prediction unit 50.
[0085]
　The mode setting unit 32 sets the prediction coding mode for each block based on the comparison of the costs input from the intra prediction unit 40 and the inter prediction unit 50. The mode setting unit 32 outputs the prediction image generated by the intra prediction unit 40 to the subtraction unit 13 and the addition unit 23 for the block for which the intra prediction mode is set, and outputs the information related to the intra prediction to the reversible coding unit 16. Output. Further, the mode setting unit 32 outputs the prediction image generated by the inter-prediction unit 50 to the subtraction unit 13 and the addition unit 23 for the block for which the inter-prediction mode is set, and outputs the information related to the inter-prediction to the reversible coding unit. Output to 16.
[0086]
　The intra prediction unit 40 executes the intra prediction process based on the original image and the decoded image. For example, the intra prediction unit 40 evaluates the cost based on the prediction error and the amount of code generated for each of the prediction mode candidates included in the search range. Next, the intra prediction unit 40 selects the prediction mode that minimizes the cost as the optimum prediction mode. In addition, the intra prediction unit 40 generates a prediction image according to the selected optimum prediction mode. Then, the intra prediction unit 40 outputs information on the intra prediction including the prediction mode information indicating the optimum prediction mode, the corresponding cost, and the prediction image to the mode setting unit 32.
[0087]
　The inter-prediction unit 50 executes inter-prediction processing (motion compensation) based on the original image and the decoded image. For example, the inter-prediction unit 50 evaluates the cost based on the prediction error and the amount of code generated for each of the prediction mode candidates included in a certain search range. Next, the inter-prediction unit 50 selects the prediction mode with the lowest cost, that is, the prediction mode with the highest compression rate, as the optimum prediction mode. In addition, the inter-prediction unit 50 generates a prediction image according to the selected optimum prediction mode. Then, the inter-prediction unit 50 outputs information about the inter-prediction, the corresponding cost, and the predicted image to the mode setting unit 32.
[0088]
　　[2-2. Image Decoding Device]
　Next, decoding of the data encoded as described above will be described. FIG. 6 is a block diagram showing an example of the configuration of the image decoding device 60, which is one aspect of the image processing device according to the present embodiment. With reference to FIG. 6, the storage buffer 61, the reversible decoding unit 62, the inverse quantization unit 63, the inverse orthogonal transformation unit 64, the addition unit 65, the in-loop filter 66, the sorting buffer 72, and the D / A (Digital to Analogue) conversion. A unit 73, a frame memory 80, selectors 81a and 81b, an intra prediction unit 90, and an inter prediction unit 100 are provided.
[0089]
　The storage buffer 61 temporarily stores a coded stream received from the image coding device 10 via a transmission unit (for example, a communication interface or a connection interface with a peripheral device) (not shown) using a storage medium.
[0090]
　The reversible decoding unit 62 decodes the coded stream input from the storage buffer 61 according to the coding method used at the time of coding to generate quantization data. The reversible decoding unit 62 outputs the generated quantization data to the inverse quantization unit 63.
[0091]
　Further, the reversible decoding unit 62 parses various parameters from the coded stream. The parameters parsed by the reversible decoding unit 62 may include, for example, information about intra-prediction and information about inter-prediction. The reversible decoding unit 62 outputs information regarding the intra prediction to the intra prediction unit 90. Further, the reversible decoding unit 62 outputs information regarding the inter-prediction to the inter-prediction unit 100.
[0092]
　Further, the parameters parsed by the reversible decoding unit 62 may include color difference related parameters. The reversible decoding unit 62 outputs the color difference-related parameters to the in-loop filter 66. The color difference-related parameters parsed by the reversible decoding unit 62 include, for example, a flag indicating the presence / absence of the significance coefficient of the U component in each TU described above, and a flag indicating the presence / absence of the significance coefficient of the V component in each TU.
[0093]
　The inverse quantization unit 63 dequantizes the quantization data input from the reversible decoding unit 62 in the same quantization step as that used at the time of coding, and restores the conversion coefficient. The inverse quantization unit 63 outputs the restored conversion coefficient to the inverse orthogonal conversion unit 64.
[0094]
　The inverse orthogonal transform unit 64 generates a prediction error by performing an inverse orthogonal transform on the conversion coefficient input from the inverse quantization unit 63 according to the orthogonal transform method used at the time of coding. The inverse orthogonal transform unit 64 outputs the generated prediction error to the addition unit 65.
[0095]
　The addition unit 65 generates a decoded image by adding the prediction error input from the inverse orthogonal transform unit 64 and the prediction image input from the selector 71b. Then, the addition unit 65 outputs the generated decoded image to the in-loop filter 66 and the frame memory 80.
[0096]
　The in-loop filter 66 applies a series of in-loop filters for the purpose of improving the image quality of the decoded image. For example, as described in "2.5. In-loop filtering" of reference REF3, four in-loop filters are applied in the order of bilateral filter, deblock filter, adaptive offset filter, and adaptive loop filter. You may. The in-loop filter 66 shown in FIG. 6 includes, for example, a bilateral filter 67, a deblocking filter 26b, an adaptive offset filter 69, and an adaptive loop filter 70, and the above four in-loop filters can be applied in order. However, the in-loop filter 66 is not limited to such a configuration, and which of the four in-loop filters is applied and in what order may be appropriately selected. The deblock filter 26b will be described in detail later.
[0097]
　The in-loop filter 66 outputs the decoded image to which the in-loop filter is applied to the sorting buffer 72 and the frame memory 80.
[0098]
　The sorting buffer 72 sorts the images input from the in-loop filter 66 to generate a series of time-series images. Then, the sorting buffer 72 outputs the generated image to the D / A conversion unit 73.
[0099]
　The D / A conversion unit 73 converts the digital format image input from the sorting buffer 72 into an analog format image signal. Then, the D / A conversion unit 73 displays an image by outputting an analog image signal to, for example, a display (not shown) connected to the image decoding device 60.
[0100]
　The frame memory 80 stores the decoded image before filtering input from the addition unit 65 and the decoded image to which the in-loop filter input from the in-loop filter 66 is applied by using the storage medium.
[0101]
　The selector 81a switches the output destination of the image from the frame memory 80 between the intra prediction unit 90 and the inter prediction unit 100 for each block in the image according to the prediction mode information acquired by the reversible decoding unit 62. .. For example, when the intra prediction mode is specified, the selector 81a outputs the decoded image before filtering supplied from the frame memory 80 to the intra prediction unit 90 as a reference image. When the inter-prediction mode is specified, the selector 81a outputs the filtered decoded image as a reference image to the inter-prediction unit 100.
[0102]
　The selector 81b switches the output source of the prediction image to be supplied to the addition unit 65 between the intra prediction unit 90 and the inter prediction unit 100 according to the prediction mode information acquired by the reversible decoding unit 62. For example, the selector 81b supplies the prediction image output from the intra prediction unit 90 to the addition unit 65 when the intra prediction mode is specified. Further, when the inter-prediction mode is specified, the selector 81b supplies the prediction image output from the inter-prediction unit 100 to the addition unit 65.
[0103]
　The intra prediction unit 90 performs intra prediction processing based on the information regarding the intra prediction input from the reversible decoding unit 62 and the reference image from the frame memory 80, and generates a prediction image. Then, the intra prediction unit 90 outputs the generated prediction image to the selector 81b.
[0104]
　The inter-prediction unit 100 performs inter-prediction processing based on the information regarding the inter-prediction input from the reversible decoding unit 62 and the reference image from the frame memory 80, and generates a prediction image. Then, the inter prediction unit 100 outputs the generated prediction image to the selector 81b.
[0105]
　<3. Deblock filter>
　　[3-1. Configuration Example of
　Deblock Filter] This section describes an example of the configuration of the deblock filter 26a of the image coding device 10 shown in FIG. 5 and the deblock filter 26b of the image decoding device 60 shown in FIG. The configurations of the deblock filter 26a and the deblock filter 26b may be the same. Therefore, in the following description, the deblock filter 26a and the deblock filter 26b are collectively referred to as the deblock filter 26 when it is not necessary to distinguish between the two.
[0106]
　As described above, the deblock filter 26 according to the present embodiment determines whether or not the deblock filter is applied to the color difference component of the decoded image based on the bS calculated using the color difference-related parameters related to the color difference. Further, as described above, the deblock filter 26 according to the present embodiment determines whether or not the deblock filter is applied to the color difference component of the decoded image based on the size in the direction orthogonal to the block boundary. Further, as described above, in the deblock filter 26 according to the present embodiment, a strong filter having a higher intensity (stronger low-pass characteristic) than the strong filter in Non-Patent Document 1 can be applied to the color difference component of the decoded image. .. Further, in order to apply such a strong filter more appropriately, in the present embodiment, the filter strength is determined by a method different from the filter strength determination process in Non-Patent Document 1. In the following, the function of the deblock filter 26 relating to the deblock filter applied mainly to the color difference component of the decoded image will be described, and the function of the deblock filter 26 relating to the deblock filter applied to the luminance component will be described. Will be omitted as appropriate.
[0107]
　FIG. 7 is a block diagram showing an example of a detailed configuration of the deblock filter 26 according to the present embodiment. Referring to FIG. 7, the deblock filter 26 has a boundary strength calculation unit 261, a determination unit 263, and a filtering unit 269.
[0108]
　(1) Boundary strength calculation unit The
　boundary strength calculation unit 261 calculates bS (boundary strength) using the color difference-related parameters related to the color difference, targeting the block boundary of the decoded image. When a signal in the YUV420 format is the target, the boundary intensity calculation unit 261 calculates bS in units of 4 lines in the luminance component of the decoded image, that is, in units of 2 lines in the color difference component of the decoded image.
[0109]
　The color difference-related parameters used by the boundary strength calculation unit 261 in the calculation of bS in the present embodiment include a flag indicating the presence / absence of the significance coefficient of the U component in each TU and a flag indicating the presence / absence of the significance coefficient of the V component in each TU. .. As shown in FIG. 7, in the boundary strength calculation unit 261, a flag indicating the presence or absence of the significance coefficient of each component (Y component, U component, V component) in each TU is set in the orthogonal transform unit 14 or the reversible decoding unit 62. Entered from.
[0110]
　The boundary strength calculation unit 261 calculates bS based on the conditions A, B1-Y, B1-U, B1-V, and B2 described with reference to FIG. That is, the boundary strength calculation unit 261 calculates the bS based on whether or not the significance coefficient of the color difference component exists in the TU that sandwiches the block boundary that is the calculation target of the bS. Further, the boundary strength calculation unit 261 according to the present embodiment independently determines whether or not the significance coefficient of each component of the Y component, the U component, and the V component exists in the TU sandwiching the block boundary for which the bS is calculated. By determining, bS can be calculated. With such a configuration, bS suitable for the U component and the V component is calculated and more appropriate than calculating the bS based on whether or not the significance coefficient of the Y component exists as described with reference to FIG. It is possible to apply a deblock filter to.
[0111]
　The calculation of bS by the boundary strength calculation unit 261 will be described in more detail with reference to FIG. FIG. 8 is a table showing an example of bS calculated by the boundary strength calculation unit 261. The bS calculated by the boundary strength calculation unit 261 can be represented by a plurality of bits. In the example shown in FIG. 8, bS is represented by 5 bits. Further, the bS may be calculated so that the plurality of bits include at least one bit corresponding to each component of the Y component, the U component, and the V component. With this configuration, when the determination unit 263, which will be described later, determines the necessity of applying the deblock filter based on bS, the determination is easily performed by referring to the bS bit corresponding to each component to be determined. It becomes possible.
[0112]
　Further, the boundary strength calculation unit 261 may calculate the bS so that each bit included in the bS corresponds to the truth of each condition. In the example shown in FIG. 8, bS is calculated so that when each condition is true, the bit corresponding to the condition is 1, and when each condition is false, the bit corresponding to the condition is 0. To. Further, in the example shown in FIG. 8, bS is represented by 5 bits, the 5th bit of bS is the condition A related to intra-prediction, the 4th bit of bS is the condition B1-Y related to the significance coefficient of the Y component, and bS. The third bit is the condition B1-U related to the significance coefficient of the U component, the second bit of bS is the condition B1-V related to the significance coefficient of the V component, and the first bit of bS is the condition B2 related to MV and the reference picture, respectively. It corresponds. However, the correspondence between each bit of bS and each condition is not limited to the example shown in FIG. For example, the order of the 4th bit, the 3rd bit, and the 2nd bit of bS corresponding to each component of the Y component, the U component, and the V component may be exchanged.
[0113]
　(2) As
　shown in FIG. 7, the determination unit determination unit 263 is applied to the application necessity determination unit 265 for determining the necessity of applying the deblock filter to the color difference component of the decoded image and the color difference component of the decoded image. It includes a filter strength determination unit 267 for determining the filter strength of the deblock filter. Hereinafter, the functions of the application necessity determination unit 265 and the filter strength determination unit 267 will be sequentially described.
[0114]
　In the following description, the necessity of applying the deblock filter to the color difference component of the decoded image and the determination of the filter intensity will be mainly described, and the determination of the luminance component will be omitted as appropriate. Further, the application necessity determination unit 265 and the filter strength determination unit 267 according to the present embodiment independently determine the application necessity and the filter strength of the deblock filter for each of the U component and the V component.
[0115]
　The application necessity determination unit 265 targets the block boundary of the decoded image, and applies the deblock filter to the color difference component of the decoded image based on the bS (boundary intensity) calculated as described above by the boundary strength calculation unit 261. Judge the necessity.
[0116]
　Further, the application necessity determination unit 265 may further determine whether or not the deblock filter is applied to the color difference component of the decoded image based on the block size of the block sandwiching the block boundary. In the following, the determination based on the block size may be referred to as a large block determination. Further, the application necessity determination unit 265 does not have to always perform a large block determination for all block boundaries, and may determine whether or not to perform a large block determination according to bS. The case where the large block determination is performed and the details of the large block determination will be described later.
[0117]
　The application necessity determination unit 265 according to the present embodiment determines the application necessity of the deblock filter by the determination of the following condition C1 and the determination of the condition C2.
[0118]
　-Condition C1: (bS == 16 || (Condition C11 && Condition C12))
　-Condition C2: d 16) || (EDGE_HOR && block_height> 16)
[0126]
　In the above condition C12, EDGE_VER means that the block boundary to be determined is a vertical boundary, and EDGE_HOR means that the block boundary to be determined is a horizontal boundary.
[0127]
　Further, since the above-mentioned condition C2 is the same as the above-mentioned condition C92, the description here will be omitted. The above-mentioned determination of the condition C2 is performed when the condition C1 is true, and when the condition C1 is false, the determination of the condition C2 is not performed and the deblock filter must be applied. It is judged. The determination of the condition C2 requires the process of calculating the variable d as in the above equations (1) to (7), and the processing amount is larger than the determination of the condition C1. Therefore, the determination of the condition C2 is performed after the condition C1. By doing so, it is possible to suppress the processing amount.
[0128]
　Further, the filter strength determination unit 267 further determines the filter strength of the deblock filter applied to the color difference component of the decoded image after the necessity of applying the deblock filter is determined by the conditions C1 and C2 as described above. .. As will be described later, the deblocking filter that can be applied in the present embodiment may be two types, a weak filter having a weaker strength and a strong filter having a stronger strength. Then, the filtering unit 269 described later applies either a weak filter or a strong filter according to the filter strength determined by the filter strength determining unit 267.
[0129]
　The filter strength determination unit 267 determines the filter strength when it is determined that the deblock filter is applied. By determining the filter strength after the determination of the necessity of applying the deblock filter, it is possible to suppress the processing related to the determination of the filter strength.
[0130]
　Further, the filter strength determination unit 267 determines the filter strength based on the waveform of the color difference component of the pixel located near the block boundary. Hereinafter, the determination based on the waveform will be described. The filter strength determination unit 267 determines the filter strength under the condition C3 based on the following waveform.
[0131]
　- conditions C3 :( conditions C31 && conditions C32 && conditions C33) -
　conditions C31: | the p- 3 -P 0 | Tasu | Q 3 -Q 0 | <(Beta >> 3)
　- conditions C32: | the p- 2 -2 * the p- 1 + p 0 | + | q 2 -2 * q 1 + q 0 | <(beta >> 2)
　-Condition C33: | p 0 -q 0 | <((t c * 5 + 1) >> 1)
[0132]
　The filter strength determination unit 267 determines the above condition C3 for the pixels included in the two lines among the pixels located near the block boundary. The condition C31, the condition C32, and the condition C33 used in the above C3 are determined for each line. Note that the p i , q k , p i ′, q k ′, beta, and t C under the condition C31, the condition C32, and the condition C33 have already been described above, and thus the description thereof will be omitted here.
[0133]
　Condition C31, condition C32, and condition C33 are conditions for determining using the pixels included in each line. More specifically, the condition C31 is a condition relating to the flatness in the block of the color difference component of the pixel included in each line. Further, the condition C32 is a condition relating to the determination of the continuity within the block of the color difference component of the pixels included in each line. Further, the condition C33 is a condition relating to the gap (difference) between blocks of the color difference component of the pixel included in each line, and more specifically, the gap between blocks is determined by using the pixel value adjacent to the block boundary. It is a condition to do.
[0134]
　When the condition C31 is true, the flatness of the waveform of the color difference component is high in each block. Further, when the condition C32 is true, the waveform of the color difference component has high continuity in each block. Further, when the condition C32 is true, the waveform of the color difference component has a large gap at the block boundary.
[0135]
　As described above, the condition C3 is determined to be true when all of the above conditions C31, C32, and C33 are true. Further, the filter strength determination unit 267 determines the above condition C3 for each line. However, as described above, the filter strength is determined in units of 2 lines. That is, the strong filter is applied to the two lines when the above condition C3 is true in both of the two consecutive lines, and the weak filter is applied to the two lines when the condition C3 is false. The filter strength is determined.
[0136]
　(3) Filtering unit The
　filtering unit 269 applies the deblocking filter to the color difference component of the pixel located near the block boundary based on the determination result of the deblocking filter application necessity by the application necessity determination unit 265. Further, as described above, the filtering unit 269 applies a weak filter or a strong filter as a deblocking filter according to the filter strength determined by the filter strength determining unit 267.
[0137]
　The weak filter applied to the color difference component by the filtering unit 269 according to the present embodiment is the same as the weak filter applied to the color difference component of the decoded image in, for example, Non-Patent Document 1 and HEVC described above. Good. On the other hand, the strong filter applied to the color difference component in the present embodiment is different from the strong filter applied to the color difference component in Non-Patent Document 1 (strong filter applied to the luminance component in HEVC). It's okay. Hereinafter, an example of a strong filter applied to the color difference component in the present embodiment will be described.
[0138]
　The coefficient of the strong filter applied to the color difference component in the present embodiment may be 2 at the center position of the application range of the strong filter and 1 at other positions. Further, the filtering unit 269 sets the application target range of the strong filter from the block boundary to 3 pixels on both sides, and sets the 3 pixels on both sides of the center position of the application target range as reference pixels, and sets the color difference component of the pixels included in the application target range. A strong filter may be applied. For example, a strong filter with p 0 at the center of the applicable range is expressed by the following equation (14).
[0139]
　p 0 ′ = Clip3 (p 0- w * t C , p 0 + w * t C , ((p 3 + p 2 + p 1 + 2 * p 0 + q 0 + q 1 + q 2 +4)> > 3))… (14)
[0140]
　In the above equation (14), w is a weight that can be appropriately set, and may be set to, for example, 1 or 2. Further, Clip3 (a, b, c) represents a clipping process in which the value c is clipped in the range of a ≦ c ≦ b as described above.
[0141]
　By applying such a strong filter, it is possible to apply a deblocking filter stronger than the strong filter applied to the color difference component in Non-Patent Document 1 described above.
[0142]
　By the way, when the center position of the application target range of the strong filter is the second pixel or the third pixel from the block boundary, the reference pixel includes a pixel separated by 5 pixels or more from the block boundary. However, pixels separated by 5 pixels or more from the block boundary are not used for determining the filter strength and may not be suitable for use as reference pixels. Therefore, the filtering unit 269 may use the pixel value of the fourth pixel from the block boundary as the pixel value of the reference pixel by padding instead of the pixel separated by 5 pixels or more from the block boundary.
[0143]
　For example, a strong filter with p 1 at the center of the applicable range is expressed by the following equation (15).
　p 1 ′ = Clip3 (p 1- w * t C , p 1 + w * t C , ((p 4 + p 3 + p 2 + 2 * p 1 + p 0 + q 0 + q 1 + 4) ＞ ＞ 3))
　　　= Clip3 (p 1- w * t C , p 1 + w * t C , ((p 3 + p 3 + p 2 + 2 * p) 1 + p 0 + q 0 + q 1 +4) >> 3))
　　　= Clip3 (p 1- w * t C , p 1 + w * t C , ((2 * p 3 + p 2 + 2 * p) 1 + p 0 + q 0 + q 1 +4) >> 3))… (15)
[0144]
　Similarly, a strong filter with p 2 at the center of the applicable range is expressed by the following equation (16).
　p 2 ′ = Clip3 (p 2- w * t C , p 2 + w * t C , ((p 5 + p 4 + p 3 + 2 * p 2 + p 1 + p 0 + q 0 +4) ＞ ＞ 3))
　　　= Clip3 (p 2- w * t C , p 2 + w * t C , ((p 3 + p 3 + p 3 + 2 * p) 2 + p 1 + p 0 + q 0 +4) >> 3)))
　　　= Clip3 (p 2- w * t C , p 2 + w * t C , ((3 * p 3 + 2 * p 2 + p) 1 + p 0 + q 0 +4) >> 3))… (16)
[0145]
　Similarly, the strong filters having q 0 to q 3 at the center position of the applicable range are expressed by the following equations (17) to (19), respectively.
　q 0 ′ = Clip3 (q 0 -w * t C , q 0 + w * t C , ((p 2 + p 1 + p 0 + 2 * q 0 + q 1 + q 2 + q 3 +4) ＞ ＞ 3))… (17)
　q 1 ′ = Clip3 (q 1 -w * t C , q 1 + w * t C , ((p 1)+ p 0 + q 0 + 2 * q 1 + q 2 + 2 * q 3 +4) >> 3))… (18)
　q 2 ′ = Clip3 (q 2 -w * t C , q 2 + w * t C , ((p 0 + q 0 + q 1 + 2 * q 2 + 3 * q 3 +4) >> 3))… (19)
[0146]
　　[3-2. Process Flow]
　The configuration example of the deblock filter 26 according to the present embodiment has been described above. Subsequently, the flow of processing by the deblock filter 26 according to the present embodiment will be described. FIG. 9 is a flowchart showing an example of the processing flow by the deblock filter 26 according to the present embodiment. In the following, among the processes by the deblock filter 26, the processes related to the features of the present embodiment will be described, and the description of other processes will be omitted as appropriate.
[0147]
　First, the boundary strength calculation unit 261 calculates bS (boundary strength) (S10). Here, the method of calculating bS will be described in more detail with reference to FIG. FIG. 10 is a flowchart for explaining the flow of the boundary strength calculation process (S10) executed by the boundary strength calculation unit 261.
[0148]
　First, the boundary strength calculation unit 261 initializes bS to 0 (S102). Subsequently, the boundary strength calculation unit 261 determines the authenticity of the condition A, which is a condition related to the intra prediction (S104). If condition A is true (YES in S104), bS is set to 16 (S106).
[0149]
　On the other hand, when the condition A is false (NO in S104), the boundary strength calculation unit 261 determines the authenticity of the condition B2, which is a condition relating to the motion vector (MV) and the reference picture (S108). If condition B2 is true (YES in S108), bS is set to 1 (S110).
[0150]
　On the other hand, when the condition B2 is false (NO in S108), the boundary strength calculation unit 261 determines the authenticity of the conditions B1-Y, which is a condition regarding the presence or absence of the significance coefficient of the Y component (S112). If the condition B1-Y is true (YES in S112), the process proceeds to step S116 after 8 is added to bS (S114). On the other hand, when the condition B1-Y is false (NO in S112), the process proceeds to step S116 as it is.
[0151]
　In step S116, the boundary strength calculation unit 261 determines the authenticity of the conditions B1-U, which is a condition regarding the presence or absence of the significance coefficient of the U component. If the condition B1-U is true (YES in S116), the process proceeds to step S120 after 4 is added to bS (S118). On the other hand, if the condition B1-U is false (NO in S116), the process proceeds to step S120 as it is.
[0152]
　In step S120, the boundary strength calculation unit 261 determines the authenticity of the conditions B1-V, which is a condition regarding the presence or absence of the significance coefficient of the V component. When the condition B1-V is true (YES in S120), the boundary strength calculation process (S10) ends after 2 is added to bS (S122). When the condition B1-V is false (NO in S120), the boundary strength calculation process (S10) ends as it is.
[0153]
　Returning to FIG. 9, the description of the processing flow by the deblock filter 26 will be continued. In step S20, the application necessity determination unit 265 of the determination unit 263 determines the authenticity of the above-mentioned condition C1. If the condition C1 is false (NO in S20), the process ends.
[0154]
　On the other hand, when the condition C1 is true (YES in S20), the application necessity determination unit 265 determines the truth of the condition C2 described above (S30). If the condition C2 is false (NO in S30), the process ends.
[0155]
　On the other hand, when the condition C2 is true (YES in S30), the filter strength determination unit 267 of the determination unit 263 determines the filter strength by determining the authenticity of the condition C3 described above (S40). When the condition C3 is true (YES in S40), the filtering unit 269 applies a strong filter to the color difference component of the pixel located near the block boundary (S50). On the other hand, when the condition C3 is false (NO in S40), the filtering unit 269 applies a weak filter to the color difference component of the pixel located near the block boundary (S60).
[0156]
　The flow of processing by the deblock filter 26 according to the present embodiment has been described above. In the case of the YUV420 format, for example, the above-described processing described with reference to FIGS. 9 and 10 can be performed in units of 4 lines in the luminance component of the decoded image, that is, in units of 2 lines in the color difference component of the decoded image.
[0157]
　
[0158]
　Hereinafter, a new DF (deblock filter) will be described.
[0159]
　The following documents are prior art as documents related to the present technology, and are incorporated in the present specification by reference.
[0160]
　[JVET-L0072 (version 1 --date 2018-09-25 00:23:50)]
　K. Andersson, Z. Zhang, R. Sjoberg: CE11: Long deblocking filters for luma (CE11.1.1) and for both luma and chroma (CE11.1.9), Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29/WG 11 12th Meeting, Macao, CN, 3-12 Oct. 2018.
[0161]
　[JVET-L0224 (version 1 --date 2018-09-25 01:59:53)]
　Anand Meher Kotra, Biao Wang, Semih Esenlik, Han Gao, Zhijie Zhao, Jianle Chen: CE11.1.8: Longer tap Luma deblocking filter, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29/WG 11 12th Meeting, Macao, CN, 3-12 Oct. 2018.
[0162]
　[JVET-L0403r1 (version 3-date 2018-10-04 05:13:00)]
　Dmytro Rusanovskyy, Marta Karczewicz: CE11: Test on long deblocking filtering from JVET-J0021 / JVET-K0334 (CE11.1.4). Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29/WG 11 12th Meeting, Macao, CN, 3-12 Oct. 2018.
[0163]
　[JVET-L0405r1 (version 2-date 2018-10-03 07:14:31)]
　Weijia Zhu, Kiran Misra, Phil Cowan, Andrew Segall: CE11: Deblocking modifications for Large CUs both luma and chroma (Test 11.1.7a and CE11.1.7b). Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29/WG 11 12th Meeting, Macao, CN, 3-12 Oct. 2018.
[0164]
　[JVET-L0327-v1 (version 1 --date 2018-09-25 02:33:13)]
　Masaru Ikeda, Teruhiko Suzuki: CE11 : Long-tap deblocking filter for luma and chroma (CE11.1.6). Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29/WG 11 12th Meeting, Macao, CN, 3-12 Oct. 2018.
[0165]
　Further, the scope disclosed in the present specification is not limited to the contents of the examples, and the contents of the following reference document REF4 known at the time of filing are also incorporated in the present specification by reference. In other words, the content described in the following reference REF4 is also a basis for determining support requirements.
REF4:
[JVET-K1002-v2 (version 3 --date 2018-10-02 16:37:03)]
Jianle Chen, Yan Ye, Seung Hwan Kim: Algorithm description for Versatile Video Coding and Test Model 2 (VTM 2), Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29/WG 11 11th Meeting, Ljubljana, SI, 10-18 July 2018.
[0166]
　In VVC, the block size of encoding is large (8 times that of AVC and 2 times that of HEVC), and the deterioration of the block boundary is remarkable.
[0167]
　For large blocks with a large block size, HEVC's DF (deblocking filter) may not be able to completely remove the block noise when there is strong block noise.
[0168]
　Further, in VVC, in the case of an intranet, different block divisions can be selected for the luminance component and the color difference component, and therefore the block size can be optimized for each of the luminance component and the color difference component. It is known that the coding efficiency is further improved by adopting a large block for the color difference component and a small block with a small block size for the luminance component, and the importance of DF for the large block of the color difference component is important. The sex is increasing.
[0169]
　For large blocks, block noise (block distortion) tends to increase. In order to properly remove large block noise, a DF having a strong filter strength, that is, a DF that blurs the image more is required.
[0170]
　Since the filter that blurs the image is a long-tap filter (with a large number of taps), a long-tap filter is used as the DF applied to the luminance component and color difference component in order to properly remove large block noise. Is required.
[0171]
　When adopting a long-tap filter as the DF, prepare a line buffer for the number of pixels required for filtering at the block boundary, that is, at the horizontal boundary when the DF is applied, for example, in the order of raster scan. There is a need. For example, when 8 pixels are arranged in the vertical direction orthogonal to the horizontal boundary, for example, 8 pixels are used for DF filtering (filtering processing), 4 of the 8 pixels are pixels of the block above the horizontal boundary. At some point, a line buffer is needed to store pixels (pixel values) for 4 lines (lines).
[0172]
　If the DF is provided with a line buffer having a large capacity, the cost becomes high. If cost is emphasized and the capacity of the line buffer is reduced, a long tap filter cannot be adopted as the DF, the performance of the DF deteriorates, and it becomes difficult to sufficiently remove the block noise. ..
[0173]
　Therefore, it is desirable to determine the DF specifications by balancing cost and performance. In AVC and HEVC, the filter design for DF does not take into account the capacity of the line buffer.
[0174]
　Therefore, in this technology, we propose a new DF (hereinafter, also referred to as a new DF).
[0175]
　Unless otherwise specified, the following description is intended for a color difference component, and the description of the luminance component will be omitted.
[0176]
　FIG. 11 is a block diagram showing a configuration example of the DF300 as a new DF.
[0177]
　The DF300 can be used as the DF26.
[0178]
　In FIG. 11, the parts corresponding to the DF 26 in FIG. 7 are designated by the same reference numerals, and the description thereof will be omitted as appropriate below.
[0179]
　In FIG. 11, the DF300 has a boundary strength calculation unit 261, a determination unit 310, a filtering unit 320, a line buffer 330, and a control unit 340.
[0180]
　Therefore, the DF300 is common to the DF26 of FIG. 7 in that it has the boundary strength calculation unit 261. However, the DF 300 is different from the DF 26 in that it has a determination unit 310 and a filtering unit 320 in place of the determination unit 263 and the filtering unit 269, respectively. Further, the DF300 differs from the DF26 in that it has a new control unit 340.
[0181]
　Although not shown in FIG. 7, the DF26 in FIG. 7 has a line buffer like the DF300. However, the capacities of the line buffer 330 included in the DF300 and the line buffer included in the DF26 may differ.
[0182]
　The determination unit 310 includes an application necessity determination unit 311 and a filter strength determination unit 312.
[0183]
　BS is supplied from the boundary strength calculation unit 261 to the application necessity determination unit 311. Further, the decoded image is supplied to the application necessity determination unit 311 from the outside of the DF 300 (addition unit 23 in FIG. 5 and addition unit 65 in FIG. 6) or the line buffer 330.
[0184]
　Similar to the application necessity determination unit 265 of FIG. 7, the application necessity determination unit 311 needs to be applied by using the bS from the boundary strength calculation unit 261 and the decoded image from the outside of the DF300 or the line buffer 330. Performs a rejection judgment process.
[0185]
　The application necessity determination unit 311 performs a step determination for determining whether or not there is a high possibility that there is a step at the block boundary according to the bS. For example, the application necessity determination unit 311 determines that there is a high possibility that there is a step at the block boundary when bS is larger than 0 (1 or more). Then, the application necessity determination unit 311 determines whether or not to apply the DF to the pixels of the color difference component near the block boundary when it is determined that there is a high possibility that the block boundary has a step in the step determination. Make an application judgment. The application necessity determination process performed by the application necessity determination unit 311 includes the above step determination and the filter application determination.
[0186]
　The application necessity determination unit 311 supplies the determination result of the filter application determination process to the filter strength determination unit 312 as the determination result of the application necessity determination process.
[0187]
　The filter strength determination unit 312 is supplied with the determination result of the filter application determination from the application necessity determination unit 311 and the decoded image from the outside of the DF 300 or the line buffer 330.
[0188]
　When the filter application determination from the application necessity determination unit 311 indicates that the DF is applied, the filter strength determination unit 312 may use the outside of the DF 300 or the line buffer 330 as in the case of the filter strength determination unit 267 of FIG. Filter strength determination to determine the filter strength of DF applied to the color difference component of the decoded image, that is, filter type determination to determine the filter type of DF applied to the color difference component of the decoded image using the decoded image from Do. Then, the filter strength determination unit 321 supplies the determination result of the filter type determination to the filtering unit 320.
[0189]
　In the new DF, the DF filter types applied to the color difference component of the decoded image include, for example, a weak filter with a weaker filter intensity (compared to a strong filter) and a stronger filter (compared to a weak filter). There are two filter types, a strong filter with filter strength.
[0190]
　The filter strength determination unit 312 supplies the determination result of the filter type determination to the filtering unit 320, and the decoded image is supplied from the outside of the DF 300 or the line buffer 330.
[0191]
　Similar to the filtering unit 269 of FIG. 7, the filtering unit 320 performs a filter process of applying the strong filter or the weak filter represented by the determination result of the filter type determination from the filter strength determination unit 312 to the decoded image. That is, the filtering unit 320 performs an operation as a filter process of the target pixel, which is a pixel of the color difference component to be filtered, in the decoded image from the outside of the DF 300 or the line buffer 330, in the vicinity of the target pixel. This is done using the pixel of the component. Here, the pixels used for the filter processing are also referred to as filter constituent pixels.
[0192]
　The filtering unit 320 outputs pixels (of color difference components) obtained by filtering the target pixels as filter pixels (pixels constituting the filtered image after the filtering).
[0193]
　A decoded image is supplied to the line buffer 330 from the outside of the DF 300. The line buffer 330 appropriately stores the pixels of the color difference component of the decoded image from the outside of the DF 300 (the color difference component of the pixel of the decoded image). The line buffer 330 has a storage capacity for storing pixels of a color difference component corresponding to a predetermined number of lines (number of lines), and when the pixels corresponding to the storage capacity are stored, the new pixels are overwritten with the oldest pixels. Memorize in the form of
[0194]
　The control unit 340 controls each block constituting the DF 300.
[0195]
　In the present embodiment, the DF 300 processes, for example, the decoded images in the order of raster scan. When processing the decoded image from top to bottom instead of the raster scan order is repeated from left to right, the horizontal (horizontal) (left and right) and vertical (vertical) (up and down) described below are used. Is the opposite (swapped).
[0196]
　FIG. 12 is a diagram showing a configuration example of a decoded image processed by the DF 300.
[0197]
　The blocks constituting the decoded image include a CTU, and the CTU includes blocks such as PU and TU.
[0198]
　The strength calculation unit 261 and the determination unit 310 and the filtering unit 320 of the DF 300 of FIG. 11 can perform processing in units of, for example, the CTU, and in this case, a buffer (hereinafter, not shown) capable of storing the CTU. , Also called an internal buffer).
[0199]
　Now, among the block boundaries, the CTU boundary is referred to as the CTU boundary, and the block boundary other than the CTU boundary is referred to as the internal boundary. Further, the CTU that is the target of processing by the DF300 is referred to as a noteworthy CTU.
[0200]
　The intensity calculation unit 261 and the determination unit 310, and the filtering unit 320 store the pixels of the line (row) included in the attention CTU in the internal buffer, and process the attention CTU.

The scope of the claims
[Claim 1]
　The
　filter strength is higher than that of the first luminance filter for the decoding unit that decodes the bit stream to generate the decoded image and the pixels of the color difference component located near the block boundary of the decoded image decoded by the decoding unit. The reduced first luminance filter with reduced filter characteristics of the second luminance filter or the reduced first luminance filter with reduced filter characteristics of the first luminance filter is a second luminance filter with stronger filter strength than the first luminance filter. An
　image processing device including a filter unit applied as a color difference filter .
[Claim 2]
　The reduced second luminance filter is a reduced second luminance vertical filter that performs the reduced second luminance filter in the vertical direction, and the second luminance filter is a second luminance vertical filter that performs the second luminance
　filter in the vertical direction. it is
　an image processing apparatus according to claim 1.
[Claim 3]

　The image processing apparatus according to claim 2, 　wherein the reduced second luminance vertical filter is a filter in which a filter coefficient or a clip parameter is changed with respect to the second luminance filter .
[Claim 4]

　The image processing apparatus according to claim 2, 　wherein the reduced second luminance vertical filter is an asymmetric filter in which the filter characteristics of the filter applied to the pixels located above the block boundary are reduced .
[Claim 5]

　The image processing apparatus according to claim 2 　, wherein the reduced first luminance filter is a reduced first luminance vertical filter that performs the reduced first luminance filter in the vertical direction .
[Claim 6]
　A control unit that controls the filter unit is further added so that the reduced second luminance vertical filter is applied as the second color difference vertical filter to the block boundary of the coding tree block, which is a block having a fixed size in sequence units. comprising
　an image processing apparatus according to claim 2.
[Claim 7]
　A
　claim that the control unit controls the filter unit so that the reduced second luminance vertical filter is applied as the second color difference vertical filter to a block boundary of a block divided from the coding tree block. The image processing apparatus according to 6.
[Claim 8]
　The control unit applies the second luminance vertical filter that performs the second luminance filter in the vertical direction to the block boundary of the block divided from the coding tree block as the second luminance vertical filter.
　The image processing apparatus according to claim 6, which controls the filter unit .
[Claim 9]

　The image processing apparatus 　according to claim 7, wherein the block is divided according to a Quad-Tree Block Structure, a Quad Tree Plus Binary Tree Block Structure, or a Multi-type Tree Coding Block Structure .
[Claim 10]

　The image processing apparatus according to claim 1, 　wherein the second luminance filter is a strong filter having a luminance component conforming to the H, 265 / HEVC standard .
[Claim 11]

　The image processing apparatus according to claim 1, 　wherein the second luminance filter is a filter in which the filter characteristics of a strong filter having a luminance component conforming to the H, 265 / HEVC standard are changed .
[Claim 12]
　The bit stream is decoded to generate a decoded image, and the
　filter of the second luminance filter, which has a stronger filter intensity than the first luminance filter, for the pixels of the color difference component located near the block boundary of the decoded image. reduced first luminance filter characteristic reduced-reduced second luminance filter or the filter characteristic of the first luminance filter is reduced, and applying a strong second chrominance filter having filter strength than the first color difference filter
　the Image processing method including.
[Claim 13]
　The filter characteristics of the second luminance filter, which has a stronger filter intensity than the first luminance filter, for the pixels of the color difference component located near the block boundary of the locally decoded image that was locally decoded when the image was encoded. A filter unit that applies the reduced reduced second luminance filter or the reduced first luminance filter with reduced filter characteristics of the first luminance filter as a second luminance filter having a stronger filter intensity than the first luminance filter.
　An
　image processing apparatus including a coding unit that encodes the image using the locally decoded image to which the second luminance filter is applied by the filter unit.
[Claim 14]
　The filter characteristics of the second luminance filter, which has a stronger filter intensity than the first luminance filter, for the pixels of the color difference component located near the block boundary of the locally decoded image that was locally decoded when the image was encoded. the reduced-reduced second luminance filter or the first reduced first luminance filter filter characteristics of the luminance filter is reduced, as a strong second chrominance filter having filter strength than the first color difference filter, and applying,
　the An
　image processing method including encoding the image using the locally decoded image to which the second luminance filter is applied .