0001]This technique relates to an image processing apparatus and an image processing method, in particular, for example, the S / N of the image, an image processing apparatus and image processing method to be able to improve significantly.
BACKGROUND
[0002]One of which is predictive coding scheme, for example, in HEVC (High Efficiency Video Coding), ILF (In Loop Filter) has been proposed. Moreover, the post HEVC (next generation of predictive coding method of HEVC), is expected to adopt the ILF.
[0003]
　The ILF, DF (Deblocking Filter) for reducing block noise, SAO (Sample Adaptive Offset) for reducing ringing (the decoded image, the error with respect to the original image) encoding error ALF to minimize there is a (Adaptive Loop Filter).
[0004]
　The ALF, is described in Patent Document 1, for SAO, are described in US Pat.
CITATION
Patent Document
[0005]
Patent Document 1: Patent No. 5485983 Patent Publication
Patent Document 2: JP-T 2014-523183 Patent Publication
Summary of the Invention
Problems that the Invention is to Solve
[0006]
　DF and as ILF currently proposed, SAO, ALF is freedom is low, since it is difficult to perform fine control of the filter, the image of the S / N (Signal to Noise Ratio), significantly improved it is difficult.
[0007]
　This technology has been made in view of such circumstances, the S / N of the image, and to be able to improve significantly.
Means for Solving the Problems
[0008]
　The image processing apparatus of the present technology, the processing target pixel in the first image obtained by adding the predicted image and the residual predictive coding, classification to classify into any class of a plurality of classes class that the pixel class taps, by selecting from the first image, and the class tap selection unit constituting the class tap, by using the class taps, performs the classification of the processed pixels to be used in a classification unit, to the first image, to filter corresponding to the class of the target pixel, and a filter processing unit that generates a second image used for the prediction of the predicted image, the class tap selection unit, the tap structure of the class tap is an image processing apparatus for updating the selected tap structure from a plurality of tap structures.
[0009]
　The image processing method of the present technology, the processing target pixel in the first image obtained by adding the predicted image and the residual predictive coding, classification to classify into any class of a plurality of classes by the pixels serving class taps, selected from the first image used in a method comprising constituting the class tap, by using the class taps, and to perform the classification of the target pixel, the the first image, to filter corresponding to the class of the target pixel, and a generating a second image used for the prediction of the predicted image, the tap structure of the class taps, a plurality of the image processing method of updating the selected tap structure from the tap structure.
[0010]
　Any class of the image processing apparatus and image processing method of the present technique, a processing target pixel in the first image obtained by adding the predicted image and the residual predictive coding, a plurality of classes pixels serving as class taps used for classifying the classification to have, by being selected from the first image, the class tap is formed. Then, by using the class taps, the classification of the target pixel is performed, the first image, a filter process is performed corresponding to the class of the target pixel, used to predict the predicted image second image to be is generated. In this case, the tap structure of the class tap, are updated to the selected tap structure from a plurality of tap structures.
[0011]
　The image processing apparatus may be an independent apparatus or may be an internal block constituting one apparatus.
[0012]
　The image processing apparatus may be realized by causing a computer to execute a program. Program via a transmission medium, by transmitting, or by being recorded on a recording medium, it can be provided.
Effect of the invention
[0013]
　According to this technique, the S / N of the image can be greatly improved.
[0014]
　Here, the advantages described in the present invention is not necessarily limited, it may be any of the effects described in the present disclosure.
BRIEF DESCRIPTION OF THE DRAWINGS
[0015]
Is a diagram illustrating a configuration example of an embodiment of FIG. 1 image processing system embodying the present technology.
It is a block diagram showing a first configuration example of an image conversion apparatus for performing the Figure 2] classification adaptive processing.
It is a block diagram showing a configuration example of FIG. 3 learning apparatus that performs learning of the tap coefficients stored in the coefficient acquisition unit 24.
It is a block diagram showing a configuration example of FIG. 4 learning unit 33.
It is a block diagram showing a second configuration example of an image conversion apparatus for performing FIG. 5 classification adaptive processing.
It is a block diagram showing a configuration example of a learning apparatus that performs learning of FIG. 6 species coefficients stored in the coefficient acquisition unit 24.
7 is a block diagram showing a configuration example of the learning unit 63.
8 is a block diagram showing another configuration example of the learning unit 63.
It is a block diagram showing a first configuration example of FIG. 9 encoder 11.
It is a block diagram showing a configuration example of FIG. 10 classification adaptive filter 111.
11 is a block diagram showing an example of the configuration of a learning device 131.
It is a diagram illustrating an example of a class tap shape of the tap structure of FIG. 12 class taps.
It is a diagram illustrating an example of FIG. 13 tap structure of a class tap constituted by pixels of a plurality of frames.
It is a diagram illustrating an example of a variation of FIG. 14 in the class tap tap structure.
It is a diagram for explaining an example of a method for determining a plurality of tap structures of FIG. 15 class tap to be stored in the tap structure selection unit 151.
[16] the classification unit 163 is a diagram showing an example of an image feature quantity of the class taps used for classification.
Is a block diagram showing a configuration example of FIG. 17 the image conversion unit 171.
18 is a flowchart illustrating an example of processing of the learning device 131.
It is a block diagram showing a configuration example of FIG. 19 image converter 133.
It is a flowchart illustrating an example of an encoding process in FIG. 20 encoding device 11.
21 is a flowchart illustrating an example of a classification adaptive processing performed in step S46.
Is a block diagram showing a first configuration example of FIG. 22 decoder 12.
It is a block diagram showing a configuration example of FIG. 23 classification adaptive filter 206.
It is a block diagram showing a configuration example of FIG. 24 image converter 231.
It is a flowchart illustrating an example of a decoding process in FIG. 25 the decoding apparatus 12.
FIG. 26 is a flowchart for explaining an example of a classification adaptive processing performed in step S122.
It is a diagram for explaining an example of a reduction method for reducing the tap coefficient for each [27] Class.
It is a block diagram showing a second configuration example of FIG. 28 encoding device 11.
It is a block diagram showing a configuration example of FIG. 29 classification adaptive filter 311.
Is a diagram illustrating an example of acquirable information used for the selection of the tap structure of FIG. 30 class tap.
[FIG. 31] is a block diagram showing an example of the configuration of a learning device 331.
Is a block diagram showing a configuration example of FIG. 32 the image conversion unit 371.
[FIG 33 is a flowchart illustrating an example of processing of the learning device 331.
It is a block diagram showing a configuration example of FIG. 34 image converter 333.
It is a flowchart illustrating an example of an encoding process in FIG. 35 encoding device 11.
[FIG. 36] is a flowchart for explaining an example of a classification adaptive processing performed in step S246.
Is a block diagram showing a second configuration example of FIG. 37 decoder 12.
It is a block diagram showing a configuration example of FIG. 38 classification adaptive filter 401.
It is a block diagram showing a configuration example of FIG. 39 image converter 431.
It is a flowchart illustrating an example of a decoding process in FIG. 40 the decoding apparatus 12.
[FIG. 41] is a flowchart for explaining an example of a classification adaptive processing performed in step S322.
[FIG. 42] is a diagram showing a multi-view image encoding method.
[FIG 43 is a diagram showing a main configuration example of the multi-view image encoding apparatus to which the present technology is applied.
[FIG. 44] is a diagram showing a main configuration example of the multi-view image decoding apparatus according to the present technology.
Is a diagram illustrating an example of FIG. 45 hierarchical image coding scheme.
[FIG. 46] is a diagram showing a main configuration example of the applied hierarchical image coding apparatus to which the present technology.
[FIG. 47] is a diagram showing a main configuration example of a hierarchical image decoding apparatus according to the present technology.
[FIG. 48] is a block diagram showing a main configuration example of a computer.
[FIG. 49] is a block diagram showing an example of a schematic configuration of a television device.
[FIG. 50] is a block diagram showing an example of a schematic configuration of a mobile phone.
[FIG. 51] is a block diagram showing an example of a schematic configuration of a recording and reproducing apparatus.
[FIG. 52] is a block diagram showing an example of a schematic configuration of an imaging apparatus.
[FIG. 53] is a block diagram showing an example of a schematic configuration of a video set.
[FIG. 54] is a block diagram showing an example of a schematic configuration of the video processor.
[FIG. 55] is a block diagram showing another example of a schematic configuration of the video processor.
DESCRIPTION OF THE INVENTION
[0016]
　
[0017]
　Figure 1 is a diagram illustrating a configuration example of an embodiment of an image processing system embodying the present technology.
[0018]
　1, the image processing system includes an encoding device 11 and decoding device 12.
[0019]
　The encoding apparatus 11, the original image of the encoding target is supplied.
[0020]
　Encoding device 11 may be, for example, by predictive coding such as HEVC or AVC (Advanced Video Coding), encodes the original image.
[0021]
　In predictive coding of the coding apparatus 11, the predicted image of the original image is generated, the residual between the original image and the prediction image is encoded.
[0022]
　Further, in the prediction encoding of the encoding device 11, the decoding process image obtained by adding the predicted image and the residual predictive coding, by performing the ILF process of applying the ILF, used to predict the predictive image reference image to be is generated.
[0023]
　Here, the filter processing as ILF processing (filtering) is an image obtained by being subjected to the decoding process image, also referred to as a post-filter image.
[0024]
　Encoding device 11, in addition to performing the predictive coding, if necessary, by performing learning, such as by using a decoding process image original image, the filtered image, ILF processing as possible close to the original image information about the filtering as a, can be obtained as the filter information.
[0025]
　ILF processing of the encoding device 11 can be performed by using filter information obtained by learning.
[0026]
　Here, learning for determining the filter information, for example, every 1 or more sequences of the original image, (from the scene change, a frame to the next scene change) one or more scenes of the original image by, in the original image 1 or each of a plurality of frames (pictures), each one or more slices of the original image, each one or more lines of blocks of units of coded pictures can be done other in arbitrary units. Also, learning for determining the filter information, for example, it can be performed if the residual is equal to or greater than a threshold value.
[0027]
　Encoder 11, the encoded data obtained by predictive coding of the original image, and transmitted via a transmission medium 13, or to record and transmit to the recording medium 14.
[0028]
　The encoding device 11, the filter information obtained by learning, and transmitted via a transmission medium 13, or may be recorded and transmitted to the recording medium 14.
[0029]
　The learning for obtaining the filter information can be performed in a different device from the encoding device 11.
[0030]
　The filter information can either be separately transmitted from the encoded data may be transmitted included in the encoded data.
[0031]
　Moreover, learning for determining the filter information, in addition carried out using the original image itself (and decoded middle image obtained from the original image), the image characteristic amount is similar to the original image is performed using a separate image from the original image be able to.
[0032]
　Decoder 12, the encoded data and the filter information required to be transmitted from the encoding device 11, (received) received via the transmission medium 13 and the recording medium 14 (acquired), the encoded data, encoding decoding in a manner that corresponds to the predictive coding apparatus 11.
[0033]
　That is, the decoding device 12, by processing the encoded data from the encoding device 11, obtains the residual predictive coding. Further, the decoding device 12, by adding the residual and the prediction image to obtain the same decoding process image and those obtained by the encoding device 11. Then, decoding device 12, the decoding process image, applies a filtering process as ILF process using as necessary filter information from the encoding device 11, it obtains the image after filtering.
[0034]
　In the decoding device 12, after filtering the image is output as a decoded image of the original image, if necessary, temporarily stored as a reference image used for prediction of the prediction image.
[0035]
　Filtering the ILF processing of the encoding device 11 and decoding device 12 may be performed by any filter.
[0036]
　Further, filtering of the encoding device 11 and decoding device 12 may be performed by classification adaptive processing (prediction operation). The following describes the classification adaptive processing.
[0037]
　
[0038]
　Figure 2 is a block diagram showing a first configuration example of an image conversion apparatus for performing classification adaptive processing.
[0039]
　Here, the classification adaptive processing, for example, it is possible to catch a first image, as an image conversion process for converting the second image.
[0040]
　Image conversion processing for converting the first image into the second image is a different signal processing depending on the definition of the first and second images.
[0041]
　That is, for example, together with the first image and the low spatial resolution image, if the second image with high spatial resolution image, the image conversion process is that the spatial resolution creation to improve the spatial resolution (improved) process be able to.
[0042]
　Further, for example, the first image with the image of the low S / N, if the second image and the image of the high S / N, the image conversion process is a noise removing processing for removing noise .
[0043]
　Furthermore, for example, while the first image and the image of a predetermined number of pixels (size), the second image, if the first number the number of pixels of the image or reduce the image, the image conversion process, it can be referred to resize processing for resizing the image (enlargement or reduction).
[0044]
　Further, for example, the first image, together with a decoded image obtained by decoding the encoded image in blocks, such as HEVC, the second image, if before encoding the original image, image conversion process is a distortion removing process for removing block distortion caused by encoding and decoding in block units.
[0045]
　Incidentally, the classification adaptive processing, other image, for example, acoustic, can be subjected to processing. Classification adaptive processing intended for the acoustics, the first acoustic (e.g., low acoustic like the S / N), and second acoustic (e.g., high acoustic like the S / N) as the sound conversion processing for converting the it can be captured.
[0046]
　In classification adaptive processing, the class obtained by classification into any class of a plurality of classes of pixel value of the target to which the pixel of interest (pixel to be processed to be processed) of the first image and the tap coefficients, the first image is selected for the pixel of interest, the prediction calculation using the pixel values ​​of the same number of pixels and the tap coefficients, the pixel value of the pixel of interest is calculated.
[0047]
　Figure 2 shows a configuration example of an image conversion apparatus for performing image conversion processing by the classification adaptive processing.
[0048]
　2, the image converting unit 20, the tap selecting section 21 and 22, the class classification unit 23, the coefficient obtaining unit 24, and includes a prediction calculation section 25.
[0049]
　The image converter 20, the first image is supplied. First image supplied to the image converter 20 is supplied to the tap selector 21 and 22.
[0050]
　Tap selection unit 21, the pixels constituting the first image, sequentially selects the pixel of interest. Further, the tap selector 21, the number of pixels constituting the first image used for predicting the corresponding pixel of the second image corresponding to the pixel of interest (pixel value of) (values ​​of pixels), the predicted It is selected as the tap.
[0051]
　Specifically, the tap selector 21, a plurality of pixels of the first image from the position of the time-space of the target pixel spatially or temporally close, select as prediction taps.
[0052]
　Tap selection unit 22, the pixel of interest, some of the pixels constituting the first image used to perform classification for classifying into one of several classes (value of the pixel), the class tap selected as. That is, the tap selection unit 22, the tap selection unit 21 in the same manner as selecting a prediction tap, selects a class tap.
[0053]
　Note that the prediction taps and the class taps may be one having the same tap structure, it may have different tap structures.
[0054]
　Prediction taps obtained by the tap selector 21 is supplied to the prediction computation unit 25, the class taps obtained by the tap selection unit 22 is supplied to a classification unit 23.
[0055]
　Class classification unit 23, according to a certain rule, the pixel of interest classification and supplies a class code corresponding to a class obtained as a result, the coefficient obtaining unit 24.
[0056]
　Namely, the class classifying unit 23, for example, using the class tap from the tap selection unit 22, the pixel of interest classification and supplies a class code corresponding to a class obtained as a result, the coefficient obtaining unit 24.
[0057]
　For example, the classification unit 23 uses the class tap to obtain the image feature quantity of the pixel of interest. Further, the class classification unit 23, in accordance with the image feature amount of the attention pixel, the target pixel classification, and supplies a class code corresponding to a class obtained as a result, the coefficient obtaining unit 24.
[0058]
　Here, as a method of performing classification, for example, it can be adopted ADRC (Adaptive Dynamic Range Coding) or the like.
[0059]
　In the method using ADRC, the pixels constituting the class taps (values ​​of pixels) are ADRC processed, in accordance with the resulting ADRC code (ADRC value), the class of the pixel of interest is determined. ADRC code represents the waveform patterns as the image feature amounts of the small region including the pixel of interest.
[0060]
　In the L-bit ADRC, for example, the maximum value MAX and the minimum value MIN of the pixel values of the pixels constituting the class tap are detected, the DR = MAX-MIN, a local dynamic range of a set, the dynamic range based on the DR, the pixel values of the pixels constituting the class tap are re-quantized to L bits. That is, the pixel value of each pixel forming the class taps, the minimum value MIN is subtracted, the subtraction value DR / 2 L divided by (re-quantization). Then, obtained as described above, the pixel value of each pixel of the L bits forming the class taps, the bit string arranged in a predetermined order is output as an ADRC code. Therefore, class taps, for example, if it is 1-bit ADRC process, a pixel value of each pixel forming the class taps is divided by the average of the maximum value MAX and the minimum value MIN (truncating decimal) , thereby, the pixel value of each pixel (binarized) are 1 bit. The bit string obtained by arranging the pixel values of the one bit in a predetermined order is output as an ADRC code.
[0061]
　Note that the class classification unit 23, for example, a pattern of level distribution of pixel values of the pixels constituting the class tap, it is also possible to directly output as the class code. However, in this case, the class tap is composed of pixel values of N pixels, the pixel value of each pixel, if the A bit is allocated, the number of cases of the class code class classification unit 23 outputs is, (2 N ) a becomes street, an enormous number exponentially proportional to the number of bits a of the pixel values of the pixels.
[0062]
　Therefore, the class classification unit 23, the information amount of the class tap, ADRC processing or above, or by compressing by vector quantization or the like is preferably performed classification.
[0063]
　Coefficient acquiring unit 24 stores the tap coefficient for each class obtained by learning described later, further, among the stored tap coefficients, the tap coefficient of the class represented by the class code supplied from the classification unit 23, that is, to obtain the tap coefficient of the class of the subject pixel. Further, the coefficient obtaining unit 24, the tap coefficients of the class of the subject pixel, and supplies the prediction computation unit 25.
[0064]
　Here, the tap coefficient in the digital filter is a coefficient corresponding to the coefficient to be multiplied with the input data in a so-called tap.
[0065]
　Prediction calculation section 25 uses the prediction taps output by the tap selector 21, and a tap coefficient coefficient acquisition unit 24 is supplied, the pixel value of the pixel of the second image corresponding to the pixel of interest (corresponding pixel) It performs predetermined prediction computation for determining the prediction value of the true value. Accordingly, the prediction computation unit 25, the pixel value of the corresponding pixel (predicted value), i.e., calculates and outputs the pixel values ​​of pixels constituting the second image.
[0066]
　Figure 3 is a block diagram showing a configuration example of a learning apparatus that performs learning of the tap coefficients stored in the coefficient acquisition unit 24.
[0067]
　Here, for example, high quality image (high quality image) with the second image, low reduced the image quality (resolution) the high-quality image by, for example filtering by LPF (Low Pass Filter) quality of images (low-quality image) as the first image, and selects the prediction taps from the low-quality image, by using the prediction tap and tap coefficients, the pixel values ​​of the pixels of the high quality image (high quality pixel), determined by a predetermined prediction calculation considering the (predicted to) be.
[0068]
　As the predetermined prediction computation, for example, when adopting a linear first-order prediction computation, pixel values ​​y of the high quality pixel can be determined by a linear expression following linear.
[0069]
[Number

　　　　　　　　　　　　　　　　　　　　　　　　　1] · · · (1)
[0070]
　However, in the formula (1), x n constitutes a prediction tap for the high image quality pixel y as the corresponding pixel, the pixel of the n-th low-quality image (hereinafter referred to as low as the image quality pixel) represents a pixel value of , w n denotes the n th n-th tap coefficient multiplied with the low-quality pixel (pixel value of). In formula (1), the prediction taps, N pieces of low-quality pixel x 1 , x 2 , · · ·, x N and be composed of.
[0071]
　Here, the pixel value y of the high quality pixel is not linear first-order equation shown in equation (1), it is also possible to obtain the second- or higher-order equations.
[0072]
　Now, the true value of the pixel value of high definition pixel of the k-th sample y k with expressed as, the true value y obtained by the equation (1) k the predicted value of y k expressed as' the prediction error e k It is expressed by the following equation.
[0073]
[Number

　　　　　　　　　　　　　　　　　　　　　　　　　2] · (2)
[0074]
　Now, equation predicted value y of (2) k 'is because it is determined according to equation (1), y of formula (2) k ' a, replacing according to equation (1), the following equation is obtained.
[0075]
[Number

　　　　　　　　　　　　　　　　　　　　　　　　　3] · (3)
[0076]
　However, in the formula (3), x n, k represents the n-th low quality pixel forming the prediction taps for the high quality pixel of the k-th sample of the corresponding pixel.
[0077]
　Prediction error e of the formula (3) (or Equation (2)) k tap coefficient w and a 0 n is is the optimal for predicting the high-quality pixel, for all the high-quality pixel, such a tap coefficient w n to seek is generally difficult.
[0078]
　Therefore, the tap coefficient w n as the standard for indicating that is optimal, for example, when adopting the method of least squares, optimum tap coefficients w n , the sum E of square errors expressed by the following formula ( the statistical error) may be determined by minimizing.
[0079]
[Number

　　　　　　　　　　　　　　　　　　　　　　　　　4] · (4)
[0080]
　However, in the formula (4), K is high definition pixel y as the corresponding pixel k and its image quality pixel y k low-quality pixels x constituting the prediction tap with respect to 1, k , x 2, k , · · · , x N, k represents a set number of samples of the (number of samples for learning).
[0081]
　The minimum value of the sum E of square errors of Equation (4) (minimum value), as shown in Equation (5), the sum E tap coefficient w n of those partially differentiated by w and 0 n is given by.
[0082]
[Number

　　　　　　　　　　　　　　　　　　　　　　　　　5] · · · (5)
[0083]
　Therefore, the above equation (3) the tap coefficient w n When partially differentiated by the following equation is obtained.
[0084]
[Number

　　　　　　　　　　　　　　　　　　　　　　　　　6] · (6)
[0085]
　From equations (5) (6), the following equation is obtained.
[0086]
[Number

　　　　　　　　　　　　　　　　　　　　　　　　　7] · · · (7)
[0087]
　E of the formula (7) k in, by substituting equation (3), equation (7) can be represented by normal equations shown in Equation (8).
[0088]
[Number

　　　　　　　　　　　　　　　　　　　　　　　　　8] · · · (8)
[0089]
　Normal equation of Equation (8), for example, by using a sweeping-out method (Gauss-Jordan elimination method) or the like, the tap coefficient w n can be solved for.
[0090]
　The normal equation of Equation (8), by solving for each class, the optimal tap coefficient (here, the tap coefficient that minimizes the sum E of square errors) w n a can be determined for each class .
[0091]
　Figure 3 is a tap coefficient w by solving the normal equations in equation (8) n shows a configuration example of a learning apparatus that performs learning for determining the.
[0092]
　3, the learning unit 30, the teacher data generating unit 31, the student data generating unit 32, and a learning unit 33.
[0093]
　The tutor data generating unit 31 and student data generating unit 32, the tap coefficients w n learning image used for learning is supplied. The learning image, for example, can be used a high resolution high-quality image.
[0094]
　Tutor data generating unit 31, from the learning image, the teacher data serving as a teacher of the tap coefficient learning (true value), i.e., as teacher data to be obtained by the classification adaptive processing, mapping of a prediction calculation according to formula (1) It generates a teacher image as a mapping destination, and supplies the learning unit 33. Here, the teacher data generating unit 31, for example, high-quality images as learning image, as a teacher image as it is supplied to the learning unit 33.
[0095]
　Student data generating unit 32, from the learning image, the student data serving as a student of the tap coefficient learning, i.e., as student data to be predictive operation of the tap coefficients in the classification adaptive processing, the prediction calculation performed by equation (1) It generates a student image to be converted by the mapping as supplied to the learning unit 33. Here, the student data generating unit 32, for example, by filtering the high-quality image as a training image LPF (low Pass Filter), by decreasing the resolution thereof, to generate a low-quality image, the low image quality an image, a student image, and supplies the learning unit 33.
[0096]
　Learning unit 33, the pixels constituting the student image as student data from the student data generating unit 32, sequentially, as a target pixel, for the pixel of interest, the same taps and the tap selector 21 of FIG. 2 is selected the pixel structure is selected as a prediction tap from the student image. Furthermore, the learning unit 33 uses the corresponding pixels constituting the teacher image corresponding to the pixel of interest, the prediction tap of the pixel of interest, for each class, by solving the normal equations in equation (8), the class determining the tap coefficients of each.
[0097]
　Figure 4 is a block diagram showing a configuration example of the learning unit 33 of FIG.
[0098]
　4, the learning unit 33, the tap selecting section 41 and 42, the classification unit 43, the adder 44, and has a coefficient calculation unit 45.
[0099]
　Student image is supplied to the tap selector 41 and 42, the teacher image is supplied to the adder 44.
[0100]
　Tap selector 41 selects the pixels forming the learner image, sequentially selected as the target pixel, information representing the target pixel, and supplies the required block.
[0101]
　Further, the tap selector 41, the pixel of interest, from the pixels constituting the student image, and select the prediction tap of the same pixel of the tap selector 21 of FIG. 2 is selected, thereby, obtained at the tap selector 21 to give the same prediction tap of the tap structure as is supplied to the adder 44.
[0102]
　Tap selection unit 42, for the pixel of interest, from the pixels constituting the student image, select the same pixel and the tap selector 22 of FIG. 2 selects the class tap, it is thereby, obtained by the tap selection unit 22 Noto obtain a class tap having the same tap structure, and supplies the class classifying unit 43.
[0103]
　Classification unit 43 uses the class tap from the tap selection unit 42 performs the same classification and the classification unit 23 of FIG. 2, a class code corresponding to the class of the pixel of interest obtained as a result of summation and outputs it to the part 44.
[0104]
　Adder 44, the configuration of pixels constituting the teacher image, and acquires the corresponding pixel corresponding to the pixel of interest (pixel value), the corresponding pixel, the prediction tap for the pixel of interest supplied from the tap selecting section 41 the summation targeting a pixel of the student image (values ​​of pixels) to be carried out for each class code supplied from the classification unit 43.
[0105]
　That is, the adder 44, the corresponding pixel y teacher image as teacher data k , the prediction tap x of the pixel of interest as student data n, k , the class code representing the class of the pixel of interest is supplied.
[0106]
　Adder 44, for each class of the subject pixel, prediction taps (student data) x n, k using, multiplication (x in each other learner data in the matrix on the left of formula (8) n, k x n ', k ) and performs an operation corresponding to the summation (sigma).
[0107]
　Further, the adder 44, again, for each class of the pixel of interest, the prediction taps (student data) x n, k and the teacher data y k using the student data x in the vector of the right side of equation (8) n, k and teacher data y k multiplication (x n, k y k performs a), a calculation corresponding to summation (sigma).
[0108]
　That is, the adder 44, the last, as teacher data, components of the matrix on the left in the determined for the corresponding pixel corresponding to the pixel of interest Equation (8) (? X n, k x n ', k ) and the right side component (? x a vector of n, k y k memory), and its internal stores (not shown), the components of the matrix (? x n, k x n ', k ) or components of a vector (? x n, k y k relative), the teacher data which is the corresponding pixel corresponding to the new pixel of interest, the teacher data y k + 1 and the student data x n, k + 1 is calculated using the corresponding component x to n, k + 1 x n ', k + 1 or x n, k + 1 y k + 1 Komu plus (performs an addition represented by the summation of equation (8)).
[0109]
　Then, the adder 44, for example, all pixels in the student image as a pixel of interest, by carrying out the summation of the above, for each class, set a normal equation shown in equation (8), the normal equation , and it supplies the coefficient calculation unit 45.
[0110]
　Coefficient calculating unit 45 solves the normal equations for each class supplied from the adder 44, for each class, the optimal tap coefficient w n outputs seeking.
[0111]
　The coefficient acquiring unit 24 of the image converter 20 of FIG. 2, the tap coefficients w for each class obtained as described above n can be stored.
[0112]
　Figure 5 is a block diagram showing a second configuration example of an image conversion apparatus for performing classification adaptive processing.
[0113]
　In the figure, parts corresponding to those in FIG. 2 are denoted with the same reference numerals, and description thereof will be omitted below as appropriate.
[0114]
　5, the image converter 20, the tap selecting section 21 and 22, the class classification unit 23, the coefficient obtaining unit 24, and includes a prediction calculation section 25.
[0115]
　Therefore, the image conversion device 20 of FIG. 5 is configured similarly to the case of FIG.
[0116]
　However, in FIG. 5, the coefficient obtaining unit 24 stores the seed coefficient to be described later. Further, in FIG. 5, the coefficient acquiring unit 24, a parameter z is supplied from the outside.
[0117]
　Coefficient acquiring unit 24, from the seed coefficients, corresponding to the parameter z, generates a tap coefficient for each class, the tap coefficient for each class, to obtain the tap coefficients of the class from the classification unit 23, prediction calculation supplied to the part 25.
[0118]
　In FIG. 2, the coefficient obtaining unit 24 is to store the tap coefficients themselves, in FIG. 5, the coefficient obtaining unit 24 stores the seed coefficient. Species coefficients by giving a parameter z (determined), it is possible to generate a tap coefficient, from this point of view, the seed coefficient can be regarded as equivalent of information and tap coefficients. In this specification, the tap coefficients, other tap coefficients themselves, and be included as necessary species coefficients capable of generating the tap coefficients.
[0119]
　Figure 6 is a block diagram showing a configuration example of a learning apparatus that performs learning of the species coefficients stored in the coefficient acquisition unit 24.
[0120]
　Here, for example, as with the case described in FIG. 3, high quality image (high quality image) with the second image, the low-quality image (low with a reduced spatial resolution of the high-quality image the quality image) as the first image, and selects the prediction taps from the low-quality image, by using the prediction tap and tap coefficients, the pixel value of high definition pixel is a pixel of the high-quality image, for example, the formula (1 determined by linear first-order prediction computation of) (considering the prediction to) be.
[0121]
　Now, the tap coefficient w n is the seed coefficient, and be produced by the following equation using the parameters z.
[0122]
[Number

　　　　　　　　　　　　　　　　　　　　　　　　　9] · · · (9)
[0123]
　However, in the equation (9), beta m, n are, n-th tap coefficient w n represents the m-th species coefficients used for determining the. In formula (9), the tap coefficient w n is, M-number of species coefficients beta 1, n , beta 2, n , · · ·, beta M, n is determined using.
[0124]
　Here, the seed coefficient beta m, n from the parameter z, the tap coefficient w n equations to obtain the, is not limited to equation (9).
[0125]
　Now, the value z depends on the parameter z in equation (9) m-1 a, a new variable t m by introducing, defined by the following equation.
[0126]
[Number

　　　　　　　　　　　　　　　　　　　　　　　　　10] · (10)
[0127]
　Equation (10) by substituting the equation (9), the following equation is obtained.
[0128]
[Number

　　　　　　　　　　　　　　　　　　　　　　　　　11] · (11)
[0129]
　According to equation (11), the tap coefficient w n is the seed coefficient beta m, n and the variable t m will be determined by a linear first-order equation with.
[0130]
　Meanwhile, the true value of the pixel values of the high-quality pixel of the k samples y k with expressed as, the true value y obtained by the equation (1) k the predicted value of y k expressed as' the prediction error e k is expressed by the following equation.
[0131]
[Number

　　　　　　　　　　　　　　　　　　　　　　　　　12] · (12)
[0132]
　Now, the prediction value y of formula (12) k 'is because it is determined according to equation (1), y of formula (12) k ' a, replacing according to equation (1), the following equation is obtained.
[0133]
[Number

　　　　　　　　　　　　　　　　　　　　　　　　　13] · (13)
[0134]
　However, in the equation (13), x n, k represents the n-th low quality pixel forming the prediction taps for the high quality pixel of the k-th sample of the corresponding pixel.
[0135]
　W of formula (13) n , the by substituting equation (11), the following equation is obtained.
[0136]
[Number

　　　　　　　　　　　　　　　　　　　　　　　　　14] · (14)
[0137]
　Prediction error e of the formula (14) k species coefficients and 0 beta m, n are, is the optimal for predicting the high-quality pixel, for all the high-quality pixel, such species coefficient beta m , n be obtained is generally difficult.
[0138]
　Therefore, the seed coefficient beta m, n as a norm representing that is optimal, for example, when adopting the method of least squares, best species coefficient beta m, n is square errors expressed by the following formula the sum E can be determined by minimizing.
[0139]
[Number

　　　　　　　　　　　　　　　　　　　　　　　　　15] · (15)
[0140]
　However, in the equation (15), K is high definition pixel y as the corresponding pixel k and its image quality pixel y k low-quality pixels x constituting the prediction tap with respect to 1, k , x 2, k , · · · , x N, k represents a set number of samples of the (number of samples for learning).
[0141]
　The minimum value of the sum E of square errors of Equation (15) (minimum value), as shown in equation (16), the sum E seed coefficient beta m, n and those partially differentiating the 0 beta m, n It is given by.
[0142]
[Number

　　　　　　　　　　　　　　　　　　　　　　　　　16] · (16)
[0143]
　Equation (13) by substituting the equation (16), the following equation is obtained.
[0144]
[Number

　　　　　　　　　　　　　　　　　　　　　　　　　17] · (17)
[0145]
　Now, X i, p, j, q and Y i, p , and is defined as shown in equation (18) and (19).
[0146]
[Number

　　　　　　　　　　　　　　　　　　　　　　　　　18] · (18)
[0147]
[Number

　　　　　　　　　　　　　　　　　　　　　　　　　19] · (19)
[0148]
　In this case, equation (17), X i, p, j, q and Y i, p can be expressed by normal equations shown in equation (20) using.
[0149]
[Number

　　　　　　　　　　　　　　　　　　　　　　　　　20] · (20)
[0150]
　Normal equation of Equation (20), for example, by using a sweeping-out method (Gauss-Jordan elimination method) or the like, seed coefficient beta m, n can be solved for.
[0151]
　In the image conversion apparatus 20 of FIG. 5, a number of high-quality pixel y 1 , y 2 , · · ·, y K with the training data, each high-quality pixel y k low-quality pixels x constituting the prediction tap with respect to 1, k , x 2, k , · · ·, x N, k as student data, the species coefficient for each class obtained by performing the vertical normal equations are solved learning of formula (20) for each class beta m, n are stored in the coefficient obtaining unit 24. Then, the coefficient obtaining unit 24, the seed coefficient beta m, n and, from the parameter z supplied from the outside, according to Equation (9), the tap coefficients w for each class n is generated, the prediction computation unit 25, the tap coefficient w n and, x (pixels of the first image) low quality pixel forming the prediction taps for the pixel of interest n using, by the formula (1) is calculated, high definition pixel (second image pixel values of the corresponding pixels) (predicted value close to) are obtained.
[0152]
　6, by solving the normal equations in equation (20) for each class, species coefficient β for each class m, n shows a configuration example of a learning apparatus that performs learning for determining the.
[0153]
　In the figure, parts corresponding to those in the case of FIG. 3 are denoted with the same reference numerals, and description thereof will be omitted below as appropriate.
[0154]
　6, the learning apparatus 30, the tutor data generating unit 31, the parameter generating unit 61, the student data generating unit 62, and a learning unit 63.
[0155]
　Thus, the learning apparatus 30 of FIG. 6, in that it has a tutor data generating unit 31, common to that of FIG.
[0156]
　However, the learning apparatus 30 of FIG. 6, in that it has a new parameter generating unit 61, differs from the case of FIG. Further, the learning apparatus 30 of FIG. 6, in place of the student data generating unit 32 and the learning unit 33, the student data generating unit 62, and in that the learning unit 63 are respectively provided, different from the case of FIG. 3 .
[0157]
　Parameter generating unit 61 generates a number of values ​​in the range that can take the parameter z, and supplies the student data generating unit 62 and the learning unit 63.
[0158]
　For example, if the value that can take the parameter z is assumed to be real numbers in the range 0 to Z, the parameter generating unit 61, for example, for example, z = 0, 1, 2, · · ·, a parameter z of the value of Z generated, and supplies the student data generating unit 62 and the learning unit 63.
[0159]
　The student data generating unit 62, the same learning image to that supplied to the tutor data generating unit 31 is supplied.
[0160]
　Student data generating unit 62, similarly to the student data generating unit 32 of FIG. 3, to generate a student image from the learning image, as student data, and supplies the learning unit 63.
[0161]
　Here, the student data generating unit 62, other learning image, several values ​​of the range that can take the parameter z is supplied from the parameter generating unit 61.
[0162]
　Student data generating unit 62, a high-quality image as a training image, for example, by filtering by LPF cutoff frequency corresponding to the parameter z supplied thereto, for each several values ​​of the parameter z , to produce a low-quality image as a student image.
[0163]
　That is, in the student data generating unit 62, the high-quality images as learning image, the Z + 1 type, the low-quality image as a different student image spatial resolution is generated.
[0164]
　Here, for example, the larger the value of the parameter z, using a high cut-off frequency LPF, filtering the high-quality image, and generating a low-quality image as a student image. In this case, for the large parameter z of value as low-quality image as a student image, a high spatial resolution.
[0165]
　Further, the student data generating unit 62, generates a low quality image in response to the parameter z, as the horizontal direction and one or both of the student image with reduced spatial resolution of the vertical direction of the high-quality images as learning image can do.
[0166]
　Furthermore, when generating the horizontal and low-quality image as a student image with a reduced both the spatial resolution of the vertical direction of the high-quality images as learning image, horizontal high-quality images as learning image and vertical spatial resolution, each separate parameter, i.e., can be two in accordance with the parameter z and z ', it is separately reduced.
[0167]
　In this case, the coefficient obtaining unit 24 of FIG. 5, two parameters z and z 'is given, the two parameters z and z' from the outside by using the seed coefficient, the tap coefficients are generated.
[0168]
　As described above, the seed coefficients, other one parameter z, 2 pieces of parameter z and z ', further, by using three or more parameters, the seed coefficients can generate tap coefficients it can be determined. However, in this specification, for simplicity of explanation, as an example species coefficients to generate tap coefficients using a single parameter z, will be described.

claims

The processing target pixel in the first image obtained by adding the residual of the prediction coding and the prediction image, a class tap used for class classification for classifying any class of a plurality of classes pixels and by selecting from said first image, and the class tap selection unit constituting the class tap,
　by using the class tap, and the class classification section that performs classification of the target pixel,
　the first the image, to filter corresponding to the class of the target pixel, the filter processing unit that generates a second image used for prediction of the predicted image
　comprises a,
　the class tap selection unit, the class tap the tap structure is updated with the selected tap structure from a plurality of tap structures
　the image processing apparatus.
[Requested item 2]
　Further comprising a transmission unit for transmitting the filter information about the filtering processing
　image processing apparatus according to claim 1.
[Requested item 3]
　The filtering process unit,
　　the pixels serving as prediction taps used in prediction calculation for obtaining the pixel value of a corresponding pixel of the second image corresponding to the processing target pixel of the first image, from the first image by selecting the prediction tap selection unit constituting the prediction tap,
　　and the student image corresponding to the first image, by learning using the teacher image corresponding to the original image corresponding to said first image determined, for each of the classes, and the prediction operation on one of the tap coefficient used, the tap coefficient acquisition unit for acquiring the tap coefficients of the class of the target pixel,
　　the tap coefficients of the class of the target pixel, the processing by performing the predictive calculation using said prediction tap of the pixel of interest, a calculation unit for determining the pixel values of the corresponding pixels
　having
　image processing apparatus according to claim 2.
[Requested item 4]
　The filter information includes a tap coefficient for each said class
　image processing apparatus according to claim 3.
[Requested item 5]
　The filter information may include a tap structure information indicating the selected tap structure from the plurality of tap structures
　image processing apparatus according to claim 2.
[Requested item 6]
　Depending on the tap structure evaluation value representing the appropriateness of using each of the plurality of tap structures in the classification, from among the plurality of tap structures, further comprising a selection unit for selecting a tap structure of the class tap
　claim 2 the image processing apparatus according to.
[Requested item 7]
　Using said acquirable information obtainable from the coded data obtained by predictive coding, in accordance with selection rules for selecting a tap structure of the class tap from the plurality of tap structures, among the plural tap structures further comprising a tap structure selection unit for selecting a tap structure of the class tap
　image processing apparatus according to claim 2.
[Requested item 8]
　The filter information may include the selection rule
　image processing apparatus according to claim 7.
[Requested item 9]
　The obtainable information, the image feature amount obtained from the first image, and, one of the coded information about the predictive coding of the processing target pixel, or both a is
　image processing according to claim 7 apparatus.
[Requested item 10]
　The filter information, as a tap structure of the class tap includes copy information indicating whether use the same tap structure as when updating the immediately preceding tap structure
　image processing apparatus according to claim 2.
[Requested item 11]
　Further comprising a receiving unit for receiving the filter information about the filtering processing
　image processing apparatus according to claim 1.
[Requested item 12]
　The filtering process unit,
　　the pixels serving as prediction taps used in prediction calculation for obtaining the pixel value of a corresponding pixel of the second image corresponding to the processing target pixel of the first image, from the first image by selecting the prediction tap selection unit constituting the prediction tap,
　　and the student image corresponding to the first image, by learning using the teacher image corresponding to the original image corresponding to said first image determined, for each of the classes, and the prediction operation on one of the tap coefficient used, the tap coefficient acquisition unit for acquiring the tap coefficients of the class of the target pixel,
　　the tap coefficients of the class of the target pixel, by performing the prediction operation using said predictive tap of the processing target pixel, and a calculation unit for determining the pixel values of the corresponding pixels
　having
　image processing apparatus according to claim 11.
[Requested item 13]
　The filter information includes a tap coefficient for each said class,
　said tap coefficient acquiring unit, wherein the tap coefficients for each of the classes included in the filter information, and acquires the tap coefficients of the class of the target pixel
　according to claim 12 the image processing apparatus according to.
[Requested item 14]
　The filter information may include a tap structure information indicating the selected tap structure from the plurality of tap structures,
　the class tap selection unit, the tap structure of the tap structure of the class tap, are included in the filter information updating the tap structure information indicating
　the image processing apparatus according to claim 11.
[Requested item 15]
　The filter information may include a selection rule for selecting a tap structure of the class tap from the plurality of tap structures,
　by using the predicted code acquirable information obtainable from the resulting encoded data by reduction, the selection rule accordance, from the plurality of tap structures, further comprising a tap structure selection unit for selecting a tap structure of the class tap
　image processing apparatus according to claim 11.
[Requested item 16]
　The obtainable information, the image feature amount obtained from the first image, and, one of the coded information about the predictive coding of the processing target pixel, or both a is
　image processing according to claim 15 apparatus.
[Requested item 17]
　The filter information, as a tap structure of the class tap includes copy information indicating whether use the same tap structure as when updating the immediately preceding tap structure,
　the class tap selection unit is configured to be included in the filter information depending on the copy information, as a tap structure of the class tap, to select the same tap structure as when updating the immediately preceding tap structure
　image processing apparatus according to claim 11.
[Requested item 18]
　The filtering process unit, ILF (In Loop Filter) constituting the DF (Deblocking Filter), SAO ( Sample Adaptive Offset), and, ALF function as one or more of (Adaptive Loop Filter)
　of claim 1 image processing apparatus.
[Requested item 19]
　The processing target pixel in the first image obtained by adding the residual of the prediction coding and the prediction image, a class tap used for class classification for classifying any class of a plurality of classes pixels and by selecting from said first image, and configuring the class tap,
　by using the class taps, and to perform the classification of the target pixel,
　the first image, the processing to filter corresponding to the class of the target pixel, and generating a second image used for the prediction of the predicted image
　comprises,
　a tap structure of the class tap is selected from a plurality of tap structures updating the tap structure
　image processing method.