Technical field
[0001]
The present disclosure relates to an image processing apparatus, an image processing method, and a program.
BACKGROUND
[0002]
Conventionally, many video encoding schemes for efficiently encoding video has been standardized. For example, H. of ITU-T and ISO / IEC jointly developed by 264 / AVC (Advanced Video Coding) is a standard encoding scheme successor MPEG-2 (see Non-Patent Document 1). H. 264 / AVC, the prediction, orthogonal transformation, using various elemental technologies such quantization and entropy coding, to encode into the bit stream image signal with improved coding efficiency. In addition, H. 264 / AVC is a standard encoding method for successor H. 265 / HEVC (High Efficiency Video Coding) is about 4-fold when compared to MPEG-2, H. Compared to 264 / AVC is said to achieve approximately twice the coding efficiency (see Non-Patent Document 2).
[0003]
　Apart from the pursuit of high coding efficiency, in recent years, in order to be able to display an image the state of the real world in a more faithful reproduction or richer brightness and color, expansion of the video signal representations It has been promoted. HDR (High Dynamic Range) is a concept of trying to represent an image or video in a wide luminance dynamic range than a conventional standard dynamic range SDR (Standard Dynamic Range). For example, HLG (Hybrid Log-Gamma), converted into an image signal light by a transfer function such ST2084 or S-log3 (also referred to as a tone curve) (and, the image signal converted into light) by, than 100nit it is known that it is possible to reproduce light of the real world with a high luminance on the display (see non-Patent Document 3 for HLG). As another example, standardized by ITU-R BT. 2020, have been used in many applications so far BT. 709 as compared to the color gamut of, defines a color gamut to be able to express a more vivid color.
CITATION
Non-patent literature
[0004]
Non-Patent Document 1: ITU-T, "H.264 : Advanced video coding for generic audiovisual services", ITU-T Recommendation H.264, November 2007
Non-Patent Document 2: ITU-T, "H.265 : High efficiency video coding ", ITU-T Recommendation H.265, 2014 10 years dated
non-Patent Document 3: Association of Radio Industries and Businesses ," ESSENTIAL PARAMETER VALUES FOR THE EXTENDED IMAGE DYNAMIC RANGE TELEVISION (EIDRTV) SYSTEM FOR PROGRAMME PRODUCTION ARIB STANDARD " , ARIB STD-B67 Version 1.0, July 3, 2015, [online], [2016 dated 24 years. 11, 2005 search],イnn center have NekoッSuites
Summary of the Invention
Problems that the Invention is to Solve
[0005]
　Coding the video (or individual images constituting the image), decoding, or signal conversion, existing devices involved in the time of imaging or playback is not possible always well adapted to the video signal representation that is being diversified . It is considered similar situation for the image processing of the still image.
[0006]
　Regardless scheme of the signal representation, it is desirable that provides a mechanism that can provide a good image quality.
Means for Solving the Problems
[0007]
　According to the present disclosure, an encoding unit for encoding an image obtained on the basis of the transfer function for the conversion between the optical and the image signal, based on the transfer function, each of the image in the encoding unit the image processing apparatus is provided and a control unit for controlling the amount of codes allocated to the partial regions.
[0008]
　Further, according to the present disclosure, encoding the image obtained on the basis of the transfer function for the conversion between the optical and the image signal, based on the transfer function, of the image during the encoding and controlling the amount of codes allocated to each partial area, an image processing method comprising is provided.
[0009]
　Further, according to the present disclosure, the processor of the image processing apparatus, an encoding unit for encoding an image obtained on the basis of the transfer function for the conversion between the optical and the image signal, based on the transfer function, program for functioning as a control unit for controlling the amount of codes allocated to each partial region of the image in the encoding unit is provided.
[0010]
　Further, according to the present disclosure, an encoding unit for encoding an image obtained on the basis of the transfer function for the conversion between the optical and the image signal for enabling the display of a higher than 100nit brightness, the the code amount assigned to each partial region of the image in the encoding unit, the image processing device is provided and a control unit for controlling in dependence on at least one of luminance components and chrominance components of the partial region .
[0011]
　Further, according to the present disclosure, encoding the image obtained on the basis of the transfer function for the conversion between the optical and the image signal for enabling the display of a higher than 100nit luminance, the coded wherein the code amount assigned to each partial area of ​​the image, the image processing method comprising controlling as a function of at least one, the one of luminance components and chrominance components of the partial region is provided in the.
[0012]
　Further, according to the present disclosure, coding the processor of the image processing apparatus, an image is obtained based on a transfer function for the conversion between the optical and the image signal for enabling the display of a higher than 100nit luminance features and coding unit, a code amount assigned to each partial region of the image in the encoding unit, as a control unit for controlling in dependence on at least one of luminance components and chrominance components of the partial areas program for causing a is provided.
[0013]
　Further, according to the present disclosure, an encoding unit for encoding an image obtained on the basis of the transfer function for the conversion between the optical and the image signal, based on the transfer function, the image in the encoding unit a control unit for controlling the prediction residual code amount or mode code amount for the mode selection when encoding, the image processing apparatus including a is provided with.
[0014]
　Further, according to the present disclosure, encoding the image obtained on the basis of the transfer function for the conversion between the optical and the image signal, based on the transfer function, at the time of encoding the image mode the image processing method comprising controlling the prediction residual code amount or mode code amount for selection, is provided.
[0015]
　Further, according to the present disclosure, the processor of the image processing apparatus, an encoding unit for encoding an image obtained on the basis of the transfer function for the conversion between the optical and the image signal, based on the transfer function, program for functioning as a control unit that controls the prediction residual code amount or mode code amount for the mode selection when encoding the image in the encoding unit is provided.
Effect of the invention
[0016]
　According to the technique of the present disclosure, regardless of the type of signal representation can provide a good image quality.
　Incidentally, the above effect is not necessarily restrictive, with the above effects, or instead of the above effects, any effects shown herein, or other effects that may be grasped from the description it may be achieved.
BRIEF DESCRIPTION OF THE DRAWINGS
[0017]
Is an explanatory diagram for describing luminance dynamic range of FIG. 1A] SDR image.
[FIG 1B] is an explanatory diagram for describing luminance dynamic range of HDR images.
Is an explanatory diagram for describing a codec distortion of the image signal in FIG. 2A] SDR image.
Is an explanatory diagram for describing a codec distortion of the image signal in FIG. 2B] HDR image.
Is an explanatory diagram for describing a codec distortion is enlarged through FIG 2C] HDR-SDR conversion.
3 is an explanatory diagram showing an example of a OETF signal format and signal format for HDR for SDR.
[Figure 4] BT for the SDR. S-log3 for HDR is a graph representing how much compressed image information to 709.
[Figure 5] BT. 709 and BT. It is an explanatory diagram for describing the color gamut defined by 2020.
[FIG 6A] is an explanatory diagram showing a first example of a configuration of an image processing system according to an embodiment.
[FIG 6B] is an explanatory diagram showing a second example of a configuration of an image processing system according to an embodiment.
[FIG 7A] is a block diagram showing a first example of a schematic configuration of an image processing apparatus according to the first embodiment.
[FIG 7B] is a block diagram showing a second example of a schematic configuration of an image processing apparatus according to the first embodiment.
8 is a block diagram showing an example of a detailed configuration of the control unit according to the first embodiment and the coding unit.
It is an explanatory diagram for describing a first example of a protection ratio for protecting FIG 9A] high-intensity part.
It is an explanatory diagram for describing a second example of the protection ratio for protecting FIG 9B] high-intensity part.
It is an explanatory diagram for FIG. 10 for the code value to be protected as Koirosa portion will be described.
11 is an explanatory diagram for describing an example of a protection ratio for protecting Koirosa portion.
12 is a flowchart showing an example of the flow of encoding control processing according to the first embodiment.
Is a flowchart showing an exemplary flow of a quantization control process to protect the gradation of FIG 13A] high-intensity part.
Is a flowchart showing an exemplary flow of a quantization control process to protect the gradation of FIG 13B] Koirosa portion.
Is a flowchart showing an exemplary flow of a quantization control process to protect both the tone of FIG. @ 13 C] high-intensity part and Koirosa moiety.
14 is a block diagram showing a modification of the configuration of an image processing apparatus according to the first embodiment.
15 is a flowchart illustrating an example of a flow of encoding control processing according to the modified example described with reference to FIG. 14.
It is a first explanatory diagram for explaining the effect of the difference of the transfer function of the FIG. 16A] mode selection.
It is a second explanatory diagram for explaining the effect of the difference of the transfer function of the FIG. 16B] mode selection.
[FIG. 17A] is a block diagram showing a first example of a schematic configuration of an image processing apparatus according to the second embodiment.
[FIG. 17B] is a block diagram showing a second example of a schematic configuration of an image processing apparatus according to the second embodiment.
18 is a block diagram showing an example of a detailed configuration of the control unit according to the second embodiment and the coding unit.
19 is an explanatory diagram for explaining an example of switching of the mode code amount based on the type of transfer function.
FIG. 20 is a flowchart showing an exemplary flow of an encoding control process according to the second embodiment.
Is a block diagram showing an example of the hardware configuration of FIG. 21 apparatus.
22 is a diagram schematically showing an overall configuration of the operating room system.
23 is a diagram showing a display example of operation screen in the intensive operation panel.
It is a diagram showing an example of the state of FIG. 24 operating room surgery system is applied.
Is a block diagram illustrating an example of FIG. 25 of the camera head and CCU shown in FIG. 24 functional configuration.
DESCRIPTION OF THE INVENTION
[0018]
　Reference will now be described in detail preferred embodiments of the present disclosure. In the specification and the drawings, components having substantially the same function and structure are a repeated explanation thereof by referring to the figures.
[0019]
　Also, a description in the following order.
　　1. Description of the Related Art
　　　1-1. SDR and
　　　HDR 1-2. Codec distortion
　　　1-3. Transfer function
　　　1-4. Color gamut
　　2. First Embodiment
　　　2-1. Introduction
　　　2-2. System Overview
　　　2-3. Schematic configuration of the image processing apparatus
　　　2-4. Detailed structure of the encoding unit and the control unit
　　　2-5. Processing of flow
　　　2-6. Modification
　　　2-7. Summary of the first embodiment
　　3. Second Embodiment
　　　3-1. Introduction
　　　3-2. System Overview
　　　3-3. Schematic configuration of the image processing apparatus
　　　3-4. Detailed structure of the encoding unit and the control unit
　　　3-5. Processing of flow
　　　3-6. Summary of the second embodiment
　　4. Hardware Configuration Example
　　5. Application Example
　　6. Summary
[0020]
　<1. Description of Related
　　Art> [1-1. SDR and HDR]
　In recent years, in order to be able to play a video the state of the real world in a more faithful reproduction or richer brightness and color, expansion of the video signal representation has been promoted. HDR is a concept attempts to represent an image or video in a wide luminance dynamic range than a conventional standard dynamic range SDR.
[0021]
　Figure 1A is an explanatory diagram for describing luminance dynamic range SDR image. The vertical axis of FIG. 1A represents luminance [nit]. Maximum brightness in nature may reach 20000Nit, luminance of a typical object is 12000nit about a maximum, for example. The upper limit of the dynamic range of the image sensor may be a maximum lower than the luminance, for example 4000nit in nature. Digital camera or digital camcorder such as imaging devices, an electric signal generated by photoelectric conversion of incident light in the image sensor, converts the subsequent signal processing circuit of the image sensor, for example to 10 bits of digital image signals. The signal format of the legacy SDR image, gradation of the high luminance portion over a 100nit during such conversion is lost. Digital image signal generated by the imaging device, for example, depending on the purpose of application such as transmission or recording, is encoded in a predetermined video coding method (also referred to as video codecs) are converted into coded bit stream. Then, when displaying the SDR video, digital image signals obtained by decoding the encoded bit stream is provided to the display device, the image is reproduced by the display luminance of the upper 100 nit.
[0022]
　Figure 1B is an explanatory diagram for describing luminance dynamic range of HDR images. Like the SDR case, the imaging device, the incident light to the image sensor and converted into an analog electrical signal, further converts the analog electrical signal into for example 10-bit digital image signal. Signal format of HDR images, when such conversion, to maintain the gradation of a high luminance portion exceeding 100 nit, makes it possible to reproduce the video the luminance of up to an upper limit of several hundred or thousand nit. Digital image signal generated by the imaging device is encoded in a predetermined video coding scheme also depending on the purpose of application, it is converted into coded bit stream. When displaying the HDR image is a digital image signal obtained by decoding the encoded bit stream is provided to the display device, the image is reproduced by the luminance dynamic range, including high display luminance than 100 nit.
[0023]
　Incidentally, as the reference for classifying the SDR and HDR, where it is assumed the case where the upper limit of the luminance dynamic range is below or equal to 100 nit SDR, a case where the upper limit is above 100 nit and the HDR. However, at some point in the future, the higher the reference value rather than the 100 nit, widespread at that time (i.e., has become the standard) and dynamic range (having a higher upper limit) newer and a dynamic range each may be be classified as HDR and SDR. The technology according to the present disclosure generally two types of dynamic range is widely applicable to the case having a different upper limit each other, but not limited to whether they dynamic range is classified by what kind of reference values. SDR, in comparison with HDR, sometimes referred to as LDR (Low Dynamic Range).
[0024]
　　[1-2. Codec Distortion
　whether SDR image or HDR image, when coded by the video coding method comprising the lossy compression of the image signal, the image reproduced on the basis of the image signal to be decoded, the deterioration of the image quality occurs. The deterioration of these image quality, herein referred to as codec distortion. The degree of codec distortion can be evaluated in the indication that PSNR (Peak Signal-to-Noise Ratio). Generally, when the coding efficiency equivalent, H. 264 / quality of the encoding / decoding images in AVC is higher than the image quality of the encoded / decoded images MPEG-2, H. 265 / quality of the encoding / decoding images in HEVC is H. Higher than the 264 / AVC. However, usually, the evaluation of the codec distortion is performed by comparing the decoded image output from the original image and the decoder input to the encoder. Signal conversion performed when the imaging or display of the HDR image, or either reduction or expansion of the dynamic range acts how the codec distortion, is not well known.
[0025]
　It, converted into an image signal a number of samples the video in signal format for HDR, H. After encoding and decoding by the encoder and decoder compliant with 264 / AVC, an experiment was conducted to verify the quality of HDR images reproduced from the image signal after decoding. As a result, the same sample in deterioration of the image quality that was not detected in the SDR video, that there is a case to be visible in HDR image through the video codec is recognized. Degradation of image quality, mainly in the form such as block noise or mosquito noise, as well as scattered throughout the image, had occurred remarkably in a portion of the image.
[0026]
　The degree of degradation resulting in encoding the image signal of the same 10-bit with the same video coding scheme is usually comparable. But the sensed no (or sensed hardly) strain in SDR image is detected in the HDR image is believed to be because the codec distortion is expanded together in extending the dynamic range of the image signal after decoding.
[0027]
　FIG 2A, the image signal of the SDR image, how codec distortion through encoding and decoding occurs is shown. Since codec distortion is not expanding when reproducing SDR video, does not strain the subjective distortion sufficiently small sensed. On the other hand, in FIG. 2B shows a state which is also codec distortion occurs in the image signal of the HDR image. When reproducing an HDR image is a result of the codec distortion with the expansion of the dynamic range is expanded, more likely to deteriorate in image quality such as block noise or mosquito noise is detected subjectively it is.
[0028]
　Codec distortion can be enlarged even when the format conversion into SDR is run from HDR an image signal represented by a signal format for HDR. The Figure 2C, the format conversion to SDR from HDR, i.e. how the codec distortion is enlarged through the HDR-SDR conversion is shown. HDR-SDR conversion is generally to an original signal corresponding with the inverse of the transfer function corresponding to the signal format for HDR (e.g. obtained by decoding the coded bit stream) image signal to an output of the image sensor including a process for restoring, and processing for re-converted into an image signal for SDR transfer function corresponding to the signal format for SDR from the restored original signal. Codec distortion is enlarged in the former of these processes are not reduced in the re-conversion to the signal format for SDR. Therefore, when playing SDR image based on the image signal after HDR-SDR conversion, enlarged codec distortion may result sensed subjectively.
[0029]
　Codec distortion as described above, if due to the performance of the video encoding method itself, the distortion should occur uniformly. However, the verification of the sample image as described above, in addition to the distortion dispersed throughout the image, be distorted in a characteristic partial regions as illustrated has become remarkable was confirmed as follows:
　　- high luminance region (e.g., cloud in the sky)
　　- vivid region color (e.g., glow lamp red or blue)
cause distortion becomes conspicuous in these partial areas is related to the signal transfer function of the signal format for HDR .
[0030]
　　[1-3. Transfer function]
　In general, characteristics of the signal conversion to the image signals from the light in the imaging device, OETF; is modeled with (Opto-Electronic Transfer Function photoelectric transfer function). Figure 3 shows examples of each of OETF typical OETF and signal format for HDR signal format for SDR. 3, the horizontal axis represents the luminance dynamic range of the pre-conversion of light, 100% corresponds to the luminance of 100 nit. The vertical axis represents the sign value of the image signal after the conversion, in the case of 10bit the code value can take a value from 0 to 1023. Signal format for SDR shown by a broken line in FIG. (E.g., BT.709) OETF (e.g., HLG, ST2084 or S-log3) for HDR shown in OETF and solid line when the comparing, code value is particularly the difference of the slope of the transfer function at a relatively large portion is remarkable. This is because, in this part, the image information compared to the SDR in HDR case is compressed at a higher compression ratio, i.e. the degree of change in code value is greater than the case of the SDR in case of HDR floors It is meant to represent a change of tone. Red (R) component in the RGB color system, a green even when analyzing each of the transfer function of the component (G) and blue (B) component, between similar HDR and SDR to the graph shown in FIG. 3 the difference of the signal transfer characteristic was confirmed.
[0031]
　4, BT for the SDR. S-log3 for HDR is a graph representing how much compressed image information to 709. 4, the horizontal axis represents the sign value of 10 bit image signal. The vertical axis, BT. It represents the ratio of the compression ratio of the S-log3 for 709 compression ratio. In the code value "600" near the S-log3 equivalent to 100% of the luminance dynamic range, the compression ratio of the S-log3 is BT. 709 is about 4 times the compression ratio, the compression ratio of the code values ​​larger the S-log3 is higher than relatively. From this graph, the code value is at a relatively large part, in the case of HDR is understood that the image information is compressed more strongly as compared with the SDR case.
[0032]
　When reproducing an HDR image is often, EOTF is an inverse function of OETF as indicated by the solid line in FIG. 3; to apply (Electro-Optical Transfer Function electric optical transfer function) to the code value of the image signal it causes the voltage level to be supplied to the display element can be determined. Then, the individual images constituting the HDR image is displayed in an expanded luminance dynamic range by the application of EOTF. Called OOTF the transfer function of the entire system including the up display from the imaging, OOTF is sometimes referred to as a system gamma. As used herein, "transfer function", unless otherwise specifically stated, OETF, refers to any one or more combinations of EOTF and OOTF. These transfer functions are sometimes referred to as the tone curve.
[0033]
　　[1-4. Gamut]
　As a technique that enables more faithful reproduction or from rich images representing the real-world situation, it is also important concept gamut with HDR. Standardized by ITU-R has been BT. 2020, have been used in many applications so far BT. 709 as compared to the color gamut of, defines a color gamut to be able to express a more vivid color. 5, BT. 709 and BT. It is an explanatory diagram for describing the color gamut defined by 2020. Referring to FIG. 5, the color gamut graph mapping the three-dimensional color space into a two-dimensional plane to a predetermined restricting condition is shown. Cross mark in the graph shows the position where the white color is mapped. The broken line in the graph, BT. It shows the range of colors that can be expressed according to 709. The solid line in the graph, BT. It shows the range of colors that can be expressed according 2020. The dotted line in the graph indicates the range of colors that can be human vision to identify. As understood from FIG. 5, BT. 2020, BT. Than 709 makes it possible to express a variety of colors. BT. 709 While it is possible representation is about 75 percent of the colors that exist in the real world, BT. 2020 is said to be possible to express the color of more than 99%. BT. 2020 may be utilized as a color gamut of SDR video, or may be utilized as the color gamut of the HDR video.
[0034]
　<2. First
　　Embodiment> [2-1. Introducing]
　Some of the above-described codec distortion becomes remarkable in the partial areas in the image when using a signal format for HDR, particularly relatively large code values of the dynamic range of each color component of the image signal in the corresponding subrange, assigned code amount for representing the gray level of the original signal is caused to be insufficient. MPEG-2, H. 264 / AVC or H. Encoder that conforms to the video encoding system such as 265 / HEVC, in order to achieve the required compression ratio is quantized in the frequency domain image signal. Typically, transform coefficients obtained by orthogonal transformation of the prediction residual after applying the prediction technique such intra prediction or inter prediction is quantized. However, the quantization step used by the optimized them encoder for encoding the SDR image is often too large when the signal format for HDR can be used. This, in sub-range corresponding to the higher code value is already the fact that the tone information when the signal conversion (than SDR case) is strongly compressed, because existing encoder does not consider.
[0035]
　Therefore, this section is to better preserve the gradation allocated more code amount by the partial area gradation is compressed strongly in the image when a signal format for HDR is used, reduce the degradation of image quality An embodiment for explaining.
[0036]
　　[2-2. System Overview
　Figure 6A is an explanatory diagram showing a first example of a configuration of an image processing system according to the present embodiment. The image processing system 10a shown in Figure 6A, includes an imaging device 11, the signal processing device 14 and server device 15,.
[0037]
　Imaging device 11, for example, a digital video camera or digital still camera, or any type of device having an image capturing function (e.g., a surveillance camera, Web cameras or data terminal, etc.). Imaging device 11, by taking a picture of the real world using an image sensor, and generates a primitive image signal. The signal processing device 14 can be, for example, BPU (Baseband Processing Unit), is connected to the imaging device 11. The signal processing unit 14, the primitive image signal generated by the imaging device 11 performs AD conversion and digital signal processing to generate an image signal in a predetermined signal format. Digital signal processing performed by the signal processor 14 may include, for example, gamma correction and color conversion. The signal processing unit 14, the imaging device 11 may be integrally constructed.
[0038]
　Characteristics of signal conversion to the image signal generated by the signal processor 14 from the light incident to the imaging device 11 is represented by OETF. For example, the signal processor 14 may generate an image signal by the transfer function (or signal format) is selected from a plurality of candidates by the user via some user interface. As an example, a plurality of candidates, one signal format for SDR (e.g., BT.709) comprise a single signal format for HDR (e.g., a BT.2020, the combination of the HLG or S-log3) and But good. As another example, a plurality of candidates may include a signal format for the plurality of HDR. As a variant, the signal processor 14 may be capable of generating an image signal only in the signal format for a single HDR.
[0039]
　The signal processing unit 14, the image signal generated as a result of the signal conversion, multiplexes the auxiliary signal including voice signals and metadata as needed, and outputs them signals multiplexed to the server device 15. Server device 15 is, for example SDI (Serial Digital Interface), or an image processing apparatus connected to the signal processor 14 via a signal line conforming to a transmission protocol such as HD-SDI. Server device 15 acquires the image signal transmitted from the signal processing unit 14, encodes the image by a predetermined image encoding method for generating a coded bit stream 17a. Coded bit stream 17a may be stored in the internal or external storage device of the server device 15, or other device connected to the server device 15 (e.g., display devices) may be transmitted to.
[0040]
　Figure 6B is an explanatory diagram showing a second example of a configuration of an image processing system according to the present embodiment. The image processing system 10b shown in FIG. 6B, includes an imaging device 12, storage device 13 and the terminal device 16.
[0041]
　Imaging device 12, for example, may be any type of device having a digital video camera, a digital camcorder or digital still camera, or a video shooting function. Imaging device 12, by taking a picture of the real world using an image sensor, and generates a primitive image signal. The imaging device 12 performs the AD conversion, a digital signal processing as described above in relation to the signal processor 14 to generate an image signal in a predetermined signal format. Imaging device 12, similarly to the signal processor 14 may generate an image signal by a transfer function selected from a plurality of candidates by the user via some user interface, or the transfer function for a single HDR the image signal may be generated only.
[0042]
　Imaging device 12 based on the image signal generated as a result of the signal conversion, encodes the image by a predetermined image encoding scheme to generate a coded bit stream 17b. Coded bit stream 17b, for example, may be stored as a video file, or may be provided to the storage device 13 or the terminal device 16 via the network.
[0043]
　Storage device 13 is a data storage for storing various image data. Storage device 13, for example, may store video files 17c generated by encoding an image at a predetermined video encoding method. The header area of ​​the video file, for example, identify the type of transfer function for the conversion between the optical and the image signal applied to the video content included in the video file, the type of color gamut, and the video encoding method, respectively parameters may be included. Storage device 13 may store a RAW image file 18 for recording the image signal before encoding (or signal before conversion) as RAW data. Storage device 13, for example in response to a request from the terminal device 16, the user provides to the terminal device 16 via the network file wishing to or editing and playback.
[0044]
　Terminal device 16 is an image processing apparatus having a function of reproducing or editing a video file that is stored by or storage device 13 is generated by the imaging device 12. Terminal device 16, for example, may generate a decoded image signal by decoding the encoded bit stream included in the video file 17b or 17c is acquired from the imaging device 12 or the storage device 13. The terminal device 16, a dynamic range conversion on the decoded image generated as such (e.g., HDR-SDR conversion or SDR-HDR conversion) may be executed. Further, the terminal device 16, the image signal included in the RAW image file 18 or the decoded image signal after dynamic range transform encoded in a predetermined video encoding method may generate a coded bit stream 17d.
[0045]
　Server device 15 in the example of FIG. 6A and the image pickup device 12 and the terminal device 16 in the example of Figure 6B, are all image processing apparatus for encoding an image (i.e., encoders) functions as a. In the present embodiment, when these image processing apparatus encodes image, based on the transfer function (e.g., type or other based on the attributes of the transfer function) controls the code amount assigned to each partial region of the image by, to reduce the deterioration of the image quality when the signal format for HDR can be used. From the next section, it will be described in detail specific and exemplary configuration of such an image processing apparatus.
[0046]
　　[2-3. Schematic configuration] of the image processing apparatus
　FIG. 7A is a block diagram showing a first example of a schematic configuration of an image processing apparatus according to this embodiment. The image processing apparatus 100a shown in FIG. 7A, for example, the server device 15 in the example of FIG. 6A, or the imaging device 12 or the terminal device 16 in the example of FIG. 6B (or, the image processing module mounted on one of those devices ) it may be. The image processing apparatus 100a includes a signal acquisition unit 101, the information acquisition unit 103, the encoding unit 110 and the control unit 140.
[0047]
　Signal acquisition unit 101 acquires an input image signal generated based on the transfer function for the conversion between the optical and the image signal. Signal acquiring unit 101, an input from obtaining input image signal from an external device, or the image processing apparatus 100a and integrally formed imaging module and a signal processing module (not shown) via a transmission interface the image signals may be obtained.
[0048]
　Information acquisition unit 103 acquires input information about transfer function applied to the image to be encoded by the encoder unit 110. As an example, the information acquisition unit 103, the input information via the user interface of the image processing apparatus 100a has may be acquired. User interface, the image processing apparatus 100a, for example, a touch panel provided on the housing, may be provided by physical input devices such as buttons or switches. Alternatively, the user interface may be provided as a GUI (Graphical User Interface) on the terminal device that is remotely connected through a communications interface. In this embodiment, the input information includes a transfer function type indicating the type of the transfer function applied to the image to be at least coded. The user interface, for example, may be selected one to be applied to the image of the two choices "SDR" and "HDR" to the user. In this case, one of the transfer function previously defined for SDR, or one of the transfer function previously defined for HDR is determined to have been applied to the image. Further, the user interface, a plurality of candidates for the transfer function (e.g., BT.709, HLG, ST2084 and S-log3) from may be selected transfer functions to be applied to the image to the user.
[0049]
　As another example, the information acquisition unit 103 may acquire the input information from the auxiliary signal inputted image signal and multiplexing. For example, the auxiliary signal is a period during which an image signal in signal line is not transmitted (e.g., the blanking period) is received by the signal receiver 101 in. Then, the information acquisition unit 103, the auxiliary signal separated in the signal acquisition unit 101 may obtain input information including the transfer function type indicating the type of the transfer function applied to the image. The information acquisition unit 103 may acquire the input information required to access to external data sources.
[0050]
　Encoding unit 110, an image represented by the image signal acquired by the signal acquiring unit 101 and encodes, to generate an encoded bit stream. Encoding unit 110, for example, MPEG-2, H. 264 / AVC or H. It may perform an encoding process in accordance with any video coding scheme, such as 265 / HEVC. Encoding process performed by the encoding unit 110 is typically prediction, orthogonal transformation, include various processing such quantization and entropy coding, among others quantization to achieve the required compression ratio a process including the lossy compression.
[0051]
　Control unit 140, based on the transfer function indicated by the input information acquired by the information acquisition unit 103, controls the code amount assigned in the encoding unit 110 to each partial region of the image. More specifically, the control unit 140, for HDR of the first transfer function corresponding to the HDR (transfer function for the SDR) a second transfer function corresponding to and SDR (transfer function for HDR) If the transfer function is applied to the image, to enable the quantization control process for reducing the degradation of the image quality of the HDR image. The quantization control process, a process of correcting a process parameter of the quantization process performed regardless of the transfer function or signal format, in order to adjust the allocation of the code amount in the case where the transfer function for the HDR is applied It may include. Here, an example will be described in which the assigned code amount based on the type of main transfer function is controlled, but such upper limit of the dynamic range associated with the transfer function, the code amount based on other attributes of the transfer function it may be controlled.
[0052]
　Figure 7B is a block diagram showing a second example of a schematic configuration of an image processing apparatus according to this embodiment. The image processing apparatus 100b shown in FIG. 7B also, for example, the server device 15 in the example of FIG. 6A, or the imaging device 12 or the terminal device 16 in the example of FIG. 6B (or, the image processing to be mounted on either of them device it may be a module). The image processing apparatus 100b is provided with a signal processing unit 102, the information acquisition unit 104, a storage unit 107, the encoding unit 110 and the control unit 140.
[0053]
　The signal processing unit 102 obtains the original image signal inputted from the imaging device via some transmission interface or device inside the signal line, or acquires the image signal from the image file stored in the storage unit 107 . The signal processing unit 102 performs digital signal processing, which may include, for example, gamma correction and color conversion for primitive image signal, and generates an image signal to be the target of coding in a predetermined signal format. Transfer function signal format and corresponding applied by the signal processing unit 102 in the image is determined based on input information acquired by the information acquisition unit 104. The signal processing unit 102 outputs the generated image signal to the encoding unit 110.
[0054]
　Information acquisition unit 104 acquires input information about the transfer function applied to the image to be encoded by the encoder unit 110. For example, the information acquisition unit 104 (provided as provided or GUI by physical input device) of the image processing apparatus 100b has may acquire the input information via the user interface. As described above, the input information includes a transfer function type indicating the type of the transfer function applied to the image to be at least coded. The user interface, for example, may be selected one to be applied to the image of the two choices "SDR" and "HDR" to the user. Further, the user interface, from the candidates of a plurality of transfer functions, the transfer function to be applied to the image may be selected by a user.
[0055]
　Storage unit 107 is a storage device for storing various image data. Storage unit 107, for example, may store image file that records a digital image signal before signal conversion. The user output interface of the image processing apparatus 100b has (not shown), it may be stored video file acquired from the external storage medium in the storage unit 107. The storage unit 107 may store the video file containing the encoded bit stream generated as a result of the encoding process performed by the encoding unit 110. Video file may be outputted to the external apparatus in response to the request.
[0056]
　Similar to the first example described with reference to FIG. 7A, the encoding unit 110, an image represented by an image signal obtained by the signal processing unit 102 encodes, for generating a coded bit stream. Control unit 140, based on the type of transfer function indicated by the input information acquired by the information acquisition unit 104, controls the code amount assigned in the encoding unit 110 to each partial region of the image. Coded bit stream generated by the coding unit 110 may be transmitted to the external apparatus of the image processing apparatus 100b, or may be stored as a video file in the storage unit 107.
[0057]
　　[2-4. Detailed Configuration of the encoding unit and the control unit
　in this section will be described in detail more specific structure of the encoding unit 110 and the control unit 140 shown in FIGS. 7A and 7B. Figure 8 is a block diagram showing an example of a detailed configuration of the encoding unit and the control unit according to the first embodiment.
[0058]
　　(1) coding unit
　Referring to Figure 8, the encoding unit 110, reordering buffer 111, the block setting unit 112, the subtraction unit 113, orthogonal transform unit 114, a quantization unit 115, a lossless coding unit 116, an inverse quantization comprising a unit 121, inverse orthogonal transform unit 122, addition unit 123, a loop filter 124, a frame memory 126, switch 127, mode selection unit 128, an intra prediction unit 130 and the inter prediction unit 135.
[0059]
　Reordering buffer 111, the image data of a series of images represented by the image signal acquired by the signal acquiring unit 101 or the signal processing unit 102 rearranges according to a GOP (Group of Pictures) structure. Reordering buffer 111 outputs the image data after the rearrangement block setting unit 112, the intra prediction unit 130 and the inter prediction unit 135.
[0060]
　Block setting section 112 divides each of the image corresponding to the picture into a plurality of blocks. MPEG-2 and H. In 264 / AVC, a picture is divided in a grid pattern into a plurality of macro blocks having a fixed size, encoding processing is executed each macro block as a processing unit. Quantization process can be performed with smaller sub-blocks from being set in each macroblock as a processing unit. H. In 265 / HEVC, picture is divided into a plurality of coding units (Coding Unit) in quadtree form with variable sizes, coding process is performed each CU as a processing unit. Quantization process can be performed as a processing unit smaller conversion unit (Transform Unit) which is set to each CU.
[0061]
　Subtraction unit 113 calculates a prediction residual data is a difference between the image data and predicted image data input from the block setting unit 112, and outputs the prediction residual data to the orthogonal transform unit 114.
[0062]
　Orthogonal transform unit 114, a prediction residual data received from subtracting section 113, converts the image data in the spatial domain to the transform coefficient data in the frequency domain. Orthogonal transform performed by the orthogonal transform unit 114, for example and the like discrete cosine transform or discrete sine transform. The orthogonal transform unit 114 outputs the transform coefficient data to the quantization unit 115.
[0063]
　Quantization unit 115, the transform coefficient data input from the orthogonal transformation unit 114, is quantized by the quantization step of required compression ratio is determined so as to achieve. For example, in many cases the free space in the buffer or the transmission path with respect to the size of the outputted encoded bit stream quantization step is set smaller, the quantization step is set large when the free space in the opposite space is low obtain. Quantization step are generally determined for each partial region in an image. For each of the three color components different quantization steps may be used. As quantization step used for a certain partial area smaller, transform coefficients of the partial region is quantized finely. This means more that the code amount is assigned to the partial region, i.e. be maintained without gradation of the image of the partial area is more impaired. Quantization unit 115 may apply different quantization steps to different frequency components of the transform coefficients using a quantization matrix. Then, the quantization unit 115, transform coefficient data after quantization (hereinafter referred to as quantized data) to the to the lossless encoding section 116 and the inverse quantization unit 121.
[0064]
　If the transfer function for the HDR input image signal is applied, the quantization unit 115, parameters for adjusting the quantization step (scale) used for the partial regions is provided from the control unit 140 . Quantization unit 115, by scaling the quantization step is divided by this parameter provided from the control unit 140 (or multiplication) quantizes the transform coefficient data by the quantization step after scaling. In some video coding scheme, instead of directly encoding the quantization step as a control value required for the inverse quantization at the decoder side, having a logarithmic relationship with the quantization step quantization parameter (QP) is coded. Scaling of quantization step, instead of dividing the quantization step by some factor (or multiplication) may be implemented by adding (or subtracting) some offset to the quantization parameter.
[0065]
　Lossless encoding unit 116, by encoding the quantized data input from the quantization unit 115, generates an encoded bit stream. Also, the lossless encoding unit 116, various parameters that are referenced by the decoder by encoding to insert the encoded parameters into the encoded bit stream. Parameters coded by the lossless coding unit 116 may include information about the transfer function, information regarding color gamut, and the above-described information on the quantization parameter. Lossless encoding unit 116, the generated encoded bit stream and outputs to an output destination in accordance with the purpose of the application.
[0066]
　Inverse quantization unit 121, inverse orthogonal transform unit 122 and the addition unit 123 constitute a local decoder. Local decoder is responsible for reconstructing the original image from the encoded data.
[0067]
　Inverse quantization unit 121 inversely quantizes the quantized data in the same quantization step as that used by the quantization unit 115, restores the transform coefficient data. When the transfer function for the HDR is applied to the input image signal is scaled quantization step using the parameters provided from the control unit 140 may be used for each partial area. Then, the inverse quantization unit 121 outputs the transform coefficient data restored to the inverse orthogonal transform unit 122.
[0068]
　Inverse orthogonal transform unit 122, by executing the inverse orthogonal transform processing for transform coefficient data input from the inverse quantization unit 121, to restore the prediction residual data. Then, the inverse orthogonal transform unit 122 outputs the restored prediction residual data to the adder 123.
[0069]
　Adding section 123 by adding the predicted image data generated by the inverse orthogonal transform unit prediction residual restored input from 122 data and the intra prediction unit 130 or the inter prediction unit 135, generates decoded image data to. The adding unit 123 outputs the generated decoded image data to the loop filter 124 and a frame memory 126.
[0070]
　Loop filter 124, an in-loop filter for the purpose of improving the quality of the decoded image. Loop filter 124 may include, for example, a deblock filter for reducing block distortion appearing in the decoded picture. Further, the loop filter 124 may include an adaptive offset filter for adding the edge offset or band offset in the decoded picture. Loop filter 124 outputs the decoded image data after filtering to the frame memory 126.
[0071]
　The frame memory 126 stores the decoded image data after the application of the in-loop filter input before filtering the decoded image data input from the addition unit 123, and the loop filter 124.
[0072]
　Switch 127 reads out the decoded image data before filtering that is used for intra prediction from the frame memory 126, and supplies the intra prediction unit 130 read decoded image data as reference image data. The switch 127 reads the decoded image data after filtering to be used for inter prediction from the frame memory 126, and supplies the inter prediction unit 135 read decoded image data as reference image data.
[0073]
　Mode selection unit 128, based on the cost comparison of input from the intra prediction unit 130 and the inter prediction unit 135 selects a prediction method for each block. Mode selection unit 128, for the block where the intra prediction, outputs the predicted image data generated by the intra prediction unit 130 to the subtraction unit 113 outputs information on the intra prediction to the lossless encoding unit 116. The mode selection unit 128, the block where the inter prediction, outputs the predicted image data generated by the inter prediction section 135 to the subtraction unit 113 outputs information on the inter prediction to the lossless coding unit 116 .
[0074]
　The intra prediction unit 130, based on the original image data and the decoded image data, to perform the intra prediction process. For example, the intra prediction unit 130, for each of a plurality of candidate modes included in the search range, to evaluate the cost is estimated to occur. Then, the intra prediction unit 130 selects a prediction mode the minimum cost as the best prediction mode. Further, the intra prediction unit 130 generates a predictive image data according to the best prediction mode selected. Then, the intra prediction unit 130, information on an intra prediction including a prediction mode information indicating the best prediction mode, the corresponding cost, and predicted image data to the mode selection unit 128.
[0075]
　Inter prediction unit 135, based on the original image data and the decoded image data, to perform the inter prediction process (motion compensation). For example, inter prediction unit 135, for each of a plurality of candidate modes included in the search range, to evaluate the cost is estimated to occur. Next, the inter prediction unit 135 selects a prediction mode the minimum cost as the best prediction mode. Further, the inter prediction unit 135 generates a predictive image data according to the best prediction mode selected. The inter prediction unit 135, information on inter prediction, corresponding cost, and predicted image data to the mode selection unit 128.
[0076]
　　(2) Control Unit
　Referring to Figure 8, the control unit 140 includes a statistical computation unit 141 and code amount control unit 143.
[0077]
　Statistical operation unit 141, for each of partial areas set in the image, calculates the statistics related to at least one of the intensity of the luminance component and chrominance components. Statistics calculated by statistical calculation unit 141, the pixel value of the partial area for one or more color components representative value (code value) (e.g., mean, median or mode value) or from histograms . The statistical computation unit 141 outputs the calculated statistic to code amount control unit 143.
[0078]
　The portion region here typically may correspond to block corresponding to the unit of processing of the quantization processing. For example, MPEG-2 or H. Macroblock or sub-block in the 264 / AVC, or H. 265 / statistics for each partial region, such as CU or TU in HEVC is calculated, then the quantization step by the code amount control unit may be controlled to be described. The invention is not limited to this example, for each partial region (which may be 1 pixel) having other shapes, quantization control process may be performed as described herein.
[0079]
　The code amount control unit 143, the type of the transfer function applied to the image to be encoded, determines based on the input information inputted from the information acquisition unit 103 or 104. Then, code amount control unit 143, if the transfer function for the SDR of the transfer function for the transfer function and SDR for HDR is applied may skip the quantization control process described below. On the other hand, the code amount control unit 143, when the image to be encoded is the transfer function for the HDR is applied, the amount of codes allocated to each partial region, of the luminance components and chrominance components of the partial region controlled as a function of at least one of. More specifically, the code amount control unit 143, the quantization step used by the quantization unit 115 for each local area, scaled depending on the intensity of the one or more color components (or quantization unit by scaling) 115, controls the code amount assigned to each subregion. Hereinafter, both the strength of the strong dependent control of control that depends on the intensity of the color difference component as the second embodiment, the luminance components and chrominance components as the third embodiment of the luminance component as the first embodiment control will be described that depend on.
[0080]
　　(3) the first embodiment - the protection of the high brightness portion
　in the first embodiment, the code amount control unit 143, a number of code amount by a stronger partial area intensity of the luminance component (i.e., the high-intensity part) as it is allocated scales the quantization steps used for the partial regions. The strength of the luminance component of each partial region is grasped from the statistics of each partial region calculated by the statistical operation unit 141. Here, the code amount control unit 143 is divided by the protection ratio that depends on the quantization step to the intensity of the luminance component of each partial area is intended to scale the quantization step. Protection ratio is a parameter indicating how much protection the image quality of the partial region. The greater the value of the protection ratio, the value of the quantization step decreases the quality of the partial region where the quantization step is applied is protected more strongly. Incidentally, the actual division by protection ratio can be carried out in the quantization unit 115 is provided a protection ratio.
[0081]
　9A is an explanatory diagram for describing a first example of a protection ratio for protecting high-luminance portion. The horizontal axis in FIG. 9A represents the sign value of the luminance component. The vertical axis represents the protection ratio. As shown here, the protection ratio can be a parameter calculated using a predetermined function with an argument strength of the luminance component of each partial area. Although a linear function is shown as an example in FIG. 9A, or higher order function, or for example be other types of functions, such as a logarithmic function is used. Thus example, that the intensity of the luminance component is higher protection ratio stronger partial region, the quantization step of such partial regions are scaled to a smaller value. Thereby, it is possible to prevent the image information of high-luminance portion that has already been strongly compressed in the conversion from optical to electric signal is excessively impaired.
[0082]
　9B is an explanatory diagram for describing a second example of the protection ratio for protecting high-luminance portion. As shown in FIG. 9B, the protection ratio can be a parameter that the intensity of the luminance component of each partial region is selectively determined by belongs to which sub-range. According to a second example, the strength is less than 200 nit luminance component of each partial area, less than 200 nit 300nit, less than 300nit 400 nit, less than 400 nit 500Nit, are classified into six sub-ranges of less than or 500nit 600nit and 600Nit more and, protection ratio corresponding to each sub-range is defined. The code amount control unit 143 may have a memory for previously storing a mapping table that maps the protection ratio corresponding to such subranges. In the second embodiment, like the first embodiment, since the protective ratio stronger partial area intensity of the luminance component is set higher, already strongly compressed during conversion to electrical signals from the light image information of high-luminance portion is can be prevented from be excessively compromised.
[0083]
　Incidentally, according to the second embodiment, the central portion of the dynamic range, while protecting ratio (or quantization step) is controlled finely, protection ratio at the end of the dynamic range is fixed. The protection ratio By setting in this way, the subjectively protection of the central portion of the sensed given easily dynamic range effect on the image quality through the human visual, reinforced efficiently while suppressing expense of coding efficiency be able to.
[0084]
　　(4) Second Embodiment - protection Koirosa portion
　in the second embodiment, the code amount control unit 143, stronger partial region the intensity of the color difference components (i.e., Koirosa portion) many code amount by as it is allocated scales the quantization steps used for the partial regions.
[0085]
　Figure 10 is an explanatory diagram for describing the code value to be protected as Koirosa portion. The horizontal axis of FIG. 10 represents the sign value of one in which the Cb component of the two color difference components. The vertical axis represents the sign value of the other is a Cr component of the two color difference components. Point P1 in the figure, the particular point code value of the R component and G component corresponds to the so-called "yellow" above 1000 in the RGB space, shows the corresponding position of the CbCr plane in the YCbCr space. Point P2, the particular point code value of the G component and the B component corresponds to the so-called "cyan" above 1000 in the RGB space, shows the corresponding position of the CbCr plane in the YCbCr space. Point P3, the particular point code value of the G component corresponds to the so-called "green" above 1000 in the RGB space, shows the corresponding position of the CbCr plane in the YCbCr space. Point P4 is the particular point code value of the R component and the B component corresponds to the so-called "magenta" above 1000 in the RGB space, shows the corresponding position of the CbCr plane in the YCbCr space. Point P5 is the particular point code value of the R component corresponds to the so-called "red" above 1000 in the RGB space, it shows the corresponding position of the CbCr plane in the YCbCr space. Point P6 is a particular point code value of the B component in the RGB space corresponds to a so-called "blue" in excess of 1000, it shows the corresponding position of the CbCr plane in the YCbCr space.
[0086]
　According to the analysis of the color component values ​​of these points, while having a relatively high Y component value in a broken line frame inside P1, P2 and P3 YCbCr space point in the HL in FIG. (E.g., 700 or more) , the points P4, P5 and P6 which is outside the broken line frame HL has a relatively low Y component value (e.g., less than 700). This is part of the "yellow" Among vivid color part, while the part of "cyan" and "green" can be protected given the luminance component, "magenta", "red" and "blue" means that it is not the case. Therefore, it is also beneficial to increase the allocation amount of codes for Koirosa portion. The intensity of the color difference components of each partial region is grasped from the statistics of each partial region calculated by the statistical operation unit 141. Here, the code amount control unit 143 is divided by the protection ratio that depends on the quantization step to the intensity of the color difference components of the partial regions (specific to common or chrominance component and a luminance component) quantization step the assumed to be scaling. Incidentally, the actual division may be performed in the quantization unit 115 is provided a protection ratio.
[0087]
　Protection ratios for protecting Koirosa moiety, as in the first example shown in FIG. 9A, a parameter which is calculated using a predetermined function with an argument intensity of the color difference component of the partial regions it may be. Alternatively, the protection ratio for protecting Koirosa moiety, as in the second example shown in FIG. 9B, is determined depending on whether the intensity of the color difference component of each partial area belongs to which subrange it may be that parameter. The code amount control unit 143 may have a memory for previously storing a mapping table that maps the protection ratio corresponding to the sub-range of the color difference component.
[0088]
　Figure 11 is an explanatory diagram for explaining an example of a protection ratio for protecting Koirosa portion. 11, in addition to the protection ratio of the luminance component corresponding to the six sub-range illustrated in FIG. 9B (solid line), the protection ratio of the color difference component corresponding to the same six sub-ranges (dashed line) are shown. By thus strength of the color difference component is set higher protection ratio stronger partial area, the image information of high color difference part is already strongly compressed in the conversion from optical to electrical signals is excessively impaired it is possible to avoid to be. Further, by setting the fine protection ratio in the central portion of the dynamic range, efficient protection of the central portion of the subjectively easy affect perceived image quality dynamic range, while suppressing the expense of coding efficiency it can be enhanced.
[0089]
　　(5) Third embodiment - the high-intensity part and Koirosa portion protection
　point of view 10 P1, P2 and P3 ( "yellow", "cyan" and "green"), as described above, colorful of a (strong Y component) (R component, one or more strong of G component and B component) belonging to the region and the high brightness belongs to region. When also be protected as a high chrominance portion while protecting the partial region as a high luminance portion having such color, there is a possibility that the assigned code amount becomes larger as inappropriate. Therefore, the code amount control unit 143, one of the quantization step of partial regions quantization step is scaled in accordance with the intensity of the luminance components and chrominance components, the other of the luminance and chrominance it may not be scaled depending on the strength.
[0090]
　As an example, the code amount control unit 143, based on the histogram is calculated for each color component by the statistical operation unit 141 for each partial region, classifies each partial region into two groups. More specifically, for example, the code amount control unit 143 is above the rate threshold of pixels having a Cb component above a certain Cb reference value, or percentage of pixels having a Cr component above a certain Cr reference value the threshold a partial region above the first group, can be classified not so (the proportion of both is less than the threshold value) partial region to the second group. The first group includes a broken line frame more partial regions of pixels located outside the HL of FIG. 10, the second group will contain more partial regions of pixels located inside the broken line frame HL. The code amount control unit 143, the partial areas belonging to the first group the protection of the high-chroma portions in accordance with a second embodiment, the partial areas belonging to the second group of high-luminance portion in accordance with a first embodiment protection may be applied.
[0091]
　By such switching of protection techniques, while reducing avoidance and process cost control of redundant quantization step, locally assigned code amount in the image can be prevented from becoming excessive.
[0092]
　　[2-5. Process Flow
　　(1) the encoding control processing
　Figure 12 is a flowchart showing an exemplary flow of an encoding control process according to the present embodiment. Encoding control process described herein may be repeated for each image constituting the video. Processing steps for obtaining or setting the parameters that do not change over a plurality of images may be skipped in the repetition of the second and subsequent. Here, for simplicity of description, description of the processing steps not directly related to the control of the code amount is omitted.
[0093]
　Referring to FIG. 12, first, the signal acquisition unit 101 or the signal processing unit 102 obtains an image signal generated based on a transfer function for the conversion between the optical and the image signal (step S110). Image signal obtained here is output to the encoding unit 110.
[0094]
　Next, the information acquisition unit 103 or 104, the input information about the transfer function applied to the image to be encoded by the encoder unit 110, acquires from the auxiliary signal or input image signals multiplexed through the user interface (step S112). Input information obtained here is outputted to the control unit 140.
[0095]
　Next, the code amount control unit 143, based on the type of transfer function indicated by the input information described above, to set the protection ratio table or function is used to set the protection ratio to each partial region (step S114). Protection ratio table or function set here, different depending on whether may be common across multiple transfer functions for HDR, or which of the plurality of transfer functions for HDR applied it may be.
[0096]
　Subsequent processing is repeated for each of a plurality of partial regions set in the image to be processed. The partial area to be processed at each iteration, herein referred to the target portion area.
[0097]
　First, the quantization unit 115 of the encoding unit 110, regardless of what the transfer function is applied to determine the quantization step of the target partial region as required compression ratio is achieved (step S130) .
[0098]
　Next, the code amount control unit 143 determines the type of the applied transfer functions, based on the input information (step S132). Then, code amount control unit 143, when the image to be encoded is the transfer function for the HDR is determined to have been applied performs quantization control process will be described in detail later (step S140). On the other hand, if it is determined that the image to be encoded transfer function for SDR is applied, the code amount control unit 143 skips the quantization control process.
[0099]
　Then, the quantization unit 115, the transform coefficient data of the target partial area is input from the orthogonal transform unit 114, (not scaled to a or SDR video) after scaling is quantized by the quantization step (Step S160 ).
[0100]
　Next, the reversible encoding unit 116 encodes the quantized data and the quantization parameter input from the quantization unit 115, generates an encoded bit stream (step S170).
[0101]
　Step S130 ~ step S170 are repeated until the processing is completed for all of the partial areas in the picture (step S180). When the processing for all the pictures completed, the encoding control process shown in FIG. 12 is terminated (step S190).
[0102]
　　(2) quantization control processing (first embodiment)
　FIG. 13A is a flowchart showing a first example of the flow of a quantization control process may be executed in step S140 of FIG. 12. The first example shows an example of the flow of the quantization control process to protect the gradation of the high-intensity part in the image.
[0103]
　Referring to FIG. 13A, first, the statistical calculation unit 141 calculates the statistics on the intensity of the luminance component of the target portion area (step S141). Here is calculated statistics, for example, the average of the pixel values ​​of the partial area of ​​the luminance component may include median or mode. The statistical computation unit 141 outputs the calculated statistic to code amount control unit 143.
[0104]
　Next, the code amount control unit 143 is determined by using a protective ratio corresponding to the luminance statistics of the target portion area inputted from the statistical calculation unit 141, a function for referring to or protect ratio calculated protection ratio table (step S144). Then, code amount control unit 143, the determined protection ratio, and outputs it to the quantization unit 115.
[0105]
　Then, the quantization unit 115, according to the protection ratio inputted from the code amount control unit 143, scales the quantization step determined at step S130 of FIG. 12 (step S146). For example, the quantization unit 115 reduces the quantization step by dividing the quantization step larger than 1 protection ratio inputted from the code amount control unit 143, or dividing the quantization step in less than one protection ratio expanding the quantization step by. Here, although described as an example of scaling in protected ratio tentatively determined quantization step as required compression ratio is achieved, at the same time taking into account both the required compression rate and protection ratios quantization step may be determined. The same applies to the other embodiments described below.
[0106]
　　(3) the quantization control process (second embodiment)
　FIG. 13B is a flowchart showing a second example of the flow of a quantization control process may be executed in step S140 of FIG. 12. The second example shows an example of the flow of the quantization control process to protect the gradation of high chrominance portion of the image.
[0107]
　Referring to FIG. 13B, first, the statistical calculation unit 141 calculates the statistics on the intensity of the color difference components of the target portion area (step S142). Statistics calculated here, for example, the average of the pixel values ​​of the partial area of ​​the color difference component may include median or mode. The statistical computation unit 141 outputs the calculated statistic to code amount control unit 143.
[0108]
　Next, the code amount control unit 143, the protection ratio corresponding to the color difference statistics of the target portion area inputted from the statistical calculation unit 141 is determined by using the function for reference or protection ratio calculation protection ratio table (step S145). Then, code amount control unit 143, the determined protection ratio, and outputs it to the quantization unit 115.
[0109]
　Then, the quantization unit 115, according to the protection ratio inputted from the code amount control unit 143, scales the quantization step determined at step S130 of FIG. 12 (step S147). For example, the quantization unit 115 reduces the quantization step by dividing the quantization step larger than 1 protection ratio inputted from the code amount control unit 143, or dividing the quantization step in less than one protection ratio expanding the quantization step by.
[0110]
　　(4) the quantization control process (Third Embodiment)
　FIG. 13C is a flowchart showing a third example of the flow of the quantization control process may be executed in step S140 of FIG. 12. The third example shows an example of the flow of the quantization control process to protect both the gradation of a high luminance portion and Koirosa portion of the image.
[0111]
　Referring to FIG. 13C, first, the statistical calculation unit 141 calculates the statistics on the intensity of the luminance component of the target portion area (step S141). The statistical calculation unit 141 calculates the statistics on the intensity of the color difference components of the target portion area (step S142). The statistical computation unit 141 outputs the calculated statistic to code amount control unit 143.
[0112]
　Next, the code amount control unit 143, statistics of the target portion area inputted from the statistical calculation unit 141 (e.g., a histogram of the color difference components), based on, or the color difference to apply the protection depends on the brightness to the target partial region It depends on determining whether to apply the protection (step S143).
[0113]
　Next, the code amount control unit 143, when determining to apply the protection depends on the brightness in the target portion area, the protection ratio corresponding to the luminance statistics of the target portion area, refers to the protection ratio table or protection ratio determined by using the function for calculating (step S144). Then, the quantization unit 115 scales the quantization step according to the protection ratio inputted from the code amount control unit 143 based on the luminance statistics (step S148).
[0114]
　On the other hand, the code amount control unit 143, when determining to apply the protection depends on the color difference to the target partial region, the protection ratio corresponding to the color difference statistics of the target portion area, refers to the protection ratio table or protection ratio calculation determined by using a function of the use (step S145). Then, the quantization unit 115 scales the quantization step according to the protection ratio inputted from the code amount control unit 143 based on the color difference statistics (step S149).
[0115]
　　[2-6. Modification
　far, an image processing apparatus having a function of encoding an image, based on the type of the transfer function applied to the image, switching the processing of the on / off to control the assigned code amount for each subregion Example It was described. However, the idea of this embodiment, without a determination of the type of the transfer function is also applicable to the case where the control of the assigned code amount for each partial region is executed. This section will be described such a modification.
[0116]
　　(1) Configuration of the image processing apparatus
　14 is a block diagram showing a modification of the configuration of an image processing apparatus according to the first embodiment. The image processing apparatus 100c shown in FIG. 14, for example, the server device 15 in the example of FIG. 6A, or the imaging device 12 or the terminal device 16 in the example of FIG. 6B (or, the image processing module mounted on one of those devices ) it may be. The image processing apparatus 100c includes a signal acquisition unit 101 includes an encoding unit 110 and the control unit 140c.
[0117]
　Similar to the example described with reference to FIG. 7A, the signal acquisition unit 101 acquires an input image signal generated based on the transfer function for the conversion between the optical and the image signal. Input image signal acquired by the signal acquiring unit 101 in this modification is converted from light by the transfer function for the HDR, a signal generated by the signal format for HDR. Here the transfer function for the HDR in, for example, for enabling the display of video at high luminance than 100 nit, may be a transfer function such HLG, ST2084 or S-log3. Encoding unit 110, an image represented by the image signal input from the signal receiver 101 and encodes and produces the encoded bit stream. The image processing apparatus 100c includes, instead of the signal acquisition unit 101 may include a signal processing unit 102 described with reference to Figure 7B.
[0118]
　In this modified example, the control unit 140c, assuming that the transfer function for the HDR to an image to be encoded is applied, the amount of codes assigned in the encoding unit 110 to each partial region of the image, the partial region controlled as a function of at least one of the luminance and chrominance. More specifically, the control unit 140c is not possible to determine the type of transfer function, the quantization step used in accordance with any, for each partial region in the embodiment described with reference to FIGS. 13A ~ FIG 13C and by scaling as a function of at least one of luminance components and chrominance components, capable of controlling the amount of codes to be allocated to the partial regions.
[0119]
　Scaling of quantization step, implemented for example by multiplication or division of the parameters are determined using the argument to a sign value of the color component as described with reference to FIG. 9A (e.g., division by the protection ratio) it may be. Alternatively, the scaling of the quantization step is achieved by pre-mapped (obtained by referring to the mapping table) parameters multiplication or division with code values ​​of the color components as described with reference to FIG. 9B example it may be. By the strength of the color components to reduce the quantization step stronger partial regions, it is possible to reduce more amount of code allocated to their partial regions, deterioration of image quality caused by the codec distortion.
[0120]
　　(2) the encoding control processing
　Figure 15 is a flowchart showing an exemplary flow of an encoding control process according to the modified example described with reference to FIG. 14. Encoding control process described herein may be repeated for each image constituting the video. Processing steps for obtaining or setting the parameters that do not change over a plurality of images may be skipped in the repetition of the second and subsequent. Here, for simplicity of description, description of the processing steps not directly related to the control of the code amount is omitted.
[0121]
　Referring to FIG. 15, first, the signal acquisition unit 101 or the signal processing unit 102 obtains an image signal transfer function is applied for HDR for the conversion between the optical and the image signal (step S111). Image signal obtained here is output to the encoding unit 110.
[0122]
　Next, the control unit 140c sets the protection ratio table or function is used to set the protection ratio to each partial area (step S115). Protection ratio table or function set here, different depending on whether may be common across multiple transfer functions for HDR, or which of the plurality of transfer functions for HDR applied it may be.
[0123]
　Subsequent processing is repeated for each of a plurality of partial regions set in the image to be processed. First, the quantization unit 115 of the encoding unit 110 determines the quantization step of the target partial region as required compression ratio is achieved (step S130).
[0124]
　Next, the control unit 140c executes one of the quantization control process described with reference to FIGS. 13A ~ FIG @ 13 C (step S140). Thereby, the quantization step of the target portion area determined in step S130 is scaled.
[0125]
　Then, the quantization unit 115, the transform coefficient data of the target partial area is input from the orthogonal transform unit 114, quantized by the quantization step after scaling (step S160).
[0126]
　Next, the reversible encoding unit 116 encodes the quantized data and the quantization parameter input from the quantization unit 115, generates an encoded bit stream (step S170).
[0127]
　Step S130 ~ step S170 are repeated until the processing is completed for all of the partial areas in the picture (step S180). When the processing for all the pictures completed, the encoding control process shown in FIG. 15 is ended (step S190).
[0128]
　　[2-7. Conclusion of first embodiment
　up to here, with reference to FIGS. 6A ~ 15, described for the first embodiment of the technology according to the present disclosure. In the embodiment described above, when coding the image acquired on the basis of the transfer function for the conversion between the optical and the image signal, based on the transfer function, the code amount control allocated to each partial region of the image It is. According to such a configuration, it is possible to what kind of transfer function is to change the assigned code amount for each partial region depending on whether applied to the image. Thereby, the cause that the assigned code amount for representing the gray level of the original signal as a result of the selection of the transfer function is insufficient, codec distortion in partial areas of the image can be prevented from becoming conspicuous Become.
[0129]
　Further, in the embodiment described above, the first transmission of the second transfer function corresponding to the narrower second dynamic range than the first transfer function and the first dynamic range corresponding to the first dynamic range If the function is applied to the image, the code amount to be allocated to the partial regions can be controlled in dependence on at least one of luminance components and chrominance components of the partial region. According to such a configuration, the assigned code amount determined regardless of the transfer function, when the transfer function corresponding to a wider dynamic range is applied, partial regions, depending on the strength of at least one color component it can be adjusted to each. Thereby, the to or tuned encoder configuration design assuming certain dynamic range, while making the order of extended dynamic range, to optimize the allocation amount of codes can be reduced deterioration of image quality.
[0130]
　As an example, the first dynamic range may be a dynamic range for enabling the display of a higher brightness than 100 nit, the second dynamic range, a dynamic range of up to brightness of 100 nit it may be. Thereby, be utilized to encode an encoder that is designed, the HDR image transfer function is applied, e.g. HLG, ST2084 or S-log3 while preventing deterioration of image quality due to the existing SDR video It can become.
[0131]
　Further, in the above embodiment, the code amount to be allocated to the partial regions, depending on at least one of luminance components and chrominance components of the partial regions is controlled by scaling the quantization step. For example, by scaling to a smaller value the quantization step determined in accordance with the application requirements (such as required compression ratio), it is possible to preserve the gradation of the image better. Further, the partial region of relatively margin allocated code quantity by scaling the quantization step to a larger value, it is possible to compensate for the decrease in the coding efficiency.
[0132]
　As an example, by adopting a structure that scaling of the quantization step for each block corresponding to the unit of processing of the quantization processing defined in each of the video coding scheme, the encoder to support them video coding scheme extends, the technology according to the above-described embodiment can be easily implemented at a low cost.
[0133]
　Further, in the above embodiment, the quantization step used for each partial region, a number of code amount at least one of the strength of the stronger part region of the luminance components and chrominance components of the partial area is assigned It is scaled so. As described above, for example, in the HDR cases, especially in partial code value is relatively large, the image information compared to the SDR of the case are compressed at a higher compression ratio, which is the time of the display of the HDR image codec distortion was caused to expand at the high intensity part and Koirosa portion of the image. In contrast, by raising the assigned code amount by reducing the quantization step in a stronger partial area intensity of the color components, that reduces codec distortion, properly reproduces the variation in gray in the original image It can become.
[0134]
　Further, in the above embodiment, based on the input information about the transfer function applied to the image, the transfer function may be determined. If the input information is acquired through the user interface can execute the control based on the transfer function as the user desires even if it can not determine the transfer function from the input signal. If the input information is acquired from the input image signal and the auxiliary signals to be multiplexed may perform control for automatically based on the transfer function without requiring user input.
[0135]
　Further, according to a variant, when encoding an image obtained on the basis of the transfer function for the conversion between the optical and the image signal, the code amount assigned to each partial region of the image, the partial region It is controlled in dependence on at least one of the luminance and chrominance. Thereby, it is possible to prevent the codec distortion becomes conspicuous in the partial region of the image as a cause that the assigned code amount for representing the gray level of the original signal is insufficient.
[0136]
　<3. Second
　　Embodiment> [3-1. Introducing]
　In many video encoding schemes, the encoder codes when encoding the image, and select the best modes in terms of coding efficiency from multiple selectable mode, the mode information indicating the selected mode It turned into to be transmitted to the decoder. Such mode selection, for example, selection of the prediction mode in the intra prediction (e.g., prediction direction and a prediction block size), the selection of the prediction mode in the inter prediction (e.g., motion vectors, prediction block size and the reference picture), and the intra prediction mode It may include selection of a prediction method between the inter prediction mode. Mode selection usually costs may correspond to the sum of the amount of code generated from the mode information of the code amount and the overhead generated by the prediction residual that remains by subtracting the predicted image data from the original image data, a plurality of candidate performed by evaluating over mode. However, the cost evaluation formula which is designed or tuned for SDR image is not necessarily optimal for the HDR image. This is because, in the image signal of the HDR image, the image information compared to the SDR video is compressed more strongly, the difference between the modes of the code amount generated from the prediction residual when the same evaluation formula is used it is because it is likely to be underestimated.
[0137]
　We, the image signal of the sample image expressed in the signal format for HDR H. In case of encoding with existing encoder that conforms to 264 / AVC, we recognize that often unnatural biased prediction mode selected occur. For example, the analysis of the prediction mode selected for an image for each prediction block as a result of the intra prediction, over the entire image, and unnatural as many blocks DC prediction (also referred to as the average value prediction) is selected Case was there. Bias of such prediction modes exacerbates the prediction accuracy, resulting in distortion dispersed throughout the image under the desired compression ratio. Cause deviation of prediction modes occurs, uniform cost evaluation formula for mode selection is not suitable for the HDR image. In particular, in the HDR cases, as a result of strong compression of the image information, the contribution of the prediction residuals in cost evaluation formula is reduced, presumably contributed mode information becomes excessively dominant.
[0138]
　In general, as a method for selecting the best mode from among a plurality of candidate modes, RD (Rate Distortion) optimization is known based on the Lagrange multiplier method. In RD optimization, encoding cost J for i-th candidate modes i can be written as follows:
[0139]
[Number 1]

[0140]
　In the formula (1), D i is, i-th strain occurring in the image in the candidate mode (hereinafter, referred to as prediction residual code amount) represent, usually the sum of absolute differences between the predicted image and the original image (SAD )be equivalent to. R i represents a code amount of i-th overhead bits occurring in the candidate mode (e.g., mode information indicating a prediction mode). λ is a factor dependent on the quantization parameter QP. On the right side of equation (1), in place of the coefficient lambda, the offset value which depends on the QP may be added (or subtracted). Furthermore, the prediction residual code amount D i as a value obtained by Hadamard converting the sum of absolute differences may be used.
[0141]
　When implementing the cost evaluation formula such as the formula (1) to the encoder, for example, in order to reduce the complexity of ensuring a sufficient processing speed or implementation, the code amount of overhead bits term R i (hereinafter, mode code as that the amount), it is beneficial to use a pre-defined fixed values are for each candidate mode. On the other hand, the difference in comparable tone in the signal before conversion of the image is compressed to the difference between the smaller code value is the case of HDR, compared with SDR case. Then, the optimized mode code amount R for SDR image i is the prediction residual code amount D generated in HDR video i will be too large to be counted to the cost evaluation formula with.
[0142]
　16A and 16B are explanatory diagrams for explaining the effect of the difference in the transfer function of the mode selected. The horizontal axis of the graph shown in the upper part of FIG. 16A represents the horizontal pixel position on the line of the image. The vertical axis represents the sign value of a color component with a pixel row on that line. The solid line of the graph represents the sign value of the original image, in the illustrated example, the code value takes a large value in the left half of the line, and reduced in the middle of the line, takes a small value in the right half of the line. Dashed left of the graph represents the sign value of the prediction image can be generated in accordance with the DC prediction which is one prediction mode of intra prediction. Right dashed graph represents the sign value of the prediction image can be generated in accordance with another one which is diagonal prediction of the prediction mode of intra prediction. Area of ​​the trajectory (dashed line) and exits the portion surrounded by the sign values ​​of the locus (solid line) and the prediction image of the code values ​​of the original image (shaded shaded portion in the figure), when the respective prediction mode is selected corresponding to the prediction error. In the example of FIG. 16A, the prediction error of the DC prediction is larger than the prediction error of the diagonal prediction. On the other hand, for example, less than DC prediction mode numbers, it is assumed that the small mode code amount than diagonal prediction (dot shaded area in the figure) is given. However, in the example of FIG. 16A, it shows the sum of the prediction error code amount and mode code amount, i.e., when the comparing between the two predictive modes cost, a low cost value towards even diagonal prediction than DC prediction. Therefore, in this case, as the prediction mode of intra prediction, diagonal prediction can be selected.
[0143]
　Also in the graph shown in the upper part of FIG. 16B, a solid line represents the sign value of the original image. However, while the transfer function for the SDR in the example of FIG. 16A are applied to the image, as a result of the transfer function for the HDR is applied to the image in the example of FIG. 16B, floors should be a comparable nature difference in tone are compressed to the difference between the smaller code value. Therefore, the area of ​​the trajectory (dashed line) and exits the portion surrounded by the sign values ​​of the locus (solid line) and the prediction image of the code values ​​of the original image, i.e. the DC prediction and diagonal (hatched shaded portion in the drawing) prediction error compared with the directional prediction, the difference between these prediction error is smaller. When adding each mode code amount prediction error code amount (dot shaded portion in the drawing), the total cost represents a lower value towards the DC prediction than diagonal prediction. Thus, the original image despite the same, in this case, DC prediction may be selected as the prediction mode of intra prediction.
[0144]
　In order to prevent the undesired codec distortion in the image as a result of this manner affect the difference mode selection of the transfer function is generated, in this section, as the second embodiment, the prediction residual is calculated into the cost evaluation formula one of the differential code amount and mode code amount, to propose a method of controlling on the basis of the transfer function.
[0145]
　　[3-2. Summary of the system
　image processing system according to the present embodiment may be configured similarly to the image processing system 10a or 10b in the first embodiment. For example imaging devices in the system, the server device or the terminal device, or the image processing module mounted on any one of them device encodes an image to be acquired on the basis of the transfer function for the conversion between the optical and the image signal the image processing apparatus (i.e., encoders) functions as a. In the present embodiment, the encoder when encoding the image, based on the transfer function, the prediction residual code amount or mode code amount for mode selection is controlled. Thereby, inappropriate mode selection when the signal format for HDR is used is avoided, deterioration of the image quality is reduced. From the next section, it will be described in detail specific and exemplary configuration of such an image processing apparatus.
[0146]
　　[3-3. Schematic configuration] of the image processing apparatus
　17A is a block diagram showing a first example of a schematic configuration of an image processing apparatus according to this embodiment. The image processing apparatus 200a shown in Figure 17A includes a signal acquisition unit 201, the information acquisition unit 203, the encoding unit 210 and the control unit 240.
[0147]
　Signal acquisition unit 201 acquires an input image signal generated based on the transfer function for the conversion between the optical and the image signal. Signal acquisition unit 201, an input from obtaining input image signal from an external device, or the image processing apparatus 200a and integrally formed imaging module and a signal processing module (not shown) via a transmission interface the image signals may be obtained.
[0148]
　Information acquisition unit 203 acquires input information about transfer function applied to the image to be encoded by the encoding unit 210. As an example, the information acquisition unit 203, the input information via the user interface of the image processing apparatus 200a has may be acquired. User interface, the image processing apparatus 200a, for example, a touch panel provided on the housing, may be provided by physical input devices such as buttons or switches. Alternatively, the user interface may be provided as a GUI on the terminal device that is remotely connected through a communications interface. In this embodiment, the input information includes a transfer function type indicating the type of the transfer function applied to the image to be at least coded. The user interface, for example, may be selected one to be applied to the image of the two choices "SDR" and "HDR" to the user. In this case, one of the transfer function previously defined for SDR, or one of the transfer function previously defined for HDR is determined to have been applied to the image. Further, the user interface, from the candidates of a plurality of transfer functions, the transfer function to be applied to the image may be selected by a user.
[0149]
　As another example, the information acquisition unit 203 may acquire the input information from the auxiliary signal inputted image signal and multiplexing. For example, the auxiliary signal, the image signal in signal line is received by the signal acquisition unit 201 in a period with no transmissions. Then, the information obtaining unit 203, the auxiliary signal separated in the signal acquisition unit 201 may obtain input information including the transfer function type indicating the type of the transfer function applied to the image.
[0150]
　Encoding unit 210, an image represented by the image signal acquired by the signal acquiring unit 201 and encodes, to generate an encoded bit stream. Encoding unit 210, for example, MPEG-2, H. 264 / AVC or H. It may perform an encoding process in accordance with any video coding scheme, such as 265 / HEVC. Encoding process performed by the encoding unit 210 is typically prediction, orthogonal transformation, include various processes such as quantization and entropy coding, various mode selection is performed in those processes. Although described herein primarily for mode selection in the intra prediction and inter prediction, the idea of ​​the present embodiment, for example, transform block size selection, or the like prediction mode selection inter-layer predictive for scalable coding, any type it may also be utilized for mode selection.

The scope of the claims
[Requested item 1]
　An encoding unit for encoding an image obtained on the basis of the transfer function for the conversion between the optical and the image signal,
　based on the transfer function, mode selection when encoding the image in the encoding unit a control unit for controlling the prediction residual code amount or mode code amount for the
　image processing apparatus comprising a.
[Requested item 2]
　Wherein, when the first transfer function corresponding to the first dynamic range is applied to the image, a second transmission corresponding to the narrower second dynamic range than the first dynamic range as the mode code amount becomes smaller than if the function is applied to the image, to control said mode code amount, the image processing apparatus according to claim 1.
[Requested item 3]
　Wherein, when the first transfer function corresponding to the first dynamic range is applied to the image, a second transmission corresponding to the narrower second dynamic range than the first dynamic range so much said prediction residual code amount than if the function is applied to the image, and controls the prediction residual code amount, the image processing apparatus according to claim 1.
[Requested item 4]
　The first dynamic range is the dynamic range for enabling the display of a higher than 100nit brightness, the image processing apparatus according to claim 2.
[Requested item 5]
　Encoding process performed by the encoding unit includes an intra prediction,
　the prediction residual code amount or the mode code quantity is controlled by the control unit, the mode selection from the plurality of candidate modes in the intra prediction It is used in the,
　image processing apparatus according to claim 1.
[Requested item 6]
　Encoding process performed by the encoding unit includes a inter prediction,
　the prediction residual code amount or the mode code quantity is controlled by the control unit, the mode selection from the plurality of candidate modes in the inter prediction It is used in the,
　image processing apparatus according to claim 1.
[Requested item 7]
　Encoding process performed by the encoding unit includes an intra prediction and inter prediction,
　the prediction residual code amount or the mode code quantity is controlled by the control unit is a the intra prediction or the inter prediction It is used in the selection of the prediction method,
　the image processing apparatus according to claim 1.
[Requested item 8]
　Wherein, based on the input information on the transfer function applied to the image, said determining the type of transfer function, the determined the type wherein the prediction residual code amount or controlling the mode code amount based on to image processing apparatus according to claim 1.
[Requested item 9]
　Among the plurality of candidate modes, the mode selection unit for selecting a mode cost including the prediction residual code amount and the mode code amount becomes smallest, further comprising a
　said coding unit is selected by the mode selection unit that the image is encoded according to the mode,
　the image processing apparatus according to claim 1.
[Requested item 10]
　The input information is information obtained through the user interface, the image processing apparatus according to claim 8.
[Requested item 11]
　The input information, the image is information obtained from the input image signal and the auxiliary signals to be multiplexed to represent an image processing apparatus according to claim 8.
[Requested item 12]
　The image processing apparatus further comprises said mode code amount of value associated with the type of transfer function, or, a storage unit, for storing parameters for controlling the prediction residual code amount or the mode code amount, wherein the image processing apparatus according to claim 8.
[Requested item 13]
　And encoding the image obtained on the basis of the transfer function for the conversion between the optical and the image signal,
　based on the transfer function, the prediction residual code for the mode selection when encoding the image and controlling the amount or mode code quantity,
　an image processing method comprising.
[Requested item 14]
　The processor of the image processing apparatus,
　an encoding unit for encoding an image obtained on the basis of the transfer function for the conversion between the optical and the image signal,
　based on the transfer function, the image in the encoding unit a control unit for controlling the prediction residual code amount or mode code amount for the mode selection when encoding
　program for functioning as a.