Technical field
[0001]
The present disclosure relates to an image processing apparatus, an image processing method, and a program.
BACKGROUND
[0002]
Recently, in order to be able to display an image the state of the real world in a more faithful reproduction or richer brightness and color, expansion of the video signal representation it has been promoted. HDR (High Dynamic Range) is a concept of trying to represent an image or video in a wide luminance dynamic range than a conventional standard dynamic range SDR (Standard Dynamic Range). For example, HLG (Hybrid Log-Gamma), converted into an image signal light by a transfer function such ST2084 or S-log3 (also referred to as a tone curve) (and, the image signal converted into light) by, than 100nit it is known that it is possible to reproduce light of the real world with a high luminance on the display (see non-Patent Document 1). As another example, standardized by ITU-R BT. 2020, have been used in many applications so far BT. 709 as compared to defines the gamut to be able to express a more vivid color.
CITATION
Non-patent literature
[0003]
Non-Patent Document 1: Association of Radio Industries and Businesses, "ESSENTIAL PARAMETER VALUES FOR THE EXTENDED IMAGE DYNAMIC RANGE TELEVISION (EIDRTV) SYSTEM FOR PROGRAMME PRODUCTION ARIB STANDARD", ARIB STD-B67 Version 1.0, July 3, 2015, [online], [November 24, 2016 search], イ nn center have Neko ッ Suites
Summary of the Invention
Problems that the Invention is to Solve
[0004]
　Video (or individual images constituting the image) is encoded or decoded, or existing devices utilizing such images have not been able always well adapted to the video signal representation that is being diversified. Optimized processing parameters specific expression technique assumes is not necessarily suitable for the processing of the image signals represented by another expression technique. It is considered similar situation for the image processing of the still image.
[0005]
　Therefore, it is desirable that a mechanism for preventing the improper processing parameters when dealing with images that can be expressed in a variety of expression technique is used are provided.
Means for Solving the Problems
[0006]
　According to the present disclosure, an image based on at least one of the color gamut applied to the transfer function and the image for the conversion between the optical and the image signal applied to the processing for the image processing of the image a control unit for determining parameters, using the processing parameter determined by the control unit, the image processing apparatus is provided comprising a processing unit for executing the image processing.
[0007]
　Further, according to the present disclosure, an image based on at least one of the color gamut applied to the transfer function and the image for the conversion between the optical and the image signal applied to, for image processing of the image determining a processing parameter, using the processing parameters determined, the image processing method comprising, and performing the image processing is provided.
[0008]
　Further, according to the present disclosure, the processor of the image processing apparatus, based on at least one of the color gamut applied to the transfer function and the image for the conversion between the optical and the image signal applied to the image, a control unit for determining a processing parameter for image processing of the image, using the processing parameter determined by the control unit, and a processing unit that executes the image processing, the program to function as is provided that.
Effect of the invention
[0009]
　According to the technique of the present disclosure, it is possible to prevent the improper processing parameters when dealing with images that can be expressed in a variety of expression technique is used.
　Incidentally, the above effect is not necessarily restrictive, with the above effects, or instead of the above effects, any effects shown herein, or other effects that may be grasped from the description it may be achieved.
BRIEF DESCRIPTION OF THE DRAWINGS
[0010]
Is an explanatory diagram for describing luminance dynamic range of FIG. 1A] SDR image.
[FIG 1B] is an explanatory diagram for describing luminance dynamic range of HDR images.
Is an explanatory diagram for describing a codec distortion of the image signal in FIG. 2A] SDR image.
Is an explanatory diagram for describing a codec distortion of the image signal in FIG. 2B] HDR image.
Is an explanatory diagram for describing a codec distortion is enlarged through FIG 2C] HDR-SDR conversion.
3 is an explanatory diagram showing an example of a OETF signal format and signal format for HDR for SDR.
[Figure 4] BT for the SDR. S-log3 for HDR is a graph representing how much compressed image information to 709.
[Figure 5] BT. 709 and BT. It is an explanatory diagram for describing the color gamut defined by 2020.
[6] The difference in the type of transfer function is an explanatory diagram for describing the impact on the coding difficulty.
[FIG 7A] is an explanatory diagram showing a first example of a configuration of an image processing system according to an embodiment.
[FIG 7B] is an explanatory diagram showing a second example of a configuration of an image processing system according to an embodiment.
[FIG 8A] is a block diagram showing a first example of a schematic configuration of an image processing apparatus according to the first embodiment.
[FIG 8B] is a block diagram showing a second example of a schematic configuration of an image processing apparatus according to the first embodiment.
9 is a block diagram showing an example of a detailed configuration of the control unit according to the first embodiment and the coding unit.
Is an explanatory view showing an example of FIG. 10 lower high block and the encoding degree of difficulty encoding difficulty block.
11 is an explanatory diagram for describing an example of the adjustment of the quantization step in the SDR case.
FIG. 12 is an explanatory diagram for describing an example of the adjustment of the quantization step in the HDR case according to existing methods.
13 is an explanatory diagram for describing an example of the adjustment of the quantization step according to the first embodiment.
14 is a flowchart showing an example of the flow of encoding control processing according to the first embodiment.
It is an explanatory diagram for explaining an example of the influence of the color gamut of the pixel classification process in FIG 15A] SDR.
[FIG. 15B] is a first explanatory diagram for explaining an example of the effect of the transfer function to the pixel classification process in HDR.
[FIG. 15C] is a second explanatory diagram for explaining an example of the effect of the transfer function to the pixel classification process in HDR.
[FIG. 15D] is an explanatory diagram for explaining an example of the influence of the color gamut of the pixel classification process in HDR.
16 is a block diagram showing an example of a schematic configuration of an image processing apparatus according to the second embodiment.
17 is a block diagram showing an example of a detailed configuration of the control unit according to the second embodiment and the coding unit.
18 is a flowchart showing an exemplary flow of an encoding control process according to the second embodiment.
19 is a block diagram showing an example of a configuration of an image processing apparatus according to a modification of the second embodiment.
FIG. 20 is a flowchart illustrating an example of an image processing flow according to the modified example described with reference to FIG. 19.
Is a block diagram showing an example of the hardware configuration of FIG. 21 apparatus.
22 is a diagram schematically showing an overall configuration of the operating room system.
23 is a diagram showing a display example of operation screen in the intensive operation panel.
It is a diagram showing an example of the state of FIG. 24 operating room surgery system is applied.
Is a block diagram illustrating an example of FIG. 25 of the camera head and CCU shown in FIG. 24 functional configuration.
DESCRIPTION OF THE INVENTION
[0011]
　Reference will now be described in detail preferred embodiments of the present disclosure. In the specification and the drawings, components having substantially the same function and structure are a repeated explanation thereof by referring to the figures.
[0012]
　Also, a description in the following order.
　　1. Description of the Related Art
　　　1-1. SDR and
　　　HDR 1-2. Codec distortion
　　　1-3. Transfer function
　　　1-4. Color gamut
　　2. First Embodiment
　　　2-1. Introduction
　　　2-2. System Overview
　　　2-3. Schematic configuration of the image processing apparatus
　　　2-4. Detailed structure of the encoding unit and the control unit
　　　2-5. Processing of flow
　　　2-6. Summary of the first embodiment
　　3. Second Embodiment
　　　3-1. Introduction
　　　3-2. System Overview
　　　3-3. Schematic configuration of the image processing apparatus
　　　3-4. Detailed structure of the encoding unit and the control unit
　　　3-5. Processing of flow
　　　3-6. Modification
　　　3-7. Summary of the second embodiment
　　4. Hardware Configuration Example
　　5. Application Example
　　6. Summary
[0013]
　<1. Description of Related
　　Art> [1-1. SDR and HDR]
　In recent years, in order to be able to play a video the state of the real world in a more faithful reproduction or richer brightness and color, expansion of the video signal representation has been promoted. HDR is a concept attempts to represent an image or video in a wide luminance dynamic range than a conventional standard dynamic range SDR.
[0014]
　Figure 1A is an explanatory diagram for describing luminance dynamic range SDR image. The vertical axis of FIG. 1A represents luminance [nit]. Maximum brightness in nature may reach 20000Nit, luminance of a typical object is 12000nit about a maximum, for example. The upper limit of the dynamic range of the image sensor may be a maximum lower than the luminance, for example 4000nit in nature. Digital camera or digital camcorder such as imaging devices, an electric signal generated by photoelectric conversion of incident light in the image sensor, converts the subsequent signal processing circuit of the image sensor, for example to 10 bits of digital image signals. The signal format of the legacy SDR image, gradation of the high luminance portion over a 100nit during such conversion is lost. Digital image signal generated by the imaging device, for example, depending on the purpose of application such as transmission or recording, is encoded in a predetermined video coding method (also referred to as video codecs) are converted into coded bit stream. Then, when displaying the SDR video, digital image signals obtained by decoding the encoded bit stream is provided to the display device, the image is reproduced by the display luminance of the upper 100 nit.
[0015]
　Figure 1B is an explanatory diagram for describing luminance dynamic range of HDR images. Like the SDR case, the imaging device, the incident light to the image sensor and converted into an analog electrical signal, further converts the analog electrical signal into for example 10-bit digital image signal. Signal format of HDR images, when such conversion, to maintain the gradation of a high luminance portion exceeding 100 nit, makes it possible to reproduce the video the luminance of up to an upper limit of several hundred or thousand nit. Digital image signal generated by the imaging device is encoded in a predetermined video coding scheme also depending on the purpose of application, it is converted into coded bit stream. When displaying the HDR image is a digital image signal obtained by decoding the encoded bit stream is provided to the display device, the image is reproduced by the luminance dynamic range, including high display luminance than 100 nit.
[0016]
　Incidentally, as the reference for classifying the SDR and HDR, where it is assumed the case where the upper limit of the luminance dynamic range is below or equal to 100 nit SDR, a case where the upper limit is above 100 nit and the HDR. However, at some point in the future, the higher the reference value rather than the 100 nit, widespread at that time (i.e., has become the standard) and dynamic range (having a higher upper limit) newer and a dynamic range each may be be classified as HDR and SDR. The technology according to the present disclosure generally two types of dynamic range is widely applicable to the case having a different upper limit each other, but not limited to whether they dynamic range is classified by what kind of reference values. SDR, in comparison with HDR, sometimes referred to as LDR (Low Dynamic Range).
[0017]
　　[1-2. Codec Distortion
　whether SDR image or HDR image, when coded by the video coding method comprising the lossy compression of the image signal, the image reproduced on the basis of the image signal to be decoded, the deterioration of the image quality occurs. The deterioration of these image quality, herein referred to as codec distortion. The degree of codec distortion can be evaluated in the indication that PSNR (Peak Signal-to-Noise Ratio). Generally, when the coding efficiency equivalent, H. 264 / quality of the encoding / decoding images in AVC is higher than the image quality of the encoded / decoded images MPEG-2, H. 265 / quality of the encoding / decoding images in HEVC is H. Higher than the 264 / AVC. However, usually, the evaluation of the codec distortion is performed by comparing the decoded image output from the original image and the decoder input to the encoder. Signal conversion performed when the imaging or display of the HDR image, or either reduction or expansion of the dynamic range acts how the codec distortion, is not well known.
[0018]
　It, converted into an image signal a number of samples the video in signal format for HDR, H. After encoding and decoding by the encoder and decoder compliant with 264 / AVC, an experiment was conducted to verify the quality of HDR images reproduced from the image signal after decoding. As a result, the same sample in deterioration of the image quality that was not detected in the SDR video, that there is a case to be visible in HDR image through the video codec is recognized. Degradation of image quality, mainly in the form such as block noise or mosquito noise, had occurred remarkably in a portion of the image.
[0019]
　The degree of degradation resulting in encoding the image signal of the same 10-bit with the same video coding scheme is usually comparable. But the sensed no (or sensed hardly) strain in SDR image is detected in the HDR image is believed to be because the codec distortion is expanded together in extending the dynamic range of the image signal after decoding.
[0020]
　FIG 2A, the image signal of the SDR image, how codec distortion through encoding and decoding occurs is shown. Since codec distortion is not expanding when reproducing SDR video, does not strain the subjective distortion sufficiently small sensed. On the other hand, in FIG. 2B shows a state which is also codec distortion occurs in the image signal of the HDR image. When reproducing an HDR image is a result of the codec distortion with the expansion of the dynamic range is expanded, more likely to deteriorate in image quality such as block noise or mosquito noise is detected subjectively it is.
[0021]
　Codec distortion can be enlarged even when the format conversion into SDR is run from HDR an image signal represented by a signal format for HDR. The Figure 2C, the format conversion to SDR from HDR, i.e. how the codec distortion is enlarged through the HDR-SDR conversion is shown. HDR-SDR conversion is generally to an original signal corresponding with the inverse of the transfer function corresponding to the signal format for HDR (e.g. obtained by decoding the coded bit stream) image signal to an output of the image sensor including a process for restoring, and processing for re-converted into an image signal for SDR in transfer function corresponding the restored original signal into a signal format for SDR. Codec distortion is enlarged in the former of these processes are not reduced in the re-conversion to the signal format for SDR. Therefore, when playing SDR image based on the image signal after HDR-SDR conversion, enlarged codec distortion may result sensed subjectively.
[0022]
　Codec distortion as described above, if due to the performance of the video encoding method itself, the distortion should occur uniformly. However, the verification of the sample image as described above, that the strain has become prominent in characteristic part regions as illustrated below was confirmed:
　　- a specific color region (e.g., skin color region)
　　- a flat area ( for example, there is no building wall) textured
cause of distortion remarkable in these partial regions, depending on the signal selection of the transfer function and the color gamut of the signal format for HDR, each code value is the meaning of the color components it is that there is a difference in tone or color in the real world to be.
[0023]
　　[1-3. Transfer function]
　In general, characteristics of the signal conversion to the image signals from the light in the imaging device, OETF; is modeled with (Opto-Electronic Transfer Function photoelectric transfer function). Figure 3 shows examples of each of OETF typical OETF and signal format for HDR signal format for SDR. 3, the horizontal axis represents the luminance dynamic range of the pre-conversion of light, 100% corresponds to the luminance of 100 nit. The vertical axis represents the sign value of the image signal after the conversion, in the case of 10bit the code value can take a value from 0 to 1023. Signal format for SDR shown by a broken line in FIG. (E.g., BT.709) OETF (e.g., HLG, ST2084 or S-log3) for HDR shown in OETF and solid line when the comparing, code value is particularly the difference of the slope of the transfer function at a relatively large portion is remarkable. This is because, in this part, the image information compared to the SDR in HDR case is compressed at a higher compression ratio, i.e. the degree of change in code value is greater than the case of the SDR in case of HDR floors It is meant to represent a change of tone. Red (R) component in the RGB color system, a green even when analyzing each of the transfer function of the component (G) and blue (B) component, between similar HDR and SDR to the graph shown in FIG. 3 the difference of the signal transfer characteristic was confirmed.
[0024]
　4, BT for the SDR. S-log3 for HDR is a graph representing how much compressed image information to 709. 4, the horizontal axis represents the sign value of 10 bit image signal. The vertical axis, BT. It represents the ratio of the compression ratio of the S-log3 for 709 compression ratio. In the code value "600" near the S-log3 equivalent to 100% of the luminance dynamic range, the compression ratio of the S-log3 is BT. 709 is about 4 times the compression ratio, the compression ratio of the code values ​​larger the S-log3 is higher than relatively. From this graph, the code value is at a relatively large part, in the case of HDR is understood that the image information is compressed more strongly as compared with the SDR case.
[0025]
　When reproducing an HDR image is often, EOTF is an inverse function of OETF as indicated by the solid line in FIG. 3; to apply (Electro-Optical Transfer Function electric optical transfer function) to the code value of the image signal it causes the voltage level to be supplied to the display element can be determined. Then, the individual images constituting the HDR image is displayed in an expanded luminance dynamic range by the application of EOTF. Called OOTF the transfer function of the entire system including the up display from the imaging, OOTF is sometimes referred to as a system gamma. As used herein, "transfer function", unless otherwise specifically stated, OETF, refers to any one or more combinations of EOTF and OOTF. These transfer functions are sometimes referred to as the tone curve.
[0026]
　　[1-4. Gamut]
　As a technique that enables more faithful reproduction or from rich images representing the real-world situation, it is also important concept gamut with HDR. Standardized by ITU-R has been BT. 2020, have been used in many applications so far BT. 709 as compared to the color gamut of, defines a color gamut to be able to express a more vivid color. 5, BT. 709 and BT. It is an explanatory diagram for describing the color gamut defined by 2020. Referring to FIG. 5, the color gamut graph mapping the three-dimensional color space into a two-dimensional plane to a predetermined restricting condition is shown. Cross mark in the graph shows the position where the white color is mapped. The broken line in the graph, BT. It shows the range of colors that can be expressed according to 709. The solid line in the graph, BT. It shows the range of colors that can be expressed according 2020. The dotted line in the graph indicates the range of colors that can be human vision to identify. As understood from FIG. 5, BT. 2020, BT. Than 709 makes it possible to express a variety of colors. BT. 709 While it is possible representation is about 75 percent of the colors that exist in the real world, BT. 2020 is said to be possible to express the color of more than 99%. BT. 2020 may be utilized as a color gamut of SDR video, or may be utilized as the color gamut of the HDR video.
[0027]
　<2. First
　　Embodiment> [2-1. Introducing]
　above codec distortion was to be pronounced in the partial region in an image when using a signal format for HDR may cause that the assigned code amount for representing the gray level of the original signal in those partial areas is insufficient It is set to. MPEG-2, H. 264 / AVC or H. Encoder that conforms to the video encoding system such as 265 / HEVC, in order to achieve the required compression ratio is quantized in the frequency domain image signal. Typically, transform coefficients obtained by orthogonal transformation of the prediction residual after applying the prediction technique such intra prediction or inter prediction is quantized. However, method of determining the quantization step used by the optimized them encoder for encoding the SDR image becomes often non-optimal in the case where the signal format for HDR can be used.
[0028]
　Method of determining a number of quantization steps that are employed in the encoder can be expressed as the following equation (1).
[0029]
[Number 1]

[0030]
　Here, Q i , regardless of the content of the image, the required compression rate according to the purpose of application is determined so as to achieve, it represents a temporary quantization step i th block in the image . Q' i denotes the quantization step after adjustment is adjusted based on the coding difficulty for each block. Act Bi represents an activity that is statistically calculated for the i-th block. Activities is one indicator of the encoding difficulty, for example, equal to the minimum value of the dispersion are calculated for a plurality of sub-blocks within the block. Instead of activity, other parameters such as the maximum difference between the code value and the minimum code value (also sometimes referred to as "dynamic range") or before adjustment of the quantization step itself, used as an indicator of the encoding difficulty it may be. Function F (Act Bi ) is the argument of the activity (or other indicators of coding difficulty) for each block, a function that returns the adjustment coefficient of the quantization step. Function F (Act Bi two examples of), F 1 and F 2 shown below as:
[0031]
[Number 2]

[0032]
　Act PIC represents typical values of the activity over the whole image (for example, mean, median or maximum value, etc.). According to equation (2), G is activity Act Bi is activity Act of the entire image PIC of equal to, provisional quantization step Q i adjusted quantization step Q'for i corresponds to the ratio of. Activity Act Bi when varies between zero and infinity, the function F 1 (Act Bi return value) varies between 2G from G / 2. Remaining terms except the G of the right side of equation (2), the activity Act Bi has a function to normalize the. According to equation (3), G, the activity Act Bi corresponds to the adjustment gain when equal to zero (i.e., when the i-th block is completely flat). Activity Act Bi when varies between zero and infinity, the function F 2 (Act Bi return value) varies between 1 / G of G. In this specification, these G of basic adjustment gain. In the case of the formula (3), G may be referred to as the maximum adjustment gain or the maximum protection ratio.
[0033]
　According to equation (2) or Formula (3), i-th block B i if image is relatively flat with respect to other blocks, adjusted quantization step Q' i is to a smaller value It is adjusted. As quantization step used for a block is smaller, transform coefficients of the block are quantized finely. This means more that the code amount is assigned to the block, i.e. the gradation of the image of the block is maintained without further impaired.
[0034]
　However, in the HDR case, tone information when the comparison the signal converted SDR case is compressed more strongly. Therefore, the complexity of the appearance in the real world (or flatness in the opposite sense) if is comparable, the coding difficulty is calculated from the code values of the image signal in HDR cases, than the case of the SDR smaller. In the upper part of FIG. 6, the image Im1 is shown as an example. Block B i is the i-th block that is set in the image Im1. Block B i for, than the coding difficulty is calculated from the image signal in the case of transfer function for the SDR is applied (Figure bottom left arrow D1), from the image signal in the case of transfer function for the HDR is applied coding difficulty to be calculated smaller (arrow D2 in figure below right). Thus for to provide protection comparable subjective image quality degrees two different image signals, such information compression function F includes a gain of formula (1) (for example, formula (2) or formula ( 3 basic adjustment gain G) of the) is desirably variably set so as to cancel variations in statistics due to the compression of information.
[0035]
　BT explained with reference to FIG. Even cases where a wider color gamut as a color gamut 2020 is used, for example, BT. Narrower color gamut as 709 compared to the cases used, the color information is compressed more strongly. Thus, for applied two image signals applied color gamut no different transfer function is also to provide protection comparable subjective image quality, compression amount of information in determining the quantization step flexible control such as to compensate for is desirably carried out.
[0036]
　Therefore, this section, what kind of flexibility to determine the quantization step transfer function or the color gamut of the type is based on whether applied to an image, the embodiment to provide an improved protected for subjective image quality It will be described.
[0037]
　　[2-2. System Overview
　FIG. 7A is an explanatory diagram showing a first example of a configuration of an image processing system according to the present embodiment. The image processing system 10a shown in FIG. 7A, includes an imaging device 11, the signal processing device 14 and server device 15,.
[0038]
　Imaging device 11, for example, a digital video camera or digital still camera, or any type of device having an image capturing function (e.g., a surveillance camera, Web cameras or data terminal, etc.). Imaging device 11, by taking a picture of the real world using an image sensor, and generates a primitive image signal. The signal processing device 14 can be, for example, BPU (Baseband Processing Unit), is connected to the imaging device 11. The signal processing unit 14, the primitive image signal generated by the imaging device 11 performs AD conversion and digital signal processing to generate an image signal in a predetermined signal format. Digital signal processing performed by the signal processor 14 may include, for example, gamma correction and color conversion. The signal processing unit 14, the imaging device 11 may be integrally constructed.
[0039]
　The signal processing unit 14 may generate an image signal by a transfer function and color gamut are selected from a plurality of candidates by the user via some user interface. As an example, a candidate of the transfer function, BT for the SDR. 709 may include HLG, ST2084 and S-log3 for the HDR. In addition, the candidate of the color gamut, BT. 709, BT. It may include 2020 and S-Gamut.
[0040]
　The signal processing unit 14, the image signal generated as a result of the signal conversion, multiplexes the auxiliary signal including voice signals and metadata as needed, and outputs them signals multiplexed to the server device 15. Server device 15 is, for example SDI (Serial Digital Interface), or an image processing apparatus connected to the signal processor 14 via a signal line conforming to a transmission protocol such as HD-SDI. Server device 15 acquires the image signal transmitted from the signal processing unit 14, encodes the image by a predetermined image encoding method for generating a coded bit stream 17a. Coded bit stream 17a may be stored in the internal or external storage device of the server device 15, or other device connected to the server device 15 (e.g., display devices) may be transmitted to.
[0041]
　Figure 7B is an explanatory diagram showing a second example of a configuration of an image processing system according to the present embodiment. The image processing system 10b shown in FIG. 7B, includes an imaging device 12, storage device 13 and the terminal device 16.
[0042]
　Imaging device 12, for example, may be any type of device having a digital video camera, a digital camcorder or digital still camera, or a video shooting function. Imaging device 12, by taking a picture of the real world using an image sensor, and generates a primitive image signal. The imaging device 12 performs the AD conversion, a digital signal processing as described above in relation to the signal processor 14 to generate an image signal in a predetermined signal format. Imaging device 12, similarly to the signal processor 14 may generate an image signal by a transfer function and color gamut are selected from a plurality of candidates by the user via some user interface.
[0043]
　Imaging device 12 based on the image signal generated as a result of the signal conversion, encodes the image by a predetermined image encoding scheme to generate a coded bit stream 17b. Coded bit stream 17b, for example, may be stored as a video file, or may be provided to the storage device 13 or the terminal device 16 via the network.
[0044]
　Storage device 13 is a data storage for storing various image data. Storage device 13, for example, may store video files 17c generated by encoding an image at a predetermined video encoding method. The header area of ​​the video file, for example, identify the type of transfer function for the conversion between the optical and the image signal applied to the video content included in the video file, the type of color gamut, and the video encoding method, respectively parameters may be included. Storage device 13 may store a RAW image file 18 for recording the image signal before encoding (or signal before conversion) as RAW data. Storage device 13, for example in response to a request from the terminal device 16, the user provides to the terminal device 16 via the network file wishing to or editing and playback.
[0045]
　Terminal device 16 is an image processing apparatus having a function of reproducing or editing a video file that is stored by or storage device 13 is generated by the imaging device 12. Terminal device 16, for example, may generate a decoded image signal by decoding the encoded bit stream included in the video file 17b or 17c is acquired from the imaging device 12 or the storage device 13. The terminal device 16, a dynamic range conversion on the decoded image generated as such (e.g., HDR-SDR conversion or SDR-HDR conversion) may be executed. Further, the terminal device 16, the image signal included in the RAW image file 18 or the decoded image signal after dynamic range transform encoded in a predetermined video encoding method may generate a coded bit stream 17d.
[0046]
　Server device 15 in the example of FIG. 7A and the image pickup device 12 and the terminal device 16 in the example of FIG. 7B, are all image processing apparatus for encoding an image (i.e., encoders) functions as a. In the present embodiment, when these image processing apparatus encodes image, based on at least one of the transfer function and the color gamut applied to the image (e.g., based on their type or other attributes) Quantum process parameters are controlled about, whereby deterioration in image quality is reduced. From the next section, it will be described in detail specific and exemplary configuration of such an image processing apparatus.
[0047]
　　[2-3. Schematic configuration] of the image processing apparatus
　FIG. 8A is a block diagram showing a first example of a schematic configuration of an image processing apparatus according to this embodiment. The image processing apparatus 100a shown in Figure 8A, for example, the server device 15 in the example of FIG. 7A, or the imaging device 12 or the terminal device 16 in the example of FIG. 7B (or, the image processing module mounted on one of those devices ) it may be. The image processing apparatus 100a includes a signal acquisition unit 101, the information acquisition unit 103, the encoding unit 110 and the control unit 140.
[0048]
　Signal acquisition unit 101 acquires an input image signal generated based on the transfer function for the conversion between the optical and the image signal. Signal acquiring unit 101, an input from obtaining input image signal from an external device, or the image processing apparatus 100a and integrally formed imaging module and a signal processing module (not shown) via a transmission interface the image signals may be obtained.
[0049]
　Information acquisition unit 103 acquires input information about transfer function and the color gamut is applied to an image to be input to the encoding section 110. As an example, the information acquisition unit 103, the input information via the user interface of the image processing apparatus 100a has may be acquired. User interface, the image processing apparatus 100a, for example, a touch panel provided on the housing, may be provided by physical input devices such as buttons or switches. Alternatively, the user interface may be provided as a GUI (Graphical User Interface) on the terminal device that is remotely connected through a communications interface. In this embodiment, the input information includes at least a transfer function indicating the type of transfer function applied to the encoding to be image type, and a color gamut indicating the type of applied color gamut to the image type. The user interface, for example, BT for the SDR. 709, HLG, ST2084 and S-log3 may be user to select one of a plurality of candidate transfer functions type may include as a HDR. In addition, the user interface, BT. 709, BT. 2020 and the plurality may include S-Gamut one of the candidate of the color gamut type may be selected by a user.
[0050]
　As another example, the information acquisition unit 103 may acquire the input information from the auxiliary signal inputted image signal and multiplexing. For example, the auxiliary signal is a period during which an image signal in signal line is not transmitted (e.g., the blanking period) is received by the signal receiver 101 in. Then, the information acquisition unit 103, the auxiliary signal separated in the signal acquisition unit 101 may obtain input information including the transfer function type and the color gamut type. The information acquisition unit 103 may acquire the input information required to access to external data sources.
[0051]
　Encoding unit 110, an image represented by the image signal acquired by the signal acquiring unit 101 and encodes, to generate an encoded bit stream. Encoding unit 110, for example, MPEG-2, H. 264 / AVC or H. It may perform an encoding process in accordance with any video coding scheme, such as 265 / HEVC. Encoding process performed by the encoding unit 110 is typically prediction, orthogonal transformation, include various processing such quantization and entropy coding, among others quantization to achieve the required compression ratio a process including the lossy compression.
[0052]
　Control unit 140, based on at least one of the transfer function type and the gamut type indicated by the input information acquired by the information acquisition unit 103 controls the quantization process performed in the encoding unit 110. For example, the control unit 140, if a transfer function is applied either to rely to different adjusted gains to the image of the kind of type, the coding unit 110 may be adjusted quantization step in the quantization process. The control unit 140, if a different adjustment gain gamut kind of type is dependent on whether applied to an image, the encoding unit 110 may be adjusted quantization step in the quantization process. Here, based primarily on the type of transfer function and the color gamut is described an example in which the process parameters such as the quantization step is determined, the process parameters may be determined based on other attributes. The same applies to the second embodiment described later.
[0053]
　Figure 8B is a block diagram showing a second example of a schematic configuration of an image processing apparatus according to this embodiment. The image processing apparatus 100b shown in FIG. 8B also, for example, the server device 15 in the example of FIG. 7A, or the imaging device 12 or the terminal device 16 in the example of FIG. 7B (or, the image processing to be mounted on either of them device it may be a module). The image processing apparatus 100b is provided with a signal processing unit 102, the information acquisition unit 104, a storage unit 107, the encoding unit 110 and the control unit 140.
[0054]
　The signal processing unit 102 obtains the original image signal inputted from the imaging device via some transmission interface or device inside the signal line, or acquires the image signal from the image file stored in the storage unit 107 . The signal processing unit 102 performs digital signal processing, which may include, for example, gamma correction and color conversion for primitive image signal, and generates an image signal to be the target of coding in a predetermined signal format. Transfer function and the color gamut by the signal processing unit 102 is applied to the image is determined based on input information acquired by the information acquisition unit 104. The signal processing unit 102 outputs the generated image signal to the encoding unit 110.
[0055]
　Information acquisition unit 104 acquires input information about transfer function and the color gamut is applied to the image to be encoded by the encoder unit 110. For example, the information acquisition unit 104 (provided as provided or GUI by physical input device) of the image processing apparatus 100b has may acquire the input information via the user interface. As described above, the input information includes at least transfer function type and the gamut type. The user interface, for example, BT for the SDR. 709, HLG, ST2084 and S-log3 may be user to select one of a plurality of candidate transfer functions type may include as a HDR. In addition, the user interface, BT. 709, BT. 2020 and the plurality may include S-Gamut one of the candidate of the color gamut type may be selected by a user.
[0056]
　Storage unit 107 is a storage device for storing various image data. Storage unit 107, for example, may store image file that records a digital image signal before signal conversion. The user output interface of the image processing apparatus 100b has (not shown), it may be stored video file acquired from the external storage medium in the storage unit 107. The storage unit 107 may store the video file containing the encoded bit stream generated as a result of the encoding process performed by the encoding unit 110. Video file may be outputted to the external apparatus in response to the request.
[0057]
　Similar to the first example described with reference to FIG. 8A, the encoding unit 110, an image represented by the image signal input from the signal processing unit 102 encodes, for generating a coded bit stream. Control unit 140, based on at least one of the transfer function type and the gamut type indicated by the input information acquired by the information acquisition unit 104 controls the quantization process performed in the encoding unit 110. Coded bit stream generated by the coding unit 110 may be transmitted to the external apparatus of the image processing apparatus 100b, or may be stored as a video file in the storage unit 107.
[0058]
　　[2-4. Detailed Configuration of the encoding unit and the control unit
　in this section will be described in detail more specific structure of the encoding unit 110 and the control unit 140 shown in FIGS. 8A and 8B. Figure 9 is a block diagram showing an example of a detailed configuration of the encoding unit and the control unit according to the first embodiment.
[0059]
　　(1) coding unit
　Referring to Figure 9, the coding unit 110, reordering buffer 111, the block setting unit 112, the subtraction unit 113, orthogonal transform unit 114, a quantization unit 115, a lossless coding unit 116, an inverse quantization comprising a unit 121, inverse orthogonal transform unit 122, addition unit 123, a loop filter 124, a frame memory 126, switch 127, mode selection unit 128, an intra prediction unit 130 and the inter prediction unit 135.
[0060]
　Reordering buffer 111, the image data of a series of images represented by the image signal acquired by the signal acquiring unit 101 or the signal processing unit 102 rearranges according to a GOP (Group of Pictures) structure. Reordering buffer 111 outputs the image data after the rearrangement block setting unit 112, the intra prediction unit 130 and the inter prediction unit 135.
[0061]
　Block setting section 112 divides each of the image corresponding to the picture into a plurality of blocks. MPEG-2 and H. In 264 / AVC, a picture is divided in a grid pattern into a plurality of macro blocks having a fixed size, encoding processing is executed each macro block as a processing unit. Quantization process can be performed with smaller sub-blocks from being set in each macroblock as a processing unit. H. In 265 / HEVC, picture is divided into a plurality of coding units (Coding Unit) in quadtree form with variable sizes, coding process is performed each CU as a processing unit. Quantization process can be performed as a processing unit smaller conversion unit (Transform Unit) which is set to each CU.
[0062]
　Subtraction unit 113 calculates a prediction residual data is a difference between the image data and predicted image data input from the block setting unit 112, and outputs the prediction residual data to the orthogonal transform unit 114.
[0063]
　Orthogonal transform unit 114, a prediction residual data received from subtracting section 113, converts the image data in the spatial domain to the transform coefficient data in the frequency domain. Orthogonal transform performed by the orthogonal transform unit 114, for example and the like discrete cosine transform or discrete sine transform. The orthogonal transform unit 114 outputs the transform coefficient data to the quantization unit 115.
[0064]
　Quantization unit 115 quantizes the transform coefficient data input from the orthogonal transformation unit 114. For example, in many cases the free space in the buffer or the transmission path with respect to the size of the outputted encoded bit stream quantization step is set smaller, the quantization step is set large when the free space in the opposite space is low obtain. More specifically, the quantization unit 115, the quantization step in accordance with the tentatively determines the quantization step further of each block coding difficulty for each block as required compression ratio is achieved adjust. The quantization unit 115 quantizes the transform coefficient data by the quantization step after the adjustment. Transform coefficient data after quantization (hereinafter referred to as quantized data) is output to the lossless encoding section 116 and the inverse quantization unit 121.
[0065]
　Adjustment of the quantization step in the quantization unit 115 may be performed in accordance with the formula described above (1). Function of equation (1) F (Act Bi ) comprises a processing parameter representative of the adjusted gain of the quantization step. Quantization step as the adjusted gain is large is adjusted to a smaller value, consequently code amount allocation to the block is increased. An example of a parameter representing the adjusted gain is a fundamental adjustment gain G included in Equations (2) and (3). In this case, the quantization unit 115, the basic adjustment gain G and the encoding difficulty Act of each block Bi by the quantization step after adjustment is adjusted based on the, performing quantization processing for each block. In the quantization unit 115, different quantization steps for each color component may be used. May also be performed using the processing parameters are adjusted quantization step differs for each color component.
[0066]
　Lossless encoding unit 116, by encoding the quantized data input from the quantization unit 115, generates an encoded bit stream. Also, the lossless encoding unit 116, various parameters that are referenced by the decoder by encoding to insert the encoded parameters into the encoded bit stream. Parameters coded by the lossless coding unit 116, information about the transfer functions may include information regarding color gamut, and the information about the quantization parameter. Lossless encoding unit 116, the generated encoded bit stream and outputs to an output destination in accordance with the purpose of the application.
[0067]
　Inverse quantization unit 121, inverse orthogonal transform unit 122 and the addition unit 123 constitute a local decoder. Local decoder is responsible for reconstructing the original image from the encoded data.
[0068]
　Inverse quantization unit 121 inversely quantizes the quantized data in the same quantization step as that used by the quantization unit 115, restores the transform coefficient data. Then, the inverse quantization unit 121 outputs the transform coefficient data restored to the inverse orthogonal transform unit 122.
[0069]
　Inverse orthogonal transform unit 122, by executing the inverse orthogonal transform processing for transform coefficient data input from the inverse quantization unit 121, to restore the prediction residual data. Then, the inverse orthogonal transform unit 122 outputs the restored prediction residual data to the adder 123.
[0070]
　Adding section 123 by adding the predicted image data generated by the inverse orthogonal transform unit prediction residual restored input from 122 data and the intra prediction unit 130 or the inter prediction unit 135, generates decoded image data to. The adding unit 123 outputs the generated decoded image data to the loop filter 124 and a frame memory 126.
[0071]
　Loop filter 124, an in-loop filter for the purpose of improving the quality of the decoded image. Loop filter 124 may include, for example, a deblock filter for reducing block distortion appearing in the decoded picture. Further, the loop filter 124 may include an adaptive offset filter for adding the edge offset or band offset in the decoded picture. Loop filter 124 outputs the decoded image data after filtering to the frame memory 126.
[0072]
　The frame memory 126 stores the decoded image data after the application of the in-loop filter input before filtering the decoded image data input from the addition unit 123, and the loop filter 124.
[0073]
　Switch 127 reads out the decoded image data before filtering that is used for intra prediction from the frame memory 126, and supplies the intra prediction unit 130 read decoded image data as reference image data. The switch 127 reads the decoded image data after filtering to be used for inter prediction from the frame memory 126, and supplies the inter prediction unit 135 read decoded image data as reference image data.
[0074]
　Mode selection unit 128, based on the cost comparison of input from the intra prediction unit 130 and the inter prediction unit 135 selects a prediction method for each block. Mode selection unit 128, for the block where the intra prediction, outputs the predicted image data generated by the intra prediction unit 130 to the subtraction unit 113 outputs information on the intra prediction to the lossless encoding unit 116. The mode selection unit 128, the block where the inter prediction, outputs the predicted image data generated by the inter prediction section 135 to the subtraction unit 113 outputs information on the inter prediction to the lossless coding unit 116 .
[0075]
　The intra prediction unit 130, based on the original image data and the decoded image data, to perform the intra prediction process. For example, the intra prediction unit 130, for each of a plurality of candidate modes included in the search range, to evaluate the cost is estimated to occur. Then, the intra prediction unit 130 selects a prediction mode the minimum cost as the best prediction mode. Further, the intra prediction unit 130 generates a predictive image data according to the best prediction mode selected. Then, the intra prediction unit 130, information on an intra prediction including a prediction mode information indicating the best prediction mode, the corresponding cost, and predicted image data to the mode selection unit 128.
[0076]
　Inter prediction unit 135, based on the original image data and the decoded image data, to perform the inter prediction process (motion compensation). For example, inter prediction unit 135, for each of a plurality of candidate modes included in the search range, to evaluate the cost is estimated to occur. Next, the inter prediction unit 135 selects a prediction mode the minimum cost as the best prediction mode. Further, the inter prediction unit 135 generates a predictive image data according to the best prediction mode selected. The inter prediction unit 135, information on inter prediction, corresponding cost, and predicted image data to the mode selection unit 128.
[0077]
　　(2) Control Unit
　With reference to Figure 9, the control unit 140 includes a statistical computation unit 141 and the quantization controller 143.
[0078]
　Statistical calculation unit 141 calculates the coding difficulty throughout the image, and the encoding difficulty for each of the blocks set on the image. For example, statistical computation unit 141, as the encoding difficulty, activity Act described above Bi and Act PIC to calculate:. The statistical computation unit 141 outputs the calculated these statistics to the quantization control unit 143.
[0079]
　Quantization control unit 143, the transfer function and the color gamut is applied to the image to be encoded, it determines based on the input information inputted from the information acquisition unit 103 or 104. Then, the quantization control unit 143, the transfer function or the color gamut, or based on a combination thereof, determines the basic adjustment gain to be used for the adjustment of the quantization step. More specifically, the quantization control unit 143, so that the change in the code value of either or image signals due to both the difference of the transfer function and the color gamut counteract the effect on the quantization processing, the basic adjustment gain It can be determined. Effect here is typically appear as variations in the degree of the codec distortion resulting from the quantization can be perceived as degradation of subjective image quality. The degree of the codec distortion can be evaluated using an index called PSNR described above. For example, by verifying the pre while changing the adjusted gain in tuning to assess changes in PSNR or subjective image quality, transfer each function for each or color gamut or for each of those combinations, for canceling the effects described above You can determine the value of the basic adjustment gain value of the so determined basic adjustment gains may be stored in the storage unit 107. Then, the quantization control unit 143 corresponds to, for example, the transfer function or the color gamut, or a combination thereof, one of those predetermined value from the storage unit 107, the adjustment of the quantization step It can be determined as the value of the basic adjustment gain to be used for.
[0080]
　Hereinafter, with reference to FIGS. 10 to 13, illustrating the significance of the dynamic determination of the basic adjustment gain. Figure 10 shows a similar image Im1 to that illustrated in FIG. Block B1 is located in the region having a relatively complex image content in the image Im1, block B2 is located in the region having a relatively flat image content in the image Im1. 11, SDR cases (e.g., the transfer function in which BT.709) shows an example of an adjustment of the quantization step in. Here, if the quantization step which is tentatively determined for blocks B1 and B2 is assumed to be equal to Q together. Higher (activity greater than) the quantization step of the block B1 of the encoding difficulty is, Q' B1 is adjusted to. On the other hand, lower (activity smaller than) the quantization step of the block B2 of the encoding difficulty, Q' B1 smaller Q'than B2 is adjusted to. The width of these adjustments, the basic adjustment gain G has been successfully designed for SDR cases SDR governed by.
[0081]
　12, the case of HDR according to the existing method (for example, the transfer function at a HLG) shows an example of an adjustment of the quantization step in. According to the existing technique, fixed basic adjustment gain irrespective transfer function and gamut is applied to the image is used. On the other hand, the coding difficulty is calculated for the image which reflects the same subject, the image information when the signal conversion becomes smaller value if it is compressed more strongly, adjustable quantization step accordingly (1) adjusting the width of also becomes smaller than. Consequently, in the case of FIG. 12, the quantization step of the block B1 is Q' B1 Q'' greater than B1 is adjusted to the quantization step of the block B2 is Q' B2 Q'' greater than B2 to It can be adjusted with. Especially for the encoding difficulty lower flat block B2, Q'in the case of SDR of Figure 11 B2 Q'' in HDR case of FIG. 12 with respect to B2 is much larger. This is insufficient, the assigned code amount in a flat region in HDR cases, a cause of image deterioration has been a remarkable.
[0082]
　Figure 13 shows an example of an adjustment of the quantization step according to the present embodiment. In the present embodiment, as described above, the basic adjustment quantization control unit 143, based on at least one of the transfer functions and gamut is applied to the image, it is used in the adjustment of the quantization step switch the gain. In the example of FIG. 13, the basic adjustment gain G for SDR designed respectively according to the degree of compression of the image information in the HDR cases SDR basic adjustment gain G for and HDR HDR basic adjustment gain G of the HDR use It is. As a result, the quantization step of the block B1 is Q' B1 is adjusted to the quantization step of the block B2 is Q' B2 may be adjusted to. By switching such processing parameters, variations on statistics coding difficulty associated with the compression of the image information is compensated, the deterioration of image quality caused by the shortage of the assigned code amount is reduced.
[0083]
　Quantization control unit 143 calculates the basic adjustment gain determined in this way, the adjustment factor by using the differentially (the entire image and each block) coding difficulty calculated by the statistical operation unit 141, calculates was the adjustment factor may be provided to the quantization unit 115. Alternatively, the coding difficulty calculated by the basic adjustment gain and statistical calculation unit 141 which is determined by the quantization controller 143 is provided to the quantization unit 115, the formula (2) in the quantization unit 115 or the formula ( adjustment factor may be calculated according to 3). Storage unit 107, the value of the basic adjustment gain associated with one or both of the transfer function and the color gamut may store. The value of the basic adjustment gain is defined for each transfer function is defined for each color gamut, or may be defined for each combination of the transfer function and the color gamut.
[0084]
　Control of such quantization step is typically, MPEG-2 or H. Macroblock or sub-block in the 264 / AVC, or H. 265 / are performed for each block of rectangular such CU or TU in HEVC. However, the idea of ​​this embodiment is also applicable to the case where quantization control process for each partial region having a non-rectangular shape is performed.
[0085]
　　[2-5. Process Flow
　FIG. 14 is a flowchart showing an exemplary flow of an encoding control process according to the present embodiment. Encoding control process described herein may be repeated for each image constituting the video. Processing steps for obtaining or setting the parameters that do not change over a plurality of images may be skipped in the repetition of the second and subsequent. Here, for simplicity of description, description of the processing steps not directly related to control of the quantization process is omitted.
[0086]
　Referring to FIG. 14, first, the signal acquisition unit 101 or the signal processing unit 102 obtains an image signal generated based on a transfer function for the conversion between the optical and the image signal (step S110). Image signal obtained here is output to the encoding unit 110.
[0087]
　Next, the information acquisition unit 103 or 104, the input information indicating the transfer function and the color gamut is applied to an image to be input to the encoding section 110, is or the input image signal and the multiplexed via the user interface obtained from the auxiliary signal (step S112). Input information obtained here is outputted to the control unit 140.
[0088]
　Next, the quantization control unit 143, based on at least one of the transfer function and the color gamut indicated by the input information described above, to determine the basic adjustment gain (step S114). The statistical calculation unit 141 calculates the encoding difficulty of the entire picture, and outputs the calculated coding difficulty to the quantization control unit 143 (step S116).
[0089]
　Subsequent processing is repeated for each of a plurality of blocks set on the image to be processed. The block to be processed in each iteration, is referred to herein as the target block.
[0090]
　First, the quantization unit 115 of the encoding unit 110, regardless of what the transfer function and the color gamut is applied to determine the provisional quantization step of the target block as required compression ratio is achieved ( step S120).
[0091]
　The statistical calculation unit 141 calculates the encoding difficulty of the target block, and outputs the calculated coding difficulty to the quantization control unit 143 (step S130). Quantization controller 143 uses the encoding difficulty calculated by the basic adjustment gain and statistical calculation unit 141 determined in step S114, it determines the adjustment factor for the target block (step S140).
[0092]
　Quantization unit 115, by using the adjustment factor provided by the quantization controller 143 adjusts the quantization step determined in step S120 (step S150). Then, the quantization unit 115, the transform coefficient data of the target block input from the orthogonal transform unit 114, quantized by the quantization step after the adjustment (step S160). Here, although described as an example be adjusted using an adjustment factor tentatively determined quantization step as required compression ratio is achieved, both required compression ratio and adjustment factor it may be determined quantization step in consideration simultaneously.
[0093]
　Next, the reversible encoding unit 116 encodes the quantized data and the quantization parameter input from the quantization unit 115, generates an encoded bit stream (step S170).
[0094]
　Step S120 ~ step S170 are repeated until all the blocks in the picture processing is completed (step S180). When the processing for all the pictures completed, the encoding control process shown in FIG. 14 is ended (step S190).
[0095]
　　[2-6. Conclusion of first embodiment
　up to here, with reference to FIGS. 6 to 14, described for the first embodiment of the technology according to the present disclosure. In the embodiment described above, based on at least one of the transfer function and the color gamut for the conversion between the optical and the image signal, processing parameters related to the quantization processing at the time of encoding the image is determined. According to such a configuration, the difference in the transfer function, or inappropriate quantization step due to the difference in the color gamut can be prevented from being used. Thus, to ensure sufficient allocation amount of codes for representing the gray level of the original signal when encoding an image, it is possible to prevent deterioration in the image quality.
[0096]
　Further, in the embodiment described above, the basic adjustment gain is used to adjust the quantization step in accordance with the coding difficulty of each partial region of the image, based on at least one of the transfer functions and gamut It is determined. According to such a configuration, by using a relatively large basic adjustment gain is smaller case in transfer function or, depending on the selection of the gamut coding difficulty on statistics, coding difficulty conversely statistics can be used a relatively small basic adjustment gain is larger case above. Thereby, the adjustment of a consistent quantization step which is not affected by the method of representing an image signal is realized.
[0097]
　Further, in the above embodiment, based on the input information about the transfer function and the color gamut, the transfer function and the color gamut can be determined. If the input information is acquired through the user interface can execute the control of the street the user desires even if it can not determine these types from the input signal. If obtained from the auxiliary signal the input information is the input image signal and multiplexing can be determined automatically proper type without requiring user input.
[0098]
　<3. Second
　　Embodiment> [3-1. Introducing]
　As explained in the previous section, it is compressed at a higher compression ratio image information compared to the SDR in HDR cases. Even if the color gamut is different, the compression ratio of the image information are different. If the compression ratio is a difference in image information, the same code values of the image signal represents the brightness or color different in the real world. In the previous section, it was proposed to flexibly determined taking the quantization step which has been determined conventionally uniformly into account the differences in the meaning of this code value. However, in addition to the parameters relating to the adjustment of the quantization step, the processing parameters to be determined taking into account the differences in the meaning of the code values exist. Parameters relating to pixel classification process performed in various situations of the image processing is also applicable to such processing parameters.
[0099]
　Here, as an example of a pixel classification process, with reference to FIGS. 15A ~ FIG 15D, a pixel is examined process for classifying whether belonging to the skin color region. In these figures, the horizontal axis represents the sign value of one in which the Cb component of the two color difference components, and the vertical axis represents the sign value of the other is a Cr component of the two color difference components.
[0100]
　Rectangular region R11 of Fig. 15A is a human by a set of real-world which is recognized as being subjectively skin tone color, transfer function and BT for SDR. As a result of mapping onto Cb-Cr plane in the color gamut 709 is a region including a set of skin color after mapping. The left and right sides of the rectangular region R11 is whether the pixel corresponds to the threshold to be compared with the sign value of the Cb component in classifying whether belonging to the skin color regions, lower and upper, the pixel belongs to the skin color region It corresponds to the threshold to be compared with the sign value of the Cr component in classifying or. Transfer function and BT for the SDR the same set of skin color in the real world. When mapped onto Cb-Cr plane in the color gamut 2020, a set of their skin color after mapping are located in a rectangular area R12. When these two compared rectangular regions R11 and R12, the SDR case, when the different color gamuts of classification has been applied to the image, using different thresholds in the pixel classification process for classifying of whether the skin color pixel that it should be understood.
[0101]
　Rectangular region R2 in FIG. 15B, the same set of skin color in the real world, S-log3 and BT is the transfer function for the HDR. As a result of mapping onto Cb-Cr plane in the color gamut 2020, a region containing a set of skin color after mapping. Rectangular region R3 in FIG. 15C, the same set of skin color in the real world, is the transfer function for the HDR HLG and BT. As a result of mapping onto Cb-Cr plane in the color gamut 2020, a region containing a set of skin color after mapping. Rectangular area R3 occupy an area different from the rectangular region R2 on Cb-Cr plane. That this is when the different transfer functions of type even gamut same is applied to the image, which should use a different threshold value in pixel classification process for classifying of whether the skin color pixel it means.
[0102]
　Rectangular region R4 in FIG. 15D, the real world the same set of skin color, as a result of mapping onto Cb-Cr plane in the color gamut of S-log3 and S-Gamut is the transfer function for the HDR, the mapped is a region that contains a set of skin color. Rectangular region R4 occupy an area different from the rectangular region R2 on Cb-Cr plane. This, in the case of HDR, when the different color gamuts of classification even the transfer function the same is applied to the image, the use of different thresholds in the pixel classification process for classifying of whether the skin color pixel which means that it should.
[0103]
　These considerations also apply to other color is not limited to skin color. Generally, when trying to implement a pixel classification process of classifying the pixels by comparing the code values ​​of the color components and some threshold, appropriate threshold for the combination of transfer functions and gamut there is always transfer function and the color gamut of not appropriate for other combinations. Therefore, in this section, as the second embodiment, instead of using a uniform threshold for pixel classification, flexibly propose a method of switching the threshold value based on one or both of the transfer function and the color gamut.
[0104]
　　[3-2. Summary of the system
　image processing system according to the present embodiment may be configured similarly to the image processing system 10a or 10b in the first embodiment. As an example, has a function as an encoder that any of the devices in the system to encode the image, the particular region in the encoder (e.g., skin color region) area detecting process is performed to enhance the protection of the quality of the it may be. As another example, the image processing system according to this embodiment include any type of device for obtaining an image signal, the pixel for various purposes of face recognition, gesture recognition, such as biometrics or augmented reality in that device classification process may be performed. In this embodiment, the process parameters for the pixel classification is controlled on the basis of at least one of the transfer function and the color gamut in such a pixel classification process. Thereby, as compared with the example uniform processing parameters are used, is it possible to obtain a more accurate pixel classification results.
[0105]
　　[3-3. Schematic configuration] of the image processing apparatus
　16 is a block diagram showing an example of a schematic configuration of an image processing apparatus according to the second embodiment. The image processing apparatus 200a shown in FIG. 16 includes a signal acquisition unit 201, the information acquisition unit 203, the encoding unit 210 and the control unit 240.
[0106]
　Signal acquisition unit 201 acquires an input image signal generated based on the transfer function for the conversion between the optical and the image signal. Signal acquisition unit 201, an input from obtaining input image signal from an external device, or the image processing apparatus 200a and integrally formed imaging module and a signal processing module (not shown) via a transmission interface the image signals may be obtained.
[0107]
　Information acquisition unit 203 acquires input information about transfer function and the color gamut is applied to an image to be input to the coding section 210. As an example, the information acquisition unit 203, the input information via the user interface of the image processing apparatus 200a has may be acquired. User interface, the image processing apparatus 200a, for example, a touch panel provided on the housing, may be provided by physical input devices such as buttons or switches. Alternatively, the user interface may be provided as a GUI on the terminal device that is remotely connected through a communications interface. In this embodiment, the input information includes at least a transfer function indicating the type of transfer function applied to the encoding to be image type, and a color gamut indicating the type of applied color gamut to the image type. The user interface, for example, BT for the SDR. 709, HLG, ST2084 and S-log3 may be user to select one of a plurality of candidate transfer functions type may include as a HDR. In addition, the user interface, BT. 709, BT. 2020 and the plurality may include S-Gamut one of the candidate of the color gamut type may be selected by a user.
[0108]
　As another example, the information acquisition unit 203 may acquire the input information from the auxiliary signal inputted image signal and multiplexing. For example, the auxiliary signal, the image signal in signal line is received by the signal acquisition unit 201 in a period with no transmissions. Then, the information obtaining unit 203, the auxiliary signal separated in the signal acquisition unit 201 may obtain input information including the transfer function type and the color gamut type.
[0109]
　Encoding unit 210, an image represented by the image signal acquired by the signal acquiring unit 201 and encodes, to generate an encoded bit stream. Encoding unit 210, for example, MPEG-2, H. 264 / AVC or H. It may perform an encoding process in accordance with any video coding scheme, such as 265 / HEVC. Encoding process performed by the encoding unit 210 includes a quantization processing including lossy compression to achieve the required compression ratio.
[0110]
　Control unit 240 performs the pixel classification process based on at least one of the transfer function type and the gamut type indicated by the input information acquired by the information acquisition section 203, coding section 210 depending on the result of the pixel classification controlling the quantization process in. For example, the control unit 240 detects a specific area in an image using what kind of transfer functions and different thresholds depending on whether the combination of the color gamut is applied to the image. Then, the control unit 240, by scaling the quantization step to be applied to the specific area detected, to enhance the protection of the quality of a specific region.
[0111]
　　[3-4. Detailed Configuration of the encoding unit and the control unit
　in this section will be described in detail more specific structure of the encoding unit 210 and the control unit 240 shown in FIG. 16. Figure 17 is a block diagram showing an example of a detailed configuration of the encoding unit and the control unit according to the second embodiment.
[0112]
　　(1) coding unit
　Referring to Figure 17, the coding unit 210, reordering buffer 111, the block setting unit 112, the subtraction unit 113, orthogonal transform unit 114, a quantization unit 215, a lossless coding unit 116, an inverse quantization comprising a unit 221, inverse orthogonal transform unit 122, addition unit 123, a loop filter 124, a frame memory 126, switch 127, mode selection unit 128, an intra prediction unit 130 and the inter prediction unit 135.
[0113]
　Quantization unit 215 quantizes the transform coefficient data input from the orthogonal transformation unit 114. More specifically, the quantization unit 215, tentatively determining the quantization step for each block as required compression ratio is achieved, further adjusts the quantization step according to the result of the pixel classification. The quantization unit 215 quantizes the transform coefficient data by the quantization step adjusted, and outputs the quantized data to the lossless encoding section 116 and the inverse quantization unit 221.
[0114]
　Adjustment of the quantization step in the quantization unit 215 applies the adjusted gain provided by the control unit 240 in the provisional quantization step (e.g., multiplying the adjustment factor to the provisional quantization step) it may be carried out by. Quantization step as the adjusted gain is large is adjusted to a smaller value, consequently code amount allocation to the target block is increased. For example, by increasing the allocation of the code amount for a block belonging to the skin color region, protecting the quality of the region, such as a human face or hands reflected it can be enhanced. Quantization unit 215 may use a different quantization steps for each color component.
[0115]
　Inverse quantization unit 221 inversely quantizes the quantized data in the same quantization step as that used by the quantization unit 215, restores the transform coefficient data. Then, the inverse quantization unit 221 outputs the transform coefficient data restored to the inverse orthogonal transform unit 122.
[0116]
　　(2) Control Unit
　Referring to Figure 17, the control unit 240 includes threshold controller 241, a pixel classification unit 243 and the quantization controller 245.
[0117]
　Threshold controller 241, the transfer function and the color gamut is applied to the image to be encoded, it determines based on the input information inputted from the information acquisition unit 203. The threshold control section 241, the transfer function or the color gamut, or on the basis of a combination thereof, to be used in the pixel classification process, to determine the threshold to be compared with the sign value of the color component. For example, pixel classification process may be an area detection process for detecting a specific color area each pixel by classifying whether representing a specific color. The pixel classification process may be an area detection process for detecting specific brightness regions each pixel is classified whether representing the specific brightness. Not limited to these examples, any pixel classification process may be employed to classify into any category pixels. Threshold controller 241, for example, the transfer function or the color gamut, or referring to a memory for storing a threshold value associated with (not shown) a combination thereof, the threshold value associated with the type indicated by the input information, use it may be determined as the threshold should do. Typically, such a change in sign value of the image signal resulting from at least one of the differences of the transfer function and the color gamut counteract the effect on the pixel classification process, each transfer function, each color gamut, or their different threshold for each combination may be used.
[0118]
　Pixel classification unit 243, using a threshold determined by the threshold value control unit 241 performs pixel classification processing for the image represented by the input image signal. For example, if the pixel classification process is the skin color area detecting process, the threshold determined by the threshold value control unit 241 includes a skin color determination threshold. Skin color determination threshold corresponds to the boundary of Figure 15A ~ FIG 15D rectangular region illustrated in R11, R12, R2, R3 or R4, the two thresholds left and right sides are compared with the Cb component, lower and upper side Cr corresponding respectively to the two thresholds to be compared with the component. For example, the transfer function and BT for SDR. In the case where the color gamut of 709 is applied, the pixel classification unit 243, if each pixel as a result of the threshold determination is determined to be located in a rectangular area R11 on Cb-Cr plane, the pixel represents a skin color It may determine that. Transfer function and BT for the SDR. In the case where the color gamut 2020 is applied, the pixel classifying unit 243, if each pixel as a result of the threshold determination is determined to be located in a rectangular area R12 on Cb-Cr plane, the pixel represents a skin color It may determine that. In cases where other transfer function and gamut is applied may be performed is similar determination using the corresponding region detection threshold. Pixel classification unit 243 outputs a result of these pixel classification to the quantization control unit 245. Here, although described here is an example of a pixel classification of a simple threshold determination mainly approach pixel classification is not limited to such an example. For example, to determine whether each pixel in the area of ​​the shape is not a rectangular position, area determining argument to a code value of one or more color components may be utilized. Then, parameters such as coefficients or constants included in the area determination function may be switched in dependence on one or both of the combination of the transfer function and the color gamut. Alternatively, defined different identification plurality information is given region determination function beforehand, region determination identified by one or, depending on the combination of both the identification information to be selected from among the transfer functions and gamut use function may be used.
[0119]
　Quantization control unit 245, based on the results of pixel classification input from the pixel classifying unit 243 controls the quantization process performed in the quantization unit 215. For example, the quantization control unit 245, when the control target pixel or block is classified as belonging to a particular area which it is desired to enhance the protection of the image quality, for the pixels or blocks, from the provisional quantization step may the quantization step adjusted also to a small value is used in the quantization unit 215. The quantization controller 245, the pixels or blocks belonging to the area to be allowed a certain degree of deterioration in the image quality, the quantization step adjusted to a value greater than the provisional quantization step by using a quantization unit 215 it may be. By quantizing unit 215 the quantization step is adjusted based on the results of performed pixel classification using the processing parameters to be flexibly set as described above to perform the quantization process, method of representing image signal protection of consistent quality which is not affected by the is realized.
[0120]
　　[3-5. Process Flow
　FIG. 18 is a flowchart showing an exemplary flow of an encoding control process according to the present embodiment. Encoding control process described herein may be repeated for each image constituting the video. Processing steps for obtaining or setting the parameters that do not change over a plurality of images may be skipped in the repetition of the second and subsequent. Here, for simplicity of description, a description of the processing steps which are not related to the pixel classification is omitted.
[0121]
　Referring to FIG. 18, first, the signal acquisition unit 201 acquires an image signal generated based on a transfer function for the conversion between the optical and the image signal (step S210). Image signal obtained here is output to the encoding unit 210.
[0122]
　Next, the information acquisition unit 203, the input information indicating the transfer function and the color gamut is applied to an image to be input to the encoding unit 210, via the user interface or the input image signal and the auxiliary signal to be multiplexed obtained from (step S212). Input information obtained here is outputted to the control unit 240.
[0123]
　Next, the threshold value control unit 241, based on at least one of the transfer function and the color gamut indicated by the input information described above, to determine the threshold for pixel classification (step S214).
[0124]
　Subsequent processing is repeated for each of a plurality of blocks set on the image to be processed. The block to be processed in each iteration, is referred to herein as the target block.
[0125]
　First, the pixel classification unit 243, using a threshold determined by the threshold value control unit 241 executes the pixel classification for the target block (step S220). For example, in certain area detection case, the pixel classifying unit 243, each pixel in the block is to classify whether located in a particular area surrounded by threshold on Cb-Cr plane. Pixel classification unit 243 calculates the ratio of the number of pixels located in a specific area to the total number of pixels in the block. The pixel classification unit 243, if the percentage calculated exceeds a predetermined threshold value, the block of interest can be determined to belong to a specific region.
[0126]
　Quantization unit 215, regardless of what the transfer function and the color gamut is applied to determine the provisional quantization step of the target block as required compression ratio is achieved (step S230). Then, the quantization unit 215, using the adjustment factor determined by the quantization control unit 245 based on the results of pixel classification, to adjust the quantization step determined in step S230 (step S240). Then, the quantization unit 215, the transform coefficient data of the target block input from the orthogonal transformation unit 214, it is quantized by the quantization step after the adjustment (step S250).
[0127]
　Next, the reversible encoding unit 216 encodes the quantized data and the quantization parameter input from the quantization unit 215, generates an encoded bit stream (step S260).
[0128]
　Step S220 ~ step S260 are repeated until all the blocks in the picture processing is completed (step S280). When the processing for all the pictures completed, the encoding control process shown in FIG. 18 is ended (step S290).
[0129]
　　[3-6. Modification
　far, mainly describes an example that uses the result of the pixel classification for control of assignment code amount in the encoder. However, the idea of this embodiment is also applicable to the case of using the results of pixel classification for other purposes. This section will be described such a modification.
[0130]
　　(1) Configuration of the image processing apparatus
　19 is a block diagram showing a modification of the configuration of an image processing apparatus according to the second embodiment. The image processing apparatus 200b shown in FIG. 19 includes a signal acquisition unit 201, the information acquisition unit 203, the threshold control unit 250, a pixel classification unit 260 and the output signal generation unit 270.
[0131]
　Threshold control section 250, the transfer function and the color gamut is applied to the image, determines, based on input information inputted from the information acquisition unit 203. The threshold control section 250, the transfer function or the color gamut, or on the basis of a combination thereof, to be used in the pixel classification process, to determine the threshold to be compared with the sign value of the color component. Threshold control section 250, for example, the transfer function or the color gamut, or referring to a memory for storing a threshold value associated with (not shown) a combination thereof, the threshold value associated with the type indicated by the input information, use it may be determined as the threshold should do. Threshold controller 250 outputs the determined threshold to the pixel classification unit 260.
[0132]
　Pixel classification unit 260, using a threshold determined by the threshold value control unit 250 performs pixel classification processing for the image represented by the input image signal. For example, if the pixel classification process is the skin color area detecting process, the threshold determined by the threshold value control unit 250 includes a skin color determination threshold. For example, the transfer function and BT for SDR. In the case where the color gamut of 709 is applied, the pixel classification unit 260, if each pixel as a result of the threshold determination is determined to be located in a rectangular area R11 on Cb-Cr plane, the pixel represents a skin color It may determine that. Transfer function and BT for the SDR. In the case where the color gamut 2020 is applied, the pixel classifying unit 260, if each pixel as a result of the threshold determination is determined to be located in a rectangular area R12 on Cb-Cr plane, the pixel represents a skin color It may determine that. In cases where other transfer function and gamut is applied may be performed is similar determination using the corresponding skin color determination threshold. Pixel classification unit 260 outputs the result of such pixel classification to the output signal generation unit 270.
[0133]
　Output signal generation unit 270 generates an output signal based on the results of pixel classification input from the pixel classifying unit 260. For example, the output signal generation unit 270 may generate an output signal for displaying an indicator indicating the position of the specific region to be detected as a result of the pixel classification on the image (e.g., a frame surrounding the region). The output signal generation unit 270 performs a face recognition using the results of the skin color area detection may generate an output signal indicating the result of face recognition. Output signal generation unit 270, a result of the gesture recognition, the result of biometric authentication, or the output signal for displaying a display object that is superimposed on the output image in the augmented reality may be generated.
[0134]
　　(2) Processing Flow
　Figure 20 is a flowchart illustrating an example of an image processing flow according to the modified example described with reference to FIG. 19.
[0135]
　Referring to FIG. 20, first, the signal acquisition unit 201 acquires an image signal generated based on a transfer function for the conversion between the optical and the image signal (step S210). Image signal obtained here is output to the pixel classification unit 260.
[0136]
　Next, the information acquisition unit 203, the input information indicating the transfer function and the color gamut is applied to an image to be processed by the pixel classification unit 260, from the auxiliary signal or input image signals multiplexed through the user interface acquiring (step S212). Input information obtained here is output to the threshold control unit 250.
[0137]
　Next, the threshold value control unit 250, based on at least one of the transfer function and the color gamut indicated by the input information described above, to determine the threshold for pixel classification (step S214).
[0138]
　Next, the pixel classification unit 260, using a threshold determined by the threshold value control unit 250, for each pixel in the image to perform the pixel classification (step S225). Pixel classification is repeated until the pixel to be processed in the image disappears (step S270).
[0139]
　Then, the output signal generation unit 270, based on the results of the executed pixel classification in step S225, and generates an output signal (step S285).
[0140]
　The process described above is repeated for each of a series of images (step S290). If the next image is not, the encoding control process shown in FIG. 20 ends.
[0141]
　　[3-7. Conclusion of second embodiment
　up to here, with reference to FIGS. 15A ~ FIG. 20 and description of a second embodiment of the technology according to the present disclosure. In the embodiment described above, based on at least one of the transfer function and the color gamut for the conversion between the optical and the image signal, processing parameters are determined regarding pixel classification process that may be performed in a variety of purposes. According to such a configuration, the difference in the transfer function, or due to differences in color gamut and prevent the pixels are classified in an inappropriate basis, to ensure proper execution of the process based on the results of pixel classification can.
[0142]
　Further, in the above embodiment, the threshold value to be compared with the sign value of the color component in the pixel classification process is determined based on at least one of the transfer function and the color gamut. According to such a configuration, even when the same brightness or color in the real world is represented by a code value different depending on the choice of transfer function or gamut, consistent threshold which is not affected by the method of representing them image signal it is possible to provide a judgment.
[0143]
　For example, pixel classification process may be a region detecting process for detecting a specific area in an image, the threshold may be a region detection threshold. In this case, it can be prevented situation region which should not be detected originally Depending on the choice of transfer function or color gamut is detected erroneously as a specific area. According to one embodiment, the quantization step is adjusted based on the results of the above-described area detecting process, the quantization process for encoding an image is performed. In this case, a specific region which it is desired to enhance the protection of the quality accurately detected, it is possible to increase the assigned code amount of the specific area.
[0144]
　In the embodiments described above, based on the input information about the transfer function and the color gamut, the transfer function and the color gamut it can be determined. If the input information is acquired through the user interface can execute the control of the street the user desires even if it can not determine the transfer function or the gamut from the input signal. If obtained from the auxiliary signal the input information is the input image signal and multiplexing can be determined automatically proper type without requiring user input.
[0145]
　<4. Hardware Configuration Example>
　embodiment described previous sections, software, hardware, and any combination of software and hardware may be implemented using. If the image processing apparatus 100a, 100b, 200a or 200b uses the software, a program constituting the software is, for example, within or provided on an external storage medium (non-transitory media: non-transitory media) of the apparatus in advance in It is stored. Each program, for example, loaded into a RAM (Random Access Memory) during execution and executed by a processor such as CPU (Central Processing Unit).
[0146]
　Figure 21 is a block diagram showing an example of a hardware configuration of the applicable device one or more of the embodiments described above. Referring to FIG. 21, the image processing apparatus 900 includes a system bus 910, the image processing chip 920 and the off-chip memory 990. The image processing chip 920, the processing circuit 930-1,930-2 of n (n is 1 or more), ..., 930-n, the reference buffer 940, a system bus interface 950 and the local bus interface 960.
[0147]
　System bus 910, the image processing chip 920 and an external module (e.g., a central control function, an application function, such as a communication interface or user interface) to provide a communication path between the. Processing circuit 930-1,930-2, ..., 930-n is connected to the system bus 910 via the system bus interface 950, and is connected to the off-chip memory 990 via the local bus interface 960. Processing circuit 930-1,930-2, ..., 930-n can also access the on-chip memory (e.g., SRAM) reference buffer 940 may correspond to. Off-chip memory 990 may be, for example, a frame memory for storing image data processed by the image processing chip 920. As an example, processing circuitry 930-1 may be used for conversion of the image signal. Processing circuitry 930-2 may be utilized for encoding the image signal. Other processing circuitry may be utilized for the classification of pixels. Note that these processing circuits are the same image processing chip 920 without, or may be formed on a separate chip.
[0148]
　<5. Applications>
　technology according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure may be applied to the operating room system as described in this section.
[0149]
　Figure 22 is a diagram schematically showing an overall configuration of the operating room system 5100 technology according to the present disclosure may be applied. Referring to FIG. 22, operating room system 5100 is constructed by apparatus group installed in an operating room is cooperate connected to each other via the audiovisual controller (AV Controller) 5107 and operating room control system 5109.
[0150]
　The operating room, various devices may be installed. In Figure 22, as an example, endoscopes with various of the devices 5101 for mirror surgery, a ceiling camera 5187 for imaging the hand are caster provided on the ceiling of the operating room, operating room provided in the ceiling of the operating room the operative field camera 5189 for imaging the whole of a state, and a plurality of display devices 5103A ~ 5103D, a recorder 5105, a patient bed 5183, illustrates a lighting 5191, the.
[0151]
　Here, among these devices, the device group 5101, which belongs to the endoscopic surgical system 5113 to be described later, and a display device for displaying the image captured by the endoscope and the endoscope. Each device that belongs to the system for endoscopic surgery 5113 is referred to as medical equipment. On the other hand, the display device 5103A ~ 5103D, recorder 5105, the patient bed 5183 and illumination 5191 is separate from the endoscopic surgical system 5113 is a device which for example are installed in the operating room. Each device does not belong to these endoscopic surgical system 5113 is referred to as non-medical devices. Audiovisual controller 5107 and / or operating room controller 5109 controls in coordination operations of these medical devices and non-medical devices with one another.
[0152]
　Audiovisual controller 5107, a process related to image display in the medical devices and non-medical devices, and controls overall. Specifically, of the device included in the operating room system 5100, device group 5101, ceiling camera 5187 and Subejo camera 5189, the information to be displayed during surgery having a function for transmitting a (hereinafter, also referred to as display information) apparatus (hereinafter, also originating device referred to) may be. The display device 5103A ~ 5103D may be a device in which the display information is output (hereinafter, also referred to as output destination device). Also, the recorder 5105 may be a device corresponding to both the source device and destination device. Audiovisual controller 5107 controls the operation of the source device and destination device, acquires the display information from a source device, and transmits the display information to the output destination device, a function of displaying or recording a. Note that a display information, various images and the captured during surgery, various information related to surgery (e.g., the patient's and body information, past examination results, information, etc. about the surgical procedure), and the like.
[0153]
　Specifically, the audio-visual controller 5107, the device group 5101, as display information, information about the operative portion of the image in the body cavity of a patient picked up by the endoscope can be transmitted. Also, the ceiling camera 5187, as display information, the information about the nearby image of the operator captured by the ceiling camera 5187 may be transmitted. Also, the operative field camera 5189, as display information, information about the image showing the state of the entire operating room is captured by the surgery field camera 5189 may be transmitted. If another device having an imaging function in the operating room system 5100 exists, audiovisual controller 5107, as display information, obtain information about the image captured by the other device from the other device it may be.
[0154]
　Alternatively, for example, the recorder 5105, the information about these images captured in the past are recorded by audiovisual controller 5107. Audiovisual controller 5107 may be a display information, to obtain information about an image captured from the recorder 5105 to the past. Note that the recorder 5105, various information regarding the surgery may be recorded in advance.
[0155]
　Audiovisual controller 5107, at least one of the display devices 5103A ~ 5103D is the output destination device, the acquired display information (i.e., or have been taken during the surgical image, various information regarding surgery) is displayed. In the illustrated example, the display device 5103A is a display device which is installed suspended from the ceiling of the operating room, the display device 5103B is a display device installed on the wall of the operating room, the display device 5103C is in the operating room a display device which is installed on a desk, the display device 5103D is a mobile device having a display function (e.g., a tablet PC (Personal Computer)).
[0156]
　Further, although not shown in FIG. 22, the operating room system 5100 may include an external device in the operating room. An external device in the operating room, for example, a server or connected to the network built in the hospital and out, medical staff using PC, may be a projector or the like installed in a hospital conference room. If such an external device is outside the hospital, audiovisual controller 5107 may be for telemedicine, via a video conference system or the like, it is also possible to display information on a display device of another hospital.
[0157]
　Operating room control unit 5109, the processing other than an image display in the non-medical devices, and controls overall. For example, an operating room control system 5109, the patient bed 5183, ceiling camera 5187, and controls the driving of the operative field camera 5189 and illumination 5191.
[0158]
　The operating room system 5100 is provided with a centralized operation panel 5111, the user, via the centralized operation panel 5111, or give instructions for image display to the audiovisual controller 5107, the operating room control system 5109 it can give an indication of the operation of the non-medical equipment for. Centralized operation panel 5111 is constituted by a touch panel provided on the display surface of the display device.
[0159]
　Figure 23 is a view showing a display example of operation screen in the intensive operation panel 5111. In Figure 23, as an example, the operating room system 5100, as the output destination device, two display devices indicates an operation screen corresponding to the case is provided. Referring to FIG. 23, the operation screen 5193, a source selection area 5195, a preview area 5197, the control area 5201, are provided.
[0160]
　The source selection region 5195 includes a source device provided in the operating room system 5100, and a thumbnail screen showing the display information in which the source device has, but is displayed tied. The user can select the display information to be displayed on the display device, from one of the source device that is displayed in the source selection region 5195.
[0161]
　The preview region 5197, a preview of the screen displayed on two display devices, which is the output destination device (Monitor1, monitor2) is displayed. In the illustrated example, four images in one display apparatus is displayed PinP. The four images are those corresponding to the originating display information from the source device selected in the source selection region 5195. Of the four images, one is relatively large display as the main image, the remaining three are displayed relatively small as a sub image. The user, by four images are appropriately selected region displayed, it is possible to replace the main image and sub-image. Further, at the lower part of the region where four images are displayed, is provided with a status display area 5199, status of operation in the area (e.g., elapsed time and surgery, the patient's body information, etc.) are displayed appropriately obtain.
[0162]
　The control area 5201, GUI parts for performing the originating operation area 5203, an operation to the output destination device GUI (Graphical User Interface) components for performing operations on the source of the device are displayed There and destination operation area 5205 to be displayed, is provided. In the illustrated example, the originating operation area 5203, various operation (pan, tilt and zoom) of GUI components for performing is provided for the camera in the originating device having an imaging function. The user selects these GUI components can be appropriately manipulating the operation of the camera in the originating device. Incidentally, although not shown, if the originating device is selected in the source selection region 5195 is recorder (i.e., in the preview area 5197, an image recorded in the past to the recorder is displayed the case), the originating operation region 5203, the reproduction of the image, playback stop, rewind, it may GUI components provided for performing operations such as fast forward.
[0163]
　Further, the output destination operation area 5205, various operation on the display in the output is the destination of the device display a GUI component for performing (swap, flip, color adjustment, contrast adjustment, switching between 2D display and 3D display) It is provided. The user selects these GUI components can be appropriately manipulating the display on the display device.
[0164]
　Note that the operation screen displayed on the centralized operation panel 5111 is not limited to the illustrated example, the user, via a centralized operation panel 5111, provided in the operating room system 5100, the audiovisual controller 5107 and operating room control system 5109 operation input to each of the devices can be controlled may be possible.
[0165]
　Figure 24 is a diagram showing an example of a state of operation in which the operating room system described above is applied. Ceiling camera 5187 and Subejo camera 5189 is provided on the ceiling of the operating room, the surgeon performing the procedure for the affected part of the patient 5185 on the patient bed 5183 (doctor) 5181 of the hand and the operating room can capture the entire state it is. The ceiling camera 5187 and Subejo camera 5189, magnification adjustment function, a focal length adjusting function may photographing direction adjusting function or the like is provided. Lighting 5191 is provided in the ceiling of the operating room, irradiating the hand of at least the operator 5181. Lighting 5191, the irradiation light amount, the irradiation direction and the like of the wavelength of the irradiated light (color) and light may be appropriately adjusted.
[0166]
　Endoscopic surgery system 5113, the patient bed 5183, ceiling camera 5187, operative field camera 5189 and the lighting 5191, as shown in FIG. 22, through the audiovisual controller 5107 and operating room control system 5109 (FIG. 24 not shown) It is linkage connected to each other Te. Surgery The chamber is provided with centralized control panel 5111, as described above, the user, via the centralized operation panel 5111, it is possible to operate these devices suitably present in the operating room.
[0167]
　Hereinafter, a detailed description of the construction of an endoscopic surgical system 5113. As shown, the endoscopic surgical system 5113 includes an endoscope 5115, other and surgical instrument 5131, the support arm device 5141 for supporting the endoscope 5115, various devices for endoscopic surgery but it consists of, the cart 5151, which is mounted.
[0168]
　In endoscopic surgery, instead of abdominal cut the abdominal wall, cylindrical apertures instrument called trocar 5139a ~ 5139d are multiple punctures the abdominal wall. Then, the trocar 5139a ~ 5139d, the barrel 5117 and the endoscope 5115, and other surgical tool 5131 is inserted into the body cavity of the patient 5185. In the illustrated example, as another surgical instrument 5131, pneumoperitoneum tube 5133, the energy treatment instrument 5135 and forceps 5137 is inserted into the body cavity of the patient 5185. The energy treatment instrument 5135 is a high-frequency current and ultrasonic vibration, tissue incision and exfoliation, or treatment instrument for performing sealing and the like of the vessel. However, the surgical instrument 5131 illustrated is only an example, as the surgical instrument 5131, for example 攝子, retractor, etc., various surgical instrument may be used in used in general endoscopic surgery.
[0169]
　Operative part of the image of the endoscope 5115 in a body cavity of a patient 5185 taken by is displayed on the display device 5155. Operator 5181 while watching the image of the operative site which is displayed on the display device 5155 in real time, using an energy treatment instrument 5135 and forceps 5137, for example, performs treatment such ablate diseased part. Incidentally, although not shown, pneumoperitoneum tube 5133, the energy treatment instrument 5135 and forceps 5137, during surgery, is supported by the operator 5181 or the assistant or the like.
[0170]
　(Support arm unit)
　supporting arm apparatus 5141 includes an arm portion 5145 extending from the base portion 5143. In the illustrated example, the arm portion 5145 is a joint portion 5147a, 5147b, 5147c, and links 5149A, are composed of 5149B, is driven under the control of the arm control device 5159. The endoscope 5115 is supported by the arm portion 5145, its position and orientation are controlled. Thereby, the fixation of the stable positions of the endoscope 5115 can be realized.
[0171]
　(Endoscope)
　endoscope 5115 includes a barrel 5117 in which a predetermined length of the region from the tip is inserted into a body cavity of a patient 5185, a camera head 5119 is connected to the proximal end of the lens barrel 5117, a constructed. In the illustrated example, it is illustrated the endoscope 5115 configured as a so-called rigid endoscope having a lens barrel 5117 of the rigid endoscope 5115 is configured as a so-called flexible scope having a lens barrel 5117 of the flexible it may be.
[0172]
　At the tip of the lens barrel 5117, an opening portion is provided with the objective lens is fitted. The endoscope 5115 and the light source device 5157 is connected, the light generated by the light source device 5157 is guided by a light guide is extended into the barrel 5117 to the tip of the barrel, the objective through the lens and is irradiated toward the observation target in the body cavity of the patient 5185. Incidentally, the endoscope 5115 may be a direct view mirror may be oblique mirror or side endoscope.
[0173]
　Inside the camera head 5119 has an optical system and an imaging device is provided, the reflected light from the observation target (observation light) is condensed to the imaging device by the optical system. Observation light by the imaging device is photoelectrically converted, an electric signal corresponding to the observation light, i.e., the image signal corresponding to the observation image is generated. The image signal, as the RAW data camera control unit: sent to (CCU Camera Control Unit) 5153. Note that the camera head 5119, by driving the optical system properly, the ability to adjust the magnification and focal length are mounted.
[0174]
　Incidentally, for example, to accommodate stereoscopic (3D display) or the like, the camera head 5119 may be an imaging device is provided with a plurality. In this case, inside the barrel 5117, in order to guide the observation light for each of the plurality of imaging devices, the relay optical system are provided a plurality of systems.
[0175]
　(Various devices mounted on the
　cart) CCU5153 is constituted by a CPU (Central Processing Unit) and GPU (Graphics Processing Unit) or the like, and performs overall control of the operation of the endoscope 5115 and a display device 5155. Specifically, CCU5153 subjects the image signal received from the camera head 5119, for example, such as development processing (demosaicing processing), various image processing for displaying an image based on the image signal subjected. CCU5153 provides an image signal subjected to the image processing to the display device 5155. Further, the CCU5153, is connected to audio-visual controller 5107 shown in FIG. 22. CCU5153 also provides an image signal subjected to image processing in audiovisual controller 5107. Further, CCU5153 transmits a control signal to the camera head 5119, and controls the drive. The said control signal, the magnification and focal length, may include information about the imaging conditions. Information on the imaging conditions may be input via an input device 5161, it may be input via a centralized operation panel 5111 as described above.
[0176]
　Display device 5155 is under the control of the CCU5153, displays an image based on the image signal subjected to image processing by the CCU5153. The endoscope 5115 is, for example, 4K (several horizontal pixels 3840 × vertical pixels 2160) or 8K if those corresponding to the high resolution imaging of such (horizontal pixels 7680 × vertical number of pixels 4320), and / or 3D display in the case in which the corresponding, as the display device 5155, in response to each, capable of displaying high-resolution, and / or what 3D capable display may be used. If those corresponding to the high resolution imaging of such 4K or 8K, obtained a more immersive feeling by using of a size over 55 inches as the display device 5155. Also, depending on the application, the resolution may be is provided a plurality of different display devices 5155 size.
[0177]
　Light source apparatus 5157 is composed of, for example, from a light source such as LED (light emitting diode), and supplies the endoscope 5115 the irradiation light when capturing the operative site.
[0178]
　Arm control device 5159 is configured by, for example, a processor such as a CPU operates in accordance with a predetermined program, and controls the driving of the arm portion 5145 of the support arm device 5141 according to a predetermined control strategy.
[0179]
　Input device 5161 is an input interface for endoscopic surgery system 5113. The user, via the input device 5161 can input and instruction input various information to the endoscopic operation system 5113. For example, the user, via the input device 5161, inputs patient and physical information, such as information about the operative procedure of the surgery, a variety of information about the operation. Further, for example, a user, via the input device 5161, an instruction and the effect for driving the arm section 5145, (type of illumination light, magnification and focal length) endoscopes 5115 by the imaging condition instruction to change the inputs an instruction such as the effect of driving the energy treatment instrument 5135.
[0180]
　Type of input device 5161 is not limited, the input device 5161 may be a variety of known input devices. The input device 5161, e.g., a mouse, a keyboard, a touch panel, switch, footswitch 5171 and / or lever or the like may be applied. If the touch panel is used as the input device 5161 is, the touch panel may be provided on the display surface of the display device 5155.
[0181]
　Alternatively, the input device 5161 is, for example, spectacle-type wearable device or HMD (Head Mounted Display) such as a device that is worn by a user, various inputs depending on the gestures and the line of sight of the user detected by these devices It is carried out. The input device 5161 comprises a detectable camera movement of the user, various input is performed according to a gesture and line of sight of the user detected from the image captured by the camera. Furthermore, the input device 5161 includes a sound pickup capable microphone the user's voice, various input is performed by a voice through the microphone. Thus, it is an input device 5161 by the input configured to be capable of various kinds of information in a non-contact, in particular belongs to the clean area user (e.g. operator 5181), operating in a non-contact devices belonging to unclean area it is possible. The user, since it is possible to operate the equipment without removing the hand from the surgical instrument in possession, the user convenience is improved.
[0182]
　Surgical instrument control unit 5163 includes, tissue ablation, to control the driving of the energy treatment instrument 5135 for such sealing of the incision or blood vessel. Insufflator 5165 is the endoscope 5115 according to the purpose of ensuring secure and surgeon's working space field, a body cavity of a patient 5185 for Fukurameru, gas into the body cavity through insufflation tube 5133 the fed. Recorder 5167 is capable of recording apparatus various types of information about the operation. The printer 5169 has a variety of information about the operation, a printing apparatus capable text, image or graph or the like various forms.
[0183]
　Hereinafter, the particular characteristic configuration in endoscopic surgery system 5113 is described in further detail.
[0184]
　(Support arm unit)
　supporting arm unit 5141 includes a base portion 5143 is a base, an arm portion 5145 extending from the base portion 5143, the. In the illustrated example, the arm portion 5145 has a plurality of joint portions 5147a, 5147b, and 5147C, a plurality of links 5149a connected by joints 5147B, and 5149B, it is composed of, in FIG. 24, for simplicity illustrates a simplified configuration of the arm portion 5145. In fact, as the arm portion 5145 has the desired flexibility, the joint portion 5147a ~ 5147c and links 5149A, the shape of the 5149B, number and arrangement, as well as the direction and the like of the rotation axis of the joint 5147a ~ 5147c are set appropriately obtain. For example, the arm portion 5145 may suitably be configured to have six degrees of freedom or more degrees of freedom. Thus, it becomes possible to freely move the endoscope 5115 in a movable range of the arm portion 5145, the insertion of the barrel 5117 of the endoscope 5115 into a body cavity of a patient 5185 from a desired direction possible to become.
[0185]
　The joints 5147a ~ 5147c and actuator are provided, the joint portion 5147a ~ 5147c is rotatably constituted by the driving of the actuator around a predetermined rotational axis. By driving of the actuator is controlled by the arm control device 5159, the rotational angle of each joint portion 5147a ~ 5147c are controlled, the driving of the arm portion 5145 is controlled. Thus, control of the position and orientation of the endoscope 5115 can be realized. At this time, the arm control device 5159 may be force control or position control, etc., by various known control method for controlling the driving of the arm portion 5145.
[0186]
　For example, the surgeon 5181 is, by performing an appropriate operation input through the input device 5161 (including the foot switch 5171), the driving of the arm unit 5145 by the arm control device 5159 in response to the operation input is properly controlled, the inner position and orientation of the endoscope 5115 may be controlled. By the control, after moving the endoscope 5115 of the tip of the arm portion 5145 to an arbitrary position from an arbitrary position can be fixedly supported at the position after the movement. Incidentally, the arm portion 5145 may be operated in a so-called master slave system. In this case, the arm portion 5145 may be remotely operated by the user via the input device 5161 which is located away from the operating room.
[0187]
　Further, when the force control is applied, the arm control device 5159, as an external force from the user, the arm section 5145 is moved smoothly in conformity to the external force, the actuator of each joint 5147a ~ 5147c be driven, it may be carried out a so-called power assist control. This makes it possible when the user moves the arm 5145 while touching the arm portion 5145 directly, moving the arm section 5145 with a relatively small force. Therefore, more intuitive, it is possible to move the endoscope 5115 in a more simple operation, thereby improving the convenience for the user.
[0188]
　Here, generally, in the endoscopic surgery, endoscopic 5115 was supported by the physician called Sukopisuto. In contrast, by using the support arm device 5141, since it becomes possible to more reliably fix the position of the endoscope 5115 without hands, it is possible to obtain an image of the operative site stably , it becomes possible to ensure a smooth surgery.
[0189]
　Incidentally, the arm control device 5159 may not necessarily be provided on the cart 5151. The arm control unit 5159 may not necessarily be a single device. For example, the arm control device 5159 may be respectively provided in the respective joint portions 5147a ~ 5147c of the arm portion 5145 of the support arm device 5141, by a plurality of arm control device 5159 is cooperate driving arm portion 5145 control may be realized.
[0190]
　(Light source device)
　light source device 5157 supplies the illumination light when capturing the operative portion to the endoscope 5115. Light source device 5157, for example LED, composed of a white light source constituted by a laser light source or combination thereof. In this case, if the white light source is constituted by the combination of the RGB laser light source, it is possible to control the output intensity and output timing of each color (each wavelength) with high accuracy, the white balance of the captured image in the light source device 5157 it is possible to perform the adjustment. Further, in this case, by irradiating the observation target by time division the laser light from each RGB laser light source, by controlling the driving of the image pickup device of the camera head 5119 in synchronization with the irradiation timing, corresponding to each RGB it is also possible to image in a time division images. According to the method, without providing a color filter on the imaging device, it is possible to obtain a color image.
[0191]
　The light source device 5157, the drive may be controlled so as to change the intensity of the output light at predetermined time intervals. Acquiring an image at the time of controlling the driving of the image pickup device of the camera head 5119 divided in synchronization with the timing of the change of the intensity of the light, by synthesizing the image, a high dynamic no so-called underexposure and overexposure it is possible to generate an image of the range.
[0192]
　The light source apparatus 5157 may be configured to be capable of supplying light of a predetermined wavelength band corresponding to the observation special light. The special light observation, for example, by utilizing the wavelength dependency of the absorption of light in body tissue, the irradiation light in normal observation (i.e., white light) by irradiation with light of a narrow band as compared to the mucosal surface shooting given tissue, such as blood vessels with high contrast, so-called narrow-band light observation (narrow band imaging) is performed. Alternatively, the special light observation, fluorescence fluorescence observation may be performed to obtain images with generated by irradiating the excitation light. The fluorescence observation, irradiated with excitation light to body tissue intended to observe the fluorescence from the body tissue (autofluorescence), or reagents such as indocyanine green (ICG) with dispenses station body tissue to the body tissue As the excitation light corresponding to the fluorescence wavelength of the reagents irradiated, etc. to obtain a fluorescent image can be made. Light source device 5157, such may be configured to provide a narrow-band light and / or the excitation light corresponding to the special light observation.
[0193]
　(Camera head and CCU)
　with reference to FIG. 25, described in more detail features of the camera head 5119 and CCU5153 endoscope 5115. Figure 25 is a block diagram showing an example of a functional configuration of the camera head 5119 and CCU5153 shown in FIG.
[0194]
　Referring to FIG. 25, the camera head 5119 has as its functions, a lens unit 5121, an imaging unit 5123, a drive unit 5125, a communication unit 5127, a camera head control unit 5129, the. Further, CCU5153 includes, as its functions, a communication unit 5173, an image processing unit 5175, a control unit 5177, the. Camera head 5119 and CCU5153 is communicatively connected bidirectionally via a transmission cable 5179.
[0195]
　First, a description is given of a functional configuration of the camera head 5119. The lens unit 5121 is an optical system provided in the connecting portion of the barrel 5117. Observation light taken from the tip of the lens barrel 5117 is guided to the camera head 5119, and is incident on the lens unit 5121. The lens unit 5121 is constituted by a plurality of lenses are combined including a zoom lens and a focus lens. The lens unit 5121, the observation light on the light receiving surface of the imaging element of the imaging unit 5123 so as to collect light, its optical characteristics are adjusted. The zoom lens and focus lens, for adjusting magnification and focus of the captured image, configured to be movable position on the optical axis.
[0196]
　Imaging unit 5123 is constituted by the image pickup device, it is arranged downstream of the lens unit 5121. Observation light passing through the lens unit 5121 is focused on the light receiving surface of the imaging element, by photoelectric conversion, an image signal corresponding to the observation image is generated. Image signal generated by the imaging unit 5123 are provided to communications unit 5127.
[0197]
　The imaging device constituting the imaging section 5123, for example, CMOS is an image sensor (Complementary Metal Oxide Semiconductor) type, capable color photography having a Bayer array is used. As the the image sensor, it may be used capable corresponds for example to the photographing of the high-resolution images of more than 4K. By image of the surgical section can be obtained with high resolution, the operator 5181 is able to grasp the state of the surgical section in more detail, it is possible to proceed more smoothly surgery.
[0198]
　The imaging device constituting the imaging unit 5123 is configured to have a pair of imaging elements for each acquired image signals for the right eye and the left eye corresponding to the 3D display. By 3D display is performed, the operator 5181 is enabled to know the depth of the living tissue in the operative site more accurately. The imaging unit 5123 when composed of multi-plate, corresponding to the imaging elements, the lens unit 5121 is also provided a plurality of systems.
[0199]
　The imaging unit 5123 may not necessarily be provided in the camera head 5119. For example, the imaging unit 5123, the interior of the barrel 5117 may be provided immediately after the objective lens.
[0200]
　Drive unit 5125 is constituted by an actuator, under the control of the camera head control unit 5129, the zoom lens and the focus lens of the lens unit 5121 along the optical axis is moved by a predetermined distance. Thus, the magnification and focus of the image captured by the imaging unit 5123 can be appropriately adjusted.
[0201]
　The communication unit 5127 is constituted by a communication device for transmitting and receiving various information to and from the CCU5153. The communication unit 5127 transmits the image signal obtained from the imaging unit 5123 in CCU5153 via a transmission cable 5179 as RAW data. In this case, the captured image of the operation site in order to display with low latency, it is preferable that image signals are transmitted by optical communication. During surgery, because the surgeon 5181 performs an operation while observing the state of the affected area by the captured image, for safer and more reliable operation is displayed in real time as possible moving image of the surgical unit it is because it is required. If the optical communication is performed, the communication unit 5127, a photoelectric conversion module for converting an electric signal into an optical signal is provided. After the image signal is converted into an optical signal by the photoelectric conversion module, it is transmitted to CCU5153 via a transmission cable 5179.
[0202]
　The communication unit 5127 from CCU5153, receives a control signal for controlling the driving of the camera head 5119. The the control signal, for example, information that specifies the frame rate of the captured image, information that specifies the exposure value at the time of imaging, and / or magnification and information, etc. indicating that specifies the focal point of the captured image, captured It contains information about the condition. The communication unit 5127 provides control signals received in the camera head control unit 5129. The control signal from CCU5153 also may be transmitted by optical communication. In this case, the communication unit 5127, a photoelectric conversion module is provided for converting an optical signal into an electrical signal, the control signal is converted into an electric signal by the photoelectric conversion module is provided to the camera head control unit 5129.
[0203]
　The above frame rate and exposure value, magnification, imaging conditions of the focus, etc., are automatically set by the control unit 5177 of CCU5153 based on the acquired image signal. That is, a so-called AE (Auto Exposure) function, AF (Auto Focus) function and AWB (Auto White Balance) function is mounted to the endoscope 5115.
[0204]
　Camera head control unit 5129 based on the control signal from CCU5153 received via the communication unit 5127, and controls the drive of the camera head 5119. For example, the camera head control unit 5129, based on the information that specifies the frame rate of a captured image and / or information that specifies the exposure at the time of imaging, and controls the driving of the image sensor of the imaging unit 5123. Further, for example, the camera head control unit 5129, based on the information that specifies the magnification and focus of the captured image, as appropriate to move to the zoom lens and the focus lens of the lens unit 5121 via the drive unit 5125. Camera head control unit 5129 may further comprise a function of storing information for identifying the lens barrel 5117 and the camera head 5119.
[0205]
　Incidentally, the configuration of such a lens unit 5121 and the imaging unit 5123, by disposing the air tightness and water resistance is high airtight structure, the camera head 5119, can have resistance to autoclave sterilization.
[0206]
　Next, description is made about the functional structure of CCU5153. The communication unit 5173 is constituted by a communication device for transmitting and receiving various information between the camera head 5119. The communication unit 5173, from the camera head 5119 receives image signals transmitted via a transmission cable 5179. At this time, as described above, the image signal can be transmitted suitably by optical communication. In this case, in response to the optical communication, the communication unit 5173 is a photoelectric conversion module for converting an optical signal into an electrical signal is provided. The communication unit 5173 provides an image signal converted into an electric signal to the image processing unit 5175.
[0207]
　The communication unit 5173, to the camera head 5119, and transmits a control signal for controlling the driving of the camera head 5119. It may be transmitted by the control signal also optical communication.
[0208]
　The image processing unit 5175 applies various kinds of image processing of the image signal is a RAW data transmitted from the camera head 5119. As the image processing, for example the development processing, high-quality image processing (band enhancement processing, the super-resolution processing, NR (Noise reduction) processing and / or hand shake correction processing, etc.), and / or expansion processing (electronic zoom processing) etc., includes various known signal processing. The image processing unit 5175 is, AE, for performing AF and AWB, the detection processing on the image signal performed.
[0209]
　The image processing unit 5175 is constituted by a processor such as a CPU or GPU, by which the processor operates in accordance with a predetermined program, the image processing described above and detection processing may be performed. Incidentally, when the image processing unit 5175 is composed of a plurality of the GPU, the image processing unit 5175 is appropriately divides the information related to the image signal, performing parallel image processing by the plurality of GPU.
[0210]
　Control unit 5177, the imaging of the surgical unit according to the endoscope 5115, and the various controls relating to the display of the captured image performed. For example, the control unit 5177 generates a control signal for controlling the driving of the camera head 5119. At this time, when the imaging conditions are entered by the user, the control unit 5177 generates a control signal based on input by the user. Alternatively, AE function to the endoscope 5115, when the AF function and AWB function is mounted, the control unit 5177 in accordance with the result of the detection processing by the image processing unit 5175, the optimum exposure value, focal length and appropriately calculated white balance, and generates a control signal.
[0211]
　The control unit 5177 based on the image signal subjected to image processing by the image processing unit 5175, and displays an image of the operation site to the display device 5155. At this time, the control unit 5177 recognizes the various objects in the operative part image by using various image recognition technology. For example, the control unit 5177 by detecting the edge of the shape and color of an object or the like contained in the operative unit image, the surgical instrument such as forceps, a specific body part, hemorrhage, energy treatment instrument 5135 when using a mist or the like it can be recognized. Control unit 5177 is, when displaying the image of the operation site to the display device 5155, using the recognition result, various operation support information is superimposed on the image of the surgical unit. Operation support information is superimposed, by being presented to the operator 5181, it is possible to proceed more safely and reliably surgery.
[0212]
　Transmission cable 5179 for connecting the camera head 5119 and CCU5153 an electric signal cable for the communication of electrical signals, optical fibers corresponding to optical communication, or these composite cable.
[0213]
　Here, in the illustrated example, although wired communication has been performed using a transmission cable 5179, communication between the camera head 5119 and CCU5153 may be performed wirelessly. When the communication between them is performed by radio, the transmission cable 5179 eliminates the need to lay the operating room to the movement of the medical staff in the operating room can be eliminated situation hindered by the transmission cable 5179.
[0214]
　Has been described above an example of an operating room system 5100 technology according to the present disclosure may be applied. Here, the case has been described where the medical system operating room system 5100 is applied as an example is an endoscope surgical system 5113, configuration of the operating room system 5100 is not limited to such an example. For example, operating room system 5100, instead of the endoscopic operation system 5113, may be applied to flexible endoscope system and microsurgical system for inspection.
[0215]
　The technology according to the present disclosure, of the configuration described above, can be suitably applied to the recorder 5105. As an example, a recorder 5105, either the camera (e.g., ceiling camera 5187, operative field camera 5189 or camera head 5119) when coding an image captured by, according to the techniques of the present disclosure relates to quantization the process parameters may be determined based on at least one of the transfer function and the color gamut for the conversion between the optical and the image signal. Thereby, it is possible to inappropriate quantization step due to the difference in transfer function or color gamut can be prevented from being used, to ensure sufficient allocation amount of codes for representing the gray level of the original signal . As another example, the recorder 5105, when encoding the image captured by either camera, in accordance with the techniques of the present disclosure, the process parameters related to pixel classification process that may be performed in a variety of purposes, light and images signal may be determined based on at least one of the transfer function and the color gamut for the conversion between. Thus, due to the difference in transfer function or gamut prevents pixels with incorrect criteria are classified, it is possible to ensure proper execution of the process based on the classification. Consequently, in each of the examples, the image can be enhanced diagnosis or treatment of accuracy using.
[0216]
　<6. Summary>
　technology according to the present disclosure, in accordance with the mechanism described in detail so far, diversification not be always well adapted to the video signal representation is becoming for example, a digital video camera, a digital camcorder, a video encoder or encoding function or pixel classification It provides improved for any type of existing device having a function. According to the technique of the present disclosure, inappropriate process parameters can be prevented from being used when dealing with HDR video.
[0217]
　In this specification, for simplicity of explanation, is used in some specific terminology to a particular video coding scheme, the technology according to the present disclosure is not limited by such terms. For example, the term luminance (luminance / luma) and color difference (chrominance / chroma), depending on the color system to be utilized, and replaced each other terms such lightness (brightness) and chroma (saturation) it may be.
[0218]
　Having described in detail preferred embodiments of the present disclosure with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such an example. It would be appreciated by those skilled in the art of the present disclosure, within the scope of the technical idea described in the claims, it is intended to cover various modifications, combinations, these for it is also understood to belong to the technical scope of the present disclosure.
[0219]
　The effects described herein are not limiting be those that only illustrative or exemplary. In other words, the technology according to the present disclosure, together with the effect, or in place of the above-mentioned effects can achieve apparent other effects to those skilled in the art from the description herein.
[0220]
　Also within the scope of the present disclosure the following configurations.
(1)
　image based on at least one of the color gamut applied to the transfer function and the image for the conversion between the optical and the image signal applied to, determine the processing parameters for image processing of the image and a control unit which,
　using the processing parameter determined by the control unit, and a processing unit for executing said image processing
　image processing apparatus comprising a.
(2)
　the control unit, on the basis of the combination of the transfer function and the color gamut, determining the processing parameter, the image processing apparatus according to (1).
(3)
　the control unit, so that the change in the code value of the transfer function and the image signal due to at least one of the differences of said color gamut counteract the effect on the image processing, determining the processing parameter to the (1) or the image processing apparatus according to (2).
(4)
　wherein the processing unit, a quantization step to be adjusted in accordance with the coding difficulty of each partial area of the image for each partial area running quantization processing,
　the processing parameter relates to the quantization process it is a parameter,
　wherein (1) the image processing apparatus according to any one of - (3).
(5)
　parameters relating to the quantization processing includes adjusting gain of the quantization step for said image, the image processing apparatus according to (4).
(6)
　the control unit, the transfer function and based on said at least one of said color gamut, determine the basic adjustment gain of the quantization step for said image, the image processing according to (5) apparatus.
(7)
　wherein the processing unit, a quantization step after adjustment is adjusted based on the coding difficulty of the basic adjustment gain and the partial regions, to perform the quantization processing for each partial region, the ( the image processing apparatus according to 6).
(8)
　wherein the processing unit executes the pixel classification process for the image using the processing parameters, the (1) or the image processing apparatus according to (2).
(9)
　the process parameter comprises a threshold to be compared with the sign value of the color component in the pixel classification process, the image processing apparatus according to (8).
(10)
　the pixel classification process includes area detection process for detecting a specific area in the image,
　the threshold value includes a region detection threshold,
　the image processing apparatus according to (9).
(11)
　the specific region is a region of a specific color,
　the area detection threshold comprises a color determination threshold,
　the image processing apparatus according to (10).
(12)
　said particular color is a skin color,
　the color determination threshold includes skin color determination threshold,
　the image processing apparatus according to (11).
(13)
　The image processing apparatus, a quantization unit to perform quantization processing for the image in the quantization step is adjusted based on the result of the area detection process, further comprising an image according to (12) processing apparatus.
(14)
　The image processing apparatus, one or a storage unit for storing the values of the process parameters associated with both of the transfer function and the color gamut, further comprising, either above (1) to (13) whether the image processing apparatus according to (1).
(15)
　the control unit, the determining the at least one type of said transfer function and the color gamut on the basis of the transfer function and the at least one related input information of the color gamut, the determined the type wherein determining the processing parameter for the image processing, wherein (1) the image processing apparatus according to any one of - (14) based on.
(16)
　wherein the input information is information obtained through the user interface, the image processing apparatus according to (15).
(17)
　the input information, the image is information obtained from the input image signal and the auxiliary signals to be multiplexed to represent an image processing apparatus according to (15).
(18)
　an image based on at least one of the color gamut applied to the transfer function and the image for the conversion between the optical and the image signal applied to, determine the processing parameters for image processing of the image it and, for
　using the process parameters determined, and performing the image processing
　image processing method comprising.
(19)
　the processor of the image processing apparatus,
　based on at least one of the color gamut applied to the transfer function and the image for the conversion between the optical and the image signal applied to an image, the image processing of the image a control unit for determining a processing parameter for,
　using the processing parameter determined by the control unit, and a processing unit for executing the image processing
　program for functioning as a.
DESCRIPTION OF SYMBOLS
[0221]
　100a, 100b the image processing apparatus
　101 the signal acquisition unit
　102 signal processing unit
　103 information acquisition unit
　107 storage unit
　110 coding unit
　115 quantization unit (processing
　unit) 140 control unit
　200a, 200b the image processing apparatus
　201 the signal acquiring unit
　203 information acquisition unit
　210 coding unit
　240 control unit
　241,250 threshold controller
　243,260 pixel classification unit (processing
　unit) 270 output signal generation unit

The scope of the claims
[Requested item 1]
　Based on at least one of the color gamut applied to the transfer function and the image for the conversion between the optical and the image signal applied to the image, the control unit for determining a processing parameter for image processing of the image When,
　using the process parameters as determined by the control unit, and a processing unit for executing said image processing
　image processing apparatus comprising a.
[Requested item 2]
　The control unit, on the basis of the combination of the transfer function and the color gamut, determining the processing parameter, the image processing apparatus according to claim 1.
[Requested item 3]
　Wherein the control unit is configured such that the change in the code value of the transfer function and the image signal due to at least one of the differences of said color gamut counteract the effect on the image processing, to determine the processing parameter, wherein the image processing apparatus according to claim 1.
[Requested item 4]
　Wherein the processing unit is configured to perform a quantization step to be adjusted in accordance with the coding difficulty of each partial region of the image for each partial area quantization processing,
　the processing parameter is a parameter related to the quantization process ,
　the image processing apparatus according to claim 1.
[Requested item 5]
　Parameters related to the quantization processing includes adjusting gain of the quantization step for said image, the image processing apparatus according to claim 4.
[Requested item 6]
　Wherein the control unit, the transfer function and based on said at least one of said color gamut, determine the basic adjustment gain of the quantization step for said image, the image processing apparatus according to claim 5.
[Requested item 7]
　Wherein the processing unit, a quantization step after adjustment is adjusted based on the coding difficulty of the basic adjustment gain and the partial regions, to perform the quantization processing for each partial region, according to claim 6 the image processing apparatus.
[Requested item 8]
　Wherein the processing unit executes the pixel classification process for the image using the processing parameters, the image processing apparatus according to claim 1.
[Requested item 9]
　The processing parameters, the comprises a threshold to be compared with the sign value of the color component in the pixel classification process, the image processing apparatus according to claim 8.
[Requested item 10]
　The pixel classification process includes area detection process for detecting a specific area in the image,
　the threshold value includes a region detection threshold,
　the image processing apparatus according to claim 9.
[Requested item 11]
　The specific region is a region of a specific color,
　the area detection threshold comprises a color determination threshold,
　the image processing apparatus according to claim 10.
[Requested item 12]
　The specific color is skin color,
　the color determination threshold includes skin color determination threshold,
　the image processing apparatus according to claim 11.
[Requested item 13]
　The image processing apparatus further includes a quantization unit, for performing a quantization process for the image in the quantization step is adjusted based on the result of the area detection process, an image processing apparatus according to claim 12.
[Requested item 14]
　The image processing apparatus, one or a storage unit for storing the values ​​of the process parameters associated with both of the transfer function and the color gamut, further comprising an image processing apparatus according to claim 1.
[Requested item 15]
　Wherein the control unit, the transfer function and determining the at least one type of said transfer function and the color gamut based on the at least one related input information of the color gamut, based on the determined the type determining the processing parameter for the image processing, the image processing apparatus according to claim 1.
[Requested item 16]
　The input information is information obtained through the user interface, the image processing apparatus according to claim 15.
[Requested item 17]
　The input information, the image is information obtained from the input image signal and the auxiliary signals to be multiplexed to represent an image processing apparatus according to claim 15.
[Requested item 18]
　Image based on at least one of the color gamut applied to the transfer function and the image for the conversion between the optical and the image signal applied to, and determining the processing parameter for the image processing of the image ,
　using the process parameters determined, and performing the image processing
　image processing method comprising.
[Requested item 19]
　The processor of the image processing apparatus,
　an image based on at least one of the color gamut applied to the transfer function and the image for the conversion between the optical and the image signal applied to, for image processing of the image a control unit for determining a processing parameter,
　using the processing parameter determined by the control unit, and a processing unit for executing the image processing
　program for functioning as a.