Title of the invention: An image processing apparatus, an image processing method, and a program.
Technical field
[0001]
　The present disclosure relates to an image processing apparatus, an image processing method, and a program. More specifically, the present invention relates to an image processing apparatus for correcting uneven brightness of an image and generating a high-quality image, an image processing method, and a program.
Background technology
[0002]
　In recent years, the use of drones, which are small aircraft that fly by remote control or autonomously fly based on GPS or the like, has been rapidly increasing.
　For example, it is used for processing such as attaching a camera to a drone and taking a picture of a landscape on the ground from the sky.
　Recently, aerial images using drones have been used for crop growth confirmation and management on agricultural land, topographical confirmation processing, surveying processing, and construction sites.
[0003]
　The drone continuously captures images of a predetermined area while moving. By stitching together a plurality of captured images, it is possible to generate one image in a wide area.
　However, drones that move in the air change their posture when shooting each image due to the influence of wind and the like. As a result, the input conditions of the light from the subject to the camera change, and the brightness and color of each captured image are not uniform and become sparse. When images having different brightness and colors are joined together to generate one composite image, a low-quality image in which the continuity of the joint of each image is lost is obtained.
　Further, even in a single photographed image, for example, in an environment where sparse clouds exist, the brightness and color of the shaded portion of the cloud and the non-shadowed portion are different.
[0004]
　As described above, in the stitching (mosaic king) processing that creates a composite image (panoramic image) by joining multiple images shot by changing the shooting position and shooting direction, the lighting state of each shot image changes. As a result, there arises a problem that the brightness value of the pixels changes discontinuously in the region corresponding to the joint of the composite images.
[0005]
　For example, the following documents disclose techniques for solving this problem.
　Non-patent Document 1 ( "shading correction and color correction of an endoscope image" Masaharu Ando al, Japan Gastroenterological endoscopy Journal, Vol. 31, No. 7, p.1727-1741,1989.)
　In this document, A signal value correction method using a smoothing filter is disclosed.
　Specifically, using a smoothed image of a captured image or a reference image, the signal value of each pixel of the captured image is divided by the signal value of each pixel value of the smoothed image to obtain uneven brightness of the captured image. The method of correction is disclosed.
[0006]
　However, the correction method described in this document has a problem that an error such as a ringing artifact occurs in the corrected image because the brightness change information of the subject other than the correction target included in the image also affects the corrected image. is there.
[0007]
　Further, Patent Document 1 (Japanese Unexamined Patent Publication No. 2004-139219) discloses a method for correcting a signal value based on known geometric conditions.
　This document calculates correction functions such as quadratic functions for image correction using the lighting of the camera and shooting environment and the geometric conditions of the shooting target, and uses the calculated correction functions to determine the brightness of the image, etc. A method for adjusting the amount of correction is disclosed.
　That is, by fitting the brightness change predicted in advance from the shooting environment with the correction function and performing the brightness correction of the shot image, the brightness change due to the illumination unevenness in the shot image is corrected.
[0008]
　However, in this method, it is possible to obtain an accurate correction function from geometric conditions, such as when it is difficult to specify the coordinates and orientation of the lighting and camera, or when the surface to be photographed is not a flat surface but a curved surface with irregularities. When it is difficult, there is a problem that high-precision correction cannot be performed.
Prior art literature
Patent documents
[0009]
Patent Document 1: Japanese Unexamined Patent Publication No. 2004-139219
Non-patent literature
[0010]
Non-Patent Document 1: "Shading Correction and Color Correction of Endoscopic Images" by Masaharu Ando, ​​Journal of Japan Gastroenterological Endoscopy Society, Vol. 31, No. 7, p. 1727-1741, 1989.
Outline of the invention
Problems to be solved by the invention
[0011]
　The present disclosure has been made in view of the above problems, for example, and when a composite image is generated by joining images having different shooting conditions, differences in output pixel values ​​such as brightness between the images are obtained. It is an object of the present invention to provide an image processing apparatus, an image processing method, and a program capable of generating a high-quality composite image by reducing the number of images.
[0012]
　For example, when performing stitching processing to create a single image by stitching images taken from a moving object such as a drone outdoors, the irradiation light conditions such as sunlight and lighting, the camera posture, the uneven state of the subject surface, etc. Even in an environment where it is difficult to specify geometric conditions, it is possible to generate a high-quality composite image in which the joints between images are inconspicuous by suppressing ringing artifacts that cause discontinuity in the joints of images by appropriate pixel value correction. And.
[0013]
　Further, the present disclosure is an image processing device capable of generating a high-quality single image by reducing this difference even when there is a difference in output pixel value such as uneven brightness in one captured image. An object of the present invention is to provide an image processing method and a program.
Means to solve problems
[0014]
　The first aspect of the present disclosure,
　the sampling point selection section for selecting a sampling point to be used for calculation of the correction function applied to the pixel value correction of the image from the image,
　the pixels of the selected sampling point of the sampling point selection section a correction function calculator for calculating the correction function by applying the value as position information,
　by applying the correction function includes an image correction unit that performs pixel value correction of the image,
　the sampling point selection section,
　wherein An image that executes clustering that divides the constituent pixels of the image into a plurality of subsets,
　determines a sampling point extraction cluster from the plurality of clusters generated by the clustering,
　and executes a process of selecting a sampling point from the sampling point extraction cluster. It is in the processing device.
[0015]
　Further, the second aspect of the present disclosure is
　an image processing method executed in the image processing apparatus, and the
　sampling points used by the sampling point selection unit to calculate the correction function applied to the pixel value correction of the image are obtained from the image. The sampling point selection step to be selected, the
　correction function calculation step in which the correction function calculation unit calculates the correction function by applying the pixel value and the position information of the sampling point selected by the sampling point selection unit, and the
　image correction unit An image correction step of applying the correction function to correct the pixel value of the image is executed, and the
　sampling point selection step includes
　a step of executing clustering for dividing the constituent pixels of the image into a plurality of subsets.
　The
　image processing method includes a step of determining a sampling point extraction cluster from a plurality of clusters generated by the clustering and a step of selecting a sampling point from the sampling point extraction cluster.
[0016]
　Further, the third aspect of the present disclosure is
　a program for executing image processing in an image processing apparatus, and the
　sampling point selection unit is provided with a sampling point used for calculating a correction function applied to pixel value correction of the image. A sampling point selection step to be selected from, a
　correction function calculation step to apply the pixel value and position information of the sampling point selected by the sampling point selection unit to the correction function calculation unit to calculate the correction function, and an
　image correction unit. To execute an image correction step of applying the correction function to correct the pixel value of the image, and in the
　sampling point selection step,
　perform clustering that divides the constituent pixels of the image into a plurality of subsets. The 　program includes a
　step, a step of determining a sampling point extraction cluster from a plurality of clusters generated by the clustering, and a step
of selecting a sampling point from the sampling point extraction cluster.
[0017]
　The program of the present disclosure is, for example, a program that can be provided by a storage medium or a communication medium that is provided in a computer-readable format to an information processing device or a computer system that can execute various program codes. By providing such a program in a computer-readable format, processing according to the program can be realized on an information processing device or a computer system.
[0018]
　Still other objectives, features and advantages of the present disclosure will become apparent by more detailed description based on the examples of the present disclosure and the accompanying drawings described below. In the present specification, the system is a logical set configuration of a plurality of devices, and the devices having each configuration are not limited to those in the same housing.
Effect of the invention
[0019]
　According to the configuration of one embodiment of the present disclosure, an image processing device and a method capable of generating a high-quality corrected image with reduced luminance unevenness and color unevenness are realized.
　Specifically, for example, a sampling point selection unit that selects a sampling point to be used for calculating a correction function applied to pixel value correction of an image from an image, and a correction function that applies the pixel value and position information of the sampling point to perform a correction function. It has a correction function calculation unit to calculate and an image correction unit to correct the pixel value of the image by applying the correction function, and the sampling point selection unit is a clustering unit that divides the constituent pixels of the image into a plurality of subsets (clusters). (Cluster division) is executed, the sampling point extraction cluster is determined from the multiple clusters generated by the clustering, and the sampling point is selected from the sampling point extraction clusters.
　With this configuration, an image processing device and method capable of generating a high-quality corrected image with reduced luminance unevenness and color unevenness are realized.
　The effects described in the present specification are merely exemplary and not limited, and may have additional effects.
A brief description of the drawing
[0020]
FIG. 1 is a block diagram showing a configuration example of a first embodiment of the image processing apparatus of the present disclosure.
FIG. 2 is a diagram showing an example in which an image to be corrected is divided into 11 clusters (K = 11) by the K-Means method.
FIG. 3 is a diagram showing a histogram showing an average value of signal values ​​of elements (pixels) belonging to each of the 11 clusters divided by the K-means method shown in FIG. 2 and their frequencies.
FIG. 4 is a diagram showing the results of randomly selecting sampling points from the sampling point extraction clusters selected using the histogram of FIG.
[Fig. 5] Fig. 5 is a diagram illustrating an example in which clustering is performed by applying semantic segmentation in a sampling point selection unit.
[Fig. 6] Fig. 6 is a diagram illustrating an example in which a captured image is divided into a plurality of areas using a boundary set by a user or a boundary defined in advance, and these divided areas are set as a cluster.
FIG. 7 is a diagram illustrating a specific difference between the processing by the image processing apparatus of the present disclosure and the image correction example by the conventional processing.
FIG. 8 is a diagram illustrating processing by the image processing apparatus of the present disclosure.
FIG. 9 is a diagram showing a flowchart illustrating a sequence of image processing executed by the image processing apparatus shown in FIG. 1.
FIG. 10 is a diagram illustrating the configuration and processing of the second embodiment of the image processing apparatus of the present disclosure.
11 is a diagram showing a flowchart illustrating a sequence of image processing executed by the image processing apparatus shown in FIG. 10. FIG.
FIG. 12 is a diagram illustrating the configuration and processing of the third embodiment of the image processing apparatus of the present disclosure.
FIG. 13 is a diagram showing a flowchart illustrating a sequence of image processing executed by the image processing apparatus shown in FIG.
FIG. 14 is a diagram illustrating the configuration and processing of the fourth embodiment of the image processing apparatus of the present disclosure.
FIG. 15 is a diagram illustrating a process to be executed according to a fourth embodiment of the image processing apparatus of the present disclosure.
FIG. 16 is a diagram illustrating a process to be executed according to a fourth embodiment of the image processing apparatus of the present disclosure.
FIG. 17 is a diagram illustrating the configuration and processing of the fifth embodiment of the image processing apparatus of the present disclosure.
FIG. 18 is a diagram illustrating a process to be executed according to a fifth embodiment of the image processing apparatus of the present disclosure.
FIG. 19 is a diagram illustrating the configuration and processing of the sixth embodiment of the image processing apparatus of the present disclosure.
FIG. 20 is a diagram illustrating a process to be executed according to a sixth embodiment of the image processing apparatus of the present disclosure.
FIG. 21 is a diagram illustrating a hardware configuration example of the image processing apparatus of the present disclosure.
Mode for carrying out the invention
[0021]
　Hereinafter, the details of the image processing apparatus, the image processing method, and the program of the present disclosure will be described with reference to the drawings. The explanation will be given according to the following items.
　1. 1.
　2. Regarding the configuration and processing of the first embodiment of the image processing apparatus of the present disclosure . About the image processing sequence executed by the image processing device
　3.
　4. Regarding the configuration and processing of the second embodiment of the image processing apparatus of the present disclosure .
　5. Regarding the configuration and processing of the third embodiment of the image processing apparatus of the present disclosure . About other examples
　6. About the hardware configuration example of the image processing device
　7. Summary of the structure of this disclosure
[0022]
　　[1. Regarding the configuration and processing of the first embodiment of the image processing apparatus of the present disclosure]
　First, the configuration and processing of the first embodiment of the image processing apparatus of the present disclosure will be described with reference to FIGS. 1 and the following.
[0023]
　FIG. 1 is a block diagram showing a configuration example of a first embodiment of the image processing apparatus of the present disclosure. As shown in FIG. 1, the image processing device 100 includes a sampling point selection unit 101, a correction function calculation unit 102, and an image correction unit (correction function application unit) 103.
[0024]
　The image processing device 100 inputs one captured image 10 to be corrected, executes the correction, and outputs the corrected image 20.
　In this embodiment, the photographed image 10 to be corrected is one photographed image taken by one shooting process of the camera. That is, it is not a composite image generated by performing a stitching process for joining a plurality of images.
[0025]
　For example, an image taken by a camera attached to a moving object such as a drone may include a cloud in the sky or an area that is shaded and an area that is not shaded by other flying objects such as an airplane or a bird. Brightness and color are different between such areas. That is, uneven brightness and uneven color occur.
　The image processing device 100 shown in FIG. 1 corrects the output unevenness of the pixel values ​​included in such one captured image 10 to generate a corrected image 20 in which the luminance unevenness, the color unevenness, and the like are reduced.
[0026]
　Each component of the image processing device 100 shown in FIG. 1 can be configured as individual hardware or integrated hardware. Further, the processing executed by each component of the image processing apparatus 100 shown in FIG. 1 can be partially or completely executed by software (program).
[0027]
　In the following description, an example in which the captured image 10 is an RGB color image will be described. However, this is only an example, and the image processing apparatus of the present disclosure performs correction processing not only on RGB images but also on various images such as color images other than RGB such as YCbCr images, monochrome images, and luminance images. It is possible.
[0028]
　The configuration and processing of the image processing apparatus 100 shown in FIG. 1 will be described.
　First, the sampling point selection unit 101 of the image processing device 100 shown in FIG. 1 inputs a captured image 10 to be corrected.
　The captured image 10 is, for example, one image captured by a drone.
[0029]
　The sampling point selection unit 101 selects pixels to be sampling points from the captured image 10 to be corrected.
　The sampling point is a pixel used to generate a correction function in the next correction function calculation unit 102.
　The sampling point selection unit 101 outputs the pixel value (luminance value, etc.) of the sampling point and its position information to the next correction function calculation unit 102.
[0030]
　The correction function calculation unit 102 generates a correction function by using the pixel value and the position information of the sampling point selected by the sampling point selection unit 101.
　In the final stage image correction unit (correction function application unit) 103, the correction function generated by the correction function calculation unit 102 is applied to all the pixels of the captured image 10, and the pixel value correction of all the pixels is executed to correct the image. 20 is generated.
[0031]
　First, the sampling point selection unit 101 executes a clustering (cluster division) process for dividing the constituent pixels of the captured image 10 to be corrected into a plurality of subsets (clusters).
[0032]
　As a method of dividing the constituent pixels of the captured image 10 into a plurality of subsets (clusters), for example, the following existing clustering method can be applied.
　(A) K-means method (K-means method),
　(b) K-NN method (K shortest distance method: K-Nearest Neighbor method),
　(c) Ward method (Ward method),
　(d) Semantic segmentation,
[0033]
　The K-means method (K-means method) is a clustering method that follows a non-hierarchical method, and is classified into K subsets (clusters) by classifying them so as to obtain the optimum evaluation value using an evaluation function. It is a method.
　The K-NN method is a clustering method according to a hierarchical method, and is a method of sorting into K classes in order of distance (high degree of similarity) from a certain reference value or standard pattern.
　The Ward method is also a clustering method according to a hierarchical method, and is a method of classifying so that the sum of squares in each cluster is minimized.
[0034]
　Semantic segmentation is the constituent pixels of an image based on the degree of matching between the objects in the image and the dictionary data (trained data) for object identification based on the shape of various actual objects and other feature information. This is a method of identifying what kind of category (people, cars, roads, plants, etc.) each object belongs to and classifying it into clusters for each category.
[0035]
　In addition, there are a plurality of different methods for semantic segmentation, for example, as follows.
　(D1) Conditional Random Fields (CRF ) a method using (P.Krahenbuhl, et.al, "Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials", NIPS'11 Proceedings of the 24th International Conference on Neural Information Processing Systems, 2011.)
[0036]
　(D2) method using a neural network (CNN) convolution (G. Lin, Et.Al, "Efficient Piecewise Training Of Deep Structured Models For Semantic", IEEE Conference On Computer Vision And Pattern Recognition (CVPR),
　2016.) ( d3) Techniques using Generative Adversarial Learning (GAN) (P. Isola, et. Al, "Image-to-Image Transition with Convolutional Neural Networks", EEE Conference Computer Vision
[0037]
　The sampling point selection unit 101 is, for example, the above clustering method, that is,
　(a) K-means method (K-means method),
　(b) K-NN method (K shortest distance method: K-Nearest Neighbor method),
　(c) 　Clustering (cluster division) processing that divides the constituent pixels of the captured image 10 to be corrected into a plurality of subsets (clusters) by applying one of the Ward method (Ward method) and
　(d) semantic segmentation.
To execute.
[0038]
　Next, the sampling point selection unit 101 selects a sampling point, that is, a sampling point (pixel) used for generating a correction function in the next correction function calculation unit 102 from the divided clusters.
[0039]
　The sampling point selection process is executed, for example, in the following procedure.
　A sampling point extraction cluster is determined from a subset (cluster) which is a plurality of pixel groups classified by the above clustering.
　Next, the pixels included in the determined sampling point extraction cluster are selected as sampling points.
　The details of these processes will be described below.
[0040]
　First, the sampling point selection unit 101 determines a sampling point extraction cluster from a subset (cluster) which is a plurality of pixel groups classified by the above clustering.
[0041]
　As a method for determining the sampling point extraction cluster, for example, any of the following methods can be applied.
　(1) One or more clusters are designated as sampling point extraction clusters in order from the cluster having the largest number of samples (elements / pixels).
　(2) A cluster in which the average value or median value of the pixel output (pixel value, brightness value, etc.) of each cluster is the center of all clusters, or a number of clusters adjacent to the cluster is defined as a sampling point extraction cluster.
　(3) The user selects a specific cluster, and the selected cluster is used as a sampling point extraction cluster.
[0042]
　In this example, the average value and the median value of each cluster shown in (2) above are the average value and the median value of the pixel values ​​and the luminance values ​​of the pixels belonging to each cluster.
　By setting the cluster that is the center of the mean or median of each cluster shown in (2) or the number of clusters adjacent to the cluster as the sampling point extraction cluster, for example, an extremely dark pixel area or a bright area in the image. The cluster containing is not set as the sampling point extraction cluster, and the sampling points will not be extracted.
[0043]
　Further, as an example of cluster selection by the user in (3) above, there is an example of selecting a cluster containing many pixels of a specific subject (object) for which a pixel value with reduced luminance unevenness and color unevenness is to be obtained.
[0044]
　Specifically, for example, when the purpose is to analyze the growth status of a crop on a farmland photographed with a drone, a cluster containing many pixels included in the image area of ​​the crop is selected as a sampling point extraction cluster. Is possible.
　If the purpose is to analyze the soil of the agricultural land photographed by the drone, a cluster containing many pixels in which the ground and agricultural products are captured may be used as a sampling point extraction cluster.
[0045]
　The sampling point extraction cluster to be selected may be one cluster, but two or more clusters may be selected as the sampling point extraction cluster.
　The pixels in the sampling point extraction cluster selected by the sampling point selection unit 101 are set as sampling points (pixels) and used by the next correction function calculation unit 102 to generate a correction function.
[0046]
　A specific example of the sampling point selection process executed by the sampling point selection unit 101 will be described with reference to FIG. 2 and below.
　FIG. 2 The process described with reference to the following is a clustering (cluster division) process for dividing the constituent pixels of the captured image 10 to be corrected into a plurality of subsets (clusters) by using the K-means method (K-means method). This is an example of applied processing.
[0047]
　As described above, the K-means method (K-means method) is a clustering method according to a non-hierarchical method, and is classified so as to obtain an optimum evaluation value using an evaluation function, and K subsets (K-means method). It is a method of classifying into clusters.
[0048]
　FIG. 2 is a diagram showing an example in which the captured image 10 to be corrected is divided into 11 clusters (K = 11) by the K-Means method.
　Although FIG. 2 is shown as a shade image, different colors are actually set corresponding to 11 clusters. Each color corresponds to each of the 11 clusters.
　Each cluster contains one or more pixels (elements), and the pixels classified into one cluster are pixels having similar feature information (for example, brightness and color) applied to the cluster classification.
[0049]
　As the pixel feature information applied to the cluster classification, various feature information such as the pixel "luminance", "color", or RGB value, for example, the pixel value of "G" can be used.
　For example, if "brightness" is used as feature information to be applied to cluster classification, it is possible to classify into a plurality of clusters by a set unit of pixels having similar brightness of each pixel.
　Further, for example, if the pixel value of "G" is used as the feature information applied to the cluster classification, it is possible to classify into a plurality of clusters by a set of pixels having similar G output values ​​of each pixel.
[0050]
　As described above, as the signal space applied as the feature information applied to the cluster classification, the brightness or color information of the captured image 10 or all three wavelengths of the RGB signal may be used, or an arbitrary wavelength signal in the RGB signal may be used. Or, the combination may be used. Further, the signal of the image obtained after converting the captured image 10 into a desired space such as a Lab space, a YUV space, or an xy chromaticity space, or a combination thereof may be used.
[0051]
　FIG. 3 is a histogram showing the average value of the signal values ​​of the elements (pixels) belonging to each of the 11 clusters divided by the K-means method shown in FIG. 2 and their frequencies.
　The horizontal axis is the average brightness (or pixel value) of the elements (pixels) in each of the 11 clusters divided by the K-means method shown in FIG. 2, and the vertical axis is the frequency of each cluster, that is, for each cluster. Indicates the number of elements (number of pixels) included.
[0052]
　The sampling point selection unit 101 can select, for example, a sampling point extraction cluster based on this histogram.
　For example, a predetermined number of clusters are selected as sampling point extraction clusters in the direction in which the average brightness value increases (on the right side of the graph in the figure) based on the cluster having the maximum frequency of the divided clusters. Furthermore, the pixels belonging to the selected cluster are randomly sampled.
　The sampling point selection unit 101 selects sampling pixels by, for example, this process.
[0053]
　In the histogram shown in FIG. 3, the cluster having the maximum power of the cluster is the second cluster from the left, and is defined in advance in the direction in which the average brightness value increases (on the right side of the graph in the figure) with reference to this cluster. Select a number of clusters, eg, 7 clusters, as sampling point extraction clusters.
　Pixels belonging to the seven sampling point extraction clusters selected in this way are randomly sampled.
[0054]
　For example, by selecting the sampling point extraction cluster in this way, the sampling points (pixels) are not selected from the clusters on the left end side and the right end side of the histogram. This is a process in which pixels whose luminance and pixel values ​​are significantly different from the average value of the entire image, that is, low-luminance pixels and high-luminance pixels are not sampled.
　By performing such processing, the sampling point can be selected from the pixel set excluding the pixels affected by shadows and specular reflection in the image.
[0055]
　The sampling point extraction cluster selection process using the histogram described with reference to FIG. 3 is an example, and as described above, the sampling point extraction cluster selection process belongs to, for example, the selected cluster. It can be performed by various processes such as selecting until the number of pixels exceeds a certain ratio in the number of pixels of the entire image.
[0056]
　FIG. 4 shows seven sampling point extraction clusters selected by using the histogram of FIG. 3, that is, seven sampling point extraction clusters selected in the direction of increasing the luminance average value based on the cluster having the maximum frequency of 11 clusters. It is a figure which shows the result of randomly selecting a sampling point from.
　White points in the image shown in FIG. 4 indicate sampling points (pixels).
[0057]
　A river is photographed on the upper right side of the image shown in FIG. 4, and the reflected light of sunlight is incident on the camera from the water surface of this river. That is, this river region is a specular reflection region. Such a specular reflection region is a high-luminance region whose brightness is far from the average value of the entire image, and in this processing example, sampling pixels are not selected from such a region.
　Similarly, sampling pixels are not selected even in a low-luminance region where the brightness is far from the average value of the entire image.
[0058]
　In this processing example, the sampling pixel is composed of pixels whose brightness and pixel value are close to the average value of the entire image, and is composed of pixels which are far from the average value and do not include low-luminance pixels and high-luminance pixels. ..
[0059]
　The pixel value information and the position information of the sampling pixel composed of the average pixel value of such an image are output to the correction function calculation unit 102 shown in FIG. 1, and the correction function calculation unit 102 is the pixel of the sampling pixel. Generate a correction function based on the value and position information.
[0060]
　As described above, the sampling point selection unit 101 outputs the pixel value (luminance value, etc.) of the sampling point and its position information to the next correction function calculation unit 102.
　The correction function calculation unit 102 generates a correction function by using the pixel value and the position information of the sampling point selected by the sampling point selection unit 101.
[0061]
　The correction function calculation unit 102 can generate a correction function based on the pixel value of the sampling pixel composed of the average pixel value of the image, and reduces the influence of the pixel having extreme brightness and the pixel value. Can be generated.
[0062]
　In the sampling point selection process described with reference to FIGS. 2 to 4, the sampling point selection unit 101 applies the K-means method (K-means method) to a subset (cluster) of K constituent pixels of the image. This is an example in which clustering is performed and sampling points are selected using the clustering results.
　As described above, the sampling point selection unit 101 can apply various clustering methods other than the K-means method (K-means method).
[0063]
　FIG. 5 shows an example in which clustering is performed by applying semantic segmentation in the sampling point selection unit 101.
[0064]
　As mentioned above, semantic segmentation is based on the degree of matching between the objects in the image and the dictionary data (trained data) for object identification based on the shape of various actual objects and other feature information. This is a method of identifying what category (people, cars, roads, plants, etc.) each constituent pixel belongs to and classifying them into clusters for each category.
[0065]
　As shown in FIG. 5, by performing semantic segmentation on the image, pixel classification (clustering) for each object as shown in FIG. 5 is executed.
　In the example shown in FIG. 5, cabbage, people, soil, rivers, houses, trees, roads, black spaces, soil, and these objects are identified. These are separate clusters.
　In addition, an unknown area (Unknown) in which the object cannot be identified may occur in the image.
[0066]
　For example, when the purpose of analysis is to analyze the growth status of cabbage, which is an agricultural product, it is necessary to accurately acquire information such as the color of cabbage. For this purpose, it is required to correct the pixel value of the image area where the cabbage is photographed to the pixel value with reduced brightness unevenness and color unevenness, that is, to the pixel value with the influence of the shadow of clouds removed. ..
[0067]
　When there is such an analysis purpose, an image area containing cabbage is selected as a sampling point extraction cluster, and elements (pixels) in the sampling point extraction cluster are selected as sampling points (pixels).
　The pixel value information of the sampling pixel is output to the correction function calculation unit 102 shown in FIG. 1, and the correction function calculation unit 102 generates a correction function based on the pixel value of the sampling pixel.
[0068]
　The correction function calculation unit 102 can generate a correction function based on the pixel value of the sampling pixel selected from the sampling point extraction cluster which is the image area of ​​the cabbage, and as a result, the correction function specialized for the image area of ​​the cabbage. Can be generated.
　In this way, by using the sampling points selected from the sampling point extraction cluster, it is possible to generate a correction function and perform the correction process without being affected by the signal values ​​of trees and rivers other than the correction target.
[0069]
　As described above, the following existing clustering method can be applied to the clustering in the sampling point selection unit 101, that is, the method of dividing the constituent pixels of the captured image 10 into a plurality of subsets (clusters).
　(A) K-means method (K-means method),
　(b) K-NN method (K shortest distance method: K-Nearest Neighbor method),
　(c) Ward method (Ward method),
　(d) Semantic segmentation,
[0070]
　Clustering in the sampling point selection unit 101 is not limited to these existing methods. For example, the captured image 10 is divided into a plurality of regions using a boundary set by the user or a predetermined boundary, and these divided regions are divided into a plurality of regions. It may be set as a cluster. A specific example is shown in FIG.
[0071]
　FIG. 6 shows the following cluster division example.
　(1) Cluster division example 1: Clustering processing example in which an image is equally divided into tiles
　(2) Cluster division example 2: Clustering processing example in which only the image edge is divided into a plurality of areas
　(3) Cluster division example 3: User Example of clustering processing in which an image area is divided using a designated boundary
　For example, various clustering may be performed in this way.
[0072]
　After performing clustering, the sampling point selection unit 101 determines a sampling point extraction cluster, and further selects a sampling point from the determined sampling point extraction cluster.
　The sampling point selection process can be executed, for example, as a process of randomly selecting an arbitrary number of pixels from the sampling point extraction cluster.
　Alternatively, a sampling point extraction cluster to be selected as a sampling point is further divided into a plurality of regions, and sampling points are randomly selected from each region so that the number of selections is less biased in each divided region. You may go.
[0073]
　As described above, the sampling point selection unit 101 includes
　(1) a clustering process for dividing the constituent pixels of the captured image 10 into a plurality of subsets (clusters), and
　(2) sampling to be sampled points from the plurality of clusters. The process of determining the point extraction cluster,
　(3) the process of selecting sampling points (pixels) from the determined sampling point extraction cluster, and
　these processes are executed.
　The pixel value (luminance value, etc.) information of the sampling point (pixel) selected by the sampling point selection unit 101 is input to the correction function calculation unit 102 together with the position information.
[0074]
　The correction function calculation unit 102 calculates a correction function to be applied to all the constituent pixels of the captured image 10 by using the pixel values ​​and position information of the sampling points (pixels) selected by the sampling point selection unit 101.
[0075]
　The correction function calculation unit 102 uses any of existing methods such as kernel regression interpolation (Kernel Regression), spline surface interpolation, polynomial approximation surface, and linear interpolation to perform correction surface fitting as a two-dimensional function of image coordinates. To calculate a correction function for correcting the pixel values ​​of the constituent pixels of the image.
[0076]
　Hereinafter, as an example of the correction function calculation process executed by the correction function calculation unit 102, an example of calculating the correction function using kernel regression interpolation will be described.
[0077]
　Kernel regression approximation is to project the data of the observation space d, which is the data observation space, onto the feature space (projection space) D, which is a space corresponding to a certain feature different from the observation space d, and project the feature space (projection space) D. The approximate plane corresponding to the data is calculated above, and the approximate plane (y (x)) on the original observation space d is obtained based on the calculated approximate plane on the projection space. From the approximate plane (y (x)) on the observation space d, a highly accurate approximate solution corresponding to all x on the original observation space d can be obtained.
[0078]
　In this example, the observation space d is composed of the positions of the sampling points (pixels) selected by the sampling point selection unit 101 and the pixel values ​​(luminance values).
　The approximate plane (y (x)) that can be calculated by the kernel regression interpolation described above is a plane that defines an approximate solution (estimated value) of pixel values ​​(luminance values) corresponding to all points including points other than sampling points. Become.
[0079]
　Let the total number of sampling points (pixels) selected by the sampling point selection unit 101 be N. Let d be a set of pixels with N sampling points.
　Assuming that x is the pixel position of N sampling points and t is the signal value, the set d can be shown as the following (Formula 1).
[0080]
[Number 1]

[0081]
　The pixel position x is a vector indicating two-dimensional coordinate data (x, y) indicating the pixel position.
　The signal value t corresponds to the feature information when clustering is performed, and is, for example, a luminance value.
　(X, t) is a combination of the pixel position x and the luminance value t of the pixel position x.
[0082]
　For example, if the feature information when clustering is performed is the RGB G signal value, the signal value t is the G signal value, and (x, t) is the pixel position x and the G of the pixel position x. It is a combination of signal values ​​t.
[0083]
　The correction function calculation unit 102 can generate an approximate plane y (x) and a correction function corresponding to the feature information applied to the clustering.
[0084]
　When the set d of pixels of N sampling points selected by the sampling point selection unit 101 is defined as in the above (Equation 1), the approximate plane y (x) (approximate plane calculation function) with respect to the input x is as follows ( It can be shown as Equation 2).
[0085]
[Number 2]

[0086]
　In the approximate plane calculation function y (x) shown in the above (Formula 2),
　λ is a regularization parameter.
　Further, K is a Gram matrix, which is a matrix shown in the following (Formula 3).
[0087]
[Number 3]

[0088]
　Further, k (x) of the above (Formula 2) is described by the following (Formula 4).
[0089]
[Number 4]

[0090]
　K (x, x') in the above (Equation 4) is a kernel function, and kernel functions such as a Gaussian kernel, a polynomial kernel, a whole subset kernel, and an ANOVA kernel can be used. In this example, an example using a Gaussian kernel will be described.
　When a Gaussian kernel is used, the kernel function: k (x, x') in the above (Equation 4) can be described as the following (Equation 5) using the standard deviation σ.
[0091]
[Number 5]

[0092]
　The correction function calculation unit 102 further applies the following (Formula 6) using the calculated approximate plane y (x) to generate a correction function for calculating the pixel value of each corrected pixel.
[0093]
[Number 6]

[0094]
　In the correction function shown in the above (Formula 6),
　Y'(x) is a signal value (pixel value or brightness value) corresponding to the corrected pixel position (x) of the captured image 10, and
　Y (x) is. The signal values ​​(pixel values ​​and brightness values) and
　y (x) corresponding to the pixel position (x) before correction of the captured image 10 are the approximate planes y (x) and
　Yref (x) represented by the above-mentioned (mathematical expression 2 ). Is a predetermined correction standard.
[0095]
　As the correction reference Yref (x), for example, the average value or the median value of the pixel values ​​of the sampling point extraction cluster, or the value selected by the user can be used. The value selected by the user may be a value whose value is changed according to the pixel position.
[0096]
　The above (Formula 6) is a 　correction reference Yref composed of the
　approximate plane y (x ) represented by the above-mentioned (Formula 2),
the average value and the median value of the pixel values ​​of the sampling point extraction cluster, and the value selected by the user. (X),
　This is a correction function in which the correction amount is set larger as the ratio of these two values ​​(Yref (x) / y (x)) is larger.
[0097]
　When Yref (x) and y (x) are equal and (Yref (x) / y (x)) = 1, the correction pixel value Y'(x) calculated according to the above (Formula 6) is before correction. It becomes equal to the pixel value Y (x) of, and
　Y'(x) = Y (x).
[0098]
　On the other hand, when the difference between the value of the correction reference Yref (x) and the value of the approximate plane y (x) is large and the ratio of these two values ​​(Yref (x) / y (x)) is large, the pixel before correction The amount of correction for the value Y (x) is set large.
　Specific examples of the correction mode will be described later.
[0099]
　The processing executed by the correction function calculation unit 102 is summarized as follows.
　(1) The approximate plane y (x) that defines the output value (pixel value, luminance value, etc.) with respect to the input x is calculated.
　(2) A correction function (Equation 6) that changes the correction amount according to the ratio between the approximate plane y (x) and the correction reference Yref (x) is calculated.
[0100]
　In this example, the "output value (pixel value, luminance value, etc.) with respect to the input x" in the above (1) is a pixel value or a luminance value corresponding to the pixel position, and the approximate plane y (x) is a pixel. It is an approximate plane y (x) that defines a pixel value and a luminance value corresponding to a position.
[0101]
　In this way, the correction function calculation unit 102 calculates the correction function corresponding to the signal value t corresponding to the feature information when clustering is performed.
　For example, when the feature information at the time of clustering is the luminance value and the signal value t at the time of generating the correction function corresponds to the luminance value, the correction function corresponding to the luminance value is calculated.
　This correction function corresponding to the brightness value is applied to the correction of the brightness value of the captured image 10 to be corrected. Further, a process of applying the correction function corresponding to the brightness value to the correction of the RGB pixel value of each of the constituent pixels of the captured image 10 may be performed. In this case, all RGB signal values ​​are corrected.
[0102]
　Further, for example, when the feature information when clustering is performed is the RGB G signal value and the signal value t at the time of generating the correction function corresponds to the pixel value of G, the correction function generated based on the G pixel value is used. It is also possible to perform processing such as applying only to the G pixel value of each of the constituent pixels of the captured image 10 to be corrected.
　In this case, among the RGB pixel values, only the G pixel value is corrected.
[0103]
　As described above, the correction function calculation unit 102 may be configured to calculate one correction function applied to all RGB pixel values ​​of each of the constituent pixels of the captured image 10, or may have three types of corrections corresponding to pixel values ​​of three RGB wavelengths. The functions may be calculated individually.
[0104]
　The next image correction unit (correction function application unit) 103 calculates the correction pixel values ​​of all the constituent pixels of the captured image 10 according to the correction function (mathematical expression 6) calculated by the correction function calculation unit 102, and calculates the calculated pixel values. The set corrected image 20 is generated.
[0105]
　The correction target is basically all the constituent pixels of the captured image 10. The image correction unit (correction function application unit) 103 applies the correction function calculated by the correction function calculation unit 102 to calculate the correction pixel values ​​of all the constituent pixels of the captured image 10, and sets the calculated pixel values. 20 is generated.
　It is also possible to set a specific image area or a specific cluster as a correction target by user setting.
[0106]
　The series of processes executed by each component of the image processing apparatus 100 shown in FIG. 1 has been described above.
　A specific difference between the process according to this embodiment and the image correction example by the conventional process will be described with reference to FIG. 7.
[0107]
　FIG. 7 shows the following two figures.
　(1) Conventional processing example (image correction processing example when spatially uniform sampling point selection is executed)
　(2) Processing example of the present disclosure ( image correction when sampling point selection based on clustering result is executed) Processing example)
[0108]
　(1) The conventional processing example is a processing example in which spatially uniform sampling points are selected from the two-dimensional plane of the image to be corrected. That is, sampling points are set at equal intervals on the image, the approximate plane y (x) is calculated based on the pixel values ​​of the sampling points, and the ratio between the approximate plane y (x) and the correction reference Yref (x). This is a processing example in which the correction is performed by applying the correction function (formula 6) that changes the correction amount according to the above. The correction reference Yref (x) uses the average brightness of the image.
[0109]
　On the other hand, (2) the processing example of the present disclosure is a processing example when sampling point selection is executed based on the clustering result. That is, it is a correction example when the correction functions calculated by the following processes (S1) to (S3) are applied.
　(S1) Clustering and sampling point extraction cluster determination process described with reference to FIGS. 2 to 4 and sampling point selection process from sampling point extraction cluster
　(S2) Approximation based on pixel values ​​and position information of sampling points. Calculation of surface y (x), calculation of
　correction function (Equation 6) that changes the correction amount according to the ratio of the calculated approximate surface y (x) and correction reference Yref (x),
　these steps S1 This is a processing example when correction is performed by applying the correction function (mathematical expression 6) generated by S3. The correction reference Yref (x) uses the average brightness of the image.
[0110]
　The example shown in FIG. 7 (2) is a processing example in which a cluster containing many pixels close to the average brightness or the intermediate brightness of the image to be corrected is set as the sampling point extraction cluster. A cluster containing many high-luminance pixels and low-luminance pixels that are far from the average brightness and intermediate brightness of the image to be corrected is not set as a sampling point extraction cluster.
[0111]
　7 (1) and 7 (2) show the figures (A), (B), and (C).
　FIG. (A) is a diagram showing a pixel position-luminance correspondence curve (= pixel luminance value before correction) and a sampling point selection mode.
　(B) is a figure which shows the approximate plane y (x) (= pixel position-luminance approximate curve) generated based on a sampling point.
　(C) is a pixel position-luminance correspondence curve (= pixel brightness value before correction), an approximate plane y (x), and a pixel position-luminance correspondence curve (= correction) which is a pixel value corrected based on a correction function. It is a figure which shows the back pixel luminance value). Further, in (C), Yref (x) in the above-described (Formula 6), that is, the average brightness value Yref (x) of the image used as a predetermined correction reference is shown.
　In each of (A) to (C), the horizontal axis is the pixel position and the vertical axis is the luminance value (pixel value).
[0112]
　FIG. 7 (1) In a conventional processing example, that is, an image correction processing example in which spatially uniform sampling point selection is executed, as shown in (A), the sampling points are horizontal axes indicating pixel positions. It is selected at equal intervals along. Pixels at the pixel position (x1) whose luminance value is rapidly increasing are also sampled.
　This high-luminance region corresponds to, for example, the river region of the image described above with reference to FIG.
[0113]
　As shown in FIGS. 7 (1) and 7 (B), the approximate plane y (x) is generated so as to smoothly connect the selected sampling points at equal intervals. The approximate surface y (x) is curved so as to approach the brightness level of the pixel position (x1) in the vicinity of the pixel position (x1) where the brightness is sharply increased.
[0114]
　7 (1) and 7 (C) show a pixel position-luminance correspondence curve (= corrected pixel brightness value) which is a pixel value corrected based on the correction function (mathematical formula 6).
　This pixel position-luminance correspondence curve (= corrected pixel brightness value) is a curve corresponding to the corrected pixel value Y'(x) calculated according to (Formula 6) described above.
　As shown in FIGS. 7 (1) and 7 (C), the pixel position-luminance correspondence curve (= corrected pixel brightness value) fluctuates greatly up and down in the vicinity of the pixel position (x1) where the brightness is rapidly increasing. It is set.
[0115]
　This is the
　Yref in (Formula 6)
　described above , that is, Y'(x) = (Yref (x) / y (x)) Y (x) ... (Formula 6) above (Formula 6). This is because the larger the difference between (x) and y (x), the larger (Yref (x) / y (x)), and the larger the correction amount of the pixel value Y (x) before correction. ..
[0116]
　As shown in FIGS. 7 (1) and 7 (C), in the vicinity of the pixel position (x1) where the brightness is sharply increased
　, a pixel position-luminance approximation curve (y (x)) corresponding to the correction function and a
　predetermined
　The difference between the average luminance value Yref (x) of the image used as the specified correction standard becomes large, and (Yref (x) / y (x)) in (Formula 6) described above becomes large, and the correction is made. The correction amount of the previous pixel value Y (x) becomes large.
[0117]
　As a result, as shown in FIGS. 7 (1) and 7 (C), the correction amount is set large in the vicinity of the pixel position (x1) where the brightness value before correction is significantly higher than the surroundings, and the pixels are abrupt. Value changes, or artifacts, occur.
　Further, the pixel value Y'(x) after the correction of the pixel position (x1) is corrected to be larger than the original pixel value Y (x) before the correction, and as a result, the brightness of the surrounding pixels is increased. The difference in brightness between the two and the image is reduced, and the dynamic range of the entire image is reduced.
[0118]
　On the other hand, the processing example of the present disclosure shown in FIG. 7 (2) is a processing example when sampling point selection is executed based on the clustering result. This is a processing example when a cluster containing many pixels close to the average brightness or intermediate brightness of the image is set as a sampling point extraction cluster. A cluster containing many high-luminance pixels and low-luminance pixels that are far from the average brightness and intermediate brightness of the image to be corrected is not set as a sampling point extraction cluster.
[0119]
　FIG. 7 (2) In the processing example of the present disclosure, as shown in (A), the sampling points are extracted from clusters containing many pixels close to the average brightness and intermediate brightness of the image, so that the brightness value is from the surroundings. The pixel at the pixel position (x1) that is rapidly increasing is not sampled.
[0120]
　As shown in FIGS. 7 (2) and 7 (B), the approximate plane y (x) is generated so as to smoothly connect the sampling points extracted from the cluster containing many pixels close to the average brightness and the intermediate brightness of the image. To. The approximate plane y (x) is not curved in the vicinity of the pixel position (x1) where the brightness is sharply increased. That is, as shown in FIGS. 7 (1) and 7 (B), a curve that approaches the brightness level of the pixel position (x1) is not formed.
[0121]
　7 (2) and 7 (C) show a pixel position-luminance correspondence curve (= after correction) which is a pixel value corrected according to a correction function (formula 6) generated based on a sampling point selected based on the clustering result. Pixel luminance value) is shown.
　This pixel position-luminance correspondence curve (= corrected pixel brightness value) is a curve corresponding to the corrected pixel value Y'(x) calculated according to (Formula 6) described above.
　As shown in FIGS. 7 (2) and 7 (C), in the vicinity of the pixel position (x1) where the brightness is rapidly increasing, the pixel position-luminance correspondence curve (= corrected pixel brightness value) is substantially before the correction. The setting is such that the pixel value (luminance value) is maintained.
[0122]
　This is the
　Yref in (Formula 6)
　described above , that is, Y'(x) = (Yref (x) / y (x)) Y (x) ... (Formula 6). This is because the smaller the difference between (x) and y (x), the smaller (Yref (x) / y (x)), and the smaller the correction amount of the pixel value Y (x) before correction. ..
[0123]
　As shown in FIGS. 7 (2) and 7 (C), in the vicinity of the pixel position (x1) where the brightness is rapidly increasing, the
　approximate plane y (x) and
　the average brightness of the image used as a predetermined correction reference are used. The difference from the value Yref (x)
　is a small setting.
　That is, (Yref (x) / y (x)) in (Formula 6) described above becomes small, and the correction amount of the pixel value Y (x) before correction becomes small.
[0124]
　This is because when the approximate plane y (x) is generated, the pixel at the pixel position (x1) whose brightness is rapidly increasing is not selected as the sampling point, and the pixel value of this high-luminance pixel is selected as the approximate plane y (x). This is because the setting is not reflected in.
[0125]
　As a result, as shown in FIGS. 7 (2) and 7 (C), the correction amount is set small at the pixel position (x1) where the brightness value before correction is significantly higher than the surroundings, and the pixel values ​​before and after the correction are set. Will be corrected so that no major change will occur.
[0126]
　As a result, a sudden change in pixel value as shown in FIGS. 7 (1) and 7 (C) described as the conventional process, that is, the occurrence of an artifact is prevented.
　Further, the pixel value Y'(x) after the correction of the pixel position (x1) does not change significantly from the original pixel value Y (x) before the correction, and as a result, the brightness difference from the brightness of the surrounding pixels is reduced. Without doing so, the decrease in the dynamic range of the entire image is also suppressed.
[0127]
　In this way, by performing image correction applying the image processing device 100 of the present disclosure shown in FIG. 1, it is possible to prevent the occurrence of artifacts in a portion where the brightness changes greatly, and also to prevent a decrease in the dynamic range of the entire image. It is possible to realize pixel value correction with less deterioration in image quality.
[0128]
　FIG. 8 shows another processing example of the present disclosure.
　(3) Processing example of the present disclosure (Image correction processing example when sampling point selection is executed based on the clustering result) In
　this processing example, for example, a part of the image is shaded by clouds and becomes dark. This is an example of image correction.
[0129]
　However, in this example, for example, the dark area behind the clouds is also the analysis target area, for example, the image area of ​​the cabbage of the crop that is the analysis target of the growth state, and the analysis target is required to be restored to the correct pixel value. This is an example when it is an area.
　In such a case, the dark area behind the clouds is also selected as a sampling point in the same manner as the other bright areas.
　That is, a cluster containing a large area of ​​cabbage to be analyzed as a subject object is set as a sampling point extraction cluster.
[0130]
　As described above, the processing example of the present disclosure shown in FIG. 8 (3) is a processing example in which an analysis target area, for example, an area including an image area of ​​cabbage, which is an agricultural product, is set as a sampling point extraction cluster.
[0131]
　FIG. 8 (3) In the processing example of the present disclosure, as shown in (A), even if the region has a decrease in brightness due to the influence of the shadow of clouds, if the region includes the image region of cabbage, other regions are used. Sampling points are extracted as in the bright areas of.
[0132]
　As shown in FIG. 8 (B), the approximate plane y (x) is generated so as to smoothly connect the sampling points. The approximate surface y (x) is curved so as to approach the brightness level at that position in the region where the brightness is reduced.
[0133]
　FIG. 8C shows a pixel position-luminance correspondence curve (= corrected pixel brightness value) which is a pixel value corrected according to a correction function (formula 6) generated based on a sampling point selected based on the clustering result. ) Is shown.
　This pixel position-luminance correspondence curve (= corrected pixel brightness value) is a curve corresponding to the corrected pixel value Y'(x) calculated according to (Formula 6) described above.
　As shown in FIG. 8C, in the vicinity of the pixel position where the brightness is reduced, the pixel position-luminance correspondence curve (= corrected pixel brightness value) is larger and brighter than the uncorrected pixel value (brightness value). It is a corrected setting.
[0134]
　This is the
　Yref in (Formula 6)
　described above , that is, Y'(x) = (Yref (x) / y (x)) Y (x) ... (Formula 6) above (Formula 6). This is because the larger the difference between (x) and y (x), the larger (Yref (x) / y (x)), and the larger the correction amount of the pixel value Y (x) before correction. ..
[0135]
　As shown in FIG. 　8C, 　the difference between the
　approximate plane y (x) and
the average brightness value Yref (x) of the image used as a predetermined correction reference in the vicinity of the pixel position where the brightness is reduced.
Is big.
　That is, (Yref (x) / y (x)) in (Formula 6) described above becomes large, and the correction amount of the pixel value Y (x) before correction becomes large.
[0136]
　This is a setting in which when the approximate surface y (x) is generated, the pixel at the pixel position where the brightness is reduced is also selected as a sampling point, and the pixel value of this low brightness pixel is reflected on the approximate surface y (x). Because it was done.
[0137]
　As a result, as shown in FIG. 8C, the correction amount is set large at the pixel position where the brightness value before correction is lower than that of the surroundings, and the correction is performed to increase the brightness value. Become.
[0138]
　As a result, the region where the brightness value is lowered due to the shadow of the cloud is brightly corrected, and a corrected image similar to that in which the influence of the cloud is eliminated is generated.
[0139]
　As described above, in the image correction to which the image processing apparatus 100 of the present disclosure shown in FIG. 1 is applied, even if the brightness fluctuates in the image area including the object to be corrected due to the influence of shadows or the like, the pixels in the image area are used. By setting the including cluster as a sampling point extraction cluster, it is possible to perform pixel value correction in which the influence of the shadow of the image area is reduced.
[0140]
　　[2. About the image processing sequence executed by the image processing apparatus]
　Next, the sequence of image processing executed by the image processing apparatus 100 shown in FIG. 1 will be described with reference to the flowchart shown in FIG.
[0141]
　The process according to the flowchart shown in FIG. 9 can be executed according to, for example, a program stored in the storage unit of the image processing apparatus 100. For example, it can be executed under the control of a data processing unit (control unit) having a CPU or the like having a program execution function.
　Hereinafter, the processing of each step of the flowchart shown in FIG. 9 will be sequentially described.
[0142]
　　(Step S101)
　First, in step S101 , the image processing device 100 inputs a captured image to be corrected.
　This captured image is the captured image 10 shown in FIG. 1, and is, for example, one image captured by a drone. The captured image is, for example, an image in which a part of the area is shaded by clouds or the like, the pixel value is set to be darker than that of the other area, and the brightness is uneven or the color is uneven.
[0143]
　　(Step S102)
　Next, in step S102 , the image processing apparatus 100 executes clustering (cluster division) in which the constituent pixels of the captured image 10 to be corrected are divided into a plurality of subsets (clusters).
　This process is executed by the sampling point selection unit 101 shown in FIG.
[0144]
　As described above, the following existing clustering methods can be applied.
　(A) K-means method (K-means method),
　(b) K-NN method (K-nearest neighbor method: K-Nearest Neighbor method),
　(c) Ward method (Ward method),
　(d) semantic segmentation, and
　also. Not limited to these existing methods, as described above with reference to FIG. 6, for example, the captured image 10 is divided into a plurality of regions using a boundary set by the user or a predetermined boundary. These segmented areas may be set as a cluster.
[0145]
　As the pixel feature information applied to the cluster classification, various feature information such as the pixel "luminance", "color", or RGB value, for example, the pixel value of "G" can be used.
　For example, if "brightness" is used as feature information to be applied to cluster classification, it is possible to classify into a plurality of clusters by a set unit of pixels having similar brightness of each pixel.
　Further, for example, if the pixel value of "G" is used as the feature information applied to the cluster classification, it is possible to classify into a plurality of clusters by a set of pixels having similar G output values ​​of each pixel.
[0146]
　　(Step S103)
　Next, in step S103 , the image processing apparatus 100 selects a sampling point from the captured image 10 to be corrected.
　This process is also executed by the sampling point selection unit 101 shown in FIG.
[0147]
　The sampling point selection unit 101 clusters the constituent pixels of the captured image 10 to be corrected in step S102, determines a sampling point extraction cluster, and further selects a sampling point from the determined sampling point extraction cluster.
[0148]
　As described above, for example, any of the following methods can be applied as a method for determining the sampling point extraction cluster.
　(1) One or more clusters are designated as sampling point extraction clusters in order from the cluster having the largest number of samples (pixels).
Let the cluster be a sampling point extraction cluster.
　(2) The cluster in which the average value or median value of each cluster is the center of all clusters, or the number of clusters adjacent to the cluster is defined as a sampling point extraction cluster.
　(3) The user selects a specific cluster, and the selected cluster is used as a sampling point extraction cluster.
[0149]
　For example, when performing the process (2) above, if the average value or median value of the pixel values ​​and brightness values ​​of the pixels belonging to each cluster is used, the cluster including the extremely dark pixel area or bright area in the image is a sampling point. It will not be set in the extraction cluster and the sampling points will not be extracted.
[0150]
　Further, when the cluster is selected by the user in (3) above, a specific subject (object) for which an accurate pixel value is to be obtained, for example, a cluster containing many pixels of agricultural land crops is selected as a sampling point extraction cluster. Perform processing.
[0151]
　　(Step S104)
　Next, in step S104 , the image processing apparatus 100 generates a correction function using the pixel values ​​of the sampling points.
　This process is executed by the correction function calculation unit 102 shown in FIG.
[0152]
　The correction function calculation unit 102 calculates a correction function to be applied to all the constituent pixels of the captured image 10 by using the pixel values ​​of the sampling points (pixels) selected by the sampling point selection unit 101.
　As described above, the correction function calculation unit 102 first uses any of the existing methods such as kernel regression interpolation, spline curved surface interpolation, polynomial approximated curved surface, and linear interpolation to obtain 2 image coordinates. The approximate plane y (x) is calculated by performing correction curved surface fitting or the like as a dimensional function.
　The approximate plane y (x) is, for example, the approximate plane y (x) shown in (Formula 2) described above.
[0153]
　Further, the correction function calculation unit 102 calculates a correction function (mathematical expression 6) that changes the correction amount according to the ratio between the approximation surface y (x) and the correction reference Yref (x).
　As the correction reference Yref (x), for example, the average value or the median value of the pixel values ​​of the sampling point extraction cluster, or the value selected by the user can be used. The value selected by the user may be a value whose value is changed according to the pixel position.
[0154]
　The correction function calculation unit 102 calculates the correction function using only the pixel values ​​of the sampling points (pixels) selected by the sampling point selection unit 101, for example, to obtain a sampling pixel composed of the average pixel value of the image. A correction function based on the pixel value can be generated. That is, it is possible to generate a correction function that eliminates the influence of pixels having extreme brightness and pixel values.
[0155]
　Further, for example, when an object to be analyzed is specified, the pixel value of the object is corrected by generating a correction function based on the pixel value of the sampling point selected from the cluster containing a large image area of ​​the object. It is also possible to generate a correction function to perform the above more accurately.
　For example, when the object to be analyzed is cabbage in a field photographed by a drone, if a correction function based on the pixel value of the sampling pixel selected from the sampling point extraction cluster, which is the image area of ​​cabbage, is generated, the image area of ​​cabbage It is possible to generate a correction function specialized for.
[0156]
　　(Step S105)
　Next, in step S105 , the image processing apparatus 100 applies a correction function (mathematical expression 6) to the pixel value of each pixel of the captured image 10 to be corrected to calculate the correction pixel value.
　This process is a process executed by the image correction unit (correction function application unit) 103 shown in FIG.
[0157]
　The image correction unit (correction function application unit) 103 applies the correction function calculated by the correction function calculation unit 102 in step S104, that is, the following (Formula 6) to calculate the pixel value of each pixel after correction. ..
　Y'(x) = (Yref (x) / y (x)) Y (x) ... (Formula 6)
　Pixel value calculation processing according to the above formula is executed.
[0158]
　In the above (Formula 6),
　Y'(x) is a signal value (pixel value, brightness value, etc.) corresponding to the corrected pixel position
　(x) of the captured image 10, and Y (x) is the captured image 10. The signal value (pixel value, brightness value, etc.) corresponding to the pixel position (x) before correction,
　y (x) is, for example, the approximate plane y (x) and
　Yref (x) represented by the above-mentioned (Formula 2). , It is a pre-defined correction standard.
　Correction standard: As Yref (x), for example, the average value or median value of the pixel values ​​of the sampling point extraction cluster, or a value selected by the user can be used. The value selected by the user may be a value whose value is changed according to the pixel position.
[0159]
　The image correction unit (correction function application unit) 103 calculated the correction pixel values ​​of all the constituent pixels of the captured image 10 according to the correction function (mathematical expression 6) calculated by the correction function calculation unit 102, and set the calculated pixel values. The corrected image 20 is generated.
[0160]
　　(Step S106)
　Finally, the image processing apparatus 100 generates and outputs a corrected image composed of the pixel values ​​corrected by the pixel value correction processing in step S105. For example, output processing for the display unit and storage processing for the storage unit are executed.
[0161]
　As a result, as described above with reference to FIGS. 7 and 8, the occurrence of artifacts and the decrease in the dynamic range of the image are prevented in the portion where the brightness change is large, and the corrected image with less deterioration in image quality is generated. Can be done.
[0162]
　　[3. Configuration and processing of the second embodiment of the image processing apparatus of the present disclosure]
　Next, the configuration and processing of the second embodiment of the image processing apparatus of the present disclosure will be described with reference to FIGS. 10 and below.
　FIG. 10 is a block diagram showing a configuration example of the image processing device 120 according to the second embodiment of the present disclosure. As shown in FIG. 10, the image processing device 120 includes a reduced image generation unit 121, a sampling point selection unit 101, a correction function calculation unit 102, and an image correction unit (correction function application unit) 103.
　The image processing device 120 shown in FIG. 10 has a configuration in which a reduced image generation unit 121 is provided in front of the sampling point selection unit 101 of the image processing device 100 of the first embodiment described above with reference to FIG.
　The sampling point selection unit 101, the correction function calculation unit 102, and the image correction unit (correction function application unit) 103, which have other configurations, execute almost the same processing as in the first embodiment.
　Hereinafter, the points different from those of the first embodiment described above will be mainly described.
[0163]
　The image processing device 120 inputs one captured image 10 to be corrected, executes correction, and outputs the corrected image 20.
　In this embodiment as well, the photographed image 10 to be corrected is one photographed image taken by one shooting process of the camera. That is, it is not a composite image generated by performing a stitching process for joining a plurality of images.
[0164]
　The reduced image calculation unit 121 inputs the captured image 10 to be corrected, and generates a reduced image obtained by reducing the captured image 10.
　As the image reduction method, existing image reduction methods such as the nearest neighbor method, the area averaging method, the bilinear method, the bicubic method, and the Lanczos method can be applied. Further, the reduced image may be calculated by using the median value (median value) or the average value in the reduced range as the signal value of the reduced image.
[0165]
　The sampling point selection unit 101 and the correction function calculation unit 102 execute the sampling point selection process and the correction function calculation process using the reduced image generated by the reduced image calculation unit 121.
　The correction function calculation unit 102 calculates the approximate plane y (x) of the reduced image size and the correction function.
[0166]
　The image correction unit (correction function application unit) 103 generates a correction image by using the correction function calculated by the correction function calculation unit 102.
[0167]
　A captured image 10 that is not reduced is input to the image correction unit (correction function application unit) 103, and the pixel values ​​of the constituent pixels of the captured image 10 are corrected.
　The image correction unit (correction function application unit) 103 first enlarges the approximate surface y (x) of the reduced image size calculated by the correction function calculation unit 102 and the correction function to the size of the captured image 10 before reduction, and enlarges the image. The corrected correction function is generated, and the enlarged correction function is applied to the captured image 10.
[0168]
　Alternatively, the reduced image generated by the reduced image generation unit 121 is input to the image correction unit (correction function application unit) 103, and the approximate surface y (x) of the reduced image size calculated by the correction function calculation unit 102 and the correction function are used. , A correction image of the reduction correction image may be generated by applying to the reduction image, and then a process of enlarging the reduction correction image may be performed. As the image enlargement method, existing enlargement methods such as the nearest neighbor method, the bilinear method, the bicubic method, and the Lanczos method can be used.
[0169]
　In the processing of the second embodiment, the reduced image generation unit 121 generates a reduced image of the captured image 10, and then the sampling point selection unit 101, the correction function calculation unit 102, and the image correction unit (correction function application unit) 103. The processing can be executed as a processing for a reduced image, the amount of calculation can be reduced, the processing speed can be improved, and the hardware and programs applied to the processing can be simplified.
[0170]
　Next, the sequence of image processing executed by the image processing apparatus 120 shown in FIG. 10 will be described with reference to the flowchart shown in FIG.
[0171]
　The process according to the flowchart shown in FIG. 11 can be executed according to, for example, a program stored in the storage unit of the image processing device 120. For example, it can be executed under the control of a data processing unit (control unit) having a CPU or the like having a program execution function.
　Hereinafter, the processing of each step of the flowchart shown in FIG. 11 will be sequentially described.
[0172]
　　(Step S201)
　First, in step S201 , the image processing device 120 inputs a captured image to be corrected.
[0173]
　This captured image is the captured image 10 shown in FIG. 10, and is, for example, one image captured by a drone. The captured image is, for example, an image in which a part of the area is shaded by clouds or the like, the pixel value is set to be darker than that of the other area, and the brightness is uneven or the color is uneven.
[0174]
　　(Step S202)
　Next, in step S202 , the image processing device 120 generates a reduced image of the captured image 10 to be corrected.
　This process is a process executed by the reduced image generation unit 121 shown in FIG.
[0175]
　The reduced image calculation unit 121 applies existing image reduction methods such as the nearest neighbor method, the area averaging method, the bilinear method, the bicubic method, and the Lanczos method to the captured image 10 to be corrected to obtain a reduced image. Generate. A reduced image may be generated in which the median value (median value) or the average value in the reduced range is used as the signal value of the reduced image.
[0176]
　　(Steps S203 to S205)
　The processes of steps S203 to S205 are the same as the processes of steps S102 to S104 of the flow in the first embodiment described above with reference to FIG.
　However, in this embodiment, the processes of steps S203 to S205 are executed for the reduced image generated in step S202.
[0177]
　In step S203, clustering (cluster division) is performed in which the constituent pixels of the reduced image of the captured image 10 to be corrected are divided into a plurality of subsets (clusters).
　In step S204, a sampling point extraction cluster is determined from the cluster generated as a clustering result, and a sampling point is further selected from the determined sampling point extraction cluster.
　These processes are executed by the sampling point selection unit 101 shown in FIG.
[0178]
　Further, in step S205, a correction function is generated using the pixel value of the sampling point.
　This process is executed by the correction function calculation unit 102 shown in FIG.
　As described above, in the second embodiment, the correction function calculation unit 102 calculates the correction function y (x) (= approximate plane calculation function y (x)) of the reduced image size.
[0179]
　　(Steps S206 to S207)
　Next, in step S206, the image processing apparatus 120 applies a correction function to the pixel value of each pixel of the captured image 10 to be corrected to calculate the correction pixel value.
　Finally, in step S207, a corrected image composed of the corrected pixel values ​​is generated and output. For example, output processing for the display unit and storage processing for the storage unit are executed.
　These processes are the processes executed by the image correction unit (correction function application unit) 103 shown in FIG.
[0180]
　As described above, the image correction unit (correction function application unit) 103 inputs, for example, the captured image 10 that is not reduced and corrects the pixel values ​​of the constituent pixels of the captured image 10.
　In this case, the image correction unit (correction function application unit) 103 first enlarges the approximate surface y (x) of the reduced image size calculated by the correction function calculation unit 102 and the correction function to the size of the captured image 10 before reduction. Then, an enlarged correction function is generated, and the enlarged correction function is applied to the captured image 10.
[0181]
　Alternatively, the image correction unit (correction function application unit) 103 inputs the reduced image generated by the reduced image generation unit 121, and the approximate surface y (x) of the reduced image size calculated by the correction function calculation unit 102 or the correction function is used. , It may be applied to the reduced image as it is to generate a corrected image of the reduced corrected image, and then a process of enlarging the reduced corrected image may be performed. As the image enlargement method, existing enlargement methods such as the nearest neighbor method, the bilinear method, the bicubic method, and the Lanczos method can be used.
[0182]
　As described above, in the processing of the second embodiment, the reduced image generation unit 121 can generate a reduced image of the captured image 10, and the subsequent processing can be executed by applying the reduced image. As a result, the amount of calculation can be reduced, the processing speed can be improved, and the hardware and programs applied to the processing can be simplified.
[0183]
　　[4. Regarding the configuration and processing of the third embodiment of the image processing apparatus of the present disclosure]
　Next, the configuration and processing of the third embodiment of the image processing apparatus of the present disclosure will be described with reference to FIGS. 12 and below.
　FIG. 12 is a block diagram showing a configuration example of the image processing device 130 according to the third embodiment of the present disclosure. As shown in FIG. 12, the image processing device 130 includes a reduced image generation unit 121, a sampling point selection unit 101, a correction function calculation unit 102, an image correction unit (correction function application unit) 103, and a composite image generation unit 131. Has.
[0184]
　The image processing device 130 shown in FIG. 12 has a configuration in which a composite image generation unit 131 is added to the image processing device 120 of the second embodiment described above with reference to FIG.
　A composite image generation unit 131 is added as a post-stage processing unit of the image correction unit (correction function application unit) 103.
　The reduced image generation unit 121, the sampling point selection unit 101, the correction function calculation unit 102, and the image correction unit (correction function application unit) 103, which have other configurations, execute almost the same processing as in the second embodiment.
[0185]
　Hereinafter, the points different from those of the second embodiment described above will be mainly described.
　The image processing device 130 shown in FIG. 12 generates and outputs a composite image 30 as a final output image, which is a combination of a plurality of images taken by a drone, for example.
　The input image is, for example, a plurality of images taken by a drone. As shown in FIG. 12, a plurality of images P1 to Pn are sequentially input to the image processing device 130 as captured images 10.
　The images P1 to Pn are, for example, images continuously taken by a drone, and each is an image obtained by taking a part of a vast area. By stitching these images together, it is possible to generate an image corresponding to one vast area.
[0186]
　However, drones that move in the air change their posture when shooting each image due to the influence of wind and the like. As a result, at the timing of shooting each image, the reflected light from the subject received by the camera changes due to the change in the relationship between the camera, the subject, and the light source, and the brightness and color of each shot image are not uniform and become sparse. It ends up. When images having different brightness and colors are joined together to generate one composite image, a low-quality image in which the continuity of the joint of each image is lost is obtained.
[0187]
　In addition, one image may include shaded areas and non-shadowed areas such as clouds in the sky or other flying objects such as airplanes and birds. The colors are different. That is, uneven brightness and uneven color occur.
[0188]
　The image processing device 130 of the third embodiment shown in FIG. 12 performs correction for reducing luminance unevenness and color unevenness in one captured image unit, and further, brightness difference and color difference between each image constituting the composite image are also obtained. It produces a reduced, high-quality composite image 30.
[0189]
　The reduced image calculation unit 121 to the image correction unit 103 of the image processing device 130 of the third embodiment shown in FIG. 12 sequentially execute processing for each of the images (P1 to Pn) constituting the composite image before correction. Then, the corrected images 20-1 to n are generated.
　The processing for each image is the same as the configuration of the second embodiment described above with reference to FIG.
　In the third embodiment, the processing of each image unit is sequentially executed for each of the images (P1 to Pn) constituting the composite image before correction. That is, the same process is repeated n times for n images and executed to generate corrected images 20-1 to n.
[0190]
　The composite image generation unit 131 inputs these n corrected images 20-1 to n and generates a composite image 30 in which the corrected images 20-1 to n are joined together.
[0191]
　The composite image generation unit 131 sets, for example, a correction reference signal as a reference when stitching images, and sets an average signal (mean value of pixel value and brightness value) of each image and an overlapping region at the time of stitching. Synthesis that reduces joint discontinuity by matching the mean signal (mean value of pixel value and brightness value) and the mean value of sampling point extraction cluster (mean value of pixel value and brightness value) with the correction reference signal. Generate an image.
[0192]
　As the correction reference signal, a preset value may be used, or an average value of a specific single or a plurality of images may be used.
　Specifically, the adjacent images are connected by performing adjustments such as matching the average brightness of adjacent images or setting the difference to a specified value range or less.
[0193]
　In addition, at the time of image connection (joining), the overlapping area of ​​the adjacent image is detected and the overlapping area is deleted from one image, or the average value of the overlapping area included in the plurality of images is calculated and one. Perform processing such as stitching images as regions to correct the image so that overlapping regions do not appear in the composite image.
[0194]
　The image processing device 130 of the third embodiment shown in FIG. 12 of the third embodiment performs correction for reducing the luminance unevenness and the color unevenness for each captured image, and further, the luminance between the images constituting the composite image. A high-quality composite image 30 with reduced differences and color differences is generated.
[0195]
　The sequence of image processing executed by the image processing apparatus 130 shown in FIG. 12 will be described with reference to the flowchart shown in FIG.
　The process according to the flowchart shown in FIG. 13 can be executed according to, for example, a program stored in the storage unit of the image processing device 130. For example, it can be executed under the control of a data processing unit (control unit) having a CPU or the like having a program execution function.
　Hereinafter, the processing of each step of the flowchart shown in FIG. 13 will be sequentially described.
[0196]
　　(Step S301)
　First, in step S301 , the image processing device 130 selects a captured image to be corrected from a plurality of captured images.
　That is, the images (P1 to Pn) constituting the uncorrected composite image shown in FIG. 12 are sequentially selected as the correction target images.
[0197]
　These images P1 to Pn are, for example, images continuously taken by a drone, and each is an image obtained by taking a part of a vast area. By stitching these images together, it is possible to generate an image corresponding to one vast area.
　However, each image is a sparse image in which the brightness and color are not uniform due to the difference in shooting conditions, and even in one image, the brightness unevenness and color unevenness due to the shadow area, the specular reflection area, etc. Exists.
[0198]
　　(Step S302)
　Next, in step S302 , the image processing device 130 inputs the correction target image selected in step S301.
[0199]
　　(Steps S303 to S308)
　The processes of steps S303 to S308 are the same as the processes of steps S202 to S207 in the flow in the second embodiment described above with reference to FIG.
　That is, in step S303, the reduction process is executed for the captured image 10 which is the correction target image.
　In step S304, clustering is performed on the reduced image.
　In step S305, the sampling point extraction cluster is determined from the clustering result, and the sampling point is selected from the sampling point extraction cluster.
[0200]
　Further, in step S306, a correction function generation process based on the pixel value of the selected sampling point is executed.
　In step S307, the generated correction function is applied to calculate the correction pixel value of the image to be corrected.
　In step S308, a corrected image composed of the corrected pixel values ​​is generated.
[0201]
　　(Step S309) In
　step S309, it is determined whether or not the generation of the corrected images for all the captured images (P1 to Pn) is completed.
　If there is an unprocessed image, the unprocessed image is selected in step S301, and the processes of steps S302 to S308 are executed on the unprocessed image.
[0202]
　If it is determined in step S309 that the generation of the corrected images for all the captured images (P1 to Pn) is completed, the process proceeds to step S310.
　At this point, the generation of the corrected images 20-1 to n in which the pixel values ​​shown in FIG. 12 are corrected is completed.
[0203]
　　(Step S310)
　Finally, in step S310 , the image processing apparatus 130 connects all the corrected images 20-1 to n to generate and output a composite image 30.
　This process is a process executed by the composite image generation unit 131 shown in FIG.
[0204]
　As described above, the composite image generation unit 131 sets, for example, a correction reference signal as a reference when stitching images, an average signal of each image, an average signal of overlapping regions at the time of stitching, and a sampling point. By matching the average value of the extracted clusters with the correction signal, an image with discontinuous steps at the joints is generated.
　As the correction reference signal, a preset value, for example, an average value of a specific single image or a plurality of images can be used. This makes it possible to generate a composite image in which the average brightness of adjacent images is matched or the difference is set to be within the specified value range.
[0205]
　At the time of image connection (joining), overlapping areas of adjacent images are detected, and processing such as deleting the overlapping areas from one image is performed. Alternatively, the average value of the overlapping regions included in the plurality of images may be calculated and the images may be stitched together as one region. Further, the pixel value of the overlapping region may be determined by α blending of the superimposed image, and the image of the superimposed region may be smoothly changed. For example, when considering two images, the purpose of α-blending is to continuously change the signal value from image A to image B. However, during α-blending, lens distortion, shading, and peripheral blurring occur. In consideration of the influence of, the value may be set so that the distance from the center of the screen is reflected in addition to the above blending rate and the ratio becomes higher toward the center of the screen.
[0206]
　By executing these processes, the image processing device 130 of the third embodiment shown in FIG. 12 reduces the luminance unevenness and the color unevenness of one captured image unit, and at the same time, between the images constituting the composite image It is possible to generate a high-quality composite image 30 in which the brightness difference and the color difference are also reduced.
[0207]
　　[5. Other Examples]
　Next, other examples will be described with reference to FIGS. 14 and 14 and below.
　The following three examples will be described.
　(Example 4) Refer to the sampling point position information of the processed surrounding image and execute the sampling point selection process in the unprocessed image
　(Example 5) Refer to the correction function information of the processed surrounding image. Then, the calculation process of the correction function in the unprocessed image is executed
　(Example 6). The example in which the calculation process of the approximate surface in the unprocessed image is executed with reference to the approximate surface information of the processed surrounding image.
[0208]
　(5-1) (Example 4) An embodiment in which a sampling point selection process for an unprocessed image is executed with reference to the sampling point position information of the processed surrounding image.
[0209]
　First, as the fourth embodiment, an embodiment in which the sampling point selection process in the unprocessed image is executed with reference to the sampling point position information of the processed surrounding image will be described.
[0210]
　FIG. 14 is a block diagram showing a configuration example of the image processing device 140 of the fourth embodiment. As shown in FIG. 14, the image processing device 140 includes a reduced image generation unit 121, a sampling point selection unit 101, a correction function calculation unit 102, an image correction unit (correction function application unit) 103, and a composite image generation unit 131. Further, it has an image-corresponding sampling point position information storage unit 141.
[0211]
　The image processing device 140 shown in FIG. 14 has a configuration in which an image-corresponding sampling point position information storage unit 141 is added to the image processing device 130 of the third embodiment described above with reference to FIG.
　The reduced image generation unit 121, the sampling point selection unit 101, the correction function calculation unit 102, the image correction unit (correction function application unit) 103, and the composite image generation unit 131, which have other configurations, are basically the third embodiment. Performs almost the same processing as.
　However, the processing executed by the sampling point selection unit 101 is different.
[0212]
　The image-corresponding sampling point position information storage unit 141 stores the sampling point position information of the image for which the sampling point selection process has been performed in the sampling point selection unit 101.
[0213]
　Further, when the sampling point selection unit 101 executes the sampling point selection process for a new image, the sampling point selection process is performed on the processed image around the image to be processed, that is, the sampling point selection unit 101. The sampling point position information of the image is acquired from the image-corresponding sampling point position information storage unit 141. The sampling point selection unit 101 refers to the information acquired from the image-corresponding sampling point position information storage unit 141, and executes a new image sampling point selection process.
[0214]
　A specific example will be described with reference to FIG.
　The upper left part of FIG. 15 shows a composite image before correction. The composite image is composed of, for example, a plurality of images continuously taken by a drone.
　Each image constituting this composite image is sequentially input to the image processing device 140 and processed.
　It is assumed that the processing is performed according to the arrow shown in FIG.
[0215]
　On the right side of FIG. 15, five adjacent images that are a part of the images constituting the composite image are shown.
　These are the image Px to which the sampling point selection unit 101 will execute the sampling point selection process, and the images Pa to Pd in ​​which the sampling point selection process has already been executed in the surrounding image of the image Px.
[0216]
　When the sampling point selection process is performed on the image Px to which the sampling point selection process is to be executed, the sampling point selection unit 101 selects the images Pa to Pd for which the sampling point selection process has already been executed, that is, the surrounding image of the image Px. Refer to the set sampling point.
　Using this reference result, the sampling point selection process of the image Px is performed.
[0217]
　For example, it is determined that the region a and the region b as shown in the figure are likely to include the image to be selected as the sampling point based on the surrounding image, and the region a and b included in the image Px are used. Select the sampling point.
[0218]
　By performing such processing, it is possible to omit or simplify the steps such as the clustering processing described above, the determination of the sampling extraction cluster, and the selection processing of the sampling points from the sampling extraction cluster.
　Further, for example, as shown in FIG. 16, by adopting the sampling points selected in the surrounding image as the sample points of the image to be selected as the sampling point at a constant ratio, the sample points of the image to be selected as the sampling point are set to the surroundings. It is possible to avoid the selection arrangement of sample points that are significantly different from the image. By performing such a process, it is possible to stably perform the correction.
[0219]
　It should be noted that these processes are performed without omitting the clustering process described above, the determination of the sampling extraction cluster, and the sampling point selection process from the sampling extraction cluster, and sampling of the processed surrounding image is performed when these processes are performed. It is preferable to perform processing such as correcting the sampling point selected based on the information of the surrounding image with reference to the point information.
[0220]
　(5-2) (Example 5) Example of executing the calculation process of the correction function in the unprocessed image with reference to the correction function information of the processed surrounding image.
[0221]
　Next, as Example 5, an example in which the calculation process of the correction function in the unprocessed image is executed with reference to the correction function information of the processed surrounding image will be described.
[0222]
　FIG. 17 is a block diagram showing a configuration example of the image processing device 150 of the fifth embodiment. As shown in FIG. 17, the image processing device 150 includes a reduced image generation unit 121, a sampling point selection unit 101, a correction function calculation unit 102, an image correction unit (correction function application unit) 103, and a composite image generation unit 131. Further, it has an image-corresponding correction function storage unit 151.
[0223]
　The image processing device 150 shown in FIG. 17 has a configuration in which an image correspondence correction function storage unit 151 is added to the image processing device 130 of the third embodiment described above with reference to FIG.
　The reduced image generation unit 121, the sampling point selection unit 101, the correction function calculation unit 102, the image correction unit (correction function application unit) 103, and the composite image generation unit 131, which have other configurations, are basically the third embodiment. Performs almost the same processing as.
　However, the processing executed by the correction function calculation unit 102 is different.
[0224]
　The image-corresponding correction function storage unit 151 stores the correction function of the image for which the correction function calculation process has been performed by the correction function calculation unit 102.
[0225]
　Further, when the correction function calculation unit 102 executes the correction function calculation process for a new image, the corrected image around the image to be processed, that is, the correction function calculation unit 102 performs the correction function calculation process. The correction function information of the image is acquired from the image correspondence correction function storage unit 151. The correction function calculation unit 102 executes a new image correction function calculation process with reference to the information acquired from the image correspondence correction function storage unit 151.
[0226]
　A specific example will be described with reference to FIG.
　The upper left part of FIG. 18 shows a composite image before correction. The composite image is composed of, for example, a plurality of images continuously taken by a drone.
　Each image constituting this composite image is sequentially input to the image processing device 150 and processed.
　It is assumed that the processing is performed according to the arrow shown in FIG.
[0227]
　On the right side of FIG. 18, five adjacent images that are a part of the images constituting the composite image are shown.
　These are the image Px for which the correction function calculation unit 102 will execute the correction function calculation process, and the images Pa to Pd for which the correction function has already been calculated in the surrounding images of the image Px.
[0228]
　When the correction function calculation unit 102 calculates the correction function for the image Px to be executed from now on, the correction function calculation unit 102 corrects the images Pa to Pd for which the correction function has already been calculated, that is, the surrounding image of the image Px. Refer to the function.
　Using this reference result, the correction function calculation process of the image Px is performed.
[0229]
　For example, the correction function of the image Px is calculated by synthesizing the correction functions of the five surrounding images shown in FIG. For example, a process such as α blending is executed to calculate a new image correction function based on a plurality of correction functions of the surrounding image.
　For example, a new image correction function is calculated based on a plurality of correction functions of the surrounding image by executing a process such as α blending in which the weight is set larger for the reference image having a shorter distance.
[0230]
　By performing such processing, it is possible to omit or simplify the calculation processing step of the correction function that involves the calculation of the approximated surface described above.
[0231]
　It should be noted that these processes are performed without omitting the calculation process of the correction function that involves the calculation of the approximate plane described above, and the peripheral image is referred to when performing these processes with reference to the correction function information of the processed peripheral image. It is preferable to perform processing such as modifying the correction function calculated based on the information in.
[0232]
　(5-3) (Example 6) An embodiment in which the calculation process of the approximate surface in the unprocessed image is executed with reference to the approximate surface information of the processed surrounding image.
[0233]
　Next, as the sixth embodiment, the calculation process of the approximate plane in the unprocessed image is executed with reference to the approximate plane information of the processed surrounding image.
[0234]
　FIG. 19 is a block diagram showing a configuration example of the image processing device 160 of the sixth embodiment. As shown in FIG. 19, the image processing device 160 includes a reduced image generation unit 121, a sampling point selection unit 101, a correction function calculation unit 102, an image correction unit (correction function application unit) 103, and a composite image generation unit 131. Further, it has an image-corresponding approximate surface storage unit 161.
[0235]
　The image processing device 160 shown in FIG. 19 has a configuration in which an image-corresponding approximate surface storage unit 161 is added to the image processing device 130 of the third embodiment described above with reference to FIG.
　The reduced image generation unit 121, the sampling point selection unit 101, the correction function calculation unit 102, the image correction unit (correction function application unit) 103, and the composite image generation unit 131, which have other configurations, are basically the third embodiment. Performs almost the same processing as.
　However, the processing executed by the correction function calculation unit 102 is different.
[0236]
　The image-corresponding approximate plane storage unit 161 stores the approximate plane y (x) of the image obtained by the correction function calculation unit 102 for calculating the approximate plane y (x).
[0237]
　Further, when the correction function calculation unit 102 executes the calculation process of the approximate surface y (x) for the new image, the processed image around the image to be processed, that is, the approximate surface y in the correction function calculation unit 102. The approximate plane y (x) information of the image subjected to the calculation process of (x) is acquired from the image-corresponding approximate plane storage unit 161. The correction function calculation unit 102 executes a new image approximation surface y (x) calculation process with reference to the information acquired from the image-corresponding approximation surface storage unit 161.
[0238]
　A specific example will be described with reference to FIG.
　The upper left part of FIG. 20 shows a composite image before correction. The composite image is composed of, for example, a plurality of images continuously taken by a drone.
　Each image constituting this composite image is sequentially input to the image processing device 160 and processed.
　It is assumed that the processing is performed according to the arrow shown in FIG.
[0239]
　On the right side of FIG. 20, five adjacent images that are a part of the images constituting the composite image are shown.
　These are the image Px for which the correction function calculation unit 102 will execute the approximation surface y (x) calculation process, and the images Pa to Pd for which the approximation surface y (x) has already been calculated in the surrounding image of the image Px.
[0240]
　The correction function calculation unit 102 has already calculated the approximate plane y (x) when calculating the approximate plane y (x) for the image Px to be executed from now on. Pa to Pd, that is, the approximate plane y (x) of the peripheral image of the image Px is referred to.
　Using this reference result, the approximate plane y (x) calculation process of the image Px is performed.
[0241]
　For example, the approximate plane y (x) of the image Px is calculated by synthesizing the approximate plane y (x) of the five surrounding images shown in FIG. 20. For example, a process such as α blending is executed to calculate the approximate plane y (x) of the new image based on the plurality of approximate planes y (x) of the surrounding image.
　For example, a process such as α-blending in which the weight is set larger as the reference image is closer is executed, and the approximate plane y (x) of the new image is calculated based on a plurality of correction functions of the surrounding image.
　By performing such processing, it is possible to omit or simplify the above-mentioned calculation processing step of the approximate plane y (x).
[0242]
　It should be noted that the calculation process of the approximate surface y (x) is performed without omitting the calculation process of the approximate surface y (x) described above, and the approximate surface y (x) of the processed surrounding image is performed when this process is performed. It is preferable to perform processing such as correcting the approximate plane y (x) calculated based on the information of the surrounding image with reference to the information.
[0243]
　The following three examples have been described above with reference to FIGS. 14 to 20.
　(Example 4) Refer to the sampling point position information of the processed surrounding image and execute the sampling point selection process in the unprocessed image
　(Example 5) Refer to the correction function information of the processed surrounding image. Then, the calculation process of the correction function in the unprocessed image is executed
　(Example 6). The example in which the calculation process of the approximate surface in the unprocessed image is executed with reference to the approximate surface information of the processed surrounding image.
[0244]
　In these examples, the information of the surrounding processed image is referred to in the sampling point selection, the correction function, or the approximate plane calculation process.
　These Examples 4 to 6 may be configured to be implemented individually, but may be configured to be implemented in combination. For example, various configurations such as a configuration in which the fourth and fifth embodiments are combined are possible.
[0245]
　It should be noted that any of the following settings can be set as the processing procedure when a plurality of reference processes are performed, such as in the fourth and fifth embodiments.
　(A) Procedure for looping captured images for each processing block of the sampling selection unit and correction function calculation unit (inner loop)
　(b) Looping the processing of the processing block for each image and sequentially Procedure to process (outer loop)
[0246]
　Further, the processed peripheral image does not exist at the start of the processing, but in that case, for example, the following processing can be performed.
　(A) Processing that recursively repeats processing using peripheral information with the sampled without peripheral image information as the initial value,
　(b) Correction when correcting the captured images in order and correcting each image Processing to obtain sampling points and correction functions using only completed information
[0247]
　Further, the image used as the reference image is not limited to the image before correction, and may be an image after correction. That is, any of the following processing modes can be used.
　(A) A method of using an image in which only sampling points and correction functions are calculated as a reference image in a captured image before
　correction (b) A method in which an image in which only sampling points and correction functions are calculated is used as a reference image in an image after correction. how to
[0248]
　In the processing example described with reference to FIGS. 15, 18, and 20, the surroundings determined from the positional relationship of a plurality of images constituting one large composite image as reference images for sampling points and correction functions. An example using an image, that is, an adjacent image has been described.
　In addition to the configuration in which such a positionally adjacent image is used as a reference image, for example, when correcting a moving image, the images before and after the image frame to be processed (for example, N frames before and after) may be used as the reference image. It is possible.
[0249]
　Further, as a specific processing example when the sampling point is determined from the sampling points of the surrounding image, for example, any of the following methods can be applied.
　(A) Pixels at the same pixel position as the sampling point (pixel) of the surrounding image are selected at a constant ratio.
　(B) Clustering, semantic segmentation, sampling point extraction cluster determination processing, and sampling point selection processing are executed by applying a plurality of images including surrounding images.
[0250]
　Further, the following method is also possible as a method of using the information of the superposed area between the surrounding image and the image to be processed.
　(C) Select a fixed percentage sampling point from the overlapping area (all the overlapping areas of the peripheral image and the image to be processed
　) (d) When replacing the signal of the overlapping area with the signal of only one of the images, the above-mentioned Select a fixed percentage sampling point only from the signal of the image to be replaced.
[0251]
　Further, in the correction function calculation process and the approximate surface calculation process, the correction function and the approximate surface may be recalculated and corrected using a weighted average whose weight is the proximity to the surrounding image. ..
[0252]
　　[6. Hardware Configuration Example of Image Processing Device]
　Next, a hardware configuration example of the image processing device of the present disclosure will be described with reference to FIG. The hardware shown in FIG. 21 is an example of a specific hardware configuration of the image processing apparatus of each embodiment described with reference to FIG. 1 and others.
[0253]
　The CPU (Central Processing Unit) 301 functions as a control unit or a data processing unit that executes various processes according to a program stored in the ROM (Read Only Memory) 302 or the storage unit 308. For example, the process according to the sequence described in the above-described embodiment is executed. The RAM (Random Access Memory) 303 stores programs and data executed by the CPU 301. These CPU 301, ROM 302, and RAM 303 are connected to each other by a bus 304.
[0254]
　The CPU 301 is connected to the input / output interface 305 via the bus 304, and the input / output interface 305 is connected to an input unit 306 consisting of various switches, a keyboard, a mouse, a microphone, a sensor, etc., and an output unit 307 consisting of a display, a speaker, and the like. Has been done.
　The CPU 301 executes various processes in response to a command input from the input unit 306, and outputs the process results to, for example, the output unit 307.
[0255]
　The storage unit 308 connected to the input / output interface 305 is composed of, for example, a hard disk or the like, and stores a program executed by the CPU 301 and various data. The communication unit 309 functions as a transmission / reception unit for Wi-Fi communication, Bluetooth (registered trademark) (BT) communication, and other data communication via a network such as the Internet or a local area network, and communicates with an external device.
[0256]
　The drive 310 connected to the input / output interface 305 drives a removable medium 311 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory such as a memory card, and records or reads data.
[0257]
　　[7. Summary of the structure of the present disclosure] The
　examples of the present disclosure have been described in detail with reference to the specific examples. However, it is self-evident that one of ordinary skill in the art can modify or substitute the examples without departing from the gist of the present disclosure. That is, the present invention has been disclosed in the form of an example, and should not be construed in a limited manner. In order to judge the gist of this disclosure, the column of claims should be taken into consideration.
[0258]
　The technology disclosed in the present specification can have the following configuration.
　(1) A sampling point selection unit that selects a sampling point to be used for calculating a correction function applied to pixel value correction of an image from the image, and
　a pixel value and position information of the sampling point selected by the sampling point selection unit are applied. It has a correction function calculation unit that calculates the correction function,
　and an image correction unit that applies the correction function to correct the pixel value of the image. The
　sampling point selection unit has a
　plurality of constituent pixels of the image. 　An image processing device that executes clustering that divides into a subset of the
　above,
determines a sampling point extraction cluster from a plurality of clusters generated by the clustering, and executes a process of selecting a sampling point from the sampling point extraction cluster.
[0259]
　(2) The sampling point selection unit uses
　(a) one or more clusters in order from the cluster with the maximum number of pixels as sampling point extraction clusters, and
　(b) the average value or the center of the pixel values ​​of the pixels included in each cluster. The clusters whose values ​​are the center of all clusters or the number of clusters adjacent to the clusters are sampling point extraction clusters, and
　(c) the clusters selected by the user are sampling point extraction clusters,
　(a) to (c) above. The image processing apparatus according to (1), wherein a sampling point extraction cluster is determined by any method.
[0260]
　(3)
　The image processing apparatus according to (1), wherein the sampling point selection unit determines a cluster containing many subjects to be analyzed in the image as a sampling point extraction cluster.
[0261]
　(4) The correction function calculation unit calculates
　an approximate surface y (x) in which pixel values ​​corresponding to pixel positions are defined,
　and the ratio of the approximate surface y (x) to a predetermined correction reference Yref (x). The image processing apparatus according to any one of (1) to (3), which generates a correction function that changes the correction amount according to the above.
[0262]
　(5)
　The image processing apparatus according to (4), wherein the correction reference Yref (x) is either an average value, a median value, or a user-selected value of the pixel values ​​of the sampling point extraction cluster.
[0263]
　(6) The correction function calculation unit uses
　at least one of kernel regression interpolation, spline curve interpolation, polynomial approximation surface, and linear interpolation for the approximation surface y (x) in which the pixel value corresponding to the pixel position is defined. (4) The image processing apparatus according to (5).
[0264]
　(7) The correction function calculation unit calculates
　an approximate surface y (x) in which pixel values ​​corresponding to pixel positions are defined,
　and the ratio of the approximate surface y (x) to a predetermined correction reference Yref (x). The following correction function,
　Y'(x) = (Yref (x) / y (x)) Y (x), which changes the correction amount according to 　the above
　, where
Y'(x) is after the correction of the image. The pixel value corresponding to the pixel position (x),
　Y (x) is the pixel value corresponding to the pixel position (x) before correction of the image, and the
　correction function is generated according to any one of (1) to (6). Image processing device.
[0265]
　(8) The sampling point selection unit performs the
　clustering by
　(a) K- means method,
　(b) K-NN method,
　(c) Ward method,
　(d) semantic segmentation, and the
　above (a) to (d). The image processing apparatus according to any one of (1) to (7), which is executed by applying any of the methods.
[0266]
　(9)
　The image processing apparatus according to any one of (1) to (8 ), wherein the sampling point selection unit executes the clustering based on specific feature information of pixels of the image.
[0267]
　(10) The feature information is the brightness of the pixels of the image, a signal of at least one of RGB, or a signal in Lab space, YUV space, xy chromaticity space, or any of these chair spaces obtained by converting the RGB image. The image processing apparatus according to (9).
[0268]
　(11) The image processing apparatus further includes
　a reduced image generation unit that reduces the image to generate a reduced image, and the
　sampling point selection unit and the correction function calculation unit include the reduced image generation unit. The image processing apparatus according to any one of (1) to (10), which executes processing on the generated reduced image.
[0269]
　(12) The image processing
　according to any one of (1) to (11) , wherein the image processing apparatus has a composite image generation unit that joins the corrected images generated by the image correction unit to generate one composite image. apparatus.
[0270]
　(13) The composite image generation unit
　previously obtains the pixel value average value of each correction image to be stitched, the pixel value average value of the overlapping region of each correction image, or the pixel value average value of the sampling point extraction cluster. The image processing apparatus according to (12), which generates a composite image in which the discontinuity of the joint is reduced by executing a process of matching the set correction reference pixel value.
[0271]
　(14) In the
　sampling point selection process , the sampling point selection unit inputs the sampled point selection information that has been executed from the surrounding image of the image to be processed, and refers to the input information to perform the sampling point selection process of the image to be processed. The image processing apparatus according to any one of (1) to (13) to be executed.
[0272]
　(15) In the
　correction function calculation process , the correction function calculation unit inputs the correction function information calculated from the surrounding image of the processing target image, and refers to the input information to calculate the correction function of the processing target image. The image processing apparatus according to any one of (1) to (14) to be executed.
[0273]
　(16) An image processing method executed in an image processing apparatus, wherein the
　sampling point selection unit selects a sampling point to be used for calculating a correction function applied to pixel value correction of the image from the image, and a sampling point selection step.
　A correction function calculation step in which the correction function calculation unit calculates the correction function by applying the pixel value and position information of the sampling point selected by the sampling point selection unit, and an
　image correction unit applies the correction function. An image correction step for correcting the pixel value of the image is executed, and the
　sampling point selection step includes
　a step of executing clustering for dividing the constituent pixels of the image into a plurality of subsets and
　a plurality of clusters generated by the clustering.
　An image processing method including a step of determining a sampling point extraction cluster from the above and a step of selecting a sampling point from the sampling point extraction cluster.
[0274]
　(17) A program that executes image processing in an image processing apparatus,
　and a sampling point selection step of causing a sampling point selection unit to select a sampling point to be used for calculating a correction function applied to pixel value correction of an image from the image. , The
　correction function calculation step of applying the pixel value and the position information of the sampling point selected by the sampling point selection unit to the correction function calculation unit to calculate the correction function , and applying the correction function to the
　image correction unit. An image correction step for correcting the pixel value of the image is executed, and in the
　sampling point selection step,
　a step of executing clustering for dividing the constituent pixels of the image into a plurality of subsets and a step
　generated by the clustering are performed. A
　program that executes a step of determining a sampling point extraction cluster from a plurality of clusters and a step of selecting a sampling point from the sampling point extraction cluster.
[0275]
　In addition, the series of processes described in the specification can be executed by hardware, software, or a composite configuration of both. When executing processing by software, install the program that records the processing sequence in the memory in the computer built in the dedicated hardware and execute it, or execute the program on a general-purpose computer that can execute various processing. It can be installed and run. For example, the program can be pre-recorded on a recording medium. In addition to installing on a computer from a recording medium, it is possible to receive a program via a network such as LAN (Local Area Network) or the Internet and install it on a recording medium such as a built-in hard disk.
[0276]
　The various processes described in the specification are not only executed in chronological order according to the description, but may also be executed in parallel or individually as required by the processing capacity of the device that executes the processes. Further, in the present specification, the system is a logical set configuration of a plurality of devices, and the devices having each configuration are not limited to those in the same housing.
Industrial applicability
[0277]
　As described above, according to the configuration of one embodiment of the present disclosure, an image processing device and a method capable of generating a high-quality corrected image with reduced luminance unevenness and color unevenness are realized.
　Specifically, for example, a sampling point selection unit that selects a sampling point to be used for calculating a correction function applied to pixel value correction of an image from an image, and a correction function that applies the pixel value and position information of the sampling point to perform a correction function. It has a correction function calculation unit to calculate and an image correction unit to correct the pixel value of the image by applying the correction function, and the sampling point selection unit is a clustering unit that divides the constituent pixels of the image into a plurality of subsets (clusters). (Cluster division) is executed, the sampling point extraction cluster is determined from the multiple clusters generated by the clustering, and the sampling point is selected from the sampling point extraction clusters.
　With this configuration, an image processing device and method capable of generating a high-quality corrected image with reduced luminance unevenness and color unevenness are realized.
Code description
[0278]
　　10 Captured image
　　20 Corrected image
　100 Image processing device
　101 Sampling point selection unit
　102 Correction function calculation unit
　103 Image correction unit
　120 Image processing device
　121 Reduced image generation unit
　130 Image processing device
　131 Composite image generation unit
　140 Image processing device
　141 Image compatible sampling Point position information storage unit
　150 Image processing device
　151 Image processing correction function storage unit
　160 Image processing device
　161 Image compatible approximate surface storage unit
　301 CPU
　302 ROM
　303 RAM
　304 Bus
　305 Input / output interface
　306 Input unit
　307 Output unit
　308 Storage unit
　309 Communication Part
　310 Drive
　311 Removable media
The scope of the claims
[Claim 1]
　The sampling point selection unit that selects the sampling point used for calculating the correction function applied to the pixel value correction of the image from the image, and
　the pixel value and position information of the sampling point selected by the sampling point selection unit are applied to the above. It has a correction function calculation unit that calculates a correction function,
　and an image correction unit that applies the correction function to correct the pixel value of the image. The
　sampling point selection unit is a
　set of a plurality of constituent pixels of the image. 　An image processing device that executes clustering that divides into the
　above,
determines a sampling point extraction cluster from a plurality of clusters generated by the clustering, and selects a sampling point from the sampling point extraction cluster.
[Claim 2]
　In the sampling point selection unit,
　(a) one or more clusters are set as sampling point extraction clusters in order from the cluster with the maximum number of pixels, and
　(b) the average value and the median value of the pixel values ​​of the pixels included in each cluster are all. One of
　the
　above (a) to (c), wherein the cluster that is the center of the cluster or several clusters adjacent to the cluster is the sampling point extraction cluster, and (c) the cluster selected by the user is the sampling point extraction cluster . The image processing apparatus according to claim 1, wherein a sampling point extraction cluster is determined by a method.
[Claim 3]

　The image processing apparatus according to claim 1, 　wherein the sampling point selection unit determines a cluster including many subjects to be analyzed in the image as a sampling point extraction cluster.
[Claim 4]
　The correction function calculation unit calculates
　an approximate surface y (x) that defines a pixel value corresponding to a pixel position, and corresponds
　to a ratio between the approximate surface y (x) and a predetermined correction reference Yref (x). The image processing apparatus according to claim 1, wherein a correction function that fluctuates the correction amount is generated.
[Claim 5]

　The image processing apparatus according to claim 4, 　wherein the correction reference Yref (x) is either an average value, a median value, or a user-selected value of the pixel values ​​of the sampling point extraction cluster.
[Claim 6]
　The correction function calculation unit calculates
　the approximate plane y (x) that defines the pixel values ​​corresponding to the pixel positions by using at least one of kernel regression interpolation, spline curved surface interpolation, polynomial approximated curved surface, and linear interpolation. The image processing apparatus according to claim 4.
[Claim 7]
　The correction function calculation unit calculates
　an approximate surface y (x) that defines pixel values ​​corresponding to pixel positions, and corresponds
　to a ratio between the approximate surface y (x) and a predetermined correction reference Yref (x). The following correction function that changes the correction amount,
　Y'(x) = (Yref (x) / y (x)) Y (x),
　where
　Y'(x) is the corrected pixel position of the image ( 　The image processing apparatus according to claim 1
　, wherein x) a corresponding pixel value and Y (x) are pixel values ​​corresponding to the pixel position (x) before correction of the image, and the
correction function is generated.
[Claim 8]
　The sampling point selection unit performs the
　clustering by
　(a) K- means method,
　(b) K-NN method,
　(c) Ward method,
　(d) semantic segmentation,
　or any of the above (a) to (d). The image processing apparatus according to claim 1, wherein the method is applied and executed.
[Claim 9]

　The image processing apparatus according to claim 1, 　wherein the sampling point selection unit executes the clustering based on specific feature information of pixels of the image.
[Claim 10]
　The feature information is the brightness of the pixels of the image, a signal of at least one of RGB, or a signal in the Lab space, the YUV space, the xy chromaticity space, or any of the spaces obtained by converting the RGB image. Item 9. The image processing apparatus according to Item 9.
[Claim 11]
　The image processing device further
　has a reduced image generation unit that reduces the image to generate a reduced image, and the
　sampling point selection unit and the correction function calculation unit are reduced images generated by the reduced image generation unit. The image processing apparatus according to claim 1, which executes processing on an image.
[Claim 12]
　The image processing device
　according to claim 1, wherein the image processing device includes a composite image generation unit that joins the corrected images generated by the image correction unit to generate one composite image.
[Claim 13]
　The composite image generation unit performs a preset
　correction of the pixel value average value of each correction image to be stitched, the pixel value average value of the overlapping region of each correction image, or the pixel value average value of the sampling point extraction cluster. The image processing apparatus according to claim 12, wherein a process of matching the reference pixel value is executed to generate a composite image in which the discontinuity of the joint is reduced.
[Claim 14]
　In the
　sampling point selection process , the sampling point selection unit inputs the sampled point selection information that has been executed from the surrounding image of the image to be processed, and executes the sampling point selection process of the image to be processed by referring to the input information. Item 1. The image processing apparatus according to item 1.
[Claim 15]
　The correction function calculation unit
　inputs the correction function information calculated from the surrounding image of the processing target image in the correction function calculation processing, and executes the calculation processing of the correction function of the processing target image with reference to the input information. Item 1. The image processing apparatus according to item 1.
[Claim 16]
　An image processing method executed in an image processing device, which is a
　sampling point selection step in which a sampling point selection unit selects a sampling point to be used for calculating a correction function applied to pixel value correction of an image from the image, and a
　correction function calculation. A correction function calculation step in which the
　unit calculates the correction function by applying the pixel value and position information of the sampling point selected by the sampling point selection unit, and an image correction unit applies the correction function to the image. An image correction step for correcting pixel values ​​is executed, and the
　sampling point selection step includes
　a step of executing clustering for dividing the constituent pixels of the image into a plurality of subsets and a
　sampling point from a plurality of clusters generated by the clustering.
　An image processing method including a step of determining an extraction cluster and a step of selecting a sampling point from the sampling point extraction cluster.
[Claim 17]
　It is a program that executes image processing in an image processing device, and has
　a sampling point selection step that causes a sampling point selection unit to select a sampling point to be used for calculating a correction function applied to pixel value correction of an image from the image, and a
　correction function. A correction function calculation step of applying the pixel value and position information of the sampling point selected by the sampling point selection unit to the calculation unit to calculate the
　correction function, and applying the correction function to the image correction unit to calculate the image. In the
　sampling point selection step,
　a step of executing clustering for dividing the constituent pixels of the image into a plurality of subsets and
　a plurality of clusters generated by the clustering are executed.
　A program that executes a step of determining a sampling point extraction cluster from the above and a step of selecting a sampling point from the sampling point extraction cluster.