Specification
FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENTS RULES, 2003
COMPLETE SPECIFICATION (See section 10, rule 13)
“DEVICE, METHOD AND PROGRAM FOR
IMAGE PROCESSING, RECORDING MEDIUM,
AND IMAGING DEVICE”
FUJIFILM Corporation., a company incorporated in Japan, of 26-30, Nishiazabu 2-chome, Minato-ku, Tokyo 106-8620, Japan
The following specification particularly describes the invention and the manner in which it is to be performed.
{Description}
{Title of Invention}
DEVICE, METHOD AND PROGRAM FOR IMAGE PROCESSING, RECORDING
MEDIUM, AND IMAGING DEVICE
{Technical Field}
The present invention relates to an image processing device, method and program, a recording medium, and an imaging device, and more particularly relates to a technology for reducing fixed patterns generated due to repeating cycles of every specified pixel group in an image sensor.
{Background Art}
Figure 20 is a view illustrating one example of a color filter array provided for an image sensor. The color filter array illustrated in Figure 20 is a primary color Bayer array.
The Bayer array includes a basic array pattern of 2 x 2 pixels, and this basic array pattern is repeatedly arranged in a horizontal direction and a vertical direction. The basic array pattern includes pixels of three primary colors: red (R), blue (B) and green (G) pixels. G pixels include Gr pixel adjacent to R pixel in the horizontal direction and Gb pixel adjacent to B pixel in the horizontal direction.
Gr pixel and Gb pixel are respectively adjacent to pixels which are different in color in the horizontal direction. For example, Gr pixel may have a pixel value larger than the pixel value of a neighboring Gb pixel due to leakage of light (color mixture) from an adjacent R pixel, which may generate a level difference in pixel value between Gr pixel and Gb pixel. This level difference may periodically be repeated in accordance with the basic array pattern, as a result of which a fixed pattern may disadvantageously appear.
Figure 21 illustrates a new mosaic-like color filter array proposed by an applicant of the present invention (The specification of Japanese Patent Application No. 2011-034627).
The basic array pattern of this color filter array is made of 6 × 6 pixels, and the basic array pattern includes 20 G pixels, 8 R pixels and 8 B pixels. In the case of this basic array pattern, the pixels of the same color are often adjacent to pixels (8 pixels) which are different in color in accordance with positions in the basic array pattern. Accordingly, a fixed pattern is easily generated due to color mixture. Moreover, since the color filter array illustrated in Figure 21 has the basic array pattern larger in size than the Bayer array, the fixed pattern becomes more notable.
Further, in CMOS (Complementary Metal-Oxide Semiconductor)-type image sensors, amplifiers shared by pixels are embedded in a CMOS substrate. In an example illustrated in Figure 22, 4 pixels of 2 × 2 arrangement share one amplifier A. Due to such substrate structure of image sensors, a difference in output level is generated depending on the positions of pixels with respect to the shared amplifier (upper left, upper right, lower left, and lower right positions with respect to the amplifier A), and this generates a fixed pattern corresponding to the repeating cycles of the substrate structure.
As a method for reducing this kind of fixed patterns, it can be considered to perform filtering with use of a filter, which has a filter size of 6 × 6 pixels and which has weighted filter coefficients by center weighting as illustrated in Figure 23A. In this case, if the filter is applied while the pixel group in a processing area is shifted, a filter coefficient of 4 is applied, for example, to a G pixel P illustrated in Figure 23B, though a filter coefficient of 2 is applied to the G pixel P when the filter is moved as illustrated in Figure 23C. When different filter coefficients are applied in this way in the state where a level difference is present between the G pixel P and G pixels around it, an image after filtering becomes uneven and successful reduction of the fixed pattern cannot be achieved.
Contrary to this, if filtering is performed with use of a filter having uniform filter coefficients as illustrated in Figure 24, the fixed pattern can be reduced. However, an image is blurred and a detail of the image is lost, which may deteriorate image quality.
PTL 1 discloses an image signal processor capable of performing noise reduction in proportion to an output signal level of a solid-state image sensor so as to reduce noise of the solid-state image sensor without losing necessary components of the image signal. The image signal processor determines the magnitude of the level of
image signals generated by the solid-state image sensor and changes the degree of low pass filtering in accordance with the magnitude.
PTL 2 discloses a method for descreening a halftone area in an image, in which a low pass filter is modified and applied at the time of descreening the halftone area.
PTL 3 discloses an imaging device in which a filter size of a low pass filter is changed based on a relative distance from an in-focus position so that the filter size monotonously increases in proportion to increase in the relative distance, and thereby the intensity of noise reduction processing performed on video signals is changed.
{Citation List}
{Patent Literature}
{PTL 1} Japanese Patent Application Laid-Open No. 2001-148797
{PTL 2} National Publication of International Patent Application No. 2005-520915
{PTL 3} Japanese Patent Application Laid-Open No. 2010-239492
{Summary of Invention} {Technical Problem}
The invention disclosed in PTL 1 is to reduce light shot noise depending on an output signal level of the solid-state image sensor. Accordingly, the invention disclosed in PTL 1 is not effective for reducing fixed patterns, and strong fixed patterns end up remaining.
The inventions disclosed in PTL 2 and PTL 3 are not adapted to reduce fixed patterns. A filter correction method disclosed in PTL 2 and a filter size changing method disclosed in PTL 3 are not adapted to reduce fixed patterns either.
The present invention has been made based on such circumstances, and an object of the present invention is to provide an image processing device, method and program, a recording medium, and an imaging device capable of reducing fixed patterns generated due to repeating cycles of pixel configuration in an image sensor and capable of leaving a detail intact.
{Solution to Problem}
In order to accomplish the above object, an invention according to one aspect of the present invention includes: image acquisition means configured to acquire an image taken by imaging means including an image sensor having pixel configuration with repeating cycles of M × N (M, N: integers of 2 or more) pixels; and filtering means having a filter with a K × L (K, L: integers of M
Figure 1A illustrates one example of a mosaic-like color filter array arranged on each of photoelectric conversion elements of an image sensor on which the photoelectric conversion elements are arrayed two-dimensionally.
The color filter array illustrated in Figure 1A includes a basic array pattern (pattern illustrated with a thick frame) which is a square array pattern corresponding to 6 × 6 pixels. This basic array pattern is arranged repeatedly in a horizontal direction and a vertical direction. That is, the color filter array is arranged so that filters of respective R, G, and B colors (R filter, G filter and B filter) are arranged in the horizontal direction and the vertical direction with cyclicity corresponding to the basic array pattern (6 × 6 pixels).
The basic array pattern illustrated in Figure 1A may also be construed as an arrangement made of an "a" array of 3 × 3 pixels and a "b" array of 3 × 3 pixels placed alternately in the horizontal direction and the vertical direction.
In each of the "a" array and the "b" array, G filters which are luminance pixels are arranged at four corners and at the center, so that they are arranged on both the diagonal lines. In the "a" array, R filters are arrayed in the horizontal direction and B filters are arrayed in the vertical direction, across the central G filter. Contrary to this, in the "b" array, B filters are arrayed in the horizontal direction and R filters are arrayed in the vertical direction, across the central G filter. In other words, in the "a" array and the "b" array, positional relationship of R filters and B filters are reversed, though other arrangement features are identical.
In an RGB mosaic image outputted from the image sensor (color image sensor) having the color filter array, a fixed pattern corresponding to repeating cycles of the basic array pattern is generated.
Figure 1B illustrates filter coefficients of a filter for reducing the fixed pattern.
The filter illustrated in Figure 1B has a 9 × 9 filter size which is larger than the basic array pattern (fixed pattern). When the filter size is divided into nine, 3 × 3 areas, filter coefficients assigned to each of the areas are set so that a filter coefficient of a central area is 4, a filter coefficient of areas on the upper, lower, left and right sides thereof is 2, and a filter coefficient of four corner areas is 1. More specifically, the filter coefficients are weighted to be larger in the vicinity of the center of the filter (pixels closer to the central section of the filter are set to have larger filter coefficients).
Further, as illustrated in Figure 2A, when the basic array pattern of 6 × 6 pixels is divided into four, 3 × 3 areas, and the respective areas are defined as A, B, C, and D, relation between the respective areas A to D and the respective areas obtained by a 9 × 9 filter divided into nine areas are as illustrated in Figure 2B.
More specifically, when the area D of the basic array pattern corresponds to the area of the central section of the filter which was divided into nine areas, the areas B of the basic array pattern correspond to the areas on the upper and lower side of the central area. And the areas C of the basic array pattern correspond to the areas on the left and right side of the central section. The areas A of the basic array pattern correspond to the areas of four corners.
In this case, the number of each of the areas A, B, C and D on the filter is 4:2:2:1. Meanwhile, the filter coefficient in each of the areas A, B, C and D is 1:2:2:4. That is, the sums of the filter coefficients corresponding to the respective areas A, B, C and D are 4 × 1=4, 2 × 2=4, 2 × 2=4, and 1 × 4=4, which are all equal to each other.
Accordingly, when convolution arithmetic operation is applied to the filter coefficients of the filter and pixel values of the mosaic image corresponding to the color filter array, uniform filter coefficients are applied to the respective areas A, B, C and D of the basic array pattern, so that the areas are smoothed. It becomes possible to reduce the fixed pattern resulting from repeating cycles of the basic array pattern.
Further, since the filter coefficients in the 9 × 9 filter are weighted to be larger in the vicinity of the center as illustrated in Figure 1B, it becomes possible to prevent losing of a detail of the filtered image.
Figure 3 is a view illustrating another example of the mosaic-like color filter array arranged on the image sensor and one example of filter coefficients for a fixed pattern reduction filter corresponding to the color filter array.
The color filter array illustrated in Figure 3 includes a basic array pattern made of a square array pattern corresponding to 3 × 3 pixels, and the basic array pattern is repeatedly arranged in the horizontal direction and the vertical direction. Note that the basic array pattern illustrated in Figure 3 corresponds to the "a" array of 3 × 3 pixels illustrated in Figure 1A.
In contrast, the filter for reducing a fixed pattern has a 5 × 5 filter size. Moreover, the filter has filter coefficients weighted to be larger in the vicinity of the center and is so set that the sums of filter coefficients in the 5 × 5 filter size, corresponding to the pixels that are in identical positional relationship on the basic array pattern of 3 × 3 pixels, are all equal to each other.
Figure 4A illustrates the filter coefficients of the 5 × 5 filter.
As illustrated in Figure 4B, a pair of a column j1 and a column j4 and a pair of a column j2 and a column j5 on this filter are in identical positional relationship on the basic array pattern. By adding up filter coefficients allocated to these columns, filter coefficients as seen in a column (j1+j4) and a column (j2+j5) illustrated on the right side of Figure 4B can be obtained.
A pair of a row i1 and a row i4 and a pair of a row i2 and a row i5 on this filter are also in identical positional relationship on the basic array pattern. By adding up the filter coefficients allocated to these rows (rows of the added filter coefficients on the right-hand side of Figure 4B), the filter coefficients as seen in a row (i1+i4) and a row (i2+i5) illustrated on the right side of Figure 4C can be obtained.
The filter coefficients corresponding to each of the positions on the basic array pattern obtained by adding up the filter coefficients of the 5 × 5 filter according to the above procedures as described in the foregoing are all equal to 4.
For example, the rows of the filter are defined as im, i(m-1), ..., i1, i0 (one or more rows which are positioned in the central section of the filter and which are identical in arrangement of filter coefficients), i-1, ..., i-(m-1), i-m, while the columns are defined as jn, j(n-1), ..., j1, j0 (one or more columns which are positioned in the central section of the filter and which are identical in arrangement of filter coefficients), j-1, ..., j-(n-1), j-n. In
this filter, the filter coefficients of the columns which are in specified positional relationship across the central column j0 are added to obtained (jn+j-1), (j(n-1)+j-2), ..., (j1+j-n). Then, the filter coefficients of the rows which are in specified positional relationship across the central row i0 are added to obtain (im+i-1), (i(m-1)+i-2), ..., (i1+i-m). The values of the filter coefficients of (m+(the number of i0 rows)) × (n+(the number of j0 columns)) obtained by the aforementioned addition are all equal to each other. Note that when m and n are odd numbers, the values of the filter coefficients obtained by addition of (j(n+1)/2+j(n+1)/2) and addition of (i(m+1)/2+i(m+1)/2) are also all equal to the above filter coefficients.
By performing filtering with use of the filter, a fixed pattern resulting from repeating cycles of the basic array pattern of 3 × 3 pixels can be reduced. Further, since the filter coefficients weighted to be larger in the vicinity of the center are assigned, it becomes possible to prevent losing of a detail of the filtered image. [First embodiment of imaging device]
Figure 5 is a block diagram illustrating a first embodiment of an imaging device according to the present invention.
An imaging device 10-1 illustrated in Figure 5 is configured to record a taken image on a recording medium 12 such as a memory card. Operation of the entire imaging device 10-1 is controlled by a central processing unit (CPU) 14 in an integrated manner.
The CPU 14 controls each unit of the imaging device 10-1 based on input signals from operation units such as a shutter button and a power button which are not illustrated. The CPU 14 performs, for example, drive control of lenses, photographing operation control, image processing control, record/reproduction control of image data, display control of a liquid crystal monitor, and the like in response to input signals from the operation units.
Subject light which has passed through a photographic lens 16 forms an image on a light receiving surface of an image sensor 18. A subject image formed on the image sensor 18 is converted, with photoelectric conversion elements, into a signal charge corresponding to an incident light amount. An image acquisition unit 20 reads signal charges, which are accumulated in each of the photoelectric conversion elements of the image sensor 18, one by one by from the image sensor 18 as voltage signals
(image signals) and outputs them to a filtering unit 22. The image signals outputted from the image acquisition unit 20 to the filtering unit 22 are R, G, and B signals (digital signals) indicating R, G and B mosaic images corresponding to the color filter array of the image sensor 18.
Note that the image sensor 18 is not limited to a CCD (charge coupled device) image sensor but may be an image sensor of other types such as a CMOS (complementary metal-oxide semiconductor) image sensor.
The filtering unit 22 has filter coefficients of a K × L filter size imparted from a filter coefficient calculation unit 26. The filtering unit 22 applies convolution arithmetic operation to pixel values of K × L pixels extracted based on a target pixel to be filtered in the image acquired from the image acquisition unit 20 and filter coefficients of the K × L filter size so as to calculate a pixel value of the target pixel.
A fixed pattern size acquisition unit 24 acquires a fixed pattern size corresponding to the repeating cycles of the pixel configuration in the image sensor 18, and outputs size information indicating the fixed pattern size to the filter coefficient calculation unit 26. Note that as the size information indicating the fixed pattern size, a size preset in accordance with the type of the image sensor 18 is inputted. For example, in the case of an image sensor having a color filter array including the basic array pattern illustrated in Figure 1B, a 6 × 6 fixed pattern size is inputted. In the case of an image sensor having a color filter array including the 3 × 3 basic array pattern illustrated in Figure 3, a 3 × 3 fixed pattern size is inputted.
The filter coefficient calculation unit 26 calculates filter coefficients of a filter size, which is larger than the fixed pattern size, based on the size information indicating the fixed pattern size inputted from the fixed pattern size acquisition unit 24.
For example, when the fixed pattern size is M × N (M, N: integers of 2 or more) pixels, the filter coefficient calculation unit 26 calculates filter coefficients corresponding to the filter size of K × L (K, L: integers of M
Documents
Application Documents
| # |
Name |
Date |
| 1 |
Spcification.pdf |
2018-08-11 |
| 2 |
Markup copy.pdf |
2018-08-11 |
| 3 |
Form 5.pdf |
2018-08-11 |
| 4 |
Form 3.pdf |
2018-08-11 |
| 5 |
Form 13.pdf |
2018-08-11 |
| 6 |
Fomr 1, Form 2 & Abstract.pdf |
2018-08-11 |
| 7 |
Drawings.pdf |
2018-08-11 |
| 8 |
Clear Copy.pdf |
2018-08-11 |
| 9 |
ABSTRACT1.jpg |
2018-08-11 |
| 10 |
424-MUMNP-2014-FORM 3(28-4-2014).pdf |
2018-08-11 |
| 11 |
424-MUMNP-2014-FORM 26(28-3-2014).pdf |
2018-08-11 |
| 12 |
424-MUMNP-2014-ENGLISH TRANSLATION(28-3-2014).pdf |
2018-08-11 |
| 13 |
424-MUMNP-2014-CORRESPONDENCE(28-4-2014).pdf |
2018-08-11 |
| 14 |
424-MUMNP-2014-CORRESPONDENCE(28-3-2014).pdf |
2018-08-11 |