Specification
Description
IMAGE PROCESSING DEVICE AND IMAGE PROCESSING METHOD, IMAGING APPARATUS, AND COMPUTER PROGRAM
Technical Field [0001]
The present invention relates to an image processing device and an image processing method for processing an output signal of a solid-state imaging device that has filters of predetermined color coding (color filters), an imaging apparatus that uses the image processing device or the image processing method, and a computer program.
Background Art
[0002]
Cameras have a long history as means for recording visual information. Recently, instead of silver salt cameras that perform photographing using a film and a photosensitive plate, digital cameras that digitize images captured by solid-state imaging devices such as a CCD (Charge Coupled Device) and a CMOS (Complementary Metal-Oxide Semiconductor) are widely spread.
[0003]
An image sensor that uses a solid-state imaging device
is constituted by a mechanism in which respective pixels
(photodiodes) arranged two-dimensionally convert light into electric charges making use of a photoelectric effect. Usually, color filters including three colors of R (red) , green
(G), and blue (B) are provided on a light-receiving surface thereof to separate incident light into the three primary colors visible by human eyes . In the respective pixels, signal charges accumulated in association with amounts of incident light through the color filters are read. This makes it possible to obtain RGB image signals visible by human eyes.
[0004]
Moreover, the RGB image signals are color-space-converted into YUV image signals and used for display output and image recording. YUV represent colors with a luminance signal Y and two chromaticities including a color difference U of red and a color difference V of blue. YUV conversion makes it easy to perform data compression making use of a human visual sensitivity characteristic that resolution for luminance is high and resolution for color is low. For example, color space conversion from RGB to YUV is applied according to the following three expressions in the NTSC (National Television Standards committee) standard.
[0005]
Y=0.6G+0.3R+0.1B (1)
Cr(R-Y)=R-(0.3R+0.6G+0.1B) (2)
Cb(B-Y)=B-(0.3R+0.6G+0.1B) (3) [0006]
For example, there is known a 3CCD imaging apparatus that obtains high-resolution RGB signals by arranging primary color filters of R (red) , G (green) , and B (blue) in the same spatial phase with respect to three solid-state imaging devices and realizes a high image quality. However, the 3CCD imaging apparatus needs to use the three solid-state imaging devices and use a prism for separating incident light into each of colors of R, G, and B. This makes it difficult to realize a reduction in size and a reduction in cost of the apparatus. [0007]
On the other hand, there is also known a 1CCD imaging apparatus that realizes a reduction in size and a reduction in cost by appropriately arranging color filters of R, G, and B in pixel units in one solid-state imaging device. In this 1CCD imaging apparatus, color coding in which respective color filters of R, G, and B are intermittently arranged in pixel units is applied. As representative color coding, there is the Bayer arrangement that has been widely used (see Fig. 24) . [0008]
When the color coding is performed, information on components of G and B is lost in pixels in which the color filer of R is arranged, information on components of R and B is lost in pixels in which the color filter of G is arranged, and
information on components of G and R is lost in pixels in which the color filter of B is arranged. When the color space conversion is performed in later signal processing, signals of all of R, G, and B are necessary in the respective pixels. This is because, when the luminance signal Y and the color difference signals Cr and Cb are generated from the RGB signals having a spatially different phase using Expressions (1) to (2) above, this causes a false color signal. Therefore, it is necessary to perform the color space conversion after restoring color signals lost in the respective pixels according to color filter arrangement interpolation and creating RGB signals in the same spatial phase (see Fig. 25) . Such an interpolation technique is called "demosaic). [0009]
Since interpolation accuracy for the RGB signal depends on color coding, characteristics of a solid-state imaging device vary according to the color coding. In the Bayer arrangement shown in Fig. 24, color filters of R and color filters of G are alternately arranged in odd number rows and color filters of G and color filters of B are alternately arranged in even number rows. Since the Bayer arrangement adopts a structure in which a larger number of the color filters of G are arranged compared with the respective color filters of R and B, it is possible to interpolate G with accuracy higher than that for R and B.
[0010]
As it is evident from Expression (1) above, G is a main component in generating a luminance signal and resolution of the luminance signal substantially depends on resolution of G. Although resolution of an image signal is proportional to a sampling rate 1/fs of pixels (fs is a sampling frequency of the pixels) , according to the Bayer arrangement, it is possible to obtain a characteristic that resolution of G is higher than resolution of R and B by improving the accuracy of color filter arrangement interpolation of G. Human beings have a visual sensitivity characteristic that resolution for luminance is high and resolution for color is low. Thus, it can be said that the Bayer arrangement is color coding that successfully utilizes the human visual sensitivity characteristic. [0011]
It is important whether it is possible to generate RGB signals in the same spatial phase from RGB signals having spatially different phases according to the interpolation technique. If interpolation processing for an image signal is not optimum, an image having high resolution is not obtained or a false color signal is generated because of the interpolation processing. [0012]
For example, there is proposed an image processing device that, in applying interpolation processing to RGB image signals
subjected to color coding in the Bayer arrangement, calculates amounts of change in eight pixels near a pixel that should be interpolated, that is, eight pixels in total above and below, on the right and the left, and at the upper right, the lower right, the upper left, and the lower left of a pixel of attention, weights the amounts of change calculated to calculate correlation values, determines interpolation coefficients on the basis of the correlation values calculated, multiplies interpolation data by the interpolation coefficients, respectively, and, then, adds up the interpolation data multiplied by the interpolation coefficients (See, for example, Patent Document 1). [0013]
According to this image processing device, it is possible to satisfactorily judge a degree of correlation even for an edge in a direction not orthogonal to a calculation direction of the correlation values by performing the interpolation processing on the basis of the four pixels of pixel information in accordance with the interpolation coefficients. Thus, it is possible to satisfactorily reproduce an oblique edge after the interpolation processing without being blurred. Further, it is possible to satisfactorily perform interpolation in a bent line, a corner part curved at the right angle, and the like. [0014]
Incidentally, in the Bayer arrangement, limiting resolutions of resolution of G and resolution of R and B determined from an arrangement of color filters are the same in an oblique 45 degree direction. G is a main component in generating a luminance signal and substantially depends on luminance resolution. R and B substantially depend on resolution of a color difference signal. Thus, a point in generating an image having high resolution is to increase the resolution of G. Human eyes have a characteristic that it is possible to recognize a high frequency concerning luminance but it is difficult to recognize a high frequency concerning a color. Thus, it is anticipated that, in the Bayer arrangement, a balance of color resolution and luminance resolution does not sufficiently match the human visual sensitivity characteristic. [0015]
In view of such a point, the applicant proposes color coding in which luminance resolution is increased to twice as large as that of the Bayer arrangement, although of the color resolution is slightly sacrificed, by arranging G components serving as main components in generating a luminance component to surround respective R and B components and increasing G instead of reducing the number of pixels of R and B to a half compared with that of the Bayer arrangement (see, for example, the specification of Japanese patent application No.
2005-107037}. Such color coding matches the human visible sensitivity characteristic better than the color coding of the Bayer arrangement. However, more advanced interpolation processing is necessary in order to realize resolution higher than that in the case of the color coding of the Bayer arrangement. [0016]
[Patent Document]
JP-A-11-177994
Disclosure of the Invention
Problems that the Invention is to Solve
[0017]
It is an object of the invention to provide an excellent imaging processing device and image processing method, imaging apparatus, and computer program that apply, with higher accuracy, color filter arrangement interpolation to RGB images subjected to color coding to make it possible to obtain higher luminance resolution.
Means for Solving the Problems [0018]
The invention has been devised taking the problems into account. A first aspect of the invention is an image processing device that processes an image imaged by imaging means that
has color filters for color coding, characterized by including:
means for inputting an image signal;
correlation-value calculating means for calculating, for the image signal, at least a first correlation value and a second correlation value indicating a degree of correlation among pixels;
interpolation-pixel specifying means for specifying, on the basis of the first correlation value and the second correlation value, other pixels used in interpolating a desired pixel; and
interpolating means for interpolating the desired pixel from the pixel specified by the interpolation-pixel specifying means. [0019]
The image processing device according to the invention applies color filter arrangement interpolation to an image signal to which color coding is applied. The color coding uses color filters that have color components serving as main components in calculating a luminance component arranged to surround each of other color components with respect to, for example, a pixel arrangement in which respective pixels are arranged in a square lattice shape at equal intervals in the horizontal direction and the vertical direction. Alternatively, on color filters for color coding, color components serving as main components in calculating a
luminance component are arranged to surround each of other color components with respect to an oblique pixel arrangement in which respective pixels are shifted by 1/2 of a pixel pitch for each row or for each column.
[0020]
In such color coding, a logical limiting resolution of G rises by increasing G instead of reducing the number of pixels of R and B to 1/2 compared with that of the Bayer arrangement. Therefore, since resolution of G is improved to near the limiting resolution by subjecting G to interpolation processing with high accuracy, it is possible to improve luminance resolution to about twice as large as that of the Bayer arrangement, although color resolution is slightly sacrificed.
[0021]
When G is interpolated in a pixel in a space position to be interpolated (i.e., in which G serving as a main component of a luminance signal is lost), the correlation-value calculating means calculates, for pixel signals around the pixel, correlation values indicating intensity of a correlation of G among pixels at least in two or more directions. [0022]
As the correlation values among the pixels, it is possible to filter image signals in two directions orthogonal
to each other such as the horizontal direction and the vertical direction of an image and calculate a correlation value in one directional characteristic from a filter output of the filtering. The filter in this context is preferably a band-pass filter that removes a DC component. It is possible to use a high-pass filter such as a differential filter as the filter. [0023]
When a band-pass filter output in the horizontal direction is BPF(H) and a band-pass filter output in the vertical direction is BPF(V) , a correlation value S(HV) in the horizontal and vertical directions HV is represented as BPF(H)/{BPF(H)+BPF(V)}. S(HV) takes a value from 0 to 1. However, when S (HV) is close to 1, there is waviness of an image signal in the vertical direction, which indicates that a correlation among pixels is high in the vertical direction. Thus, it is preferable to specify a pixel located in the vertical direction among pixels surrounding a desired pixel as a pixel for interpolation. Conversely, when S (HV) is close to 0, there is waviness of an image signal in the vertical direction, which indicates that a correlation among pixels is high in the horizontal direction. Thus, it is preferable to specify a pixel located in the horizontal direction of the desired pixel as a pixel for interpolation. [0024]
The correlation-value calculating means can calculate the correlation value S(HV) in the horizontal and vertical directions as a first correlation value. However, in the case in which only a correlation value calculated with a single directional characteristic is used, when the correlation value S(HV) takes a value near 0.5, it is unclear whether a correlation among pixels in a direction of +45 degrees with respect to a vertical axis is high or a correlation among pixels in a direction of -45 degrees with respect to the vertical axis is high. Thus, it is impossible to specify a pixel for interpolation. [0025]
Thus, the correlation-value calculating means further calculates a second correlation value in a directional characteristic different from that of the first correlation value and compensates for an intermediate value taken by the first correlation value to improve resolution for specifying a direction in which a pixel is interpolated in the interpolation-pixel specifying means. [0026]
In the correlation-value calculating means, another pair of filters orthogonal to each other that have a directional characteristic different from that of a pair of filters orthogonal to each other used in calculating the first correlation value are disposed. The correlation-value
calculating means calculates the second correlation value on
the basis of outputs of these filters.
[0027]
When the correlation value S(HV) in the horizontal and vertical directions HV is calculated as the first correlation value as described above, it is possible to use, as the second correlation value, a correlation value S(NH/NV) in a non-horizontal and non-vertical directions NH/NV calculated on the basis of a band-pass filter output BPF(NH) in the non-horizontal direction NH orthogonal to band-pass filter outputs BPF(NH) and BPF(HN) in the non-horizontal direction NH. S(NH/NV) is represented as BPF(NH)/{BPF(NH)+BPF(NV)}. [0028]
When directions rotated by ±45 degrees with respect to the vertical direction H are set as the non-horizontal direction NH and the non-vertical direction NV and the second correlation value is calculated from outputs of band-pass filters arranged in the NH and NV directions orthogonal to each other, the first correlation value and the second correlation value can compensate for each other when one of the correlation values is a correlation value near 0.5. Thus, resolution for specifying an interpolation direction is maximized. [0029]
In other words, according to the image processing device according to the invention, the interpolation-pixel
specifying means can judge a direction of correlation over all directions (360°) by comparing plural correlation values calculated by the correlation-value calculating means and specify, in a finer granularity, a direction that in which interpolation should be performed. The interpolating means only has to apply interpolation processing to a pixel to be interpolated on the basis of information on pixels around a pixel present in the direction specified. [0030]
A second aspect of the invention is an image processing device that processes an image imaged by imaging means that have color filters for color coding, the image processing device characterized by including:
means for inputting an image signal;
correlation-value calculating means for calculating, for the image signal, at least a first correlation value and a second correlation value indicating a degree of correlation among pixels;
reliability calculating means for calculating reliability concerning the respective correlation values calculated by the correlation-value calculating means; and
interpolating means for interpolating a desired pixel in interpolation processing corresponding to the reliability. [0031]
According to the first aspect of the invention, it is
possible to calculate two or more correlation values having different directional characteristics concerning pixels values around a pixel to be interpolated, judge a direction of correlation over all directions (360°) by comparing these plural correlation values, and specify, in a finer granularity, a direction in which interpolation should be performed. However, such a method of specifying a direction to be interpolated is performed on condition that the correlation values calculated in the respective directional characteristics is reliable. [0032]
Thus, the image processing device according to the second aspect of the invention calculates reliability concerning the first correlation value and the second correlation value and interpolates a desired pixel signal using an interpolation processing method corresponding to the reliability. For example, it is possible to calculate reliability on the basis of a sum of respective correlation values calculated by the correlation value calculating means. [0033]
When the calculated correlation values are reliable, in the same manner as the image processing device according to the first aspect of the invention, the image processing device judges a direction of correlation over all directions (360 degrees) by comparing the plural correlation values with a
correlation line to thereby determine a direction in which interpolation should be performed and perform pixel interpolation with higher accuracy. As a result, it is possible to obtain a luminance signal having high resolution on the basis of a G signal having high resolution. [0034]
On the other hand, when the calculated correlation values lack reliability, the image processing device performs interpolation using an average of information on pixels around a pixel to be interpolated. In this case, accuracy of the pixel to be interpolated is controlled and it is impossible to obtain a luminance signal having high resolution. On the other hand, it is possible to improve SN according to averaging processing. [0035]
A third aspect of the invention is a computer program described in a computer-readable format to execute, on a computer, processing for an image imaged by imaging means that has color filters for color coding, the computer program characterized by causing the computer to execute:
a correlation-value calculating procedure for calculating, for an image signal imaged by the imaging means, at least a first correlation value and a second correlation value indicating a degree of correlation among pixels;
an interpolation-pixel specifying procedure for specifying, on the basis of the first correlation value and
the second correlation value, other pixels used in interpolating a desired pixel; and
an interpolating procedure for interpolating the desired pixel from the pixel specified by executing the interpolation-pixel specifying procedure. [0036]
A fourth aspect of the invention is a computer program described in a computer-readable format to execute, on a computer, processing for an image imaged by imaging means that has color filters for color coding, the computer program characterized by causing the computer to execute:
a correlation-value calculating procedure for calculating, for an image signal imaged by the imaging means, at least a first correlation value and a second correlation value indicating a degree of correlation among pixels;
a reliability calculating procedure for calculating reliability concerning the respective correlation values calculated by executing the correlation-value calculating procedure; and
an interpolating procedure for interpolating a desired pixel in interpolation processing corresponding to the reliability. [0037]
The computer programs according to the third and the fourth aspects of the invention define computer programs
described in a computer-readable format to realize predetermined processing on a computer. In other words, by installing the computer programs according to the third and the fourth aspects of the invention, cooperative actions are shown on the computer. It is possible to obtain operational effects same as those of the image processing devices according to the first and the second aspects of the invention, respectively.
Advantages of the Invention [0038]
According to the invention, it is possible to provide an excellent image processing device and image processing method, imaging apparatus, and computer program that apply, with higher accuracy, color filter arrangement interpolation to RGB images subjected to color coding to make it possible to obtain higher luminance resolution. [0039]
According to the invention, in particular, it is possible to highly accurately apply color filter arrangement interpolation of main components for calculating a luminance component to an image signal subjected to color coding that uses color filters on which color components serving as main components in calculating a luminance component are arranged to surround each of other color components in a pixel
arrangement in which respective pixels are arranged in a square lattice shape to be arranged at equal intervals in the horizontal direction and the vertical direction. [0040]
The image processing device according to the invention can judge a direction of correlation over all directions (360 degrees) with respect to a pixel to be interpolated. Thus, it is possible to perform appropriate interpolation processing on the basis of the direction judged. [0041]
Other objects and characteristic advantages of the invention will be apparent according to more detailed explanations based on embodiments of the invention described later and drawings attached hereto.
Brief Description of the Drawings [0042]
[Fig. 1] Fig. 1 is a block diagram showing an example of a structure of an imaging apparatus to which an image processing device or an image processing method according to the invention is applied.
[Fig. 2] Fig. 2 is a diagram showing a pixel arrangement in which only G is extracted in the Bayer arrangement.
[Fig. 3] Fig. 3 is a diagram showing a resolution chart.
[Fig. 4] Fig. 4 is a graph showing a frequency
characteristic of a band-pass filter.
[Fig. 5] Fig. 5 is a graph showing limiting resolutions of G resolution and RB resolution in the Bayer arrangement.
[Fig. 6] Fig. 6 is a diagram showing a color coding example subjected to interpolation processing according to a first embodiment of the invention.
[Fig. 7] Fig. 7 is a graph showing limiting resolutions of G resolution and RB resolution in the color coding example.
[Fig. 8] Fig. 8 is a diagram showing a pixel arrangement in which only G is extracted in the color coding example.
[Fig. 9] Fig. 9 is a diagram showing a relation among an H direction, a V direction, an NH direction, and an NV direction.
[Fig. 10] Fig. 10 is a flowchart showing a procedure of the interpolation processing according to the first embodiment of the invention.
[Fig. 11] Fig. 11 is a graph showing a frequency characteristic of a band-pass filter in H and V directions.
[Fig. 12] Fig. 12 is a graph showing a frequency characteristic of band-pulse filters in the NH and the NV directions.
[Fig. 13] Fig. 13 is a diagram showing a resolution chart.
[Fig. 14] Fig. 14 is a correlation line diagram showing a relation between respective points in a frequency chart and
correlation values of the points.
[Fig. 15] Fig. 15 is a diagram showing a relation between a correlation value and an interpolation value.
[Fig. 16] Fig. 16 is a block diagram showing an example of a structure of an interpolation processor that executes the interpolation processing according to the first embodiment.
[Fig. 17] Fig. 17 is a diagram showing a color coding example subjected to interpolation processing according to a second embodiment of the invention.
[Fig. 18] Fig. 18 is a diagram representing the color coding example as a square lattice.
[Fig. 19] Fig. 19 is a block diagram showing an example of a structure of an interpolation processor that executes the interpolation processing according to the second embodiment.
[Fig. 20] Fig. 20 is a diagram showing a result of interpolation processing by a pre-stage interpolation processor.
[Fig. 21] Fig. 21 is a diagram showing a result of interpolation processing by a post-stage interpolation processor.
[Fig. 22] Fig. 22 is a diagram showing spatial resolution characteristics in the Bayer arrangement, color coding 1, and color coding 2.
[Fig. 23] Fig. 23 is a block diagram showing a modification of the interpolation processor that executes the
interpolation processing according to the second embodiment.
[Fig. 24] Fig. 24 is a diagram showing color coding of the Bayer arrangement.
[Fig. 25] Fig. 25 is a diagram showing a state in which a picture subjected to color coding is subjected to demosaic processing.
[Fig. 26] Fig. 26 is a diagram showing a state in which correlation values in the H and the V directions with four pixels around an interpolation pixel X set as the centers, respectively, are calculated.
[Fig. 27] Fig. 27 is a diagram showing a state in which S_H_G4 and S_H_G13 are selected as reliable correlation values .
[Fig. 28] Fig. 28 is a diagram showing a state in which a correlation value in the H and the V directions with the interpolation pixel X set as the center are calculated.
[Fig. 29] Fig. 29 is a diagram showing a state in which correlation values in the NH and the NV directions with four pixels around the interpolation pixel X set as the centers, respectively, are calculated.
[Fig. 30] Fig. 30 is a diagram showing a state in which S_NH_G5 and S_NH_G9 are selected as reliable correlation values.
[Fig. 31] Fig. 31 is a diagram showing a state in which correlation values in the NH and the NV directions with the interpolation pixel X set as the center are calculated.
[Fig. 32] Fig. 32 is a graph in which a correlation line formed by an absolute value obtained by deducting 0.5 from complementary S_H (a correlation line (A) in Fig. 14) and a correlation line formed by an absolute value obtained by deducting 0.5 from S_NH (a correlation line (B) in Fig. 14) are plotted, respectively.
Description of Reference Numerals and Signs [0043]
11 Imaging lens
12 Imaging device
14 Camera signal processor
21 Optical system correction circuit
22 WB (White Balance) circuit
23, 23A, 23B, 23B' Interpolation processors
24 Gamma correction circuit
25 Y (luminance) signal processor
26 C (chroma) signal processor
27 Band limiting LPF (Low-Pass Filter)
28 Thinning-out processor
31 to 34, 37 to 40 Correlation-value calculating circuits
35, 41 Selection circuits
36, 42 Average-value calculating circuits
43 Comparator
44 Judging circuit
45 Interpolation circuit
Best Mode for Carrying out the Invention
[0044]
Embodiments of the invention will be hereinafter explained in detail with referenced to the drawings.
[0045]
An example of a structure of an imaging apparatus that uses an image processing device or an image processing method according to the invention is shown in Fig. 1.
[0046]
The imaging apparatus in this context includes camera modules including a solid-state imaging device serving as an imaging device, an optical system that focuses image light of a subject on an imaging surface (a light-receiving surface) of the solid-state imaging device, and a signal processor for the solid-state imaging device, camera apparatuses such as a digital still camera and a video camera mounted with the camera modules, and electronic apparatuses such as a cellular phone.
[0047]
In Fig. 1, an image light from a subject (not shown) is focused on an imaging surface of an imaging device 12 by an optical system, for example, an imaging lens 11. Asolid-state imaging device, in which a large number of pixels including
photoelectric conversion elements are two-dimensionally arranged in a matrix shape and color filters including color components serving as main components in creating a luminance component and other color components are arranged on the surface of the pixels, is used as the imaging device 12. [0048]
The solid-state imaging device that has the color filters may be any one of a charge transfer solid-state imaging device represented by a CCD (Charge Coupled Device), an X-Y addressable solid-state imaging device represented by a MOS (Metal Oxide Semiconductor), and the like. [0049]
In the following explanation, in a color filter, color components serving as main components in creating a luminance (Y) component are explained with green (G) as an example and other color components are explained with red (R) and blue (B) as examples. However, the gist of the invention is not limited to a combination of these color components. It is also possible to use, for example, white, cyan, or yellow as the color components serving as the main components in creating the Y component and use, for example, magenta, cyan, or yellow as the other color components. [0050]
Only light of the respective color components of the image light made incident on the imaging device 12 passes
through the color filters to be made incident on the respective pixels. The light made incident on the respective pixels is photoelectrically converted by a photoelectric conversion element such as a photodiode. The light is read out as analog image signals from the respective pixels, converted into digital image signals by an A/D converter 13, and, then, inputted to a camera signal processor 14, which is the image processing device according to the invention. [0051]
The camera signal processor 14 includes an optical system correction circuit 21, a White Balance (WB) circuit 22, an interpolation processor 23, a gamma correction circuit 24, a Y (luminance) signal processor 25, a C (Chroma) signal processor 26, a band limiting LPF (Low-Pass Filter) 27, and a thinning-out processor 28. [0052]
The optical system correction circuit 21 performs correction for the imaging device 12 and the optical system such as digital clamp for adjusting a black level to a digital image signal inputted to the camera signal processor 14, defect correction for correcting a defect of the imaging device 12, and shading correction for correcting a fall in an amount of ambient light of the imaging lens 11. [0053]
The WB circuit 22 applies processing for adjusting a
white balance to an image signal, which has passed the optical system correction circuit 21, to make R, G, and B the same for a white subject. The interpolation processor 23 creates pixels with different spatial phases according to interpolation. In other words, the interpolation processor 23 creates three planes (RGB signals in the same spatial position) from RGB signals with phases shifted from one another spatially. Specific interpolation processing in the interpolation processor 23 is a characteristic of the invention. Details of the interpolation processing will be described later. [0054]
The gamma correction circuit 24 applies gamma correction to the RGB signals in the same spatial position and, then, supplies the RGB signals to the Y signal processor 25 and the C signal processor 26. The gamma correction is processing for, in order to correctly represent gradations of colors of a subject, multiplying color signals of R, G, and B outputted from the WB circuit 22 by predetermined gains such that the photoelectric conversion characteristics of an entire system including the imaging device 12 and video reproducing means at a later stage thereof to be I. [0055]
The Y signal processor 25 generates a luminance (Y) signal according to Expression (1) described above. The C
signal processor 26 generates color difference signals Cr (R-Y) and Cb(B-Y) according to Expressions (2) and (3) described above. [0056]
The band limiting LPF 27 is a filter that lias a put-off frequency of 1/8 of a sampling frequency fs. The band limiting LPF 27 reduces a pass band from (1/2) fs to (1/8) fs for the color difference signals Cr and Cb. This is an output adjusted to a. TV signal format. When the color difference signals Cr and Cb are outputted without being subjected to band limitation, a frequency signal equal to or higher than 1/8 fs is outputted as a false color signal. The thinning-out circuit 28 performs thinning-out for sampling of the color difference signals Cr and Cb. [0057]
What is important is that the color (C) signal (the color difference signals Cr and Cb) needs only a band equal to or smaller than 1/4 of a band for the luminance (Y) signal. This is because human eyes have a characteristic that it is possible to recognize a high frequency for luminance but it is difficult to recognize a high frequency for a color (as described above) and a format of a TV signal is also determined accordingly. [0058]
Actually, when a difference between an output image in the case of Y: Cr: Cb=4 :4 :4 (output bands of Y and C are the same)
and an output image in the case of Y: Cr: Cb=4 :1:1 is observed, it is difficult to identify the difference in a usual subject other than a special subject, for example, point sources of light of red and blue. In other words, it is seen that, as defined in the TV signal format, sufficient resolution is obtained for the C signal with only a band 1/4 of a band width of the Y signal. [0059]
The interpolation processing in the interpolation processor 23, which is the characteristic part of the invention, will be described. [0060]
As described above, the interpolation processing is, as described above, color filter arrangement interpolation for generating a signal of a color component, which is missing in a pixel because of intermittent arrangement of color filters, according to interpolation that uses signals of pixels around the pixel, that is, pixels having different spatial phases. The interpolation processing is also called demosaic. This interpolation processing is processing extremely important for obtaining an image having high resolution. This is because, when a pixel cannot be successfully interpolated by the interpolation processing, a false signal is generated to cause deterioration in resolution and occurrence of false colors. If it is impossible to interpolate G components serving as main
components of a luminance signal with high accuracy, resolution of the luminance signal is deteriorated. [0061]
As the interpolation processing for obtaining high resolution, interpolation processing that uses correlation processing has been performed. The "correlation processing" is processing for interpolating a pixel to be interpolated using information on pixels in a direction in which correlation is high. [0062]
For example, the interpolation processing for a pixel is considered with an input image according to a resolution chart shown in Fig. 3 as an example. The resolution chart is a chart in which the center indicates a signal having a low frequency and positions farther apart from the center indicate signals having higher frequencies. In the resolution chart signals having the same frequency have various directions. It is possible to analyze what kind of processing is suitable for various signals by inputting a signal of the resolution chart to a signal processor. In the figure, when a subject is a subject of horizontal lines as at an A point on the ordinate, waviness is observed in an image signal in the vertical direction. On the other hand, since a correlation of image signals in the horizontal direction is high, the pixel is interpolated using pixels in the horizontal direction. When
a subject is a subject of vertical lines as at a C point on the abscissa, waviness is observed in the horizontal direction. On the other hand, since a correlation of image signals in the vertical direction is high, the pixel is interpolated using pixels in the vertical direction. It is possible to realize higher resolution by performing the interpolation processing using images around the pixel in a direction in which a correlation is high in this way. [0063]
The interpolation processing that uses the correlation processing is effective for processing for generating a G signal in a spatial position of a pixel in a spatial position where G is not present, for example, a pixel X in the figure in color coding like the Bayer arrangement in which color components G are arranged in a checkered pattern. It goes without saying that the interpolation processing is usually applied not only to the G signal but also to remaining R and B signals. In the explanation of this specification, attention is paid to the interpolation processing for the G signal for realizing high resolution for a luminance signal. However, it is also possible to apply interpolation processing same as that for G to R and B, or other colors such as cyan and yellow. [0064]
The interpolation processing is processing for
generating a signal of a pixel, which should be interpolated, according to interpolation that uses signals of pixels around the pixel having different spatial phases. Thus, a procedure of the interpolation processing depends on a layout of the pixels around the pixel, that is, color coding. First, interpolation processing for the Bayer arrangement will be explained. As it is evident from Fig. 2, color components G of the Bayer arrangement are arranged in a checkered pattern. The interpolation processing is applied to portions where the color components G are not arranged in the Bayer arrangement
(hereinafter referred to as "interpolation pixels").
[0065]
Filtering by a band-pass filter is applied in the horizontal and the vertical directions with the spatial position (the interpolation pixel) X as the center to calculate amplitude of an image signal observed from the horizontal and the vertical directions (i.e., waviness of image signals in the horizontal and the vertical directions). The band-pass filter is a filter that has a peak at 1/4 fs (fs is sampling frequency) and outputs a value with respect to a signal having a frequency up to a frequency near limiting resolution of 1/2 fs, as shown in Fig. 4 . When an output of the band-pass filter is large, a signal having large amplitude is present in a direction in which the output is large. Conversely, when an output of the band-pass filter is small, fluctuation in a signal
is small, or a signal of a low frequency is present, in a direction in which the output is small. For example, it is possible to use a high-pass filter such as a differential filter. However, any filter may be used as long as the filter has a band limiting characteristic for removing a DC component. [0066]
When a filter output of a band-pass filter arranged in the horizontal direction with the spatial position (the interpolation pixel) X as the center is Bpf_H and a filter output of a band-pass filter arranged in the vertical direction in the same manner is Bpf_V, these filter outputs are represented as follows, respectively. [0067]
Bpf_H=-(G3+G8)+2(G4+G9)-(G5+G10)
Bpf_V=-(Gl+G2)+2(G6+G7)-(G11+G12) [0068]
Outputs Bpf_H and Bpf_V of these two band-pass filters are observed in the respective points A to C in Fig. 3. At the A point, since there is no amplitude of an image signal in the horizontal direction, Bpf_H=0. At the B point, since there is amplitude of an image signal in an oblique 45 degree direction, Bpf_H=Bpf_V. At the C point, since there is no amplitude of an image signal in the vertical direction, Bpf_V=0. [0069]
Subsequently, respective correlation values S_H and S_V in the horizontal and the vertical directions are calculated from the following expressions by using the outputs Bpf_H and Bpf_V of the band-pass filter. [0070]
S_H=Bpf_V/(Bpf_H+Bpf_V)
S_V=1-S_H [0071]
The correlation values S_H and S_V in the horizontal and the vertical directions represent intensity of a correlation of image signals among adjacent pixels in the horizontal and the vertical directions. "Correlation" means a fluctuation ratio of a signal. As indicated by the expressions described above, the correlation values S_H and S_V are represented by a ratio of the filter outputs Bpf_H and Bpf_V of the band-pass filters located in the horizontal and the vertical directions, respectively. When the fluctuation ratio of a signal is large, a correlation is low. When the fluctuation ratio of a signal is small, a correlation is high. [0072]
In the example shown in Fig. 3, at the A point, since S_H=1 and S_V=0, it is possible to consider that a straight line component is present in the horizontal direction (no fluctuation in a signal or a signal is flat). Since a fluctuation ratio of a signal is small, a correlation is high
in the horizontal direction and there is no correlation in the vertical direction. Accuracy of interpolation is improved by interpolating the interpolation pixel X with pixels in a direction in which a correlation value is large, that is, a correlation is high. If accurate interpolation is realized for the G signal, resolution of a luminance signal including G as a main component is also high. [0073]
At the B point, S_H=S_V=0.5. This indicates that correlations in the horizontal and the vertical directions are identical, or the same change in an image occurs in both the horizontal and the vertical directions. At the C point, S_H=0 and S_V=1. This indicates that correlation in the vertical direction is high. [0074]
Subsequently, an interpolation value X of the interpolation pixel X is calculated in accordance with the following expression using G signals of pixels above and below and on the left and the right of the interpolation pixel X. [0075]
X={(G6+G7)xS_H+(G4+G9)xS_V}/2 [0076]
As it is seen from the above expression, interpolation processing is applied to the interpolation pixel X by applying a large weight in a direction in which a correlation is high.
For example, at the A point, since X=(G6+G7)/2, the interpolation pixel X is interpolated using pixels in the horizontal direction. At the B point, since X= (G6+G7+G4+G9) /4 , the interpolation pixel X is interpolated with an equal weight using pixels in the horizontal and the vertical directions. At the C point, since X=(G4+G9}/2, the interpolation pixel X is interpolated using pixels in the vertical direction. [0077]
It should be fully understood that it is possible to realize high resolution by interpolating the interpolation pixel X while applying weight to (weighting) components of pixels in a direction in which a change in amplitude of an image signal is small, or a direction in which a correlation is high, using the correlation processing as described above. [0078]
Respective limiting resolutions of G resolution and RB resolution in the Bayer arrangement are shown in Fig. 5. As it is seen from the figure, the resolution of G is 1/2 fs in the horizontal and the vertical directions and is (1/2^2) fs in the oblique 45 degree direction. In particular, the resolution of G and the resolution of RB are the same in the oblique 45 degree direction. [0079]
As it is seen from Expressions (1) to (3) for color space conversion, G is a main component in creating a luminance signal
and substantially depends on luminance resolution and RB substantially depends on resolution of a color difference signal. Therefore, an increase in resolution for G is important in generating an image having high resolution. Human eyes have a characteristic that it is possible to recognize a high frequency for luminance but it is difficult to recognize a high frequency for a color. Thus, in the Bayer arrangement, a balance between color resolution and luminance resolution does not match the human visual sensitivity characteristic. [0080]
In view of such a point, in the specification of Japanese Patent Application No. 2005-107037 already assigned to the applicant, the applicant proposes color coding in which G components serving as main components in creating a luminance component are arranged to surround respective RB components (as described above). According to this color coding, color resolution is slightly sacrificed but luminance resolution is improved to be about twice as large as that of the Bayer arrangement by increasing G instead of halving the number of pixels of R and B compared with the Bayer arrangement. [0081]
The inventors consider that the color coding in which the RB components are surrounded by the G components matches the human visual sensitivity characteristic better than the
color coding of the Bayer arrangement. In particular, in a signal processing system complying with an output format of Y:Cr:Cb=4:l:l, for example, in a signal processing system of a video camera or the like, the color coding in the band described above is more desirable. However, to obtain a luminance signal including G as a main component at high resolution, interpolation processing more advanced than that in the case of the color coding of the Bayer arrangement is required. [0082]
In the following description, two examples of color coding are explained as color coding in which components serving as main components in creasing a luminance component, for example, G, are arranged to surround respective components of R and B. Interpolation processing for the respective examples of color coding will be explained as a first embodiment and a second embodiment, respectively. [0083] [First embodiment]
A diagram showing a color coding example be subjected to interpolation processing according to a first embodiment of the invention is shown in Fig. 6 . In the color coding example shown in the figure, pixels are arranged in a square lattice shape such that the pixels are arranged at equal intervals (pixel pitches) d in the horizontal direction (a row direction
along pixel rows) and the vertical direction (a column direction along pixel columns). In a first row, pixels are arranged in repetition of RGBG with four pixels in the horizontal direction as a unit. In a second row, only pixels of G are arranged. In a third row, pixels are arranged in repetition of BGRG with four pixels in the horizontal direction as a unit. In a fourth row, only pixels of G are arranged. The following rows are arranged as repetition of these four rows as a unit. As it is seen from the figure, color components
(in this example, G) serving as main components in creating a luminance (Y) component and other color components (in this example, R and B) are arranged such that the color components G surround the color components R and B. The color components R and B are arranged at intervals of 4d in the horizontal and the vertical directions.
[0084]
Considering sampling rates in the horizontal and the vertical directions when a sampling rate among the pixels is set to d corresponding to the pixel pitches, the color components R and B are arranged in every other column (in the first embodiment, odd number columns) and every other row (in a second embodiment described later, odd number rows) such that sampling rates in the horizontal and the vertical directions are rates that are half the sampling rate for G. In other words, a sampling rate for G is d and a sampling rate for R and B is
2d and resolution in the horizontal and the vertical directions of the color components G is twice as large as that of the color components R and B. Considering a sampling rate in the oblique 45 degree direction, a sampling rate for G is d/2^2 and a sampling rate for R and B is 2d/V2. [0085]
A spatial frequency characteristic will be considered. In the horizontal and the vertical directions, since the sampling rate for G is d, it is possible to catch a G signal having a frequency up to (1/2) fs on the basis of the sampling theorem. In the oblique 45 degree direction, since the sampling rate for G is d/2^2, it is possible to catch a G signal having a frequency up to (1/V2) fs on the basis of the sampling theorem. [0086]
The spatial frequency characteristic of the color components R and B will be considered in the same manner. However, since intervals of pixel arrangement of R and B are the same and can be considered in the same way, only the component R will be described below. In the horizontal and the vertical directions, since the sampling rate for R is 2d, it is possible to catch a R signal having a frequency up to 1/4 fs on the basis of the sampling theorem. In the oblique 45 degree direction, since the sampling rate for R is d/2V2, it is possible to catch a signal having a frequency up to (1/4^2)
fs on the basis of the sampling theorem. [0087]
A spatial frequency characteristic in the color coding example shown in Fig. 6 is shown in Fig. 7 . The color components G can catch a signal having a frequency up to (1/2) fs in the horizontal and the vertical directions. The color components R and B can catch a signal having a frequency up to (1/V2) fs in the oblique 45 degree direction. The color components R and B can catch a signal having a frequency up to (1/4) fs in the horizontal and the vertical direction. The color
components G can catch a signal having a frequency up to (1/4^2) fs in the oblique 45 degree direction. In other words, as it is evident from a comparison of Figs. 5 and 7, the G limiting resolution is substantially improved from that in the Bayer arrangement by using the color coding example shown in Fig. 6. Thus, resolution of a luminance signal including G signals as main components is about twice as large as that of the Bayer arrangement. [0088]
Subsequently, correction processing for the color coding example shown in Fig. 6 will be specifically explained below. A pixel arrangement in which only the color components G in the color coding example shown in Fig. 6 are extracted is shown in Fig. 8. [0089]
As shown in the figure, G signals are present in eight pixels G in total in the horizontal, the vertical, and the oblique directions surrounding the interpolation pixel X, that is, pixels G4, G5, G6 , G8, G9, Gil, G12, and G13 . Interpolation processing for the G signal of the interpolation pixel X is performed using these G signals. In the color coding of the Bayer arrangement, the pixels G are only present in the vertical and the horizontal four directions. On the other hand, in the color coding example shown in Fig. 6, the eight pixels are present in the horizontal, the vertical, and the oblique directions. When the interpolation pixel X is observed, it is seen that the pixels G are arranged in a grid pattern. This is very important in realizing high resolution. [0090]
In the first embodiment, a correlation between the interpolation pixel X and the pixels around the interpolation pixel X is calculated not only in the horizontal and the vertical directions but also in the oblique directions. The interpolation processing is performed while it is judged which pixels should actually be used for interpolation from a relation between the correlation in the horizontal and the vertical directions and the correlation in the oblique directions. [0091]
A procedure of the interpolation processing carried out
in the interpolation processor 23 is shown in Fig. 10 in a form of a flowchart. In the following description, as shown in Fig. 9, the horizontal direction is described as an H direction, the vertical direction is described as a V direction, an axial direction rotated to the right by 45 degrees with respect to the H direction is described as an NH direction, and an axial direction rotated to the left by 45 degrees with respect to the H direction is described as an NV direction. The interpolation processing will be hereinafter explained in detail with reference to Fig. 10. [0092]
The pixel X in Fig. 8 is set as an interpolation pixel (a pixel to be interpolated) (step Sll). First, correlation values in the H and the V directions are calculated for respective pixels around this interpolation pixel X (stepS12) . Specifically, correlation values in the H and the V directions are calculated by applying filtering by a band-pass filter in the H and the V directions with the pixel G4 on the obliquely upper left of the interpolation pixel X as the center. A frequency characteristic (a filter characteristic) of the band-pass filter in the H and the V directions is shown in Fig. 11. [0093]
When an output in the horizontal direction of the band-pass filter is Bpf_H_G4 and an output in the vertical
direction of the band-pass filter is Bpf_V_G4, a filtering result represented by the following equations is obtained. [0094]
Bpf_H_G4=-G3+2G4-G5
Bpf_V_G4=-Gl+2G4-G8 [0095]
Subsequently, a correlation value S_H_G4 in the H direction for the pixel G4 is calculated in accordance with the following expression. A correlation value S_H_G4 in the H direction indicates a ratio of filter outputs Bpf_H and Bpf_V in the horizontal and the vertical directions of band-pass filters that have the same filter characteristic. [0096]
S_H_G4=Bpf_V/(Bpf_H+Bpf_V) [0097]
Concerning a correlation value S_V_G4 in the V direction, S_V_G4 = 1-S_H_G4. If only the correlation value S_H_G4 in the H direction is calculated, it is possible to easily calculate the correlation value S_V_G4. Thus, it is not particularly necessary to calculate the correlation value S_V_G4. This means that, at this point, the correlation values S_H_G4 and S_V_G4 in the H and the V directions with the pixel G4 as the center are calculated. For example, when the correlation value S_H_G4 in the H direction is 1. 0 and the correlation value S_V_G4 in the V direction is 0.0, a favorable interpolation
result is obtained if the pixel X is interpolated with the correlation value S_V_G4 in the V direction. [0098]
In the same manner as the calculation of the correlation value S_H_G4 in the H direction with the pixel G4 on the obliquely upper left of the interpolation pixel X as the center, correlation values S_H_G6, S_H_G11, and S_H_G13 in the H direction with the pixel G6 on the obliquely upper right, the pixel Gil on the obliquely lower left, and the pixel G13 on the obliquely lower right as the centers, respectively, are also calculated. [0099]
According to the processing described above, that is, the processing in step S12, the correlation values S_H_G4, S_H_G6, S_H_G11, and S_H_G13 in the H and the V directions with the four pixels G4, G6, Gil, and G13 surrounding the interpolation pixel X as the centers are calculated (see Fig. 26) . [0100]
Subsequently, two correlation values suitable for substituting the interpolation pixel X are selected out of the four correlation values S_H_G4, S_H_G6, S_H_G11, and S_H_G13 calculated. Specifically, two correlation values having a highest correlation among the four correlation values, that is, two correlation values calculated having highest
reliability, are adopted as correlation values of the
interpolation pixel X (step S13).
[0101]
As it is seen from Fig. 26, the correlation values S_H_G4 , S_H_G6, S_H_G11, and S_H_G13 calculated in step S12 are correlation values in the H and the V directions with the four pixels G4, G6, Gil, and G13 around the interpolation pixel X as the centers, respectively, and are not correlation values S_H in the H and the V directions with the desired interpolation pixel X as the center. When there is inconvenience in implementation, for example, it is impossible to form a band-pass filter with the interpolation pixel X as a center, it is impossible to directly calculate a correlation value in the interpolation pixel X. Thus, it is the purpose of step S13 to substitute, on the basis of an idea that correlation values are substantially equal in adjacent pixels, a correlation value in the H and the V directions having high reliability calculated in the adjacent pixels for the correlation value S_H in the H and the V directions of the interpolation pixel X. [0102]
It is possible to represent reliability for selecting a correlation value using an output value of the band-pass filter calculated in the course of calculating the four correlation values. For example, a correlation reliability
value Bpf_Max of the pixel G4 is calculated from the following expression.
[0103]
Bpf_Max=|Bpf_H_G4|+|Bpf_V_G4|
[0104]
This calculation of the correlation reliability value Bpf_Max is applied to four points of the other three pixels G6, Gil, and G13 . When an output of the band-pass filter for a pixel is large, it can be said that a signal having large amplitude is present around the pixel and the signal is generated by an image rather than by noise. Conversely, when an output of the band-pass filter is small, a signal is buried under noise and reliability of a correlation is low. Thus, it is difficult to trust a correlation value. In other words, when the correlation reliability value Bpf_Max calculated from a sum of absolute values of outputs of band-pass filters in the horizontal and the vertical orthogonal two directions is large, it is possible to estimate that reliability of a correlation value calculated from the outputs of these band-pass filters is high.
[0105]
The respective correlation reliability values Bpf_Max calculated in the pixels at the four points around the interpolation pixel X are compared to select pixels at two points having larger values around the interpolation pixel X.
Correlation values in the H and the V directions calculated with these pixels at the two points around the interpolation pixel X as the centers are selected as values substituting for the correlation value in the H and the V directions with the interpolation pixel X as the center. A state in which S_H_G4 and S_H_G13 are selected as reliable correlation values is shown in Fig. 27. [0106]
In the correlation reliability calculation, correlation values with a large result of |Bpf_H-Bpf_V| , that is, a large difference value of filter outputs may be selected instead of correlation values with a large correlation reliability value Bpf_Max, that is, a large total value of filter outputs. This has an object of adopting correlation values in places with a strong correlation because it can be said that the correlation values have a large difference between the output Bpf_H in the H direction and the output Bpf_V in the V direction of the band-pass filter, that is, a strong correlation in the horizontal and the vertical directions. [0107]
Subsequently, the correlation values in the higher order two places adopted are averaged to obtain one correlation value (step S14) . The correlation value averaged is treated as a correlation value in the H and the V directions with the interpolation pixel X as the center in the following processing
steps. For example, when reliability of correlation values in two places of G4 and G6 is selected because the reliability is high, correlation values in the H and the V directions with these two points as the centers are averaged and regarded as a correlation value in the H and the V directions for the interpolation pixel X (see Fig. 28) . [0108]
In this case, it is possible to select one correlation value in the horizontal and the vertical directions out of the four correlation values by selecting a correlation value of a pixel at one point having a largest correlation reliability value Bpf_Max around the interpolation pixel X. However, a satisfactory result is obtained by adopting an average of the correlation values in the higher order two places. A method of adopting correlation values in the higher order three places from the four correlation values and calculating an average of the correlation values is also conceivable. [0109]
When a band-pass filter with the interpolation pixel X as the center is formed, it is possible to directly calculate a correlation value in the interpolation pixel X rather than an average of correlation values of pixels having high reliability around the interpolation pixel X. Thus, the processing in steps S12 to S14 is simplified. [0110]
In parallel to the processing for calculation of a correlation value in the H and the V directions of the interpolation pixel X in steps S12 to S14, processing for calculation of correlation values in the NH and the NV directions of the interpolation pixel X in steps S15 to S17 is executed. The NH direction is an axial direction rotated to the right by 45 degrees with respect to the H direction. The NV direction is an axial direction rotated to the left by 45 degrees with respect to the H direction (as described above) . [0111]
First, correlation values in the NH and the NV directions are calculated for respective pixels around the interpolation pixel X (step S15) . Specifically, filtering by the band-pass filter is applied in the oblique directions with the pixel G5 above the interpolation pixel X as the center to calculate correlation values in the NH and the NV directions . A frequency characteristic (a filter characteristic) of the band-pass filters in the NH and the NV directions is shown in Fig. 12. [0112]
When an output in the NH direction of the band-pass filter is Bpf_NH_G5 and an output in the NV direction of the band-pass filter is Bpf_NV_G5, a filtering result represented by the following expressions is obtained. [0113]
Bpf_NH_G5=-Gl+2G5-G9
Bpf_NV_G5=-G2+2G5-G8 [0114]
Subsequently, a correlation value S_NH_G5 in the NH direction for the pixel G5 is calculated from the following expression. A correlation value S_NH_G5 in the NH direction represents a ratio of filter outputs Bpf_NH and Bpf_NV in the NH and the NV directions of band-pass filters having the same filter characteristic. [0115]
S_NH_G5=Bpf_NV_G5/(Bpf_NH_G5+Bpf_NV_G5) [0116]
Concerning the correlation value S_NV_G5 in the NV direction, S_NV_G5=1-S_NH_G5. Thus, if only the correlation value S_NH_G5 in the NH direction is calculated, it is possible to easily calculate the correlation value S_NV_G5. Thus, it is not particularly necessary to calculate the correlation value S_NV_G5 here. This means that the correlation values S_NH_G5 and S_NV_G5 in the NH and the NV directions with the pixel G5 as the center are calculated at this point. For example, when the correlation value S_NH_G5 in the NH direction is 1.0 and the correlation value S_V_G5 in the NV direction is 0.0, a satisfactory interpolation result is obtained by interpolating the pixel X with the correlation value S_NV_G5 in the NV direction. [0117]
As in the case of the calculation of the correlation value S_NH_G5 in the NH direction with the pixel G5 above the interpolation pixel X as the center, correlation values S_NH_G8, S_NH_G9, and S_NH_G12 in the NH direction with the pixel G8 on the left, the pixel G9 on the right, and the pixel G12 below the interpolation pixel X as the centers, respectively, are also calculated. [0118]
According to the processing described above, that is, the processing in step S15, the correlation values S_NH_G5, S_NH_G8, S_NH_G9, and S_NH_G12 in the NH direction with the four pixels G5, G8, G9, and G12 at the four points above, on the left, on the right, and below the interpolation pixel X as the centers, respectively, are calculated (see Fig. 29). [0119]
Subsequently, two correlation values substituted for the interpolation pixel X are selected out of the four correlation values S_NH_G5, S_NH_G8, S_NH_G9, and S_NH_G12 calculated. Specifically, two correlation values having higher reliability among the four correlation values are adopted as correlation values of the interpolation pixel X (step S16). [0120]
As it is seen from Fig. 29, the respective correlation values S_NH_G5, S_H_G8, S_H_G9, and S_H_G12 calculated in step S15 are correlation values in the H and the V directions with
the four pixels G5, G8, G9, and G12 around the interpolation pixel X as the centers, respectively, and are not correlation values with the desired interpolation pixel X as the center. When there is inconvenience in implementation, for example, it is impossible to form a band-pass filter with the interpolation pixel X as a center, it is impossible to directly calculate a correlation value in the interpolation pixel X. Thus, it is the purpose of step S16 to substitute, on the basis of an idea that correlation values are substantially equal in adjacent pixels, a correlation value in the NH direction having high reliability calculated in the pixels around the interpolation pixel X for the correlation value in the NH direction of the interpolation pixel X. [0121]
Reliability for selecting a correlation value is represented using an output value of a band-pass filter calculated in the process for calculating the four correlation values. For example, a correlation reliability value Bpf_Max of the pixel G5 is calculated from the following expression. The correlation reliability value Bpf_Maxis is calculated for the other pixels G8, G9, and G12 at the three points around the interpolation pixel X in the same manner. When the correlation reliability value Bpf_Max calculated from a sum of absolute values of outputs of band-pass filters in the horizontal and the vertical orthogonal two directions is large,
it is possible to estimate that reliability of a correlation value calculated from the outputs of these band-pass filters is high (as described above).
[0122]
Bpf_Max= Bpf_NH_G5 +|Bpf_NV_G5|
[0123]
The correlation values in the higher order two places adopted are averaged to obtain one correlation value (step SI7) . A state in which S_NH__G5 and S_NH_G9 are selected as reliable correlation values is shown in Fig. 30.
[0124]
The correlation value averaged is treated as a correlation value in the NH and the NV directions with the interpolation pixel X as the center in the following processing steps. For example, when reliability of correlation values in two places of G5 and G9 is selected because the reliability is high, correlation values in the H and the V directions with these two points as the centers are averaged and regarded as a correlation value in the NH and the NV directions with the interpolation pixel X as the center (see Fig. 31).
[0125]
When a band-pass filter with the interpolation pixel X is formed, it is possible to directly calculate a correlation value in the interpolation pixel X rather than an average of correlation values of pixels having high reliability around
the interpolation pixel X. The processing in steps S15 to S17
is simplified (as described above).
[0126]
According to the processing described above, a correlation value S_H in the H direction and a correlation value S_NH in the NH direction for the interpolation pixel X are calculated (see Figs. 28 and 31). [0127]
Subsequently, on the basis of the respective correlation values S_H and S_NH in the H direction and the NH direction for the interpolation pixel X, a direction of pixels around the interpolation pixel X with which the interpolation pixel X has a strong correlation is judged, that is, directional properties of a correlation are judged (step S18). [0128]
Here, the correlation values S_H and S_NH calculated in the H direction and the NH direction, respectively, for the interpolation pixel X and a degree of correlation between the interpolation pixel X and pixels surrounding the interpolation pixel X will be considered with an input image formed by a resolution chart shown in Fig. 13 as an example . The resolution chart is a chart in which the center indicates a signal having a low frequency and positions farther apart from the center indicate signals having higher frequencies . In the resolution chart, signals having the same frequency have various
directions. Therefore, it is possible to analyze what kind of processing is suitable for various signals by inputting signals in the resolution chart into a signal processor. [0129]
For example, when signals at an (a) point to an (e) point in Fig. 13 are inputted as the interpolation pixel X and the processing in step S12 to S17 described above is executed, the two correlation values S_H and S_NH at the (a) point are S_H=1. 0 (S_V=0.0) and S_NH=0.5 (S_NV=0.5). It is seen from these correlation values that, in the H and the V directions, the interpolation pixel X has a strong correlation with pixels around the interpolation pixel X in the H direction. It is seen from this that, in the NH and the NV directions, the interpolation pixel X has the same correlation in the NH and the NV directions, or has no correlation with the pixels around the interpolation pixel X in the oblique directions. [0130]
At the (b) point, S_H=0.5 (S_V=0.5) and S_NH=1.0 (S_NV=0.0). It is seen from this that, in the H and the V directions, the interpolation pixel X has no correlation with the pixels around the interpolation pixel X but, in the NH and the NV directions, the interpolation pixel X has a strong correlation with the pixels around the interpolation pixel X in the NH direction. [0131]
At the (c) point, S_H=0.0 (S_V=1.0), S_NH=0.5 (S_NV=0.5). It is seen from this that, in the H and the V directions, the interpolation pixel X has a strong correlation with the pixels around the interpolation pixel X in the V direction. In the NH and the NV directions, the interpolation pixel X has no correlation in the pixels around the interpolation pixels X. [0132]
At the (d) point, S_H=0.5 (S_V=0.5) and S_NH=0.0 (S_NV=1.0) . It is seen from this that, in the H and the V directions, the interpolation pixel X has no correlation with the pixels around the interpolation pixel X but, in the NH and the NV directions, the interpolation pixel X has a strong correlation with the pixels around the interpolation pixel X in the NV direction. [0133]
At the (e) point, S_H=1.0 (S_V=0.0) and S_NH=0.5 (S_NV=0.5) . It is seen from this that, as at the point (a), in the H and the V directions, the interpolation pixel X has a strong correlation with the pixels around the interpolation pixel in the H direction but, in the NH and the NV directions, the interpolation pixel X has no correlation with the pixels around the interpolation pixel X. [0134]
A frequency chart representing a relation of correlation values between a spatial phase of an interpolation pixel in
the case in which the resolution chart shown in Fig. 13 is set as an input image and pixels around the interpolation pixels in the H and the V directions and the NH and the NV directions is shown in Fig. 14. In the following explanation, a straight line in the figure representing the relation of correlation values between the interpolation pixel and the pixels around the interpolation pixel in the H and the V directions and the NH and the NV directions is also referred to as a "correlation line" . It is possible to obtain this correlation line by calculating at least two patterns of correlation values in different directions and plotting the correlation values of the at least two patterns on straight lines of various angles. In a correlation line diagram in Fig. 14, an alternate long and short dash line (A) is equivalent to the correlation value S_H in the H and the V directions and an alternate long and two short dashes line (B) is equivalent to the correlation value S_NH in the NH and the NV directions . Phases of the correlation lines (A) and (B) are shifted by 45 degrees. [0135]
When only the correlation value S_H in the H and the V directions is used, it is possible to specify that the interpolation pixel X has a strong correlation in the H direction or the V direction when S_H is close to 0 or 1. However, it is likely that the interpolation pixel X is present at the (a) point and the (c) point on the resolution chart when
S_H takes a value near 0.5. It is impossible to specify directional properties in which the interpolation pixel X has a strong correlation simply by referring to the correlation line (A) . On the other hand, even when S_H takes a value near 0.5, it is possible to specify, at high resolution, in which oblique directions the interpolation pixel X has a strong correlation by referring to the correlation line (B) plotted with the correlation value S_NH calculated from outputs of two band-pass filters orthogonal to each other that are rotated by 45 degrees with respect to the horizontal and the vertical axes, that is, in the NH direction rotated to the right by 45 degrees with respect to the H direction and in the NV direction rotated to the left by 45 degrees with respect to the H direction. In other words, by comparing the correlation lines of S_H and S_NH, it is possible to judge, over all directions (360 degrees) , a direction in which the interpolation pixel X has a strong correlation with pixels around the interpolation pixel X. In this embodiment, as an example, the two band-pass filters orthogonal to each other rotated by 45 degrees as a predetermined angle are used. However, an angle formed by the H and the V directions and the NH/NV directions may be other angles such as 40 degrees to 50 degrees as long as an effect of the same degree as above is realized. [0136]
When the resolution chart shown in Fig. 13 is set as an
input image, in step S18, it is possible to find what degree of correlation the interpolation pixel has with the pixels around the interpolation pixel in the H and the V directions and the NH and the NV directions, respectively, that is, directional properties in which a correlation is strong by comparing the correlation values indicated by the alternate long and short dash line (A) and the alternate long and short two dashes line (B) in the special phase in which the interpolation pixel X on the frequency chart shown in Fig. 14 is located. [0137]
When directional properties of a correlation of the interpolation pixel X is found in step S18, subsequently, it is judged whether the correlation values S_H and S_NH calculated in steps S12 to S14 and steps S15 to S17 have reliability (step S19). [0138]
As described already, in steps S12 to S14 and steps S15 to S17, correlation values in the respective directions with the interpolation pixel X as the center are not directly calculated. Correlation values having high reliability among correlation values in the H direction and the NH direction calculated with the respective pixels surrounding the interpolation pixel X as the center are averaged and substituted for a correlation value of the interpolation pixel
X. In such a case, it is particularly important to check reliability of a correlation. [0139]
A specific method of checking reliability of a correlation will be explained. The frequency chart in the case in which the resolution chart shown in Fig. 13 is set as an input image, that is, the relation of correlation values between the interpolation pixel and the pixels around the interpolation pixel in the H and the V directions and the NH and the NV directions is as shown in Fig. 14. It is possible to arrange two pairs of band-pass filters orthogonal to each other in the horizontal and the vertical directions and the directions rotated by 45 degrees with respect to the horizontal and the vertical directions, respectively, and calculate the correlation value S__H in the H direction and the correlation value S_NH in the NH direction from outputs of the respective pairs of band-pass filters. Phases of the correlation lines
(A) and (B) of S_H and S_NH are shifted by 45 degrees from each
other.
[0140]
In Fig. 32, a correlation line formed by an absolute value calculated by deducting 0.5 from S_H (the correlation line (A) in Fig. 14) and a correlation line formed by an absolute value calculated by deducting 0.5 from S_NH (the correlation line
(B) in Fig. 14) are plotted, respectively. As it is seen from
the figure, if the correlation values S_H and S_NH are ideal correlation values on the correlation line shown in Fig. 14, that is, correlation values having high reliability, a sum of the absolute value calculated by deducting 0.5 from S_H and the absolute value calculated by deducting 0.5 from S_NH is a value close to 0.5. Thus, it is possible to substitute the correlation values S_H and S_NH calculated in steps S12 to S14 and steps S15 to S17 in a conditional expression including inequalities and judge reliability of the correlation values S_H and S_NH according to whether the condition is satisfied. TH1 and TH2 are values close to 0.5 that satisfy the relation of TH1/2d in the horizontal and the vertical directions. This color coding is exactly color coding in which the color arrangement in the color coding example shown in Fig. 6 is inclined 45 degrees. [0183]
When sampling rates in the color coding example shown in Fig. 17 are considered in the horizontal and the vertical directions, a sampling rate for G is d/^2 and a sampling rate for R and B is 2^2d. In other words, R and B are arranged in every other column {in this embodiment, odd number columns) and every other row (in this embodiment, odd number rows) such that sampling rates in the horizontal and the vertical directions are rates that are 1/4 of the sampling rate for G. Therefore, resolution in the horizontal and the vertical directions of G is four times as large as that of the color components R and B. Considering a sampling rate in the oblique 45 degree direction, a sampling rate for G is d and a sampling rate for R and B is 2d. [0184]
Here, a spatial frequency characteristic will be considered. In the horizontal and the vertical directions, since the sampling rate for G is d^2, it is possible to catch
a G signal having a frequency up to (1/^2) fs according to the sampling theorem. In the oblique 45 degree direction, since the sampling rate for G is d, it is possible to catch a G signal having a frequency up to (1/4) fs according to the sampling theorem. [0185]
The color components R and B will be considered in the same manner. However, since intervals of pixel arrangement of R and B are the same, only R will be described here. Concerning a spatial frequency characteristic of R, in the horizontal and the vertical directions , since the sampling rate for R is 2^2di it is possible to catch an R signal having a frequency up to (1/4V2) fs according to the sampling theorem. In the oblique 45 degree direction, since the sampling rate for R is 2d, it is possible to catch an R signal having a frequency up to (1/2) fs according to the sampling theorem. [0186]
Incidentally, a solid-state imaging device that has the oblique pixel arrangement can obtain high resolution because pixel pitches are narrow compared with the pixel arrangement of the square lattice shape. When resolution is the same as resolution of the pixel arrangement of the square lattice shape, it is possible to arrange pixels at pixel pitches wider than the pixel pitches of the pixel arrangement of the square lattice shape. Thus, it is possible to form openings of the pixels
wide. As a result, it is possible to improve S/N.
[0187]
Concerning details of the color coding shown in Fig. 17, see, for example, paragraphs 0055 to 0072 of the specification of Japanese patent application No. 2005-107037 already assigned to the applicant.
[0188]
The second embodiment is characterized by interpolation processing for the color coding example shown in Fig. 17. The interpolation processing will be hereinafter specifically explained.
[0189]
Fig. 18 is a diagram representing the color coding example shown in Fig. 17 as a square lattice. It is seen that the arrangement of G in the color coding example shown in Fig. 6 and an arrangement of G in Fig. 18 are in a relation in which the one arrangement is rotated from the other arrangement by 45 degrees. Consequently, it could be seen that it is possible to accurately interpolate G pixels in spatial positions of R and B pixels by performing the interpolation processing according to the first embodiment in a positional relation rotated by 45 degrees with respect to the color coding example shown in Fig. 17. [0190]
An example of a structure of an interpolation processor
23B that executes the interpolation processing according to the second embodiment is shown in Fig. 19. As shown in the figure, the interpolation processor 23B according to this embodiment has a two-stage structure including a pre-stage interpolation processor 231 and a post-stage interpolation processor 232. [0191]
The pre-stage interpolation processor 231 basically performs interpolation processing in the same manner as the first embodiment. Consequently, the pre-stage interpolation processor 231 can interpolate G pixels in spatial positions of R and B pixels in the pixel arrangement of G in Fig. 18. A result of the interpolation processing by the pre-stage interpolation processor 231, that is, a result of interpolating the G pixels in the spatial positions of the R and B pixels in the pixel arrangement of G in Fig. 18 is shown in Fig. 20. AG arrangement in Fig. 20 is accurately interpolated according to this interpolation processing. When attention is paid to the arrangement of G in Fig. 20, it is seen that G is arranged in a checkered pattern. [0192]
The post-stage interpolation processor 232 basically performs interpolation processing in the same procedure as the interpolation processing for the Bayer arrangement described above to apply interpolation processing to a pixel arrangement
of G in Fig. 20. According to this interpolation processing for the Bayer arrangement, G is generated for all the pixels from G arranged in the checkered pattern. A result of the interpolation processing by the post-stage interpolation processor 232, that is, a result of interpolating G arranged in the checkered pattern to generate G for all pixels is shown in Fig. 21. [0193]
As described above, for the color coding example shown in Fig. 17, it is also possible to obtain resolution of G up to limiting resolution by combining the interpolation processing by the pre-stage interpolation processor 231, that is, the interpolation processing according to the first embodiment, and the interpolation processing by the post-stage interpolation processor 232, that is, the interpolation processing for the Bayer arrangement. [0194]
Spatial resolution characteristics in the Bayer arrangement, the color coding shown in Fig. 6, and the color coding shown in Fig. 17 are shown in Fig. 22. When resolution of the color coding shown in Fig. 6 and resolution of the color coding shown in Fig. 17 are compared, the resolution of the color coding shown in Fig. 17 is more advantageous in the horizontal and the vertical directions. In general, in photographing a subject with a camera, images including linear
components are often distributed in the horizontal and the vertical directions. From this viewpoint, it can be said that the color coding shown in Fig. 17 is more advantageous than the color coding shown in Fig. 8. [0195]
Considering that it is necessary to output an image signal in a square lattice shape as an output of the camera signal processor 14 (see Fig. 1), the color coding shown in Fig. 17 has an advantage that it is possible to set an output size twice as large as the number of pixels. It goes without saying that it is seen from Fig. 22 that the resolution is remarkably high compared with that of the Bayer arrangement. [0196]
In the camera signal processor 14, it is possible to obtain the operational effects described above by using the interpolation processor 23B in the second embodiment directly as the interpolation processor 23 . It is also possible to adopt a structure like an interpolation processor 23B' shown in Fig. 23 formed by modifying the interpolation processor 23B. [0197]
In the interpolation processor 23B' shown in Fig. 23, a changeover switch 233 is provided on an input side of the post-stage interpolation processor 232 to make it possible to selectively input one of an output signal of the pre-stage interpolation processor 231 and an output signal of the WB
circuit 22 (see Fig. 1) (an input signal of the pre-stage interpolation processor 231) to the post-stage interpolation processor 232. By adopting such a structure, it is possible to cope with solid-state imaging devices of two types, namely, the color coding shown in Fig. 17 and the Bayer arrangement, using one camera system.
Industrial Applicability [0198]
The invention has been explained in detail with reference to the specific embodiments. However, it is obvious that those skilled in the art can perform modification and substitution of the embodiments without departing from the spirit of the invention. [0199]
The invention can be suitably applied, in particular, to processing for applying color filter arrangement interpolation of main components for calculating a luminance component to an image signal to which color coding is applied. The color coding.uses color filters that have color components serving as main components in calculating a luminance component arranged to surround each of other color components with respect to a pixel arrangement in which respective pixels are arranged in a square lattice shape at equal intervals in the horizontal direction and the vertical direction. However, the
gist of the invention is not limited to specific color coding. [0200]
In the embodiments mainly explained in this specification, the two color coding examples shown in Figs. 8 and 17 are used as color coding in which each of R and B is surrounded by G serving as a main component in creating a luminance component. However, color coding to which the invention is applicable is not limited to these two color coding examples. For example, it is also possible to apply the interpolation processing according to the invention even to, for example, color coding in which, with respect to a pixel arrangement of a square lattice shape, pixels are arranged in repetition of RGGG with four pixels in the horizontal direction as a unit a the first row, only G pixels are arranged in a second row, pixels are arranged in repetition of GGBG with four pixels in the horizontal direction as a unit in a third row, only G pixels are arranged in a fourth row, and, in the following rows, pixels are arranged with the four rows as a unit. [0201]
In this specification, the explanation has been made with attention paid to the interpolation processing for a G signal for realizing high resolution for a luminance signal. However, it is also possible to apply interpolation processing same as that for G to R and B or other colors, for example, cyan and yellow.
[0202]
In short, the invention has been disclosed in a form of illustration. The contents described in this specification should not be limitedly interpreted. In order to judge the gist of the invention, description of claims should be taken into account.
Claims
1. An image processing device that processes an image
imaged by imaging means that has color filters for color coding,
the image processing device characterized by comprising:
means for inputting an image signal;
correlation-value calculating means for calculating, for the image signal, at least a first correlation value and a second correlation value indicating a degree of correlation among pixels,-
interpolation-pixel specifying means for specifying, on the basis of the first correlation value and the second correlation value, other pixels used in interpolating a desired pixel; and
interpolating means for interpolating the desired pixel from the pixel specified by the interpolation-pixel specifying means.
2. An image processing device according to claim 1,
characterized in that the correlation-value calculating means
calculates the first correlation value on the basis of outputs
of two filters orthogonal to each other and calculates the
second correlation value on the basis of outputs of two filters
orthogonal to each other that have a directional characteristic
different from that in calculating the first correlation value.
3. An image processing device according to claim 2,
characterized in that the correlation-value calculating means
uses, in the calculation of the second correlation value, a filter that has a directional characteristic shifted by a predetermined angle from a filter used for the calculation of the first correlation value.
4. An image processing device according to claim 2,
characterized in that the interpolation-pixel specifying
means specifies, by comparing a correlation between the first
correlation value and the second correlation value, the other
pixels used in interpolating the desired pixel on the basis
of correlation properties at 360 degrees around the desired
pixel to be interpolated.
5. An image processing device according to claim 1,
characterized in that color components serving as main
components in calculating a luminance component are arranged
on the color filters for color coding to surround each of other
color components with respect to a pixel arrangement in which
respective pixels are arranged in a square lattice shape at
equal intervals in a horizontal direction and a vertical
direction.
6. An image processing device according to claim 1,
characterized in that color components serving as main
components in calculating a luminance component are arranged
on the color filters for color coding to surround each of other
color components with respect to an oblique pixel arrangement
in which respective pixels are shifted by 1/2 of a pixel pitch
for each row and for each column.
7. An image processing device according to claim 1,
characterized by comprising:
pre-stage interpolation processing means for changing color coding for the oblique pixel arrangement to color coding of a Bayer arrangement according to interpolation processing; and
post-stage interpolation processing means for applying Bayer interpolation processing that uses interpolation to the color coding of the Bayer arrangement.
8. An image processing device that processes an image
imaged by imaging means that has color filters for color coding,
the image processing device characterized by comprising:
means for inputting an image signal;
correlation-value calculating means for calculating, for the image signal, at least a first correlation value and a second correlation value indicating a degree of correlation among pixels;
reliability calculating means for calculating reliability concerning the respective correlation values calculated by the correlation-value calculating means; and
interpolating means for interpolating a desired pixel in interpolation processing corresponding to the reliability.
9. An image processing device according to claim 8,
characterized in that the reliability calculating means
calculates reliability on the basis of a sum of the respective correlation values calculated by the correlation-value calculating means.
10. An image processing device according to claim 8,
characterized in that, when the reliability is outside a
predetermined range, the interpolating means performs
interpolation using an average of information on pixels around
the desired pixel to be interpolated.
11. An image processing device according to claim 8,
characterized in that, when the reliability is within a
predetermined range, the interpolating means specifies, on the
basis of the first correlation value and the second correlation
value, a pixel used for interpolation out of pixels around the
desired pixel to be interpolated and performs interpolation
using the pixel specified.
12. An image processing method of processing an image
imaged by imaging means that has color filters for color coding,
the image processing method characterized by comprising:
a correlation-value calculating step of calculating, for an image signal imaged by imaging means, at least a first correlation value and a second correlation value indicating a degree of correlation among pixels;
an interpolation-pixel specifying step of specifying, on the basis of the first correlation value and the second correlation value, other pixels used in interpolating a desired
pixel; and
an interpolating step of interpolating the desired pixel from the pixel specified in the interpolation-pixel specifying step.
13. An image processing method according to claim 12,
character^, zed in that, in the correlation-value calculating
step, the first correlation value is calculated on the basis
of outputs of two filters orthogonal to each other and the
second coefficient value is calculated on the basis of outputs
of two filters orthogonal to each other that have a directional
characteristic different from that in calculating the first
correlation value.
14. An image processing method according to claim 13,
characterized in that, in the correlation-value calculating
step, in the calculation of the second correlation value, a
filter that has a directional characteristic shifted by a
predetermined angle from a filter used for the calculation of
the first correlation value is applied.
15. An image processing method according to claim 13,
characterized in that, in the interpolation-pixel specifying
step, by comparing a correlation between the first correlation
value and the second correlation value, the other pixels used
in interpolating the desired pixel is specified on the basis
of correlation properties at 360 degrees around the desired
pixel to be interpolated.
16. An image processing method according to claim 12,
characterized in that the respective steps are carried out for
an image signal to which color coding is applied by color
filters on which color components serving as main components
in calculating a luminance component are arranged to surround
each of other color components with respect to a pixel
arrangement in which respective pixels are arranged in a square
lattice shape at equal intervals in a horizontal direction and
a vertical direction, and color filter arrangement
interpolation for the color components serving as main
components in calculating a luminance component is performed.
17. An image processing method according to claim 12,
characterized in that the respective steps are carried out for
an image signal to which color coding is applied by color
filters on which color components serving as main components
in calculating a luminance component are arranged to surround
each of other color components with respect to an oblique pixel
arrangement in which respective pixels are shifted by 1/2 of
a pixel pitch for each row and for each column, and color filter
arrangement interpolation for the color components serving as
main components in calculating a luminance component is
performed.
18. An image processing method according to claim 12,
characterized by comprising:
a pre-stage interpolation processing step of changing
color coding for the oblique pixel arrangement to color coding of a Bayer arrangement according to interpolation processing; and
a post-stage interpolation processing step of applying Bayer interpolation processing that uses interpolation to the color coding of the Bayer arrangement.
19. An image processing method of processing an image
imaged by imaging means that has color filters for color coding,
the image processing method characterized by comprising:
a correlation-value calculating step of calculating, for an image signal imaged by the imaging means, at least a first correlation value and a second correlation value indicating a degree of correlation among pixels;
a reliability calculating step of calculating reliability concerning the respective correlation values calculated in the correlation-value calculating step; and
an interpolating step of interpolating a desired pixel in interpolation processing corresponding to the reliability.
20. An image processing method according to claim 19,
characterized in that, in the reliability calculating step,
reliability is calculated on the basis of a sum of the
respective correlation values calculated in the
correlation-value calculating step.
21. An image processing method according to claim 19,
characterized in that, in the interpolating step, when the
reliability is outside a predetermined range, interpolation is performed using an average of information on pixels around the desired pixel to be interpolated.
22. An image processing method according to claim 19,
characterized in that, in the interpolating step, when the
reliability is within a predetermined range, a pixel used for
interpolation out of pixels around the desired pixel to be
interpolated is specified on the basis of the first correlation
value and the second correlation value and interpolation is
performed using the pixel specified.
23. An imaging apparatus characterized by comprising:
imaging means that have color filters for color coding;
correlation-value calculating means for calculating,
for an image signal imaged by the imaging means, at least a first correlation value and a second correlation value indicating a degree of correlation among pixels;
interpolation-pixel specifying means for specifying, on the basis of the first correlation value and the second correlation value, other pixels used in interpolating a desired pixel;
interpolating means for interpolating the desired pixel from the pixel specified by the interpolation-pixel specifying means; and
signal processing means for applying color space separation or other signal processing to image information with
the pixel interpolated by the interpolating means.
24. An imaging apparatus characterized by comprising:
imaging means that have color filters for color coding;
correlation-value calculating means for calculating, for an image signal imaged by the imaging means, at least a first correlation value and a second correlation value indicating a degree of correlation among pixels;
reliability calculating means for calculating reliability concerning the respective correlation values calculated by the correlation-value calculating means;
interpolating means for interpolating a desired pixel in interpolation processing corresponding to the reliability; and
signal processing means for applying color space separation or other signal processing to image information with the pixel interpolated by the interpolating means.
25 . A computer program described in a computer-readable format to execute, on a computer, processing for an image imaged by imaging means that has color filters for color coding, the computer program characterized by causing the computer to execute:
a correlation-value calculating procedure for calculating, for an image signal imaged by the imaging means, at least a first correlation value and a second correlation value indicating a degree of correlation among pixels;
an interpolation-pixel specifying procedure for specifying, on the basis of the first correlation value and the second correlation value, other pixels used in interpolating a desired pixel; and
an interpolating procedure for interpolating the desired pixel from the pixel specified by executing the interpolation-pixel specifying procedure.
26 . A computer program described in a computer-readable format to execute, on a computer, processing for an image imaged by imaging means that has color filters for color coding, the computer program characterized by causing the computer to execute:
a correlation-value calculating procedure for calculating, for an image signal imaged by the imaging means, at least a first correlation value and a second correlation value indicating a degree of correlation among pixels;
a reliability calculating procedure for calculating reliability concerning the respective correlation values calculated by executing the correlation-value calculating procedure,- and
an interpolating procedure for interpolating a desired pixel in interpolation processing corresponding to the reliability.
27. An image processing device characterized by comprising:
correlation value calculating means for calculating plural correlation values indicating degrees of correlation of an image for a pixel to be interpolated on a solid-state imaging device that has filters of predetermined color coding;
judging means for judging a direction in which interpolation should be performed by calculating, in different directions, at least two patterns of the plural correlation values calculated by the correlation value calculating means and comparing the correlation values of at least two patterns with correlation lines obtained by plotting the correlation values with respect to straight lines of various angles; and
interpolating means for applying interpolation processing to the pixel to be interpolated on the basis of information on pixels around the pixel to be interpolated present in the direction judged by the judging means.
28. An image processing device according to claim 27,
characterized in that the correlation value calculating means
calculates the plural correlation values with pixels around
the pixel to be interpolated as centers.
29. An image processing device according to claim 27,
characterized in that
the correlation value calculating means calculates an average of higher order plural correlation values with high correlation among the plural correlation values, and
the judging means judges, on the basis of the average
of the higher order plural correlation values, a direction in which interpolation should be performed.
30. An image processing device according to claim 27,
characterized in that the correlation value calculating means
calculates the plural correlation values from a ratio of a first
filter output obtained by applying filtering processing by a
first band-pass filter to a first direction and a second filter
output obtained by applying filtering processing by a second
band-pass filter, which has a same filter characteristic as
the first band-pass filter, to a second direction orthogonal
to the first direction.
31. An image processing device according to claim 27,
characterized in that the different directions are directions
shifted from each other by 45°.
32. An image processing device according to claim 31,
characterized in that the judging means considers, when two
correlation values for the different directions are on the
correlation line, that reliability of correlation is high and
considers, when the two correlation values are not on the
correlation line, that reliability of correlation is low.
33. An image processing device according to claim 32,
characterized in that the interpolating means performs, when
the judging means judges that reliability of correlation is
high, interpolation using information on pixels around the
pixel to be interpolated present in the direction judged by
the judging means and performs, when the judging means judges that reliability of correlation is low, interpolation using an average of the information on the pixels around the pixel to be interpolated.
34. An image processing device according to claim 27,
characterized in that, in the predetermined color coding, with
respect to a pixel arrangement in which pixels are arranged
in a square lattice shape at equal intervals in a horizontal
direction and a vertical direction, color components serving
as main components in creating a luminance component are
arranged to surround respective other color components.
35. An image processing device according to claim 27,
characterized in that, in the predetermined color coding, with
respect to an oblique pixel arrangement in which pixels are
shifted by 1/2 of pixel pitches for each row and for each column,
color components serving as main components in creating a
luminance component are arranged to surround respective other
color components.
36. An image processing device according to claim 35,
characterized by further comprising:
pre-stage interpolation processing means for changing color coding for the oblique pixel arrangement to color coding of a Bayer arrangement according to interpolation processing; and
post-stage interpolation processing means for applying
Bayer interpolation processing that uses interpolation to the color coding of the Bayer arrangement, and in that
the pre-stage interpolation processing means includes:
correlation value calculating means for calculating plural correlation values indicating degrees of correlation of an image for the pixel to be interpolated;
judging means for judging a direction in which interpolation should be performed by calculating, in different directions, at least two patterns of the plural correlation values calculated by the correlation value calculating means, and comparing the correlation values of at least two patterns with correlation lines obtained by plotting the correlation values with respect to straight lines of various angles; and
interpolating means for applying interpolation processing to the pixel to be .interpolated on the basis of information on pixels around the pixel to be interpolated present in the direction judged by the judging means.
37. An image processing method characterized by comprising:
a first step of calculating plural correlation values indicating degrees of correlation of an image for a pixel to be interpolated on a solid-state imaging device that has filters of predetermined color coding;
a second step of judging a direction in which interpolation should be performed by calculating, in different
directions, at least two patterns of the plural correlation values calculated in the first step and comparing the correlation values of at least two patterns with correlation lines obtained by plotting the correlation values with respect to straight lines of various angles; and
a third step of applying interpolation processing to the pixel to be interpolated on the basis of information on pixels around the pixel to be interpolated present in the direction judged in the second step.
38. An imaging apparatus characterized by comprising:
a solid-state imaging device that has a filter of predetermined color coding; and
an image processing device that processes an output signal of the solid-state imaging device, and in that
the image processing device includes:
correlation value calculating means for calculating plural correlation values indicating degrees of correlation of an image for a pixel to be interpolated on the solid-state imaging device;
judging means for judging a direction in which interpolation should be performed by calculating, in different directions, at least two patterns of the plural correlation values calculated by the correlation value calculating means and comparing the correlation values of at least two patterns with correlation lines obtained by plotting the correlation
values with respect to straight lines of various angles; and interpolation processing means for applying interpolation processing to the pixel to be interpolated on the basis of information on pixels around the pixel to be interpolated present in the direction judged by the judging means.