DESCRIPTION
Title of Invention: IMAGE PROCESSING DEVICE, IMAGE
SP262426
PROCESSING METHOD, AND PRORGRAM
Technical Field
[0001]
The present invention relates to an image processi
device, an image processing method, and a program. The
present invention relates particularly to an image
g
processing device, a method and a program for performing a
signal process on the output of an imaging device having an
RGBW arrangement.
Background Art
[0002]
For example, an imaging device (image sensor) used in
an imaging device has a configuration in which a color
filter permitting a specific wavelength component of light
(R, G, B) to penetrate a pixel unit is attached to the
surface of the device. For example, a filter having an RGB
arrangement, as shown in Fig. 1(a), is used. In a process
of generating a color image using an output signal of the
imaging device, the process of restoring a necessary color
component using a set of a plurality of pixels. The color
arrangement of the color filter comes in many types, and a
Bayer arrangement, configured by 3 kinds of filters, which
SP262426
permit only a specific wavelength light, red (R), green (G),
and blue (B) shown in Fig. 1(a), to penetrate, is frequently
used.
[0003]
In recent years, the pixels of the imaging device
(image sensor) have become increasingly smaller in size, and
accordingly a problem has occurred in that the amount of
light incident to each pixel has decreased, and thus the S/N
ratio has deteriorated. To solve this problem, as shown in
Fig. l(b), in addition to a filter permitting, for example,
only a specific wavelength light of RGB, or the like to
penetrate, an image sensor (imaging device) having a white
filter (W: White) which additionally permits light in a
visible light range to penetrate widely, has been proposed.
In Fig. l(b), one example of a filter having an RGBW
arrangement is illustrated. A W pixel in the RGBW
arrangement shown in Fig. 1(b) is a filter which permits
light in a visible light range to penetrate widely.
[0004]
Inthis way, an imaging device equipped with a color
filter having white (W: White) pixels is disclosed in, for
example, PTL 1 (U.S Unexamined Patent Application
Publication No. 2007/24879) and PTL 2 (U,S Unexamined Patent
Application Publication No. 2007/0024934).
As shown in Fig. l(b), an imaging device (image sensor)
SP262426
having the color filter having the white (W: White) pixel is
used, and thus the penetration light rate of the filter is
increased and high sensitivity may be accomplished.
[0005]
However, the problems with an RGBW type device are as
follows.
Both the RGB arrangement shown in Fig. 1(a) and the
RGBW arrangement as shown in Fig. l(b) are single elements
in which a filter of R, G, B or W is arranged in a mosaic
state, that is, a single plate type image sensor. Therefore,
when generating a color image, it is necessary to perform a
de-mosaicing process as a color coding that generates an RGB
pixel value corresponding to each pixel.
[0006]
In the RGBW arrangement shown in Fig. l(b), the
sampling rates of R, G, and B components are decreased,
compared with the RGB arrangement shown in Fig. l(a). As a
result, in a case where obtainment data obtained by an
element with the RGBW arrangement shown in Fig. l(b) is used
when performing a process of generating the color image,
there is a problem in that false color may easily occur,
compared with the RGB arrangement shown in Fig. 1(a).
Furthermore, because a wavelength component of the white (W)
includes all of the wavelength components of R, G and B,
there is a problem in that the concentration rate of an RGB
SP262426
wavelength component is decreased and resolution is
decreased compared with that of a single color component
when an optical lens, having a large amount of chromatic
aberration, is used. As the pixels are smaller in size,
this problem is more remarkable.
[0007]
As a technique to prevent a deterioration in resolution
resulting from chromatic aberration in the optical lens, it
is effective to suppress an occurrence of chromatic
aberration by combining lenses which are different in
refractive index, but in this case, there is another problem
in that costs are increased because an increase in the
number of optical lenses. Furthermore, in this
configuration, a problem also occurs in that the abovedescribed
problem of false color resulting from a decrease
in the sampling rate of the RGB component is more remarkable.
[0008]
Furthermore, because each pixel-of a single plate type
image sensor only has information for a single color
component, a de-mosaicing process of obtaining the RGB pixel
values corresponding to all of the pixels is performed to
obtain the color image from the R, G, B, and W signals which
are discretely obtained. When performing the de-mosaic
process, an interpolation process is performed based on the
assumption that a color rate is almost uniformly retained
- 5 -
SP262426
and there is a strong color correlation in the local region.
Specifically, when calculating the pixel value of a specific
color of a pixel, a method of interpolating using the pixel
values of the neighboring pixels is widely used. This
method is disclosed in, for example, PTL 3 (Japanese
Unexamined Patent Application Publication No. 2009-17544).
However, the above-described assumption that the color rate
is almost uniformly retained and color correlation exists in
the local region is not true for the neighborhood of the
edge. As a result, there is a problem in that the false
color easily occurs in the neighborhood of the edge.
Citation List
Patent Literature
[0009]
PTL 1: US Unexamined Patent Application Publication No.
2007/0024879
PTL 2: US Unexamined Patent Application Publication No.
2007/0024934
PTL 3: Japanese Unexamined Patent Application
Publication No. 2009-17544
Summary of Invention
Technical Problem
[0010]
An object of the present invention is to provide an
image processing device for, an image processing method of,
- 6
SP262426
and a program for accomplishing generation of a high-quality
color image with little false colors in a color image
generation process to which obtainment data obtained by an
imaging device (image sensor) having, for example, an RGBW
type color filter including white (W: White) is applied.
[0011]
According to a first aspect of the present invention,
there is provided an image processing device, including a
data conversion process unit performing pixel conversion by
interpreting a two-dimensional pixel arrangement signal in
which pixel which is main components of a brightness signal
are in a checkered state, and pixels of a plurality of
colors which are color information components are arranged
in the remaining region,
the data conversion process unit including a parameter
calculation process unit calculating a parameter which are
applied to a pixel conversion process, by interpreting the
two-dimensional pixel arrangement signal, and
the parameter calculation process unit performing a
calculation process of calculating the parameter using at
least any of a first correlation information and a second
correlation information, the first correlation information
on a correlation between the pixel which is the main
component of the brightness signal included in the twodimensional
pixel arrangement signal and the color
SP262426
information component of the pixel after conversion, and the
second correlation information on a correlation between one
selection color information component selected from among
the color information components and the color information
component of the pixel after the conversion, in the
calculation of calculating the parameter which are applied
to the pixel conversion process.
[0012]
Furthermore, in one embodiment according to the image
processing- device according to the present invention, the
parameter calculation unit uses the correlation information
on a correlation between the color information component of
which a distribution rate is the highest among the color
information components and the color information component
of the pixel after the conversion, as the second correlation
information.
[0013]
Furthermore, in one embodiment of the image processing
device according to the present invention, the data
conversion process unit includes an edge detection unit
generating edge information by interpreting the twodimensional
pixel arrangement signal, and the parameter
calculation unit performs the calculation process of
calculating the parameter by selectively using any of the
first correlation information and the second correlation
SP262426
information, according to an edge direction which the edge
detection unit detects.
[0014]
Furthermore, in one embodiment of the image processing
device according to the present invention, a color of the
main component of the brightness signal is white, and a
color of the color information component of which the
distribution rate is the highest is green, and the parameter
calculation unit performs the calculation process of
calculating the parameter using any of the first correlation
information on a correlation between white and the color
information component of the pixel after the conversion and
the second correlation information on a correlation between
green and the color information component of the pixel after
the conversion, according to the edge direction which the
edge detection unit detects, in the calculation process of
calculating the parameter which is applied to the pixel
conversion process.
[0015]
Furthermore, in one embodiment of the image processing
device according to the present invention, the data
conversion process unit includes a texture detection unit
generating texture information by interpreting the twodimensional
pixel arrangement signal, and a blend process
unit inputting the parameter which the parameter calculation
SP262426
unit calculates and the edge information and the texture
information, and determining a conversion pixel value by
performing a blend process by changing a blend ratio of the
parameter which the parameter calculation unit, according to
the edge information and the texture information
corresponding to the conversion pixel.
[0016]
Furthermore, in one embodiment of the image processing
device according to the present invention, the edge
detection unit generates the edge information including edge
direction and strength information corresponding to each
pixel by interpreting an RGB arrangement signal which is
generated from an RGB pixel and a white (W) pixel, the
texture detection unit generates texture information
indicating texture extent corresponding to each pixel by
interpreting the RGBW arrangement signal, the parameter
calculation unit is the parameter calculation unit which
calculates a parameter for converting the RGBW arrangement
to the RGB arrangement, and generates the parameter
corresponding to an interpolation pixel value calculated by
an interpolation process of changing an application pixel
position according to the edge direction corresponding to
the conversion pixel, and the blend process unit inputs the
parameter which the parameter calculation unit calculates
and the edge information and the texture information and
- 10 -
SP262426
performs the process of determining the conversion pixel
value by performing the blend process by changing the blend
ratio of the parameter which the parameter calculation unit
calculates, according to the edge information and the
texture information corresponding to the conversion pixel.
[0017]
Furthermore, in one embodiment of the image processing
device according to the present invention, the parameter
calculation unit has a configuration which generates the
parameter by the interpolation process of defining the pixel
position which is applied to the interpolation process as
the pixel position which is along the edge direction.
[0018]
Furthermore, in one embodiment of the image processing
device according to the present invention, the parameter
calculation unit has a configuration which generates the
parameter by the interpolation process of using any of a
correlation in a local region between the W pixel
configuring the RGBW arrangement and the other RGB pixel or
a correlation in the local region between the G pixel
configuring the RGBW arrangement and the other RGB pixel.
[0019]
Furthermore, in one embodiment of the image processing
device according to the present invention, the image
processing device further includes a temporary pixel setting
- 11 -
SP262426
unit setting the pixel value of the pixel of any of RGB to a
W pixel position by the interpolation process of using the
correlation in the local region between the W pixel
configuring the RGBW arrangement and the other RGB pixel,
and the parameter calculation unit has a configuration which
generates the parameter by the interpolation process of
applying temporary pixel setting data.
[0020]
Furthermore, in one embodiment of the image processing
device according to the present invention, the parameter
calculation unit selects which one of the first correlation
information on a correlation between the W pixel configuring
the RGBW arrangement and the color information component of
the pixel after the conversion and the second correlation
information on a correlation between the G pixel configuring
the RGBW arrangement and the color information component of
the pixel after the conversion is used, according to 4 kinds
of the edge directions, longitudinal, traverse, left
gradient upward, and right gradient upward, which the edge
detection unit detects.
[0021]
Furthermore, in one embodiment of the image processing
device according to the present invention, the parameter
calculation unit generates a plurality of parameters by
setting a reference pixel position to the pixel position
- 12 -
SP262426
along the edge direction and using the first correlation
information and the second correlation information, and the
blend process unit performs the blend process of changing
the blend ratios of the plurality of parameters according to
the comparison result, by performing a strength comparison
of the 4 kinds of the edge directions, longitudinal,
traverse, left gradient upward, and right gradient upward.
[0022]
Furthermore, in one embodiment of the image processing
device according to the present invention, the blend process
unit performs a blend process of calculating an edge
direction ratio (ratioFlat) of the longitudinal and traverse
direction edge and the gradient direction edge corresponding
to the conversion pixel, additionally calculating
longitudinal and traverse direction edge' direction weight
(weightHV) indicating that the greater the value is, the
stronger the longitudinal and traverse direction edge is
than the gradient direction edge, and the smaller the value
is, the stronger the gradient direction edge is than the
longitudinal and traverse direction edge, based on the edge
direction ratio (ratioFlat), increasing a blend ratio of the
parameter calculated by setting the edge direction to the
longitudinal or traverse direction in a case where the
longitudinal and traverse direction edge corresponding to
the conversion pixel is stronger than the gradient direction
- 13 -
SP262426
edge, and increasing the blend ratio of the parameter
calculated by setting the edge direction to the gradient
direction edge in a case where the longitudinal and traverse
direction edge corresponding to the conversion pixel is
weaker than the gradient direction edge.
[0023]
Furthermore, in one embodiment of the image processing
device according to the present invention, the texture
detection unit calculates a flatness weight (weightFlat)
corresponding to each pixel, indicating a high value for a
pixel area of which textures are small in number and of
which the flatness is high and a low value for a pixel area
of which textures are large in number and of which the
flatness is low, as the texture information, and the
parameter calculation unit calculates a contrast enhancement
process application parameter for performing a contrast
enhancement process on the interpolation pixel value, Bind a
contrast enhancement process non-application parameter for
not performing the contrast enhancement process on the
interpolation pixel value, and the blend process unit
performs the blend process of setting the blend ratio of the
contrast enhancement process non-application parameter to be
high for the pixel of which the flatness weight is great,
and setting the blend ratio of the contrast enhancement
process application parameter to be high for the pixel of
- 14 -
SP262426
which the flatness weight is small.
[0024]
Furthermore, in one embodiment of the image processing
device according to the present invention, the edge
detection unit has a configuration which generates the edge
information corresponding to each pixel, by an
interpretation process using only the white (W) pixel of the
RGBW arrangement signal, and generates the edge information
including the edge direction and the strength information
corresponding to each pixel by calculating a signal value
gradient of the W pixel in the neighborhood of the process
object pixel.
[0025]
Furthermore, in one embodiment of the image processing
device according to the present invention, the texture
detection unit generates texture information indicating a
texture extent corresponding to each pixel, by the
interpretation process using only the white (W) pixel of the
RGBW arrangement signal.
[00261
Furthermore, according to a second aspect of the
invention, there is provided an image processing method of
performing an image signal process in the image processing
device, performing an edge detection step of enabling an
edge detection unit to generate an edge information
- 15 -
S2262426
including an edge direction and a strength information
corresponding to each pixel by interpreting an RGBW
arrangement signal which is generated from an RGB pixel and
a white (W) pixel,
a texture detection step of enabling a texture
detection unit to generate texture information indicating
texture extent corresponding to each pixel by interpreting
the RGBW arrangement signal,
a parameter calculation step of enabling a parameter
calculation unit to be the parameter calculation step of
calculating the parameter for converting the RGBW
arrangement to the RGB arrangement, and to generate the
parameter corresponding to an interpolation pixel value
calculated by an interpolation process of changing an
application pixel position according to the edge direction
corresponding to the conversion pixel, and
a blend process step of enabling a blend process unit
to input the parameter which the parameter calculation unit
calculates and the edge information and the texture
information and determine the conversion pixel value by
performing the blend process by changing the blend ratio of
the parameter which the parameter calculation unit
calculates, according to the edge information and the
texture information corresponding to the conversion pixel,
wherein the parameter calculation step performs the
- 16 -
SP262426
calculation process of calculating the parameter using at
least any of a first correlation information and a second
correlation information , the first correlation information
on a correlation between the pixel which is the main
component of the brightness signal included in the twodimensional
pixel arrangement signal and the color
information component of the pixel after conversion , and the
second correlation information on a correlation between the
color information component of which a distribution rate is
the highest among the color information components and the
color information component of the pixel after the
conversion , in the calculation of calculating the parameter
which are applied to the pixel conversion process.
[0027]
Furthermore, according to a third aspect of the present
invention , there is provided a program for causing the
performing of an image signal process in the image
processing device, the image signal process including an
edge detection step of enabling an edge detection unit to
generate an edge information including an edge direction and
a strength information corresponding to each pixel by
interpreting an RGBW arrangement signal which is generated
from an RGB pixel and a white (W) pixel,
a texture detection step of enabling a texture
detection unit to generate texture information indicating
- 17 -
SP262426
texture extent corresponding to each pixel by interpreting
the RGBW arrangement signal,
a parameter calculation step of enabling a parameter
calculation unit to be the parameter calculation step of
calculating the parameter for converting the RGBW
arrangement to the RGB arrangement, and to generate the
parameter corresponding to an interpolation pixel value
calculated by an interpolation process of changing an
application pixel position according to the edge direction
corresponding to the conversion pixel, and
a blend process step of enabling a blend process unit
to input the parameter which the parameter calculation unit
calculates and the edge information and the texture
information and determine the conversion pixel value by
performing the blend process by changing the blend ratio of
the parameter which the parameter calculation unit
calculates, according to the edge information and the
texture information corresponding to the conversion pixel,
wherein the parameter calculation step is caused to
perform the calculation process of calculating the parameter
using at least any of a first correlation information and a
second correlation information, the first correlation
information on a correlation between the pixel which is the
main component of the brightness signal included in the twodimensional
pixel arrangement signal and the color
18 -
SP262426
information component of the pixel after conversion, and the
second correlation information on a correlation between the
color information component of which the distribution rate
is the highest among the color information components and
the color information component of the pixel after the
conversion, in the calculation of calculating the parameter
which are applied to the pixel conversion process.
[0028]
Furthermore, the program according to the present
invention is, for example, a program which may be provided
by a memory media provided in a computer-readable format and
over the communication media, for an image processing device
and a computer system which are capable of running various
program codes. This program is provided in a computerreadable
form, and thus the process according to the program
is accomplished on the image processing device and the
computer system.
[0029]
Other object, characteristics and advantages of the
present invention will be clarified by embodiments of the
present invention, described below, and the description of
the present invention based on the accompanying drawings.
Furthermore, the system in the present specification is a
logical set configuration of a plurality of apparatuses, and
the apparatus in each configuration is not limited to one
- 19 -
SP262426
within the same housing.
Advantageous Effects of Invention
[0030]
According to the configuration of one embodiment
according to the present invention, the RGB arrangement data
for generating a high-quality color image with few false
colors by inputting obtainment data obtained by the imaging
device (image sensor) having, for example, the RGBW-type
color filter, including white (W: White) may be generated.
Specifically, the edge detection unit interprets the output
signal of the imaging device with the RGBW arrangement, and
thus obtains the edge information corresponding to each
pixel, and the texture detection unit generates the texture
information. Furthermore, the parameter calculation unit
generates the parameter corresponding to, the interpolation
pixel value, by performing the interpolation process of
changing the application pixel position corresponding to the
edge direction corresponding to the conversion pixel. The
blend process unit inputs the parameter which the parameter
calculation unit generates, the edge information and the
texture information, and determines a conversion pixel value,
by performing a blend process by changing the blend ratio of
the parameter which the parameter calculation unit
calculates, according to the edge information and the
texture information corresponding to the conversion pixel.
- 20 -
SP262426
By this process, the RGB arrangement data with which the
high-quality color image with few false colors can be
generated may be generated.
Brief Description of Drawings
[0031]
[Fig. 1] is a view explaining examples of a Bayer
arrangement as a color arrangement used in a general color
filter and an RGBW arrangement applied to the present
invention.
[Fig. 2] Fig. 2 is a view explaining a re-mosaicing
process performed as a conversion process of converting an
RGBW arrangement to an RGB arrangement, which is a process
in one embodiment according to the present invention.
[Fig. 3] Fig. 3 is a view explaining an individual
process of the re-mosaic process performed as the conversion
process of converting the RGBW arrangement to the RGB
arrangement, which is a process of the present invention.
[Fig. 4] Fig. 4 is a view explain ng an individual
process of the re-mosaic process performed as the conversion
proce--sof converting the RGBW arrangement to the RGB
arrangement, which is the process of the present invention.
[Fig. 5] Fig. 5 is a view explaining a configuration
example of an imaging apparatus relating to one example of
the image processing device of the present invention.
[Fig. 6] Fig. 6 is a view explaining a configuration
- 21 -
SP262426
and a process of a data conversion process unit.
[Fig. 7] Fig. 7 is a view explaining a process which a
noise removal unit 201 shown in Fig. 6 performs.
[Fig. 8] Fig. 8 is a view explaining an edge detection
process which an edge detection unit 209 shown in Fig. 6
performs.
[Fig. 9] Fig. 9 is a view explaining the edge detection
process which the edge detection unit 209 as shown in Fig. 6
performs.
[Fig. 10] Fig. 10 is a view explaining the edge
detection process which the edge detection unit 209 as shown
in Fig. 6 performs.
[Fig. 11] Fig. 11 is a view explaining the edge
detection process which the edge detection unit 209 as shown
in Fig. 6 performs.
[Fig. 12] Fig. 12 is a view explaining a texture
detection process which a texture detection unit 210 as
shown in Fig. 6 performs.
[Fig. 13] Fig. 13 is a view explaining the texture
detection process which the texture detection unit 210 as
shown in Fig. 6 performs.
[Fig. 14] Fig. 14 is a view explaining the texture
detection process which the texture detection unit 210 as
shown in Fig. 6 performs.
[Fig. 15] Fig. 15 is a view explaining a process which
- 22 -
SP262426
a first pixel interpolation parameter calculation unit
(GonW) 202 as shown in Fig. 6 performs.
[Fig. 16] Fig. 16 is a view explaining the process
which the first pixel interpolation parameter calculation
unit (GonW) 202 as shown in Fig. 6 performs.
[Fig. 17] Fig. 17 is a view explaining the process
which the first pixel interpolation parameter calculation
unit (GonW) 202 as shown in Fig. 6 performs.
[Fig. 18] Fig. 18 is a view explaining the process
which the first pixel interpolation parameter calculation
unit (GonW) 202 as shown in Fig. 6 performs.
[Fig. 19] Fig. 19 is a view explaining a process which
a first temporary pixel setting unit (RBonWaroundG) 203 as
shown in Fig. 6 performs.
[Fig. 20] Fig. 20 is a view explaining the process
which the first temporary pixel setting unit (RBonWaroundG)
203 as shown in Fig. 6 performs.
[Fig. 21] Fig. 21 is a view explaining the process
which the first temporary pixel setting unit (RBonWaroundG)
203 as shown in Fig. 6 performs.
[Fig. 22] Fig. 22 is a view explaining a process which
a second pixel interpolation parameter calculation unit
(RBonGofHV) 204 as shown in Fig. 6 performs.
[Fig. 23] Fig. 23 is a view explaining the process
which the second pixel interpolation parameter calculation
- 23 -
SP262426
unit (RBonGofHV) 204 as shown in Fig. 6 performs.
[Fig. 24] Fig. 24 is a view explaining the process
which the second pixel interpolation parameter calculation
unit (RBonGofHV) 204 as shown in Fig. 6 performs.
[Fig. 25] Fig. 25 is a view explaining a process which
a third pixel interpolation parameter calculation unit
(RBonGofAD) 205 as shown in Fig. 6 performs..
[Fig. 26] Fig. 26 is a view explaining the process
which the third pixel interpolation parameter calculation
unit (RBonGofAD) 205 as shown in Fig. 6 performs.
[Fig. 27] Fig. 27 is a view explaining a process which
a second temporary pixel setting unit (RBonWaroundRB) 206 as
shown in Fig. 6 performs.
[Fig. 28] Fig. 28 is a view explaining a process which
a fourth pixel interpolation parameter calculation unit
(RBonRBofHV) 207 as shown in Fig. 6 performs.
[Fig. 29] Fig. 29 is a view explaining a process which
a fifth pixel interpolation parameter calculation unit
(RBonRBofAD) 208 as shown in Fig. 6 performs.
[-Fig. 30] Fig. 30 is a view explaining a process which
a blend process unit 211 as shown in Fig. 6 performs.
[Fig. 31] Fig. 31 is a view explaining the process
which the blend process unit 211 as shown in Fig. 6 performs.
[Fig. 32] Fig. 32 is a view explaining the process
which the blend process unit 211 as shown in Fig. 6 performs.
- 24 -
SP262426
[Fig. 33] Fig. 33 is a view illustrating a flow chart
explaining a sequence of processes which a data conversion
process unit 200 as shown in Fig. 6 performs.
Description of Embodiments
[0032]
Referring to the drawings, an image processing device,
an image processing method, and a program according to the
present invention are described below. The order of the
description is as follows.
1. Outline of Processes according to the Present Invention
2. Configuration Examples and Process Examples of an
Imaging Device and an Image Processing Device
3. Description of a Process of a Data Conversion Unit
3-1. Process of Noise Removal Unit
3-2. Process of Edge Detection Unit
3-3. Process by a Texture Detection Unit
3-4. Process by a Pixel Interpolation Parameter Calculation
Unit
3-5. -Process by a Blend Process unit
4. Sequence of Mosaic Processes which a Data Conversion
Process Unit of the Image Processing Device Performs
[00331
[1. Outline of Processes according to the Present Invention]
First, referring to Fig. 2, the outline of the
- 25 -
SP262426
processes which an image processing device according to the
present invention, such as an imaging device, performs, is
described. The image processing device according to the
present invention performs the process of obtainment data
obtained by an imaging device having an RGBW-type color
filter permitting all of each wavelength light of RGB,
including white (W: White), to penetrate in addition to an
RGB filter selectively to permit wavelength light of each
color of RGB to penetrate. Specifically, a pixel conversion
is performed by analyzing a two-dimensional pixel
arrangement signal in which pixels which are main components
of a brightness signal are arranged in a checkered state and
pixels with a plurality of colors which are color
information components are arranged in the remaining region.
Furthermore, a color which is the main component of the
brightness signal is white or green.
[0034]
The image processing device according to the present
invention performs the parameter calculation process, which
is applied to the process in which the obtainment data
obtained by the imaging device (image sensor) having, for
example, the RGBW-type color filter, including the white (W:
White), as shown in Fig. 2(l), is converted to an RGB
arrangement (for example, Bayer arrangement) shown in Fig.
2(2). Furthermore, in this conversion process, the process
- 26 -
SP262426
is concurrently performed for decreasing an occurrence of a
false color.
[0035]
The obtainment data obtained by the imaging device
(image sensor) having, for example, the RGBW-type color
filter, including the white (W: White), as shown in Fig.
2(1) is the two-dimensional pixel arrangement signal in
which W pixels which are the main components of the
brightness signal are arranged in a checkered state and the
pixels with the plurality of colors which are color
information components are arranged in the remaining region.
[0036]
The image processing device according to the present
invention includes a data conversion process unit analyzing
this image signal and thus performing the pixel conversion.
The data conversion process unit performs, for example, the
calculation of the conversion parameter, using a correlation
information on a correlation between the pixel (for example,
the W pixels) which are the main components of the
brightness signal included in the two-dimensional pixel
arrangement signal, and the color information component of
the pixel after conversion, or a correlation between one
selection color information component selected from among
the color information components, for example, a color
information component (for example, G) of which the
- 27 -
SP262426
distribution rate is the highest and the color information
component of the pixel after conversion, as the parameter
which is applied to the pixel conversion process, by
analyzing the input image signal, and performs the setting
process of setting the conversion pixel value to which the
calculated parameter is applied.
[0037]
The image processing device according to the present
invention, as shown in Fig. 2, performs the process of
converting at least a part of each pixel of RGBW which are
set to the RGBW color arrangement to the other color (any of
RGB), or compensating at least the part of each pixel of
RGBW. Specifically, in the conversion process of converting
the RGBW arrangement to the RGB Bayer arrangement, five of
the following conversion or compensation processes are
performed.
(a) Convert a W pixel position to a G pixel (Estimate a G
pixel value) = (GonW)
(b) Convert a G pixel position to an R pixel (Estimate an R
pixel-value) = (RonG)
(c) Convert a G pixel position to a B pixel (Estimate a B
pixel value) = (BonG)
(d) Convert a B pixel position to an R pixel (Estimate an R
pixel value) = (RonB)
(e) Convert an R pixel position to a B pixel (Estimate a B
- 28 -
SP262426
pixel value) _ (BonR)
[0038]
Each of the above-described conversion processes (a) to
(e) is performed as a pixel value estimation or a
compensation, process for converting each pixel of RGBW in
the RGBW arrangement to an RGB pixel in the RGB arrangement.
By performing this process, the RGB arrangement shown in Fig.
2(2) is generated from the RGBW color arrangement shown in
Fig. 2(l).
[0039]
This conversion process of converting the color
arrangement is hereinafter referred to "re-mosaic."
In the following embodiments, the configuration is
described which performs the re-mosaic process of converting
the RGBW-type color arrangement having the white (W) to the
RGB-type color arrangement (Bayer arrangement) and further
performs the process of decreasing an occurrence of a false
color at the time of this re-mosaic process.
[0040]
In the pixel conversion process by the image processing
device according to the present invention, the conversion
process in which a rectangular pixel region with n x n
pixels is input as an input pixel unit is performed. That
is, the pixel information on the rectangular pixel region
with the n x n pixels is used to determine the conversion
- 29 -
SP262426
pixel value of one pixel in the center of the n x n pixels.
Specifically, for example, the processing is performed by a
unit of 7 x 7 pixels in an image (7 pixels in width and 7
pixels in height) . For example; in a case where the
processing is performed by a process unit of 7 x 7 pixels,
the image processing device inputs the pixel information by
a unit of 7 x 7 pixels and determines the conversion pixel
value of the central pixel using the pixel information on
the 7 x 7 pixels. The central pixel of the 7 x 7 pixels is
a pixel of any of RGBW and the RGBW are converted according
to the conversion patterns (a) to (e) described above.
[0041]
Rectangular pixels with n x n pixels which is a unit of
the pixel conversion process is hereinafter referred to as
"an input pixel unit." In a case where the process) of one
pattern (a) of the conversion patterns (a) to (e) described
above, that is,
the conversion process which is "(a) Convert a W pixel
position to a G pixel (Estimate a G pixel value) = (GonW)",
is performed, four different patterns (al) to (a4) shown in
Fig. 3 exist in the pixel pattern of 7 x 7 pixels which is
the input pixel unit.
In Figs. 3 (al) to (a4), the input pixel unit (process
unit) in which a W pixel is the central position of the 7 x
7 pixels is illustrated. On the left side of Fig. 3, is the
- 30 -
SP262426
input pixel unit, and on the right side is RGB arrangement
data which is a final processing result.
[0042]
In the input with the W pixel being the central
position of the 7 x 7 pixels, there are four different
patterns shown in Figs. 3 (al) to (a4). In a case where the
7 x 7 pixels with any of these patterns is input,
the process which is "(a) Convert a W pixel position to
a G pixel (Estimate a G pixel value) = (GonW)" is performed.
Furthermore, the result of the final change is shown on
the right side of Figs. 3 (al) to (a4), and only the process
of converting the central pixel W to the G pixel is
performed on the input pixel unit is shown on the left side
of Figs. 3 (al) to (a4) . Thereafter, the process units are
moved one by one and the conversion process (any of the
conversion processes (a) to (e) is performed, and thus the
final change results shown on the right side ofFigs. :3 (al)
to (a4) is obtained.
[0043]
Furthermore, in a case where the process (b) to (e)
among the conversion patterns (a) to (e) described above,
that is,
(b) Convert a G pixel position to an R pixel (Estimate an P.
pixel value) = (RonG),
(c) Convert a G pixel position to a B'pixel (Estimate a B
- 31 -
SP262426
pixel value) _ (BonG),
(d) Convert an R pixel position to a B pixel (Estimate a B
pixel value) = (Bong), and
(e) Convert a B pixel position to an R pixel (Estimate an R
pixel value) = (BonB) are performed, a relationship between
the input pixel unit (process unit) and a final output of
the conversion process is shown in Figs. 4(b) to (e).
[0044]
Fig. 4(b) illustrates a process example of "Convert a G
pixel position to an R pixel (estimate an R pixel value) _
(RonG)".
Fig. 4(c) illustrates a process example of "Convert a G
pixel position to a B pixel (Estimate a B pixel value) _
(BonG)".
Fig. 4(d) illustrates a process example of "Convert an
R pixel position to a B pixel (Estimate a B pixel value) _
(BonR)".
Fig. 4(e) illustrates a process example of "Convert a B
pixel position to an R pixel (Estimate an R pixel value) =
(RonB-°.
[0045]
[2. Configuration Examples and Process Examples of the
Imaging Device and the Image Processing Device]
Referring to Figs. 5 and 6, the configuration example
and the process example of the imaging device and the image
- 32 -
SP262426
processing device relating to one embodiment according to
the present invention.
Fig. 5 is a view illustrating a configuration example
of an imaging device 100 relating to one embodiment
according to the present invention. The imaging device 100
includes an optical lens 105, an imaging device (image
sensor) 110, a signal process unit 120, memory 130, and a
control unit 140. Furthermore, the imaging device is one
embodiment of the image processing device. The image
processing device includes a device such as a PC. The image
processing device, such as a PC, has a configuration which
includes other components, without the optical lens 105 of
the imaging device 100 shown in Fig. 3 and the imaging
device 110, and includes an input unit of inputting
obtainment data obtained by the imaging device 100, or a
memory unit. Specifically, the imaging device 100 is a
still camera, a video camera, and the likes. The imaging
device 100 includes an information process device capable of
processing an image, such as a PC.
[00461
The imaging device 100, which is a typical example of
the image processing device according to the present
invention, is described below. The imaging device (image
sensor) 110 of the imaging device 100 shown in Fig. 5 has a
configuration which includes a filter with an RGBW
- 33 -
SP262426
arrangement 181 having white (W) , described referring to Fig.
1(b) and Fig. 2(1). Specifically, the imaging device (image
sensor) 110 analyzes a two-dimensional pixel arrangement
signal in which pixels which are main components of a
brightness signal are arranged in a checkered state and
pixels with a plurality of colors which are color
information components are arranged in the remaining region,
and thus performs a pixel conversion. Furthermore, a color
which is the main component of the brightness signal is
white or green.
The imaging device (image sensor) 110 is an imaging
device which includes the filter having four kinds of
spectral characteristic, red (R) penetrating a wavelength
near red, green (G) penetrating a wavelength near green,
blue (B) penetrating a wavelength near blue, and in addition
white (W) penetrating all of RGB.
[0047]
The imaging device 110 including this RGBW arrangement
181 filter receives any light of RGBW by the pixel unit
through the optical lens 105, and generates an electric
signal corresponding to a light receiving signal strength,
by photoelectric conversion, thereby outputting the result.
A mosaic image, which is made from light analysis of four
kinds of RGBW, may be obtained by the imaging device 110.
[0048]
- 34 --
SP262426
An output signal of the imaging device (image sensor)
110 is input to a data conversion process unit 200 of the
signal process unit 120.
The data conversion process unit 200, as described
above referring to Fig. 2, performs the conversion process
of converting the RGBW arrangement 181 to an RGB arrangement
182. When performing this conversion process, as described
above, the five conversion or compensation processes,
"Convert a W pixel position to a G pixel (Estimate a G pixel
value) = (GbnW)"
"Convert a G pixel position to an R pixel (Estimate an R
pixel value) = (RonG)"
"Convert a G pixel position to a B pixel (Estimate a B pixel
value) = (Bone)"
"Convert a B pixel position to an R pixel(Estimate an R
pixel value) = (Rona)", and
"Convert an R pixel position to a B pixel (Estimate a B
pixel value) = (BonR), are performed.
In this conversion/compensation process, a process for
suppressing a false color is concurrently performed.
[0049]
The RGB arrangement 182 generated by the data
conversion process unit 200, that is, data in the Bayer
arrangement, is data in the color arrangement obtainable by
the imaging device such as a conventional camera. This
- 35 -
SP262426
color arrangement data are input to an RGB signal process
unit 250.
[0050]
The RGB signal process unit 250 performs the same
process as the signal process unit provided in, for example,
the conventional camera does. Specifically, a color image
183 is generated by performing a de-mosaic process, a white
balance adjustment process, a y compensation process, and
the like. The generated color image 183 is stored in the
memory 130.
[0051]
The control unit 140 performs a sequence of these
processes. For example, a program causing the sequence of
processes to be performed is stored in the memory 130, and
the control unit 140 reads the program from the memory 130,
and thus controls the sequence of the processes.
[0052]
Referring to Fig. 6, a detailed configuration of the
data conversion process unit 200 is described. The data
conversion process unit 200 performs the conversion process
of converting the RGBW color arrangement to the RGB
arrangement 182. Furthermore, in this process, the process
for suppressing the false color is concurrently performed.
[0053]
The data conversion process unit 200, as shown in Fig.
- 36 -
SP262426
6, a noise removal unit 201, first to fifth pixel
interpolation parameter calculation units 202 to 208, an
edge detection unit 209, a texture detection unit 210, and a
blend process unit 211. The data conversion process unit
200 sequentially inputs the pixel values by the process unit
of n x n pixels from the RGBW arrangement 181 which is a
process image, and determines the conversion pixel value of
the central pixel of n x n pixels, thereby outputting the
result. When completing the conversion process on all of
the pixels, the RGB arrangement 182 is accomplished and is
provided to the RGB signal process unit 250 shown in Fig. 5,
[0054]
A noise removal unit 201 performs a noise removal on
the W pixel in the center of the input pixel unit.
A first pixel interpolation parameter calculation unit
(GonW) 202 performs the parameter calculation process which
is applied to the process of converting the W pixel to the G
pixels.
A first temporary pixel setting unit (RBonWaroundG) 203
performs the process of converting the W pixel in the
neighborhood of the G pixel to a temporary pixel (R') (B')
.of an R or B pixel, as a preparatory process prior to the
process of converting the position of the W pixel adjacent
to the G pixel to the R or B pixel.
A second pixel interpolation parameter calculation unit
- 37 -
SP262426
(RBonGofHV) 204 calculates, for example, a parameter
corresponding to a longitudinal or traverse edge, as a
parameter which is applied to the process of converting the
G pixel to the R pixel or the Bpixel.
A third pixel interpolation parameter calculation unit
(RBonGofAD) 205 calculates, for example, a parameter
corresponding to a gradient edge, as a parameter which is
applied to the process of converting the G pixel to the R
pixel or the B pixel.
A second temporary pixel setting unit (RBonWaroundRB)
206 performs the process of converting the W pixel in the
neighborhood of the R or B pixel ,to the temporary pixel (R')
(B') of the R or B pixel, as a preparatory process prior to
the process of converting the position of the W pixel
adjacent to the R pixel or the B pixel to the R pixel or the
B pixel.
A fourth pixel interpolation parameter calculation unit
(RBonRBofHV) 207 calculates, for example, the parameter
corresponding to the longitudinal or traverse edge, as a
parameter which is applied to the process of converting the
R pixel to the B pixel, or the B pixel to the R pixel.
A fifth pixel interpolation parameter calculation unit
(RBonRBofAD) 208 calculates, for example, the parameter
corresponding to the gradient edge, as a parameter which is
applied to the process of converting the R pixel to the B
- 38 -
SP262426
pixel, or the B pixel to the R pixel.
[0055]
An edge detection unit 209 performs an edge direction
detection process, using the W pixel.
A texture detection unit 210 performs a texture
detection process, using the W pixel.
A blend process unit 211 performs a process of blending
the pixel output in each of the processes described above.
[0056]
[3. Description of the Process of the Data Conversion Unit]
Next, the process is described below, which each of the
process units configuring the data conversion process unit
200 shown in Fig. 6 performs.
(0057]
Furthermore, as described above referring to Fig. 6,
the data conversion process unit 200 sequentially inputs the
pixel values by a process unit of n x n pixels from the RGBW
arrangement 181and determines the conversion pixel value of
the central pixel of n x n pixels, thereby outputting the
result. In the embodiments described below, a configuration
is described which performs the process by inputting the
pixel region with 7 x 7 pixels as one process unit in the
data conversion process unit 200, on the basis of n = 7.
[0058]
However, the noise removal unit 201, the edge detection
39 -
SP262426
unit 209, and the texture detection unit 210, among the
process units shown in Fig. 6, may be set to perform the
process, by any of the 7 x 7 pixel region unit, or the 5 x 5
pixel region unit in the center part of the pixel region
with 7 x 7 pixels. The first pixel interpolation
calculation unit 202 to the fifth pixel interpolation
parameter calculation unit 208 which are the other process
units performs the process by the 7 x 7 pixel region unit.
[0059]
The processes of the edge detection, the noise removal,
or the texture detection generates information which is used
as supplementary information in the pixel value conversion.
The pixel region for this information generation may be
variously set. The smaller the process unit is, the more
efficient the process unit is in terms of the process load.
However, in a case where the process capability is
sufficient, the larger pixel region may be set as the
process unit. For example, the configuration may be
possible which performs the process by the 7 x 7 pixel unit.
[00601
In this way, the process unit for the processes of the
edge detection, the noise removal, and the texture detection
may be variously set. In the following embodiments, a
process example at the following setting described as one
example.
- 40 -
SP262426
[0061]
The first pixel interpolation calculation unit 202 to
the fifth pixel interpolation parameter calculation unit 208
perform the process on the RGBW arrangement 181, which are
input to the data conversion process unit 200, by a 7 x 7
pixel region unit.
The edge detection unit 209 performs also the edge
detection process by the same process unit as the 7 x 7
process unit for the RGBW arrangement 181 which is input to
the data conversion process unit 200.
In the noise removal unit 201 and the texture detection
unit 210, the 5 x 5 pixel region is extracted from the
center part of the pixel region with 7 x 7 pixels, and the
process is formed on the extracted 5 x 5pixel region as the
process unit.
In each of the process units, the process unit is set
in this manner and the process is performed. A process by
each of the process units is sequentially described below.
[0062]
(3-1.-Process by a Noise Removal Unit)
First, a process by the noise removal unit 201 is
described referring to Fig. 7. The noise removal unit 201
performs the noise removal process on data in which the
central pixel of the input pixel unit (7 x 7 pixels) to the
data conversion process unit 200 is the W pixel. As
- 41 -
SP262426
described above, the noise removal unit 201 extracts the 5 x
5 pixel region from the center part of the pixel region with
7 x 7 pixels which the data conversion process unit 200
inputs, and the process is performed on the extracted 5 x 5
pixel region as the process unit. The noise removal is
performed as the calculation process of calculating the
noise reduction pixel value for the W pixel which is the
center of the input pixel unit.
[0063]
A variety of techniques for noise removal are
applicable. Here, a noise reduction process example using a
bilateral filter is described referring to Fig. 7.
Fig. 7(a) illustrates that the 5 x 5 pixel region which
is set in the center part of the 7 x 7 pixels which is the
input pixel unit is defined as a process object data 301 on
which the noise removal process is performed. Fig. 7 is a
view explaining an example in which the noise removal is
applied to the W pixel (pixel p) in the center of the
process object data 301.
[0064}
Fig. 7 illustrates (a) process object data, (b) a noise
reduction pixel calculation formula, and (c) a linear
approximation example of a function ^(r).
[0065]
As shown in (a) process object data, the noise removal
- 42 -
SP262426
unit 201 performs the process in a case where the central
pixel (p) of the noise removal process object data 301 with
the 5 x 5 pixels in the center part of the input pixel unit
(7 x 7 pixels) having the RGBW arrangement is the W pixel.
A gray region shown in 7(a) is the W pixel, and the other
white region is any of the RGB pixels. Furthermore, in the
other drawings which are referred to in the following
description, the gray region is also defined as the W pixel,
and the other white region as any of the RGB pixels.
[0066]
The noise removal unit 201 uses a pixel value I (p) of
the W pixel which is the central pixel (p) of the process
object data 301 and a pixel value I (q) of the pixel which
is included in the process object data 301 (5 x 5 pixels),
and thus calculates a noise reduction pixel value INR (P)
according to a noise reduction pixel calculation formula
shown in Fig. 7(2). That is, the noise reduction pixel
value INR (p) is calculated according to the following
formula (Formula 1).
[00671
[Math 1]
- 43 -
I(q).AI(q)-I(P)I)
INR (P)-
vEQP
^o( i(q)- I(p)I)
9EQp
SP262426
...Formula 1
[0068]
In the above-described formula, Op is a set of pixels
included in (5 x 5 pixels) which is the process object data
301, i(q) is its pixel value, and I (p) is a pixel value of
the central pixel p (= W pixel).
The function 4(r) generally uses an exponential function.
However, to suppress an amount of operation, as shown in Fig.
7(3), the function 4(r) may be a function to which the
linear approximation is applied.
The linear approximation shown in Fig. 7(3) is a linear
approximation example, where, r = 0 to Thl(2.0) -> 4(r) _
1.0, r = Thl (2.0) to Th2 (3.0) -> +(r) = 1.0 to 0 (linear
change), and r = equal to or more than Th2 -> $(r) = 0, with
Thl =-2.0 and Th2 = 3.0 being set as threshold values.
[0069]
In this way, the noise removal unit 201 applies the
bilateral filter, and thus calculates the noise reduction
pixel value INR (p) of the W pixel in the center of the noise
removal process unit (5 x 5 pixels) according to the above-
44 -
SP262426
described formula (Formula 1). The calculated noise
reduction W pixel (INR(p)) is output to the first pixel
interpolation parameter calculation unit (GonW) 202 as shown
in Fig. 6.
[0070]
Furthermore, the process to which the bilateral filter
is applied, described referring to Fig. 7, is one example of
the noise removal process, and the noise removal unit 201 is
not limited to the process to which the bilateral filter is
applied, described referring to Fig. 7 and may have a
configuration which uses other noise removal methods.
[0071]
(3-2. Process by an Edge Detection Unit)
Next, a process by an edge detection unit 209 is
described. The edge detection unit 209 verifies a discrete
white (W) signal included in the RGBW arrangement which is
an input signal, and generates edge information included in
the image, for example, edge information including the edge
direction and the edge strength, thereby outputting the
result to the blend process unit 211 and additionally to the
plurality of pixel interpolation parameter calculation units.
[0072]
A judgment method of judging the edge direction and the
edge direction using the W pixel, which the edge detection
unit 209 performs, is described referring to Fig. 8 and the
- 45 -
SP262426
subsequent figures.
[0073]
The edge detection unit 209 judges the edge direction
and the edge strength using only the white (W) signal, among
signals of the RGBW arrangement 181, which are input. In
the present embodiment, the edge detection unit 209, as
described above, performs the edge detection process by the
same process unit as the 7 x 7 process unit for the RGBW
arrangement 181 which is input to the data conversion
process unit 200. The edge detection unit 209 sequentially
performs the edge detection process on the 7 x 7 pixel
region while moving the 7 x 7 pixel region one by one. The
edge information (edge direction and edge strength) is
obtained which corresponds to the central pixel of the 7 x '7
pixel regions, by performing the process on one 7 x 7 pixel
region. The edge information corresponding to each pixel is
output to the blend process unit 211, and the first to fifth
pixel interpolation parameter calculation units.
[0074]
P, variety of techniques are applicable to the edge
detection process. One such technique is described
referring to Figs. 8 and 9. In the method described below,
4 x 4 pixels in the neighborhood of the center of the 7 x 7
pixel region are used.
[0075]
- 46 -
SP262426
As shown in Fig. 8, the central pixel of the 7 x 7
pixels which is the process object comes in two cases. One
is (a) the W pixel and the other is (b) a pixel other than
the W pixel.
Furthermore, in Fig. 8, the pixel marked with gray is
the W pixel and the other corresponds to any of the ROB.
The edge judgment process is performed on these two (a)
and (b) cases, to estimate whether which of the four
directions including horizontal, vertical, right gradient,
and left gradient directions is the edge direction, or to
estimate its strength, using the different calculation
formulas.
[0076]
The specific process is described referring to Fig. 9.
In Fig. 9, the calculation formulae are shown which are
applied to the judgment process of judging the edge
direction and the edge strength in the following two cases
(a) and (b): (a) the case where the central pixel is the W
pixel and (b) the case where the central pixel is a pixel
other than the W pixel.
The formula which is applied is a formula for calculating
the gradient of the pixel in the specific direction in the
image and is a formula for calculating each of the following
values.
grades: gradient absolute value average in the horizontal
- 47 -
SP262426
direction, gradV: gradient absolute value average in the
vertical direction, gradA: gradient absolute value average
in the right gradient upward direction, and gradD: gradient
absolute value average left gradient upward direction.
The grades, gradV, gradA, and gradD correspond to the
absolute values of gradient (difference) of the pixel values
in the different directions. The specific calculation
process is described below.
[0077]
(a) Process in a Case where a Central Pixel is a W Pixel
First, referring to Fig. 9(a), the process performed in
a case where the central pixel is the W pixel is described.
In Fig. 9(a), (al) to (a4) illustrate calculation process
examples of calculating gradH, gradV, gradA, and gradD in
the case where the central pixel is the•W pixel.
A position marked with a double circle "O" is a
position of the central pixel of the 7 x 7 pixels.
Furthermore, a position marked with a single circle "0"
is a position of an edge centroid.
[0078]
In a case where the central pixel is the W pixel, gradH,
gradV, gradA, and gradD are calculated using the following
calculation formula (Formula 2).
[0079]
[Math 2]
- 48 -
gradH = I W22 - W 02 I + I W31 - WI i
2
gradV = W22 - W20 + W13 Wll
2
gradA = I W22 - W31 I + W13 - W22I
2
gradD
W22 - WI + W33 W22
2
SP262426
...Formula 2
[0080]
Furthermore, Wxy indicates a W pixel value in the x-y
coordinate position, in the coordinate system in which
coordinates of the uppermost leftmost pixel of the 4 x 4
pixels shown in Fig. 9 is set to (0,0), and coordinates of
the lowermost rightmost pixel is set to (3,3), with the
horizontal direction being defined as (x) and the vertical
direction as (y).
[0081]
gradH is a gradient absolute value average in the
horizontal direction, and corresponds to an average value of
difference absolute values of the W pixels close to the
horizontal direction.
- 49 --
SP262426
As shown in Fig. 9 (al), the average value of the
difference absolute values of two of the W pixels close to
the horizontal direction of two horizontal lines in the
central part of the 4 x 4 pixel region is calculated as
grades.
[00821
gradV is the gradient absolute value average in the
vertical direction, and corresponds to the average value of
the difference absolute values of the W pixels close to the
vertical direction.
As shown in Fig. 9 (a2), the average value of the
difference absolute values of two of the W pixels close to
the vertical direction of two vertical lines in the central
part of the 4 x 4 pixel region is calculated as gradV.
[00831
gradA is the gradient absolute value average in the
right gradient upward direction, and corresponds to the
average value of the difference absolute values of the W
pixels close to the right gradient upward direction.
Asshown in Fig. 9 (a3), the average value of the
difference absolute values of two of the W pixels close to
the right gradient upward direction of one line in the right
gradient upward direction, which is in the central part of
the 4 x 4 pixel region is calculated as gradA.
[0084]
- 50 -
SP262426
gradD is the gradient absolute value average in the
left gradient upward direction, and corresponds to the
average value of the difference absolute values of the W
pixels close to the left gradient upward direction.
As shown in Fig. 9(a4), the average value of the
difference absolute values of two of the W pixels close to
the left gradient- upward direction of one line in the ]eft
gradient upward direction, which is in the central part of
the 4 x 4 pixel region is calculated as gradD.
[0085]
The greater the value of the gradient absolute value
average gradH in the horizontal direction, the more likely
the edge strength in the vertical direction is to be great.
The greater the value of the gradient absolute value
average gradV in the vertical direction,'the more likely the
edge strength in the horizontal direction is to be great.
The greater the value of the gradient absolute value
average gradA in the right gradient upward direction, the
more likely the edge strength in the left gradient upward
direction is to be great.
The greater the value of the gradient absolute value
average gradD in the left gradient upward direction, the
more likely the edge strength in the right gradient upward
direction is to be great.
In this way, the edge direction and the edge strength
- 51 -
SP262426
may be judged based on each value of the calculated values,
gradH, gradV, gradA, and gradD.
[0086]
(b) Process in a Case where the Central Pixel is a Pixel
other than the W Pixel
Next, the process in a case where the central pixel is
a pixel other than the W pixel is described referring to Fig.
9(b). In Figs. 9(b), (bl) to (b4), illustrate calculation
process examples of calculating gradH, gradV, gradA, and
gradD in the case where the central pixel is a pixel other
than the W pixel.
A position marked with "O" is a position of the central
pixel of the 7 x 7 pixels.
Furthermore, a position marked with "0" is an edge
centroid position.
[0087]
In a case where the central pixel is a pixel other than
the W pixel, grades, gradV, gradA, and gradD are calculated
using the following calculation formula (Formula 3).
[00881
[Math 3]
- 52 --
grades =
gradV =
gradA =
TT 32 -- W12
W23 - TV21
TIV23 -
gradD = 1TV23 - Wit
+
2
W21 - YVo I
2
T^VI2 - ui<
2
Wit-W21
+
2
TV32 - Y
SP262426
•.. Formula 3
[0089]
Furthermore, Wxy indicates a,W pixel value in the x-y
coordinate position, in the coordinate system in which
coordinates of the uppermost leftmost pixel of the 4 x 4
pixels shown in Fig. 9 is set to (0,0), and coordinates of
the lowermost rightmost pixel is set to (3,3), with the
horizontal direction being defined as (x) and the vertical
direction as (y).
[0090]
g-radH is the gradient absolute value average in the
horizontal direction, and corresponds to the average value
of difference absolute values of the W pixels close to the
horizontal direction.
As shown in Fig. 9 (bl), the average value of the
difference absolute values of two of the W pixels close to
- 53 -
SP262426
the horizontal direction of two of the horizontal lines in
the central part of the 4 x 4 pixel region is calculated as
gradH.
[0091]
gradV is the gradient absolute value average in the
vertical direction, and corresponds to the average value of
the difference absolute values of the W pixels close to the
vertical direction.
As shown in Fig. 9 (b2), the average value of the
difference absolute values of two of the W pixels close to
the vertical direction of two of the vertical lines in the
central part of the 4 x 4 pixel region is calculated as
grade.
[0092]
gradA is the gradient absolute value average in the
right gradient upward direction, and corresponds to the
average value of the difference absolute values of the W
pixels close to the right gradient upward direction.
As shown in Fig. 9 (b3), the average value of the
difference absolute values of two of the W pixels close to
the right gradient upward direction of two lines in the
right gradient upward direction, which are in the central
part of the 4 x 4 pixel region is calculated as gradA.
[0093]
gradD is the gradient absolute value average in the
- 54 -
SP262426
left gradient upward direction, and corresponds to the
average value of the difference absolute values of the W
pixels close to the left gradient upward direction.
As shown in Fig. 9 (b4), the average value of the
difference absolute values of two of the W pixels close to
the left gradient upward direction of two lines in the left
gradient upward direction, which are in the central part of
the 4 x 4 pixel region is calculated as gradD.
[0094]
The greater the value of the gradient absolute value
average grades in the horizontal direction, the more likely
the edge strength in the vertical direction is to be great.
The greater the value of the gradient absolute value
average gradV in the vertical direction, the more likely the
edge strength in the horizontal direction is to be great.
The greater the value of the gradient absolute value
average gradA in the right gradient upward direction, the
more likely the edge strength in the left gradient upward
direction is to be great.
The greater the value of the gradient absolute value
average gradD in the left gradient upward direction, the
more likely the edge strength in the right gradient upward
direction is to be great.
In this way, the edge direction and the edge strength
may be estimated based on each of the calculated values,
- 55 -
SP262426
gradH, gradV, gradA, and gradD.
[0095]
In this way, the edge detection unit 209 obtains the
edge information (edge direction and edge strength)
corresponding to each pixel, based on each of gradH, gradV,
gradA, and gradD. The obtained edge information is output
to the blend process unit 211.
Furthermore, the edge direction-strength detection
method described above is one example, and a configuration
which uses other edge detection methods may be possible.
For example, since the edge detection method described above
referring to Fig. 9 uses the specific pixel value
information in an extremely narrow range, the misjudgment is
expected to occur when the noise is high. An obtainment
process example of obtaining the edge information to prevent
this misjudgment is described referring to Fig. 10.
[0096]
The obtainment process example of obtaining the edge
information shown in Fig. 10 is a technique which uses a
processin which a weighting addition is performed on the
gradient absolute value average, which is the calculation
value described referring to Fig. 9. In Fig. 10, (1) an
obtainment process example of obtaining the edge information
on the horizontal-vertical component, and (2) an obtainment
process example of obtaining the edge information on the
56 -
89262426
gradient component are shown.
[0097]
In Figs. 10(l) and 10(2), image data on the 7 x 7
pixels are illustrated. This data are data which the edge
detection unit 209 defines as the process object and in
which only the W pixel values are discretely arranged in the
same manner as shown in Fig. 9. The edge information (edge
direction and edge strength) corresponding to the central
pixel positioned in the center of the 7 x 7 pixels is
obtained.
[0098]
The obtainment process of obtaining the edge
information on the horizontal-vertical component shown in
Fig. 10(l) is described. In Fig. 10(l), 16 edge centroids
are illustrated. These correspond to the edge centroids
shown in Figs. 9 (al), (a2), (bl), and (b2). That is, the
edge centroids correspond to edge centroids at the time of
calculating the values, gradH: the gradient absolute value
average in the horizontal direction, and gradV: the gradient
absolute value average in the vertical direction.
[0099]
The 4 x 4 pixel region shown in Figs. 9 (al), (a2),
(b1), and (b2) is set in the 7 x 7 pixel region shown in Fig.
10(1). The setting-possible 4 x 4 pixel regions are 16,
from the uppermost leftmost 4 x 4 pixel region 302 to the
- 57 -
SP262426
lowermost rightmost 4 x 4 pixel region 304 shown in the
figure. The 16 edge centroids shown in Fig. 10 (1) are edge
centroids which correspond to 16 of the 4 x 4 pixel regions,
that is, edge centroids which have the same positions as 16
of the 4 x 4 pixel regions shown in Figs. 9(al), (a2), (bl),
and (b2).
[0100]
In a case where the coordinate position of the edge
centroid is expressed as (i,j), with the horizontal axis
being defined as i and the vertical axis as j on a scale of
0, 1, 2, and 3 as shown in the figure, the edge centroid (0,
0) 301 is an edge centroid which corresponds to the 4 x 4
pixel region 302. In a case where the 4 x 4 pixel region
302 is defined as the 4 x 4 pixel region shown in Fig. 9
(al), (a2), (b1), and (b2), the edge centroid (0,0) 301
corresponds to the centroid shown in Fig. 9(al), (a2), (bl),
and (b2).
[0101]
Furthermore, the edge centroid (3,3) 303 is an edge
centroid which is set by corresponding to the 4 x 4 pixel
region 304. In a case where the 4 x 4 pixel region 304 is
defined as the 4 x 4 pixel region shown in Figs. 9 (al),
(a2),(bl), and (b2), the edge centroid (3,3) 303
corresponds to the centroid shown in Figs. 9(al), (a2), (bl),
and (b2).
- 58 -
SP262426
[0102]
In the 7 x 7 pixel region shown in Fig. 10(1), 16 sets,
each set consisting of the 4 x 4 pixel region and the edge
centroid, are set. Furthermore, the calculation formulae
described referring to Figs. 9(al), (a2), (bl), and (b2) are
used, and thus the values of grades: the gradient absolute
value average in the horizontal direction, and gradV: the
gradient absolute value average in the vertical direction
may be calculated by 16 for each, for these 16 sets.
[0103]
The gradient absolute value averages (gradH) and
(gradV) which are calculated using the 4 x 4 pixel region
corresponding to the edge centroid (i, j) are expressed as
gradHi,j, and gradVj,j, respectively. In addition to using
these, the weighting addition value of the gradient absolute
value average, that is, dirH: horizontal gradient
information and dirV: vertical gradient information is
calculated using the following calculation formula (Formula
4).

CLAIMS
SP262426
[Claim 1]
An image processing device, comprising:
a data conversion process unit performing pixel
conversion by interpreting a two-dimensional pixel
arrangement signal in which pixels which are main components
of a brightness signal are arranged in a checkered state,
and pixels of a plurality of colors which are color
information components are arranged in the remaining region,
wherein the data conversion process unit includes a
parameter calculation unit calculating a parameter which is
applied to a pixel conversion process by interpreting the
two-dimensional pixel arrangement signal and the parameter
calculation process unit performs a calculation process of
calculating the parameter using at least of one of first
correlation information for a correlation between the pixel
which is the main component of the brightness signal
included in the two-dimensional pixel arrangement signal and
the color information component of the pixel after
conversion, and second correlation information for a
correlation between one selection color information
component selected from among the color information
components and the color information component of the pixel
after the conversion, in the calculation process of the
parameters which are applied to the pixel conversion process.
- 173 -
SP262426
[Claim 2]
The image processing device according to claim 1,
wherein the parameter calculation unit uses the
correlation information on a correlation between the color
information component having the highest distribution rate
among the color information components and the color
information component of the pixel after the conversion, as
the second correlation information.
[Claim 3]
The imaging processing device according to claim 1 or
claim 2,
wherein the data conversion process unit includes an
edge detection unit generating edge information by
interpreting the two-dimensional pixel arrangement signal,
and
the parameter calculation unit performs the calculation
process of calculating the parameter by selectively using
any of the first correlation information and the second
correlation information, according to an edge direction
which-the edge detection unit detects.
[Claim 4]
The image processing device according to claim 3,
wherein a color of the main component of the brightness
signal is white, and the color information component of
which the distribution rate is the highest is green, and
- 174 -
SP262426
the parameter calculation unit performs the calculation
process of calculating the parameter using any of the first
correlation information on a correlation between white and
the color information component of the pixel after the
conversion and the second correlation information on a
correlation between green and the color information
component of the pixel after the conversion, according to
the edge direction which the edge detection unit detects, in
the calculation process of calculating the parameter which
is applied to the pixel conversion process.
[Claim 5]
The image processing device according to claim 3 or
claim 4,
wherein the data conversion process unit includes
a texture detection unit generating' texture information
by interpreting the two-dimensional pixel arrangement signal,
and
a blend process unit inputting the parameter which the
parameter calculation unit calculates, the edge information
and the texture information, and determining a conversion
pixel value by performing a blend process by changing a
blend ratio of the parameter which the parameter calculation
unit calculates, according to the edge information and the
texture information corresponding to the conversion pixel.
[Claim 6]
- 175 -
SP262426
The image processing device according to claim 5,
wherein the edge detection unit generates the edge
information including edge direction and strength
information corresponding to each pixel by interpreting an
RGBW arrangement signal which is generated from an RGB pixel
and a white (W) pixel,
the texture detection unit generates texture
information indicating texture extent corresponding to each
pixel by interpreting the RGBW arrangement signal,
the parameter calculation unit is a parameter
calculation unit which calculates a parameter for converting
the RGBW arrangement to the RGB arrangement, and generates
the parameter corresponding to an interpolation pixel value
calculated by an interpolation process of changing an
application pixel position according to the edge direction
corresponding to the conversion pixel, and
the blend process unit inputs the parameter which the
parameter calculation unit calculates, the edge information
and the texture information and performs the process of
determining the conversion pixel value by performing the
blend process by changing the blend ratio of the parameter
which the parameter calculation unit calculates, according
to the edge information and the texture information
corresponding to the conversion pixel.
[Claim 7]
- 176 -
SP262426
The image processing device according to claim 6,
wherein the parameter calculation unit has a
configuration which generates the parameter by the
interpolation process of defining the pixel position which
is applied to the interpolation process as the pixel
position which is along the edge direction.
[Claim 8]
The image processing device according to claim 6 or
claim 7,
wherein the parameter calculation unit has a
configuration which generates the parameter by the
interpolation process of using any of a correlation in a
local region between the W pixel configuring the RGBW
arrangement or the other RGB pixel or a correlation in the
local region between the G pixel configuring the RGBW
arrangement and the other RGB pixel.
[Claim 9]
The image processing device according to any of claims
6 to 8,
wherein the image processing device further includes a
temporary pixel setting unit setting the pixel value of the
pixel of any of RGB to a W pixel position by the
interpolation process of using the correlation in the local
region between the W pixel configuring the RGBW arrangement
and the other RGB pixel, and
- 177 -
SP262426
the parameter calculation unit has a configuration
which generates the parameter by the interpolation process
of applying temporary pixel setting data.
[Claim 10]
The image processing device according to any of claims
6 to 9,
wherein the parameter calculation unit selects which
one of first correlation information on a correlation
between the W pixel configuring the RGBW arrangement and the
color information component of the pixel after the
conversion and second correlation information on a
correlation between the G pixel configuring the RGBW
arrangement and the color information component of the pixel
after the conversion is used, according to 4 kinds of edge
directions, longitudinal, traverse, left gradient upward,
and right gradient upward, which the edge detection unit
detects.
[Claim 11]
The image processing device according to claim 10,
w-herein the parameter calculation unit generates a
plurality of parameters by setting a reference pixel
position to the pixel position along the edge direction and
using the first correlation information and the second
correlation information, and
the blend process unit performs the blend process of
- 178 -
SP262426
changing the blend ratios of the plurality of parameters
according to the comparison result, by performing a strength
comparison of the 4 kinds of the edge directions,
longitudinal, traverse, left gradient upward, and right
gradient upward.
[Claim 12]
The image processing device according to claim 11,
wherein the blend process unit performs a blend process
of calculating an edge direction ratio (ratioFlat) of a
longitudinal and traverse direction edge and a gradient
direction edge corresponding to the conversion pixel,
additionally calculating longitudinal and traverse direction
edge direction weight (weightHV) indicating that the greater
the value is, the stronger the longitudinal and traverse
direction edge is than the gradient direction edge, and the
smaller the value is, the stronger the gradient direction
edge is than the longitudinal and traverse direction edge,
based on the edge direction ratio (ratioFlat),
increasing a blend ratio of the parameter calculated by
setting the edge direction to the longitudinal or traverse
direction in a case where the longitudinal and traverse
direction edge corresponding to the conversion pixel is
stronger than the gradient direction edge, and
increasing the blend ratio of the parameter calculated
by setting the edge direction to the gradient direction edge
- 179 -
SP262426
in a case where the longitudinal and traverse direction edge
corresponding to the conversion pixel is weaker than the
gradient direction edge.
[Claim 13]
The image processing device according to any of claims
6 to 12,
wherein the texture detection unit calculates a
flatness weight (weightFlat) corresponding to each pixel,
indicating a high value for a pixel area of which textures
are small in number and of which the flatness is high and a
low value for a pixel area of which textures are large in
number and of which the flatness is low, as the texture
information,
the parameter calculation unit calculates a contrast
enhancement process application parameter for performing a
contrast enhancement process on the interpolation pixel
value, and a contrast enhancement process non-application
parameter for not performing the contrast enhancement
process on the interpolation pixel value, and
the blend process unit performs the blend process of
setting the blend ratio of the contrast enhancement process
non-application parameter to be high for the pixel of which
the flatness weight is great, and setting the blend ratio of
the contrast enhancement process application parameter to be
high for the pixel of which the flatness weight is small.
- 180 -
SP262426
[Claim 141
The image processing device according to any of claims
3 to 13,
wherein the edge detection unit has a configuration
which generates the edge information corresponding to each
pixel, by an interpretation process using only the white (W)
pixel of the RGBW arrangement signal, and generates the edge
information including the edge direction and the strength
information corresponding to each pixel by calculating a
signal value gradient of the W pixel in the neighborhood of
the process object pixel.
[Claim 15]
The image processing device according to any of claims
5 to 14,
wherein the texture detection unit generates texture
information indicating a texture extent corresponding to
each pixel, by the interpretation process using only the
white (W) pixel of the RGBW arrangement signal.
[Claim 16]
knimage processing method of performing an image
signal process in an image processing device comprising:
an edge detection step of enabling an edge detection
unit to generate edge information including an edge
direction and strength information corresponding to each
pixel by interpreting an RGBW arrangement signal which is
- 181 -
SP262426
generated from an RGB pixel and a white (W) pixel;
a texture detection step of enabling a texture
detection unit to generate texture information indicating
texture extent corresponding to each pixel by interpreting
the RGBW arrangement signal;
a parameter calculation step of being a parameter
calculation step of enabling a parameter calculation unit to
calculate the parameter for converting the RGBW arrangement
to the RGB arrangement, and to generate the parameter
corresponding to an interpolation pixel value calculated by
an interpolation process of changing an application pixel
position according to the edge direction corresponding to
the conversion pixel; and
a blend process step of enabling a blend process unit
to input the parameter which the parameter calculation unit
calculates and the edge information and the texture
information and determine the conversion pixel value by
performing the blend process by changing the blend ratio of
the parameter which the parameter calculation unit
calculates, according to the edge information and the
texture information corresponding to the conversion pixel,
wherein the parameter calculation step performs the
calculation process of calculating the parameter using at
least any of first correlation information and second
correlation information, the first correlation information
- 182 -
SP262426
on a correlation between the pixel which is the main
component of the brightness signal included in the twodimensional
pixel arrangement signal or the color
information component of the pixel after conversion, and the
second correlation information on a correlation between the
color information component of which a distribution rate is
the highest among the color information components and the
color information component of the pixel after the
conversion, in the calculation process of calculating the
parameter which are applied to the pixel conversion process.
[Claim 17]
A program for causing the performing of an image signal
process in an image processing device, the image signal
process comprising:
an edge detection step of enabling an edge detection
unit to generate edge information including an edge
direction and strength information corresponding to each
pixel by interpreting an RGBW arrangement signal which is
generated from an RGB pixel and a white (W) pixel,
a texture detection step of enabling a texture
detection unit to generate texture information indicating
texture extent corresponding to each pixel by interpreting
the RGBW arrangement signal,
a parameter calculation step of being a parameter
calculation step of enabling a parameter calculation unit to
- 183 -
SP262426
calculate the parameter for converting the RGBW arrangement
to the RGB arrangement, and to generate the parameter
corresponding to an interpolation pixel value calculated by
an interpolation process of changing an application pixel
position according to the edge direction corresponding to
the conversion pixel, and
a blend process step of enabling a blend process unit
to input the parameter which the parameter calculation unit
calculates, the edge information and the texture information
and determine the conversion pixel value by performing the
blend process by changing the blend ratio of the parameter
which the parameter calculation unit calculates, according
to the edge information and the texture information
corresponding to the conversion pixel,
wherein the parameter calculation step is caused to
perform the calculation process of calculating the parameter
using at least any of first correlation information and
second correlation information, the first correlation
information on a correlation between the pixel which is the
main component of the brightness signal included in the twodimensional
pixel arrangement signal and the color
information component of the pixel after conversion, and the
second correlation information on a correlation between the
color information component of which the distribution rate
is the highest among the color information components and
- 184 -
SP262426
the color information component of the pixel after the
conversion, in the calculation of calculating the parameter
which are applied to the pixel conversion process.