• SP342362XXOO
1/29
IMAGE PROCESSING APPARATUS, METHOD, AND PROGRAM
BACKGROUND
[0001]
5 The present technology relates to an image processing apparatus, method,
and program, and more specifically, to an image processing apparatus, method, and
program capable ofreducing noise more simply.
[0002]
Recently, due to market demands for higher resolution, smaller size, and
10 lower cost, the pixels in an image sensor have been becoming more miniature. For
example, if about 8,000 pixels are arranged on an image sensor with a width in the
lateral direction of 8 mm, the pixel pitch, namely, the interval or width between
pixels, is about 1 /..lm.
[0003]
15 Thus, as the pixel pitch on the image sensor becomes smaller, the image
signal SIN ratio (signal to noise ratio) obtained by the image sensor deteriorates, so
that the quality of image deteriorates.
[0004]
In view of this, a technology has been proposed for removing the noise
20 component inherent in the imaging apparatus from an image during image capture by
capturing an image for noise component removal and determining in advance the
noise component from that image (e.g., JP 2007-274064A).
[0005]
25 SUMMARY
[0006]
However, in the above-described technology, if the noise component
included in the image signal varies due to the imaging conditions and the like, noise
can no longer be sufficiently reduced. Further, in the above-described technology,
30 since circuits are required for the extraction, recording, and removal of the noise
component, the configuration of the imaging apparatus becomes complex.
• SP342362XXOO
2/29
[0007]
According to an embodiment of the present technology, which was made in
view of such circumstances, noise can be reduced more simply.
[0008]
5 According to an embodiment of the present disclosure, there is provided an
image processing apparatus including a binning unit configured to calculate, based
on, as a binning area, an area including two or more adjacent pixels in a unit area
including m pixels in a lateral direction and n pixels in a perpendicular direction on
an image sensor, a product of output values of the pixels in the binning area as an
10 addition output value for each ofm x n binning areas that are different to each other,
and a calculation unit configured to calculate a pixel value of each pixel in the
binning area based on m x n addition output values obtained for the unit area.
[0009]
The calculation unit may be configured to calculate the pixel value of each
15 pixel in the unit area by calculating q = wp, where a matrix obtained by arranging the
m x n addition output values is q, a matrix obtained by arranging the pixel value of
each pixel in the unit area is p, and a matrix obtained by arranging a weighting of
each pixel in the unit area is w.
[0010]
20 The calculation unit may be configured to calculate the pixel value of each
pixel in the unit area by calculating p = w·1q, where w-I is an inverse matrix of the
matrix w.
[0011]
The matrix w may be a matrix obtained by setting a weighting of pixels
25 used to determine the addition output values as 1, and a weighting of pixels not used
to determine the addition output values as O.
[0012]
The binning unit may be configured to calculate the sum of digitized output
values of the pixels as the addition output value.
30 [0013]
According to an embodiment of the present disclosure, there is provided an
• ill9
SP342362XXOO
image processing method or a program including calculating, based on an area
including two or more adjacent pixels in a unit area including m pixels in a lateral
direction and n pixels in a perpendicular direction on an image sensor as a binning
area, a sum of output values of the pixels in the binning area as an addition output
5 value for each of m x n binning areas that are different to each other, and
calculating a pixel value of each pixel in the binning area based on m x n addition
output values obtained for the unit area.
[0014]
According to an embodiment of the present disclosure, based on an area
10 including two or more adjacent pixels in a unit area including m pixels in a lateral
direction and n pixels in a perpendicular direction on an image sensor as a binning
area, a sum of output values of the pixels in the binning area may be calculated as an
addition output value for each ofm x n binning areas that are different to each other,
and a pixel value of each pixel in the binning area may be calculated based on m x n
15 addition output values obtained for the unit area.
[0015]
According to an aspect ofthe present technology, noise can be reduced more
simply.
20 BRIEF DESCRIPTION OF THE DRAWINGS
[0016]
FIG 1 is a diagram illustrating binning and restoration of a pixel value;
FIG 2 is a diagram illustrating binning and restoration of a pixel value;
FIG 3 is a diagram illustrating a configuration example of an image
25 processing apparatus;
FIG 4 is a flowchart illustrating image generation processing; and
FIG 5 is a diagram illustrating a configuration example of a computer.
DETAILED DESCRIPTION OF THE EMBODIMENT(S)
30 [0017]
Hereinafter, preferred embodiments of the present disclosure will be
• 4/29
SP342362XXOO
described in detail with reference to the appended drawings. Note that, in this
specification and the appended drawings, structural elements that have substantially
the same function and structure are denoted with the same reference numerals, and
repeated explanation ofthese structural elements is omitted.
5 [0017]
An embodiment to which the present technology is applied will now be
described with reference to the drawings.
[0018]
(First Embodiment)
10 [Binning and Pixel Value Restoration]
The present technology reduces image signal noise more simply by
restoring a pixel value of each pixel based on an addition output value obtained by
adding the output value of several pixels on a light-receiving surface of an image
sensor, for example.
15 [0019]
As indicated by arrow All in FIG. 1, an area including pixels GI ,\, GI,2, G2,1,
and G2,2, which are arranged adjacent to each other on the light-receiving surface of
the image sensor, will be referred to as a unit area, and the ultimately output image
will be referred to as an output image, for example.
20 [0020]
In the present technology, as indicated by arrow A12, for example, the pixel
value Pij of a pixel Gjj (wherein i = 1 to 2, and j = 1 to 2) on the a sensor layer,
namely, the unit area, is calculated based on the addition output value qij (wherein i =
1 to 2, and j = 1 to 2) of the addition output layer. Note that in the drawings, the
25 vertical direction may also be referred to as the perpendicular direction (i direction),
and the horizontal direction may also be referred to as the lateral direction (j
direction).
[0021]
More specifically, in the present technology, an area including two or more
30 pixels adjacent to each other in a unit area of the image sensor is referred to as a
binning area, and the sum of the output values (pixel values) of the pixels in a
• 5/29
SP342362XXOO
binning area is referred to as the addition output value of that binning area. Further,
the ultimate pixel value of each pixel in the unit area is calculated based on the
addition output value of a plurality of binning areas.
[0022]
5 For example, as indicated by arrow A21, an area including the three pixels
G1,J, G 1,2, and G 2,1 that are adjacent to each other among the four pixels in the unit
area is one binning area, and the sum of the pixel values of these three pixels is the
addition output value ql,l.
[0023]
10 Further, as indicated by arrow A22, an area including the three pixels G1,J,
G 1,2, and G 2,2 that are adjacent to each other among the four pixels in the unit area is
one binning area, and the sum of the pixel values of these three pixels is as addition
output value ql,2.
[0024]
15 In addition, as indicated by arrow A23, an area including the three pixels
Gl,l, G 2,J, and G 2,2 that are adjacent to each other among the four pixels in the unit
area is one binning area, and the sum of the pixel values of these three pixels is
addition output value q2,1.
[0025]
20 Similarly, as indicated by arrow A24, an area including the three pixels G1,2,
G 2,J, and G 2,2 that are adjacent to each other among the four pixels in the unit area is
one binning area, and the sum of the pixel values of these three pixels is addition
output value Q2,2.
[0026]
25 Based on the thus-obtained four addition output values for one unit area,
addition output values QI,J, QI,2, Q2,\, and Q2,2, the pixel value of each pixel in the unit
area to be ultimately obtained can be determined. Namely, the pixel value Pl,l of
the pixel Gl,l, the pixel value PI,2 of the pixel G 1,2, the pixel value P2,1 of the pixel
G 2,1, and the pixel value P2,2 ofthe pixel G2,2 in the unit area are calculated.
30 [0027]
For example, among the respective areas on the image sensor indicated by
• 6/29
SP342362XXOO
arrows A 11 and A21 to A24, the hatched areas are areas where light is incident, and
the non-hatched areas are areas where light is not incident. In this example, light is
incident on the whole area of pixel G1,I and on half of the areas of pixels G1,2 and
G2,I, respectively.
5 [0028]
Therefore, as can be seen from the respective pixels in the unit area
indicated by arrow A21, the addition output value qI,I is the value of the light
components incident on pixels G1,I, G1,2, and G2,I. Further, as can be seen from the
respective pixels in the unit area indicated by arrow A22, the addition output value
10 ql,2 is the value of the light components incident on pixels G1,I and G1,2.
[0029]
Similarly, as can be seen from the respective pixels in the unit area indicated
by arrow A23, the addition output value q2,I is the value of the light components
incident on pixels G1,I and G2,I. In addition, as can be seen from the respective
15 pixels in the unit area indicated by arrow A24, the addition output value q2,2 is the
value ofthe light components incident on pixels G1,2 and G2,I.
[0030]
Further, using the respective addition output value qij (wherein i = 1 to 2,
andj = 1 to 2), the respective pixel value Pij can be obtained.
20 [0031]
Specifically, the simultaneous equation shown in the following formula (l)
can be obtained from the respective addition output value qij (wherein i = 1 to 2, and
j = 1 to 2).
[0032]
25 [Formula 1]
ql.l=WllPl,1+W12Pl.2+W13P2.1+W14P2,2
Ql. 2=W2l Pl. 1+W22Pl. 2+W23P2. 1+W24P2, 2
Q2, 1=W31Pl, 1+W32Pl,2+W33P2.l+W34P2,2
Q2,2=W41Pl, 1+W42P1.2+W43P2, 1+W44P2, 2 ••• (1)
[0033]
• 7/29
SP342362XXOO
In formula (l), each addition output value qij gives the same weighting to
the pixel value Pij for all ofthe pixels in the unit area. Ws,t (wherein s = 1 to 4, and t
= 1 to 4) in formula (l) represents the weighting for the pixel value Pij of each pixel.
[0034]
5 Specifically, when pixel Gij is used in the calculation of addition output
value qij, the weighting Ws,t for the pixel value Pij of that pixel Gij is set as "I", and
when pixel Gij is not used in the calculation of addition output value qij, the
weighting Ws,t for the pixel value Pij of that pixel Gij is set as "0".
[0035]
10 In other words, if a pixel Gij is included in the binning area for determining
the addition output value qij, the weighting Ws,t of the pixel value Pij is set as "I",
and if a pixel Gij is not included in the binning area, the weighting Ws,t of the pixel
value Pij is set as "0".
[0036]
15 Specifically, since all of the pixels except for pixel G2,2 are used in the
calculation of addition output value ql,J, for example, the weighting WI,I of pixel GI,1,
the weighting WI,2 of pixel G1,2, and the weighting WI,3 of pixel G2,1 are "1", and the
weighting WI,4 of pixel G2,2 is "0".
[0037]
20 By solving such a simultaneous equation shown in formula (l), the pixel
value of each pixel in the unit area can be obtained. For example, when solving the
simultaneous equation of formula (l), the pixel value Pij of each pixel can be easily
obtained by performing a matrix calculation.
[0038]
25 Specifically, a matrix obtained by arranging the weighting Ws,t shown in
formula (1) as shown in the following formula (2) is used as a weighting matrix w.
[0039]
[Formula 2]
• 8/29
SP342362XXOO
Wll W12 W13 W14 1 1 1 0
W21 W22 W23 W24 - 1 1 0 1 =w W31 W32 W33 W34 1 0 1 1
W41 W42 W43 W44 0 1 1 1 • • • (2)
[0040]
Further, the matrix (horizontal vector) obtained by arranging the addition
output values qij in order is used as a matrix q. For example, in formula (1), the
5 horizontal vector in which the addition output values ql,I, ql,2, q2,I, and q2,2 are
arranged in order is used as the matrix q.
[0041]
In addition, the horizontal vector obtained by arranging the pixel values Pij
in order is used as a matrix p. For example, in formula (1), the horizontal vector in
10 which the pixel values PI, I, PI,2, P2,I, and P2,2 are arranged in order is used as the
matrixp.
[0042]
Using the matrix p, matrix q, and weighting matrix w, formula (1) can be
expressed as the following formula (3).
15 [0043]
[Formula 3]
q=wp
[0044]
• • • (3)
In addition, formula (4) can be obtained by multiplying formula (3) by
20 matrix w-I
, which is the inverse matrix of the weighting matrix w.
[0045]
[Formula 4]
p=w-1q
[0046]
• • • (4)
25 Therefore, the ultimate pixel value Pij of each pixel in the unit area, which is
the target value, can be obtained from formula (4). More specifically, using formula
(2), formula (4) can be calculated as the following formula (5).
• 9/29
SP342362XXOO
[0047]
[Formula 5]
Pl, 1= -ql, 1+q1, 2+q2, 1-2/ 3 (-2ql, 1+ql, 2+q2, 1+q2, 2)
Pl, 2=ql, l-q2, 1+ 1/3 (-2ql, 1+ql, 2+q2, 1+q2, 2)
P2, 1=ql, l-q1, 2+ 1/3 (-2ql, 1+ql, 2+q2, 1+q2, 2)
P2,2=1/3(-2q1,1+q1,2+q2.l+q2,2) ... (5)
[0048]
5 Note that during an actual calculation, the value substituted for pixel value
Pij in formula (l) is the output value actually output from the pixel, and that the pixel
value Pij calculated in formula (5) is the pixel value of the pixel that was restored by
the calculation, namely, the pixel value ofthe pixel that is to be ultimately obtained.
[0049]
10 Further, each element in the weighting matrix w is set as "I" or "0" based
on whether the target pixel is used in the calculation of the addition output value,
namely, whether that pixel is in the binning area. Consequently, when determining
the pixel value Pij of each pixel in the binning area based on the calculation of
formula (4), the combination of the pixels forming the binning area in the unit area
15 needs to be a combination in which the weighting matrix x is an inverse matrix.
Namely, the combination of the pixels forming each binning area can be determined
so that the weighting matrix w is a non-specific matrix.
[0050]
Thus, by using the sum of the output values of the pixels in a binning area
20 including a plurality of pixels as an addition output value, and determining the
ultimate pixel value of each pixel based on a calculation from a plurality of addition
output values, image signal noise can be reduced based on a simpler configuration
and a simpler calculation.
[0051]
25 For example, since the shot noise included in an image captured by an
image sensor, namely, the image signal of the image, increases if the size of the
image sensor pixels is reduced (number of pixels is increased), the image signal SIN
• 10/29
SP342362XXOO
ratio deteriorates. In this case, the dominant noise included in the image signal will
be shot noise.
[0052]
Shot noise occurs due to increased statistical variation in the number of
5 photons (particles) when the number of particles incident on the image sensor is
small. The magnitude of shot noise is proportional to the mean of the current (light
intensity). Further, the number of photons detected over a given time has a Poisson
distribution centered at this mean. From the properties of a Poisson distribution, the
standard deviation of photon noise is equal to the square root of the number of
10 photons.
[0053]
From such properties of shot noise, if the output value of a plurality of
pixels is read as an addition output value, like in the present technology, the apparent
pixel pitch increases. Consequently, the noise component included in the signal
15 (addition output value) decreases, and the SIN ratio improves. Namely, although a
biased shot noise component is included in the output value of each pixel, if these
pixel output values are taken as the addition output value, the noise component ofthe
respective output values averages out. Consequently, the signal component
included in the addition output value can be relatively increased, which enables the
20 SIN ratio to be improved.
[0054]
By binning a plurality of pixels in this manner, the noise component in the
addition output value is reduced, so that if the pixel value of each pixel is calculated
(restored) from such an addition output value with reduced noise, the SIN ratio can
25 be improved more than when the pixel value of each pixel is directly read.
[0055]
Further, with binning, since the apparent pixel pitch increases, a higher
resolution image can be obtained compared with when the pixel value of each pixel
is directly read, even if there is a large number of pixels on the image sensor, which
30 can make resolution more difficult due to the diffraction limit. For example, in the
example illustrated in FIG I, compared with when the pixel value of each pixel is
• 11/29
SP342362XXOO
directly read, the apparent pixel pitch, namely, the width of the binning area,
approximately doubles in both the lateral and the perpendicular directions. The
present technology is especially preferred when the pitch of the pixels on the image
sensor is 4 /lm or less.
5 [0056]
In the above, as described with reference to FIG. 1, a case was described in
which a 2 pixel x 2 pixel area was the unit area. However, the unit area may be any
area, as long as it is formed from adjacent pixels.
[0057]
10 To generalize, the unit area may be an area including m pixels in the lateral
direction and n pixels in the perpendicular direction of the image sensor. In this
case, m x n binning areas for the unit area is determined. The combination of the
pixels forming the binning area may be any combination, as long as the combinatio~
combines 2 or more pixels adjacent to each other in the unit area. Further, the unit
15 area may be a rectangular area.
[0058]
However, when determining the pixel value of each pixel based on a
calculation performed using formula (4), the combinations of pixels forming each
binning area need to be a combination in which the weighting matrix w is a non-
20 specific matrix.
[0059]
For example, as the unit area of m pixels x n pixels, if a 3 pixel x 3 pixel
area is the unit area, as indicated by arrow A41 in FIG. 2, in such a case the pixel
value Pij that is the ultimate determination target of the pixel Gij on the sensor layer,
25 namely, the unit area, is calculated based on the addition output value qij (wherein i =
1 to 3, andj = 1 to 3) of the addition output layer.
[0060]
Note that in the following, parts in FIG. 2 that correspond to the case
illustrated in FIG. 1 are denoted using the same reference numerals, and a description
30 thereof is omitted as appropriate. Further, in FIG. 2, the horizontal direction and the
vertical direction represent the perpendicular direction (i direction) and the lateral
•
direction (j direction), respectively.
[0061]
12/29
SP342362XXOO
Thus, when a 3 pixel x 3 pixel area serves as the unit area, as indicated by
arrows A51 to A59, 9 (= 3 x 3) binning areas are determined for the unit area, and the
5 addition output value ofeach binning area is read.
[0062]
Namely, as indicated by arrow A51 for example, an area including all 9
pixels in the unit area is the binning area, and the sum of the pixel values of these 9
pixels is the addition output value qt,l. Here, the 9 pixels in the unit area are pixels
10 GI,I, G I,2, G I,3, G 2,1, G 2,2, G 2,3, G 3,I, G 3,2, and G 3,3.
[0063]
Further, as indicated by arrow A52 for example, of the 9 pixels in the unit
area, an area including the four pixels Gl,I, G 1,2, G 2,I, and G 2,2 that are adjacent to
each other is the binning area, and the sum of the pixel values of these pixels is the
15 addition output value ql,2.
[0064]
In addition, as indicated by arrow A53 for example, of the 9 pixels in the
unit area, an area including the four pixels G I,2, G1,3, G 2,2, and G 2,3 that are adjacent
to each other is the binning area, and the sum of the pixel values ofthese pixels is the
20 addition output value QI,3.
[0065]
As indicated by arrow A54 for example, of the 9 pixels in the unit area, an
area including the four pixels G 2,I, G 2,2, G 3,1, and G 3,2 that are adjacent to each other
is the binning area, and the sum of the pixel values of these pixels is the addition
25 output value Q2,1.
[0066]
Further, as indicated by arrow A55 for example, of the 9 pixels in the unit
area, an area including the four pixels G 2,2, G 2,3, G3,2, and G 3,3 that are adjacent to
each other is the binning area, and the sum of the pixel values of these pixels is the
30 addition output value Q2,2.
[0067]
• 13/29
SP342362XXOO
In addition, as indicated by arrow A52 for example, of the 9 pixels in the
unit area, an area including the six pixels GI,l, G I,2, G I,3, G 2,I, G 2,2, and G 2,3 that are
adjacent to each other is the binning area, and the sum of the pixel values of these
pixels is the addition output value q2,3.
5 [0068]
As indicated by arrow A57 for example, of the 9 pixels in the unit area, an
area including the six pixels G I,2, G I,3, G 2,2, G 2,3, G 3,2, and G 3,3 that are adjacent to
each other is the binning area, and the sum of the pixel values of these pixels is the
addition output value q3,1.
10 [0069]
As indicated by arrow A58 for example, of the 9 pixels in the unit area, an
area including the six pixels G 2,I, G 2,2, G 2,3, G 3,I, G 3,2, and G 3,3 that are adjacent to
each other is the binning area, and the sum of the pixel values of these pixels is the
addition output value q3,2.
15 [0070]
Further, as indicated by arrow A59 for example, of the 9 pixels in the unit
area, an area including the six pixels GI,I, G I,2, G 2,1, G 2,2, G 3,I, and G 3,2 that are
adjacent to each other is the binning area, and the sum of the pixel values of these
pixels is the addition output value q3,3.
20 [0071]
Based on the thus-obtained nine addition output values QI,I, QI,2, QI,3, Q2,I,
Q2,2, Q2,3, Q3,I, Q3,2, and Q3,3, the pixel value of each pixel in the unit area to be
ultimately obtained can be determined. Namely, the pixel values PI,I to P3,3 of
pixels GI,I to G 3,3, respectively, in the unit area are calculated.
25 [0072]
For example, among the areas on the image sensor indicated by arrows A51
to A59, the hatched areas are areas where light is incident, and the non-hatched areas
are areas where light is not incident. In this example, light is incident on the whole
area ofpixel GI,l and on half ofthe areas ofpixels G I,2 and G 2,I, respectively.
30 [0073]
Therefore, for the binning area indicated by arrow A51, the addition output
• SP342362XXOO
14/29
value ql,1 is the value of the light components incident on pixels Gl,l, G1,2, and G2,1.
Similarly, for the binning areas indicated by arrows A52, A56, and A59, the addition
output values ql,2, q2,3, and q3,3 are the value of the light components incident on
pixels G1,I. G1,2, and G2,1.
5 [0074]
Further, for the binning areas indicated by arrows A53 and A57, the addition
output values ql,3 and Q3,1 are the value of the light component incident on pixel Gl,2.
For the binning areas indicated by arrows A54 and A58, the addition output values
Q2,1 and Q3,2 are the value of the light component incident on pixel G2,1. In addition,
10 for the binning areas indicated by arrow A55, since no light is incident on the binning
area, no light component is included in the addition output value Q2,2.
[0075]
Thus, even if the unit area is an area including 3 pixels x 3 pixels, using the
respective addition output value Qij (wherein i = 1 to 3, and j = 1 to 3), the respective
15 pixel value Pij can be obtained. Specifically, the simultaneous equation shown in
the following formula (6) can be obtained from the respective addition output value
qij (wherein i = 1 to 3, and j = 1 to 3).
[0076]
[Formula 6]
ql, 1=Wl1 Pl, 1+W12Pl, 2+W13Pl, 3+W14P2, 1+W1SP2, 2+W16P2, 3+W17P3, 1+W18P3, 2+W19P3, 3
ql, 2=W2l Pl, 1+W22Pl, 2+W23Pl, 3+W24P2, 1+W2SP2, 2+W26P2, 3+W27P3, 1+W28P3, 2+W29P3, 3
ql, 3=W3l Pl, 1+W32Pl, 2+W33Pl, 3+W34P2, 1+W3SP2, 2+W36P2, 3+W37P3, 1+W38P3, 2+W39P3, 3
q2, 1=W4l Pl, 1+W42Pl, 2+W43Pl, 3+W44P2, 1+W4SP2, 2+W46P2, 3+W47P3, 1+W48P3, 2+W49P3, 3
q2, 2=WSl Pl, 1+WS2Pl, 2+WS3Pl, 3+WS4P2, 1+WSSP2, 2+WS6P2, 3+WS7P3, 1+WS8P3, 2+WS9P3, 3
q2, 3=W6l Pl, 1+W62Pl, 2+W63Pl, 3+W64P2, 1+W6SP2, 2+W66P2, 3+W67P3, 1+W68P3, 2+W69P3, 3
q3, 1=W7l Pl, 1+W72Pl, 2+W73Pl, 3+W74P2, 1+W7SP2, 2+W76P2, 3+WnP3, 1+W78P3, 2+W79P3, 3
q3, 2=W8l Pl, 1+W82Pl, 2+W83Pl, 3+W84P2, 1+W8SP2, 2+W86P2, 3+W87P3, 1+W88P3, 2+W89P3, 3
Q3, 3=W9l Pl, 1+W92Pl, 2+W93Pl, 3+W94P2, 1+W9SP2, 2+W96P2, 3+W97P3, 1+W98P3, 2+W99P3, 3
20 • • • (6)
•
[0077]
15/29
SP342362XXOO
In formula (6), similar to the case of formula (1), each addition output value
qij gives the same weighting to the pixel value Pij for all ofthe pixels in the unit area.
Ws,t (wherein s = 1 to 9, and t = I to 9) in formula (6) represents the weighting for
5 each pixel value Pij.
[0078]
When pixel G ij is used in the calculation of addition output value qij, the
weighting Ws,t of each these pixel values Pij is set as "1 ", and when pixel Gij is not
used in the calculation of addition output value qij, the weighting Ws,t of each these
10 pixel values Pij is set as "0".
[0079]
Therefore, if the matrix obtained by arranging the weighting Ws,t shown in
formula (6) in the manner in formula (7) is a weighting matrix w, by solving the
above formula (3) or formula (4), the pixel value Pij of each pixel can be obtained.
15 [0080]
[Formula 7]
WI I W12 W13 W14 W15 W16 W17 W18 W19
W21 W22 W23 W24 W25 W26 W27 W28 W29
W31 W32 W33 W34 W35 W36 W31 W38 W39
W41 W42 W43 W44 W45 W46 W41 W48 W49
W51 W52 W53 W54 W55 W56 W51 W58 W59
W61 W62 W63 W64 W65 W66 W61 W68 W69
Wl1 Wn W13 W14 W15 W16 Wn W18 W19
W81 W82 W83 W84 W85 W86 W81 W88 W89
W91 W92 W93 W94 W95 W96 W91 W98 W99
[0081]
11111111
101 1 0 000
o 1 101 1 000
000110110
o 0 0 0 1 1 0 1 1 =w
111111000
011011011
000 1 1 1 111
110110110
• (7)
Further, in such a case, the horizontal vector in which the addition output
20 values ql,), ql,2, ql,3, q2,), q2,2, q2,3, q3,), q3,2, and Q3,3 are arranged in order is used as a
• SP342362XXOO
16/29
matrix q. In addition, the horizontal vector in which the pixel values PI,\' PI,2, PI,3,
P2,I, P2,2, P2,3, P3,\' P3,2, and P3,3 are arranged in order is used as a matrix p.
[0082]
10
Therefore, the following formula (8) can be obtained by performing the
5 calculation of formula (4) in the example illustrated in FIG. 2. Based on the
calculation of this formula (8), the pixel value Pij (wherein i = 1 to 3, and j = 1 to 3)
that is the ultimate determination target is obtained.
[0083]
[Formula 8]
Pl, 1=ql. 1+q2, 2-q3, l-q3. 2
Pl,2=-ql.l-q2.l-q2.2+q3.l+q3.2+q3,3
Pl,3=ql.l+q2,l-q3.2-q3.3
P2, 1= -ql. l-ql, 3-q2. 2+q2. 3+q3. 1+q3. 2
P2, 2=ql, 1+ql, 2+ql, 3+q2. 1+q2. 2-q2. 3-q3, l-q3. 2-q3, 3
P2,3=-ql,l-ql,2-q2,l+q2,3+q3,2+q3.3
P3, 1=ql, 1+Ql, 3-Q2, 3-Q3, 1
P3, 2= -Ql, l-Ql, 2-Ql, 3+Q2, 3+Q3, 1+Q3, 3
P3, 3=Ql, 1+Ql, 2-Q2, 3-Q3. 3 • • • (8)
[0084]
Note that the combinations of the pixels forming the area serving as the unit
area and the binning areas, for example, may be determined in an arbitrary manner.
However, the following description will be based on the 2 pixel x 2 pixel area
15 illustrated in FIG. 1 as the unit area, and the areas indicated by the arrows All to A24
as the binning areas.
[0085]
[Image Processing Apparatus Configuration Example]
Next, a specific embodiment to which the present technology is applied will
20 be described. FIG 3 is a diagram illustrating a configuration example of an
embodiment of an image processing apparatus to which the present technology is
•
applied.
[0086]
17/29
SP342362XXOO
An image processing apparatus 11 illustrated in FIG. 3 is configured from a
light-receiving portion 21, a conversion unit 22, a binning unit 23, and a calculation
5 unit 24.
[0087]
The light-receiving portion 21, which is formed from pixels that receive and
photoelectrically convert light incident from an object, and output an electric signal
based on the strength of the received light, supplies an electric signal (output value)
10 obtained from each pixel to the conversion unit 22. For example, in the example
illustrated in FIG. 1, the area including the four pixels GI,I, Gi ,2, G2,i, and G2,2
provided on the light-receiving portion 21 serves as one unit area, and a plurality of
unit areas are provided on the light-receiving portion 21. Namely, the lightreceiving
portion 21 is divided into a plurality of unit areas.
15 [0088]
The data from the electric signal (output value) output from each pixel of
the light-receiving portion 21 is an image signal of an image captured by the lightreceiving
portion 21.
[0089]
20 The conversion unit 22 converts the electric signals supplied from each of
the pixels ofthe light-receiving portion 21 from an analog signal into a digital signal,
and supplies the converted signals to the binning unit 23. For example, the
conversion unit 22 is configured from a plurality of AID (analog/digital) conversion
circuits provided for each pixel of the light-receiving portion 21. Each AID
25 conversion circuit digitizes the output value supplied from the connected pixel, and
outputs the digitized value to the binning unit 23.
[0090]
The binning unit 23 calculates the addition output value of each binning
area determined for the unit area based on the output value of each pixel supplied
30 from the conversion unit 22, and supplies the calculated value to the calculation unit
24.
•
[0091]
18/29
SP342362XXOO
The calculation unit 24 is configured from a plurality of adders. For
example, in the example illustrated in FIG 1, an AID conversion circuit as the
conversion unit 22 is provided for each of the four pixels in the unit area serving as
5 the light-receiving portion 21. Each of these four AID conversion circuits is
connected to one adder as the binning unit 23. Each adder records the weighting
matrix w shown in formula (2), calculates the addition output value from the output
value of each pixel supplied from the AID conversion circuit based on this weighting
matrix w, and supplies the calculated value to the calculation unit 24.
10 [0092]
Further, the light-receiving portion 21, conversion unit 22, and binning unit
23 are provided on one image sensor, such as a CCD (charge-coupled device). Note
that the conversion unit 22 and the binning unit 23 may also be provided externally
to the image sensor.
15 [0093]
The calculation unit 24 calculates the pixel value of each pixel based on the
addition output value of the binning area for the unit area supplied from the binning
unit 23, and outputs the thus-obtained image signal of the output image.
[0094]
20 [Description of Image Generation Processing]
In the image processing apparatus 11, the output image can be obtained by
either a simultaneous reading method that outputs the addition output value of each
binning area by capturing an image of the object once, or a successive reading
method that captures an image of the object for each binning area and then outputs
25 the addition output value.
[0095]
For example, in the example illustrated in FIG 1, in a simultaneous reading
method, based on each of the output values (pixel values) supplied at the same time
from the four pixels Gl,1 to G2,2 of the light-receiving portion 21, the binning unit 23
30 performs the calculation of formula (1), and outputs the addition output values ql,1 to
q2,2. Namely, in the simultaneous reading method, the addition output values ql,1 to
• SP342362XXOO
19/29
q2,2 are simultaneously calculated based on the pixel values output from each pixel at
the same time.
[0096]
In contrast, in the successive reading method, for example, first the binning
5 unit 23 calculates only the addition output value ql,1 by performing the calculation of
formula (1) based on each of the pixel values supplied from the four pixels GI,I to
G2,2 of the light-receiving portion 21. Then, the binning unit 23 calculates only the
addition output value ql,2 by performing the calculation of formula (1) based on each
of the pixel values supplied from the four pixels G1,1 to G2,2 of the light-receiving
10 portion 21 at the next time instant.
[0097]
Next, the binning unit 23 subsequently calculates only the addition output
value q2,1 by performing the calculation of formula (1) based on each of the pixel
values supplied from the four pixels GI,I to G2,2 of the light-receiving portion 21.
15 After that, the binning unit 23 further subsequently calculates only the addition
output value q2,2 by performing the calculation of formula (1) based on each of the
pixel values supplied from the four pixels G1,1 to G2,2 of the light-receiving portion
21.
[0098]
20 Thus, in the successive reading method, each addition output value is
calculated based on pixel values output from each pixel at different time instants to
each other. Namely, an image is captured of the object the same number of times
that an addition output value is output. For example, in the example illustrated in
FIG. 1, image capturing is carried out four times, so that an addition output value of
25 each binning area indicated by arrows A21 and A22 is obtained.
[0099]
Next, the image processing performed by the image processing apparatus 11
will be described. This image processing is started when an instruction to generate
an output image is issued by the user.
30 [0100]
In step S11, the image processing apparatus 11 determines whether to
• 20/29
generate an output image based on the simultaneous reading method.
[0101]
SP342362XXOO
If it is determined in step S11 to generate an output image based on the
simultaneous reading method, in step S12, the image sensor captures an image of the
5 object. Namely, each of the pixels forming the light-receiving portion 21
photoelectrically converts light incident from the object, and the output values
obtained from those results are supplied to the conversion unit 22.
[0102]
In step S13, the conversion unit 22 digitizes the output value of each pixel
10 supplied from the light-receiving portion 21, and supplies the digitized value to the
binning unit 23.
[0103]
In step S14, the binning unit 23 calculates an addition output value for each
binning area determined for the unit area based on the output value of each pixel
15 supplied from the conversion unit 22, and supplies the calculated value to the
calculation unit 24.
[0104]
For example, the binning unit 23 simultaneously calculates addition output
values ql,1 to q2,2 by performing the calculation of formula (1) based on a
20 predetermined weighting matrix wand the output value of each pixel. Note that in
the calculation performed in step S14, the actual output value obtained from each
pixel ofthe light-receiving portion 21 is substituted for the pixel value Pij.
[0105]
In step S15, the calculation unit 24 calculates the pixel value Pij (wherein i =
25 1 to 2, and j = 1 to 2) of each pixel in the output image based on each addition output
value qij (wherein i = 1 to 2, andj = 1 to 2) supplied from the binning unit 23.
[0106]
For example, the calculation unit 24 calculates the pixel value Pij by
performing the calculation of formula (4) based on each addition output value qij and
30 a matrix w-I recorded in advance.
[0107]
• 21/29
SP342362XXOO
In step S16, the calculation unit 24 outputs the output image, more
specifically the image signal of the output image, formed from the pixel value Pij of
each pixel obtained from the processing performed in step S15, and then the image
generation processing finishes.
5 [0108]
Further, if it is determined in step S11 to not generate an output image based
on the simultaneous reading method, namely, to generate the output image based on
the successive reading method, the processing proceeds to step S17.
[0109]
10 In step S17, the image sensor captures an image of the object. Namely,
each of the pixels forming the light-receiving portion 21 photoelectrically converts
light incident from the object, and the output values obtained from those results are
supplied to the conversion unit 22.
[0110]
15 In step S18, the conversion unit 22 digitizes the output value of each pixel
supplied from the light-receiving portion 21, and supplies the digitized value to the
binning unit 23.
[0111 ]
In step S19, the binning unit 23 calculates an addition output value for one
20 binning area determined for the unit area based on the output value of each pixel
supplied from the conversion unit 22, and supplies the calculated value to the
calculation unit 24.
[0112]
For example, although basically the same processing is performed in step
25 S19 as in step S14, in step S19, one addition output value not yet calculated from
among ql,1 to q2,2 is calculated.
[0113]
In step S20, the image processing apparatus 11 determines whether the
addition output value for all binning areas has been determined.
30 [0114]
For example, if the addition output values for the four binning areas
• 22/29
SP342362XXOO
indicated by the respective arrows A21 to A24 in FIG 1 have been calculated, the
image processing apparatus 11 determines that the addition output value for all
binning areas has been determined.
[0115]
5 If it is determined in step S20 that the addition output value for all binning
areas has not yet been determined, the processing returns to step S17, and the abovedescribed
processing is repeated. Namely, an addition output value of the binning
area to be processed is calculated from the output value of each pixel obtained from
the next image capture process.
10 [0116]
More specifically, when an image of the object is captured and the output
value of each pixel is output from the light-receiving portion 21, the next image
capture process is started at the light-receiving portion 21.
[0117]
15 On the other hand, if it is determined in step S20 that the addition output
value for all binning areas has been determined, the processing proceeds to step S15.
Then, the processing of steps S15 and S16 is performed, and the image generation
processing finishes.
[0118]
20 As described above, the image processing apparatus 11 determines an
addition output value for a plurality of determined binning areas for a unit area, and
calculates the ultimate pixel value of each pixel from those addition output values.
Thus, by determining the pixel value of each pixel from the addition output value,
noise can be easily reduced and the SIN ratio can be improved even for a compact
25 and inexpensive image sensor.
[0119]
Meanwhile, the above-described series of processes may be performed by
hardware or software. When the series of processes is performed by the software,
programs constituting the software are installed in a computer. Here, the computer
30 includes a computer incorporated into dedicated hardware, or, for example, a general
purpose personal computer or the like which can execute various functions by
•
installing various kinds of programs.
[0120]
23/29
SP342362XXOO
FIG 5 is a block diagram illustrating a hardware configuration example of a
computer which executes the series of processes using a program.
5 [0121]
In the computer, a CPU (Central Processing Unit) 201, a ROM (Read Only
Memory) 202, and a RAM (Random Access Memory) 203 are connected to each
other via a bus 204.
[0122]
10 An input/output interface 205 is further connected to the bus 204. The
input/output interface 205 is connected to an input unit 206, an output unit 207, a
recording unit 208, a communication unit 209, and a drive 210.
[0123]
The input unit 206 may include a keyboard, a mouse, a microphone, an
15 imaging element, or the like. The output unit 207 may include a display, a speaker,
and the like. The recording unit 208 may include a hard disk, a nonvolatile memory,
or the like. The communication unit 209 may include a network interface or the
like. The drive 210 drives a removable medium 211 such as a magnetic disk, an
optical disc, a magneto-optical disc, or a semiconductor memory.
20 [0124]
In the computer with the above-described configuration, the CPU 201 loads
a program recorded in, for example, the recording unit 208 to the RAM 203 via the
input/output interface 205 and the bus 204 and executes the program, thereby
performing the above-described series of processes.
25 [0125]
The program executed by the computer (the CPU 201) may be recorded, for
example, on the removable medium 211 which is a package medium. In addition,
the program may be provided via wired or wireless transmission media such as a
local area network, the Internet, and digital satellite broadcasting.
30 [0126]
In the computer, when the removable medium 211 is mounted on the drive
• 24/29
SP342362XXOO
210, the program may be installed to the recording unit 208 via the input/output
interface 205. In addition, the program may be received by the communication unit
209 via a wired or wireless transmission media and may be installed to the recording
unit 208. Further, the program may be installed to the ROM 202 or the recording
5 unit 208 in advance.
[0127]
The program executed by the computer may be a program for executing the
processes on a time-sequential basis in the order described herein. Alternatively, it
may be a program for executing the processes in parallel or at timing when the
10 process is necessary, e.g., when a process call is made.
[0128]
It should be understood by those skilled in the art that various modifications,
combinations, sub-combinations and alterations may occur depending on design
requirements and other factors insofar as they are within the scope of the appended
15 claims or the equivalents thereof.
[0129]
For example, the embodiments of the present technology may employ cloud
computing in which a single function is distributed to a plurality of devices via a
network and is processed in cooperation.
20 [0130]
In addition, each step described in the above flowchart may be not only
executed by a single device, but may also be distributed to a plurality of devices and
be executed.
[0131]
25 Further, in a case where a single step includes a plurality of processes, the
plurality of processes included in the step may be not only executed by a single
device, but may also be distributed to a plurality of devices and be executed.
[0132]
Additionally, the present technology may also be configured as below.
30 [0133]
[1]
SP342362XXOO
An image processing apparatus including:
a binning unit configured to calculate, based on, as a binning area, an area
including two or more adjacent pixels in a unit area including m pixels in a lateral
direction and n pixels in a perpendicular direction on an image sensor, a sum of
5 output values of the pixels in the binning area as an addition output value for each of
m x n binning areas that are different to each other; and
a calculation unit configured to calculate a pixel value of each pixel in the
binning area based on m x n addition output values obtained for the unit area.
[2]
10 The image processing apparatus according to [1], wherein the calculation
unit is configured to calculate the pixel value of each pixel in the unit area by
calculating q = wp, where a matrix obtained by arranging the m x n addition output
values is q, a matrix obtained by arranging the pixel value of each pixel in the unit
area is p, and a matrix obtained by arranging a weighting of each pixel in the unit
15 area is w.
[3] The image processing apparatus according to [2], wherein the calculation unit is
configured to calculate the pixel value of each pixel in the unit area by calculating p
= w-1q, where w-1is an inverse matrix ofthe matrix w.
[4] The image processing apparatus according to [2] or [3], wherein the matrix w is a
20 matrix obtained by setting a weighting of pixels used to determine the addition
output values as 1, and a weighting of pixels not used to determine the addition
output values as O.
[5] The image processing apparatus according to anyone of [1] to [4], wherein the
binning unit is configured to calculate the sum of digitized output values ofthe pixels
25 as the addition output value.
[0000]
It should be understood by those skilled in the art that various modifications,
combinations, sub-combinations and alterations may occur depending on design
requirements and other factors insofar as they are within the scope of the appended
30 claims or the equivalents thereof
[0000]
• SP342362XXOO
26/29
The present disclosure contains subject matter related to that disclosed in
Japanese Priority Patent Application JP 2012-050212 filed in the Japan Patent Office
on March 07,2012, the entire content ofwhich is hereby incorporated by reference.

SP342362XXOO
What is claimed is:
1. An image processing apparatus comprising:
a binning unit configured to calculate, based on, as a binning area, an area
including two or more adjacent pixels in a unit area including m pixels in a lateral
5 direction and n pixels in a perpendicular direction on an image sensor, a sum of
output values of the pixels in the binning area as an addition output value for each of
m x n binning areas that are different to each other; and
a calculation unit configured to calculate a pixel value of each pixel in the
binning area based on m x n addition output values obtained for the unit area.
10
2. The image processing apparatus according to claim 1, wherein the calculation unit
is configured to calculate the pixel value of each pixel in the unit area by calculating
q = wp, where a matrix obtained by arranging the m x n addition output values is q, a
matrix obtained by arranging the pixel value of each pixel in the unit area is p, and a
15 matrix obtained by arranging a weighting of each pixel in the unit area is w.
3. The image processing apparatus according to claim 2, wherein the calculation unit
is configured to calculate the pixel value of each pixel in the unit area by calculating
p = w-1q, where w-I is an inverse matrix of the matrix w.
20
4. The image processing apparatus according to claim 3, wherein the matrix w is a
matrix obtained by setting a weighting of pixels used to determine the addition
output values as 1, and a weighting of pixels not used to determine the addition
output values as O.
25
5. The image processing apparatus according to claim 4, wherein the binning unit is
configured to calculate the sum of digitized output values of the pixels as the
addition output value.
30 6. An image processing method comprising:
calculating, based on an area including two or more adjacent pixels in a unit
• 28/29
SP342362XXOO
area including m pixels in a lateral direction and n pixels in a perpendicular direction
on an image sensor as a binning area, a product of output values of the pixels in the
binning area as an addition output value for each of m x n binning areas that are
different to each other; and
5 calculating a pixel value of each pixel in the binning area based on m x n
add~tion output values obtained for the unit area..
7. A program that causes a computer to execute processing comprising:
calculating, based on an area including two or more adjacent pixels in a unit
10 area including m pixels in a lateral direction and n pixels in a perpendicular direction
on an image sensor as a binning area, a product of output values of the pixels in the
binning area as an addition output value for each of m x n binning areas that are
~
different to each other; and
calculating a pixel value of each pixel in the binning area based on ill x n
15 addition output values obtained for the unit area.