1. An image pickup apparatus comprising: a control unit generating exposure time control data in which an exposure 5 time of an image sensor is set in pixel region units; and an image sensor inputting the exposure time control data and carrying out image acquisition based on exposure time control in the pixel region units, wherein the control unit successively and sequentially generates the exposure time control data in region units based on luminance information in the pixel region 10 units that form part of a preceding picked-up image and outputs the exposure time control data to the image sensor, and the image sensor inputs the exposure time control data in the region units from the control unit and successively and sequentially uses the inputted exposure time control data to cany out the image acquisition based on the exposure time 15 control in the pixel region units.

2. An image pickup apparatus according to claim 1, wherein the control unit successively and sequentially generates the exposure time control data in the region units based on luminance information in units of at 20 least one line of the preceding picked-up image and outputs the exposure time control data to the image sensor, and the image sensor inputs the exposure time control data in region units from the control unit and successively and sequentially uses the inputted exposure time control data to carry out the image acquisition based on the exposure time control in 25 units of at least one line.

3. An image pickup apparatus according to claim 1, wherein the control unit generates the exposure time control data by calculation that refers to only one block as a pixel region that is a unit for acquiring 30 luminance information of the preceding picked-up image, and outputs the exposure time control data to the image sensor, and the image sensor carries out the image acquisition based on the exposure time control that uses the exposure time control data.

4. An image pickup apparatus according to claim 1, 5 wherein the control unit generates first exposure time control data calculated by referring to only one block as a pixel region that is a unit for acquiring luminance information of the preceding picked-up image and second exposure time control data calculated by referring to a plurality of blocks and outputs the first exposure time control data and the second exposure time control data to the image sensor, and 10 the image sensor carries out the image acquisition based on the exposure time control that selectively uses one of the first exposure time control data and the second exposure time control data.

5. An image pickup apparatus according to claim 4, 15 wherein the image sensor is operable, when carrying out the image acquisition selectively using the first exposure time control data and the second exposure time control data, to carry out a reset process of accumulated pixel charge to set a selectively used exposure time. 20 6. An image pickup apparatus according to claim 1, wherein the control unit successively and sequentially generates the exposure time control data in the region units based on luminance information in pixel region units that form part of an image frame N that is the preceding picked-up image and outputs the exposure time control data to the image sensor, and 25 the image sensor inputs the exposure time control data in the region units from the control unit and successively and sequentially uses the inputted exposure time control data to carry out exposure time control in the pixel region units of an image fiame N+1 that is a consecutively picked-up image following the image frame N. 30

7. An image pickup apparatus according to claim 1, further comprising: a high dynamic range processing unit generating a high dynamic range image based on pixel information of long exposure pixels and short exposure pixels.

8. An image pickup apparatus control method executed by an image pickup apparatus 5 comprising: generating, by way of a control unit, exposure time control data in which an exposure time of an image sensor is set in pixel region units; and inputting, by way of an image sensor, the exposure time control data and carrying out image acquisition based on exposure time control in the pixel region 10 units, wherein, in the step of generating, the exposure time control data is successively and sequentially generated in region units based on luminance information in the pixel region units that form part of a preceding picked-up image and the exposure time control data is outputted to the image sensor, and in the step of inputting, the exposure time control data in the region units is inputted from the control unit and the inputted exposure time control data is successively and sequentially used to carry out the image acquisition based on the exposure time control in the pixel region units. 20 9. A program causing an image pickup apparatus to carry out an image pickup control process comprising: generating, by way of a control unit, exposure time control data in which an exposure time of an image sensor is set in pixel region units; and inputting, by way of an image sensor, the exposure time control data and 25 carrying out image acquisition based on exposure time control in the pixel region units, wherein, in the step of generating, the exposure time control data is successively and sequentially generated in region units based on luminance information in the pixel region units that form part of a preceding picked-up image 30 and the exposure time control data is outputted to the image sensor, and in the step of inputting, the exposure time control data in the region units is inputted fiom the control unit and the inputted exposure time control data is successively and sequentially used to carry out the image acquisition based on the exposure time control in the pixel region units. 5 Dated this 06/06/2012 ATTORNEY FOR THE AP@ICANT[S]

The present disclosure relates to an image pickup apparatus, an image pickup
apparatus control method, and a program. In more detail, the present disclosure
relates to an image pickup apparatus, an image pickup apparatus control method, and
a program that carry out exposure control in region units.
5 [0002]
An example of the configuration and image pickup process of a CMOS image
sensor as one example of an image sensor is described below with reference to FIG
1. FIG 1 is a diagram showing the partial configuration of an image sensor (CMOS
image sensor) 101. The image sensor (CMOS image sensor) 101 includes a vertical
10 scan circuit 102, a horizontal scan circuit 103, and a plurality of pixels 104 disposed
in an array.
[0003]
When an exposure process is carried out to pick up an image of a subject,
charge accumulates in a photodiode inside each of the pixels 104. The charge
15 accumulated in the photodiode in each pixel is outputted via an amp transistor and a
transfer transistor to a vertical signal wire 113. The signal current outputted to the
vertical signal wire 113 is also supplied to the horizontal scan circuit 103 and, after
being subjected to specified signal processing, is outputted via a signal output wire
1 14 to the periphery.
20 [0004]
Since each vertical signal wire 113 is commonly connected to pixels that are
aligned in the vertical direction, to read out the signals of the individual pixels, it is
necessary to output the signal of only one pixel at a time to the vertical signal wire
113.
25 [0005]
With the image sensor (CMOS image sensor) 101, signals can be read out
separately from the individual pixels for example by first reading out signals from
the respective pixels 104d aligned on the bottom line as shown in FIG. 2A, then
reading out signals from a line of pixels 104c as shown in FIG 2B, and after that
30 reading out signals while successively changing the line being read out. As one
example, the control signals for such pixel reads are outputted fkom horizontal reset

[OO lo]
For this reason, in regions where the subject is bright, pixel values of pixels
where the exposure time is reduced are acquired as effective pixel values.
Meanwhile, in regions where the subject is dark, pixel values of pixels where the
5 exposure time is increased are acquired as effective pixels. One method of
expanding the dynamic range is to generate an output image by combining regions
with reduced and increased exposure times. Note that when outputting the final pixel
values, a pixel value adjusting process is carried out based on the respective exposure
times.
10 [OOll]
In Japanese Laid-Open Patent Publication No. 201 0-1 36205 and Jenwei Gu et
al., "Coded Rolling Shutter Photography: Flexible Space-Time Sampling",
Computational Photography (ICCP), 2010, a technique for expanding the dynamic
range of picked-up images by setting different exposure times for each line of pixels
15 on an image sensor is disclosed. One example is a configuration that sets a short
exposure line and a long exposure line alternately for the lines of pixels on an image
sensor
[OO 1 21
For example, Publication No. 201 0-136205 and the Jenwei Gu et al. article
20 disclose a method of carrying out exposure time control in line units to set the
exposure time in line units in keeping with the brightness of pixels on a screen. The
configuration in the Publication No. 2010-136205 and the Jenwei Gu et al. article
carries out exposure time control in units of image regions with a configuration that
uses an electronic shutter.
25 [0013]
An algorithm that adaptively sets the exposure time in keeping with the
luminance of the subject has been described for example in Shuji Shimizu, et al., "A
New Algorithm for Exposure Control Based on Fuzzy Logic for Video Cameras",
IEEE Transactions on Consumer Electronics, Vol. 38-3, (1992). In this way,
30 configurations for controlling the exposure time in units of regions, such as lines, in
keeping with the brightness of a subject have already been disclosed. However, to
carry out exposure control in keeping with subject luminance, it is necessary to
decide the exposure time in pixel units before the start of image pickup. Here, there
is a problem in that a delay is produced between the following two processes.
Process 1 : a process of acquiring a preceding image to be used in a process
5 that decides the exposure time in pixel units, and
Process 2: subsequent to Process 1, an image pickup process that carries out
exposure control that reflects the information produced by Process 1.
[00 1 41
The longer the delay mentioned above, the greater the difficulty in tracking
10 changes in brightness. As a result, there is the problem that images that are too
bright or too dark will be acquired for some time after a change in brightness. In
particular, if exposure control is carried out for respective regions such as line units
or pixel units, compared to when exposure control is carried out over the entire
screen as in the past, it becomes necessary to cany out more severe exposure control.
15 When a single exposure time is decided for the entire screen as in the past, local
movement of objects cause very little change in brightness. However, when
exposure control is carried out on a region-by-region basis, it is necessary to track
such local movement of objects, with the delay in such tracking having a larger effect
than before.
20 [0015]
The delay in reflecting movement in exposure control will now be described
with reference to FIGS. 3 and 4. In FIGS. 3 and 4, time is shown on the horizontal
axis and lines of pixels are shown on the vertical axis. As one example, in FIG 3, the
charge read timing shifts in line units as shown by the dotted lines 15 la, 15 1 b in the
25 drawing.
[00 1 61
The plurality of rectangular blocks shown in FIG. 3 show the exposure time
of a single picked-up image frame, and represent the exposure time of line block
units that are each composed of a line or a plurality of lines. The exposure process
30 starts immediately after the timing shown by the read line 151a for the preceding
picked-up image frame. As shown by the read line 151% the exposure start time
gradually shifts in line units. As shown by the graph in FIG 3, exposure begins
earlier for upper lines but only begins after an increasing delay for lines toward the
bottom. For the line at the very top, the time (tl), is the exposure start time, while
for the line at the very bottom, the time (t2) is the exposure start time.
5 [0017]
The right ends of the plurality of rectangular blocks shown in FIG 3 are the
timing at which a read out process is carried out for the exposed image, with the
accumulated charge of the pixels on each line being read at the timing shown by the
read line 1 5 1 b. In this example, since the exposure end time is approximately equal
10 to the read out process time, as shown by the read line 15 1 b in FIG 3, the read out
process is successively carried out for the pixels on each line starting from the first
line.
[00 1 81
For the line at the very top, the time (t2) is the exposure end time, while for
15 the line at the very bottom, the time (t3) is the exposure end time. Note that in this
example, since the exposure start and exposure end of the respective lines shift by the
same amount in line units, the exposure time for every line is the same.
[00 1 91
In FIG 4, the exposure process and read out timing corresponding to three
20 consecutive image pickup frames (numbered N to N+2) during image pickup of
video images is shown. Afier read out of the final line in frame N ends, read out of
the first line in frame N+l begins. Acquisition of the entire image of frame N
becomes possible at the read out timing of the final line. However, since the
exposure of frame N+l will have already started at such time, it is not possible for
25 exposure control of frame N+l to reflect the image acquired for frame N, and such
control only becomes possible for frame N+2 onwards (i.e., a delay of two frames).
[0020]
However, since the interval from the read out of the final line in frame N to
the start of exposure of the first line in h e N+2 is short, if the calculation of
30 exposure time and communication of control data take some time, it may not be
possible to reflect frame N in the exposure control of frame N+2, resulting in an even
longer delay.
[0021]
In this way, if exposure control is carried out in accordance with the
brightness of a subject, a delay is produced between the following two processes.
5 Process 1: a process of acquiring a preceding image to be used in a process
that decides the exposure time in pixel units, and
Process 2: subsequent to Process 1, an image pickup process that carries out
exposure control that reflects the information produced by Process 1.
Due to such delay, there is the problem that optimal exposure control may not
10 be possible.
SUMMARY
[0024]
The present disclosure aims to provide an image pickup apparatus, an image
15 pickup apparatus control method, and a program capable of exposure time control
with little delay.
[0025]
According to an embodiment of the present disclosure, there is provided an
image pickup apparatus which includes a control unit generating exposure time
20 control data in which an exposure time of an image sensor is set in pixel region units,
and an image sensor inputting the exposure time control data and carrying out image
acquisition based on exposure time control in the pixel region units. The control unit
successively and sequentially generates the exposure time control data in region units
based on luminance information in the pixel region units that form part of a
25 preceding picked-up image and outputs the exposure time control data to the image
sensor. The image sensor inputs the exposure time control data in the region units
from the control unit and successively and sequentially uses the inputted exposure
time control data to carry out the image acquisition based on the exposure time
control in the pixel region units.
30 [0026]
In an embodiment of the image pickup apparatus according to the present
disclosure, the control unit successively and sequentially generates the exposure time
control data in the region units based on luminance information in units of at least
one line of the preceding picked-up image and outputs the exposure time control data
to the image sensor, and the image sensor inputs the exposure time control data in
5 region units from the control unit and successively and sequentially uses the inputted
exposure time control data to carry out the image acquisition based on the exposure
time control in units of at least one line.
[0027]
In an embodiment of the image pickup apparatus according to the present
10 disclosure, the control unit generates the exposure time control data by calculation
that refers to only one block as a pixel region that is a unit for acquiring luminance
information of the preceding picked-up image, and outputs the exposure time control
data to the image sensor, and the image sensor carries out the image acquisition
based on the exposure time control that uses the exposure time control data.
15 [0028]
In an embodiment of the image pickup apparatus according to the present
disclosure, the control unit generates first exposure time control data calculated by
referring to only one block as a pixel region that is a unit for acquiring luminance
information of the preceding picked-up image and second exposure time control data
20 calculated by referring to a plurality of blocks and outputs the first exposure time
control data and the second exposure time control data to the image sensor, and the
image sensor carries out the image acquisition based on the exposure time control
that selectively uses one of the first exposure time control data and the second
exposure time control data.
25 [0029]
In an embodiment of the image pickup apparatus according to the present
disclosure, the image sensor is operable, when carrying out the image acquisition
selectively using the first exposure time control data and the second exposure time
control data, to carry out a reset process of accumulated pixel charge to set a
30 selectively used exposure time.
[0030]
In an embodiment of the image pickup apparatus according to the present
disclosure, the control unit successively and sequentially generates the exposure time
control data in the region units based on luminance information in pixel region units
that form part of an image frame N that is the preceding picked-up image and outputs
5 the exposure time control data to the image sensor, and the image sensor inputs the
exposure time control data in the region units from the control unit and successively
and sequentially uses the inputted exposure time control data to carry out exposure
time control in the pixel region units of an image fkame N+l that is a consecutively
picked-up image following the image fiame N.
10 [0031]
In an embodiment of the image pickup apparatus according to the present
disclosure, the image pickup apparatus further includes a high dynamic range
processing unit generating a high dynamic range image based on pixel information of
long exposure pixels and short exposure pixels.
15 [0032]
According to an embodiment of the present disclosure, there is provided an
image pickup apparatus control method executed by an image pickup apparatus
which includes generating, by way of a control unit, exposure time control data in
which an exposure time of an image sensor is set in pixel region units, and inputting,
20 by way of an image sensor, the exposure time control data and carrying out image
acquisition based on exposure time control in the pixel region units. In the step of
generating, the exposure time control data is successively and sequentially generated
in region units based on luminance information in the pixel region units that form
part of a preceding picked-up image and the exposure time control data is outputted
25 to the image sensor, and in the step of inputting, the exposure time control data in the
region units is inputted from the control unit and the inputted exposure time control
data is successively and sequentially used to carry out the image acquisition based on
the exposure time control in the pixel region units.
[0033]
30 According to an embodiment of the present disclosure, there is provided a
program causing an image pickup apparatus to cany out an image pickup control
process which includes generating, by way of a control unit, exposure time control
data in which an exposure time of an image sensor is set in pixel region units, and
inputting, by way of an image sensor, the exposure time control data and carrying out
image acquisition based on exposure time control in the pixel region units. In the
5 step of generating, the exposure time control data is successively and sequentially
generated in region units based on luminance information in the pixel region units
that form part of a preceding picked-up image and the exposure time control data is
outputted to the image sensor, and in the step of inputting, the exposure time control
data in the region units is inputted fiom the control unit and the inputted exposure
10 time control data is successively and sequentially used to carry out the image
acquisition based on the exposure time control in the pixel region units.
[0034]
Note that the program according to the present disclosure is a program
provided, via a storage medium for example, to an information processing apparatus
15 or a computer system capable of executing various program code. By having the
program executed by a program executing unit in such information processing
apparatus or computer system, processing according to the program is realized.
[0035]
Other objects, features, and advantages of the technology according to the
20 present disclosure should be apparent from the following detailed description of the
embodiments and the appended drawings. Note that the expression "system" in this
specification refers to a logical grouping of a plurality of apparatuses and is not
limited to having the various apparatuses present inside a single housing.
[0036]
25 According to the embodiments of the present disclosure described above, an
exposure time control process with a reduced delay is realized in a configuration that
controls exposure time in pixel region units of an image. More specifically, a control
unit generating exposure time control data in which an exposure time of an image
sensor is set in pixel region units and an image sensor inputting the exposure time
30 control data and carrying out image acquisition based on exposure time control in the
pixel region units are included. The control unit successively and sequentially
generates the exposure time control data in region units based on luminance
information in the pixel region units that form part of a preceding picked-up image
and outputs the exposure time control data to the image sensor. The image sensor
inputs the exposure time control data in the region units fiom the control unit and
5 successively and sequentially uses the inputted exposure time control data to carry
out the image acquisition based on the exposure time control in the pixel region
units. By carrying out a process that successively uses the exposure time control data
in region units, an exposure time control process with a reduced delay is realized.
10 BRIEF DESCRIPTION OF THE DRAWINGS
[0037]
FIG 1 is a diagram useful in explaining the configuration and image pickup
process of an image sensor;
FIGS. 2A and 2B are diagrams useful in explaining the configuration and
15 image pickup process of the image sensor;
FIG. 3 is a diagram useful in explaining an image pickup processing
sequence;
FIG 4 is a diagram usell in explaining an image pickup processing
sequence;
20 FIG. 5 is a diagram useful in explaining the configuration and processing of
an image pickup apparatus;
FIG 6 is a diagram useful in explaining the configuration and image pickup
processing of an image sensor;
FIG 7 is a diagram useful in explaining an image pickup processing
25 sequence;
FIG 8 is a diagram useful in explaining the configuration of an exposure time
calculating process of a control unit;
FIG 9 is a diagram useful in explaining the configuration of an exposure time
calculating process of a control unit;
30 FIGS. 10A to 10C are diagrams useful in explaining an example of an
exposure time setting process carried out by analyzing luminance;
FIG 11 is a diagram useful in explaining the configuration and processing of
an exposure time control unit;
FIG 12 is a diagram useful in explaining the configuration and example
processing of an exposure time latch unit;
5 FIG 13 is a timing chart showing an example of an operation sequence of the
exposure time latch unit shown in FIG 12;
FIG 14 is a diagram useful in explaining an example ~ o ~ g u r a t i oofn a
vertical scan circuit;
FIG 15 is a diagram useful in explaining the processing of a read timing
10 generating unit;
FIG 16 is a diagram useful in explaining the relationship between the read
timing, reset timing, and exposure time of an arbitrary line;
FIGS. 17A and 17B are diagrams useful in explaining an existing exposure
time control sequence;
FIGS. 18A and 18B are diagrams useful in explaining an exposure time
control sequence of an image pickup apparatus according to an embodiment of the
present disclosure;
FIG 19 is a diagram showing an example setting of blocks that are units for
exposure control;
20 FIG 20 is a diagram showing an example setting of blocks that are units for
exposure control;
FIGS. 21A to 21C are diagrams useful in explaining an example of an
exposure control process;
FIGS. 22A and 22B are diagrams useful in explaining a problem with an
25 exposure time control sequence of an image pickup apparatus;
FIG 23 is a diagram useful in explaining the configuration and example
processing of an exposure time latch unit;
FIGS. 24A and 24B are diagrams useful in explaining an example of an
exposure time control sequence of an image pickup apparatus according to an
30 embodiment of the present disclosure; and
FIGS. 25A and 25B are diagrams useful in explaining an example of an
exposure time control sequence of an image pickup apparatus according to an
embodiment of the present disclosure.
DETAILED DESCRIPTION OF THE EMBODIMENT(S)
5 [0038]
Hereinafter, preferred embodiments of the present disclosure will be
described in detail with reference to the appended drawings. Note that, in this
specification and the appended drawings, structural elements that have substantially
the same function and structure are denoted with the same reference numerals, and
10 repeated explanation of these structural elements is omitted.
An image pickup apparatus, an image pickup apparatus control method, and a
program according to the present disclosure will now be described in detail with
reference to the drawings. The description is given in the order indicated below.
1. Configuration and Processing of Image Pickup Apparatus
2. Exposure Time Control Process in Region Units
3. Example Operation of Image sensor according to Exposure Control in
Region Units
4. Comparison of Exposure Control Process According to an Embodiment of
the Present Disclosure and Existing Exposure Control Process
20 5. Modifications
6. Configuration of Exposure Control Based on Process Referring to a
Plurality of Blocks
7. Conclusion
[0039]
1. Configuration and Processing of Image Pickup Apparatus
First, an example configuration of an image pickup apparatus will be
described with reference to FIG 5. As shown in FIG 5, the image pickup apparatus
includes an optical lens 201, an image sensor 202 constructed of a CMOS image
sensor or the like, a high dynamic range (HDR) processing unit 204, a signal
30 processing unit 206, and a control unit 208. The image pickup apparatus shown in
FIG 5 is configured to carry out high dynamic range (HDR) image pickup by picking
up an image while changing the exposure time on a region-by-region basis.
[0040]
The image pickup apparatus is an image pickup apparatus that is capable of
picking up video and/or still images and realizes exposure time control in region
5 units according to the luminance of the subject. When picking up video images, for
example, luminance information is acquired in region units for a preceding image
frame and exposure time control is carried out for the next picked-up frame in region
units based on measured luminance information for such region units. When picking
up still images, luminance information in region units is acquired using an image
10 picked up immediately before the present image to be picked up and when picking
up the present still image, exposure time control in region units is carried out based
on measured luminance information for such region units.
[004 11
Light that is incident via the optical lens 201 is subjected to photoelectric
15 conversion by the image sensor 202 to obtain a picked-up image 203. In the HDR
processing unit 204, the difference in sensitivity due to differences in exposure time
in pixel region units is corrected to obtain an HDR image 205.
[0042]
Note that the exposure time control specifically sets a short exposure time for
20 bright subject regions and a long exposure time for dark subject regions. As a result,
pixel values of the respective pixels of the picked-up image 203 are pixel values in
keeping with the different exposure times. In the HDR processing unit 204, by
referring to the exposure times of the respective pixels, pixel values for a case where
all of the pixels have the same exposure time are calculated and an HDR image 205
25 in which such calculated pixel values are set is generated. The signal processing unit
206 carries out processing that improves image quality and thereby obtains an output
image 207.
[0043]
The control unit 208 carries out control over the various processes of the
30 image pickup apparatus. For example, an optimal exposure time is calculated in
pixel region units based on luminance information of a previously-acquired image
and is outputted to the image sensor 202 as control data 209. The image sensor 202
carries out an image pickup process (charge accumulation) based on such control
data 209 according to exposure time control in region units using an electronic
shutter. Note that the control unit 208 carries out overall control of the various
5 component elements in accordance with a program stored for example in a memory,
not shown.
[0044]
The image sensor 202 is constructed of a CMOS image sensor, for example,
and is an image sensor capable of exposure control in units of specified pixel regions.
10 One specific example configuration is shown in FIG 6. FIG. 6 is a diagram showing
the configuration of part of the image sensor (CMOS image sensor) 202. As shown
in FIG 6, the image sensor (CMOS image sensor) 202 is constructed of an exposure
time control unit 202a into which the control data 209 outputted from the control unit
208 is inputted, a vertical scan circuit 202b, a horizontal scan circuit 202c, and a
15 plurality of pixels 202d disposed in an array.
[0045]
Due to an exposure process carried out for image pickup of a subject, charge
accumulates in a photodiode in each pixel 202d. The charge accumulated in the
photodiode in each pixel is outputted via an amp transistor and a transfer transistor to
20 a vertical signal wire 202g. The signal current outputted to the vertical signal wire
202g is also supplied to the horizontal scan circuit 202c and, after being subjected to
specified signal processing, is outputted via a signal output wire 202h to the
periphery.
[0046]
25 As shown in FIG 7, for example, the image sensor 202 is a CMOS image
sensor where the exposure time can be controlled arbitrarily in line units. The
configuration of such image sensor 202 is described for example in the Jenwei Gu et
al. article mentioned earlier.
[0047]
30 As described earlier, the HDR processing unit 204 corrects differences in
sensitivity due to differences in the exposure time in pixel region units to obtain the
HDR image 205. More specifically, by multiplying the respective pixel values of the
picked-up image 203 by the reciprocal of the exposure time in each region, an image
with linear luminance characteristics is generated and set as the HDR image 205.
[0048]
5 In the image pickup apparatus shown in FIG 5, the control unit 208
successively and sequentially generates exposure time control data (i.e., the control
data 209) in region units based on luminance information in units of pixel regions
that form part of a preceding picked-up image and outputs such exposure time
control data to the image sensor 202. The image sensor 202 inputs the exposure time
10 control data in region units from the control unit 208 and successively and
sequentially uses the inputted exposure time control data to carry out image
acquisition based on exposure time control in pixel region units.
[0049]
2. Exposure Time Control Process in Region Units
15 Next, an exposure time control process in region units in keeping with subject
luminance carried out by the control unit 208 will be described with reference to
FIGS. 8 to 10C. FIG 8 is a diagram showing an example configuration for carrying
out the exposure time control process executed by the control unit 208. A luminance
calculating unit 21 1 inputs the HDR image 205 generated by the HDR processing
20 unit 204 based on the preceding picked-up image and calculates the luminance in
pixel region units from the input image.
[0050]
As one example, as shown in FIG 9, a 2x2 pixel region that constructs the
HDR image 205 is set as a pixel region unit 215 for luminance calculation. A
25 maximum value (MAX) of R, and B (where R is the pixel value of the R (red)
pixel, G is an average value (Gr+Gb)/2 of the G (green) pixels, and B is the pixel
value of the B (blue) pixel) is found, and is set as the luminance L of the pixel region
unit 2 1 5 for luminance calculation.
[005 11
30 A block luminance statistic calculating unit 212 calculates a luminance
statistic such as that shown in FIG 10B for each block composed of a plurality of
lines as shown in FIG 10A. In FIG. 10B the vertical axis shows lines and the
horizontal axis shows a statistic that represents the brightness of each line block.
[0052]
A block exposure time calculating unit 21 3 calculates an exposure time (Texl
5 to TexM) to be set for each line block in FIG 10C based on the luminance statistic of
each block. By setting a short exposure time in bright regions, saturation of pixel
values is suppressed. By setting a long exposure time in dark regions, control that
improves the S/N ratio is carried out. Such processing makes it possible to pick up
images with a wide dynamic range and a favorable S/N ratio.
10 [0053]
According to the present configuration, when exposure control is carried out
for each region, it is possible to decide the exposure time by merely referring to the
periphery of such region. That is, to calculate the exposure time Texl of block 1, for
example, only image information of block 1 is used. It is possible to set the exposure
15 time of each block using only information in block units without carrying out a
process that refers to the entire image of one fhme or refers to an image region that
covers a large part of such frame. This means that the control unit 208 can output the
control data 209 immediately and successively from the blocks for which the
calculation of exposure time has been completed to the image sensor 202 constructed
20 for example of a CMOS image sensor.
[0054]
That is, as described earlier, the control unit 208 sequentially and
successively generates exposure time control data in block units based on luminance
information in pixel regions, that is, block units (for example, line units or units of a
25 plurality of lines) that form part of a preceding picked-up image and outputs the
exposure time control data to the image sensor 202. The image sensor 202 inputs the
exposure time control data in block units from the control unit 208 and carries out
image acquisition based on exposure time control in block units that successively and
sequentially uses the inputted exposure time control data.
30 COO551
3. Example Operation of Image sensor according to Exposure Control in
Region Units
Next, an example operation where the image sensor 202 inputs the control
data 209 outputted from the control unit 208 and carries out an image pickup process
by carrying out exposure control in region units will be described. FIG 11 is a
5 diagram useful in explaining an example configuration of the exposure time control
unit 202a of the image sensor 202. The exposure time control unit 202a of the image
sensor 202 inputs the control data 209 composed of exposure time setting
information in region units from the control unit 208 and controls the vertical scan
circuit 202b to set the exposure time in image region units (block units (for example,
10 line units)).
[0056]
A decoder 221 inputs the control data 209 composed of the exposure time
setting information in region units from the control unit 208 and generates exposure
time data 222 in block units.
15 A line counter 226 uses a vertical sync signal 224 and a horizontal sync signal
225 generated by a timing generating unit 223 to generate the line number that shows
the present processing target.
[0057]
An exposure time latch unit 227 inputs the line number generated by the line
20 counter 226, generates exposure time latch timing for each block (pixel region), and
outputs the exposure time data 222 to the vertical scan circuit 202b at different
timing for each block (pixel region).
[0058]
The vertical scan circuit 202b carries out electronic shutter control in block
25 (pixel region) units in accordance with the exposure time data in block (pixel region)
units outputted by the exposure time latch unit 227. More specifically, the vertical
scan circuit 202b generates a line reset and line selection signal in line units, for
example, and carries out electronic shutter control that realizes exposure time control
in line units.
30 [0059]
FIG 12 is a diagram showing an example of the detail configuration of the
exposure time latch unit 227 of one block. The exposure time latch unit 227
corresponding to one block outputs an exposure time control signal for one block to
the vertical scan circuit 202b. As one example, if a block is one line, the exposure
time control process is carried out in line units.
5 [0060]
As shown in FIG 12, the exposure time latch unit 227 fetches exposure time
data (in) 222in in block units generated by the decoder 221 into a storage unit 234
and inputs the present line number 23 1 generated by the line counter 226.
[006 11
10 The exposure time latch unit 227 stores a latch line number 232 of a line
controlled using this particular exposure time latch unit 227 and determines, in a
match detecting unit 233, whether the line number 23 1 inputted fiom the line counter
226 matches the latch line number 232 of the line controlled using this exposure time
latch unit 227. If a match has been detected, exposure time data (out) 222out is
15 outputted from the storage unit 234 to the vertical scan circuit 202b and exposure
time control is carried out in block units in keeping with the exposure time data 222.
Note that when implemented by a logic circuit, the storage unit 234 can be realized
for example by a D flip-flop circuit with an enable port.
[0062]
20 FIG. 13 is a timing chart usefbl in explaining an operation sequence of the
exposure time latch unit 227 shown in FIG 12. The horizontal axis represents time
(t) and therefore changes in the data mentioned below are shown on a time axis.
(a) b'Exposure time data (in) 222in" inputted into the exposure time latch unit
227
25 (b) "Present line number 23 1" inputted into the exposure time latch unit 227
(c) "Latch line number 232" stored in the exposure time latch unit 227
(d) "Match detection signal output" of the match detecting unit 233
(e) Output from storage unit 234 (i.e., exposure time data (out) 222out)
[0063]
30 Note that to simplifl the explanation, the number of lines in each frame is set
at 10. In FIG 13, out of the period where an exposure time setting process is carried
out for lines 0 to 9 of frame N and lines 0 to 9 of frame N+1, the processing sequence
of a single exposure time latch unit 227 used for processing the fifth line is shown.
[0064]
(a) The "exposure time data (in) 222in" inputted into the exposure time latch unit 227
5 is fetched at the timing when (d) the "match detection signal output" of the match
detecting unit 233 detects that (b) the "present line number 231" inputted into the
exposure time latch unit 227 matches (c) the "latch line number 232" stored by the
exposure time latch unit 227, and is outputted fiom the storage unit 234 to the
vertical scan circuit 202b and stored in the storage unit 234 until the same timing in
10 the next fiame.
[0065]
FIG. 14 shows an example configuration of the vertical scan circuit 202b. A
decoder 251 is used to extract the timing data 252 fiom the control data 209 provided
fiom the periphery As one example, information that is necessary for timing
15 purposes, such as the number of output lines of the image, is extracted as the timing
data 252.
[0066]
A read timing generating unit 253 successively outputs read timing to a
horizontal (line) select wire 202f of each line based on the timing data 252. More
20 specifically, as shown in FIG 15 for example, the read timing corresponding to each
line is generated inside the read timing generating unit 253 and the read timing is
successively outputted to the horizontal (line) select wire 202f of each line.
[0067]
A reset timing generating unit 254 inputs read timing corresponding to the
25 respective lines generated by the read timing generating unit 253, calculates a reset
time as an exposure start time in line units based on the exposure time data (out)
222out inputted fiom the exposure time latch unit 227 in order to control the
exposure start time so as to set the exposure time given in the inputted exposure time
data, generates a reset signal in line units, and outputs the reset signal to a horizontal
30 (line) reset wire 202e.
[0068]
FIG 16 shows the relationship between the read timing, the reset timing, and
the exposure time of an arbitrary line. The reset timing (Treset) shown in FIG. 16 is
the exposure start time from which charge accumulation starts for the pixel and the
read timing (Tread) is the exposure end time when the accumulated signal is read
5 out. That is, a relationship where the exposure time (Texp) = (Tread) - (Treset) is
established.
[0069]
Since exposure time data 241 and the read timing have been obtained, the
reset timing generating unit 254 uses the reset timing (Treset) = Tread - Texp
10 relationship to generate the reset timing and outputs a reset signal in accordance with
the reset timing to the horizontal (line) reset wire 202e on a line-by-line basis.
[0070]
4. Comparison of Exposure Control Process According to an Embodiment of
the Present Disclosure and Existing Exposure Control Process
15 Next, the exposure control process according to an embodiment of the present
disclosure and an existing exposure control process will be compared with reference
to FIGS. 17A to 18B. FIGS. 17A and 17B are diagrams useful in explaining an
example operation of an image sensor (CMOS image sensor) according to existing
exposure control. FIGS. 18A and 18B are diagrams useful in explaining an example
20 operation of the image sensor (CMOS image sensor) of the image pickup apparatus
according to an embodiment of the present disclosure.
[007 11
First, an example operation of an image sensor (CMOS image sensor)
according to existing exposure control will be described with reference to FIGS. 17A
25 and 17B. FIGS. 17A and 17B show an exposure control sequence carried out during
an image pickup process for three frames numbered N to N+2 that are consecutively
picked up. In the graph in FIG 17A showing the exposure timing the vertical axis
shows lines, the horizontal axis shows time, and exposure time in line units is shown
for each line as a rectangular region. As described earlier with reference to FIGS. 3
30 and 4, the exposure time is set with a delay of a specified time from lines at the top to
lines at the bottom.
[0072]
FIG. 17B shows an exposure control process sequence on a time axis for the
exposure control process that uses the image of frame N. The processes numbered
301 to 305 are processes executed as time passes. The content of such processes is
5 as follows.
Measurement acquisition 301: the timing at which the image of frame N is
outputted.
Exposure time calculation 302: the timing at which the exposure time is
calculated based on the image acquired by the measurement acquisition 30 1.
10 [0073]
Exposure time transmission 303: the timing at which the exposure time
calculated in the exposure time calculation 302 is outputted to the image sensor
(image sensor).
Exposure time latch timing 304: the timing at which the exposure time data
15 inputted by the exposure time transmission 303 is latched inside the image sensor
(image sensor). Here, the latch timing is generated before the start of exposure of the
first line.
Exposure time control according to reset control 305: reset timing is
generated based on the data latched in the exposure time latch timing 304 and is the
20 period during which exposure control is carried out.
[0074]
Although the processing in processes 301 to 305 is exposure time control
carried out based on the picked-up image of frame N, image pickup of the frame N+l
will have already started at the start time of the exposure time control according to
25 reset control 305. Accordingly, the exposure time control according to reset control
305 is not applied to the image pickup process of fiame N+1. As a result, image
acquisition 306 according to image pickup with such exposure control is carried out
for the image of frame N+2.
100751
30 In this way, in the exposure time control shown in FIG 17, an image pickup
process according to exposure time control that uses exposure time control data
generated based on the picked-up image of frame N cannot be carried out for the
fiame (frame N+1) picked up immediately afterward and is instead carried out for the
frame N+2 that follows two frames after frame N. In this way, with the existing
method, a delay of at least two frames was produced.
5 [0076]
FIGS. 18A and 18B are diagrams useful in explaining an example operation
of an image sensor (CMOS image sensor) in the image pickup apparatus according to
the present embodiment. In the same way as FIGS. 17A and 17B, FIG. 18A and 18B
show an exposure control sequence carried out during an image pickup process for
10 three frames numbered N to N+2 that are consecutively picked up.
[0077]
In the graph in FIG 18A showing the exposure timing the vertical axis shows
lines, the horizontal axis shows time, and exposure time in line units is shown for
each line as a rectangular region.
15 [0078]
FIG 18B shows an exposure control process sequence on a time axis in
region (block) units for the exposure control process that uses the image of frame N.
In the processing according to the present embodiment, by carrying out calculation,
transmission, and latching for exposure time control separately for specified unit
20 regions (in the present embodiment, lines) of an image, the delay time is reduced.
[0079]
In FIGS. 18A and 18B, exposure control of the frame N+l that is the
following image pickup frame is realized using the image of frame N that is the
preceding image pickup frame. Such exposure control is carried out in units of pixel
25 regions (blocks). In the present embodiment, exposure control is carried out in line
units. In FIGS. 18A and 18B, as representative examples, processing examples
where exposure control (the setting of exposure time) is carried out for block 1 (line
1) of frame N+l based on the image information of block 1 (line 1) of frame N and
exposure control (the setting of exposure time) is carried out for block M (line M) of
30 b e N+l based on the image information of block M (line M) of frame N are
shown.
[OOSO]
In FIG 18B, in the same way as FIG 17B, an exposure control processing
sequence on a time axis is shown for an exposure control process that uses the image
of frame N. The processes numbered 311 to 316 are processes executed as time
5 passes. Note that the processing proceeds with the same sequence (processes 3 11 to
3 16) for both block 1 and block M. The content of such processes is as follows.
[OOS 11
Measurement acquisition 311: the timing at which the image of a block in
frame N is outputted.
10 Exposure time calculation 312: the timing at which the exposure time of a
block (line) is calculated based on the image (one block in frame N, in the present
embodiment the image of one line unit) acquired by the measurement acquisition
311.
[0082]
15 Exposure time transmission 313: the timing at which the exposure time of a
block calculated in the exposure time calculation 3 12 is transmitted.
Exposure time latch timing 3 14: the timing at which the exposure time data of
one processing block transmitted by the exposure time transmission 3 13 is latched
inside the image sensor (image sensor). Here, the latch timing is generated before
20 the start of exposure of such block.
[0083]
Exposure time control 315: reset timing is generated in block (line) units
based on the exposure time latch timing 314 and is the timing at which exposure
control (control to start exposure) is carried out.
25 Image acquisition 316: the time at which the image of the block (line)
currently subject to control is obtained by the exposure control in block (line) units
carried out by the exposure control 3 15 described above.
[0084]
For block 1, block M, and all of the other blocks (lines) shown in FIG 18A,
30 processing is carried out according to the same sequence described above.
[0085]
With the configuration according to the present embodiment shown in FIGS.
18A and 18B, unlike FIGS. 17A and 17B, the exposure time control data is
calculated for specified units, that is pixel region (or "block", or line in the present
embodiment) units and exposure control successively starts in pixel region units.
5 According to such processing, a configuration that carries out exposure control
which applies exposure control data based on the preceding picked-up image to the
following image pickup frame is realized.
[0086]
In the exposure time control shown in FIGS. 18A and 18B, the image pickup
10 process according to exposure time control which uses the exposure time control data
generated based on the picked-up image of frame N can be carried out for the
immediately following image pickup frame (frame N+l), which means it is possible
to carry out exposure control with no delay.
[0087]
15 To realize the control shown in FIGS. 18A and 18B, the image sensor 202 has
a configuration that is capable of exposure control for each image region (or block
(for example, line)) and also carries out control having fetched control data with
different timing for each specified unit.
[0088]
20 5. Modifications
Modifications to the embodiment described above will now be described. In
the embodiment described above, an example configuration of an image pickup
apparatus was described with reference to FIG 5. The configuration shown in FIG 5
is merely one example and other configurations are also possible. As one example, a
25 configuration may be used where all of the processing shown in FIG. 5 is carried out
inside the image sensor.
[0089]
The image pickup apparatus according to the present embodiment is
configured to carry out exposure control that sets a different exposure time in each
30 region and as one example an example configuration that carries out exposure
control in line units was described above with reference to FIG 7. However, the
regions (blocks) as exposure control units are not limited to line units or units of a
plurality of lines, and may be set differently.
[0090]
For example, as shown in FIG. 19, a configuration that carries out exposure
5 control with blocks that are rectangular regions as units may be used. Also, as shown
in FIG 20, a configuration may be used which, in addition to exposure control based
on the preceding picked-up image, sets short exposure blocks and long exposure
blocks to cyclically carry out control over a plurality of sensitivities and generates an
HDR image based on the resulting image. In addition, a configuration may be used
10 which, in addition to exposure control based on the preceding picked-up image,
carries out a short exposure image pickup process and a long exposure image pickup
process for each region as shown in FIGS. 21A to 21C and generates an HDR image
based on the resulting images.
[009l]
Also, in the embodiment described above, as an example of an exposure
control configuration of an image sensor, an example configuration of the exposure
time latch unit 227 was described with reference to FIG 12. In the above
description, an example is described where the exposure time latch unit 227 stores
the latch line number 232 showing the block to be controlled by this particular
20 exposure time latch unit 227. However, the latch line number 232 does not have to
be fixed and may be set so as to change in accordance with a luminance statistic for
an image. Such changing process makes it possible to dynamically change the block
being controlled.
[0092]
25 6. Configuration of Exposure Control Based on Process Referring to a
Plurality of Blocks
In the embodiment described above, a configuration is used where the
exposure time is set based on luminance information of the preceding picked-up
image in block (for example, line) units as described above with reference to FIG 18
30 and exposure time control over corresponding blocks in the next picked-up image is
carried out.
[0093]
That is, a configuration was described where the control unit 208
successively outputs exposure time control data to the image sensor 202 in block
units based on luminance information in single block units of the preceding picked-
5 up image and the image sensor 202 sequentially and successively uses the exposure
time control data in single block units to carry out image acquisition based on
exposure time control in block units.
[0094]
That is, in the example shown in FIG 18, for blocks 1 to M set in an image,
10 exposure control is carried out corresponding to the blocks so that exposure control
of an image region of block 1 of frame N+l is carried out according to exposure
control data based on the image region of block 1 of frame N, exposure control of an
image region of block 2 of frame N+l is carried out according to exposure control
data based on the image region of block 2 of frame N, . . . , and exposure control of an
15 image region of block M of frame N+l is carried out according to exposure control
data based on the image region of block M of frame N.
[0095]
However, if exposure control is carried out in this way by referring to only
local information, in cases such as when a bright object has made a large movement,
20 the same subject may not be present in the same block region in frame N and fiame
N+1. In such a case, an exposure time set based on a preceding picked-up image is
no longer appropriate, and problems such as the saturation of pixel values can occur.
[0096]
To solve such problem, in some cases it is effective to set the exposure time
25 by referring not only to local information but also a wider area, for example, a
plurality of blocks. That is, by using a configuration that sets-the exposure time by
referring to a plurality of blocks, it is possible to precisely predict changes in
luminance. However, when a wide area is referred to, there is the problem that
image acquisition is time-consuming.
30 [0097]
The above problem will first be described with reference to FIGS. 22A and
22B and then a configuration and processing sequence that solves the problem will
be described with reference to FIGS. 23A to 25B.
[0098]
First, the above problem, that is, the problem that image acquisition is time-
5 consuming when a wide area is referred to will be described with reference to FIGS.
22A and 22B. FIG 22A is a timing chart for a case where a setting is made to refer
to a plurality of blocks, that is, blocks 1 to 3 when calculating the exposure control
data of block 1 in FIG 18A. FIG 22B shows an exposure control sequence carried
out during an image pickup process for two frames numbered N and N+l that are
10 consecutively picked up.
[0099]
In the graph in FIG 22A showing the exposure timing the vertical axis shows
lines, the horizontal axis shows time and exposure time in line units is shown for
each line as a rectangular region. FIG 22B shows an exposure control process
15 sequence on a time axis in region (block) units for the exposure control process that
uses the image of frame N. Note that the numbers 1 to M are identifiers of pixel
regions (or blocks, in the present example "lines") in a picked-up image.
[O 1 001
In the same way as the example described with reference to FIGS. 18A and
20 18B, the settings shown in FIGS. 22A and 22B have exposure control successively
started in specified units, that is, pixel region (or "block", in the present example,
line) units. By carrying out such processing, this configuration carries out exposure
control by applying exposure control data based on a preceding picked-up image to
the following image pickup frame.
25 [OlOl]
However, the calculation process for exposure time control data
corresponding to each block refers to a plurality of blocks. That is, the exposure time
of one block is decided based on the image information of a plurality of blocks. The
example shown in FIG 22B is an example where a process is carried out to decide
30 the exposure time of one block in the following image pickup frame by referring to
three blocks in the preceding image pickup frame.
[O 1021
In FIG 22B, in the same way as FIG. 18B, an example of an exposure control
process for block 1 (line 1) that is one block (line) is shown. Processes 311 to 316
shown in FIG. 22B are processes that are the same as processes 3 11 to 316 in FIG
5 18B described earlier. However, in the example shown in FIG. 22B, the exposure
time of one block of the next image pickup fiame is decided with reference to three
blocks of the preceding image pickup fi-ame. The content of the respective processes
is described below.
[0 1031
10 Image acquisition 3 11 : the timing at which the image of a block of h e N is
outputted. At this timing, an image of three consecutive blocks, that is, pixel region
1 (block 1) to pixel region 3 (block 3), that are an image region that forms part of the
picked-up image N is outputted from the image sensor.
Exposure time calculation 3 12: the timing at which the exposure time of one
15 block (line) of a following fiame subjected to exposure control is calculated based on
the image (the three blocks of frame N (in the present example, an image in line
units)) acquired by the image acquisition 3 11. The exposure time of block 1 of a
following picked-up image is calculated.
[O 1 041
20 Exposure time transmission 3 13: the timing at which the exposure time of a
block calculated by the exposure time calculation 3 12 is transmitted.
Exposure time latch timing 3 14: the timing at which the exposure time data of
one processing block transmitted by the exposure time transmission 313 process is
latched inside the image sensor (image sensor). Here, the latch timing is generated
25 before the start of exposure of such block.
[O 1 051
Exposure time control 315: reset timing is generated in block (line) units
based on the exposure time latch timing 314 and is the timing at which exposure
control (control to start exposure) is carried out.
30 Image acquisition 316: the time at which the image of the block (line)
currently subject to control is obtained by exposure control in block (line) units
carried out by the exposure control 3 15 described above.
[0 1061
In this example, the image acquisition 3 11 overlaps into output of block 3 of
frame N. As a result, the exposure time calculation 312 and the exposure time
5 transmission 3 13 are both delayed compared to the setting in FIG 18B. As a result,
if the exposure time latch timing 314 and the exposure time control 315 were set in
the same way as in FIG. 18B, the exposure time latch timing 314 and the exposure
time control 3 15 would be set at an earlier time than the exposure time transmission
313. Such processing cannot be executed in reality. That is, with the sequence
10 shown in FIG 22, exposure control based on the image in frame N cannot be carried
out in the following h e N+l and an exposure time calculated based on the image
of a plurality of blocks in frame N cannot be used during image pickup of frame N+l
and can only be used after the processing advances to image pickup of frame N+2.
[0 1 071
15 The configuration of an exposure time latch unit 227 that solves this problem
will now be described with reference to FIG. 23. The exposure time latch unit 227
shown in FIG 23 is a configuration of one exposure time latch unit 227 of the
exposure time control unit 202a shown in FIG 11 described earlier. By using an
exposure time latch unit 227 such as that shown in FIG 23 in place of the exposure
20 time latch unit 227 described with reference to FIG 12, the problem described above
with reference to FIGS. 22A and 22B is solved.
[O 1081
The exposure time latch unit 227 shown in FIG 23 receives exposure control
time data corresponding to the respective blocks at a plurality of different timings,
25 stores the exposure control time data in the storage units 245 and 246, and outputs
the exposure control time data via a selector 249 to the vertical scan circuit 202b.
[0 1091
More specifically, (1) first exposure control time data calculated from only
the image data of one block in the preceding picked-up image frame is outputted
30 from the first storage unit 245 shown in FIG. 23. (2) Second exposure control time
data calculated from the image data of a plurality of blocks in the preceding pickedup
image fiame is outputted fiom the second storage unit 246 shown in FIG 23.
Such exposure time control data is generated and selectively used. Tbe data (1) is the
same as the data described in the embodiment given earlier.
[Ol lo]
5 For example, when the brightness of the subject does not change, processing
that uses only the first storage unit 245, that is, the same processing as that described
earlier with reference to FIG 12 is carried out. Meanwhile in a case where it is
expected that the brightness of the subject will change and saturation will occur in
the next exposure, second exposure control time data (the exposure time data 222)
10 calculated fiom the image data of a plurality of blocks is sent, the second storage unit
246 latches such data at the timing of a second latch line number 242, and exposure
control is carried out according to an exposure time calculated based on a plurality of
blocks.
[Olll]
If the block to be processed using this particular exposure time latch unit 227
is the first line, for example, the first latch line number 241 of block X shown in FIG.
23 is set so that the line number is 1. Also, if the exposure time data stored in the
second storage unit 246 is exposure time data calculated based on an image in units
of a plurality of blocks (for example, three blocks (three lines)) the second latch line
20 number 242 of block X shown in FIG 23 is set so that the line number is 3.
[0112]
Note that since the vertical scan circuit 102 may have to carry out reset
control again when the exposure time data in the first storage unit 245 and the second
storage unit 246 differs, a match detecting unit 247 detects a match and outputs
25 information showing the match.
[0113]
FIGS. 24A and 25A are timing charts like FIG. 22A, and are timing charts for
processing that uses the exposure time latch unit 227 with the configuration shown in
FIG. 23. Like FIG 22B, FIGS. 24B and 25B show exposure control sequences
30 during image pickup of the two frames numbered N and N+l that are consecutively
picked up.
[0114]
In the graphs in FIGS. 24A and 25A showing the exposure timing, the vertical
axis shows lines, the horizontal axis shows time, and exposure time in line units is
shown for each line as a rectangular region. FIGS. 24B and 25B show an exposure
5 control process sequence on a time axis in region (block) units for the exposure
control process that uses the image of frame N. Note that the numbers 1 to M are
identifiers of pixel regions (or "blocks", in the present example, lines) in a picked-up
image.
[0115]
10 In the same way as in the example described with reference to FIGS. 18A and
18B, the settings shown in FIGS. 24A to 25B have exposure control successively
started in specified units, that is, in pixel region (or "block", in the present example,
line) units. By carrying out such processing, this configuration carries out exposure
control by applying exposure control data based on a preceding picked-up image to
15 the following image pickup frame.
[0116]
The processing sequences shown in FIGS. 24B and 25B will now be
described in brief. First, the exposure time is decided by referring to only one block,
with such exposure time being outputted as first data from the first storage unit 245
20 via the selector 249 to the vertical scan circuit 202b. After this, the exposure time is
decided by referring to a larger block, in the example in FIGS. 24B and 25B, three
blocks, and if saturation for example appears possible, such exposure time is
outputted as second data from the second storage unit 246 via the selector 249 to the
vertical scan circuit 202b.
25 [0117]
FIG 24B is a timing chart showing an example of an exposure control
process that discards the exposure time decided by referring to only one block and
instead uses the exposure time decided by referring to three blocks during image
pickup of the next frame. FIG 25B is a timing chart showing an example of an
30 exposure control process set so as to use the exposure time decided by referring to
only one block and to not use the exposure time decided by referring to three blocks.
[0118]
The processing shown in FIG. 24B will now be described. FIG. 24B is a
timing chart showing an example of an exposure control process that discards the
exposure time decided by referring to only one block and instead uses the exposure
5 time decided by referring to three blocks during image pickup of the next fiame.
[0119]
In the exposure control shown in FIG 24B that uses the image of frame N, a
processing example is shown where the exposure time decided by referring to only
one block (i.e., line 1) of the fiame N carried out as the "first exposure control" is
10 discarded and an exposure time calculation result found by referring to three blocks
(i.e., lines 1 to 3) carried out as the "second exposure control" is used during image
pickup of the next fi-ame.
[O 1 201
Processes 3 11 to 325 carried out as the "first exposure control" will now be
15 described.
Image acquisition 3 11 : the timing at which the image of block 1 (i.e., line 1)
of frame N is outputted. At this timing, an image of block 1 (line 1) that is a pixel
region forming part of the picked-up image N is outputted fiom the image sensor.
Exposure time calculation 3 12: the timing at which the exposure time of one
20 block (line) of a following frame subjected to exposure control is calculated based on
the image (one block of frame N (in the present example, line 1)) acquired by the
image acquisition 3 11. Here, the exposure time of block 1 of the following pickedup
image is calculated.
[0121]
25 Exposure time transmission 3 13: the timing at which the exposure time of the
block calculated by the exposure time calculation 3 12 is transmitted.
Exposure time latch timing 3 14: the timing at which the exposure time data of
one processing block transmitted by the exposure time transmission 3 13 process is
latched inside the image sensor (image sensor). Here, the latch timing is generated
30 before the start of exposure of such block.
[O 1221
First exposure time control 325: reset timing is generated in block (line) units
based on the exposure time latch timing 314 and is the timing at which exposure
control (control to start exposure) is carried out. The processing up to this point
corresponds to "first exposure control".
5 [0123]
In parallel with such processing, the "second exposure control" shown at the
bottom of FIG 24B is carried out. In this process, the exposure time decided by
referring to a plurality of blocks (in the present example, blocks 1 to 3, (lines 1 to 3))
is used as the exposure control time for block 1 (line 1) of the next frame N+1.
10 [0124]
Image acquisition 321 : the timing at which the image of a block of fiame N is
outputted. At this timing, an image of three consecutive blocks, that is, pixel region
1 (block 1) to pixel region 3 (block 3), that are an image region forming part of the
picked-up image N is outputted from the image sensor.
15 Exposure time calculation 322: the timing at which the exposure time of one
block (line) in a following fiame subjected to exposure control is calculated based on
the image (the three blocks in frame N (in the present example, an image in line
units)) acquired by the image acquisition 3 11. Here, the exposure time of block 1 of
the following picked-up image is calculated.
20 [0125]
Exposure time transmission 323: the timing at which the exposure time of the
block calculated by the exposure time calculation 322 is transmitted.
Exposure time latch timing 324: the timing at which the exposure time data of
one processing block transmitted by the exposure time transmission 323 process is
25 latched inside the image sensor (image sensor). Here, the latch timing is generated
before the start of exposure of such block.
[0 1261
Second exposure time control 327: reset timing is generated in block (line)
units based on the exposure time latch timing 324 and is the timing at which
30 exposure control (control to start exposure) is carried out.
Image acquisition 328: the time at which the image of the block (line)
currently subject to control is obtained by exposure control in block (line) units
carried out by the exposure control 327 described above. This process is carried out
as exposure control of the next image pickup h e based on the second exposure
control, that is, control based on an exposure time decided with reference to a
5 plurality of blocks.
[0 1271
Next, an example of an exposure control process set so as to use the exposure
time decided by referring to only one block and to not use the exposure time decided
by referring to three blocks will be described with reference to FIG 25B.
10 [0128]
In FIG 25B, as exposure control that uses the image of h e N, an example
is shown where an exposure time decided by referring to only one block (line 1) of
frame N carried out as the "first exposure control" is used in the next exposure time
control and the exposure time calculation process that refers to three blocks (line 1 to
15 3) carried out as the "second exposure control" is stopped.
[0 1291
Processes 3 11 to 328 carried out as the "first exposure control" will now be
described.
Image acquisition 3 11: the timing at which the image of block 1 (i-e., line 1)
20 of frame N is outputted. At this timing, an image of block 1 (line 1) that is a pixel
region forming part of the picked-up image N is outputted from the image sensor.
Exposure time calculation 3 12: the timing at which the exposure time of one
block (line) of a following frame subjected to exposure control is calculated based on
the image (one block of frame N (in the present example, line 1)) acquired by the
25 image acquisition 3 11. Here, the exposure time of block 1 of the following pickedup
image is calculated.
[0130]
Exposure time transmission 3 13: the timing at which the exposure time of the
block calculated by the exposure time calculation 3 12 is transmitted.
30 Exposure time latch timing 3 14: the timing at which the exposure time data of
one processing block transmitted by the exposure time transmission 3 13 process is
latched inside the image sensor (image sensor). Here, the latch timing is generated
before the start of exposure of such block.
[0131]
First exposure time control 325: reset timing is generated in block (line) units
5 based on the exposure time latch timing 314 and is the timing at which exposure
control (control to start exposure) is carried out.
Image acquisition 328: an image acquisition process of block 1 (line 1) of the
b e N+l picked up with the exposure time decided by the second exposure control.
The processing up to this point corresponds to the "first exposure control".
10 [0132]
In parallel with such processing, the "second exposure control" shown at the
bottom of FIG 25B is carried out. In this process, the exposure time decided with
reference to a plurality of blocks (in the present example blocks 1 to 3 (lines 1 to 3)
is used as the exposure control time for block 1 (line 1) of the next frame N+1.
15 [0133]
Image acquisition 321: the timing at which the image of a block in frame N is
outputted. At this timing, an image of three consecutive blocks, that is, pixel region
1 (block 1) to pixel region 3 (block 3), that are a pixel region forming part of the
picked-up image N is outputted from the image sensor.
20 Exposure time calculation 322: the timing at which the exposure time of one
block (line) of a following frame subjected to exposure control is calculated based on
the image (the three blocks of frame N (in the present example, an image in line
units)) acquired by the image acquisition 3 11. Here, the exposure time of block 1 of
the following picked-up image is calculated. At this time, since image pickup of
25 frame N+1 will have started based on the first exposure control, the processing of the
second exposure control is cancelled.
[O 1341
In the processing described with reference to FIGS. 24B and 25B, the
exposure control signal output timing is set at two different timings. One is the
30 output of exposure control data calculated by referring to one block and the other is
the output of exposure control data calculated by referring to a plurality of blocks.
The output timing of the second control signal corresponding to the exposure time
generated by referring to a plurality of blocks is later than the output timing of the
first control signal corresponding to the exposure time generated by referring to a
single block.
5 [0135]
For example, if the brightness of the subject does not change, a process that
uses the output of the exposure control data calculated by referring to a single block,
that is, only the first storage unit 245 shown in FIG 23 is carried out in the same way
as described earlier with reference to FIG 12. Meanwhile, if the brightness of the
10 subject changes and saturation is expected to occur during the next exposure,
exposure control data calculated based on a plurality of blocks, that is data from the
second storage unit 246 shown in FIG. 23 is latched at the timing of the second latch
line number 242 and exposure control is carried out according to the exposure time
calculated based on such plurality of blocks.
15 [0136]
By carrying out such control, even if the luminance of the same pixel region
in consecutively picked-up image frames changes due to the presence of a moving
subject or the like, it will be possible to calculate the luminance and set the exposure
time based on the information of a plurality of blocks and thereby realize appropriate
20 exposure control.
[0137]
7. Conclusion
The configuration of the present disclosure has been described in detail above
with reference to specific embodiments. It should be understood by those skilled in
25 the art that various modifications, combinations, sub-combinations and alterations
may occur depending on design requirements and other factors insofar as they are
within the scope of the appended claims or the equivalents thereof.
[0138]
Additionally, the technology disclosed in the present specification may be
30 configured as follows.
(1) An image pickup apparatus including:
a control unit generating exposure time control data in which an exposure
time of an image sensor is set in pixel region units; and
an image sensor inputting the exposure time control data and carrying out
image acquisition based on exposure time control in the pixel region units,
5 wherein the control unit successively and sequentially generates the exposure
time control data in region units based on luminance information in the pixel region
units that form part of a preceding picked-up image and outputs the exposure time
control data to the image sensor, and
the image sensor inputs the exposure time control data in the region units
10 from the control unit and successively and sequentially uses the inputted exposure
time control data to carry out the image acquisition based on the exposure time
control in the pixel region units.
[0139]
(2) An image pickup apparatus according to (I),
15 wherein the control unit successively and sequentially generates the exposure
time control data in the region units based on luminance information in units of at
least one line of the preceding picked-up image and outputs the exposure time
control data to the image sensor, and
the image sensor inputs the exposure time control data in region units from
20 the control unit and successively and sequentially uses the inputted exposure time
control data to carry out the image acquisition based on the exposure time control in
units of at least one line.
(3) An image pickup apparatus according to (1) or (2),
wherein the control unit generates the exposure time control data by
25 calculation that refers to only one block as a pixel region that is a unit for acquiring
luminance information of the preceding picked-up image, and outputs the exposure
time control data to the image sensor, and
the image sensor carries out the image acquisition based on the exposure time
control that uses the exposure time control data.
30 [0140]
(4) An image pickup apparatus according to any of (1) to (3),
wherein the control unit generates first exposure time control data calculated
by referring to only one block as a pixel region that is a unit for acquiring luminance
information of the preceding picked-up image and second exposure time control data
calculated by referring to a plurality of blocks and outputs the first exposure time
5 control data and the second exposure time control data to the image sensor, and
the image sensor carries out the image acquisition based on the exposure time
control that selectively uses one of the first exposure time control data and the
second exposure time control data.
[0141]
10 (5) An image pickup apparatus according to (4),
wherein the image sensor is operable, when carrying out the image
acquisition selectively using the first exposure time control data and the second
exposure time control data, to carry out a reset process of accumulated pixel charge
to set a selectively used exposure time.
15 (6) An image pickup apparatus according to any of (1) to (9,
wherein the control unit successively and sequentially generates the exposure
time control data in the region units based on luminance information in pixel region
units that form part of an image frame N that is the preceding picked-up image and
outputs the exposure time control data to the image sensor, and
20 the image sensor inputs the exposure time control data in the region units
from the control unit and successively and sequentially uses the inputted exposure
time control data to carry out exposure time control in the pixel region units of an
image frame N+l that is a consecutively picked-up image following the image frame
N.
25 (7) An image pickup apparatus according to any of (1) to (6), Mher including:
a high dynamic range processing unit generating a high dynamic range image
based on pixel information of long exposure pixels and short exposure pixels.
[0 1421
Furthermore, a method of processing executed by the apparatus described
30 above and the like, and programs for executing the processing are also included in
the configuration of the present disclosure.
[0 1431
Also, a series of processes described in the specification can be performed by
hardware, software, or a combination of both. In the case of performing softwarebased
processing, a program storing a processing sequence can be installed in a
5 memory in a computer embedded in dedicated hardware and be executed, or the
program can be installed in a general-purpose computer capable of performing
various processes and be executed. For example, the program can be stored in
advance in a recording medium. Besides installing the program in a computer from
the recording medium, the program can be received over a network such as a LAN
10 (Local Area Network) or the Internet and be installed in a recording medium such as
an embedded hard disk or the like.
[0 1441
Additionally, various processes described in the specification may be
performed, not only chronologically according to the description, but also in parallel
15 or individually according to the processing capability of an apparatus performing the
process or as appropriate. Moreover, the system in the present specification is a
logical group of a plurality of apparatuses, and the apparatuses of respective
configurations are not necessarily in one housing.
[0 1451
20 As described above, according to the embodiments of the present disclosure,
an exposure time control process with a reduced delay is realized in a configuration
that controls the exposure time for each pixel region of an image.
More specifically, in the above configuration, a control unit generating
exposure time control data in which exposure time of an image sensor is set in pixel
25 region units and an image sensor inputting the exposure time control data and
carrying out image acquisition based on exposure time control in the pixel region
units are included. The control unit successively and sequentially generates the
exposure time control data in region units based on luminance information in the
pixel region units that form part of a preceding picked-up image and outputs the
30 exposure time control data to the image sensor. The image sensor inputs the
exposure time control data in the region units from the control unit and successively
and sequentially uses the inputted exposure time control data to carry out the image
acquisition based on the exposure time control in the pixel region units.
By carrying out processing that successively applies the exposure time
control information in region units, exposure time control processing with a reduced
5 delay is realized.
The present disclosure contains subject matter related to that disclosed in
Japanese Priority Patent Application JP 20 1 1 - 13 1 1 16 filed in the Japan Patent Ofice
on June 13,20 1 1, the entire content of which is hereby incorporated by reference.
10 In so far as the embodiments of the invention described above are implemented, at
least in part, using software-controlled data processing apparatus, it will be
appreciated that a computer program providing such software control and a
transmission, storage or other medium by which such a computer program is
provided are envisaged as aspects of the present invention.

We claim:
1. An image pickup apparatus comprising:
a control unit generating exposure time control data in which an exposure
5 time of an image sensor is set in pixel region units; and
an image sensor inputting the exposure time control data and carrying out
image acquisition based on exposure time control in the pixel region units,
wherein the control unit successively and sequentially generates the exposure
time control data in region units based on luminance information in the pixel region
10 units that form part of a preceding picked-up image and outputs the exposure time
control data to the image sensor, and
the image sensor inputs the exposure time control data in the region units
from the control unit and successively and sequentially uses the inputted exposure
time control data to cany out the image acquisition based on the exposure time
15 control in the pixel region units.
2. An image pickup apparatus according to claim 1,
wherein the control unit successively and sequentially generates the exposure
time control data in the region units based on luminance information in units of at
20 least one line of the preceding picked-up image and outputs the exposure time
control data to the image sensor, and
the image sensor inputs the exposure time control data in region units from
the control unit and successively and sequentially uses the inputted exposure time
control data to carry out the image acquisition based on the exposure time control in
25 units of at least one line.
3. An image pickup apparatus according to claim 1,
wherein the control unit generates the exposure time control data by
calculation that refers to only one block as a pixel region that is a unit for acquiring
30 luminance information of the preceding picked-up image, and outputs the exposure
time control data to the image sensor, and
the image sensor carries out the image acquisition based on the exposure time
control that uses the exposure time control data.
4. An image pickup apparatus according to claim 1,
5 wherein the control unit generates first exposure time control data calculated
by referring to only one block as a pixel region that is a unit for acquiring luminance
information of the preceding picked-up image and second exposure time control data
calculated by referring to a plurality of blocks and outputs the first exposure time
control data and the second exposure time control data to the image sensor, and
10 the image sensor carries out the image acquisition based on the exposure time
control that selectively uses one of the first exposure time control data and the
second exposure time control data.
5. An image pickup apparatus according to claim 4,
15 wherein the image sensor is operable, when carrying out the image
acquisition selectively using the first exposure time control data and the second
exposure time control data, to carry out a reset process of accumulated pixel charge
to set a selectively used exposure time.
20 6. An image pickup apparatus according to claim 1,
wherein the control unit successively and sequentially generates the exposure
time control data in the region units based on luminance information in pixel region
units that form part of an image frame N that is the preceding picked-up image and
outputs the exposure time control data to the image sensor, and
25 the image sensor inputs the exposure time control data in the region units
from the control unit and successively and sequentially uses the inputted exposure
time control data to carry out exposure time control in the pixel region units of an
image fiame N+1 that is a consecutively picked-up image following the image frame
N.
30
7. An image pickup apparatus according to claim 1, further comprising:
a high dynamic range processing unit generating a high dynamic range image
based on pixel information of long exposure pixels and short exposure pixels.
8. An image pickup apparatus control method executed by an image pickup apparatus
5 comprising:
generating, by way of a control unit, exposure time control data in which an
exposure time of an image sensor is set in pixel region units; and
inputting, by way of an image sensor, the exposure time control data and
carrying out image acquisition based on exposure time control in the pixel region
10 units,
wherein, in the step of generating, the exposure time control data is
successively and sequentially generated in region units based on luminance
information in the pixel region units that form part of a preceding picked-up image
and the exposure time control data is outputted to the image sensor, and
in the step of inputting, the exposure time control data in the region units is
inputted from the control unit and the inputted exposure time control data is
successively and sequentially used to carry out the image acquisition based on the
exposure time control in the pixel region units.
20 9. A program causing an image pickup apparatus to carry out an image pickup
control process comprising:
generating, by way of a control unit, exposure time control data in which an
exposure time of an image sensor is set in pixel region units; and
inputting, by way of an image sensor, the exposure time control data and
25 carrying out image acquisition based on exposure time control in the pixel region
units,
wherein, in the step of generating, the exposure time control data is
successively and sequentially generated in region units based on luminance
information in the pixel region units that form part of a preceding picked-up image
30 and the exposure time control data is outputted to the image sensor, and
in the step of inputting, the exposure time control data in the region units is
inputted fiom the control unit and the inputted exposure time control data is
successively and sequentially used to carry out the image acquisition based on the
exposure time control in the pixel region units.
5 Dated this 06/06/2012
ATTORNEY FOR THE AP@ICANT[S]