Title of Invention: Imaging Apparatus, Imaging System, Imaging Method, and Imaging Program
Technical field
[0001]
　The present disclosure relates to an imaging device, an imaging system, an imaging method, and an imaging program.
Background technology
[0002]
　In recent years, along with the high performance of imaging devices such as digital still cameras, digital video cameras, compact cameras installed in multifunctional mobile phones (smartphones), etc., an image recognition function that recognizes a predetermined object contained in a captured image has been developed. is being developed.
prior art documents
patent literature
[0003]
Patent Document 1: JP 2017-112409 A
SUMMARY OF THE INVENTION
Problems to be Solved by the Invention
[0004]
　In general, an image suitable for recognition processing in an image recognition function is different from an image suitable for human visual recognition. Therefore, when trying to improve the accuracy of recognition in an imaging device equipped with an image recognition function, it may be difficult for the image captured for recognition processing to provide sufficient information as an image for visual recognition. There is
[0005]
　An object of the present disclosure is to provide an imaging device, an imaging system, an imaging method, and an imaging program capable of performing both imaging for recognition processing and imaging for visual recognition.
Means to solve problems
[0006]
　An imaging device according to the present disclosure includes an imaging unit having a pixel region in which a plurality of pixels are arranged, a readout control unit that controls readout of pixel signals from pixels included in the pixel region, and a readout control unit that controls the readout of pixel signals. a first readout unit setting unit that sets a readout unit as a part of a pixel region that performs readout; an image output unit that outputs a first image based on pixel signals read out from the readout unit to a subsequent stage; A second readout unit setting unit for setting a readout unit in which the control unit reads out pixel signals, learns teacher data for each readout unit, performs recognition processing on pixel signals for each readout unit, and performs recognition processing results. and a recognition unit that outputs the recognition result of.
Brief description of the drawing
[0007]
1 is a block diagram showing the configuration of an example of an imaging device applicable to each embodiment of the present disclosure; FIG.
2A is a schematic diagram showing an example of a hardware configuration of an imaging device according to each embodiment; FIG.
2B is a schematic diagram showing an example of the hardware configuration of an imaging device according to each embodiment; FIG.
3A is a diagram showing an example in which an imaging device according to each embodiment is formed by a laminated CIS having a two-layer structure; FIG.
3B is a diagram showing an example in which an imaging device according to each embodiment is formed by a laminated CIS having a three-layer structure; FIG.
4 is a block diagram showing an example configuration of a sensor section applicable to each embodiment; FIG.
5A is a schematic diagram for explaining a rolling shutter method; FIG.
5B is a schematic diagram for explaining a rolling shutter method; FIG.
5C is a schematic diagram for explaining a rolling shutter method; FIG.
6A is a schematic diagram for explaining thinning of lines in the rolling shutter method; FIG.
6B is a schematic diagram for explaining thinning of lines in the rolling shutter method; FIG.
6C is a schematic diagram for explaining thinning of lines in the rolling shutter method; FIG.
7A is a diagram schematically showing an example of another imaging method in the rolling shutter system; FIG.
7B is a diagram schematically showing an example of another imaging method in the rolling shutter system; FIG.
8A is a schematic diagram for explaining a global shutter method; FIG.
8B is a schematic diagram for explaining a global shutter method; FIG.
8C is a schematic diagram for explaining a global shutter method; FIG.
9A] A diagram schematically showing an example of a sampling pattern that can be realized in the global shutter method. [FIG.
9B] A diagram schematically showing an example of a sampling pattern that can be realized in the global shutter method. [FIG.
10 is a diagram for schematically explaining image recognition processing by CNN; FIG.
11 is a diagram for schematically explaining image recognition processing for obtaining a recognition result from a part of an image to be recognized; FIG.
12A is a diagram schematically showing an example of identification processing by DNN when time-series information is not used; FIG.
12B is a diagram schematically showing an example of identification processing by DNN when time-series information is not used; FIG.
13A is a diagram schematically showing a first example of identification processing by DNN when time-series information is used; FIG.
13B is a diagram schematically showing a first example of identification processing by DNN when time-series information is used; FIG.
14A is a diagram schematically showing a second example of identification processing by DNN when time-series information is used; FIG.
14B is a diagram schematically showing a second example of identification processing by DNN when time-series information is used; FIG.
15A is a diagram for explaining the relationship between the driving speed of a frame and the readout amount of pixel signals. FIG.
15B is a diagram for explaining the relationship between the driving speed of a frame and the readout amount of pixel signals; FIG.
16 is a schematic diagram for schematically explaining recognition processing according to each embodiment of the present disclosure; FIG.
17 is a flowchart of an example of recognition processing by a recognition processing unit according to the first embodiment; FIG.
18 is a diagram showing an example of image data for one frame; FIG.
19 is a diagram for explaining the flow of machine learning processing executed by the recognition processing unit according to the first embodiment; FIG.
20A is a schematic diagram for explaining an application example of the first embodiment; FIG.
20B is a schematic diagram for explaining an application example of the first embodiment; FIG.
21 is a functional block diagram of an example for explaining functions of an imaging device according to a second embodiment; FIG.
22 is a schematic diagram showing in more detail an example of processing in a recognition processing unit according to the second embodiment; FIG.
23 is a functional block diagram of an example for explaining functions according to the second embodiment; FIG.
24 is a schematic diagram for explaining frame readout processing according to the second embodiment; FIG.
25 is a schematic diagram schematically showing recognition processing according to the second embodiment; FIG.
26 is a diagram for explaining an example of terminating recognition processing in the middle of frame reading; FIG.
27 is a diagram for explaining an example of terminating recognition processing in the middle of frame reading; FIG.
28 is a flow chart showing an example of recognition processing according to the second embodiment; FIG.
29A is a time chart showing an example of control of reading and recognition processing according to the second embodiment; FIG.
29B is an example time chart showing an example of control of reading and recognition processing according to the second embodiment; FIG.
30 is an example time chart showing another example of control of reading and recognition processing according to the second embodiment. FIG.
31 is a schematic diagram for explaining frame readout processing according to the first modification of the second embodiment; FIG.
32 is a schematic diagram for explaining frame readout processing according to a second modification of the second embodiment; FIG.
33 is a schematic diagram for explaining frame readout processing according to the third modification of the second embodiment; FIG.
34 is a schematic diagram schematically showing recognition processing according to a third modification of the second embodiment; FIG.
35] Fig. 35 is a diagram for explaining an example of terminating recognition processing in the middle of frame reading when the reading unit is an area. [Fig.
36] Fig. 36 is a diagram for explaining an example of terminating recognition processing in the middle of frame reading when the reading unit is an area. [Fig.
37 is a schematic diagram for explaining frame readout processing according to the fourth modification of the second embodiment; FIG.
38 is a schematic diagram schematically showing recognition processing applicable to the fourth modification of the second embodiment; FIG.
39 is an example time chart showing an example of readout and control according to the fourth modification of the second embodiment; FIG.
40 is a diagram for more specifically explaining the frame readout process according to the fourth modification of the second embodiment; FIG.
41 is a diagram for more specifically explaining the frame readout process according to the fourth modification of the second embodiment; FIG.
42 is a schematic diagram for explaining frame readout processing according to the fifth modification of the second embodiment; FIG.
43 is a schematic diagram for explaining frame readout processing according to the sixth modification of the second embodiment; FIG.
44 is a diagram showing an example of patterns for reading and recognition processing according to the sixth modification of the second embodiment; FIG.
45 is a schematic diagram for explaining a first example of frame readout processing according to the seventh modification of the second embodiment; FIG.
46 is a schematic diagram for explaining frame readout processing according to a first alternative example of the seventh modification of the second embodiment; FIG.
47 is a schematic diagram for explaining frame readout processing according to a second alternative example of the seventh modification of the second embodiment; FIG.
48 is a schematic diagram for explaining frame readout processing according to a third alternative example of the seventh modification of the second embodiment; FIG.
49 is a functional block diagram of an example for explaining functions according to an eighth modification of the second embodiment; FIG.
50 is a flowchart of an example of recognition processing according to an eighth modification of the second embodiment; FIG.
51A is a diagram for explaining a first example of reading and recognition processing according to the eighth modification of the second embodiment; FIG.
51B is a diagram for explaining a first example of reading and recognition processing according to the eighth modification of the second embodiment; FIG.
52 is a diagram showing an example of an exposed pattern according to a ninth modification of the second embodiment; FIG.
53 is a functional block diagram of an example for explaining functions according to a tenth modification of the second embodiment; FIG.
54 is a flowchart of an example showing processing according to a tenth modification of the second embodiment; FIG.
55 is a schematic diagram for explaining the first process according to the tenth modification of the second embodiment; FIG.
56 is a schematic diagram for explaining second processing according to the tenth modification of the second embodiment; FIG.
57A is a schematic diagram for explaining third processing according to the tenth modification of the second embodiment; FIG.
57B is a schematic diagram for explaining third processing according to the tenth modification of the second embodiment; FIG.
58A is a schematic diagram for explaining third processing according to the tenth modification of the second embodiment; FIG.
58B is a schematic diagram for explaining third processing according to the tenth modification of the second embodiment; FIG.
59A is a schematic diagram for explaining third processing according to the tenth modification of the second embodiment; FIG.
59B is a schematic diagram for explaining third processing according to the tenth modification of the second embodiment; FIG.
60 is a schematic diagram showing in more detail an example of processing in the recognition processing unit according to the tenth modification of the second embodiment; FIG.
61 is a functional block diagram of an example for explaining functions according to a third embodiment; FIG.
62 is a schematic diagram showing an example of a readout unit pattern applicable to the third embodiment; FIG.
63 is a schematic diagram showing an example of a readout order pattern applicable to the third embodiment; FIG.
64 is a functional block diagram of an example for explaining functions according to the first modification of the third embodiment; FIG.
65 is a schematic diagram for explaining a first setting method of the first modification of the third embodiment; FIG.
66 is a schematic diagram for explaining a second setting method of the first modification of the third embodiment; FIG.
67 is a schematic diagram for explaining a third setting method of the first modification of the third embodiment; FIG.
68 is a functional block diagram of an example for explaining functions of an imaging device according to a third modified example of the third embodiment; FIG.
69 is a functional block diagram of an example for explaining functions of an imaging device according to a fourth embodiment; FIG.
70 is a schematic diagram for schematically explaining image processing according to the fourth embodiment; FIG.
71 is a diagram showing an example of read processing according to the fourth embodiment; FIG.
72 is a flow chart showing an example of processing according to the fourth embodiment; FIG.
73 is a diagram for explaining a third example of control of the image data storage unit according to the fourth embodiment; FIG.
74 is a diagram for explaining a first modification of the fourth embodiment; FIG.
75 is a diagram for explaining a second modification of the fourth embodiment; FIG.
76 is a diagram for explaining a first example in the third modified example of the fourth embodiment; FIG.
77] Fig. 77 is a diagram illustrating a usage example of an imaging device to which the technology of the present disclosure is applied. [Fig.
78 is a block diagram showing an example of a schematic configuration of a vehicle control system; FIG.
79 is an explanatory diagram showing an example of installation positions of the vehicle exterior information detection section and the imaging section. FIG.
MODE FOR CARRYING OUT THE INVENTION
[0008]
　Hereinafter, embodiments of the present disclosure will be described in detail based on the drawings. In addition, in the following embodiments, the same parts are denoted by the same reference numerals, thereby omitting redundant explanations.
[0009]
　Hereinafter, embodiments of the present disclosure will be described according to the following order.
1.
Configuration example 2 according to each embodiment of the present disclosure . Examples of existing technology applicable to the present disclosure
　2-1. Outline of rolling shutter
　2-2. Outline of global shutter
　2-3. DNN (Deep Neural Network)
　　2-3-1. Outline of CNN (Convolutional Neural Network)
　　2-3-2. Outline of RNN (Recurrent Neural Network)
　2-4. 3. Driving speed
. Overview of the present disclosure
4. First Embodiment
　4-1. Operation example by recognition processing unit
　4-2. Concrete example of operation by recognition processing unit
　4-3. Application example of the first embodiment
5. Second Embodiment
　　5-0-1.
　　Configuration example 5-0-2 according to the second embodiment . Example of processing in the recognition processing unit according to the second embodiment
　　5-0-3. Details of recognition processing according to the second embodiment
　　5-0-4. Control example of reading and recognition processing according to the second embodiment
　5-1. First modification of the second embodiment
　5-2. Second modification of second embodiment
　5-3. Third modification of second embodiment
　5-4. Fourth Modification of Second Embodiment
　5-5. Fifth modification of second embodiment
　5-6. Sixth Modification of Second Embodiment
　5-7. Seventh modification of second embodiment
　5-8. Eighth modification of the second embodiment
　5-9. Ninth Modification of Second Embodiment
　5-10. 6. Tenth modification of the second embodiment
. Third Embodiment
　　6-0-1. Method of determining readout unit pattern and readout order pattern
　　6-0-1-1. Examples of readout unit pattern and readout order pattern
　　6-0-1-2. Specific example of read unit pattern priority setting method
　　6-0-1-3. Concrete example of priority setting method for readout order pattern
　6-1. First Modification of Third Embodiment
　6-2. Second Modification of Third Embodiment
　6-3. Third modified example of the third embodiment7
. Fourth Embodiment
　7-1. First Modification of Fourth Embodiment
　7-2. Second Modification of Fourth Embodiment
　7-3. Third modification of the fourth embodiment
8. Fifth embodiment
[0010]
[1. Configuration Example According to Embodiments of Present Disclosure]
　The configuration of an imaging apparatus according to the present disclosure will be schematically described. FIG. 1 is a block diagram showing the configuration of an example of an imaging device applicable to each embodiment of the present disclosure. In FIG. 1, the imaging device 1 includes a sensor unit 10, a sensor control unit 11, a recognition processing unit 12, a memory 13, a visual recognition processing unit 14, and an output control unit 15, each of which is a CMOS ( It is a CMOS image sensor (CIS) integrally formed using Complementary Metal Oxide Semiconductor. Note that the imaging device 1 is not limited to this example, and may be another type of optical sensor such as an infrared light sensor that performs imaging using infrared light.
[0011]
　The sensor unit 10 outputs pixel signals according to the light applied to the light receiving surface through the optical unit 30 . More specifically, the sensor unit 10 has a pixel array in which pixels each including at least one photoelectric conversion element are arranged in a matrix. A light-receiving surface is formed by pixels arranged in rows and columns in the pixel array. The sensor unit 10 further includes a drive circuit for driving each pixel included in the pixel array, and a signal that performs predetermined signal processing on a signal read from each pixel and outputs the signal as a pixel signal of each pixel. and a processing circuit. The sensor unit 10 outputs a pixel signal of each pixel included in the pixel area as digital image data.
[0012]
　Hereinafter, in the pixel array of the sensor unit 10, a region in which pixels effective for generating pixel signals are arranged is called a frame. Frame image data is formed by pixel data based on pixel signals output from pixels included in a frame. Each row in the array of pixels of the sensor unit 10 is called a line, and line image data is formed by pixel data based on pixel signals output from each pixel included in the line. Further, the operation of the sensor unit 10 to output pixel signals according to the light applied to the light receiving surface is called imaging. The sensor unit 10 is controlled in terms of exposure during imaging and gain (analog gain) for pixel signals in accordance with imaging control signals supplied from a sensor control unit 11, which will be described later.
[0013]
　The sensor control unit 11 is composed of, for example, a microprocessor, controls reading of pixel data from the sensor unit 10, and outputs pixel data based on pixel signals read from pixels included in a frame. The pixel data output from the sensor control section 11 is passed to the recognition processing section 12 and the visual recognition processing section 14 .
[0014]
　The sensor control unit 11 also generates an imaging control signal for controlling imaging in the sensor unit 10 . The sensor control unit 11 generates imaging control signals, for example, according to instructions from the recognition processing unit 12 and the visual recognition processing unit 14, which will be described later. The imaging control signal includes information indicating the exposure and analog gain at the time of imaging in the sensor section 10 described above. The imaging control signal further includes control signals (vertical synchronizing signal, horizontal synchronizing signal, etc.) used by the sensor unit 10 to perform an imaging operation. The sensor control section 11 supplies the generated imaging control signal to the sensor section 10 .
[0015]
　The optical unit 30 is for irradiating the light receiving surface of the sensor unit 10 with light from a subject, and is arranged at a position corresponding to the sensor unit 10, for example. The optical unit 30 includes, for example, a plurality of lenses, a diaphragm mechanism for adjusting the size of an opening for incident light, and a focus mechanism for adjusting the focus of light irradiated onto the light receiving surface. The optical section 30 may further include a shutter mechanism (mechanical shutter) that adjusts the time during which the light receiving surface is irradiated with light. The aperture mechanism, focus mechanism, and shutter mechanism of the optical unit 30 can be controlled by the sensor control unit 11, for example. Without being limited to this, the aperture and focus in the optical unit 30 can be controlled from outside the imaging device 1 . It is also possible to configure the optical unit 30 integrally with the imaging device 1 .
[0016]
　Based on the pixel data passed from the sensor control unit 11, the recognition processing unit 12 performs recognition processing of an object included in the image based on the pixel data. In the present disclosure, for example, a DSP (Digital Signal Processor) reads and executes a program that is pre-learned with teacher data and stored in the memory 13 as a learning model, thereby performing recognition processing using a DNN (Deep Neural Network). A recognition processing unit 12 is configured as a machine learning unit that performs the above. The recognition processing unit 12 can instruct the sensor control unit 11 to read pixel data necessary for recognition processing from the sensor unit 10 . A recognition result obtained by the recognition processing unit 12 is passed to the output control unit 15 .
[0017]
　The visual recognition processing unit 14 executes processing for obtaining an image suitable for human visual recognition on the pixel data passed from the sensor control unit 11, and outputs image data composed of, for example, a group of pixel data. do. For example, the visual recognition processing unit 14 is configured by an ISP (Image Signal Processor) reading and executing a program stored in advance in a memory (not shown).
[0018]
　For example, if each pixel included in the sensor unit 10 is provided with a color filter and the pixel data has color information of R (red), G (green), and B (blue), the visual recognition processing unit 14 performs demosaic processing. processing, white balancing, etc. can be performed. Further, the visual recognition processing unit 14 can instruct the sensor control unit 11 to read pixel data necessary for visual recognition processing from the sensor unit 10 . Image data obtained by subjecting pixel data to image processing by the visual recognition processing unit 14 is passed to the output control unit 15 .
[0019]
　The output control unit 15 is configured by, for example, a microprocessor, and outputs one or both of the recognition result delivered from the recognition processing unit 12 and the image data delivered as the viewing processing result from the viewing processing unit 14 to the imaging device. Output to the outside of 1. The output control unit 15 can output image data to, for example, a display unit 31 having a display device. Thereby, the user can visually recognize the image data displayed by the display unit 31 . Note that the display unit 31 may be built in the imaging device 1 or may be configured outside the imaging device 1 .
[0020]
　2A and 2B are schematic diagrams showing an example of the hardware configuration of the imaging device 1 according to each embodiment. 2A, one chip 2 includes the sensor unit 10, the sensor control unit 11, the recognition processing unit 12, the memory 13, the visual recognition processing unit 14, and the output control unit 15 among the components shown in FIG. For example. Note that the memory 13 and the output control unit 15 are omitted in FIG. 2A to avoid complication.
[0021]
　In the configuration shown in FIG. 2A, the recognition result by the recognition processing unit 12 is output to the outside of the chip 2 via the output control unit 15 (not shown). 2A, the recognition processing unit 12 can acquire pixel data to be used for recognition from the sensor control unit 11 via the internal interface of the chip 2. FIG.
[0022]
　2B, one chip 2 includes the sensor unit 10, the sensor control unit 11, the visual recognition processing unit 14, and the output control unit 15 of the configuration shown in FIG. not shown) is placed outside the chip 2 . Also in FIG. 2B, the memory 13 and the output control unit 15 are omitted to avoid complication, as in FIG. 2A described above.
[0023]
　In the configuration of FIG. 2B, the recognition processing unit 12 acquires pixel data to be used for recognition via an interface for communication between chips. Further, although FIG. 2B shows that the recognition result by the recognition processing unit 12 is output directly from the recognition processing unit 12 to the outside, this is not limited to this example. That is, in the configuration of FIG. 2B , the recognition processing unit 12 may return the recognition result to the chip 2 and output it from the output control unit 15 (not shown) mounted on the chip 2 .
[0024]
　In the configuration shown in FIG. 2A, the recognition processing unit 12 is mounted on the chip 2 together with the sensor control unit 11, and communication between the recognition processing unit 12 and the sensor control unit 11 can be performed at high speed by the internal interface of the chip 2. . On the other hand, in the configuration shown in FIG. 2A, the recognition processing unit 12 cannot be replaced, and it is difficult to change the recognition processing. On the other hand, in the configuration shown in FIG. 2B, since the recognition processing unit 12 is provided outside the chip 2, communication between the recognition processing unit 12 and the sensor control unit 11 is performed via an interface between chips. There is a need. Therefore, communication between the recognition processing unit 12 and the sensor control unit 11 is slower than in the configuration of FIG. 2A, and control may be delayed. On the other hand, replacement of the recognition processing unit 12 is easy, and various recognition processes can be realized.
[0025]
　Hereinafter, unless otherwise specified, the imaging device 1 includes the sensor unit 10, the sensor control unit 11, the recognition processing unit 12, the memory 13, the visual recognition processing unit 14, and the output control unit 15 in one chip 2 of FIG. 2A. shall be adopted.
[0026]
　In the configuration shown in FIG. 2A described above, the imaging device 1 can be formed on one substrate. The imaging device 1 is not limited to this, and may be a laminated CIS in which a plurality of semiconductor chips are laminated and integrally formed.
[0027]
　As an example, the imaging device 1 can be formed with a two-layer structure in which semiconductor chips are stacked in two layers. FIG. 3A is a diagram showing an example in which the imaging device 1 according to each embodiment is formed by a laminated CIS having a two-layer structure. In the structure of FIG. 3A, the pixel section 20a is formed in the semiconductor chip of the first layer, and the memory+logic section 20b is formed in the semiconductor chip of the second layer. The pixel section 20 a includes at least the pixel array in the sensor section 10 . The memory+logic unit 20b includes, for example, the sensor control unit 11, the recognition processing unit 12, the memory 13, the visual recognition processing unit 14, the output control unit 15, and an interface for communicating between the imaging device 1 and the outside. . The memory+logic section 20b further includes a part or all of a drive circuit that drives the pixel array in the sensor section 10. FIG. Although not shown, the memory+logic unit 20b can further include a memory used by the visual recognition processing unit 14 to process image data, for example.
[0028]
　As shown on the right side of FIG. 3A, the imaging device 1 is configured as one solid-state imaging device by bonding the semiconductor chips of the first layer and the semiconductor chips of the second layer in electrical contact with each other.
[0029]
　As another example, the imaging device 1 can be formed with a three-layer structure in which semiconductor chips are stacked in three layers. FIG. 3B is a diagram showing an example in which the imaging device 1 according to each embodiment is formed by a laminated CIS having a three-layer structure. In the structure of FIG. 3B, the pixel section 20a is formed in the semiconductor chip of the first layer, the memory section 20c is formed in the semiconductor chip of the second layer, and the logic section 20b' is formed in the semiconductor chip of the third layer. . In this case, the logic unit 20b' includes, for example, the sensor control unit 11, the recognition processing unit 12, the visual recognition processing unit 14, the output control unit 15, and an interface for communicating between the imaging device 1 and the outside. Further, the memory unit 20c can include the memory 13 and a memory used by the visual recognition processing unit 14 for processing image data, for example. The memory 13 may be included in the logic portion 20b'.
[0030]
　As shown on the right side of FIG. 3B , the first layer semiconductor chip, the second layer semiconductor chip, and the third layer semiconductor chip are laminated while being in electrical contact with each other, whereby the imaging device 1 is integrated into one. configured as one solid-state imaging device.
[0031]
　FIG. 4 is a block diagram showing an example configuration of the sensor unit 10 applicable to each embodiment. 4, the sensor unit 10 includes a pixel array unit 101, a vertical scanning unit 102, an AD (Analog to Digital) conversion unit 103, a pixel signal line 106, a vertical signal line VSL, a control unit 1100, a signal and a processing unit 1101 . 4, the control unit 1100 and the signal processing unit 1101 can be included in the sensor control unit 11 shown in FIG. 1, for example.
[0032]
　The pixel array unit 101 includes a plurality of pixel circuits 100 each including a photoelectric conversion element such as a photodiode that performs photoelectric conversion on received light, and a circuit that reads charges from the photoelectric conversion element. In the pixel array portion 101, the plurality of pixel circuits 100 are arranged in a matrix in the horizontal direction (row direction) and vertical direction (column direction). In the pixel array portion 101, the arrangement of the pixel circuits 100 in the row direction is called a line. For example, when an image of one frame is formed by 1920 pixels×1080 lines, the pixel array section 101 includes at least 1080 lines each including at least 1920 pixel circuits 100 . An image (image data) of one frame is formed by pixel signals read from the pixel circuits 100 included in the frame.
[0033]
　Hereinafter, the operation of reading out pixel signals from each pixel circuit 100 included in a frame in the sensor unit 10 will be described as appropriately reading pixels from the frame. Also, the operation of reading out pixel signals from the pixel circuits 100 of the lines included in the frame is described as appropriately reading out the lines.
[0034]
　Further, in the pixel array section 101, the pixel signal line 106 is connected to each row and column of each pixel circuit 100, and the vertical signal line VSL is connected to each column. The ends of the pixel signal lines 106 that are not connected to the pixel array section 101 are connected to the vertical scanning section 102 . The vertical scanning unit 102 transmits control signals such as drive pulses for reading out pixel signals from pixels to the pixel array unit 101 via the pixel signal lines 106 under the control of the control unit 1100 to be described later. An end of the vertical signal line VSL that is not connected to the pixel array unit 101 is connected to the AD conversion unit 103 . A pixel signal read from the pixel is transmitted to the AD conversion unit 103 via the vertical signal line VSL.
[0035]
　Readout control of pixel signals from the pixel circuit 100 will be schematically described. A pixel signal is read out from the pixel circuit 100 by transferring charges accumulated in a photoelectric conversion element due to exposure to a floating diffusion layer (FD) and converting the transferred charges in the floating diffusion layer into a voltage. conduct. A voltage resulting from charge conversion in the floating diffusion layer is output to the vertical signal line VSL via an amplifier.
[0036]
　More specifically, in the pixel circuit 100, the space between the photoelectric conversion element and the floating diffusion layer is turned off (opened) during exposure, and the light generated by photoelectric conversion in the photoelectric conversion element is generated according to incident light. charge is accumulated. After the exposure is finished, the floating diffusion layer and the vertical signal line VSL are connected according to the selection signal supplied through the pixel signal line 106 . Furthermore, the floating diffusion layer is connected to the power supply voltage VDD or the black level voltage supply line for a short period of time in response to a reset pulse supplied through the pixel signal line 106 to reset the floating diffusion layer. A reset level voltage (assumed to be voltage A) of the floating diffusion layer is output to the vertical signal line VSL. Thereafter, a transfer pulse supplied through the pixel signal line 106 turns on (closes) the space between the photoelectric conversion element and the floating diffusion layer, thereby transferring the charge accumulated in the photoelectric conversion element to the floating diffusion layer. A voltage (referred to as voltage B) corresponding to the charge amount of the floating diffusion layer is output to the vertical signal line VSL.
[0037]
　The AD conversion unit 103 includes an AD converter 107 provided for each vertical signal line VSL, a reference signal generation unit 104, and a horizontal scanning unit 105. The AD converter 107 is a column AD converter that performs AD conversion processing on each column of the pixel array unit 101 . The AD converter 107 performs AD conversion processing on the pixel signal supplied from the pixel circuit 100 via the vertical signal line VSL, and performs correlated double sampling (CDS) processing for noise reduction. Two digital values ​​(values ​​corresponding to voltage A and voltage B, respectively) are generated.
[0038]
　The AD converter 107 supplies the two generated digital values ​​to the signal processing section 1101 . The signal processing unit 1101 performs CDS processing based on the two digital values ​​supplied from the AD converter 107 to generate pixel signals (pixel data) as digital signals. Pixel data generated by the signal processing unit 1101 is output to the outside of the sensor unit 10 .
[0039]
　Based on the control signal input from the control unit 1100, the reference signal generation unit 104 generates, as a reference signal, a ramp signal used by each AD converter 107 to convert the pixel signal into two digital values. A ramp signal is a signal whose level (voltage value) decreases with a constant slope with respect to time, or a signal whose level decreases stepwise. The reference signal generator 104 supplies the generated ramp signal to each AD converter 107 . The reference signal generator 104 is configured using, for example, a DAC (Digital to Analog Converter).
[0040]
　When the reference signal generator 104 supplies a ramp signal in which the voltage drops stepwise according to a predetermined slope, the counter starts counting according to the clock signal. The comparator compares the voltage of the pixel signal supplied from the vertical signal line VSL with the voltage of the ramp signal, and stops counting by the counter when the voltage of the ramp signal straddles the voltage of the pixel signal. The AD converter 107 converts the analog pixel signal into a digital value by outputting a value corresponding to the count value of the time when the counting is stopped.
[0041]
　The AD converter 107 supplies the two generated digital values ​​to the signal processing section 1101 . The signal processing unit 1101 performs CDS processing based on the two digital values ​​supplied from the AD converter 107 to generate pixel signals (pixel data) as digital signals. A digital pixel signal generated by the signal processing unit 1101 is output to the outside of the sensor unit 10 .
[0042]
　Under the control of the control unit 1100, the horizontal scanning unit 105 performs selective scanning to select each AD converter 107 in a predetermined order, thereby scanning each digital value temporarily held by each AD converter 107. The signals are sequentially output to the signal processing unit 1101 . The horizontal scanning unit 105 is configured using, for example, a shift register and an address decoder.
[0043]
　The control unit 1100 drives and controls the vertical scanning unit 102 , the AD conversion unit 103 , the reference signal generation unit 104 , the horizontal scanning unit 105 and the like according to the imaging control signal supplied from the sensor control unit 11 . The control unit 1100 generates various drive signals that serve as references for operations of the vertical scanning unit 102 , AD conversion unit 103 , reference signal generation unit 104 and horizontal scanning unit 105 . For example, the control unit 1100 controls the vertical scanning unit 102 to supply signals to the pixel circuits 100 via the pixel signal lines 106 based on the vertical synchronization signal or the external trigger signal included in the imaging control signal and the horizontal synchronization signal. Generate control signals. The control unit 1100 supplies the generated control signal to the vertical scanning unit 102 .
[0044]
　Also, the control unit 1100 passes information indicating the analog gain included in the imaging control signal supplied from the sensor control unit 11 to the AD conversion unit 103, for example. The AD converter 103 controls the gain of the pixel signal input to each AD converter 107 included in the AD converter 103 via the vertical signal line VSL according to the information indicating the analog gain.
[0045]
　Based on control signals supplied from the control unit 1100, the vertical scanning unit 102 applies various signals including drive pulses to the pixel signal lines 106 of the selected pixel rows of the pixel array unit 101 to the pixel circuits 100 line by line. Then, each pixel circuit 100 outputs a pixel signal to the vertical signal line VSL. The vertical scanning unit 102 is configured using, for example, shift registers and address decoders. Also, the vertical scanning unit 102 controls exposure in each pixel circuit 100 according to information indicating exposure supplied from the control unit 1100 .
[0046]
　The sensor unit 10 configured in this manner is a column AD type complementary metal oxide semiconductor (CMOS) image sensor in which the AD converters 107 are arranged for each column.
[0047]
[2. Examples of Existing Technologies Applicable to the Present Disclosure ]
　Before describing each embodiment according to the present disclosure, existing technologies applicable to the present disclosure will be briefly described for easy understanding.
[0048]
(2-1. Overview of Rolling Shutter)
　As imaging methods for imaging by the pixel array section 101, a rolling shutter (RS) method and a global shutter (GS) method are known. First, the rolling shutter method will be briefly described. 5A, 5B and 5C are schematic diagrams for explaining the rolling shutter method. In the rolling shutter method, as shown in FIG. 5A, images are sequentially captured line by line from, for example, an upper end line 201 of a frame 200 .
[0049]
　It should be noted that, in the above description, it has been explained that “imaging” refers to the operation of the sensor unit 10 outputting pixel signals according to the light with which the light-receiving surface is irradiated. More specifically, “imaging” refers to a series of operations from exposing a pixel to transferring a pixel signal based on charges accumulated in a photoelectric conversion element included in the pixel due to the exposure to the sensor control unit 11. and Also, as described above, a frame indicates an area in the pixel array section 101 in which the pixel circuits 100 effective for generating pixel signals are arranged.
[0050]
　For example, in the configuration of FIG. 4, exposure is performed simultaneously for each pixel circuit 100 included in a line. After the exposure is completed, the pixel signals based on the charges accumulated by the exposure are simultaneously transferred in each pixel circuit 100 included in the line through each vertical signal line VSL corresponding to each pixel circuit 100 . By sequentially executing this operation on a line-by-line basis, imaging with a rolling shutter can be realized.
[0051]
　FIG. 5B schematically shows an example of the relationship between imaging and time in the rolling shutter method. In FIG. 5B, the vertical axis indicates line position, and the horizontal axis indicates time. In the rolling shutter method, the exposure of each line is performed line by line. Therefore, as shown in FIG. 5B, the timing of the exposure of each line shifts according to the position of the line. Therefore, for example, when the horizontal positional relationship between the imaging device 1 and the subject changes at high speed, distortion occurs in the captured image of the frame 200 as illustrated in FIG. 5C. In the example of FIG. 5C, the image 202 corresponding to the frame 200 is an image tilted at an angle corresponding to the speed and direction of change in the horizontal positional relationship between the imaging device 1 and the subject.
[0052]
　In the rolling shutter method, it is also possible to pick up an image by thinning lines. 6A, 6B, and 6C are schematic diagrams for explaining thinning of lines in the rolling shutter method. As shown in FIG. 6A, as in the example of FIG. 5A described above, imaging is performed line by line from the line 201 at the top end of the frame 200 toward the bottom end of the frame 200 . At this time, imaging is performed while skipping lines every predetermined number.
[0053]
　Here, for the sake of explanation, it is assumed that every other line is picked up by thinning one line. That is, after imaging the nth line, the (n+2)th line is imaged. At this time, it is assumed that the time from imaging the nth line to imaging the (n+2)th line is equal to the time from imaging the nth line to imaging the (n+1)th line when thinning is not performed.
[0054]
　FIG. 6B schematically shows an example of the relationship between imaging and time when one line is thinned out in the rolling shutter method. In FIG. 6B, the vertical axis indicates line position, and the horizontal axis indicates time. In FIG. 6B, exposure A corresponds to the exposure in FIG. 5B without thinning, and exposure B shows exposure with one line thinning. As shown in exposure B, by performing line thinning, it is possible to reduce exposure timing lag at the same line position as compared with the case where line thinning is not performed. Therefore, as exemplified as an image 203 in FIG. 6C, the distortion in the tilt direction that occurs in the captured image of the frame 200 is smaller than in the case where line thinning is not performed as shown in FIG. 5C. On the other hand, when line thinning is performed, the image resolution is lower than when line thinning is not performed.
[0055]
　In the above description, an example has been described in which images are captured line by line from the top end to the bottom end of the frame 200 in the rolling shutter method, but this is not limited to this example. 7A and 7B are diagrams schematically showing examples of other imaging methods in the rolling shutter system. For example, as shown in FIG. 7A, line-sequential imaging can be performed from the bottom end to the top end of the frame 200 in the rolling shutter method. In this case, the horizontal direction of the distortion of the image 202 is reversed compared to the case where the image is captured line by line from the top end to the bottom end of the frame 200 .

The scope of the claims
[Claim 1]
　an imaging unit having a pixel region in which a plurality of pixels are arranged;
　a readout control unit that controls readout of pixel signals from pixels included in the pixel region; and the readout
　control unit reads out the pixel signals. a first readout unit setting unit that sets a readout unit as part of a pixel region;
　an image output unit that outputs a first image based on the pixel signal read out from the readout unit to a subsequent stage; and the
　readout control . a second readout unit setting unit for setting the readout unit for reading out the pixel signal; and a second readout unit setting unit
　that learns teacher data for each readout unit and performs recognition processing on the pixel signal for each readout unit. and a recognition unit that outputs a recognition result of the recognition processing
.
[Claim 2]
2. The imaging apparatus according to claim 1 , wherein 　said first readout unit setting section
　determines said readout unit according to said recognition result .
[Claim 3]
　The first readout unit setting unit
　causes the image output unit to output the first image at a timing different from the output of the second image based on the pixel signals for the recognition unit to perform the recognition processing.
2. The imaging device of claim 1, wherein the imaging device performs:
[Claim 4]
　2. The imaging apparatus according to claim 1 , wherein said first
　readout unit setting unit does not set said readout unit to a region within a frame image in which a specific object has been recognized by said recognition unit .
[Claim 5]

　2. The imaging apparatus according to claim 1 , wherein 　said image output unit masks a region within a frame image in which a specific object has been recognized by said recognition unit.
[Claim 6]
5. The imaging apparatus according to claim 4 , wherein 　said image output unit
　masks an area read out after said recognition, out of areas within a frame image in which a specific object has been recognized by said recognition unit .
[Claim 7]
2. The imaging apparatus according to claim 1 , 　wherein said first readout unit setting section
　sets said readout unit according to a detection output of another sensor device.
[Claim 8]
　wherein the first readout unit setting unit sets the readout unit to
　an area in the frame image in which the specific object is recognized by the recognition unit, with priority over other areas in the frame image.
Item 1. The imaging device according to item 1.
[Claim 9]
　The first readout unit setting unit sets the readout unit
　in the other area so that the pixel signals are read out at a resolution lower than the resolution of the area in the frame image where the specific object is recognized. 9. The imaging device according to claim 8.
[Claim 10]
　9. The image capturing apparatus according to claim 8, wherein said first
　readout unit setting section sets said readout unit in said other area after reading of the area in the frame image in which said specific object has been recognized is completed.
.
[Claim 11]
2. The method according to claim 　1, wherein said first readout unit setting section sets,
　for said readout unit, imaging conditions different from imaging conditions set for said readout unit by said second readout unit setting section.
imaging device.
[Claim 12]
　Further comprising an accumulation control section for controlling accumulation of the pixel signals read from the readout unit in an accumulation section,
　wherein the accumulation control section controls
　at least one of the pixel signals and the recognition result of the recognition section. 2.
The imaging apparatus according to claim 1 , wherein the storage of said pixel signals in said storage unit is controlled according to said storage unit .
[Claim 13]
　The accumulation control unit,
　when a change amount of a second recognition result after a predetermined time from the first recognition result by the recognition unit with respect to the first recognition result by the recognition unit is equal to or less than a threshold,
13. The imaging apparatus according to claim 12, wherein the pixel signals read out correspondingly are not stored in the storage unit.
[Claim 14]
　The accumulation control section
　supplies the pixel signals to the accumulation section when the pixel signals for a unit for the image output section to output the first image to the subsequent stage are not accumulated in the accumulation section. 13. The imaging device according to claim 12, which stores the .
[Claim 15]
　3. The storage control unit
　discards the pixel signals stored in the storage unit when it is determined that there is a change in a scene to be imaged based on the pixel signals read by the readout control unit
. 13. The imaging device according to 12.
[Claim 16]
　3. The image output unit
　sequentially outputs the first image according to the pixel signals read out from the readout unit, when the processing unit of the first image is not frame unit
. 1. The imaging device according to 1.
[Claim 17]
　an imaging unit having a pixel region in which a plurality of pixels are arranged;
　a readout control unit that controls readout of pixel signals from pixels included in the pixel region; and the readout
　control unit reads out the pixel signals. a first readout unit setting unit that sets a readout unit as part of a pixel region;
　an image output unit that outputs a first image based on the pixel signal read out from the readout unit; and the
　readout control . a second readout unit control unit for setting the readout unit for reading out the
pixel
　signal ;
and an information processing device
including a recognition unit that performs recognition processing on an object and outputs a recognition result of the recognition processing .
[Claim 18]
　a readout control step , executed by a processor, of
　controlling readout of pixel signals from pixels included in a pixel region in which a plurality of pixels are arranged; and readout of the pixel signals by the readout control step
　. a first readout unit setting step of setting a readout unit as part of the pixel region;
　an image output step of outputting a first image based on the pixel signals read from the readout unit to a subsequent stage
　; a second readout unit control step of controlling the readout unit in which the pixel signal is read out by a readout control step; and
　learning teacher data for each readout unit, and performing recognition processing on the pixel signal for each readout unit. and a recognition step of outputting a recognition result as a result of the recognition processing
.
[Claim 19]
　a readout control step , executed by a processor, of
　controlling readout of pixel signals from pixels included in a pixel region in which a plurality of pixels are arranged; and readout of the pixel signals by the readout control step
　. a first readout unit setting step of setting a readout unit as part of the pixel region;
　an image output step of outputting a first image based on the pixel signals read from the readout unit to a subsequent stage
　; a readout unit control step of controlling the readout unit in which the pixel signal is read out by a readout control step; and
　learning teacher data for each readout unit, and performing recognition processing on the pixel signal for each readout unit.
an imaging program for causing a processor to execute a recognition step of outputting a recognition result as a result of recognition processing ;