Title of Invention: Imaging Apparatus, Imaging System, Imaging Method, and Imaging Program
Technical field
[0001]
The present disclosure relates to imaging devices, imaging systems, imaging methods, and imaging programs.
Background technology
[0002]
In recent years, along with the high performance of imaging devices such as digital still cameras, digital video cameras, compact cameras installed in multifunctional mobile phones (smartphones), etc., an image recognition function that recognizes a predetermined object contained in a captured image has been developed. is being developed.
prior art documents
patent literature
[0003]
Patent Document 1: JP 2017-112409 A
SUMMARY OF THE INVENTION
Problems to be Solved by the Invention
[0004]
However, conventionally, in order to execute the image recognition function, it was necessary to perform image processing on image data for one to several frames. Therefore, there is a problem that the processing time and power consumption for function realization increase.
[0005]
An object of the present disclosure is to provide an imaging device, an imaging system, an imaging method, and an imaging program capable of suppressing processing time and power consumption associated with function realization.
Means to solve problems
[0006]
An imaging device according to the present disclosure includes an imaging unit having a pixel region in which a plurality of pixels are arranged, a readout control unit that controls readout of pixel signals from pixels included in the pixel region, and a readout control unit that performs readout. , a readout unit control unit that controls the readout unit set as a part of the pixel region; and a recognition unit that has learned teacher data for each readout unit. performs recognition processing, and outputs the recognition result of the recognition processing.
Brief description of the drawing
[0007]
1 is a block diagram showing the configuration of an example of an imaging device applicable to each embodiment of the present disclosure; FIG.
2A is a schematic diagram showing an example of a hardware configuration of an imaging device according to each embodiment; FIG.
2B is a schematic diagram showing an example of the hardware configuration of an imaging device according to each embodiment; FIG.
3A is a diagram showing an example in which an imaging device according to each embodiment is formed by a laminated CIS having a two-layer structure; FIG.
3B is a diagram showing an example in which an imaging device according to each embodiment is formed by a laminated CIS having a three-layer structure; FIG.
4 is a block diagram showing an example configuration of a sensor section applicable to each embodiment; FIG.
5A is a schematic diagram for explaining a rolling shutter method; FIG.
5B is a schematic diagram for explaining a rolling shutter method; FIG.
5C is a schematic diagram for explaining a rolling shutter method; FIG.
6A is a schematic diagram for explaining thinning of lines in the rolling shutter method; FIG.
6B is a schematic diagram for explaining thinning of lines in the rolling shutter method; FIG.
6C is a schematic diagram for explaining thinning of lines in the rolling shutter method; FIG.
7A is a diagram schematically showing an example of another imaging method in the rolling shutter system; FIG.
7B is a diagram schematically showing an example of another imaging method in the rolling shutter system; FIG.
8A is a schematic diagram for explaining a global shutter method; FIG.
8B is a schematic diagram for explaining a global shutter method; FIG.
8C is a schematic diagram for explaining a global shutter method; FIG.
9A] A diagram schematically showing an example of a sampling pattern that can be realized in the global shutter method. [FIG.
9B] A diagram schematically showing an example of a sampling pattern that can be implemented in the global shutter method. [FIG.
10 is a diagram for schematically explaining image recognition processing by CNN; FIG.
11 is a diagram for schematically explaining image recognition processing for obtaining a recognition result from a part of an image to be recognized; FIG.
12A is a diagram schematically showing an example of identification processing by DNN when time-series information is not used; FIG.
12B is a diagram schematically showing an example of identification processing by DNN when time-series information is not used; FIG.
13A is a diagram schematically showing a first example of identification processing by DNN when time-series information is used; FIG.
13B is a diagram schematically showing a first example of identification processing by DNN when time-series information is used; FIG.
14A is a diagram schematically showing a second example of identification processing by DNN when time-series information is used; FIG.
14B is a diagram schematically showing a second example of identification processing by DNN when time-series information is used; FIG.
15A is a diagram for explaining the relationship between the driving speed of a frame and the readout amount of pixel signals; FIG.
15B is a diagram for explaining the relationship between the driving speed of a frame and the readout amount of pixel signals; FIG.
16 is a schematic diagram for schematically explaining recognition processing according to each embodiment of the present disclosure; FIG.
17 is a flowchart of an example of recognition processing by a recognition processing unit according to the first embodiment; FIG.
18 is a diagram showing an example of image data for one frame; FIG.
19 is a diagram for explaining the flow of machine learning processing executed by the recognition processing unit according to the first embodiment; FIG.
20A is a schematic diagram for explaining an application example of the first embodiment; FIG.
20B is a schematic diagram for explaining an application example of the first embodiment; FIG.
21 is a functional block diagram of an example for explaining functions of an imaging device according to a second embodiment; FIG.
22 is a schematic diagram showing in more detail an example of processing in a recognition processing unit according to the second embodiment; FIG.
23 is a functional block diagram of an example for explaining functions according to the second embodiment; FIG.
24 is a schematic diagram for explaining frame readout processing according to the second embodiment; FIG.
25 is a schematic diagram schematically showing recognition processing according to the second embodiment; FIG.
26 is a diagram for explaining an example of terminating recognition processing in the middle of frame reading; FIG.
27 is a diagram for explaining an example of terminating recognition processing in the middle of frame reading; FIG.
28 is a flow chart showing an example of recognition processing according to the second embodiment; FIG.
29A is a time chart showing an example of control of reading and recognition processing according to the second embodiment; FIG.
29B is a time chart showing an example of control of reading and recognition processing according to the second embodiment; FIG.
30 is an example time chart showing another example of control of reading and recognition processing according to the second embodiment. FIG.
31 is a schematic diagram for explaining frame readout processing according to the first modification of the second embodiment; FIG.
32 is a schematic diagram for explaining frame readout processing according to a second modification of the second embodiment; FIG.
33 is a schematic diagram for explaining frame readout processing according to the third modification of the second embodiment; FIG.
34 is a schematic diagram schematically showing recognition processing according to a third modification of the second embodiment; FIG.
35] Fig. 35 is a diagram for explaining an example of terminating recognition processing in the middle of frame reading when the reading unit is an area. [Fig.
36] Fig. 36 is a diagram for explaining an example of terminating recognition processing in the middle of frame reading when the reading unit is an area. [Fig.
37 is a schematic diagram for explaining frame readout processing according to the fourth modification of the second embodiment; FIG.
38 is a schematic diagram schematically showing recognition processing applicable to the fourth modification of the second embodiment; FIG.
39 is an example time chart showing an example of readout and control according to the fourth modification of the second embodiment; FIG.
40 is a diagram for more specifically explaining the frame readout process according to the fourth modification of the second embodiment; FIG.
41 is a diagram for more specifically explaining the frame readout process according to the fourth modification of the second embodiment; FIG.
42 is a schematic diagram for explaining frame readout processing according to the fifth modification of the second embodiment; FIG.
43 is a schematic diagram for explaining frame readout processing according to the sixth modification of the second embodiment; FIG.
44 is a diagram showing an example of patterns for reading and recognition processing according to the sixth modification of the second embodiment; FIG.
45 is a schematic diagram for explaining a first example of frame readout processing according to the seventh modification of the second embodiment; FIG.
46 is a schematic diagram for explaining frame readout processing according to a first alternative example of the seventh modification of the second embodiment; FIG.
47 is a schematic diagram for explaining frame readout processing according to a second alternative example of the seventh modification of the second embodiment; FIG.
48 is a schematic diagram for explaining frame readout processing according to a third alternative example of the seventh modification of the second embodiment; FIG.
49 is a functional block diagram of an example for explaining functions according to an eighth modification of the second embodiment; FIG.
50 is a flowchart of an example of recognition processing according to an eighth modification of the second embodiment; FIG.
51A is a diagram for explaining a first example of reading and recognition processing according to the eighth modification of the second embodiment; FIG.
51B is a diagram for explaining a first example of reading and recognition processing according to the eighth modification of the second embodiment; FIG.
52 is a diagram showing an example of an exposed pattern according to a ninth modification of the second embodiment; FIG.
53 is a functional block diagram of an example for explaining functions according to a tenth modification of the second embodiment; FIG.
54 is a flowchart of an example showing processing according to a tenth modification of the second embodiment; FIG.
55 is a schematic diagram for explaining the first process according to the tenth modification of the second embodiment; FIG.
56 is a schematic diagram for explaining second processing according to the tenth modification of the second embodiment; FIG.
57A is a schematic diagram for explaining third processing according to the tenth modification of the second embodiment; FIG.
57B is a schematic diagram for explaining third processing according to the tenth modification of the second embodiment; FIG.
58A is a schematic diagram for explaining third processing according to the tenth modification of the second embodiment; FIG.
58B is a schematic diagram for explaining third processing according to the tenth modification of the second embodiment; FIG.
59A is a schematic diagram for explaining third processing according to the tenth modification of the second embodiment; FIG.
59B is a schematic diagram for explaining third processing according to the tenth modification of the second embodiment; FIG.
60 is a schematic diagram showing in more detail an example of processing in the recognition processing unit according to the tenth modification of the second embodiment; FIG.
61 is a functional block diagram of an example for explaining functions according to a third embodiment; FIG.
62 is a schematic diagram showing an example of a readout unit pattern applicable to the third embodiment; FIG.
63 is a schematic diagram showing an example of a readout order pattern applicable to the third embodiment; FIG.
64 is a functional block diagram of an example for explaining functions according to the first modification of the third embodiment; FIG.
65 is a schematic diagram for explaining a first setting method of the first modification of the third embodiment; FIG.
66 is a schematic diagram for explaining a second setting method of the first modification of the third embodiment; FIG.
67 is a schematic diagram for explaining a third setting method of the first modification of the third embodiment; FIG.
68 is a functional block diagram of an example for explaining functions of an imaging device according to a third modified example of the third embodiment; FIG.
69 is a functional block diagram of an example for explaining functions of an imaging device according to a fourth embodiment; FIG.
70 is a schematic diagram for schematically explaining image processing according to the fourth embodiment; FIG.
71 is a diagram showing an example of read processing according to the fourth embodiment; FIG.
72 is a flow chart showing an example of processing according to the fourth embodiment; FIG.
73 is a diagram for explaining a third example of control of the image data storage unit according to the fourth embodiment; FIG.
74 is a diagram for explaining a first modification of the fourth embodiment; FIG.
[Fig. 75] Regarding the second modification of the fourth embodimentIt is a figure for explaining.
76 is a diagram for explaining a first example in the third modified example of the fourth embodiment; FIG.
77] Fig. 77 is a diagram illustrating a usage example of an imaging device to which the technique of the present disclosure is applied. [Fig.
78 is a block diagram showing an example of a schematic configuration of a vehicle control system; FIG.
79 is an explanatory diagram showing an example of installation positions of the vehicle exterior information detection section and the imaging section. FIG.

The scope of the claims
[Claim 1]
an imaging unit having a pixel area in which a plurality of pixels are arranged;
a readout control unit that controls readout of pixel signals from pixels included in the pixel region;
a readout unit control unit that controls a readout unit set as a part of the pixel region in which the readout control unit performs the readout;
a recognition unit that has learned the teacher data for each readout unit;
with
The recognition unit
perform recognition processing on the pixel signal for each readout unit, and output the recognition result of the recognition processing;
Imaging device.
[Claim 2]
The recognition unit
　Performs machine learning processing using RNN (Recurrent Neural Network) on the pixel data of the plurality of readout units in the same frame image, and executes the recognition processing based on the results of the machine learning processing
The imaging device according to claim 1 .
[Claim 3]
　The read unit control unit
When the recognition unit outputs the recognition result that satisfies a predetermined condition, the readout control unit is instructed to end the readout;
The imaging device according to claim 1 .
[Claim 4]
　The read unit control unit
The readout is performed so that, when the recognition unit acquires the recognition result candidate satisfying a predetermined condition, the readout unit is read at a position where acquisition of the recognition result satisfying the predetermined condition is expected. instruct the control unit,
The imaging device according to claim 1 .
[Claim 5]
　The read unit control unit
instructing the readout control unit to thin out the pixels included in the pixel region in the readout unit and read out the pixel signals, and when the candidate is output, the thinned out readout; instructing to read a readout unit that is expected to satisfy a predetermined condition among the units;
The imaging device according to claim 4.
[Claim 6]
　The read unit control unit
The readout control unit controls at least one of an exposure in the pixels included in the pixel region and a gain for the pixel signals read out from the pixels included in the pixel region based on the recognition result. instruct the control unit,
The imaging device according to claim 1 .
[Claim 7]
　The read unit control unit
The imaging according to claim 1, wherein the readout unit is controlled based on at least one of pixel information based on the pixel signal, recognition information generated by the recognition processing, and external information obtained from the outside. Device.
[Claim 8]
　The read unit control unit
The imaging device according to claim 1, wherein a line composed of a plurality of said pixels arranged in one row of said array is set as said readout unit.
[Claim 9]
　The read unit control unit
The imaging device according to claim 1, wherein a pattern composed of a plurality of pixels including non-adjacent pixels is set as the readout unit.
[Claim 10]
　The read unit control unit
arranging the plurality of pixels according to a predetermined rule to form the pattern,
The imaging device according to claim 9 .
[Claim 11]
　The read unit control unit
Priority is set for each of the plurality of readout units based on at least one of pixel information based on the pixel signal, recognition information generated by the recognition processing, and external information obtained from the outside. do,
The imaging device according to claim 1 .
[Claim 12]
　The read unit control unit
an area within the pixel area indicated by the recognition result is acquired based on the recognition information acquired in the recognition process, the readout unit is controlled based on the acquired area, and the recognition unit performs the recognition process next determine the area in which to
The imaging device according to claim 1 .
[Claim 13]
　The read unit control unit
Whether or not the recognizing unit reads out the pixel signals for further performing the recognition process on the area acquired based on the recognition information acquired in the recognition process, depending on the reliability of the recognition result of the area. instructing the read control unit whether
The imaging device according to claim 1 .
[Claim 14]
　The read unit control unit
Based on at least one of pixel information based on the pixel signal, recognition information generated by the recognition process, and external information obtained from the outside, exposure in the pixel included in the pixel region; the readout control to control at least one of a gain for the pixel signal read out from the pixel included in the pixel region, and read out the pixel signal from each of the pixels included in the pixel region. direct the department
The imaging device according to claim 1 .
[Claim 15]
　The read unit control unit
Among the readout unit, the exposure in the pixels included in the pixel region, and the gain for the pixel signals read out from the pixels included in the pixel region, according to an operation mode instructed from the outside. instructing the readout control unit to control at least one to read out pixel signals from each of the pixels included in the pixel region;
The imaging device according to claim 1 .
[Claim 16]
The imaging unit, the readout control unit, the readout unit control unit, and the recognition unit are integrally formed
The imaging device according to claim 1 .
[Claim 17]
The imaging section, the readout control section, and the readout unit control section are integrally formed, and the recognition section is placed outside the integrally formed structure.
The imaging device according to claim 1 .
[Claim 18]
an imaging unit having a pixel area in which a plurality of pixels are arranged;
a readout control unit that controls readout of pixel signals from pixels included in the pixel region;
a readout unit control unit that controls a readout unit set as a part of the pixel region in which the readout control unit performs the readout;
an imaging device comprising
the recognition unit that has learned the teacher data for each readout unit,
an information processing device comprising
including
The recognition unit
perform recognition processing on the pixel signal for each readout unit, and output the recognition result of the recognition processing;
imaging system.
[Claim 19]
　Executed by the processor,
a readout control step of controlling readout of pixel signals from pixels included in a pixel region in which a plurality of pixels are arranged, which the imaging unit has;
a readout unit control step of controlling a readout unit set as a part of the pixel region in which the readout is performed by the readout control step;
a recognition step in which the teacher data for each readout unit is learned;
including
The recognition step is
perform recognition processing on the pixel signal for each readout unit, and output the recognition result of the recognition processing;
Imaging method.
[Claim 20]
a readout control step of controlling readout of pixel signals from pixels included in a pixel region in which a plurality of pixels are arranged, which the imaging unit has;
a readout unit control step of controlling a readout unit set as a part of the pixel region in which the readout is performed by the readout control step;
a recognition step in which the teacher data for each readout unit is learned;
is executed by the processor, and
The recognition step is
perform recognition processing on the pixel signal for each readout unit, and output the recognition result of the recognition processing;
Imaging program for.