Title of Invention: Imaging Device, Control Method, and Program
Technical field
[0001]
　TECHNICAL FIELD The present technology relates to an imaging device, a control method, and a program, and more particularly, to an imaging device, a control method, and a program that can easily provide an image suitable for driving a vehicle, for example.
Background technology
[0002]
　For example, for a vehicle such as an automobile, a viewing system has been proposed in which a camera as a photographing device is installed at the rear of the vehicle and an image of the rear of the vehicle photographed by the camera is displayed.
[0003]
　The image of the rear of the vehicle provided by the viewing system includes an image of the area immediately behind the rear of the vehicle and an image of the area immediately behind the rear of the vehicle.
[0004]
　Here, the image that shows the rear from the rear of the vehicle is, for example, an image that corresponds to an image that is reflected in a class I mirror called a so-called interior rearview mirror (room mirror), and is hereinafter also referred to as a BM (Back Mirror) image. . An image showing the rear part of the vehicle and the area immediately behind the vehicle is an image showing the rear part of the vehicle and the area immediately behind it, and is hereinafter also referred to as an RV (Rear View) image.
[0005]
　When the vehicle approaches a slope or is traveling on an uneven road surface, the vehicle leans. In addition, the vehicle may tilt depending on the load to be loaded on the vehicle and the condition of the passengers of the vehicle.
[0006]
　When the vehicle tilts, the camera installed in the vehicle also tilts, and as a result, the ratio (percentage) of the sky, roads, etc. in the BM image changes from the state when the vehicle is not tilted.
[0007]
　When the viewing system is adjusted so that the BM image suitable for driving the vehicle is displayed when the vehicle is not tilted, the sky, roads, etc. appearing in the BM image may be affected by the tilt of the vehicle. If the ratio changes, BM images suitable for driving a vehicle may not be displayed.
[0008]
　For example, when the vehicle is traveling on a flat road surface, the BM image enables the driver to fully recognize other vehicles approaching from behind the vehicle and obstacles behind the vehicle. was displayed, but when the vehicle is tilted, there may be less information required for the driver to recognize such other vehicles or obstacles in the BM image.
[0009]
　Therefore, for example, a technology has been proposed in which an image captured by a camera is stored in a storage device at a later stage while driving on a slope, and a necessary image is cut out from the captured image stored in the storage device (for example, Patent Document 1). ).
prior art documents
patent literature
[0010]
Patent Document 1: Patent No. 6245274
SUMMARY OF THE INVENTION
Problems to be Solved by the Invention
[0011]
　When extracting a necessary image from a captured image stored in a storage device, it is necessary to read out from the camera a captured image whose size (the number of pixels) greatly exceeds the size of the required image. A large size of the captured image read out from the camera causes a drop in the frame rate.
[0012]
　In addition to the technology for extracting necessary images from the captured images stored in the storage device behind the camera, there is a demand for proposals for technology that can easily provide images suitable for driving a vehicle.
[0013]
　The present technology has been made in view of such a situation, and is intended to easily provide an image suitable for driving a vehicle.
Means to solve problems
[0014]
　An imaging device of the present technology includes an image sensor that captures an image displayed on a display unit of a vehicle, and a control unit that controls reading of an image from the image sensor based on vehicle information acquired by the vehicle. It is a photographing device provided.
[0015]
　A control method or program according to the present technology includes a control step of controlling readout of an image from an image sensor that captures an image displayed on a display unit of the vehicle, based on vehicle information acquired by the vehicle. A method or a program for causing a computer to function as a control unit that performs such control.
[0016]
　In the imaging device, control method, and program of the present technology, readout of an image from an image sensor that captures an image displayed on a display unit of the vehicle is controlled based on vehicle information acquired by the vehicle. .
[0017]
　The imaging device may be an independent device, or may be an internal block forming one device.
[0018]
　Also, the program can be provided by transmitting it via a transmission medium or by recording it on a recording medium.
Brief description of the drawing
[0019]
1 is a perspective view showing an external configuration example of a vehicle 10 equipped with a viewing system to which the present technology is applied; FIG.
2 is a perspective view showing a configuration example of the interior of the vehicle 10. FIG.
3 is a block diagram showing a first configuration example of a viewing system mounted on a vehicle 10; FIG.
4 is a diagram for explaining an example of readout control of a BM image and an RV image as readout images from a captured image by the control unit 43. FIG.
5 is a diagram explaining a first example of the relationship between the state of the vehicle 10 and the BM image; FIG.
6 is a diagram for explaining an example of readout control of a readout image according to tilt information by the control unit 43. FIG.
7 is a diagram showing a configuration example of an image sensor 32; FIG.
8 is a flowchart for explaining an example of display processing for displaying a BM image and performed by the first configuration example of the viewing system; FIG.
9 is a block diagram showing a second configuration example of the viewing system mounted on the vehicle 10; FIG.
10 is a diagram illustrating a second example of the relationship between the state of the vehicle 10 and the BM image; FIG.
11A and 11B are diagrams for explaining an example of readout control of a readout image according to tilt information by the control unit 71. FIG.
12 is a diagram for further explaining an example of readout control of a readout image according to tilt information by the control unit 71. FIG.
13 is a flowchart for explaining an example of display processing for displaying a BM image and performed by the second configuration example of the viewing system; FIG.
14 is a block diagram showing a third configuration example of the viewing system mounted on the vehicle 10. FIG.
15A and 15B are diagrams for explaining an example of readout control of a readout image according to tilt information by the control unit 71. FIG.
16 is a flowchart for explaining an example of display processing for displaying a BM image performed by the third configuration example of the viewing system; FIG.
17 is a block diagram showing a fourth configuration example of the viewing system mounted on the vehicle 10. FIG.
18 is a diagram for explaining an example of an image that can be output from the image sensor 32; FIG.
19 is a diagram illustrating an example of vehicle transmission bands that can be used for data transmission in vehicle 10. FIG.
20 is a diagram for explaining a first example of adjustment control of the data amounts of the BM image and the RV image by the control unit 136; FIG.
21 is a diagram for explaining a second example of adjustment control of the data amounts of the BM image and the RV image by the control unit 136; FIG.
22 is a flowchart illustrating an example of display processing for displaying BM images and RV images performed by the viewing system; FIG.
23 is a block diagram showing a fifth configuration example of the viewing system mounted on the vehicle 10. FIG.
24 is a diagram for explaining an example of extraction control of a BM image and an RV image from a photographed image by the control unit 181; FIG.
25 is a flowchart for explaining an example of display processing for displaying BM images and RV images performed by the viewing system; FIG.
26 is a block diagram showing a configuration example of an embodiment of a computer to which the present technology is applied; FIG.
MODE FOR CARRYING OUT THE INVENTION
[0020]
　
[0021]
　FIG. 1 is a perspective view showing an external configuration example of a vehicle 10 equipped with a viewing system to which the present technology is applied.
[0022]
　For example, a vehicle 10, which is a (four-wheeled) automobile, is provided with one camera unit 11 as a photographing device for photographing the rear of the vehicle 10 at its rear portion. In FIG. 1 , the camera unit 11 is installed above the rear glass of the vehicle 10 .
[0023]
　The camera unit 11 is a wide-angle camera unit (for example, the angle of view is 120 degrees or more) so that an image containing both the BM image range and the RV image range can be captured as one image. It has become. Furthermore, the camera unit 11 is a high-resolution (for example, resolution of 4K or higher) camera unit so that a distant subject can be clearly captured in the BM image. Therefore, the camera unit 11 can capture wide-angle and high-resolution images.
[0024]
　As will be described later, the camera unit 11 extracts a BM image and an RV image from images captured by the camera unit 11 (hereinafter also referred to as captured images).
[0025]
　In the BM image, if an interior rearview mirror (rearview mirror. Class I mirror in Regulation No.46 defined by the United Nations Economic Commission for Europe ) is installed in the vehicle 10, the interior rearview mirror The RV image includes an image of the rear portion of the vehicle 10 and the image immediately behind it (the rear portion) so as to include an image of the state behind the rear portion of the vehicle 10 that can be observed. The camera unit 11 is installed with the orientation of the optical axis adjusted so as to capture the image.
[0026]
　Therefore, if the vehicle 10 is equipped with an interior rearview mirror, the BM image is an image that captures the situation behind the rear of the vehicle 10, which can be observed through the interior rearview mirror. Also, the RV image is an image showing the rear portion of the vehicle 10 and the state immediately after that. Since the RV image shows the rear part of the vehicle 10, which is a blind spot in the interior rearview mirror, the RV image is particularly useful when the vehicle 10 is moving backward. Also, the RV image can be used to generate a bird's-eye view image of the vehicle 10 viewed from above.
[0027]
　Note that the position where the camera unit 11 is installed is not limited to the top of the rear glass of the vehicle 10 as long as it is possible to capture a captured image from which a BM image and an RV image can be extracted as described above. For example, the camera unit 11 can be installed above the rear glass of the vehicle 10, or at a position P11 above the license plate at the rear of the vehicle 10, or the like.
[0028]
　FIG. 2 is a perspective view showing a configuration example inside the vehicle 10 of FIG.
[0029]
　A BM display unit 21 for displaying a BM image is provided at a position where an interior rearview mirror is installed inside the vehicle 10 . The BM display unit 21 is a display unit that can replace the indoor rearview mirror.
[0030]
　An RV display unit 22 for displaying an RV image is provided at the central position of the dashboard inside the vehicle 10 .
[0031]
　In addition, an in-vehicle camera 23 for capturing an image of the driver is provided on the driver's seat side of the dashboard in the vehicle 10 . The in-vehicle camera 23 takes an image of the driver and outputs an image of the driver. In the vehicle 10, the line of sight and the position of the head of the driver are detected from the image of the driver.
[0032]
　Here, the in-vehicle camera 23 that captures the image of the driver can be provided at any position other than the dashboard, for example, the position P21 above the BM display section 21 or the like.
[0033]
　
[0034]
　FIG. 3 is a block diagram showing a first configuration example of the viewing system mounted on the vehicle 10. As shown in FIG.
[0035]
　The viewing system has the camera unit 11, BM display section 21, and RV display section 22 described in FIGS.
[0036]
　The camera unit 11 has an optical system 31 , an image sensor 32 , an output section 33 , an acquisition section 41 , a detection section 42 and a control section 43 .
[0037]
　The optical system 31 is composed of optical components such as a condenser lens and a diaphragm, and condenses light incident on the optical system 31 onto the image sensor 32 .
[0038]
　The image sensor 32 receives light from the optical system 31 and photoelectrically converts the light to capture a captured image including an image that becomes a BM image and an RV image. Then, the image sensor 32 reads and outputs the BM image and the RV image (images to become) from the captured image under the control of the control unit 43 . Here, an image that the image sensor 32 reads out from the captured image and outputs is also called a readout image. A read image output by the image sensor 32 is supplied to the output unit 33 .
[0039]
　The output unit 33 is an output IF (Interface) that transmits the BM image and the RV image as readout images from the image sensor 32 to the outside of the camera unit 11, and transmits the BM image to the BM display unit 21. RV images are transmitted to the RV display unit 22 . The BM display unit 21 displays the BM image from the output unit 33 according to the specifications of the BM display unit 21, and the RV display unit 22 displays the RV image from the output unit 33 according to the specifications of the RV display unit 22. . The output unit 33 can perform format conversion and other image processing of the BM image and the RV image as necessary.
[0040]
　The acquisition unit 41 acquires (receives) vehicle information acquired by the vehicle 10 from the vehicle 10 via a network (vehicle information network) built in the vehicle 10 and supplies the vehicle information to the detection unit 42 .
[0041]
　Here, the vehicle information includes, for example, gyro information obtained from a gyro included in the vehicle 10, suspension information about the suspension of the vehicle 10, a front camera image obtained from a front camera that captures the front of the vehicle 10, and GPS (Global GPS information obtained from a positioning system), driving information representing the driving state such as the vehicle speed and driving direction (forward or backward) of the vehicle 10, the line of sight and head of the driver of the vehicle 10 obtained from the image captured by the in-vehicle camera 23 parts, 3D (Dimension) maps used in the navigation system of the vehicle 10, high-definition maps used in ADAS (Advanced Driving Assist System) / automatic driving systems, etc. can be acquired by the vehicle 10. Including any information. As the vehicle speed, for example, when the vehicle 10 is equipped with a speed sensor, speed information output by the speed sensor can be used. Gear information representing the state of the transmission, for example, can be used as the traveling direction.
[0042]
　The acquisition unit 41 acquires one or more pieces of vehicle information as necessary, and supplies the information to the detection unit 42 .
[0043]
　In the first configuration example of the viewing system, the acquisition unit 41 acquires, for example, gyro information or GPS information and a 3D map (high-definition map) as vehicle information, and supplies the information to the detection unit 42 .
[0044]
　The detection unit 42 detects (calculates) tilt information representing the tilt of the vehicle 10 using the vehicle information from the acquisition unit 41 and supplies the tilt information to the control unit 43 .
[0045]
　The control unit 43 controls reading of readout images from the image sensor 32 based on vehicle information.
[0046]
　For example, the control unit 43 controls readout of the readout image from the image sensor 32 according to tilt information detected by the detection unit 42 from the vehicle information.
[0047]
　That is, the control unit 43 calculates the readout start position of an image at which reading is started as a readout image from the captured image captured by the image sensor 32 according to the tilt information supplied from the detection unit 42, and By setting the image size (the number of pixels) and supplying the image sensor 32 with the readout start position and the readout position specified by the size, readout control for controlling the readout of the readout image from the image sensor 32 is performed. conduct.
[0048]
　The image sensor 32 reads the pixel signal of the pixel at the readout position from the control unit 43 and outputs a readout image having the pixel signal as the pixel value.
[0049]
　The readout position can be specified by the readout start position and the size, and can also be specified by, for example, the readout start position and the readout end position where the readout of the readout image ends.
[0050]
　Here, in the first configuration example of the viewing system, the readout image output by the image sensor 32 matches the BM image displayed on the BM display section 21 and the RV image displayed on the RV display section 22 . Therefore, in the first configuration example of the viewing system, the BM image displayed on the BM display section 21 and the RV image displayed on the RV display section 22 are set as the size of the readout image.
[0051]
　
[0052]
　FIG. 4 is a diagram illustrating an example of readout control of a BM image and an RV image as readout images from captured images by the control unit 43. In FIG.
[0053]
　The captured image captured by the optical system 31 and the image sensor 32 is, for example, as shown in FIG. It has a positional relationship that
[0054]
　In the readout control, the control unit 43 selects a predetermined area in which the rear from the rear of the vehicle 10 is captured (if the vehicle 10 is equipped with an interior rearview mirror, the area observed by the interior rearview mirror). ) Controls readout of pixel signals from the image sensor 32 so as to extract R11 as a BM image. That is, it controls readout of pixel signals from the image sensor 32 according to the display device and the type (/application) of the image display function.
[0055]
　Further, in the readout control, the control unit 43 supplies the readout position of the readout image to the image sensor 32, so that the rear part of the vehicle 10 and immediately after that are detected from the captured image (the image within the image circle). Readout of pixel signals from the image sensor 32 is controlled so as to extract a predetermined region R12 in which is captured as an RV image.
[0056]
　Under the control of the control unit 43, the image sensor 32 reads the pixel signals of the region R11 that will be (the pixel values ​​of) the BM image from the photographed image obtained by photographing, and outputs the readout image having the pixel signals as the pixel values. At the same time, it reads out the pixel signals of the region R12 that will be the RV image, and outputs the readout image with the pixel signals as the pixel values.
[0057]
　The regions R11 and R12 are designated by the readout positions of the readout image supplied from the control section 43 to the image sensor 32. FIG.
[0058]
　In addition, in the readout control of the BM image as the readout image, the control unit 43 controls the readout position for specifying the region R11 to be extracted as the BM image according to the line of sight and the position of the head of the driver included in the vehicle information. can be calculated.
[0059]
　That is, if the vehicle 10 is provided with a rearview mirror, the range of the image that the driver can see through the rearview mirror changes as the driver moves his or her line of sight or head. The control unit 43 reads the BM image as a readout image so that the driver can see the BM image showing the same range as the image that can be observed when the rearview mirror is installed in the vehicle 10. In the readout control, it is possible to calculate the readout position that designates the region R11 to be extracted as the BM image according to the line of sight and the position of the head of the driver.
[0060]
　To simplify the explanation, the explanation of the RV image is omitted below.
[0061]
　
[0062]
　FIG. 5 is a diagram illustrating a first example of the relationship between the state of the vehicle 10 and the BM image.
　FIG. 5A1 is a diagram showing the state of the vehicle 10 when the vehicle 10 is traveling on a flat road surface. Also, FIG. 5A2 is an example of a BM image when the vehicle 10 is traveling on a flat road surface.
　FIG. 5(B1) is a diagram showing the state of the vehicle 10 when the vehicle 10 is about to go downhill. FIG. 5(B2) is an example of a BM image when the vehicle 10 is about to go downhill.
　FIG. 5(C1) is a diagram showing the state of the vehicle 10 when the vehicle 10 is approaching an uphill. FIG. 5(C2) is an example of a BM image when the vehicle 10 is approaching an uphill.
[0063]
　Now, let us say that the range of the three-dimensional space captured in the BM image is the BM range. Also, in the following description, it is assumed that the line of sight and the head position of the driver are fixed in order to simplify the explanation. Therefore, it is assumed that (the position of) the region R11 extracted as the BM image from the photographed image and, by extension, the BM range does not change due to the driver's line of sight or the position of the head.
[0064]
　Further, in the control unit 43, when the vehicle 10 is positioned on a flat road surface (when the vehicle 10 is not tilted), as shown in FIG. However, it is assumed that readout control, that is, calculation of the readout position (readout start position and size) is performed so that a BM image captured substantially in the center of the screen is extracted.
[0065]
　In this case, when the vehicle 10 approaches a downhill, the vehicle 10 inclines forward in the pitch direction, and the optical axis of the camera unit 11 installed at the rear of the vehicle 10, and thus the BM range, becomes the flat road surface of the vehicle 10. 5 (B1), 5 (B2)).
[0066]
　As a result, when the vehicle 10 is positioned on a flat road surface (when the vehicle is traveling on a flat road surface), the vehicle behind the vehicle 10, which is a predetermined distance behind the vehicle 10, appears almost in the center of the screen of the BM image. In the case of camera setting, when the vehicle 10 is approaching a downhill, a BM image is obtained in which the vehicle behind the vehicle 10 is shown at the bottom of the screen at a predetermined distance behind the vehicle 10 . As shown in FIG. 5, this BM image has a higher ratio of the sky (the region in which the image is captured) and a smaller ratio of the road compared to the BM image in which the vehicle 10 is positioned on a flat road surface. becomes an image.
[0067]
　Therefore, in the BM image when the vehicle 10 is approaching a downhill, the amount of information changes compared to the BM image when the vehicle 10 is positioned on a flat road surface. That is, for example, in the BM image when the vehicle 10 is approaching a downhill, the amount of information regarding the road is reduced compared to the BM image when the vehicle 10 is positioned on a flat road surface. Also, depending on the relative positions of the vehicle 10 and the vehicle behind and the inclination of the downhill, part of the vehicle behind or the vehicle itself behind may not be captured.
[0068]
　On the other hand, when the vehicle 10 approaches an uphill slope, the vehicle 10 inclines to the rear side in the pitch direction, and the optical axis of the camera unit 11 installed at the rear of the vehicle 10 and the BM range of the vehicle 10 become flat on the road surface. (C1 in FIG. 5, C2 in FIG. 5).
[0069]
　As a result, when the vehicle 10 is positioned on a flat road surface, when the camera setting is such that the vehicle behind the vehicle 10 at a predetermined distance behind the vehicle 10 is captured in the center of the screen of the BM image, the vehicle 10 is on an uphill. In the approaching state, a BM image is obtained in which the rear vehicle, which is a predetermined distance behind the vehicle 10, appears in the upper part of the screen. As shown in FIG. 5, this BM image has a smaller ratio of the sky (the region in which the image is captured) and a larger ratio of the road compared to the BM image when the vehicle 10 is positioned on a flat road surface. becomes an image.
[0070]
　Therefore, in the BM image when the vehicle 10 is approaching an uphill, the amount of information changes compared to the BM image when the vehicle 10 is positioned on a flat road surface. That is, for example, in the BM image when the vehicle 10 is approaching an uphill, the sky ratio is smaller than in the BM image when the vehicle 10 is positioned on a flat road surface. decreases. Also, depending on the relative positions of the vehicle 10 and the vehicle behind, or the inclination of the uphill, a part of the vehicle behind or the vehicle itself behind may not be captured.
[0071]
　As described above, in the BM image when the vehicle 10 is approaching a downhill or an uphill, the change in the amount of information from the BM image when the vehicle 10 is positioned on a flat road surface indicates that the vehicle 10 is driving. This is not preferable from the viewpoint of providing a person with a BM image suitable for driving the vehicle 10 .
[0072]
　Therefore, in the readout control of the BM image, the control unit 43 controls readout of the BM image as a readout image from the image sensor 32 in accordance with the tilt information from the detection unit 42 , so that regardless of the state of the vehicle 10 . First, a BM image having the same amount of information as the BM image obtained when the vehicle 10 is positioned on a flat road surface is obtained.
[0073]
　
[0074]
　FIG. 6 is a diagram for explaining an example of readout control of a readout image according to tilt information by the control unit 43. As shown in FIG.
[0075]
　Now, when the vehicle 10 is positioned on a flat road surface, the control unit 43 extracts (pixel signals of pixels of) a rectangular area R101 from an image captured by the image sensor 32 (the light receiving surface of the image sensor 32). It is assumed that read control is performed so that The size of the region R101 matches the size of the BM image as the readout image.
[0076]
　For example, when the vehicle 10 is approaching an uphill slope and the vehicle 10 is tilted backward, the control unit 43 selects the number of pixels from the region R101 according to the tilt angle of the vehicle 10 to the rear side according to the tilt information. A readout position that designates a region R102 having the same size as the region R101, which is shifted upward by the amount, is calculated, and readout control is performed so as to read out pixel signals at the readout position.
[0077]
　Further, for example, when the vehicle 10 is approaching a downhill and the vehicle 10 is tilted forward, the control unit 43 selects the number of pixels from the region R101 according to the tilt angle to the front of the vehicle 10 according to the tilt information. A readout position that designates a region R103 having the same size as the region R101, which is shifted downward by the amount, is calculated, and readout control is performed so as to read out pixel signals at the readout position.
[0078]
　In the calculation of the readout position according to the tilt information, the readout image including the road and the sky is read out at the same ratio as the BM image when the vehicle 10 is positioned on a flat road surface. The readout position is calculated so that the road/sky ratio is maintained at (or close to) the ratio of the BM image when the vehicle 10 is positioned on a flat road surface.
[0079]
　By performing the readout control as described above in the control unit 43, an image suitable for driving the vehicle 10 can be easily obtained. That is, regardless of the pitch direction tilt of the vehicle 10, it is possible to easily obtain a BM image having the same amount of information as the BM image obtained when the vehicle 10 is positioned on a flat road surface.
[0080]
　In the first configuration example of the viewing system, the detection unit 42 detects a (steep) slope from gyro information as vehicle information supplied from the acquisition unit 41, GPS information, a 3D map, or the like, and Inclination information of the vehicle 10 when the vehicle 10 approaches a slope, that is, inclination information mainly representing (degree of) inclination of the vehicle 10 toward the front side and rear side in the pitch direction is detected (calculated). The detection unit 42 then supplies the tilt information to the control unit 43 .
[0081]
　The detection unit 42 detects that the vehicle 10 has approached a slope using, for example, gyro information, and further detects inclination information representing the inclination of the vehicle 10 toward the front and rear sides in the pitch direction at that time. can do.
[0082]
　Further, the detection unit 42 detects (estimates) that the vehicle 10 is approaching a slope based on the current location and the 3D map obtained using GPS information, and further detects (estimates) the inclination of the slope obtained using the 3D map. can be used to detect (estimate) inclination information representing the inclination of the vehicle 10 to the front and rear sides in the pitch direction when the vehicle 10 approaches a slope.
[0083]
　The control unit 43 calculates the readout position of the image to be read as the readout image from the image sensor 32 according to the tilt information from the detection unit 42, and supplies it to the image sensor 32, thereby reading out the readout image from the image sensor 32. to control.
[0084]
　The image sensor 32 reads the pixel signal of the pixel at the readout position from the control unit 43 and outputs a readout image having the pixel signal as the pixel value.
[0085]
　
[0086]
　FIG. 7 is a diagram showing a configuration example of the image sensor 32 of FIG.
[0087]
　The image sensor 32 has a pixel array 51 , an input circuit 52 , a row selection circuit 53 , a column selection circuit 54 , an AD (Analog to Digital) converter 55 , a line buffer 56 and an output circuit 57 .
[0088]
　The pixel array 51 is configured by arranging a plurality of pixels 61 on a two-dimensional plane. The area where the pixels 61 are arranged in the pixel array 51 is the light receiving surface of the image sensor 32 .
[0089]
　The pixel 61 converts light incident thereon into a pixel signal as an electrical signal corresponding to the light amount of the light. From the pixel array 51 , the pixel signals of the pixels 61 in the row selected by the row selection circuit 53 and the column selected by the column selection circuit 54 are read out through the column selection circuit 54 and sent to the AD converter 55 . supplied.
[0090]
　The input circuit 52 is supplied from the control unit 43 with a readout position specified by a readout start position of a readout image to be read out from the captured image and a size of the readout image (hereinafter also referred to as a readout size).
[0091]
　The input circuit 52 uses the readout start position and the readout size as the readout position from the control unit 43 to, for example, in raster scan order, the pixel 61 from among the pixels 61 of the pixel array 51 to start reading out the pixel signal. are calculated as readout start coordinates (X_STA, Y_STA), and the coordinates of the pixel 61 at which the readout of the pixel signal ends are calculated as readout end coordinates (X_END, Y_END).
[0092]
　The input circuit 52 supplies the y-coordinate Y_STA of the read start coordinates (X_STA, Y_STA) and the y-coordinate Y_END of the read end coordinates (X_END, Y_END) to the row selection circuit 53, and supplies the read start coordinates (X_STA, Y_STA). ) and the x-coordinate X_END of the read end coordinates (X_END, Y_END) are supplied to the column selection circuit 54 (process PR1).
[0093]
　The row selection circuit 53 sequentially selects each row from the row of the pixels 61 indicated by the y-coordinate Y_STA from the input circuit 52 to the row of the pixels 61 indicated by the y-coordinate Y_END (process PR2).
[0094]
　In the pixel array 51 , pixel signals are read from the pixels 61 in the row selected by the row selection circuit 53 and supplied to the column selection circuit 54 .
[0095]
　The column selection circuit 54 selects the pixels 61 in each column from the column of the pixels 61 indicated by the x-coordinate X_STA from the input circuit 52 to the column of the pixels 61 indicated by the x-coordinate X_END in the pixel signals read from the pixels 61 . is selected and supplied to the AD converter 55 (process PR3).
[0096]
　The AD converter 55 AD-converts the pixel signal from the column selection circuit 54, for example, in units of one row, and supplies the pixel signal after AD conversion to the line buffer 56 (process PR4).
[0097]
　The line buffer 56 temporarily stores pixel signals from the AD converter 55 .
[0098]
　The output circuit 57 reads the pixel signals stored in the line buffer 56 pixel by pixel (process PR5), and outputs them to the outside of the image sensor 32 as pixel values ​​of the read image.
[0099]
　As described above, in the image sensor 32, the pixel 61 at the readout start coordinates (X_STA, Y_STA) is set as the upper left vertex, and the pixel 61 at the readout end coordinates (X_END, Y_END) is set as the lower right vertex. A rectangular area is used as a readout area, pixel signals of pixels 61 in the readout area are read out, and an image having the pixel signals as pixel values ​​is output as a readout image.
[0100]
　
[0101]
　FIG. 8 is a flowchart for explaining an example of display processing for displaying a BM image performed by the first configuration example of the viewing system in FIG.
[0102]
　In step S11, the acquisition unit 41 acquires, for example, gyro information or GPS information and a 3D map as vehicle information, supplies them to the detection unit 42, and the process proceeds to step S12.
[0103]
　In step S12 , the detection unit 42 uses the vehicle information from the acquisition unit 41 to detect a slope. Furthermore, the detection unit 42 uses the vehicle information from the acquisition unit 41 to obtain tilt information of the vehicle 10 when the vehicle 10 approaches a slope (mainly, tilt information representing tilts to the front and rear sides in the pitch direction). ) is detected (calculated) and supplied to the control unit 43, and the process proceeds from step S12 to step S13.
[0104]
　In step S13, the control unit 43 calculates the readout start position of the readout image from the image sensor 32 according to the tilt information from the detection unit 42, and calculates the readout start position and the size of the BM image as the readout position. , is supplied to the image sensor 32, and the process proceeds to step S14.
[0105]
　In step S14, the image sensor 32 reads the pixel signal of the pixel at the readout position from the control unit 43, acquires and outputs a readout image having the pixel signal as the pixel value. The readout image output by the image sensor 32 is supplied to the output unit 33, and the process proceeds from step S14 to step S15.
[0106]
　In step S15, the output unit 33 transmits the image read from the image sensor 32 as a BM image to the BM display unit 21 for display. As a result, the BM image is displayed on the BM display unit 21, and the display processing ends.
[0107]
　As described above, the first configuration example of the viewing system mainly detects tilt information representing tilts toward the front side and the rear side as the pitch direction, and reads from the image sensor 32 according to the tilt information. Since image reading is controlled, an image suitable for driving the vehicle 10 can be easily provided. That is, when the vehicle 10 approaches a slope, it is possible to easily provide a BM image with the same amount of information as the BM image when the vehicle 10 is positioned on a flat road surface.
[0108]
　Furthermore, in the first configuration example of the viewing system, a readout image having the same size as the BM image is read out from the image sensor 32 in readout control, so an image having a size larger than the BM image is read out from the image sensor 32. Compared to the case, it is possible to suppress the possibility that the frame rate of the BM image is lowered.
[0109]
　As the vehicle information in the first configuration example, the gyro information, the GPS information, and the 3D map have been described. However, as vehicle information in the first configuration example, suspension information, a front camera image, or the like may be used as in the second configuration example and the third configuration example described later. The inclination of the vehicle may be detected using the suspension information, the front camera image, and the like.
[0110]
　
[0111]
　FIG. 9 is a block diagram showing a second configuration example of the viewing system mounted on the vehicle 10. As shown in FIG.
[0112]
　In the figure, parts corresponding to those in FIG. 3 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.
[0113]
　In FIG. 9, the viewing system has a camera unit 11, a BM display section 21, and an RV display section 22. FIG. Furthermore, in FIG. 9 , the camera unit 11 has an optical system 31 , an image sensor 32 , an output section 33 , an acquisition section 41 , a detection section 42 , a control section 71 and a processing section 72 .
[0114]
　Therefore, the second configuration example of the viewing system shown in FIG. 9 is common to the case shown in FIG.
[0115]
　However, in the second configuration example of the viewing system of FIG. It is different from the case of 3.
[0116]
　Similar to the control unit 43 , the control unit 71 controls readout of readout images from the image sensor 32 in accordance with the tilt information supplied from the detection unit 42 .
[0117]
　However, the control unit 71 calculates the readout start position and the readout size of the readout image from the captured image captured by the image sensor 32 according to the tilt information supplied from the detection unit 42, and calculates the readout start position of the readout image. and the readout size are supplied to the image sensor 32 .
[0118]
　That is, in the first configuration example of the viewing system, the size of the readout image matches the size of the BM image, so the readout size is set to the size of the BM image. In the example, the readout size is calculated according to the tilt information.
[0119]
　Here, in the second configuration example of the viewing system, the acquisition unit 41 acquires vehicle information, such as suspension information, and supplies it to the detection unit 42 .
[0120]
　In this case, the detection unit 42 uses the suspension information from the acquisition unit 41 to detect leftward and rightward tilts in the roll direction, and detects (calculates) tilt information representing (degree of) the tilts. The detection unit 42 then supplies the tilt information to the control unit 71 .
[0121]
　The control unit 71 calculates a rotation angle for rotating the readout image output by the image sensor 32 according to the tilt information from the detection unit 42 and supplies the rotation angle to the processing unit 72 . As described above, by supplying the rotation angle to the processing unit 72, the control unit 71 controls the rotation of the readout image by the processing unit 72 so that the readout image is rotated by the rotation angle.
[0122]
　A read image output by the image sensor 32 is supplied to the processing unit 72 .
[0123]
　The processing unit 72 rotates the read image according to the rotation angle from the control unit 71 . Further, the processing unit 72 cuts out an image of the size of the BM image (hereinafter also referred to as BM size) from the readout image after rotation, and supplies it to the output unit 33 as the BM image.
[0124]
　As described above, the processing unit 72 rotates the readout image, and cuts out a BM size image from the rotated readout image. Therefore, the readout image needs to be an image having a size such that a BM size image can be cut out from the rotated readout image.
[0125]
　Therefore, the control unit 71 calculates, as the readout size, the minimum size of the readout image with which a BM size image can be cut out from the readout image after rotation, for example, according to the tilt information.
[0126]
　
[0127]
　FIG. 10 is a diagram explaining a second example of the relationship between the state of the vehicle 10 and the BM image.
　FIG. 10(A1) is a diagram showing the state of the vehicle 10 when the vehicle 10 is traveling on a flat road surface. Also, FIG. 10A2 is an example of a BM image when the vehicle 10 is traveling on a flat road surface.
　FIG. 10(B1) is a diagram showing the state of the vehicle 10 when the vehicle 10 is positioned on a road surface with a higher step on the right side than on the left side with respect to the forward direction. FIG. 10(B2) is an example of a BM image when the vehicle 10 is positioned on a road surface with a step higher on the right side than on the left side with respect to the forward direction.
　FIG. 10(C1) is a diagram showing the state of the vehicle 10 when the vehicle 10 is positioned on a road surface with a higher step on the left side than on the right side with respect to the forward direction. FIG. 10(C2) is an example of a BM image when the vehicle 10 is positioned on a road surface with a higher step on the left side than on the right side with respect to the forward direction.
[0128]
　In FIG. 10 as well, as in FIG. 5, it is assumed that the region R11 extracted as the BM image from the captured image does not change due to the driver's line of sight or head position movement.
[0129]
　In this case, when the vehicle 10 is positioned on a road surface with a higher step on the right side than the left side with respect to the forward direction, the vehicle 10 tilts to the left side in the roll direction, and the camera unit 11 installed at the rear of the vehicle 10 also does the same. tilts in the roll direction.
[0130]
　As a result, when the vehicle 10 is positioned on a flat road surface (FIG. 10(A)), a horizontal line appears in the BM image so as to extend horizontally. When the vehicle 10 is positioned on a road surface with a high level difference on the right side, a BM image that appears downward to the right (rising to the left) is obtained (FIG. 10(B)). As shown in FIG. 10, this BM image is an image in which the range of the sky and the road shown in the BM image has changed compared to the BM image when the vehicle 10 is positioned on a flat road surface. That is, the BM image (FIG. 10B) when the vehicle 10 is positioned on a road surface with a high level difference on the right side is compared with the BM image when the vehicle 10 is positioned on a flat road surface. As the ratio of roads decreases, the ratio of sky on the right side increases, and as the ratio of roads on the left side increases, the ratio of sky on the left side decreases.
[0131]
　Therefore, in the BM image when the vehicle 10 is positioned on a road surface with a high step on the right side, the content changes compared to the BM image when the vehicle 10 is positioned on a flat road surface.
[0132]
　On the other hand, when the vehicle 10 is positioned on a road surface with a higher level difference on the left side than the right side with respect to the forward direction, the vehicle 10 tilts to the right side in the roll direction, and the camera unit 11 installed at the rear of the vehicle 10 similarly. tilt in the roll direction.
[0133]
　As a result, when the vehicle 10 is positioned on a road surface with a high level difference on the left side, a BM image is obtained in which the horizontal line rises to the right (falls to the left) (FIG. 10(C)). As shown in FIG. 10, this BM image is an image in which the range of the sky and the road shown in the BM image has changed compared to the BM image when the vehicle 10 is positioned on a flat road surface. That is, in the BM image when the vehicle 10 is positioned on a road surface with a high step on the left side (FIG. 10C), the BM image when the vehicle 10 is positioned on a flat road surface is compared with the BM image when the vehicle 10 is positioned on a flat road surface. As the ratio of roads increases, the ratio of sky on the right side decreases, and as the ratio of roads on the left side decreases, the ratio of sky on the left side increases.
[0134]
　Therefore, in the BM image when the vehicle 10 is positioned on a road surface with a high level difference on the left side, the content changes compared to the BM image when the vehicle 10 is positioned on a flat road surface.
[0135]
　As described above, the fact that the BM image when the vehicle 10 is positioned on a road surface with a high step on the left or right side is different from the BM image when the vehicle 10 is positioned on a flat road surface is From the viewpoint of providing the driver of the vehicle 10 with a BM image suitable for driving the vehicle 10, this is not preferable.
[0136]
　Therefore, in the readout control of the BM image, the control unit 71 controls readout of the BM image as a readout image from the image sensor 32 in accordance with the tilt information from the detection unit 42, thereby enabling the image to be read regardless of the state of the vehicle 10. First, a BM image having the same amount of information as the BM image obtained when the vehicle 10 is positioned on a flat road surface is obtained.
[0137]
　
[0138]
　11A and 11B are diagrams for explaining an example of readout control of a readout image according to the tilt information by the control unit 71. FIG.
　FIG. 11A on the left side shows the readout area on the image sensor 32 in each case. FIG. 11(B) on the right side is a diagram showing a cut-out region in the processing portion 72. As shown in FIG.
[0139]
　As described with reference to FIG. 6, in the control unit 71, when the vehicle 10 is positioned on a flat road surface, the captured image captured by the image sensor 32 (the light receiving surface of the image sensor 32) matches the BM image. Readout control is performed so that (the pixel signals of the pixels in) a rectangular area R101 of the size is read out.
[0140]
　For example, when the vehicle 10 is located on a road surface with a step and the vehicle 10 tilts in the roll direction, the control unit 71 determines the number of pixels corresponding to the tilt angle of the vehicle 10 in the roll direction according to the tilt information. A readout position that designates a region R111 larger than the region R101 by the same amount is calculated, and readout control is performed so as to read out pixel signals at the readout position.
[0141]
　Here, in the BM image when the vehicle 10 is tilted in the roll direction, as shown in FIG. 10, a horizontal line extending horizontally appears in the BM image when the vehicle 10 is positioned on a flat road surface. tilt.
[0142]
　Therefore, in the processing unit 72, the readout image output by the image sensor 32 according to the readout control is rotated so that the horizontal line appearing in the readout image extends in the horizontal direction, and the BM image (becomes) is obtained from the readout image after rotation. image) is cut out.
[0143]
　In the calculation of the readout position according to the tilt information in the control unit 71, for example, a region R111 centered at the center (center of gravity) of the region R101 with the minimum size that allows the BM image to be cut out from the readout image after rotation. A readout start position and a readout size as the designated readout position are calculated and supplied to the image sensor 32 .
[0144]
　Furthermore, the control unit 71 rotates the readout image region R111 according to the tilt information, for example, so that the horizontal line appearing in the readout image region R111 output by the image sensor 32 extends horizontally. A rotation angle is calculated and supplied to the processing unit 72 .
[0145]
　In the image sensor 32 , a rectangular area R 111 specified by the readout start position and the readout size as the readout position from the control unit 71 is read out as a readout image and supplied to the processing unit 72 .
[0146]
　In the processing unit 72 , the region R111 as the readout image from the image sensor 32 is rotated by the rotation angle from the control unit 71 . Then, in the processing unit 72, a region R113 having a rotation angle of 0 degrees and having the same size as the BM image is cut out as a BM image from the region R112 after rotation of the region R111 (FIG. 11B).
[0147]
　In the readout image of FIG. 11A, the vehicle 10 is tilted to the right in the roll direction, and the horizontal line appearing in the readout image is tilted upward to the right. Therefore, in the processing unit 72, the region R111 as the readout image is rotated to the right (clockwise) so that the horizontal line inclined upward to the right extends in the horizontal direction, and the region R112 as the BM image is changed from the rotated region R112. R113 is cut out.
[0148]
　As described above, an image suitable for driving the vehicle 10 can be easily obtained as the BM image. That is, regardless of the tilt of the vehicle 10 in the roll direction, it is possible to easily obtain a BM image with the same amount of information as the BM image when the vehicle 10 is positioned on a flat road surface.
[0149]
　12A and 12B are diagrams for further explaining an example of readout control of the readout image according to the tilt information by the control unit 71. FIG.
　FIG. 12A is a diagram explaining that the readout size of the readout image is larger than the size of the BM image when the vehicle 10 is tilted in the roll direction. FIG. 12B is a diagram for explaining rotation in the processing section 72. As shown in FIG. FIG. 12C is a diagram for explaining how the processing unit 72 cuts out the BM image.
[0150]
　In FIG. 12A, the vehicle 10 is tilted to the right in the roll direction, so the horizontal line appearing in the readout image is tilted upward to the right.
[0151]
　In the second configuration example of the viewing system, the detection unit 42 detects that the vehicle 10 is tilted in the roll direction from the suspension information as the vehicle information supplied from the acquisition unit 41, and detects the tilt of the vehicle 10. Inclination information to represent, that is, inclination information that mainly represents (degree of) inclination of the vehicle 10 to the left and right sides as the roll direction is detected (calculated). The detection unit 42 then supplies the tilt information to the control unit 71 .
[0152]
　The control unit 71 calculates the readout start position and the readout size of the region R111 indicated by the readout position of the readout image read out from the image sensor 32 according to the tilt information from the detection unit 42 , and supplies them to the image sensor 32 . When the vehicle 10 is tilted in the roll direction (when the tilt in the roll direction is other than 0 degrees), the readout size is larger than the size of the region R101, which has the same size as the BM image.
[0153]
　Further, the control unit 71 calculates a rotation angle for rotating the region R111 as the readout image according to the tilt information from the detection unit 42, and supplies the rotation angle to the processing unit 72.
[0154]
　The image sensor 32 reads and outputs a region R111 represented by the readout position (readout start position and readout size) from the control unit 71 as a readout image.
[0155]
　A region R111 as a readout image output by the image sensor 32 is supplied to the processing section 72 .
[0156]
　In the processing unit 72, the region R111 is rotated by the rotation angle from the control unit 71, thereby generating a region R112 in which a horizontal line appears to extend in the horizontal direction (FIG. 12(B)). Furthermore, in the processing unit 72, a region R113 having the same size as the region R101 is cut out from the region R112 as a BM image (FIGS. 12(B) and 12(C)).
[0157]
　
[0158]
　FIG. 13 is a flowchart for explaining an example of display processing for displaying a BM image performed by the second configuration example of the viewing system in FIG.
[0159]
　In step S21, the acquisition unit 41 acquires, for example, suspension information as vehicle information, supplies it to the detection unit 42, and the process proceeds to step S22.
[0160]
　In step S22 , the detection unit 42 detects the tilt of the vehicle 10 in the roll direction using the vehicle information from the acquisition unit 41 . Further, the detection unit 42 uses the vehicle information from the acquisition unit 41 to obtain tilt information of the vehicle 10 when the vehicle 10 is tilted in the roll direction (mainly tilts to the left and right in the roll direction). tilt information) is detected (calculated) and supplied to the control unit 71, and the process proceeds from step S22 to step S23.
[0161]
　In step S23, the control unit 71 calculates the readout start position and readout size of the readout image from the image sensor 32 according to the tilt information from the detection unit 42, and uses the readout start position and readout size as the readout position. , to the image sensor 32 . Further, the control unit 71 calculates a rotation angle for rotating the readout image according to the tilt information from the detection unit 42, supplies it to the processing unit 72, and the process proceeds from step S23 to step S24.
[0162]
　In step S24, the image sensor 32 reads the pixel signal of the pixel at the readout position from the control unit 71, acquires and outputs a readout image having the pixel signal as the pixel value. The read image output by the image sensor 32 is supplied to the processing unit 72, and the process proceeds from step S24 to step S25.
[0163]
　In step S25 , the processing unit 72 rotates the image read from the image sensor 32 by the rotation angle from the control unit 71 . Further, the processing unit 72 cuts out the BM image from the readout image after rotation, supplies it to the output unit 33, and the process proceeds from step S25 to step S26.
[0164]
　In step S26, the output unit 33 transmits the BM image from the processing unit 72 to the BM display unit 21 for display. As a result, the BM image is displayed on the BM display unit 21, and the display processing ends.
[0165]
　As described above, the second configuration example of the viewing system mainly detects tilt information representing tilts to the left and right in the roll direction, and reads out an image from the image sensor 32 according to the tilt information. controls the reading of Furthermore, in the second configuration example of the viewing system, the rotation of the readout image is controlled according to the tilt information, and the BM image is cut out from the readout image after rotation. Therefore, according to the second configuration example of the viewing system, an image suitable for driving the vehicle 10 can be easily provided. That is, when the vehicle 10 is positioned on a road surface with steps on the left and right sides, it is possible to easily provide a BM image having the same amount of information as the BM image when the vehicle 10 is positioned on a flat road surface. can.
[0166]
　Furthermore, in the second configuration example of the viewing system, in the readout control, a readout image of the minimum size that allows the BM image to be cut out from the readout image after rotation is read out from the image sensor 32. Therefore, the BM image frame It is possible to suppress the risk of the rate dropping.
[0167]
　Suspension information has been described as the vehicle information in the second configuration example. However, as the vehicle information, gyro information, GPS information and 3D map, front camera image, etc. may be used as in the first configuration example described above and the third configuration example described later. Then, the inclination of the vehicle may be detected using these gyro information, GPS information, 3D map, front camera image, and the like.
[0168]
　
[0169]
　FIG. 14 is a block diagram showing a third configuration example of the viewing system mounted on the vehicle 10. As shown in FIG.
[0170]
　In the figure, the parts corresponding to those in FIG. 3 or 9 are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.
[0171]
　14, a camera unit 11, a BM display section 21, and an RV display section 22 are provided. Furthermore, in FIG. 14 , the camera unit 11 has an optical system 31 , an image sensor 32 , an output section 33 , an acquisition section 41 , a detection section 42 , a control section 71 and a processing section 72 .
[0172]
　Therefore, the third configuration example of the viewing system in FIG. 14 is configured similarly to the second configuration example of the viewing system in FIG.
[0173]
　However, in the third configuration example of the viewing system shown in FIG.
[0174]
　In this case, the detection unit 42 uses the suspension information from the acquisition unit 41 or the front camera image to detect the unevenness of the road surface. Further, the detection unit 42 detects the inclination of the vehicle 10 to the front and rear sides in the pitch direction and the inclination to the left and right sides in the roll direction due to the unevenness of the road surface, and detects (degree of) the inclination. It detects (calculates) the tilt information to represent. The detection unit 42 then supplies the tilt information to the control unit 71 .
[0175]
　Here, in the first configuration example of the viewing system, readout of the readout image from the image sensor 32 is controlled according to the inclination of the vehicle 10 toward the front and rear sides in the pitch direction. In the second configuration example of the viewing system, the reading of the readout image from the image sensor 32 and the rotation of the readout image in the processing unit 72 are performed according to the tilt of the vehicle 19 to the left and right in the roll direction. controlled.
[0176]
　On the other hand, in the third configuration example of the viewing system, the image sensor 32 outputs the , and the rotation of the read image in the processing unit 72 are controlled.
[0177]
　Therefore, in the third configuration example of the viewing system, the readout from the image sensor 32 performed in the first configuration example and the second configuration example of the viewing system is controlled as the readout control of the readout image from the image sensor 32. Control for image readout is performed in such a manner that each control is combined. Furthermore, in the third configuration example of the viewing system, as the control of the rotation of the readout image in the processing unit 72, the same control as the rotation control of the readout image performed in the second configuration example of the viewing system is performed. .
[0178]
　
[0179]
　15A and 15B are diagrams for explaining an example of readout control of a readout image according to the tilt information by the control unit 71. FIG.
　FIG. 15A is a diagram showing readout of a readout image having a size larger than the BM image from the image sensor 32. FIG. FIG. 15B is a diagram showing rotation and clipping of the readout image in the processing unit 72. As shown in FIG.
[0180]
　As described with reference to FIGS. 6 and 11, when the vehicle 10 is positioned on a flat road surface, the control unit 71 reads the read image from the image captured by the image sensor 32 (the light receiving surface of the image sensor 32). Readout control is performed so that (the pixel signals of pixels in) a rectangular region R101 having a size that matches the BM image as .
[0181]
　For example, when the vehicle 10 is located on an uneven road surface and the vehicle 10 tilts in the pitch direction or the roll direction, the control unit 71 controls the tilt angle of the vehicle 10 in the roll direction according to the tilt information. A readout position that designates an area R121 that is larger than the area R111 by the number of pixels and is vertically shifted from the area R111 by the number of pixels corresponding to the angle of inclination of the vehicle 10 in the pitch direction is calculated, and the readout position is calculated. The readout control of the image sensor 32 is performed so as to read out the pixel signal at the position.
[0182]
　Furthermore, the control unit 71 rotates the readout image region R121 according to the tilt information, for example, so that the horizontal line appearing in the readout image region R121 output by the image sensor 32 extends horizontally. A rotation angle is calculated and supplied to the processing unit 72 .
[0183]
　In the image sensor 32 , the rectangular area R121 is read out as a readout image according to the readout control from the control section 71 and supplied to the processing section 72 .
[0184]
　In the processing unit 72 , the region R121 as the readout image from the image sensor 32 is rotated by the rotation angle from the control unit 71 . Then, in the processing unit 72, a region R123 having a rotation angle of 0 degrees and having the same size as the BM image is cut out as a BM image from the region R122 after rotation of the region R121.
[0185]
　In FIG. 15, the vehicle 10 is tilted to the rear in the pitch direction and to the right in the roll direction. A region R121 is read out as a readout image. Furthermore, since the vehicle 10 is tilted to the right in the roll direction, the horizontal line reflected in the region R121 as the readout image is tilted upward to the right. A region R121 as a readout image is rotated clockwise so as to extend. Then, in the processing unit 72, a region R123 as a BM image is cut out from a region R122 after rotation of the region R121.
[0186]
　As described above, an image suitable for driving the vehicle 10 can be easily obtained as the BM image. That is, regardless of the tilt of the vehicle 10 in the pitch direction and roll direction, it is possible to easily obtain a BM image that is the same image representation as the BM image when the vehicle 10 is positioned on a flat road surface. As a result, it is possible to display an image in which the horizontal line is horizontal regardless of the tilt of the vehicle 10, so that a difference in the displayed image (difference in the position of the horizontal line) due to a temporary difference in tilt with respect to the vehicle behind the vehicle can be caused. None. The driver can grasp the situation behind the vehicle without worrying about the difference in the displayed image (difference in the position of the horizon line) while driving.
[0187]
　In addition, in the third configuration example of the viewing system, the detection unit 42 can detect the inclination of the vehicle 10 in the pitch direction and the roll direction from the suspension information, for example.
[0188]
　Further, the detection unit 42, for example, recognizes the unevenness of the road surface in front of the vehicle 10 from the front camera image, and, in accordance with the recognition result of the unevenness, detects when the vehicle 10 passes the recognized uneven road surface. can be detected by estimating the inclinations in the pitch and roll directions.
[0189]
　
[0190]
　FIG. 16 is a flowchart for explaining an example of display processing for displaying a BM image performed by the third configuration example of the viewing system in FIG.
[0191]
　In step S31, the acquisition unit 41 acquires, for example, suspension information or a front camera image as vehicle information, supplies it to the detection unit 42, and the process proceeds to step S32.
[0192]
　In step S32 , the detection unit 42 uses the vehicle information from the acquisition unit 41 to detect the unevenness of the road surface on which the vehicle 10 is located. Furthermore, the detection unit 42 uses the vehicle information from the acquisition unit 41 to obtain tilt information representing the tilt of the vehicle 10 due to the unevenness of the road surface (the tilt to the front and rear sides as the pitch direction, and the tilt to the left as the roll direction). and tilt information representing the tilt to the right) is detected (calculated) and supplied to the control unit 71, and the process proceeds from step S32 to step S33.
[0193]
　In step S33, the control unit 71 calculates the readout start position and readout size of the readout image from the image sensor 32 according to the tilt information from the detection unit 42, and uses the readout start position and readout size as the readout position. , to the image sensor 32 . Further, the control unit 71 calculates a rotation angle for rotating the read image according to the tilt information from the detection unit 42, supplies it to the processing unit 72, and the process proceeds from step S33 to step S34.
[0194]
　In step S34, the image sensor 32 reads the pixel signal of the pixel at the readout position from the control unit 71, acquires and outputs a readout image having the pixel signal as the pixel value. The read image output by the image sensor 32 is supplied to the processing unit 72, and the process proceeds from step S34 to step S35.
[0195]
　In step S35 , the processing unit 72 rotates the read image from the image sensor 32 by the rotation angle from the control unit 71 . Further, the processing unit 72 cuts out the BM image from the readout image after rotation, supplies it to the output unit 33, and the process proceeds from step S35 to step S36.
[0196]
　In step S36, the output unit 33 transmits the BM image from the processing unit 72 to the BM display unit 21 for display. As a result, the BM image is displayed on the BM display unit 21, and the display processing ends.
[0197]
　As described above, the third configuration example of the viewing system detects tilt information representing forward and rearward tilts in the pitch direction and leftward and rightward tilts in the roll direction. Readout of the readout image from the image sensor 32 is controlled according to the information. Furthermore, in the third configuration example of the viewing system, the rotation of the readout image is controlled according to the tilt information, and the BM image is cut out from the rotated readout image. Therefore, according to the third configuration example of the viewing system, an image suitable for driving the vehicle 10 can be easily provided. That is, when the vehicle 10 is positioned on an uneven road surface, it is possible to easily provide a BM image with the same amount of information as the BM image when the vehicle 10 is positioned on a flat road surface.
[0198]
　Furthermore, in the third configuration example of the viewing system, as in the second configuration example of the viewing system, in the readout control, a readout image of the minimum size that allows the BM image to be cut out from the readout image after rotation is selected. , the possibility that the frame rate of the BM image is lowered can be suppressed by reading from the image sensor 32 .
[0199]
　The suspension information and the front camera image have been described as the vehicle information in the third configuration example. However, as the vehicle information, gyro information, GPS information and 3D map, etc. may be used as in the above-described first configuration example and second configuration example. Then, the tilt of the vehicle may be detected using the gyro information, GPS information, 3D map, and the like.
[0200]
　In addition to the case where the vehicle 10 inclines due to the condition of the road surface on which the vehicle 10 is located, the present technology can also be applied to the case where the vehicle 10 inclines due to the load to be loaded on the vehicle 10, the state of the passengers of the vehicle 10, and the like. can do. That is, the present technology can be applied regardless of the cause of tilting of the vehicle 10 .
[0201]
　
[0202]
　FIG. 17 is a block diagram showing a fourth configuration example of the viewing system mounted on the vehicle 10. As shown in FIG.
[0203]
　The viewing system has the camera unit 11, BM display section 21, and RV display section 22 described in FIGS.
[0204]
　The camera unit 11 has an optical system 31 , an image sensor 32 , a data amount adjustment section 133 , an output section 134 , an acquisition section 135 and a control section 136 .
[0205]
　The optical system 31 is composed of optical components such as a condenser lens and a diaphragm, and condenses light incident on the optical system 31 onto the image sensor 32 .
[0206]
　The image sensor 32 receives light from the optical system 31 and performs photoelectric conversion to capture a captured image. Then, the image sensor 32 extracts and outputs the BM image and the RV image from the captured image under the control of the control unit 136 . The BM image and RV image output by the image sensor 32 are supplied to the data amount adjustment unit 133 .
[0207]
　The data amount adjustment unit 133 adjusts the data amount of the BM image and the RV image output by the image sensor 32 under the control of the control unit 136, and outputs the BM image and the RV image after the data amount is adjusted to the output unit 134. supply to
[0208]
　The output unit 134 is an output IF (Interface) that transmits the BM image and the RV image from the data amount adjustment unit 133 to the outside of the camera unit 11, transmits the BM image to the BM display unit 21, and transmits the RV image. is transmitted to the RV display unit 22 . The BM display unit 21 displays the BM image from the output unit 134 according to the specifications of the BM display unit 21, and the RV display unit 22 displays the RV image from the output unit 134 according to the specifications of the RV display unit 22. . The output unit 134 can perform format conversion and other image processing of the BM image and the RV image as necessary.
[0209]
　The acquisition unit 135 acquires vehicle information from the vehicle 10 and supplies it to the control unit 136 .
[0210]
　The vehicle information acquired by the acquisition unit 135 includes, for example, travel information, specifications of the BM display unit 21 and the RV display unit 22, the line of sight and head position of the driver of the vehicle 10, gyro information, and the like.
[0211]
　The travel information is information representing the travel state of the vehicle 10, and specifically represents the vehicle speed and the travel direction (forward or backward). For example, if the vehicle 10 is equipped with a speed sensor, the vehicle speed can be obtained from the output of the speed sensor. The direction of travel can be obtained, for example, from the state of the transmission.
[0212]
　The specifications of the BM display section 21 and the RV display section 22 are, for example, the resolutions of the BM display section 21 and the RV display section 22, and can be obtained from the BM display section 21 and the RV display section 22, respectively.
[0213]
　The line of sight and the position of the head of the driver of the vehicle 10 can be obtained from the image captured by the in-vehicle camera 23 .
[0214]
　The gyro information is information representing the attitude of the vehicle 10 (/angle of inclination of the vehicle). If the vehicle 10 is equipped with a gyro sensor, the gyro information can be obtained from the output of the gyro sensor. According to the gyro information, it can be recognized whether the vehicle 10 is positioned on a slope.
[0215]
　The control unit 136 controls the image sensor 32 and the data amount adjustment unit 133 according to the vehicle information supplied from the acquisition unit 135 .
[0216]
　That is, the control unit 136 performs extraction control for controlling the extraction of the BM image and the RV image from the image captured by the image sensor 32 by, for example, performing readout control similar to that of the control unit 43 according to the vehicle information. conduct. An example of the reading range of the BM image and the RV image is the range shown in FIG. Further, the control unit 136 performs adjustment control for controlling the adjustment of the data amount of the BM image and the RV image in the data amount adjustment unit 133 according to the vehicle information.
[0217]
　Therefore, the image sensor 32 extracts the BM image and the RV image from the captured image according to the vehicle information, and the data amount adjustment unit 133 adjusts the data amount of the BM image and the RV image according to the vehicle information. , can be said.
[0218]
　
[0219]
　18A and 18B are diagrams illustrating examples of images that can be output from the image sensor 32. FIG.
[0220]
　Now, let us say that the captured image with the highest resolution that the image sensor 32 can output is the highest resolution image. The image sensor 32, for example, has a performance capable of outputting a maximum resolution image with a resolution (the number of pixels) Rmax at (a frame rate of) 60 fps (frame per second) or higher.
[0221]
　Here, it is assumed that the resolution RBM of the highest resolution (number of pixels) BM image extracted from the highest resolution image is 1/2 or less of the resolution Rmax of the highest resolution image. It is also assumed that the resolution RRV of the highest resolution RV image extracted from the highest resolution image is equal to or less than the resolution RBM of the BM image.
[0222]
　In the present embodiment, for example, it is assumed that the sum RBM+RRV of the resolution RBM of the BM image and the resolution RRV of the RV image is less than or equal to half the resolution Rmax of the highest resolution image. In this case, if an image sensor 32 capable of outputting a maximum resolution image of 60 fps (or more) at a resolution Rmax is used, a BM image of resolution RBM obtained by partial readout of a part of the maximum resolution image and an RV image of resolution RRV can be obtained. Both images can be output at 120fps.
[0223]
　
[0224]
　FIG. 19 is a diagram illustrating an example of vehicle transmission bands that can be used for data transmission in vehicle 10. In FIG.
[0225]
　That is, FIG. 19 shows examples of BM images and RV images that can be output by the camera unit 11 without adjustment of the data amount by the data amount adjustment section 133 .
[0226]
　The camera unit 11 can output, as the BM image, for example, a color image with a resolution of RBM, a YUV format of 4:2:2, and a bit number per pixel of 8 bits (for each of luminance and color difference). . In addition, the camera unit 11 can output, as the RV image, for example, a color image with a resolution of RRV, a YUV format of 4:2:2, and a bit number per pixel of 8 bits.
[0227]
　A BM image with resolution RBM, YUV 4:2:2 format, and 8 bits per pixel is called the highest quality BM image. An RV image in which the number of bits per pixel is 8 bits is called an RV image with the highest image quality.
[0228]
　In the present embodiment, the vehicle transmission band, which is the band in which data is transmitted from the camera unit 11, is a transmission band capable of transmitting (in real time) only two screens of BM images with the highest image quality of 60 fps, for example. It is assumed that In the present embodiment, since RBM>=RRV, according to the vehicle transmission band that can transmit only two screens of 60 fps highest quality BM images, for example, two screens of 60 fps highest quality RV images can be transmitted. minutes can be transmitted. Further, according to the vehicle transmission band, for example, it is possible to transmit one screen of 60 fps highest quality BM image and one screen of 60 fps highest quality RV image, for a total of two screens.
[0229]
　As described with reference to FIGS. 18 and 19, the camera unit 11 can output both the highest quality BM image and RV image at a maximum of 120 fps.
[0230]
　However, in this embodiment, the vehicle transmission band is 60 fps, and only two screens of BM images (or RV images) with the highest image quality can be transmitted.
[0231]
　By increasing the vehicle transmission band, it is possible to transmit both the highest quality BM and RV images at 120 fps, which is the output that the camera unit 11 can output. to cost.
[0232]
　In this technology, in order to prevent the cost of the viewing system from increasing, the camera unit 11 appropriately adjusts the data amount of the BM image and the RV image, and transmits the BM image and the RV image within the vehicle transmission band. do.
[0233]
　
[0234]
　FIG. 20 is a diagram illustrating a first example of adjustment control of the data amount of the BM image and the RV image by the control unit 136. In FIG.
[0235]
　FIG. 20A shows adjustment control when the vehicle 10 is moving forward or when the vehicle is moving backward at a speed equal to or higher than the first speed threshold.
[0236]
　In this case, the control unit 136 performs adjustment control as the control of the data amount adjustment unit 133 so that the BM image of resolution RBM and 120 fps is output and the output of the RV image is restricted. The data amount adjustment unit 133 adjusts the amount of data of the BM image and the RV image from the image sensor 32 according to the adjustment control of the control unit 136, and outputs the BM image of 120 fps with the resolution RBM and outputs the RV image. Restrict.
[0237]
　Therefore, in this case, the camera unit 11 outputs a BM image with a resolution of RBM and 120 fps, and does not output an RV image. As a result, the BM display unit 21 displays the BM image at a resolution of RBM and 120 fps, and the RV display unit 22 does not display the RV image.
[0238]
　From the above, when the vehicle 10 is moving forward or when the vehicle is moving backward at high speed, the driver uses a BM image with a resolution of RBM and 120 fps, that is, a BM image with a high resolution and a high frame rate. , the rear of the vehicle 10 can be confirmed from immediately behind.
[0239]
　Note that when the vehicle 10 is moving forward or when the vehicle is reversing at high speed, the RV image including the image immediately after the rear portion of the vehicle 10 is not displayed.
[0240]
　Also, a 120 fps BM image with a resolution RBM output from the camera unit 11 can be transmitted using a vehicle transmission band capable of transmitting only two screens of a 60 fps maximum image quality (resolution RBM) BM image.
[0241]
　Here, in the data amount adjustment unit 133, the data amount adjustment method for adjusting the data amount of the image includes the method of limiting the output of the image (not outputting the image) as described above, as well as, for example, the resolution (pixel number), a method of reducing the gradation (the number of bits per pixel), a method of reducing the frame rate, a method of compressing with a predetermined compression encoding method, and the like.
[0242]
　Among the data amount adjustment methods, limiting the image output, reducing the resolution, reducing the gradation, and reducing the frame rate are performed by the data amount adjustment unit 133. In addition, the control unit 136 The image sensor 32 can be made to perform by performing control such as extraction control of the image sensor 32 .
[0243]
　That is, the control unit 136 controls the reading of data from the image sensor 32 so as to limit the extraction of the RV image from the captured image, and the pixel 51 (FIG. 7), by not reading out the pixel signals to be the RV image. The control unit 136 controls the extraction of the image sensor 32 so as to limit the extraction of the RV image from the captured image according to the vehicle information when the vehicle 10 is moving forward or moving backward at high speed. It can be performed. Of course, the data amount adjustment unit 133 may limit the output of the RV image without the limitation by the image sensor 32 .
[0244]
　Reducing the resolution of the image is performed by the control unit 136, for example, by thinning out the pixels 51 from which pixel signals are read, or by so-called SF (Source Follower) addition, FD (Floating Diffusion) addition, or the like. can be performed by controlling the image sensor 32 so as to perform binning for adding pixel signals of .
[0245]
　Reducing the gradation can be performed by controlling the image sensor 32 in the control unit 136, for example, so as to reduce the AD conversion bit number of the AD converter 55 (FIG. 7).
[0246]
　Reducing the frame rate is performed by controlling the image sensor 32 in the control unit 136 so as to reduce, for example, the rate at which pixel signals are read from the pixels 51 and the rate at which the AD converter 55 performs AD conversion. be able to.
[0247]
　B of FIG. 20 shows adjustment control when the vehicle 10 is moving backward at a medium speed that is equal to or more than the second threshold, which is less than the first threshold, and less than the first threshold.
[0248]
　In this case, the control unit 136 controls the data amount adjustment unit 133 so that a 120 fps BM image is output with a resolution RBMM lower than the resolution RBM, and a 30 fps RV image is output with a resolution RRVM lower than the resolution RRV. adjustment control as the control of The data amount adjustment unit 133 adjusts the data amount of the BM image and the RV image from the image sensor 32 according to the adjustment control of the control unit 136, and outputs the BM image of 120 fps with the resolution RBMM and 30 fps with the resolution RRVM. output RV images.
[0249]
　Therefore, in this case, the camera unit 11 outputs a BM image with a resolution of RBMM and 120 fps and an RV image with a resolution of RRVM and 30 fps. As a result, the BM display unit 21 displays a BM image with a resolution of RBMM and a resolution of 120 fps, and the RV display unit 22 displays an RV image with a resolution of RRVM and a resolution of 30 fps.
[0250]
　From the above, when the vehicle 10 is moving backward at a medium speed, the driver uses a BM image with a resolution of RBMM and 120 fps, that is, a BM image with a medium resolution and a high frame rate. You can check the rear from immediately after. In addition, the driver can see directly behind the rear of the vehicle 10 with a 30 fps RV image at resolution RRVM, ie, a medium resolution, low frame rate RV image.
[0251]
　Note that the transmission bandwidth required to transmit 30fps RV images with resolution RRVM is the maximum transmission rate (here, the transmission rate required when transmitting BM images with resolution RBM at 120fps without compression) and resolution RBMM is equal to or less than the transmission band of the difference in the transmission rate (first transmission rate) required when transmitting at 120 fps. In this case, for both the BM image of 120 fps with the resolution RBMM and the RV image of 30 fps with the resolution RRVM output by the camera unit 11, only two screens of the highest quality BM image of 60 fps can be transmitted. It can be transmitted in the available vehicle transmission band.
[0252]
　Here, when the transmission band necessary for transmitting the RV image of 30 fps with the resolution RRVM is not equal to or less than the transmission band of the difference between the maximum transmission rate and the first transmission rate, the data amount adjustment unit 133 selects the BM image and the RV image. One or both of the images are set so that the transmission bandwidth required to transmit RV images at 30 fps with resolution RRVM is equal to or less than the transmission bandwidth of the difference between the maximum transmission rate and the first transmission rate (RV images with resolution RRVM can be compressed (encoded) so that it can be transmitted). For example, the amount of data can be reduced by compressing (encoding) part or all of a 120 fps BM image with a resolution of RBMM. The amount of data can be reduced by compressing and encoding RV images of resolution RRVM and 30 fps as they are in color, or by converting them into black-and-white images.
[0253]
　FIG. 20C shows adjustment control when the vehicle 10 is moving backward at a low speed that is less than the second threshold that is less than the first threshold.
[0254]
　In this case, the control unit 136 performs adjustment control as control of the data amount adjustment unit 133 so that the BM image of 60 fps is output with the resolution RBM and the RV image of 60 fps is output with the resolution RRV. . The data amount adjustment unit 133 adjusts the data amount of the BM image and the RV image from the image sensor 32 according to the adjustment control of the control unit 136, and outputs the BM image of 60 fps with the resolution RBM and 60 fps with the resolution RRV. output RV images.
[0255]
　Therefore, in this case, the camera unit 11 outputs a BM image with a resolution of RBM and 60 fps and an RV image with a resolution of RRV and 60 fps. As a result, the BM display unit 21 displays a BM image with a resolution of RBM and 60 fps, and the RV display unit 22 displays an RV image with a resolution of RRV and 60 fps.
[0256]
　From the above, when the vehicle 10 is moving backward at a low speed, the driver can use a BM image with resolution RBM and 60 fps, that is, a BM image with a high resolution and a medium frame rate. You can check further behind. In addition, the driver can see directly behind the rear of the vehicle 10 with a 60 fps RV image at resolution RRV, ie, a high resolution, medium frame rate RV image.
[0257]
　A case where the vehicle 10 is moving backward at a low speed is, for example, a case where the vehicle 10 is about to be parked, and it is important for the driver to check immediately behind the rear portion of the vehicle 10 which is a blind spot. Therefore, when the vehicle 10 is reversing at a low speed, the RV image is displayed with a higher resolution and a higher frame rate than when the vehicle 10 is reversing at a high speed or medium speed. This makes it easier to confirm the blind spot area and to control the vehicle according to the situation of the blind spot.
[0258]
　In addition, for both the 60 fps BM image with a resolution RBM output by the camera unit 11 and the 60 fps RV image with a resolution RRV, only two screens of the BM image with the highest image quality of 60 fps can be transmitted. It can be transmitted by the vehicle transmission band.
[0259]
　
[0260]
　FIG. 21 is a diagram illustrating a second example of adjustment control of the data amount of the BM image and the RV image by the control unit 136. In FIG.
[0261]
　Here, in the second example of the adjustment control of the data amount of the BM image and the RV image, the vehicle transmission band is, for example, 60 fps (or more) and the maximum image quality (resolution RBM) BM image is transmitted for one screen. It is assumed that the transmission band is such that
[0262]
　FIG. 21A shows adjustment control when the vehicle 10 is moving forward, or when the vehicle is moving backward at a speed equal to or higher than the first speed threshold.
[0263]
　In this case, the control unit 136 performs adjustment control as control of the data amount adjustment unit 133 so that a BM image with resolution RBM and 60 fps (or higher) is output and the output of the RV image is restricted. The data amount adjustment unit 133 adjusts the data amount of the BM image and the RV image from the image sensor 32 according to the adjustment control of the control unit 136, and outputs the BM image of 60 fps with the resolution RBM and outputs the RV image. Restrict.
[0264]
　Therefore, in this case, the camera unit 11 outputs a BM image with a resolution of RBM and 60 fps, and does not output an RV image. As a result, the BM display unit 21 displays a BM image with a resolution of RBM and 60 fps, and the RV display unit 22 does not display the RV image.
[0265]
　From the above, when the vehicle 10 is moving forward or when the vehicle is moving backward at high speed, the driver uses a BM image of resolution RBM and 60 fps, that is, a BM image of high resolution and medium frame rate. , the rear of the vehicle 10 can be confirmed from immediately behind.
[0266]
　Note that when the vehicle 10 is moving forward or when the vehicle is moving backward at high speed, the RV image including the image immediately after the rear portion of the vehicle 10 is not displayed as described with reference to A of FIG. 20 .
[0267]
　In addition, the BM image of 60 fps at the resolution RBM output by the camera unit 11 can be transmitted using a vehicle transmission band capable of transmitting only one screen of the BM image with the highest image quality (resolution RBM) of 60 fps.
[0268]
　B of FIG. 21 shows adjustment control when the vehicle 10 is moving backward at a medium speed that is equal to or more than the second threshold, which is less than the first threshold, and less than the first threshold.
[0269]
　In this case, the control unit 136 controls the data amount adjustment unit 133 so that a 60 fps BM image is output with a resolution RBMM lower than the resolution RBM, and a 30 fps RV image is output with a resolution RRVM lower than the resolution RRV. adjustment control as the control of The data amount adjustment unit 133 adjusts the amount of data of the BM image and the RV image from the image sensor 32 according to the adjustment control of the control unit 136, and outputs the BM image of 60 fps with the resolution RBMM and 30 fps with the resolution RRVM. output RV images.
[0270]
　Therefore, in this case, the camera unit 11 outputs a BM image with a resolution of RBMM and 60 fps and an RV image with a resolution of RRVM and 30 fps. As a result, the BM display unit 21 displays a BM image with a resolution of RBMM and 60 fps, and the RV display unit 22 displays an RV image with a resolution of RRVM and 30 fps.
[0271]
　From the above, when the vehicle 10 is moving backward at a medium speed, the driver uses a BM image with a resolution of RBMM and 60 fps, that is, a BM image with a medium resolution and a medium frame rate. You can check the rear from immediately after. In addition, the driver can see directly behind the rear of the vehicle 10 with a 30 fps RV image at resolution RRVM, ie, a medium resolution, low frame rate RV image.
[0272]
　In addition, the transmission band (hereinafter also referred to as the required transmission band) required to transmit both the BM image of 60 fps with the resolution RBMM and the RV image of 30 fps with the resolution RRVM output by the camera unit 11 is 60 fps. If the maximum image quality of the BM image is not within the vehicle transmission band that can be transmitted for only one screen, the BM image is compressed to the first compression for medium speed so that the necessary transmission band is within the vehicle transmission band. or compressing the RV image as a color image or converting it to a black-and-white image and compressing it with a second compression ratio for medium speed that is higher than the first compression ratio for medium speed. be able to.
[0273]
　FIG. 21C shows adjustment control when the vehicle 10 is moving backward at a low speed that is less than the second threshold that is less than the first threshold.
[0274]
　In this case, the control unit 136 controls the data amount adjustment unit 133 so that a 60 fps BM image is output with a resolution RBML lower than the resolution RBMM, and a 30 fps RV image is output with a resolution RRVM. Perform adjustment control. The data amount adjustment unit 133 adjusts the data amount of the BM image and the RV image from the image sensor 32 according to the adjustment control of the control unit 136, and outputs the BM image of 60 fps with the resolution RBML and 30 fps with the resolution RRVM. output RV images.
[0275]
　Therefore, in this case, the camera unit 11 outputs a BM image with a resolution of RBML and 60 fps and an RV image with a resolution of RRV and 30 fps. As a result, the BM display unit 21 displays a BM image with a resolution of RBML and 60 fps, and the RV display unit 22 displays an RV image with a resolution of RRVM and 30 fps.
[0276]
　From the above, when the vehicle 10 is moving backward at a low speed, the driver can use a BM image with a resolution of RBML and 60 fps, that is, a BM image with a low resolution and a medium frame rate. You can check further behind. In addition, the driver can see directly behind the rear of the vehicle 10 with a 30 fps RV image at resolution RRVM, ie, a medium resolution, low frame rate RV image.
[0277]
　In addition, the required transmission band required for transmitting both the 60 fps BM image with the resolution RBML output by the camera unit 11 and the 30 fps RV image with the resolution RRVM is 1 for the highest quality BM image of 60 fps. When the vehicle transmission band that can be transmitted for the screen cannot be accommodated, the BM image can be compressed at the first compression rate for low speed, or the RV image can be compressed so that the required transmission band can be accommodated within the vehicle transmission band. can be compressed (as a color image) with a second compression ratio for low speed that is higher than the first compression ratio for low speed.
[0278]
　Here, as the first compression rate for low speed, a higher compression rate than the first compression rate for medium speed can be adopted. The same compression ratio can be adopted as the second compression ratio for medium speed and the second compression ratio for low speed. In this case, the relationship between the first compression ratio and second compression ratio for low speed and the first compression ratio and second compression ratio for medium speed is: second compression ratio for low speed=for medium speed second compression ratio>first compression ratio for low speed>first compression for medium speed. However, here, it is assumed that the higher the value of the compression ratio, the higher the compression. The relationship between the amount of data after compression at the first compression ratio and second compression ratio for low speed, and the first compression ratio and second compression ratio for medium speed, respectively, is the second compression ratio for low speed. Amount of data after compression by ratio=Amount of data after compression by second compression ratio for medium speed
[0283]
　22 is a flowchart illustrating an example of display processing for displaying BM and RV images performed by the viewing system of FIG. 17. FIG.
[0284]
　In step S111, the image sensor 32 captures a captured image, and the process proceeds to step S112.
[0285]
　In step S112, the acquisition unit 135 acquires vehicle information from the vehicle 10, supplies it to the control unit 136, and the process proceeds to step S113.
[0286]
　In step S113 , the control unit 136 performs extraction control of the image sensor 32 according to the vehicle information from the acquisition unit 135 . The image sensor 32 extracts the BM image and the RV image from the captured image according to extraction control of the control unit 136 . Then, the image sensor 32 supplies the BM image and the RV image to the data amount adjustment unit 133, and the process proceeds from step S113 to step S114.
[0287]
　In step S114 , the control unit 136 performs adjustment control of the data amount adjustment unit 133 according to the vehicle information from the acquisition unit 135 . The data amount adjustment unit 133 adjusts the data amounts of the BM image and the RV image from the image sensor 32 according to the adjustment control of the control unit 136 . Then, the data amount adjustment unit 133 supplies the BM image and the RV image whose data amount has been adjusted to the output unit 134, and the process proceeds from step S114 to step S115.
[0288]
　In step S115, the output unit 134 outputs the BM image and the RV image with the data amount adjusted from the data amount adjustment unit 133 to the outside of the camera unit 11, and transmits the BM image to the BM display unit 21. , RV images are transmitted to the RV display unit 22, and the process proceeds to step S116.
[0289]
　In step S116, the BM display unit 21 displays the BM image from the output unit 134 according to the specifications of the BM display unit 21, and the RV display unit 22 displays the RV image from the output unit 134 on the RV display unit 22. Display according to specifications.
[0290]
　
[0291]
　FIG. 23 is a block diagram showing a fifth configuration example of the viewing system mounted on the vehicle 10. As shown in FIG.
[0292]
　In the figure, parts corresponding to those in FIG. 17 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.
[0293]
　23, the viewing system has a camera unit 11, a BM display section 21, an RV display section 22, and an extraction section 182. The camera unit 11 includes an optical system 31, an image sensor 32, a data amount adjustment section 133 , an output unit 134 , an acquisition unit 135 , and a control unit 181 .
[0294]
　Therefore, the viewing system of FIG. 23 has the camera unit 11, the BM display section 21, and the RV display section 22, and the camera unit 11 has the optical system 31 through the acquisition section 135. common with
[0295]
　However, the viewing system of FIG. 23 is different from that of FIG. different from the case.
[0296]
　Note that the extractor 182 is provided outside the camera unit 11 in FIG. 23 , but can be provided inside the camera unit 11 .
[0297]
　Vehicle information is supplied to the control unit 181 from the acquisition unit 135 . Vehicle information includes, for example, travel information, specifications of the BM display section 21 and the RV display section 22, gyro information, and the like. However, in this example, the vehicle information supplied from the acquisition unit 135 does not include the line of sight of the driver and the position of the head. Note that the line of sight and the position of the head of the driver of the vehicle 10 are input to the extraction unit 182 as part of the vehicle information.
[0298]
　Similar to the control unit 136 , the control unit 181 performs extraction control of the image sensor 32 and adjustment control of the data amount adjustment unit 133 according to the vehicle information from the acquisition unit 135 .
[0299]
　However, in the extraction control, the control unit 181 controls (the position of) the region R11 to be extracted as the BM image according to one or both of the line of sight and the position of the head of the driver, instead of controlling the region R11. A region with a large size is extracted as a BM image.
[0300]
　Therefore, the size of the BM image output by the output unit 134 in FIG. 23 is larger than the size of the BM image output by the output unit 134 in FIGS. 17 and 12 .
[0301]
　In FIG. 23 , the BM image larger than the area R11 output by the output unit 134 is supplied to the extraction unit 182 .
[0302]
　The extraction unit 182 is supplied with the BM image having a size larger than the region R11 from the output unit 134, as well as the line of sight and the head position of the driver among the vehicle information.
[0303]
　The extraction unit 182 extracts a partial area of ​​the BM image larger than the area R11 from the output unit 134, that is, the same size as the area R11, depending on one or both of the line of sight and the position of the head of the driver. is extracted as the final BM image to be displayed on the BM display section 21 and supplied to the BM display section 21 .
[0304]
　
[0305]
　24A and 24B are diagrams for explaining an example of extraction control of the BM image and the RV image from the captured image by the control unit 181. FIG.
[0306]
　24 shows the image circle of the optical system 31 and the photographed image (the light receiving surface of the image sensor 32) photographed by the image sensor 32, as in FIG.
[0307]
　Similar to the control unit 136, the control unit 181 controls reading of data from the image sensor 32 so as to extract the region R12 as an RV image in the extraction control.
[0308]
　Further, in the extraction control, the control unit 181 reads data from the image sensor 32 so as to extract not the region R11 but the region R31 having a size larger than the size of the region R11 from the captured image as the BM image. Control.
[0309]
　If the vehicle 10 were equipped with an interior rearview mirror, the region R31 includes the maximum range that the driver can see through the interior rearview mirror by moving the driver's line of sight or head. This is the area where The region R11 is a variable region whose position is changed according to the line of sight and the position of the driver's head, while the region R31 is a fixed region.
[0310]
　In the extraction unit 182, from the region R31 as described above, a region of a position corresponding to one or both of the line of sight and the position of the head of the driver and having the same size as the region R11 is selected from the BM display unit. 21 is extracted as the final BM image to be displayed. That is, the extraction unit 182 extracts, from the region R31, a region R11 that the driver would observe through the interior rearview mirror if the vehicle 10 were equipped with the interior rearview mirror, as a BM image.
[0311]
　
[0312]
　25 is a flowchart illustrating an example of display processing for displaying BM and RV images performed by the viewing system of FIG. 23. FIG.
[0313]
　In step S121, the image sensor 32 captures a captured image, and the process proceeds to step S122.
[0314]
　In step S122, the acquisition unit 135 acquires vehicle information from the vehicle 10, supplies it to the control unit 136, and the process proceeds to step S123.
[0315]
　In step S123 , the control unit 136 performs extraction control of the image sensor 32 . The image sensor 32 extracts the regions R31 and R12 from the captured image as the BM image and the RV image, respectively, according to the extraction control of the control unit 136, as described with reference to FIG. Then, the image sensor 32 supplies the BM image and the RV image to the data amount adjustment unit 133, and the process proceeds from step S123 to step S124.
[0316]
　In step S124 , the control unit 136 performs adjustment control of the data amount adjustment unit 133 according to the vehicle information from the acquisition unit 135 . The data amount adjustment unit 133 adjusts the data amounts of the BM image and the RV image from the image sensor 32 according to the adjustment control of the control unit 136 . Then, the data amount adjustment unit 133 supplies the BM image and the RV image whose data amount has been adjusted to the output unit 134, and the process proceeds from step S124 to step S125.
[0317]
　In step S125, the output unit 134 outputs the BM image and the RV image with the data amount adjusted from the data amount adjustment unit 133 to the outside of the camera unit 11, and the process proceeds to step S126. Thereby, in FIG. 23, the BM image is supplied to the extraction unit 182 and the RV image is transmitted to the RV display unit 22 .
[0318]
　In step S126, the extraction unit 182 acquires the line of sight and the head position of the driver included in the vehicle information from the vehicle 10, and the process proceeds to step S127.
[0319]
　In step S127, the extracting unit 182 causes the BM display unit 21 to display, from the BM image from the output unit 134, an area having the same size as the area R11 and located at a position corresponding to the line of sight and the position of the head of the driver. Extract as a typical BM image. The extraction unit 182 then transmits the final BM image to the BM display unit 21, and the process proceeds from step S127 to step S128.
[0320]
　In step S128, the BM display unit 21 displays the BM image from the extraction unit 182 according to the specifications of the BM display unit 21, and the RV display unit 22 displays the RV image from the output unit 134 on the RV display unit 22. Display according to specifications.
[0321]
　
[0322]
　Next, the series of processes described above can be performed by either hardware or software. When a series of processing is performed by software, a program that constitutes the software is installed in a general-purpose computer or the like.
[0323]
　FIG. 26 is a block diagram showing a configuration example of an embodiment of a computer in which a program for executing the series of processes described above is installed.
[0324]
　The program can be recorded in advance in a hard disk 905 or ROM 903 as a recording medium built into the computer.
[0325]
　Alternatively, the program can be stored (recorded) in removable recording medium 911 driven by drive 909 . Such a removable recording medium 911 can be provided as so-called package software. Here, the removable recording medium 911 includes, for example, a flexible disk, CD-ROM (Compact Disc Read Only Memory), MO (Magneto Optical) disk, DVD (Digital Versatile Disc), magnetic disk, semiconductor memory, and the like.
[0326]
　The program can be installed in the computer from the removable recording medium 911 as described above, or can be downloaded to the computer via a communication network or broadcasting network and installed in the hard disk 905 incorporated therein. That is, the program is, for example, transferred from the download site to the computer wirelessly via an artificial satellite for digital satellite broadcasting, or wired to the computer via a network such as a LAN (Local Area Network) or the Internet. be able to.
[0327]
　The computer incorporates a CPU (Central Processing Unit) 902 , and an input/output interface 910 is connected to the CPU 902 via a bus 901 .
[0328]
　The CPU 902 executes a program stored in a ROM (Read Only Memory) 903 according to a command input by the user through the input/output interface 910 by operating the input unit 907 or the like. . Alternatively, the CPU 902 loads a program stored in the hard disk 905 into a RAM (Random Access Memory) 904 and executes it.
[0329]
　As a result, the CPU 902 performs the processing according to the above-described flowchart or the processing performed by the configuration of the above-described block diagram. Then, the CPU 902 outputs the processing result from the output unit 906 via the input/output interface 910, transmits it from the communication unit 908, or records it in the hard disk 905 as necessary.
[0330]
　The input unit 907 is composed of a keyboard, mouse, microphone, and the like. Also, the output unit 906 is composed of an LCD (Liquid Crystal Display), a speaker, and the like.
[0331]
　Here, in this specification, the processing performed by the computer according to the program does not necessarily have to be performed in chronological order according to the order described as the flowchart. In other words, processing performed by a computer according to a program includes processing that is executed in parallel or individually (for example, parallel processing or processing by objects).
[0332]
　Also, the program may be processed by one computer (processor), or may be processed by a plurality of computers in a distributed manner. Furthermore, the program may be transferred to a remote computer and executed.
[0333]
　Furthermore, in this specification, a system means a set of a plurality of components (devices, modules (parts), etc.), regardless of whether or not all components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a single device housing a plurality of modules in one housing are both systems. .
[0334]
　The embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.
[0335]
　For example, the present technology can take a configuration of cloud computing in which one function is shared by a plurality of devices via a network and processed jointly.
[0336]
　Further, each step described in the flowchart above can be executed by one device, or can be shared by a plurality of devices and executed.
[0337]
　Furthermore, when one step includes a plurality of processes, the plurality of processes included in the one step can be executed by one device or shared by a plurality of devices.
[0338]
　Moreover, the effects described in this specification are merely examples and are not limited, and other effects may be provided.
[0339]
　In addition, this technique can take the following configurations.
[0340]
　<1> A photographing device comprising
　an image sensor for photographing an image displayed on a display unit of a
　vehicle, and a control unit for controlling readout of the image from the image sensor based on vehicle information acquired by the vehicle
　.
　<2> Described in 　<1>,
　further comprising a detection unit that detects tilt information of the vehicle using the vehicle information, and
　wherein the control unit controls reading of an image from the image sensor according to the tilt information.
camera equipment.
　<3> 　The photographing device
　according to <2>, wherein the detection unit detects the tilt information indicating the tilt of the vehicle toward the front side and the rear side . 　<4> 　The photographing device 　according to <2>, wherein the detection unit detects the tilt information representing tilts of the vehicle to the left and right . 　<5> 　The photographing device 　according to <2>, wherein the detection unit detects the tilt information representing tilts of the vehicle to the front and rear sides and tilts of the vehicle to the left and right sides . 　<6> 　The photographing apparatus according to any one of <1> to <5> 　, wherein the control unit controls a reading position from which an image is read in the image sensor .

　16. The program according to claim 15, 　wherein the vehicle information includes at least one of vehicle speed information and gear information .