FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENTS RULES, 2003
COMPLETE SPECIFICATION
(See section 10, rule 13)
“INFORMATION PROCESSING APPARATUS, MOBILE OBJECT,
CONTROL SYSTEM, INFORMATION PROCESSING METHOD, AND
PROGRAM”
SONY CORPORATION, of 1-7-1, Konan, Minato-ku,
Tokyo 108-0075, Japan
The following specification particularly describes the invention and the manner in which it is to
be performed.
2
Description
Title of Invention: INFORMATION PROCESSING APPARATUS,
MOBILE OBJECT, CONTROL SYSTEM, INFORMATION PROCESSING
METHOD, AND PROGRAM
5
Technical Field
[0001] The present technology relates to an
information processing apparatus, a mobile object,
control system, an information processing method, and a
10 program, and in particular to an information processing
apparatus, a mobile object, a system, an information
processing method, and a program that are optimal for
obtaining a proper image upon estimating a selflocation
of the mobile object.
15 Background Art
[0002] In the past, a method for estimating a selflocation
of an autonomously behaving mobile object has
been proposed, the method including, for example,
extracting a plurality of characteristic points (also
20 referred to as landmarks) from an image of the
surroundings of the mobile object that is captured by a
camera; and estimating three-dimensional positions of
the plurality of characteristic points (for example,
refer to Patent Literature 1).
25 Citation List
Patent Literature
3
[0003] Patent Literature 1: Japanese Patent
Application Laid-open No. 2005-315746
Disclosure of Invention
Technical Problem
[0004] When a self-location is 5 estimated using an
image, sight of a landmark may be lost or the landmark
may be falsely recognized due to various disturbance
factors existing in the surrounding environment, and
this may result in there being a decrease in the
10 accuracy in estimating a self-location.
[0005] In view of the circumstances described above,
it is an object of the present technology to provide an
information processing apparatus, a mobile object, a
control system, an information processing method, and a
15 program that make it possible to reduce an impact of a
disturbance factor.
Solution to Problem
[0006] In order to achieve the object described
above, an information processing apparatus according to
20 the present technology includes an information
acquisition unit and a control prediction unit.
The information acquisition unit acquires
information regarding a location and a posture of a
first mobile object that includes a sensing device.
25 The control prediction unit predicts control
performed with respect to the sensing device, on the
4
basis of the information regarding the location and the
posture and map information, the information regarding
the location and the posture being acquired by the
information acquisition unit.
[0007] According to such a configuration, 5 on the
basis of information regarding a location and a posture
of the first mobile object at a time T and map
information, control that is performed with respect to
the sensing device and is suitable for the mobile
10 object at T+N, that is, N seconds after the time T, is
predicted. Since control with respect to the sensing
device is performed in the mobile object at T+N, on the
basis of information regarding the control prediction,
it is possible to perform control without a time lag.
15 [0008] The sensing device may include an imagecapturing
device. The information processing apparatus
may further include a unit for predicting a location
and a posture of a mobile object that predicts the
location and the posture of the first mobile object on
20 the basis of the information regarding the location and
the posture, the information regarding the location and
the posture being acquired by the information
acquisition unit; and a unit for predicting a position
of a disturbance factor that predicts a position of a
25 disturbance factor in an image captured by the imagecapturing
device, on the basis of the map information
5
and a result of the prediction performed by the unit
for predicting a location and a posture of a mobile
object. On the basis of a result of the prediction
performed by the unit for predicting a position of a
disturbance factor, the control prediction 5 unit may
predict control performed with respect to the imagecapturing
device.
[0009] According to such a configuration, on the
basis of the information regarding a location and a
10 posture of the first mobile object at a time T and the
map information, a position of a disturbance factor in
an image predicted to be captured by the imagecapturing
device at T+N, that is, N seconds after the
time T, is predicted. Then, control performed with
15 respect to the image-capturing device such that an
image in which an impact due to the disturbance factor
is reduced is obtained, is predicted on the basis of a
result of the prediction of the position of the
disturbance factor.
20 [0010] In the first mobile object, a control
condition for the image-capturing device capturing an
image of the mobile object at T+N, is set on the basis
of information regarding the control prediction, and
this results in being able to obtain, at T+N and
25 without a time lag, an image that is suitable for a
state of the mobile object at T+N and in which an
6
impact due to a disturbance factor is reduced.
[0011] The control prediction unit may predict an
exposure control performed with respect to the imagecapturing
device.
[0012] The control prediction 5 unit may predict a
photometry-region control performed with respect to the
image-capturing device.
[0013] The sensing device may include an imagecapturing
device. The first mobile object may include a
10 self-location estimation system that estimates the
location and the posture of the first mobile object
using a characteristic point that is extracted from
image information from the image-capturing device. The
information processing apparatus may further include a
15 unit for predicting a location and a posture of a
mobile object that predicts the location and the
posture of the first mobile object on the basis of the
information regarding the location and the posture, the
information regarding the location and the posture
20 being acquired by the information acquisition unit; and
a unit for predicting a position of a disturbance
factor that predicts a position of a disturbance factor
in an image captured by the image-capturing device, on
the basis of the map information and a result of the
25 prediction performed by the unit for predicting a
location and a posture of a mobile object. On the basis
7
of a result of the prediction performed by the unit for
predicting a position of a disturbance factor, the
control prediction unit may predict control performed
with respect to the self-location estimation system.
[0014] According to such a configuration, 5 on the
basis of the information regarding a location and a
posture of the first mobile object at a time T and the
map information, a position of a disturbance factor in
an image predicted to be captured by the image10
capturing device at T+N, that is, N seconds after the
time T, is predicted. Then, control performed with
respect to the self-location estimation system is
predicted on the basis of a result of the prediction of
the position of the disturbance factor, such that an
15 impact due to the disturbance factor is reduced.
[0015] In the first mobile object, control performed
at T+N with respect to the self-location estimation
system is set on the basis of information regarding the
control prediction. This makes it possible to perform
20 processing of estimating a self-location and a posture
at T+N, using, for example, information regarding an
image that is suitable for a state of the mobile object
at T+N and in which an impact due to a disturbance
factor is reduced. This results in an improvement in
25 the estimation accuracy.
[0016] On the basis of the result of the prediction
8
performed by the unit for predicting a position of a
disturbance factor, the control prediction unit may
predict a region in which extraction of the
characteristic point in the image is not performed in
the self-location 5 estimation system.
[0017] The sensing device may include the imagecapturing
device and a mobile-object-state detection
sensor that detects a state of the first mobile object.
The first mobile object may include the self-location
10 estimation system that estimates the location and the
posture of the first mobile object using at least one
of the image information, or mobile-object-state
information from the mobile-object-state detection
sensor. On the basis of the result of the prediction
15 performed by the unit for predicting a position of a
disturbance factor, the control prediction unit may
predict how the image information and the mobileobject-
state information are respectively weighted, the
image information and the mobile-object-state
20 information being used when the location and the
posture of the first mobile object are estimated in the
self-location estimation system.
[0018] The sensing device may include a plurality of
the image-capturing devices. On the basis of the result
25 of the prediction performed by the unit for predicting
a position of a disturbance factor, the control
9
prediction unit may predict how respective pieces of
image information from the plurality of the imagecapturing
devices are weighted, the respective pieces
of image information being used when the location and
the posture of the first mobile object 5 are estimated in
the self-location estimation system.
[0019] On the basis of the result of the prediction
performed by the unit for predicting a position of a
disturbance factor, the control prediction unit may
10 predict control performed with respect to the imagecapturing
device.
[0020] The disturbance factor may be the sun, and,
on the basis of the map information, the result of the
prediction performed by the unit for predicting a
15 location and a posture of a mobile object, and sun
position information, the unit for predicting a
position of a disturbance factor may predict a position
of the sun in the image captured by the image-capturing
device.
20 [0021] The disturbance factor may be a tunnel, and,
on the basis of the map information and the result of
the prediction performed by the unit for predicting a
location and a posture of a mobile object, the unit for
predicting a position of a disturbance factor may
25 predict a position of the tunnel in the image captured
by the image-capturing device.
10
[0022] The disturbance factor may be a shadow of a
structure, and, on the basis of the map information,
the result of the prediction performed by the unit for
predicting a location and a posture of a mobile object,
and sun position information, the unit 5 for predicting a
position of a disturbance factor may predict a position
of the shadow created due to the structure in the image
captured by the image-capturing device.
[0023] The information acquisition unit may acquire
10 information regarding a location and a posture of a
second mobile object that includes a sensing device and
is different from the first mobile object. The control
prediction unit may set, to be prediction of control
performed with respect to the sensing device of the
15 second mobile object, the prediction of the control
performed with respect to the sensing device of the
first mobile object, the prediction of the control
performed with respect to the sensing device of the
first mobile object being performed by the control
20 prediction unit on the basis of the information
regarding the location and the posture of the first
mobile object and the map information, the information
regarding the location and the posture of the first
mobile object being acquired by the information
25 acquisition unit.
[0024] In order to achieve the object described
11
above, a mobile object according to the present
technology includes a sensing device and an acquisition
unit.
The acquisition unit acquires information
regarding a self-location and a posture 5 of the mobile
object.
The sensing device is controlled according to
control prediction information regarding prediction of
control performed with respect to the sensing device,
10 the control performed with respect to the sensing
device being predicted on the basis of the information
regarding the self-location and the posture and map
information, the information regarding the selflocation
and the posture being acquired by the
15 acquisition unit.
[0025] In order to achieve the object described
above, a control system according to the present
technology includes a mobile object, an information
acquisition unit, a control prediction unit, and a
20 control unit.
The mobile object includes a sensing device.
The information acquisition unit acquires
information regarding a location and a posture of the
mobile object.
25 The control prediction unit predicts control
performed with respect to the sensing device, on the
12
basis of the information regarding the location and the
posture and map information, the information regarding
the location and the posture being acquired by the
information acquisition unit.
The control unit performs the control 5 with respect
to the sensing device on the basis of control
prediction information regarding the control prediction
performed by the control prediction unit.
[0026] In order to achieve the object described
10 above, an information processing method according to
the present technology includes acquiring information
regarding a location and a posture of a mobile object
that includes a sensing device; and predicting control
performed with respect to the sensing device, on the
15 basis of the information regarding the location and the
posture and map information.
[0027] In order to achieve the object described
above, a program according to the present technology
causes an information processing apparatus to perform a
20 process including acquiring information regarding a
location and a posture of a mobile object that includes
a sensing device; and predicting control performed with
respect to the sensing device, on the basis of the
information regarding the location and the posture and
25 map information.
Advantageous Effects of Invention
13
[0028] As described above, according to the present
technology, it is possible to provide an information
processing apparatus, a mobile object, a system, an
information processing method, and a program that make
it possible to reduce an impact of a disturbance 5 factor.
Note that the effect described here is not
necessarily limitative and any of the effects described
in the present disclosure may be provided.
Brief Description of Drawings
10 [0029]
[Fig. 1] Fig. 1 schematically illustrates a sensingdevice
control system according to a first embodiment
of the present technology.
[Fig. 2] Fig. 2 is a block diagram of an example of a
15 schematic functional configuration of a vehicle control
system in the sensing-device control system.
[Fig. 3] Fig. 3 is a block diagram of an example of a
schematic functional configuration of a self-location
estimation unit in the self-location estimation system.
20 [Fig. 4] Fig. 4 is a block diagram of an example of a
schematic functional configuration of a camera included
in a vehicle.
[Fig. 5] Fig. 5 is a block diagram of an example of a
schematic functional configuration of a server
25 apparatus in the sensing-device control system.
[Fig. 6] Fig. 6 is a flowchart describing processing
14
of generating a control signal in the image processing
apparatus.
[Fig. 7] Fig. 7 is a block diagram of a functional
configuration of a server apparatus in a sensing-device
control system according to a second 5 embodiment of the
present technology.
[Fig. 8] Fig. 8 is a flowchart describing processing
of generating a control signal in an image processing
apparatus according to the second embodiment.
10 [Fig. 9] Fig. 9 is a block diagram of a functional
configuration of an image processing apparatus in a
sensing-device control system according to a third
embodiment of the present technology.
[Fig. 10] Fig. 10 is a flowchart describing processing
15 of generating a control signal in the image processing
apparatus according to the third embodiment.
Mode(s) for Carrying Out the Invention
[0030] Embodiments for carrying out the present
technology will now be described below.
20 According to the present technology, on the basis
of information regarding a self-location and a posture
of a mobile object at a time T and map information, a
location of the mobile object at T+N (N>0), that is, N
seconds after the time T is predicted, and control
25 performed at T+N with respect to a sensing device
included in the mobile object, is predicted.
15
[0031] In first to third embodiments described below,
the descriptions are made taking a vehicle such as an
automobile as an example of a first mobile object, and
taking a camera (an image-capturing device) and a
vehicle-state detection sensor 5 as an example of a
sensing device.
[0032] Note that, here, the descriptions are made
using an automobile as a mobile object, but is not
limitative. Examples of the mobile object include a
10 bicycle, a motorcycle, an unmanned aerial vehicle (UAV)
such as a drone, and various robots.
[0033] The vehicle includes a camera, a vehiclestate
detection sensor, and a self-location estimation
system that is used to assist a vehicle in traveling
15 with automated driving.
[0034] In the first and third embodiments described
below, on the basis of information regarding a selflocation
and a posture of a vehicle at a time T and map
information, a location of the vehicle at T+N, that is,
20 N seconds after the time T is predicted.
[0035] Next, on the basis of information regarding
the mobile-object-location prediction, the map
information, and information regarding a date and
time/weather, a position of a disturbance factor in an
25 image predicted to be captured at T+N is predicted.
[0036] Next, control performed with respect to the
16
camera and control performed with respect to the selflocation
estimation system are predicted on the basis
of a result of the prediction of the position of the
disturbance factor. The descriptions are made taking
the sun as an example of the disturbance 5 factor in the
first embodiment, and taking a shadow of a structure
such as a building as an example of the disturbance
factor in the third embodiment.
[0037] Further, in the second embodiment, on the
10 basis of information regarding a self-location of a
mobile object at a time T and map information, a
location and a posture of the vehicle at T+N, that is,
N seconds after the time T are predicted.
[0038] Next, on the basis of information regarding
15 the mobile-object-location prediction and the map
information, a position of a tunnel, a disturbance
factor, in an image predicted to be captured at T+N is
predicted.
[0039] Next, control performed with respect to the
20 camera and control performed with respect to the selflocation
estimation system are predicted on the basis
of a result of the prediction of the position of the
disturbance factor.
[0040] The respective embodiments are described
25 below in detail.
[0041]
17
[Example of Configuration of Control System]
Fig. 1 is a block diagram of an example of a
schematic functional configuration of a control system
1 for a sensing device to which the present technology
is applicable. Here, only a primary 5 configuration is
described, and a detailed configuration will be
described later using Fig. 2 and the figures subsequent
to Fig. 2.
[0042] The control system 1 includes a vehicle
10 control system 100 that is installed in a vehicle 10
that is a mobile object, and a server apparatus 500
that serves as an information processing apparatus. The
vehicle control system 100 and the server apparatus 500
are capable of communicating with each other, for
15 example, through a wireless communication network.
[0043] Note that an example of providing the server
apparatus 500 serving as an information processing
apparatus outside of the vehicle 10 is described in the
present embodiment, but the server apparatus 500 may be
20 installed in the vehicle 10.
[0044] The vehicle control system 100 includes a
self-location estimation system 200 and a data
acquisition unit 102.
[0045] The data acquisition unit 102 includes a
25 camera 300 and a vehicle-state detection sensor 400
that serve as a sensing device. The camera 300 is used
18
to capture an image of the outside of the vehicle 10,
and a plurality of cameras 300 is installed in the
vehicle 10.
[0046] In the self-location estimation system 200, a
self-location and a posture of the vehicle 5 10 that are
necessary to perform automated driving are estimated.
In the self-location estimation system 200, the selflocation
and the posture of the vehicle 10 are
estimated using information from the sensing device,
10 such as image information regarding an image captured
by the camera 300, and vehicle-state detection
information from the vehicle-state detection sensor 400.
[0047] In the self-location estimation system 200,
when the self-location and the posture of the vehicle
15 10 are estimated on the basis of image information from
the camera 300, the location is estimated on the basis
of a result of tracking a characteristic point (a
landmark) in the image.
[0048] The server apparatus 500 includes an
20 information acquisition unit 502, a unit 503 for
predicting a location and a posture of a vehicle at T+N,
a unit 504 for predicting a position of a disturbance
factor in an image, and a control prediction unit 505.
[0049] In the data acquisition unit 102 of the
25 vehicle 10, information regarding the vehicle 10 at a
time T is acquired using the camera 300 and the
19
vehicle-state detection sensor 400. The information
regarding the vehicle 10 at the time T is transmitted
to the server apparatus 500.
[0050] The information acquisition unit 502 of the
server apparatus 500 acquires the information 5 regarding
the vehicle 10 at the time T that is transmitted from
the vehicle control system 100.
[0051] On the basis of the information regarding the
vehicle 10 at the time T that is acquired by the
10 information acquisition unit 502, the unit 503 for
predicting a location and a posture of a vehicle at T+N,
which is a unit for predicting a self-location and a
posture of a mobile object, predicts a self-location
and a posture of the vehicle 10 at T+N, that is, N
15 seconds after the time T.
[0052] In the present embodiment, the unit 504 for
predicting a position of a disturbance factor in an
image predicts a position of the sun, a disturbance
factor, in an image.
20 The unit 504 for predicting a position of a
disturbance factor (the sun) in an image (hereinafter
referred to as a unit for predicting a position of the
sun in an image) predicts a position of the sun in an
image captured at T+N, on the basis of information
25 regarding the predicted self-location and posture of
the vehicle 10 at T+N, the map information, and
20
information regarding a date and time/weather.
[0053] The control prediction unit 505 predicts
control performed with respect to the camera 300 at T+N
and control performed with respect to the self-location
estimation system 200 at T+N, on the 5 basis of a result
of the prediction of the position of the sun in the
image captured at T+N. Information regarding the
control prediction is transmitted to the vehicle 10. In
the vehicle 10, control with respect to the camera 300
10 and control with respect to the self-location
estimation system 200 are performed at T+N on the basis
of the information on the control prediction.
[0054] Examples of the control performed with
respect to the camera 300 include an exposure control
15 and a photometry-region control.
[0055] Examples of the control performed with
respect to the self-location estimation system 200
include control performed with respect to the sensing
device upon estimating a self-location and a posture of
20 the vehicle 10.
[0056] Specifically, the control prediction unit 505
predicts how image information from the camera 300 and
detection information from the vehicle-state detection
sensor 400 are respectively weighted in the self25
location estimation system 200 to be used for
estimating a location and a posture of the vehicle.
21
[0057] Further, when a self-location and a posture
are estimated in the self-location estimation system
200 on the basis of image information regarding an
image captured by the camera 300, the control
prediction unit 505 predicts how a mask 5 region is set,
the mask region being a characteristic-point extraction
region not used for a characteristic point extraction
performed to estimate a self-location and a posture.
[0058] Furthermore, when a self-location is
10 estimated in the self-location estimation system 200 on
the basis of image information regarding an image
captured by the camera 300, the control prediction unit
505 predicts how respective pieces of image information
from a plurality of cameras 300 are weighted to be used
15 upon estimating a self-location and a posture.
[0059] As described above, an operation regarding a
sensing device of the vehicle 10 is controlled on the
basis of control prediction information transmitted
from the server apparatus 500, the control prediction
20 information being information regarding prediction of
control performed with respect to a sensing device such
as the camera 300 and the vehicle-state detection
sensor 400 that are installed in the vehicle 10.
[0060] This results in obtaining a highly robust
25 control system in which an impact of a disturbance
factor that is the sun is reduced.
22
Each structural element is described below in
detail.
[0061] [Example of Configuration of Vehicle Control
System]
Fig. 2 is a block diagram 5 of an example of a
schematic functional configuration of the vehicle
control system 100 that is an example of a mobile
object control system to which the present technology
is applicable.
10 [0062] Note that, when a vehicle provided with the
vehicle control system 100 is to be distinguished from
other vehicles, the vehicle provided with the vehicle
control system 100 will be hereinafter referred to as
an own automobile or an own vehicle.
15 [0063] The vehicle control system 100 includes an
input unit 101, a data acquisition unit 102, a
communication unit 103, in-vehicle equipment 104, an
output control unit 105, an output unit 106, a
drivetrain control unit 107, a drivetrain system 108, a
20 body-related control unit 109, a body-related system
110, a storage unit 111, and an automated driving
control unit 112. The input unit 101, the data
acquisition unit 102, the communication unit 103, the
output control unit 105, the drivetrain control unit
25 107, the body-related control unit 109, the storage
unit 111, and the automated driving control unit 112
23
are connected to each other through a communication
network 121. For example, the communication network 121
includes a bus or a vehicle-mounted communication
network compliant with any standard such as a
controller area network (CAN), a 5 local interconnect
network (LIN), a local area network (LAN), or FlexRay
(registered trademark). Note that the respective
structural elements of the vehicle control system 100
may be directly connected to each other without using
10 the communication network 121.
[0064] Note that the description of the
communication network 121 will be omitted below when
the respective structural elements of the vehicle
control system 100 communicate with each other through
15 the communication network 121. For example, when the
input unit 101 and the automated driving control unit
112 communicate with each other through the
communication network 121, it will be simply stated
that the input unit 101 and the automated driving
20 control unit 112 communicate with each other.
[0065] The input unit 101 includes an apparatus used
by a person on board to input various pieces of data,
instructions, or the like. For example, the input unit
101 includes an operation device such as a touch panel,
25 a button, a microphone, a switch, or a lever; an
operation device with which input can be performed by a
24
method other than a manual operation, such as sound or
a gesture; or the like. Alternatively, for example, the
input unit 101 may be externally connected equipment
such as a remote-control apparatus using infrared or
another radio wave, or mobile equipment 5 or wearable
equipment compatible with operation of the vehicle
control system 100. The input unit 101 generates an
input signal on the basis of data, an instruction, or
the like input by a person on board, and supplies the
10 generated input signal to the respective structural
elements of the vehicle control system 100.
[0066] The data acquisition unit 102 includes
various sensors or the like for acquiring data used for
a process performed by the vehicle control system 100,
15 and supplies the acquired data to the respective
structural elements of the vehicle control system 100.
[0067] For example, the data acquisition unit 102
includes various sensors for detecting a state and the
like of the own automobile. Specifically, for example,
20 the data acquisition unit 102 includes a gyro sensor;
an acceleration sensor; an inertial measurement unit
(IMU); and a sensor or the like for detecting an amount
of operation of an accelerator pedal, an amount of
operation of a brake pedal, a steering angle of a
25 steering wheel, the number of revolutions of an engine,
the number of revolutions of a motor, a speed of wheel
25
rotation, or the like.
[0068] Further, for example, the data acquisition
unit 102 includes various sensors for detecting
information regarding the outside of the own automobile.
Specifically, for example, the data 5 acquisition unit
102 includes an image-capturing device such as a timeof-
flight (ToF) camera, a stereo camera, a monocular
camera, an infrared camera, and other cameras.
Furthermore, for example, the data acquisition unit 102
10 includes an environment sensor for detecting weather, a
meteorological phenomenon, or the like, and a
surrounding-information detection sensor for detecting
an object around the own automobile. For example, the
environment sensor includes a raindrop sensor, a fog
15 sensor, a sunshine sensor, a snow sensor, or the like.
The surrounding-information detection sensor includes
an ultrasonic sensor, a radar, a LiDAR (light detection
and ranging, laser imaging detection and ranging)
sensor, a sonar, or the like.
20 [0069] Moreover, for example, the data acquisition
unit 102 includes various sensors for detecting a
current location of the own automobile. Specifically,
for example, the data acquisition unit 102 includes a
global navigation satellite system (GNSS) receiver or
25 the like. The GNSS receiver receives a GNSS signal from
a GNSS satellite.
26
[0070] Further, for example, the data acquisition
unit 102 includes various sensors for detecting
information regarding the inside of a vehicle.
Specifically, for example, the data acquisition unit
102 includes an image-capturing device 5 that captures an
image of a driver, a biological sensor that detects
biological information of the driver, a microphone that
collects sound in the interior of a vehicle, and the
like. For example, the biological sensor is provided to
10 a seat surface, the steering wheel, or the like, and
detects biological information of a person on board
sitting on a seat, or a driver holding the steering
wheel.
[0071] The communication unit 103 communicates with
15 the in-vehicle equipment 104 as well as various pieces
of vehicle-exterior equipment, a server, a base station,
and the like, transmits data supplied by the respective
structural elements of the vehicle control system 100,
and supplies the received data to the respective
20 structural elements of the vehicle control system 100.
Note that a communication protocol supported by the
communication unit 103 is not particularly limited. It
is also possible for the communication unit 103 to
support a plurality of types of communication protocols.
25 [0072] For example, the communication unit 103
wirelessly communicates with the in-vehicle equipment
27
104 using a wireless LAN, Bluetooth (registered
trademark), near-field communication (NFC), a wireless
USB (WUSB), or the like. Further, for example, the
communication unit 103 communicates with the in-vehicle
equipment 104 by wire using a universal 5 serial bus
(USB), a high-definition multimedia interface (HDMI)
(registered trademark), a mobile high-definition link
(MHL), or the like through a connection terminal (not
illustrated) (and a cable if necessary).
10 [0073] Further, for example, the communication unit
103 communicates with equipment (for example, an
application server or a control server) situated in an
external network (for example, the Internet, a cloud
network, or a carrier-specific network) through a base
15 station or an access point. Furthermore, for example,
the communication unit 103 communicates with a terminal
(for example, a terminal of a pedestrian or a store, or
a machine-type communication (MTC) terminal) situated
near the own automobile, using a peer-to-peer (P2P)
20 technology. Moreover, for example, the communication
unit 103 performs V2X communication such as vehicle-tovehicle
communication, vehicle-to-infrastructure
communication, vehicle-to-home communication between
the own automobile and a home, and vehicle-to25
pedestrian communication. Further, for example, the
communication unit 103 includes a beacon receiver,
28
receives a radio wave or an electromagnetic wave
transmitted from, for example, a radio station
installed on a road, and acquires information regarding,
for example, the current location, traffic congestion,
traffic regulation, 5 or necessary time.
[0074] Examples of the in-vehicle equipment 104
include mobile equipment or wearable equipment of a
person on board, information equipment that is brought
in or attached to the own automobile, and a navigation
10 apparatus that searches for a route to any destination.
[0075] The output control unit 105 controls output
of various pieces of information to a person on board
of the own automobile or to the outside of the own
automobile. For example, the output control unit 105
15 generates an output signal that includes at least one
of visual information (such as image data) or audio
information (such as sound data), supplies the output
signal to the output unit 106, and thereby controls
output of the visual information and the audio
20 information from the output unit 106. Specifically, for
example, the output control unit 105 combines pieces of
data of images captured by different image-capturing
devices of the data acquisition unit 102, generates a
bird's-eye image, a panoramic image, or the like, and
25 supplies an output signal including the generated image
to the output unit 106. Further, for example, the
29
output control unit 105 generates sound data including,
for example, a warning beep or a warning message
alerting a danger such as collision, contact, or
entrance into a dangerous zone, and supplies an output
signal including the generated sound 5 data to the output
unit 106.
[0076] The output unit 106 includes an apparatus
capable of outputting the visual information or the
audio information to a person on board of the own
10 automobile or to the outside of own automobile. For
example, the output unit 106 includes a display
apparatus, an instrument panel, an audio speaker,
headphones, a wearable device such as an eyeglass-type
display used to be worn on the person on board, a
15 projector, a lamp, or the like. Instead of an apparatus
including a commonly used display, the display
apparatus included in the output unit 106 may be an
apparatus, such as a head-up display, a transparent
display, or an apparatus including an augmented reality
20 (AR) display function, that displays the visual
information in the field of view of a driver.
[0077] The drivetrain control unit 107 generates
various control signals, supplies them to the
drivetrain system 108, and thereby controls the
25 drivetrain system 108. Further, the drivetrain control
unit 107 supplies the control signals to the structural
30
elements other than the drivetrain system 108 as
necessary to, for example, notify them of a state of
controlling the drivetrain system 108.
[0078] The drivetrain system 108 includes various
apparatuses related to the drivetrain 5 of the own
automobile. For example, the drivetrain system 108
includes a driving force generation apparatus, such as
an internal-combustion engine and a driving motor, that
generates driving force, a driving force transmitting
10 mechanism for transmitting the driving force to wheels,
a steering mechanism that adjusts the steering angle, a
braking apparatus that generates braking force, an
antilock braking system (ABS), an electronic stability
control (ESC) system, an electric power steering
15 apparatus, and the like.
[0079] The body-related control unit 109 generates
various control signals, supplies them to the bodyrelated
system 110, and thereby controls the bodyrelated
system 110. Further, the body-related control
20 unit 109 supplies the control signals to the structural
elements other than the body-related system 110 as
necessary to, for example, notify them of a state of
controlling the body-related system 110.
[0080] The body-related system 110 includes various
25 body-related apparatuses provided to a vehicle body.
For example, the body-related system 110 includes a
31
keyless entry system, a smart key system, a power
window apparatus, a power seat, a steering wheel, an
air conditioner, various lamps (such as headlamps, tail
lamps, brake lamps, direction-indicator lamps, and fog
lamps), 5 and the like.
[0081] For example, the storage unit 111 includes a
read only memory (ROM), a random access memory (RAM), a
magnetic storage device such as a hard disc drive (HDD),
a semiconductor storage device, an optical storage
10 device, a magneto-optical storage device, and the like.
The storage unit 111 stores therein various programs,
data, and the like that are used by the respective
structural elements of the vehicle control system 100.
For example, the storage unit 111 stores therein map
15 data such as a three-dimensional high-accuracy map, a
global map, and a local map. The high-accuracy map is a
dynamic map or the like. The global map is less
accurate and covers a wider area than the high-accuracy
map. The local map includes information regarding the
20 surroundings of the own automobile.
[0082] The automated driving control unit 112
performs control related to automated driving such as
autonomous traveling or a driving assistance.
Specifically, for example, the automated driving
25 control unit 112 performs a cooperative control
intended to implement a function of an advanced driver32
assistance system (ADAS) including collision avoidance
or shock mitigation for the own automobile, traveling
after a leading vehicle based on a distance between
vehicles, traveling while maintaining a vehicle speed,
a warning of collision of the own automobile, 5 a warning
of deviation of the own automobile from a lane, or the
like. Further, for example, the automated driving
control unit 112 performs a cooperative control
intended to achieve, for example, automated driving
10 that is autonomous traveling without an operation
performed by a driver. The automated driving control
unit 112 includes a detection unit 131, a self-location
estimation unit 132, a state analysis unit 133, a
planning unit 134, and a movement control unit 135.
15 [0083] The detection unit 131 detects various pieces
of information necessary to control automated driving.
The detection unit 131 includes a vehicle-exteriorinformation
detection unit 141, a vehicle-interiorinformation
detection unit 142, and a vehicle-state
20 detection unit 143.
[0084] The vehicle-exterior-information detection
unit 141 performs a process of detecting information
regarding the outside of the own automobile on the
basis of data or a signal from each structural element
25 of the vehicle control system 100. For example, the
vehicle-exterior-information detection unit 141
33
performs processes of detecting, recognizing, and
tracking an object around the own automobile, and a
process of detecting a distance to the object. Examples
of the detection-target object include a vehicle, a
person, an obstacle, a structure, 5 a road, a traffic
light, a traffic sign, and a road sign. Further, for
example, the vehicle-exterior-information detection
unit 141 performs a process of detecting an environment
surrounding the own automobile. Examples of the
10 detection-target surrounding environment includes
weather, temperature, humidity, brightness, and a road
surface condition. The vehicle-exterior-information
detection unit 141 supplies data indicating a result of
the detection process to, for example, the self15
location estimation unit 132; a map analysis unit 151,
a traffic-rule recognition unit 152, and a state
recognition unit 153 of the state analysis unit 133;
and an emergency event avoiding unit 171 of the
movement control unit 135.
20 [0085] The vehicle-interior-information detection
unit 142 performs a process of detecting information
regarding the inside of a vehicle on the basis of data
or a signal from each structural element of the vehicle
control system 100. For example, the vehicle-interior25
information detection unit 142 performs processes of
authenticating and recognizing a driver, a process of
34
detecting a state of the driver, a process of detecting
a person on board, and a process of detecting a vehicle
interior environment. Examples of the detection-target
state of a driver include a physical condition, a
degree of arousal, a degree of concentration, 5 a degree
of fatigue, and a direction of a line of sight.
Examples of the detection-target vehicle interior
environment include temperature, humidity, brightness,
and odor. The vehicle-interior-information detection
10 unit 142 supplies data indicating a result of the
detection process to, for example, the state
recognition unit 153 of the state analysis unit 133 and
the emergency event avoiding unit 171 of the movement
control unit 135.
15 [0086] The vehicle-state detection unit 143 performs
a process of detecting a state of the own automobile on
the basis of data or a signal from each structural
element of the vehicle control system 100. Examples of
the detection-target state of the own automobile
20 include speed, acceleration, a steering angle, the
presence or absence of anomaly and its details, a
driving operation state, a position and an inclination
of a power seat, a state of a door lock, and states of
other pieces of vehicle-mounted equipment. The vehicle25
state detection unit 143 supplies data indicating a
result of the detection process to, for example, the
35
state recognition unit 153 of the state analysis unit
133 and the emergency event avoiding unit 171 of the
movement control unit 135.
[0087] The self-location estimation unit 132
performs a process of estimating a location, 5 a posture,
and the like of the own automobile on the basis of data
or signals from the respective structural elements of
the vehicle control system 100, such as the vehicleexterior-
information detection unit 141, and the state
10 recognition unit 153 of the state analysis unit 133.
Further, the self-location estimation unit 132
generates, as necessary, a local map (hereinafter
referred to as a self-location estimation map) used to
estimate a self-location. For example, the self15
location estimation map is a high-accuracy map using a
technology such as simultaneous localization and
mapping (SLAM). The self-location estimation unit 132
supplies data indicating a result of the estimation
process to, for example, the map analysis unit 151, the
20 traffic-rule recognition unit 152, and the state
recognition unit 153 of the state analysis unit 133.
Further, the self-location estimator 132 stores the
self-location estimation map in the storage unit 111.
[0088] The state analysis unit 133 performs a
25 process of analyzing states of the own automobile and
its surroundings. The state analysis unit 133 includes
36
the map analysis unit 151, the traffic-rule recognition
unit 152, the state recognition unit 153, and the state
prediction unit 154.
[0089] Using, as necessary, data or signals from the
respective structural elements of the 5 vehicle control
system 100, such as the self-location estimation unit
132 and the vehicle-exterior-information detection unit
141, the map analysis unit 151 performs a process of
analyzing various maps stored in the storage unit 111,
10 and constructs a map including information necessary
for an automated driving process. The map analysis unit
151 supplies the constructed map to, for example, the
traffic-rule recognition unit 152, the state
recognition unit 153, and the state prediction unit 154,
15 as well as a route planning unit 161, a behavior
planning unit 162, and a movement planning unit 163 of
the planning unit 134.
[0090] The traffic-rule recognition unit 152
performs a process of recognizing traffic rules around
20 the own automobile on the basis of data or signals from
the respective structural elements of the vehicle
control system 100, such as the self-location
estimation unit 132, the vehicle-exterior-information
detection unit 141 , and the map analysis unit 151. The
25 recognition process makes it possible to recognize a
location and a state of a traffic light around the own
37
automobile, the details of traffic control performed
around the own automobile, and a travelable lane. The
traffic-rule recognition unit 152 supplies data
indicating a result of the recognition process to, for
example, the state prediction 5 unit 154.
[0091] The state recognition unit 153 performs a
process of recognizing a state related to the own
automobile on the basis of data or signals from the
respective structural elements of the vehicle control
10 system 100, such as the self-location estimation unit
132, the vehicle-exterior-information detection unit
141, the vehicle-interior-information detection unit
142, the vehicle-state detection unit 143, and the map
analysis unit 151. For example, the state recognition
15 unit 153 performs a process of recognizing a state of
the own automobile, a state of the surroundings of the
own automobile, a state of a driver of the own
automobile, and the like. Further, the state
recognition unit 153 generates, as necessary, a local
20 map (hereinafter referred to as a state recognition
map) used to recognize the state of the surroundings of
the own automobile. The state recognition map is, for
example, an occupancy grid map.
[0092] Examples of the recognition-target state of
25 the own automobile include a location, a posture, and
movement (such as speed, acceleration, and a movement
38
direction) of the own automobile, as well as the
presence or absence of anomaly and its details.
Examples of the recognition-target state of the
surroundings of the own automobile include the type and
a location of a stationary object 5 around the own
automobile; the type, a location, and movement (such as
speed, acceleration, and a movement direction) of a
moving object around the own automobile; a structure of
a road around the own automobile and a condition of the
10 surface of the road; and weather, temperature, humidity,
and brightness around the own automobile. Examples of
the recognition-target state of a driver include a
physical condition, a degree of arousal, a degree of
concentration, a degree of fatigue, movement of a line
15 of sight, and a driving operation.
[0093] The state recognition unit 153 supplies data
indicating a result of the recognition process
(including a state recognition map as necessary) to,
for example, the self-location estimation unit 132 and
20 the state prediction unit 154. Further, the state
recognition section 153 stores the state-recognition
map in the storage unit 111.
[0094] The state prediction unit 154 performs a
process of predicting a state related to the own
25 automobile on the basis of data or signals from the
respective structural elements of the vehicle control
39
system 100, such as the map analysis unit 151, the
traffic-rule recognition unit 152, and the state
recognition unit 153. For example, the state prediction
unit 154 performs a process of predicting a state of
the own automobile, a state of the surroundings 5 of the
own automobile, a state of a driver, and the like.
[0095] Examples of the prediction-target state of
the own automobile include the behavior of the own
automobile, the occurrence of anomaly in the own
10 automobile, and a travelable distance of the own
automobile. Examples of the prediction-target state of
the surroundings of the own automobile include the
behavior of a moving object, a change in a state of a
traffic light, and a change in environment such as
15 weather around the own automobile. Examples of the
prediction-target state of a driver include the
behavior and the physical condition of the driver.
[0096] The state prediction unit 154 supplies data
indicating a result of the prediction process to, for
20 example, the route planning unit 161, the behavior
planning unit 162, and the movement planning unit 163
of the planning unit 134 together with the data from
the traffic-rule recognition unit 152 and the state
recognition unit 153.
25 [0097] The route planning unit 161 plans a route to
a destination on the basis of data or signals from the
40
respective structural elements of the vehicle control
system 100, such as the map analysis unit 151 and the
state prediction unit 154. For example, the route
planning unit 161 sets a route from a current location
to a specified destination on the basis 5 of a global map.
Further, for example, the route planning unit 161
changes a route as appropriate on the basis of the
states of, for example, traffic congestion, an accident,
traffic regulation, and a construction, as well as the
10 physical condition of a driver. The route planning unit
161 supplies data indicating the planned route to, for
example, the behavior planning unit 162.
[0098] On the basis of data or signals from the
respective structural elements of the vehicle control
15 system 100, such as the map analysis unit 151 and the
state prediction unit 154, the behavior planning unit
162 plans the behavior of the own automobile in order
to travel safely on the route planned by the route
planning unit 161 within a time planned by the route
20 planning unit 161. For example, the behavior planning
unit 162 makes plans about, for example, a start to
move, a stop, a travel direction (such as a forward
movement, a backward movement, a left turn, a right
turn, and a change in direction), a lane for traveling,
25 a traveling speed, and passing. The behavior planning
unit 162 supplies data indicating the planned behavior
41
of the own automobile to, for example, the movement
planning unit 163.
[0099] On the basis of data or signals from the
respective structural elements of the vehicle control
system 100, such as the map analysis 5 unit 151 and the
state prediction unit 154, the movement planning unit
163 plans movement of the own automobile in order to
achieve the behavior planned by the behavior planning
unit 162. For example, the movement planning unit 163
10 makes plans about, for example, acceleration,
deceleration, and a traveling course. The movement
planning unit 163 supplies data indicating the planned
movement of the own automobile to, for example, an
acceleration/deceleration control unit 172 and a
15 direction control unit 173 of the movement control unit
135.
[0100] The movement control unit 135 controls
movement of the own automobile. The movement control
unit 135 includes the emergency event avoiding unit 171,
20 the acceleration/deceleration control unit 172, and the
direction control unit 173.
[0101] On the basis of a result of the detections
performed by the vehicle-exterior-information detection
unit 141, the vehicle-interior-information detection
25 unit 142, and the vehicle-state detection unit 143, the
emergency event avoiding unit 171 performs a process of
42
detecting emergency events such as collision, contact,
entrance into a dangerous zone, something unusual in a
driver, and anomaly in the vehicle. When the emergency
event avoiding unit 171 detects the occurrence of an
emergency event, the emergency event 5 avoiding unit 171
plans movement of the own automobile such as a sudden
stop or a quick turning for avoiding the emergency
event. The emergency event avoiding unit 171 supplies
data indicating the planned movement of the own
10 automobile to, for example, the
acceleration/deceleration control unit 172 and the
direction control unit 173.
[0102] The acceleration/deceleration control unit
172 controls acceleration/deceleration to achieve the
15 movement of the own automobile planned by the movement
planning unit 163 or the emergency event avoiding unit
171. For example, the acceleration/deceleration control
unit 172 computes a control target value for a driving
force generation apparatus or a braking apparatus to
20 achieve the planned acceleration, the planned
deceleration, or the planned sudden stop, and supplies
a control instruction indicating the computed control
target value to the drivetrain control unit 107.
[0103] The direction control unit 173 controls a
25 direction to achieve the movement of the own automobile
planned by the movement planning unit 163 or the
43
emergency event avoiding unit 171. For example, the
direction control unit 173 computes a control target
value for a steering mechanism to achieve the traveling
course planned by the movement planning unit 163 or the
quick turning planned by the emergency 5 event avoiding
unit 171, and supplies a control instruction indicating
the computed control target value to the drivetrain
control unit 107.
[0104] [Example of Configuration of Self-Location
10 Estimation System]
Fig. 3 is a block diagram of an example of a
configuration of the self-location estimation system
200 to which the present technology has been applied.
The self-location estimation system 200 relates
15 primarily to processes performed by the self-location
estimation unit 132, the vehicle-exterior-information
detection unit 141, and the state recognition unit 153
from among the vehicle control system 100, and a
process of generating a map used to perform processing
20 of estimating a self-location.
[0105] The self-location estimation system 200 is a
system that estimates a self-location and a posture of
the vehicle 10.
The self-location estimation system 200 includes a
25 control unit 201, an image acquisition unit 202, a
vehicle-state-information acquisition unit 203, a
44
characteristic point extracting unit 204, a
characteristic point tracking unit 205, a map
generation unit 206, an estimation unit 207, and a map
information storing unit 208.
[0106] In the self-location estimation 5 system 200, a
self (a vehicle 10's) location necessary for control
processing performed by the vehicle control system 100
is estimated. In the self-location estimation system
200, image information regarding an image captured by
10 the camera 300, vehicle state information from the
vehicle-state detection sensor 400, and the like are
used to estimate the location of the vehicle 10.
[0107] On the basis of control prediction
information from the server apparatus 500, the control
15 unit 201 outputs assignment-of-weights information
regarding assignment of weights upon estimation to the
estimation unit 207, the assignment-of-weights
information indicating how the image information and
the vehicle-state detection information are
20 respectively weighted to be used for estimating a selflocation
and a posture of the vehicle 10.
[0108] Further, on the basis of the control
prediction information, the control unit 201 controls
how the respective pieces of image information acquired
25 from a plurality of cameras 300 are weighted to be used
for estimating a self-location and a posture of the
45
vehicle 10. The control unit 201 outputs the
assignment-of-weights information regarding assignment
of weights to each camera to the image acquisition unit
202.
[0109] Furthermore, on the basis 5 of the control
prediction information, the control unit 201 performs
control with respect to masking of a characteristic
extraction region not used for a characteristic point
extraction performed to estimate a self-location and a
10 posture of the vehicle 10. The control unit 201 outputs
information regarding masking of a characteristic-point
extraction region to the characteristic point
extracting unit 204.
[0110] The image acquisition unit 202
15 chronologically acquires images captured by the camera
300 being installed in the vehicle 10 and serving as a
sensing device. In the present embodiment, a plurality
of cameras 300 is installed, and this makes it possible
to acquire range images obtained due to the parallax.
20 Note that a single camera may be installed to acquire
range images from chronologically captured multipleframe
images. The camera is included in the data
acquisition unit 102 described above.
[0111] The image acquisition unit 202 outputs the
25 chronological images that are selected images to the
characteristic point extracting unit 204 together with
46
the assignment-of-weights information regarding
assignment of weights to each camera from the control
unit 201.
The assignment of weights to the camera 300
includes, for example, selection of 5 a camera 300, from
among a plurality of cameras 300, that is not used upon
estimating a self-location and a posture. In this case,
a weight assigned to image information regarding an
image captured by the unused camera 300 is zero.
10 [0112] The vehicle-state-information acquisition
unit 203 chronologically acquires results of a vehicle
state detection performed by the vehicle-state
detection sensor 400 installed in the vehicle 10.
[0113] Examples of the vehicle-state detection
15 sensor 400 include a GNSS that detects a current
location of the vehicle 10, and sensors that detect a
state and the like of the vehicle 10 such as a gyro
sensor; an acceleration sensor; an inertial measurement
unit (IMU); and a sensor or the like for detecting an
20 amount of operation of an accelerator pedal, an amount
of operation of a brake pedal, a steering angle of a
steering wheel, the number of revolutions of an engine,
the number of revolutions of a motor, a speed of wheel
rotation, or the like.
25 [0114] On the basis of the information regarding
masking of a characteristic-point extraction region
47
from the control unit 201, the characteristic point
extracting unit 204 masks, as necessary, a
characteristic-point extraction region in the
chronological images input by the image acquisition
5 unit 202.
For example, a region of a position of the sun in
an image is a mask region obtained by masking a
characteristic-point extraction region, and a
characteristic point situated in this region is not
10 extracted.
[0115] The characteristic point extracting unit 204
extracts a characteristic point of a stationary object
from a region other than the mask region obtained by
masking a characteristic-point extraction region in the
15 image, and outputs the extracted characteristic point
to the characteristic point tracking unit 205.
Specifically, the characteristic point extracting
unit 204 extracts a stationary object (a landmark) from
the successively input chronological images, and
20 extracts a characteristic point of the extracted
stationary object.
[0116] The characteristic point tracking unit 205
tracks, in the sequentially input chronological images,
the characteristic point extracted by the
25 characteristic point extracting unit 204, and provides
the map generation unit 206 with information regarding
48
the tracking.
[0117] The map generation unit 206 generates map
information that is applicable to estimation of a selflocation
and includes a three-dimensional coordinate
(three-dimensional point) of a 5 stationary object
(characteristic point) in a world coordinate system of
the object.
[0118] The map generation unit 206 updates an
initial map at preset intervals. In other words, the
10 map generation unit 206 initializes, at the preset
intervals, map information of an object (a coordinate
of the object in the world coordinate system) that is
stored in the map information storing unit 208, so as
to update the map information.
15 [0119] On the basis of the assignment-of-weights
information regarding assignment of weights upon
estimation that is output by the control unit 201, the
estimation unit 207 respectively assigns weights to the
map information generated by the map generation unit
20 206 and the vehicle-state detection information output
by the vehicle-state-information acquisition unit 203,
and estimates a self-location and a posture of the
vehicle 10 using these pieces of information.
[0120] On the basis of the coordinate of the object
25 in the world coordinate system (the three-dimensional
coordinate of the characteristic point), the estimation
49
unit 207 estimates a transformation matrix representing
transformation from the world coordinate system to a
camera coordinate system that represents a coordinate
system based on a camera, and estimates, on the basis
of the transformation matrix, an angle 5 of rotation that
represents a location and a posture of the vehicle 10
with respect to the stationary object (the
characteristic point).
[0121] For example, when the assignment-of-weights
10 information indicates that a weight assigned to image
information is 100 and a weight assigned to vehiclestate
detection information is 0, the estimation unit
207 estimates a self-location and a posture of the
vehicle 10 only on the basis of map information
15 generated by the map generation unit 206 on the basis
of the image information.
[0122] On the other hand, when the assignment-ofweights
information indicates that the weight assigned
to the image information is 0 and the weight assigned
20 to the vehicle-state detection information is 100, the
estimation unit 207 estimates the self-location and the
posture of the vehicle 10 only on the basis of the
vehicle-state detection information.
For example, when it is difficult to acquire,
25 using the camera 300, an image suitable to estimate a
self-location and a posture, due to sunlight due to the
50
sun being reflected in an image, the self-location and
the posture of the vehicle 10 are estimated using the
vehicle-state detection information and the image
information in a state in which the vehicle-state
detection information is more heavily 5 weighted than the
image information.
[0123] The map information storing unit 208 stores
therein map information that includes a coordinate of
an object in a world coordinate system.
10 [0124] [Example of Configuration of Camera]
Fig. 4 is a block unit of an example of a
configuration of the camera 300 serving as an imagecapturing
device.
The camera 300 is an image sensor that captures an
15 image of the surroundings of the vehicle 10 at a
specified frame rate, and detects image information
regarding the image of the surroundings of the vehicle
10.
[0125] For example, two front cameras and two rear
20 cameras are installed in the vehicle 10, the two front
cameras capturing an image of the view to the front of
the vehicle 10 and being respectively provided on the
left and on the right in the front of a vehicle body,
the two rear cameras capturing an image of the view to
25 the rear of the vehicle 10 and being respectively
provided on the left and on the right in the rear of
51
the vehicle body.
[0126] For example, an RGB camera that includes an
image sensor such as a CCD or a CMOS is used as the
camera 300. The camera 300 is not limited to this, and,
for example, an image sensor that 5 detects infrared
light or polarization light may be used as appropriate.
The use of infrared light or polarization light makes
it possible to generate, for example, image information
regarding an image in which there is not a great change
10 in how the image looks even if there is a change in
weather.
[0127] In the camera 300 according to the present
embodiment, image information is generated by
performing, for example, an exposure control and a
15 photometry-region control on the basis of control
prediction information from the server apparatus 500.
[0128] The camera 300 includes a camera control unit
301, a lens-unit drive circuit 302, a lens unit 303, a
mechanical shutter (hereinafter abbreviated to a
20 mechano-shutter, and the same applies to the figure)
drive circuit 304, a mechano-shutter 305, an imagesensor
drive circuit 306, an image sensor 307, an
automatic-gain control (AGC) circuit 308, a signal
processing unit 309, and a photometry unit 310.
25 [0129] The camera control unit 301 includes an
exposure control unit 3011 and a photometry-region
52
control unit 3012.
[0130] The exposure control unit 3011 generates a
control signal that controls light exposure, on the
basis of control prediction information supplied by the
server apparatus 500 and a result 5 of photometry
performed by the photometry unit 310.
[0131] On the basis of the control prediction
information supplied by the server apparatus 500, the
exposure control unit 3011 generates a control signal
10 that controls a lens position and a stop of the lens
unit 303 through the lens-unit drive circuit 302,
controls driving of the mechano-shutter 305 through the
mechano-shutter drive circuit 304, controls an
electrical operation of the image sensor 307 through
15 the image-sensor drive circuit 306, and controls an
operation timing of the AGC circuit 308. Accordingly,
light exposure is controlled.
[0132] On the basis of the control prediction
information supplied by the server apparatus 500, the
20 photometry-region control unit 3012 supplies a control
signal that defines a photometry region to the
photometry unit 310.
[0133] For example, the photometry-region control
unit 3012 generates a control signal that defines a
25 region for the photometry unit 310 in accordance with
the control prediction information indicating that a
53
region, in an image, in which it has been predicted by
the server apparatus 500 that the sun will be reflected
is not to be set to be a photometry target (masking
processing regarding a photometry region), and the
other region is to be set to be a 5 photometry target.
[0134] The lens-unit drive circuit 302 includes a
motor and the like, and, on the basis of the control
signal supplied by the camera control unit 301, the
lens-unit drive circuit 302 adjusts a focal position
10 and a stop by moving the lens position of the lens unit
303.
[0135] The mechano-shutter drive circuit 304
controls a shutter speed and a shutter timing of the
mechano-shutter 305 on the basis of the control signal
15 supplied by the camera control unit 301.
[0136] The mechano-shutter 305 is arranged on an
entire surface of the image sensor 307, and is opened
or closed according to control performed by the
mechano-shutter drive circuit 304 so that light passing
20 through the lens unit 303 is transmitted through the
mechano-shutter 305 or is blocked by the mechanoshutter
305.
Note that the example of using a mechano-shutter
is described in the present embodiment, but an
25 electronic shutter may be used.
[0137] The image-sensor drive circuit 306 generates
54
a signal that drives the image sensor 307 on the basis
of, for example, a timing signal supplied by the camera
control unit 301, and adjusts, for example, a timing of
capturing an image.
[0138] The image sensor 307 includes 5 a solid-state
imaging element such as a complementary metal oxide
semiconductor (CMOS) image sensor or a charged coupled
device (CCD) image sensor.
[0139] The image sensor 307 receives light from a
10 subject that enters through the lens unit 303 to
photoelectrically convert the received light, and
outputs an analog image signal depending on the amount
of light received to the AGC circuit 308 and the
photometry unit 310.
15 [0140] The AGC circuit 308 adjusts a gain of the
image signal on the basis of the control signal from
the exposure control unit 3011, and outputs, to the
signal processing unit 309, the image signal of which
the gain has been adjusted.
20 [0141] The signal processing unit 309 performs
analog/digital (A/D) conversion with respect to the
analog image signal from the AGC circuit 308. Further,
with respect to image data represented by a digital
signal obtained by the A/D conversion, the signal
25 processing unit 309 applies denoising processing or the
like, and outputs, to the image acquisition unit 202 of
55
the self-location estimation system 200, image data
(image information) obtained as a result of applying
the denoising processing or the like.
[0142] The photometry unit 310 performs photometry
on the basis of the image signal from 5 the image sensor
307. Upon performing photometry, the photometry unit
310 performs photometry with respect to a defined
photometry region, on the basis of the control signal
supplied by the camera control unit 301. The photometry
10 unit 310 outputs a result of the photometry to the
camera control unit 301.
[0143] [Example of Configuration of Server
Apparatus]
Fig. 5 is a block diagram of an example of a
15 configuration of the server apparatus 500.
As illustrated in Fig. 5, the server apparatus 500
includes a communication unit 501, the information
acquisition unit 502, the unit 503 for predicting a
location and a posture of a vehicle at T+N, the unit
20 504 for predicting a position of a disturbance factor
(the sun) in an image, and the control prediction unit
505.
[0144] The communication unit 501 communicates with
equipment (for example, an application server or a
25 control server) situated in the vehicle 10 or an
external network (for example, the Internet, a cloud
56
network, or a carrier-specific network) through a base
station or an access point.
[0145] The communication unit 501 receives various
information from the vehicle 10, and outputs the
received information to the information 5 acquisition
unit 502. The communication unit 501 transmits, to the
vehicle 10, control prediction information received
from the control prediction unit 505.
[0146] The information acquisition unit 502 acquires,
10 from the vehicle 10 and through the communication unit
501, parameter information regarding a parameter of the
camera 300, information regarding a location and a
posture of the vehicle 10 at a time T, and vehicle
state information of the vehicle 10 at the time T.
15 The parameter information regarding a parameter of
the camera 300 is output to the unit 504 for predicting
a position of the sun in an image.
[0147] The information regarding a location and a
posture of the vehicle 10 at the time T, and the
20 vehicle state information of the vehicle 10 at the time
T are output to the unit 503 for predicting a location
and a posture of a vehicle at T+N.
The information regarding a location of the
vehicle 10 is, for example, information of a GNSS
25 signal detected by the data acquisition unit 102.
The vehicle state information includes pieces of
57
information detected by a gyro sensor, an acceleration
sensor, and an inertial measurement unit (IMU); and
information such as an amount of operation of an
accelerator pedal, an amount of operation of a brake
pedal, a steering angle of a steering 5 wheel, the number
of revolutions of an engine, the number of revolutions
of a motor, or a speed of wheel rotation.
[0148] The parameters of the camera 300 include an
internal parameter and an external parameter.
10 The internal parameter of the camera 300 is
camera-specific information, such as a focal length of
a camera lens, distortion characteristics of a lens,
and an error in a position of mounting a lens, that is
determined independently of a vehicle status.
15 The external parameter of the camera 300 is
information regarding a position and an orientation of
mounting each camera 300, with a self-location and a
posture of the vehicle 10 being the center of the
vehicle 10, that is, information regarding a position
20 and a posture of the camera 300, with the center of the
vehicle 10 being used as a reference.
[0149] The unit 503 for predicting a location and a
posture of a vehicle at T+N predicts a location and a
posture of the vehicle 10 at T+N on the basis of the
25 information regarding a location and a posture of the
vehicle 10 at the time T and the vehicle state
58
information of the vehicle 10 at the time T that are
output from the information acquisition unit 502. The
unit 503 for predicting a location and a posture of a
vehicle at T+N outputs a result of the prediction to
the unit 504 for predicting a position 5 of the sun in an
image.
[0150] The unit 504 for predicting a position of the
sun in an image predicts a position of the sun in an
image predicted to be captured by the camera 300 at T+N
10 on the basis of the parameter information of the camera
300, information regarding the location and the posture
of the vehicle at T+N, information 506 regarding a date
and time/weather, and map information 507.
[0151] Specifically, first, using the date-and-time
15 information, information regarding the predicted
location and posture of the vehicle 10 at T+N, and the
map information 507, the unit 504 for predicting a
position of the sun in an image calculates the
elevation (elevation angle) and azimuth of the sun in a
20 location in which the vehicle 10 is predicted to exist
at T+N.
[0152] Next, the unit 504 for predicting a position
of the sun in an image determines whether the sun will
be out on the basis of the weather information
25 regarding weather in the location in which the vehicle
10 is predicted to exist at T+N.
59
For example, when the unit 504 for predicting a
position of the sun in an image has determined, from
the weather information, that the sun will be out in
clear weather, the unit 504 for predicting a position
of the sun in an image predicts a position 5 of the sun
in an image predicted to be captured by the camera 300
at T+N, on the basis of information regarding the
elevation and azimuth of the sun at T+N, the
information regarding the location and posture of the
10 vehicle 10, and the parameter information of the camera
300.
[0153] On the other hand, when the unit 504 for
predicting a position of the sun in an image has
determined, from the weather information, that the sun
15 will not be out in rainy weather, the unit 504 for
predicting a position of the sun in an image predicts
that the sun will not be reflected in the image
captured by the camera 300 at T+N.
[0154] Here, the information 506 regarding a date
20 and time/weather can be acquired by, for example, the
server apparatus 500 communicating with an application
server that exists in an external network.
The map information 507 is stored in the server
apparatus 500 in advance and updated as necessary at
25 preset intervals.
[0155] A result of the prediction performed by the
60
unit 504 for predicting a position of the sun in an
image with respect to the position of the sun in an
image, is output to the control prediction unit 505.
[0156] On the basis of the input result of the
prediction performed with respect 5 to the position of
the sun in an image, the control prediction unit 505
predicts control performed with respect to the sensing
device. Control prediction information regarding the
control predicted by the control prediction unit 505 is
10 transmitted to the vehicle 10 through the communication
unit 501.
[0157] The control performed with respect to the
sensing device includes control performed with respect
to the camera 300 and control performed in the self15
location estimation system 200 with respect to the
sensing device.
[0158] The control performed with respect to the
camera 300 includes an exposure control and a
photometry-region control.
20 First, the exposure control that is the control
performed with respect to the camera 300 is described.
[0159] Generally, in the camera 300, an image sensor
included in the camera 300 receives light from a
subject, and an image signal depending on the amount of
25 light received is supplied to the photometry unit. In
the photometry unit, photometry is performed on the
61
basis of the image signal, and an adequate exposure
value is calculated on the basis of a brightness value
obtained as a photometry value. The exposure control is
performed on the basis of the calculated exposure value.
[0160] When the exposure control is 5 performed on the
basis of an exposure value calculated at the time T, an
image captured under an exposure condition proper at
the time T is obtained at T+M (M>0), that is, after a
lapse of M minutes from the time T. In other words, it
10 is not possible to obtain an image perfect for the time
T under the exposure condition proper at the time T,
and thus a time lag occurs.
[0161] On the other hand, in the camera 300 of the
vehicle 10, it is possible to set an exposure condition
15 at T+N for the camera 300 in advance to perform imagecapturing,
on the basis of control prediction
information regarding an exposure control performed
with respect to the camera 300 at T+N that is predicted
by the server apparatus 500. This results in being able
20 to perform image-capturing, without a time lag, at T+N
under a proper exposure condition for a state at T+N.
[0162] Examples of the exposure control include
control such as adjustment of a shutter speed and
adjustment of a gain of an image signal. For example,
25 when the sun is reflected in an image, the control
prediction unit 505 predicts an exposure control such
62
as increasing a shutter speed and lowering a gain,
since the sun makes the image too clear.
[0163] Information regarding the control prediction
is transmitted to the vehicle 10, and the exposure
control with respect to the camera 300 5 is performed on
the basis of the transmitted information. This is an
exposure condition upon performing image-capturing at
T+N.
[0164] Here, when a self-location and a posture are
10 estimated using an image in the self-location
estimation system 200, sight of a characteristic point
in an image may be lost or the characteristic point in
the image may be falsely recognized due to the sun
being reflected in the image upon extracting the
15 characteristic point, and this may result in there
being a decrease in the accuracy in estimating a selflocation
and a posture.
[0165] On the other hand, in the present embodiment,
the reflection of the sun in an image at T+N is
20 predicted in advance to predict an exposure control
such that a proper image can be obtained. It is
possible to capture an image that is to be captured at
T+N under an exposure condition based on information
regarding the prediction.
25 [0166] This enables the vehicle 10 to obtain an
image that is captured with an adequate exposure and is
63
proper for a state at T+N. Thus, upon estimating a
self-location and a posture using an image, it is
possible to use an image that is captured under a
proper exposure condition and in which sight of a
characteristic point is less likely 5 to be lost, and
this results in an improvement in the accuracy in
estimating a self-location and a posture.
[0167] Next, the photometry-region control that is
the control performed with respect to the camera 300 is
10 described.
For example, when the sun is reflected in an image,
the control prediction unit 505 predicts the
photometry-region control performed such that a region,
in the image, in which the sun is reflected is excluded
15 from a photometry region. Control prediction
information regarding the control predicted by the
control prediction unit 505 is transmitted to the
vehicle 10 through the communication unit 501.
[0168] Here, when the sun is reflected in an image,
20 a region in which the sun is reflected and the
surroundings of the region in the image are excessively
bright, and a region other than the excessively bright
region is dark, which results in blocked-up shadows.
When a self-location and a posture are estimated in the
25 self-location estimation system 200 using such an image,
sight of a characteristic point in the image may be
64
lost or the characteristic point in the image may be
falsely recognized.
[0169] On the other hand, in the present embodiment,
the reflection of the sun is predicted in advance, and
a region, in the image, in which the 5 sun is reflected
is excluded from a photometry region to perform
photometry. This makes it possible to provide a proper
image without blocked-up shadows. Thus, upon estimating
a self-location and a posture using an image, it is
10 possible to provide a proper image in which sight of a
characteristic point is less likely to be lost, and
this results in an improvement in the accuracy in
estimating a self-location and a posture.
[0170] The control performed in the self-location
15 estimation system 200 with respect to the sensing
device includes control that is performed upon
estimating a self-location and a posture and is related
to assignment of weights to image information and
vehicle-state detection information, control of a
20 characteristic-point-extraction mask region that is
performed upon estimating the self-location and the
posture, and control that is performed upon estimating
the self-location and the posture and is related to
assignment of weights to respective pieces of image
25 information from a plurality of cameras 300.
[0171] First, the control related to assignment of
65
weights to image information and vehicle-state
detection information is described.
For example, when the sun is reflected in an image,
a region in which the sun is reflected and the
surroundings of the region in the image 5 are excessively
bright, and a region other than the excessively bright
region is dark, which results in blocked-up shadows.
When a self-location and a posture are estimated in the
self-location estimation system 200 using such an image,
10 sight of a characteristic point may be lost or the
characteristic point may be falsely recognized.
[0172] In such a case, the control prediction unit
505 predicts control performed such that the vehiclestate
detection information is more heavily weighted
15 than the image information to estimate a self-location
and a posture in the self-location estimation system
200. Since a self-location and a posture are estimated
in the vehicle 10 on the basis of information regarding
this control prediction, there is an improvement in the
20 accuracy in estimating a self-location and a posture.
[0173] Next, the control of a characteristic-pointextraction
mask region is described.
[0174] For example, when the sun is reflected in an
image, sight of a characteristic point in the image may
25 be lost or the characteristic point in the image may be
falsely recognized in a region in the image in which
66
the sun is reflected, due to the brilliance of the sun.
[0175] In such a case, the control prediction unit
505 predicts control performed such that, upon
estimating a self-location and a posture in the selflocation
estimation system 200 using 5 an image, a region
in the image in which the sun is reflected is set to be
a region not used to extract a characteristic point
(characteristic-point-extraction mask region).
[0176] In the vehicle 10, on the basis of
10 information regarding this control prediction, a region
in which the sun is reflected is first masked to be set
a region from which a characteristic point is not
extracted, and then a characteristic point is extracted
to estimate a self-location and a posture. This results
15 in an improvement in the accuracy in estimating a selflocation
and a posture using an image.
[0177] Next, the control related to assignment of
weights to respective pieces of image information from
a plurality of cameras 300, is described.
20 [0178] For example, the control prediction unit 505
predicts control performed such that, upon estimating a
self-location and a posture using an image, image
information from a camera 300 that captures an image in
which the sun is reflected, is lightly weighted, and
25 image information from a camera 300 other than the
camera 300 capturing an image in which the sun is
67
reflected, is heavily weighted.
[0179] In the vehicle 10, on the basis of
information regarding this control prediction, an image
acquired by a camera 300 that captures an image in
which the sun is not reflected, is 5 primarily used to
estimate a self-location and a posture. This results in
an improvement in the accuracy in estimating a selflocation
and a posture using an image.
[0180] [Control Prediction Processing]
10 Fig. 6 illustrates a flow of control prediction
processing performed in the server apparatus 500 to
generate control prediction information regarding
prediction of control performed with respect to the
sensing device. S represents Step.
15 [0181] When the control prediction processing is
started, the information acquisition unit 502 acquires,
from the vehicle 10 and through the communication unit
501, information regarding the vehicle 10 such as
parameter information regarding a parameter of the
20 camera 300, information regarding a location and a
posture of the vehicle 10, and information regarding a
state of the vehicle 10 (S1).
[0182] Next, the unit 503 for predicting a location
and a posture at T+N predicts a location and a posture
25 of the vehicle 10 at T+N on the basis of the
information regarding a location and a posture of the
68
vehicle 10 at a time T and the vehicle state
information of the vehicle 10 at the time T that are
output from the information acquisition unit 502 (S2).
[0183] Next, the unit 504 for predicting a position
of the sun in an image predicts a position 5 of the sun
in an image predicted to be captured by the camera 300
at T+N, on the basis of the parameter information of
the camera 300, information regarding the location and
the posture of the vehicle at T+N, the information 506
10 regarding a date and time/weather, and the map
information 507 (S3).
[0184] Next, on the basis of information regarding
the predicted position of the sun in the image, the
control prediction unit 505 predicts control performed
15 with respect to the sensing device, and generates
control prediction information (a control prediction
signal) (S4).
Next, the communication unit 501 transmits the
generated control prediction information to the vehicle
20 10 (S5).
[0185] The server apparatus 500 includes hardware,
such as a central processing unit (CPU), a read only
memory (ROM), a random access memory (RAM), and a hard
disk drive (HDD), that is necessary for a configuration
25 of a computer.
[0186] In the server apparatus 500, the above69
described control prediction processing to generate
control prediction information regarding prediction of
control performed with respect to the sensing device is
performed by loading, into the RAM, a program stored in
the ROM and executing 5 the program.
[0187] As described above, in the control system 1
according to the present embodiment, control performed
with respect to the camera 300 and the self-location
estimation system 200 that are installed in the vehicle
10 10 such that an impact due to the reflection of the sun
in an image captured at T+N is reduced, is predicted on
the basis of information regarding a location and a
posture of the vehicle 10 at a time T, the map
information 507, and the information 506 regarding a
15 date and time/weather.
[0188] In the vehicle 10, control performed with
respect to the camera 300 and the self-location
estimation system 200 that are installed in the vehicle
10 is performed on the basis of information regarding
20 this control prediction. Thus, an image that is proper
for a state of the vehicle at T+N and in which an
impact due to the reflection of the sun, a disturbance
factor, is reduced, can be obtained at T+N without a
time lag. This results in obtaining a highly robust
25 control system.
[0189]
70
[Example of Configuration of Control System]
In the present embodiment, the description is made
taking a tunnel as an example of a disturbance factor.
Fig. 1 is the block diagram of an example of a
schematic functional configuration of 5 a control system
1000 for the sensing device to which the present
technology is applicable.
Fig. 8 is a block diagram of a functional
configuration of a server apparatus (an information
10 processing apparatus) in the control system 1000.
In the following description, the same structural
element as the first embodiment may be denoted by the
same reference symbol and a description thereof may be
omitted.
15 [0190] The control system 1000 includes the vehicle
control system 100 that is installed in the vehicle 10
that is a mobile object, and a server apparatus 1500
that serves as an information processing apparatus. The
vehicle control system 100 and the server apparatus
20 1500 are capable of communicating with each other, for
example, through a wireless communication network.
[0191] The server apparatus 1500 includes the
communication unit 501, the information acquisition
unit 502, the unit 503 for predicting a location and a
25 posture of a vehicle at T+N, a unit 1504 for predicting
a position of a disturbance factor (a tunnel) in an
71
image (hereinafter referred to as a unit for predicting
a position of a tunnel in an image), and a control
prediction unit 1505.
[0192] The information acquisition unit 502 acquires,
from the vehicle 10 and through the 5 communication unit
501, parameter information regarding a parameter of the
camera 300, information regarding a location and a
posture of the vehicle 10, and vehicle state
information of the vehicle 10.
10 The parameter information regarding a parameter of
the camera 300 is output to the unit 504 for predicting
a position of a tunnel in an image.
[0193] In the present embodiment, the unit 1504 for
predicting a position of a tunnel in an image predicts
15 a position of a tunnel, a disturbance factor, in an
image.
[0194] The unit 1504 for predicting a position of a
tunnel in an image predicts a position of a tunnel in
an image captured at T+N, on the basis of information
20 regarding the predicted self-location and posture of
the vehicle 10 at T+N, and the map information 507.
Tunnel-position-prediction information is output to the
control prediction unit 1505. The map information 507
includes tunnel position information.
25 [0195] On the basis of the input information
regarding the position of the tunnel in the image, the
72
control prediction unit 1505 predicts control performed
with respect to the sensing device. Control prediction
information regarding the control predicted by the
control prediction unit 1505 is transmitted to the
vehicle 10 through the communication 5 unit 501.
[0196] The control performed with respect to the
sensing device includes control performed with respect
to the camera 300, control performed in the selflocation
estimation system 200 with respect to the
10 sensing device, and control performed with respect to
both the camera 300 and the self-location estimation
system 200.
[0197] The control performed with respect to the
camera 300 includes an exposure control and a
15 photometry-region control.
[0198] The control prediction unit 1505 predicts an
exposure control performed with respect to the camera
300 at T+N, on the basis of a result of the prediction
of the position of the tunnel in the image captured at
20 T+N. Information regarding the control prediction is
transmitted to the vehicle 10. An exposure control with
respect to the camera 300 is performed in the vehicle
10 on the basis of the transmitted information. This is
an exposure condition upon performing image-capturing
25 at T+N.
[0199] For example, an exposure control such as
73
reducing a shutter speed and increasing a gain is
predicted such that a dark image is not obtained and
the inside of a tunnel appears in a proper image, since
it becomes dark in the tunnel.
[0200] Further, the control prediction 5 unit 1505
predicts a photometry-region control performed with
respect to the camera 300 at T+N, on the basis of the
result of the prediction of a position of a tunnel in
the image captured at T+N. Information regarding the
10 control prediction is transmitted to the vehicle 10,
and the photometry-region control with respect to the
camera 300 is performed on the basis of the transmitted
information. This is a photometry-region condition upon
performing image-capturing at T+N.
15 [0201] For example, when there is some distance
between an entrance of a tunnel and the vehicle 10
before the vehicle 10 enters the tunnel, the control
prediction unit 1505 predicts control performed such
that a region other than a region inside the tunnel is
20 set to be a photometry region and such that the region
other than the region inside the tunnel appears in a
proper image although a region, in the image, in which
the tunnel is situated exhibits blocked-up shadows. In
other words, the control prediction unit 1505 predicts
25 control performed such that an image based on
brightness outside the tunnel and not brightness inside
74
the tunnel, is obtained.
[0202] Likewise, when an exit of the tunnel appears
in an image in a state of the vehicle being situated
inside the tunnel and when the vehicle gets close to
the exit of the tunnel to some extent, 5 the control
prediction unit 1505 predicts control performed such
that the region other than the region inside the tunnel
is set to be a photometry region.
[0203] When the vehicle 10 gets close to the
10 entrance of the tunnel and when the vehicle 10 gets
close to the tunnel by a certain distance or more, the
control prediction unit 1505 predicts control performed
such that the region inside the tunnel is set to be a
photometry region and such that the inside of the
15 tunnel appears in a proper image. In other words, the
control prediction unit 1505 predicts control performed
such that an image based on brightness inside the
tunnel is obtained.
[0204] Likewise, when the exit of the tunnel does
20 not yet appear in an image in a state of the vehicle
being situated inside the tunnel, or when the exit of
the tunnel appears in the image in the state of the
vehicle being situated inside the tunnel, but the
vehicle is still distant from the exit of the tunnel,
25 the control prediction unit 1505 predicts control
performed such that the region inside the tunnel is set
75
to be a photometry region.
[0205] With respect to control performed such that a
region outside a tunnel is set to be a photometry
region and control performed such that a region inside
the tunnel is set to be the photometry 5 region, which of
the controls is to be selected may be determined, for
example, depending on the proportion of a region, in an
image, in which the tunnel appears.
[0206] Further, with respect to an image captured
10 when a region outside a tunnel is set to be a
photometry region and an image captured when a region
inside the tunnel is set to be the photometry region,
the control prediction unit 1505 predicts control
performed such that the two images are interpolated so
15 that there is not a sharp change between the images.
[0207] Note that, when a region outside a tunnel is
set to be a photometry region, the control according to
the first embodiment described above that is performed
with respect to the camera or the self-location
20 estimation system on the basis of a position of the sun,
is performed, or control according to a third
embodiment described later that is performed with
respect to the camera or the self-location estimation
system on the basis of a shadow of a structure, is
25 performed.
[0208] This makes it possible to perform image76
capturing in the vehicle 10 under a photometry
condition or an exposure condition that is proper for a
state at T+N.
[0209] The control performed in the self-location
estimation system 200 with respect 5 to the sensing
device includes control that is performed upon
estimating a self-location and a posture and is related
to assignment of weights to image information and
vehicle-state detection information, and control that
10 is performed upon estimating the self-location and the
posture and is related to assignment of weights to
pieces of image information that are respectively
acquired from a plurality of cameras 300.
[0210] The control prediction unit 1505 predicts how
15 image information and vehicle-state detection
information are respectively weighted in the selflocation
estimation system 200 to be used for
estimating a location of the vehicle 10.
[0211] For example, since it is dark in a tunnel,
20 sight of a characteristic point may be lost or the
characteristic point may be falsely recognized when a
self-location and a posture are estimated in the selflocation
estimation system 200 using image information.
In such a case, the control prediction unit 1505
25 predicts control performed such that, upon estimating a
self-location and a posture in the self-location
77
estimation system 200, a result of the estimation
performed using image information is lightly weighted
and a result of the estimation performed using vehiclestate
detection information is heavily weighted.
[0212] Since a self-location 5 and a posture are
estimated in the vehicle 10 on the basis of information
regarding this control prediction, there is an
improvement in the accuracy in estimating a selflocation
and a posture of the vehicle 10.
10 [0213] Further, the control prediction unit 1505
predicts control performed with respect to how
respective pieces of image information acquired by a
plurality of cameras 300 are weighted in the selflocation
estimation system 200 to be used for
15 estimating a location and a posture of the vehicle 10.
Examples of the control with respect to the assignment
of weights to the camera 300 includes selection of the
camera 300 used upon estimating a self-location and a
posture.
20 [0214] For example, control is predicted that is
performed such that image information regarding an
image captured by the camera 300 oriented toward a
tunnel is not used or is lightly weighted.
[0215] Next, the control performed with respect to
25 both the camera 300 and the self-location estimation
system 200 is described.
78
[0216] The control prediction unit 1505 predicts
control performed such that the camera 300 performs
image-capturing alternately in a first mode and in a
second mode for each frame. In addition, the control
prediction unit 1505 predicts control 5 performed with
respect to which of an image captured in the first mode
and an image captured in the second mode is used to
estimate a self-location at T+N in the self-location
estimation system 200.
10 [0217] Information regarding the control prediction
is transmitted to the vehicle 10. In the vehicle 10, a
self-location and a posture of the vehicle 10 are
estimated in the self-location estimation system 200 on
the basis of this information.
15 [0218] The first mode is an image-capturing mode
when a region other than a region inside a tunnel is
set to be a photometry region. In this mode, setting is
performed such that the region other than the region
inside the tunnel appears in an image captured with an
20 adequate exposure although a region, in the image, in
which the tunnel is situated exhibits blocked-up
shadows.
[0219] The second mode is an image-capturing mode
when the region inside the tunnel is set to be the
25 photometry region. In this mode, setting is performed
such that the inside of the tunnel appears in the image
79
captured with the adequate exposure. In the second mode,
control such as reducing a shutter speed and increasing
a gain is performed.
[0220] A pattern of a shutter speed and a gain is
switched between the first mode and 5 the second mode.
[0221] Processing of estimating a self-location and
a posture of the vehicle 10 is performed in the vehicle
10 on the basis of control prediction information that
includes image information and information that
10 indicates which of an image captured in the first mode
and an image captured in the second mode is used upon
estimating a self-location, the image information being
information regarding an image obtained by performing
image-capturing alternately in the first mode and in
15 the second mode for each frame.
[0222] In the case of processing of estimating a
self-location and a posture that is performed on the
basis of control prediction information indicating use
of an image captured in the first mode, an image
20 captured in the first mode is extracted from the
acquired image information in the self-location
estimation system 200, and the processing of estimating
a self-location and a posture is performed on the basis
of the extracted image.
25 [0223] On the other hand, in the case of processing
of estimating a self-location and a posture that is
80
performed on the basis of control prediction
information indicating use of an image captured in the
second mode, an image captured in the second mode is
extracted from the acquired image information in the
self-location estimation system 200, 5 and the processing
of estimating a self-location and a posture is
performed on the basis of the extracted image.
[0224] Note that, here, the example in which the
control prediction information includes information
10 that indicates which of an image in the first mode and
an image in the second mode is used to perform
processing of estimating a self-location and a posture,
and one of an image captured in the first mode and an
image captured in the second mode is used upon
15 estimating a self-location, has been described.
[0225] In addition, the control prediction
information may include information indicating how a
result of the processing of estimating a self-location
and a posture performed using an image captured in the
20 first mode, and a result of the processing of
estimating a self-location and a posture performed
using an image captured in the second mode are
respectively weighted to be integrated with each other.
[0226] In the case of this control prediction
25 information, processing of estimating a self-location
and a posture is performed for each mode in the self81
location estimation system 200, that is, processing of
estimating a self-location and a posture is performed
using image information regarding an image captured in
the first mode, and processing of estimating a selflocation
and a posture is performed 5 using image
information regarding an image captured in the second
mode. Then, processing of estimating a self-location
and a posture is performed in the self-location
estimation system 200 using a result obtained by
10 assigning weights to processing results in the
respective modes and integrating the weighted
processing results.
[0227] [Control Prediction Processing]
Fig. 8 illustrates a flow of control prediction
15 processing performed in the server apparatus 1500 to
generate control prediction information regarding
prediction of control performed with respect to the
sensing device.
[0228] When the control prediction processing is
20 started, the information acquisition unit 502 acquires,
from the vehicle 10 and through the communication unit
501, parameter information regarding a parameter of the
camera 300, information regarding a location and a
posture of the vehicle 10 at a time T, and vehicle
25 state information of the vehicle 10 at the time T (S11).
[0229] Next, the unit 503 for predicting a location
82
and a posture at T+N predicts a location and a posture
of the vehicle 10 at T+N on the basis of the
information regarding a location and a posture of the
vehicle 10 at the time T and the vehicle state
information of the vehicle 10 at the 5 time T that are
output from the information acquisition unit 502 (S12).
[0230] Next, the unit 1504 for predicting a position
of a tunnel in an image predicts a position of a tunnel
in an image predicted to be captured by the camera 300
10 at T+N, on the basis of the parameter information of
the camera 300, information regarding the position and
the posture of the vehicle at T+N, and the map
information 507, and predicts whether the tunnel will
appear in the image (S13).
15 [0231] When determination performed in S13 is No,
the process returns to S11 to be repeatedly performed.
When the determination performed in S13 is Yes,
the process moves on to S14.
[0232] In S14, on the basis of tunnel position
20 information, the control prediction unit 1505 predicts
control performed with respect to the sensing device,
and generates control prediction information (a control
prediction signal).
Next, the communication unit 501 transmits the
25 generated control prediction information to the vehicle
10 (S15).
83
[0233] The server apparatus 1500 includes hardware,
such as a CPU, a ROM, a RAM, and an HDD, that is
necessary for a configuration of a computer.
In the server apparatus 1500, the above-described
control prediction processing to 5 generate control
prediction information regarding prediction of control
performed with respect to the sensing device is
performed by loading, into the RAM, a program stored in
the ROM and executing the program.
10 [0234] As described above, in the control system
1000 according to the present embodiment, control
performed with respect to the camera 300 and the selflocation
estimation system 200 that are installed in
the vehicle 10 such that an impact due to the
15 appearance of a tunnel in an image captured at T+N is
reduced, is predicted on the basis of information
regarding a location and a posture of the vehicle 10 at
a time T, and the map information 507.
[0235] In the vehicle 10, control performed with
20 respect to the camera 300 and the self-location
estimation system 200 that are installed in the vehicle
10 is performed on the basis of information regarding
this control prediction. Thus, an image that is proper
for a state of the vehicle at T+N and in which an
25 impact due to the appearance of a tunnel, a disturbance
factor, is reduced, can be obtained at T+N without a
84
time lag. This results in obtaining a highly robust
control system.
[0236]
In the present embodiment, the description is made
taking a shadow of a structure that 5 is a stationary
object such as an architectural structure as an example
of a disturbance factor. In the following description,
a building is taken as an example of the structure.
[0237] Fig. 1 is the block diagram of an example of
10 a schematic functional configuration of a control
system 2000 for the sensing device to which the present
technology is applicable.
Fig. 9 is a block diagram of a functional
configuration of a server apparatus 2500 that serves as
15 an information processing apparatus in the control
system 2000.
In the following description, the same structural
element as the first embodiment may be denoted by the
same reference symbol and a description thereof may be
20 omitted.
[0238] The control system 2000 includes the vehicle
control system 100 that is installed in the vehicle 10
that is a mobile object, and the server apparatus 2500
that serves as an information processing apparatus. The
25 vehicle control system 100 and the server apparatus
2500 are capable of communicating with each other, for
85
example, through a wireless communication network.
[0239] The server apparatus 2500 includes the
communication unit 501, the information acquisition
unit 502, the unit 503 for predicting a location and a
posture of a vehicle at T+N, a unit 2504 5 for predicting
a position of a disturbance factor (a shadow of a
building) in an image (hereinafter referred to as a
unit for predicting a position of a shadow of a
building in an image), and a control prediction unit
10 2505.
[0240] The information acquisition unit 502 acquires,
from the vehicle 10 and through the communication unit
501, parameter information regarding a parameter of the
camera 300, information regarding a location and a
15 posture of the vehicle 10, and vehicle state
information of the vehicle 10.
The parameter information regarding a parameter of
the camera 300 is output to the unit 2504 for
predicting a position of a shadow of a building in an
20 image.
[0241] The unit 2504 for predicting a position of a
shadow of a building in an image predicts a position of
a shadow of a building, a disturbance factor, in an
image.
25 [0242] The unit 2504 for predicting a position of a
shadow of a building in an image predicts a position of
86
a shadow of a building in an image predicted to be
captured by the camera 300 at T+N, on the basis of the
parameter information of the camera 300, information
regarding the self-location and the posture of the
vehicle 10 at T+N, the information 506 5 regarding a date
and time/weather, and the map information 507.
[0243] Specifically, first, using the date-and-time
information, information regarding the predicted selflocation
and posture of the vehicle 10 at T+N, and the
10 map information 507, the unit 2504 for predicting a
position of a shadow of a building in an image
calculates the elevation (elevation angle) and azimuth
of the sun at T+N. The map information 507 includes
building position information.
15 [0244] Next, the unit 2504 for predicting a position
of a shadow of a building in an image determines
whether the sun will be out on the basis of the weather
information regarding weather in the location of the
vehicle 10 at T+N.
20 For example, when the unit 2504 for predicting a
position of a shadow of a building in an image has
determined that the sun will be out in clear weather,
the unit 2504 for predicting a position of a shadow of
a building in an image predicts a position of a shadow
25 of a building in an image captured by the camera 300 at
T+N, on the basis of information regarding the
87
elevation and azimuth of the sun at T+N, the
information regarding the self-location and posture of
the vehicle 10, the parameter information of the camera
300, and the map information 507.
[0245] A result of the prediction 5 performed by the
unit 2504 for predicting a position of a shadow of a
building in an image with respect to the position of a
shadow of a building in an image, is output to the
control prediction unit 2505.
10 [0246] On the basis of the input information
regarding a position of a shadow of a building in an
image, the control prediction unit 2505 predicts
control performed with respect to the sensing device.
Control prediction information regarding the control
15 predicted by the control prediction unit 2505 is
transmitted to the vehicle 10 through the communication
unit 501.
[0247] The control performed with respect to the
sensing device includes control performed with respect
20 to the camera 300, control performed in the selflocation
estimation system 200 with respect to the
sensing device, and control performed with respect to
both the camera 300 and the self-location estimation
system 200.
25 [0248] The control performed with respect to the
camera 300 includes an exposure control and a
88
photometry-region control.
[0249] On the basis of information regarding a
position of a shadow of a building in an image at T+N,
the control prediction unit 2505 predicts control
performed with respect to exposure, 5 such as adjustment
of a shutter speed and adjustment of a gain of an image
signal.
[0250] For example, when a shadow of a building
appears in an image, the control prediction unit 2505
10 predicts an exposure control such as reducing a shutter
speed and increasing a gain, since the image is likely
to become dark.
Information regarding the control prediction is
transmitted to the vehicle 10. An exposure control with
15 respect to the camera 300 is performed in the vehicle
10 on the basis of the transmitted control prediction
information, and this is an exposure condition upon
performing image-capturing at T+N.
[0251] This enables the vehicle 10 to obtain an
20 image that is captured with an adequate exposure and is
proper for a state at T+N.
Further, when a self-location and a posture are
estimated using an image in the self-location
estimation system 200, it is possible to use an image
25 that is captured with an adequate exposure and in which
sight of a characteristic point is less likely to be
89
lost, and this results in being able to improve the
accuracy in estimating a self-location and a posture.
[0252] On the basis of the information regarding a
position of a shadow of a building in an image at T+N,
the control prediction unit 2505 5 predicts control
performed with respect to a photometry region. Control
prediction information regarding the control predicted
by the control prediction unit 2505 is transmitted to
the vehicle 10 through the communication unit 501. A
10 photometry-region control with respect to the camera
300 is performed in the vehicle 10 on the basis of the
transmitted control prediction information, and this is
a photometry condition upon performing image-capturing
at T+N.
15 [0253] For example, when a shadow of a building
appears in an image, the control prediction unit 2505
predicts the photometry-region control performed such
that a region, in the image, in which the shadow of the
building appears is excluded from a photometry region,
20 since the region in which the shadow of the building
appears becomes dark.
[0254] Specifically, as long as there is some
distance or more between a shadow of a building and the
vehicle, the control prediction unit 2505 predicts
25 control performed such that a region other than a
region of the shadow of the building in an image is set
90
to be a photometry region and such that the region
other than the region of the shadow of the building
appears in a proper image although the region of the
shadow of the building exhibits blocked-up shadows.
[0255] On the other hand, 5 when a shadow of a
building and the vehicle 10 get close to each other by
a certain distance or more, the control prediction unit
2505 predicts control performed such that the region of
the shadow of the building is set to be the photometry
10 region and such that a shaded region appears in a
proper image.
[0256] Further, with respect to an image captured
when a shadow of a building is set to be a photometry
region and an image captured when a region other than
15 the shadow of the building is set to be the photometry
region, the control prediction unit 2505 predicts
control performed such that the two images are
interpolated so that there is not a sharp change
between the images.
20 [0257] This makes it possible to obtain an image
suitable for a state.
Further, a self-location and a posture can be
estimated in the self-location estimation system 200
using an image in which sight of a characteristic point
25 is less likely to be lost, and this results in being
able to improve the accuracy in estimating a self91
location and a posture.
[0258] The control performed in the self-location
estimation system 200 with respect to the sensing
device includes control that is performed upon
estimating a self-location and a posture 5 and is related
to assignment of weights to image information and
vehicle-state detection information, and control that
is performed upon estimating the self-location and the
posture and is related to assignment of weights to
10 respective pieces of image information that are
respectively acquired from a plurality of cameras 300.
[0259] On the basis of information regarding a
position of a shadow of a building in an image at T+N,
the control prediction unit 2505 predicts how image
15 information and vehicle-state detection information are
respectively weighted in the self-location estimation
system 200 to be used for estimating a self-location
and a posture of the vehicle 10.
[0260] For example, when the control prediction unit
20 2505 predicts that a screen will become dark due to
shadow of a building and there will be a decrease in
the accuracy in extracting a characteristic point, the
control prediction unit 2505 predicts control performed
such that, upon estimating a self-location and a
25 posture in the self-location estimation system 200, a
result of the estimation performed using image
92
information is lightly weighted and a result of the
estimation performed using vehicle-state detection
information is heavily weighted.
[0261] Information regarding the control prediction
is transmitted to the vehicle 10. Since 5 a self-location
and a posture are estimated in the vehicle 10 on the
basis of information regarding this control prediction,
there is an improvement in the accuracy in estimating a
self-location and a posture.
10 [0262] Further, the control prediction unit 2505
predicts control performed with respect to how
respective pieces of image information acquired by a
plurality of cameras 300 are weighted in the selflocation
estimation system 200 to be used for
15 estimating a location and a posture of the vehicle.
[0263] For example, control is predicted that is
performed such that image information regarding an
image captured by the camera 300 oriented toward a
shadow of a building is not used or is lightly weighted.
20 [0264] Next, the control performed with respect to
both the camera 300 and the self-location estimation
system 200 is described.
[0265] The control prediction unit 2505 predicts
control performed such that the camera 300 performs
25 image-capturing alternately in a first mode and in a
second mode for each frame. In addition, the control
93
prediction unit 2505 predicts control performed with
respect to which of an image captured in the first mode
and an image captured in the second mode is used for
estimating a self-location at T+N in the self-location
estimation system 200, or control 5 performed with
respect to how a result of estimation of the selflocation
at T+N that is performed using the image
captured in the first mode, and a result of estimation
of the self-location at T+N that is performed using the
10 image captured in the second mode are respectively
weighted to be integrated with each other for
estimating a self-location at T+N in the self-location
estimation system 200.
[0266] Information regarding the control prediction
15 is transmitted to the vehicle 10. In the vehicle 10, on
the basis of this information, an image is captured by
the camera 300 and, further, a self-location and a
posture of the vehicle 10 are estimated in the selflocation
estimation system 200.
20 [0267] The first mode is an image-capturing mode
when a region other than a region of a shadow of a
building is set to be a photometry region. In this mode,
setting is performed such that the region other than
the region of the shadow of the building appears in an
25 image captured with an adequate exposure although the
region of the shadow of the building in the image
94
exhibits blocked-up shadows.
[0268] The second mode is an image-capturing mode
when the region of the shadow of the building is set to
be the photometry region. In this mode, setting is
performed such that the region of 5 the shadow of the
building appears in the image captured with the
adequate exposure. In the second mode, an exposure
control such as reducing a shutter speed and increasing
a gain is performed.
10 [0269] A pattern of a shutter speed and a gain is
switched between the first mode and the second mode.
[0270] Processing of estimating a self-location and
a posture of the vehicle 10 is performed in the vehicle
10 on the basis of control prediction information, the
15 control prediction information including image
information and information that indicates which of an
image captured in the first mode and an image captured
in the second mode is used upon estimating a selflocation,
or indicates how results of estimating a
20 self-location and a posture in the respective modes are
weighted to be integrated with each other, the image
information being information regarding an image
obtained by performing image-capturing alternately in
the first mode and in the second mode for each frame,
25 the results of estimating a self-location and a posture
in the respective modes being obtained using images
95
captured in the two modes.
[0271] In the case of processing of estimating a
self-location and a posture that is performed on the
basis of control prediction information indicating use
of an image captured in the first 5 mode, an image
captured in the first mode is extracted from the
acquired image information in the self-location
estimation system 200, and the processing of estimating
a self-location and a posture is performed on the basis
10 of the extracted image.
[0272] In the case of processing of estimating a
self-location and a posture that is performed on the
basis of control prediction information indicating use
of an image captured in the second mode, an image
15 captured in the second mode is extracted from the
acquired image information in the self-location
estimation system 200, and the processing of estimating
a self-location and a posture is performed on the basis
of the extracted image.
20 [0273] In the case of processing of estimating a
self-location and a posture that is performed on the
basis of control prediction information indicating use
of a result obtained by assigning weights to results of
estimating a self-location and a posture in the
25 respective modes and integrating the weighted results,
processing of estimating a self-location and a posture
96
is performed for each mode in the self-location
estimation system 200, that is, processing of
estimating a self-location and a posture is performed
using image information regarding an image captured in
the first mode, and processing of 5 estimating a selflocation
and a posture is performed using image
information regarding an image captured in the second
mode. Then, processing of estimating a self-location
and a posture is performed in the self-location
10 estimation system 200 using a result obtained by
assigning weights to processing results in the
respective modes and integrating the weighted
processing results.
[0274] [Control Prediction Processing]
15 Fig. 10 illustrates a flow of control prediction
processing performed in the server apparatus 2500 to
generate control prediction information regarding
prediction of control performed with respect to the
sensing device.
20 [0275] When the control prediction processing is
started, the information acquisition unit 502 acquires,
from the vehicle 10 and through the communication unit
501, parameter information regarding a parameter of the
camera 300, information regarding a location and a
25 posture of the vehicle 10, and vehicle state
information of the vehicle 10 (S21).
97
[0276] Next, the unit 503 for predicting a location
and a posture at T+N predicts a location and a posture
of the vehicle 10 at T+N on the basis of information
regarding a location and a posture of the vehicle 10 at
a time T and vehicle state information 5 of the vehicle
10 at the time T that are output from the information
acquisition unit 502 (S22).
[0277] Next, the unit 2504 for predicting a position
of a shadow of a building in an image predicts a
10 position of a shadow of a building in an image
predicted to be captured by the camera 300 at T+N, on
the basis of the parameter information of the camera
300, information regarding the position and the posture
of the vehicle at T+N, the information 506 regarding a
15 date and time/weather, and the map information 507, and
predicts whether the shadow of the building will appear
in an image (S23).
[0278] When determination performed in S23 is No,
the process returns to S21 to be repeatedly performed.
20 When the determination performed in S23 is Yes,
the process moves on to S24.
[0279] In S24, on the basis of information regarding
a position of a shadow of a building, the control
prediction unit 2505 predicts control performed with
25 respect to the sensing device, and generates control
prediction information (a control prediction signal).
98
Next, the communication unit 501 transmits the
generated control prediction information to the vehicle
10 (S25).
[0280] The server apparatus 2500 includes hardware,
such as a CPU, a ROM, a RAM, and 5 an HDD, that is
necessary for a configuration of a computer.
In the server apparatus 2500, the above-described
control prediction processing to generate control
prediction information regarding prediction of control
10 performed with respect to the sensing device is
performed by loading, into the RAM, a program stored in
the ROM and executing the program.
[0281] As described above, in the control system
2000 according to the present embodiment, control
15 performed with respect to the camera 300 and the selflocation
estimation system 200 that are installed in
the vehicle 10 such that an impact due to the
appearance of a shadow of a structure in an image
captured at T+N is reduced, is predicted on the basis
20 of information regarding a location and a posture of
the vehicle 10 at a time T, the map information 507,
and the information 506 regarding a date and
time/weather.
[0282] In the vehicle 10, control performed with
25 respect to the camera 300 and the self-location
estimation system 200 that are installed in the vehicle
99
10 is performed on the basis of information regarding
this control prediction. Thus, an image that is proper
for a state of the vehicle at T+N and in which an
impact due to the appearance of a shadow of a structure,
a disturbance factor, is reduced, 5 can be obtained at
T+N without a time lag. This results in obtaining a
highly robust control system.
[0283]
Embodiments of the present technology are not
10 limited to the embodiments described above, and various
modifications may be made thereto without departing
from the spirit of the present technology.
[0284] For example, in the embodiments described
above, the example in which control prediction
15 information generated by the server apparatus 500 (1500,
2500) is transmitted to the vehicle 10 in which a
camera acquiring image information is installed, has
been described. However, the control prediction
information may be transmitted to another vehicle other
20 than the vehicle 10. Here, the vehicle 10 is referred
to as the own vehicle 10 in order to distinguish the
own vehicle 10 from the other vehicle.
[0285] The other vehicle, which is a second vehicle,
follows the same route as a route along which the own
25 vehicle 10, which is a first vehicle, travels, and the
other vehicle passes through a certain location at a
100
time T+N', the certain location being the same as a
location through which the vehicle 10 passes at the
time T+N (N'>N), the time T+N' being a time after only
a short period of time from the time T+N.
[0286] Control prediction information 5 regarding
prediction of control performed at T+N with respect to
the vehicle 10 is transmitted to the other vehicle, the
prediction of the control being performed by the server
apparatus 500 (1500, 2500).
10 When the other vehicle reaches the same location
as a location in which the vehicle 10 is predicted to
exist at T+N, processing with respect to the sensing
device that is similar to the processing performed in
the vehicle 10 and described in the respective
15 embodiments above, is performed in the other vehicle,
on the basis of the received control prediction
information.
[0287] The server apparatus 500 (1500, 2500) only
acquires, from the other vehicle, information regarding
20 a location and a posture of the other vehicle. As
control prediction information regarding prediction of
control performed at the time T+N' with respect to the
sensing device of the other vehicle, the server
apparatus 500 (1500, 2500) transmits, to the other
25 vehicle, control prediction information regarding
prediction of control performed at T+N with respect to
101
the own vehicle 10.
[0288] In other words, on the basis of information
regarding a location and a posture of the vehicle 10 at
the time T that is acquired by the information
acquisition unit 502, and on 5 the basis of map
information, the control prediction unit 505 (1505,
2505) sets, to be prediction of control performed with
respect to the sensing device of the other vehicle,
prediction of control performed at T+N with respect to
10 the sensing device of the own vehicle 10, the
prediction of control performed at T+N with respect to
the sensing device of the own vehicle 10 being
predicted by the control prediction unit 505 (1505,
2505).
15 As described above, similar control prediction
information may be shared by a plurality of vehicles.
[0289] Further, in the embodiments described above,
the example in which the control prediction information
includes control prediction information regarding
20 prediction of control performed with respect to the
camera, control prediction information regarding
prediction of control performed in the self-location
estimation system with respect to the sensing device,
and control prediction information regarding prediction
25 of control performed with respect to both the camera
and the self-location estimation system, has been
102
described. However, one of these pieces of control
prediction information may be used, or a combination
thereof may be used.
[0290] Furthermore, in the embodiments described
above, the example of applying the 5 present technology
in order to capture an image used in the self-location
estimation system, has been described, but the
application is not limited to this. For example, the
present technology may be used to perform an exposure
10 control and a photometry control that are performed
with respect to the camera used to capture a video
stored in a dashcam, and this makes it possible to
obtain a video that is less affected by a disturbance
factor.
15 [0291] Moreover, the control systems according to
the respective embodiments described above may be
combined.
[0292] Further, in the embodiments described above,
a system refers to a set of a plurality of structural
20 elements (such as devices and modules (components)),
and whether all of the structural elements are in a
single housing is no object. Thus, a plurality of
devices accommodated in separate housings and connected
to one another through a network, and a single device
25 in which a plurality of modules is accommodated in a
single housing are both systems.
103
[0293] Note that the present technology may also
take the following configurations.
(1) An information processing apparatus including:
an information acquisition unit that acquires
information regarding a location 5 and a posture of a
first mobile object that includes a sensing device; and
a control prediction unit that predicts control
performed with respect to the sensing device, on the
basis of the information regarding the location and the
10 posture and map information, the information regarding
the location and the posture being acquired by the
information acquisition unit.
(2) The information processing apparatus according to
(1), in which
15 the sensing device includes an image-capturing
device,
the information processing apparatus further
includes
a unit for predicting a location and a
20 posture of a mobile object that predicts the location
and the posture of the first mobile object on the basis
of the information regarding the location and the
posture, the information regarding the location and the
posture being acquired by the information acquisition
25 unit, and
a unit for predicting a position of a
104
disturbance factor that predicts a position of a
disturbance factor in an image captured by the imagecapturing
device, on the basis of the map information
and a result of the prediction performed by the unit
for predicting a location and a posture 5 of a mobile
object, and
on the basis of a result of the prediction
performed by the unit for predicting a position of a
disturbance factor, the control prediction unit
10 predicts control performed with respect to the imagecapturing
device.
(3) The information processing apparatus according to
(2), in which
the control prediction unit predicts an exposure
15 control performed with respect to the image-capturing
device.
(4) The information processing apparatus according to
(2) or (3), in which
the control prediction unit predicts a photometry20
region control performed with respect to the imagecapturing
device.
(5) The information processing apparatus according to
any one of (1) to (4), in which
the sensing device includes an image-capturing
25 device,
the first mobile object includes a self-location
105
estimation system that estimates the location and the
posture of the first mobile object using a
characteristic point that is extracted from image
information from the image-capturing device,
the information processing 5 apparatus further
includes
a unit for predicting a location and a
posture of a mobile object that predicts the location
and the posture of the first mobile object on the basis
10 of the information regarding the location and the
posture, the information regarding the location and the
posture being acquired by the information acquisition
unit, and
a unit for predicting a position of a
15 disturbance factor that predicts a position of a
disturbance factor in an image captured by the imagecapturing
device, on the basis of the map information
and a result of the prediction performed by the unit
for predicting a location and a posture of a mobile
20 object, and
on the basis of a result of the prediction
performed by the unit for predicting a position of a
disturbance factor, the control prediction unit
predicts control performed with respect to the self25
location estimation system.
(6) The information processing apparatus according to
106
(5), in which
on the basis of the result of the prediction
performed by the unit for predicting a position of a
disturbance factor, the control prediction unit
predicts a region in which 5 extraction of the
characteristic point in the image is not performed in
the self-location estimation system.
(7) The information processing apparatus according to
(5) or (6), in which
10 the sensing device includes the image-capturing
device and a mobile-object-state detection sensor that
detects a state of the first mobile object,
the first mobile object includes the self-location
estimation system that estimates the location and the
15 posture of the first mobile object using at least one
of the image information, or mobile-object-state
information from the mobile-object-state detection
sensor, and
on the basis of the result of the prediction
20 performed by the unit for predicting a position of a
disturbance factor, the control prediction unit
predicts how the image information and the mobileobject-
state information are respectively weighted, the
image information and the mobile-object-state
25 information being used when the location and the
posture of the first mobile object are estimated in the
107
self-location estimation system.
(8) The information processing apparatus according to
any one of (5) to (7), in which
the sensing device includes a plurality of the
image-capturing 5 devices, and
on the basis of the result of the prediction
performed by the unit for predicting a position of a
disturbance factor, the control prediction unit
predicts how respective pieces of image information
10 from the plurality of the image-capturing devices are
weighted, the respective pieces of image information
being used when the location and the posture of the
first mobile object are estimated in the self-location
estimation system.
15 (9) The information processing apparatus according to
any one of (5) to (8), in which
on the basis of the result of the prediction
performed by the unit for predicting a position of a
disturbance factor, the control prediction unit
20 predicts control performed with respect to the imagecapturing
device.
(10) The information processing apparatus according to
any one of (2) to (9), in which
the disturbance factor is the sun, and
25 on the basis of the map information, the result of
the prediction performed by the unit for predicting a
108
location and a posture of a mobile object, and sun
position information, the unit for predicting a
position of a disturbance factor predicts a position of
the sun in the image captured by the image-capturing
5 device.
(11) The information processing apparatus according to
any one of (2) to (9), in which
the disturbance factor is a tunnel, and
on the basis of the map information and the result
10 of the prediction performed by the unit for predicting
a location and a posture of a mobile object, the unit
for predicting a position of a disturbance factor
predicts a position of the tunnel in the image captured
by the image-capturing device.
15 (12) The information processing apparatus according to
any one of (2) to (9), in which
the disturbance factor is a shadow of a structure,
and
on the basis of the map information, the result of
20 the prediction performed by the unit for predicting a
location and a posture of a mobile object, and sun
position information, the unit for predicting a
position of a disturbance factor predicts a position of
the shadow created due to the structure in the image
25 captured by the image-capturing device.
(13) The information processing apparatus according to
109
any one of (1) to (9), in which
the information processing apparatus according to
claim 1, wherein
the information acquisition unit acquires
information regarding a location 5 and a posture of a
second mobile object that includes a sensing device and
is different from the first mobile object, and
the control prediction unit sets, to be prediction
of control performed with respect to the sensing device
10 of the second mobile object, the prediction of the
control performed with respect to the sensing device of
the first mobile object, the prediction of the control
performed with respect to the sensing device of the
first mobile object being performed by the control
15 prediction unit on the basis of the information
regarding the location and the posture of the first
mobile object and the map information, the information
regarding the location and the posture of the first
mobile object being acquired by the information
20 acquisition unit.
(14) A mobile object including:
a sensing device; and
an acquisition unit that acquires information
regarding a self-location and a posture of the mobile
25 object, in which
the sensing device is controlled according to
110
control prediction information regarding prediction of
control performed with respect to the sensing device,
the control performed with respect to the sensing
device being predicted on the basis of the information
regarding the self-location and the 5 posture and map
information, the information regarding the selflocation
and the posture being acquired by the
acquisition unit.
(15) A control system including:
10 a mobile object that includes a sensing device;
an information acquisition unit that acquires
information regarding a location and a posture of the
mobile object;
a control prediction unit that predicts control
15 performed with respect to the sensing device, on the
basis of the information regarding the location and the
posture and map information, the information regarding
the location and the posture being acquired by the
information acquisition unit; and
20 a control unit that performs the control with
respect to the sensing device on the basis of control
prediction information regarding the control prediction
performed by the control prediction unit.
(16) An information processing method including:
25 acquiring information regarding a location and a
posture of a mobile object that includes a sensing
111
device; and
predicting control performed with respect to the
sensing device, on the basis of the information
regarding the location and the posture and map
5 information.
(17) A program that causes an information processing
apparatus to perform a process including:
acquiring information regarding a location and a
posture of a mobile object that includes a sensing
10 device; and
predicting control performed with respect to the
sensing device, on the basis of the information
regarding the location and the posture and map
information.
15 Reference Signs List
[0294]
1, 1000, 2000 control system
10 vehicle (first mobile object)
102 data acquisition unit (acquisition unit)
20 200 self-location estimation system
201 control unit of self-location estimation system
(control unit)
300 camera (sensing device, image-capturing device)
301 camera control unit (control unit)
25 400 vehicle-state detection sensor (sensing device,
mobile-object-state detection sensor)
112
500, 1500, 2500 server apparatus (information
processing apparatus)
502 information acquisition unit
503 unit for predicting location and posture of
vehicle at T+N (unit for predicting 5 location and
posture of mobile object)
504 unit for predicting position of sun in image (unit
for predicting position of disturbance factor)
505, 1505, 2505 control prediction unit
10 506 information regarding date and time/weather
507 map information
1504 unit for predicting position of tunnel in image
(unit for predicting position of disturbance factor)
2504 unit for predicting position of shadow of building
15 in image (unit for predicting position of disturbance
factor)
113
Claims
[1] An information processing apparatus comprising:
an information acquisition unit that acquires
information regarding a location 5 and a posture of a
first mobile object that includes a sensing device; and
a control prediction unit that predicts control
performed with respect to the sensing device, on a
basis of the information regarding the location and the
10 posture and map information, the information regarding
the location and the posture being acquired by the
information acquisition unit.
[2] The information processing apparatus according to
claim 1, wherein
15 the sensing device includes an image-capturing
device,
the information processing apparatus further
comprises
a unit for predicting a location and a
20 posture of a mobile object that predicts the location
and the posture of the first mobile object on a basis
of the information regarding the location and the
posture, the information regarding the location and the
posture being acquired by the information acquisition
25 unit, and
a unit for predicting a position of a
114
disturbance factor that predicts a position of a
disturbance factor in an image captured by the imagecapturing
device, on a basis of the map information and
a result of the prediction performed by the unit for
predicting a location and a posture of 5 a mobile object,
and
on a basis of a result of the prediction performed
by the unit for predicting a position of a disturbance
factor, the control prediction unit predicts control
10 performed with respect to the image-capturing device.
[3] The information processing apparatus according to
claim 2, wherein
the control prediction unit predicts an exposure
control performed with respect to the image-capturing
15 device.
[4] The information processing apparatus according to
claim 2, wherein
the control prediction unit predicts a photometryregion
control performed with respect to the image20
capturing device.
[5] The information processing apparatus according to
claim 1, wherein
the sensing device includes an image-capturing
device,
25 the first mobile object includes a self-location
estimation system that estimates the location and the
115
posture of the first mobile object using a
characteristic point that is extracted from image
information from the image-capturing device,
the information processing apparatus further
5 comprises
a unit for predicting a location and a
posture of a mobile object that predicts the location
and the posture of the first mobile object on a basis
of the information regarding the location and the
10 posture, the information regarding the location and the
posture being acquired by the information acquisition
unit, and
a unit for predicting a position of a
disturbance factor that predicts a position of a
15 disturbance factor in an image captured by the imagecapturing
device, on a basis of the map information and
a result of the prediction performed by the unit for
predicting a location and a posture of a mobile object,
and
20 on a basis of a result of the prediction performed
by the unit for predicting a position of a disturbance
factor, the control prediction unit predicts control
performed with respect to the self-location estimation
system.
25 [6] The information processing apparatus according to
claim 5, wherein
116
on the basis of the result of the prediction
performed by the unit for predicting a position of a
disturbance factor, the control prediction unit
predicts a region in which extraction of the
characteristic point in the image is 5 not performed in
the self-location estimation system.
[7] The information processing apparatus according to
claim 5, wherein
the sensing device includes the image-capturing
10 device and a mobile-object-state detection sensor that
detects a state of the first mobile object,
the first mobile object includes the self-location
estimation system that estimates the location and the
posture of the first mobile object using at least one
15 of the image information, or mobile-object-state
information from the mobile-object-state detection
sensor, and
on the basis of the result of the prediction
performed by the unit for predicting a position of a
20 disturbance factor, the control prediction unit
predicts how the image information and the mobileobject-
state information are respectively weighted, the
image information and the mobile-object-state
information being used when the location and the
25 posture of the first mobile object are estimated in the
self-location estimation system.
117
[8] The information processing apparatus according to
claim 5, wherein
the sensing device includes a plurality of the
image-capturing devices, and
on the basis of the result 5 of the prediction
performed by the unit for predicting a position of a
disturbance factor, the control prediction unit
predicts how respective pieces of image information
from the plurality of the image-capturing devices are
10 weighted, the respective pieces of image information
being used when the location and the posture of the
first mobile object are estimated in the self-location
estimation system.
[9] The information processing apparatus according to
15 claim 5, wherein
on the basis of the result of the prediction
performed by the unit for predicting a position of a
disturbance factor, the control prediction unit
predicts control performed with respect to the image20
capturing device.
[10] The information processing apparatus according to
claim 2, wherein
the disturbance factor is the sun, and
on a basis of the map information, the result of
25 the prediction performed by the unit for predicting a
location and a posture of a mobile object, and sun
118
position information, the unit for predicting a
position of a disturbance factor predicts a position of
the sun in the image captured by the image-capturing
device.
[11] The information processing apparatus 5 according to
claim 2, wherein
the disturbance factor is a tunnel, and
on the basis of the map information and the result
of the prediction performed by the unit for predicting
10 a location and a posture of a mobile object, the unit
for predicting a position of a disturbance factor
predicts a position of the tunnel in the image captured
by the image-capturing device.
[12] The information processing apparatus according to
15 claim 2, wherein
the disturbance factor is a shadow of a structure,
and
on a basis of the map information, the result of
the prediction performed by the unit for predicting a
20 location and a posture of a mobile object, and sun
position information, the unit for predicting a
position of a disturbance factor predicts a position of
the shadow created due to the structure in the image
captured by the image-capturing device.
25 [13] The information processing apparatus according to
claim 1, wherein
119
the information acquisition unit acquires
information regarding a location and a posture of a
second mobile object that includes a sensing device and
is different from the first mobile object, and
the control prediction unit sets, 5 to be prediction
of control performed with respect to the sensing device
of the second mobile object, the prediction of the
control performed with respect to the sensing device of
the first mobile object, the prediction of the control
10 performed with respect to the sensing device of the
first mobile object being performed by the control
prediction unit on the basis of the information
regarding the location and the posture of the first
mobile object and the map information, the information
15 regarding the location and the posture of the first
mobile object being acquired by the information
acquisition unit.
[14] A mobile object comprising:
a sensing device; and
20 an acquisition unit that acquires information
regarding a self-location and a posture of the mobile
object, wherein
the sensing device is controlled according to
control prediction information regarding prediction of
25 control performed with respect to the sensing device,
the control performed with respect to the sensing
120
device being predicted on a basis of the information
regarding the self-location and the posture and map
information, the information regarding the selflocation
and the posture being acquired by the
5 acquisition unit.
[15] A control system comprising:
a mobile object that includes a sensing device;
an information acquisition unit that acquires
information regarding a location and a posture of the
10 mobile object;
a control prediction unit that predicts control
performed with respect to the sensing device, on a
basis of the information regarding the location and the
posture and map information, the information regarding
15 the location and the posture being acquired by the
information acquisition unit; and
a control unit that performs the control with
respect to the sensing device on a basis of control
prediction information regarding the control prediction
20 performed by the control prediction unit.
[16] An information processing method comprising:
acquiring information regarding a location and a
posture of a mobile object that includes a sensing
device; and
25 predicting control performed with respect to the
sensing device, on a basis of the information regarding
121
the location and the posture and map information.
[17] A program that causes an information processing
apparatus to perform a process comprising:
acquiring information regarding a location and a
posture of a mobile object that 5 includes a sensing
device; and
predicting control performed with respect to the
sensing device, on a basis of the information regarding
the location and the posture and map information.
10