FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENTS RULES, 2003
COMPLETE SPECIFICATION
(See section 10, rule 13)
“INFORMATION PROCESSING DEVICE, MOVEMENT DEVICE,
METHOD, AND PROGRAM”
SONY CORPORATION, of 1-7-1, Konan, Minato-ku, Tokyo
108-0075, Japan;
SONY SEMICONDUCTOR SOLUTIONS
CORPORATION, of 4-14-1 Asahi-cho, Atsugi-shi, Kanagawa
2430014, Japan
The following specification particularly describes the invention and the manner in which it is to
be performed.
1
SP371466WO00
DESCRIPTION
INFORMATION PROCESSING DEVICE, MOVEMENT DEVICE, AND
METHOD, AND PROGRAM
5 TECHNICAL FIELD
[0001]
The present disclosure relates to an information
processing device, a movement device, and a method, and a
program. More specifically, the present disclosure
10 relates to an information processing device, a movement
device, and a method, and a program for calculating a
distance to an object in a leftward or rightward
direction that is orthogonal to a traveling direction of
a movement device such as an automobile, by using a
15 camera captured image.
BACKGROUND ART
[0002]
In recent years, a variety of driving assistance
20 systems for automatic braking, automatic speed control,
obstacle detection, or the like have been developed, and
in the future, it is expected that an automatic driving
vehicle that does not require a driver’s operation or a
vehicle that is mounted with a driving assistance system
25 that reduces a driver’s operation will increase.
For safe traveling of an automobile, distances to a
variety of objects, such as a vehicle, a pedestrian, or a
wall, that hinder movement need to be calculated.
[0003]
30 Examples of a distance measurement device that
calculates an object distance include the devices
2
SP371466WO00
described below.
For example, the following distance measurement
devices are known:
(a) Light detection and ranging or laser imaging
5 detection and ranging (LiDAR) that acquires peripheral
information by using a pulse laser beam;
(b) Radar that detects reflected waves of radio
waves and measures a distance to a reflection object; and
(c) Stereo camera that calculates a distance to an
10 object in a captured image by analyzing corresponding
points in images captured by two cameras.
[0004]
However, all of these distance measurement devices
are expensive.
15 An example of an inexpensive distance measurement
instrument is a distance sensor that uses a laser beam,
infrared light, or the like that has a low output.
However, in this inexpensive distance sensor, a
measurable distance range is limited. For example, a
20 distance of about 10 to 15 m can be measured.
Accordingly, even if such an inexpensive distance
sensor is attached to an automobile, the inexpensive
distance sensor fails to be used to detect a distance to,
for example, a vehicle that is approaching from a distant
25 place at high speed.
[0005]
In order to measure distances in all directions,
forward, backward, leftward, and rightward directions, of
an automobile by using a high-precision distance
30 measurement instrument, as described above as (a) to (c),
at least four distance measurement devices need to be
3
SP371466WO00
respectively attached on front, rear, left-hand, and
right-hand sides, and a cost increases.
Accordingly, in a case where a distance measurement
device is attached to an automobile, a configuration is
5 often employed in which the distance measurement device
is only attached on a front side of the automobile.
[0006]
As a specific example, a distance measurement
device, such as a LiDAR or a stereo camera, is only
10 attached, for example, on a front side of an automobile,
and a relatively low-cost camera is attached in four
positions, front, rear, left-hand, and right-hand
positions, of the automobile. As the camera, for
example, an around view imaging camera using a wide-angle
15 lens, or the like is used.
[0007]
By attaching a camera using a wide-angle lens on
each of the left-hand and right-hand sides of an
automobile, a vehicle or the like that is approaching
20 from a leftward or rightward direction of the automobile
can be captured in a field of view. However, a captured
image of this camera using a wide-angle lens has
distortion in contrast to an image of a monofocal length.
For example, even if a driver views the image, it is
25 difficult for the driver to intuitively grasp a distance
to each subject. Furthermore, there is a problem in
which it is difficult for a driver to intuitively grasp
an accurate distance to each subject from an image, even
if distortion is corrected and the captured image is
30 converted into a central projection image.
[0008]
4
SP371466WO00
Examples of a prior art that discloses a
configuration that detects danger in a direction that is
different from a traveling direction of an automobile
include Patent Document 1 (Japanese Patent Application
5 Laid-Open No. 2017-191471) and Patent Document 2
(Japanese Patent Application Laid-Open No. 2009-067292).
[0009]
Patent Document 1 discloses a blind spot assistance
system in entering a crossing road from a narrow
10 crossroads. However, in this disclosed technology, only
a timing at which an approach situation of an approaching
vehicle can be visually observed is reported.
Patent Document 2 discloses a configuration that
cuts out and displays a video in a specified orientation
15 from an omnidirectional camera.
[0010]
In both Patent Documents 1 and 2 described above,
in a case where there is an automobile or the like that
is approaching from a leftward or rightward direction of
20 an automobile, an image of the approaching automobile or
the like is provided to a driver, and then, the driver
needs to judge a degree of danger on the basis of the
image. In a case where the driver does not have a
sufficient sense of judging the degree of danger based on
25 the image, there is a possibility of causing danger.
Stated another way, Patent Documents 1 and 2 only
disclose a configuration that provides a driver with an
image indicating that a dangerous automobile or the like
is approaching, and distance information of an object,
30 such as an automobile, that is approaching from a
leftward or rightward direction is not provided.
5
SP371466WO00
CITATION LIST
PATENT DOCUMENT
[0011]
5 Patent Document 1: Japanese Patent Application Laid-Open
No. 2017-191471
Patent Document 2: Japanese Patent Application Laid-Open
No. 2009-067292
10 SUMMARY OF THE INVENTION
PROBLEMS TO BE SOLVED BY THE INVENTION
[0012]
The present disclosure has been made in view of,
for example, the problems described above, and it is an
15 object of one example of the present disclosure to
provide an information processing device, a movement
device, and a method, and a program that are capable of
calculating a distance to an object in a leftward or
rightward direction that is orthogonal to or crosses a
20 traveling direction of a movement device such as an
automobile, by only using a camera captured image.
[0013]
Furthermore, it is an object of one example of the
present disclosure to provide an information processing
25 device, a movement device, and a method, and a program
that are capable of calculating a distance to an object
in a leftward or rightward direction that is orthogonal
to a traveling direction of a movement device such as an
automobile, by using an inexpensive distance sensor and a
30 camera captured image.
6
SP371466WO00
SOLUTIONS TO PROBLEMS
[0014]
A first aspect of the present disclosure is an
information processing device including:
5 a data processing unit that calculates an object
distance on the basis of a captured image of a camera
that captures an image in a direction that is orthogonal
to a movement direction of a movement device or in a
direction having an orthogonal component,
10 in which the data processing unit
detects a point at infinity from the captured
image, and
calculates the object distance by using information
relating to a positional relationship between a position
15 of the point at infinity that has been detected and a
position of an object in the captured image.
[0015]
Moreover, a second aspect of the present disclosure
is an information processing device including:
20 a data processing unit that calculates an object
distance on the basis of a captured image of a camera
that captures an image in a direction that is orthogonal
to a movement direction of a movement device or in a
direction having an orthogonal component,
25 in which the data processing unit calculates the
object distance by using the captured image and
information relating to a distance to a reference point P
that is located closer to the camera than the object, and
the data processing unit calculates the object
30 distance according to (Formula 2) described below:
L = Lref × (Wref/W) (Formula 2),
7
SP371466WO00
when the object distance is L,
where
Lref: an actual distance from the camera to the
reference point P,
5 Wref: a width on an image of a reference object
that is located in an image lateral direction of the
reference point P, and
W: a width on the image of the reference object
that is located in the image lateral direction of a
10 distance calculation target object.
[0016]
Moreover, a third aspect of the present disclosure
is an information processing device including:
a data processing unit that calculates an object
15 distance on the basis of a captured image of a camera
that captures an image in a direction that is orthogonal
to a movement direction of a movement device or in a
direction having an orthogonal component,
in which the data processing unit calculates the
20 object distance according to (Formula 3) described below:
L = f × (Wrw/W) (Formula 3),
when the object distance is L,
where
f: a focal length of the camera,
25 Wrw: an actual size of a component having a known
actual size, the component being included in an object
image, and
W: an image size of the component having the known
actual size, the component being included in the object
30 image.
[0017]
8
SP371466WO00
Moreover, a fourth aspect of the present disclosure
is a movement device including:
a camera that captures an image in a direction that
is orthogonal to a movement direction of the movement
5 device or in a direction having an orthogonal component;
and
a data processing unit that calculates an object
distance on the basis of a captured image of the camera,
in which the data processing unit
10 detects a point at infinity from the captured
image, and
calculates the object distance by using information
relating to a positional relationship between a position
of the point at infinity that has been detected and a
15 position of an object in the captured image, and
the data processing unit includes:
a planning unit that determines a route of the
movement device on the basis of the object distance that
has been calculated; and
20 a motion controller that controls a motion of the
movement device according to the route that has been
determined by the planning unit.
[0018]
Moreover, a fifth aspect of the present disclosure
25 is an information processing method performed by an
information processing device,
in which the information processing device includes
a data processing unit that calculates an object distance
on the basis of a captured image of a camera that
30 captures an image in a direction that is orthogonal to a
movement direction of a movement device or in a direction
9
SP371466WO00
having an orthogonal component, and
the data processing unit
detects a point at infinity from the captured
image, and
5 calculates the object distance by using information
relating to a positional relationship between a position
of the point at infinity that has been detected and a
position of an object in the captured image.
[0019]
10 Moreover, a sixth aspect of the present disclosure
is an information processing method performed by a
movement device,
in which the movement device includes:
a camera that captures an image in a direction that
15 is orthogonal to a movement direction of the movement
device or in a direction having an orthogonal component;
and
a data processing unit that calculates an object
distance on the basis of a captured image of the camera,
20 the data processing unit
detects a point at infinity from the captured
image, and
calculates the object distance by using information
relating to a positional relationship between a position
25 of the point at infinity that has been detected and a
position of an object in the captured image,
a planning unit determines a route of the movement
device on the basis of the object distance that has been
calculated, and
30 a motion controller controls a motion of the
movement device according to the route that has been
10
SP371466WO00
determined by the planning unit.
[0020]
Moreover, a seventh aspect of the present
disclosure is a program that causes an information
5 processing device to perform information processing,
in which the information processing device includes
a data processing unit that calculates an object distance
on the basis of a captured image of a camera that
captures an image in a direction that is orthogonal to a
10 movement direction of a movement device or in a direction
having an orthogonal component, and
the program causes the data processing unit to:
detect a point at infinity from the captured image;
and
15 calculate the object distance by using information
relating to a positional relationship between a position
of the point at infinity that has been detected and a
position of an object in the captured image.
[0021]
20 Note that the program according to the present
disclosure is, for example, a program that can be
provided for an information processing device or a
computer system that can execute a variety of program
codes by a storage medium or a communication medium that
25 provides the program in a computer-readable form. By
providing such a program in the computer-readable form,
processing according to the program is performed on the
information processing device or the computer system.
[0022]
30 Yet other objects, features, or advantages of the
present disclosure will be clarified by a more detailed
11
SP371466WO00
description based on the examples described later of the
present disclosure or the attached drawings. Note that a
system described herein is a logical aggregative
configuration of a plurality of devices, and does not
5 always include devices having respective configurations
in the same housing.
EFFECTS OF THE INVENTION
[0023]
10 By employing a configuration in one example of the
present disclosure, a configuration is achieved that
calculates an object distance on the basis of a captured
image of a camera that captures an image in a direction
that is orthogonal to a movement direction of a movement
15 device or in a direction having an orthogonal component.
Specifically, for example, a data processing unit
detects a point at infinity from a captured image of a
camera that captures an image in a direction that is
orthogonal to a movement direction of a movement device
20 or in a direction having an orthogonal component, and
calculates an object direction by using information
relating to a positional relationship between a position
of the detected point at infinity and a position of an
object in the captured image. The data processing unit
25 detects, from the captured image, a plurality of parallel
lines on a real world that extends in a direction away
from a camera position, and determines an intersection
point on extended lines of the detected plurality of
parallel lines to be the point at infinity.
30 Alternatively, an intersection point on respective
extended lines of the straight lines in an image frame
12
SP371466WO00
unit is determined to be the point at infinity,
directions of the straight lines changing on the captured
image in accordance with a movement of the camera.
By employing the present configuration, a
5 configuration is achieved that calculates an object
distance on the basis of a captured image of a camera
that captures an image in a direction that is orthogonal
to a movement direction of a movement device.
Note that effects described herein are only
10 illustrative, and are not restrictive. Furthermore,
additional effects may be exhibited.
BRIEF DESCRIPTION OF DRAWINGS
[0024]
15 Fig. 1 is a diagram explaining an example of a
configuration of a movement device.
Fig. 2 is a diagram explaining an example of
setting of a measurable area of a distance sensor
attached to the movement device and an image capturing
20 area of a camera.
Fig. 3 is a diagram explaining an example of
processing for calculating an object distance.
Fig. 4 is a diagram explaining an example of an
object serving as a distance calculation target.
25 Fig. 5 is a diagram explaining one example of
processing for detecting a point at infinity that is
applied to calculation of an object distance.
Fig. 6 is a diagram explaining one example of
object distance calculation processing.
30 Fig. 7 is a diagram explaining one example of
processing for detecting a point at infinity that is
13
SP371466WO00
applied to calculation of an object distance.
Fig. 8 is a diagram explaining one example of
object distance calculation processing.
Fig. 9 is a diagram explaining one example of
5 object distance calculation processing.
Fig. 10 is a diagram explaining one example of
object distance calculation processing.
Fig. 11 is a diagram explaining one example of
object distance calculation processing.
10 Fig. 12 is a diagram explaining one example of
object distance calculation processing.
Fig. 13 is a diagram explaining one example of
object distance calculation processing.
Fig. 14 is a diagram explaining one example of
15 object distance calculation processing.
Fig. 15 is a diagram explaining one example of
object distance calculation processing.
Fig. 16 is a diagram illustrating a flowchart
explaining a sequence of object distance calculation
20 processing performed by an information processing device.
Fig. 17 is a diagram illustrating a flowchart
explaining a sequence of object distance calculation
processing performed by the information processing
device.
25 Fig. 18 is a diagram explaining one example of a
configuration of a vehicle control system of the movement
device.
Fig. 19 is a diagram explaining an example of a
hardware configuration of the information processing
30 device.
14
SP371466WO00
MODE FOR CARRYING OUT THE INVENTION
[0025]
Details of an information processing device, a
movement device, and a method, and a program according to
5 the present disclosure are described below with reference
to the drawings. Note that description is provided
according to the items described below.
1. Example of configuration of movement device
according to the present disclosure
10 2. Specific examples of point-at-infinity
calculation processing and object distance calculation
processing
2-1. (Processing Example 1) Example of processing
for detecting position of point at infinity by using a
15 plurality of parallel lines included in camera captured
image
2-2. (Processing Example 2) Example of processing
for detecting position of point at infinity by using one
line segment included in camera captured image
20 3. Other examples of object distance calculation
processing
3-1. (Processing Example 1) Example of object
distance calculation processing in case where image
having certain width, such as road, is included in
25 respective positions of distance calculation target
object and reference point object
3-2. (Processing Example 2) Example of object
distance calculation processing in case where distance
calculation target object includes component having known
30 actual size
4. Sequence of processing performed by information
15
SP371466WO00
processing device
5. Example of configuration of movement device
6. Example of configuration of information
processing device
5 7. Summary of configuration according to the
present disclosure
[0026]
[1. Example of configuration of movement device
according to the present disclosure]
10 First, an example of a configuration of a movement
device according to the present disclosure is described
with reference to Fig. 1 and the drawings that follow.
Fig. 1 illustrates an automobile 10 that is one
example of the movement device according to the present
15 disclosure.
Note that, in the example described below, as one
example of the movement device, an example is described
where the movement device is the automobile 10. However,
a configuration or processing according to the present
20 disclosure can be used in a variety of movement devices
other than an automobile.
For example, the configuration or the processing
according to the present disclosure can be applied to a
variety of movement devices such as a robot that travels
25 in a warehouse, an office, or the like.
[0027]
As illustrated in Fig. 1, the automobile 10 is
attached with a plurality of cameras and a plurality of
distance sensors.
30 Note that the distance sensors are not an essential
configuration in some processes according to the present
16
SP371466WO00
disclosure, and some processes according to the present
disclosure can be performed even in a configuration that
does not include any distance sensors.
[0028]
5 The attached cameras are described below.
The following two cameras are attached:
a leftward-direction camera 11L that images a
leftward direction of the automobile 10; and
a rightward-direction camera 11R that images a
10 rightward direction of the automobile 10.
These cameras capture an image in a direction that
is orthogonal to a movement direction of the automobile
10.
Note that, as these cameras 11L and 11R, a camera
15 that captures a normal image or a camera (a monocular
camera) that includes a wide-angle lens such as a fisheye
lens can be used.
[0029]
The automobile 10 is further attached with the
20 following two distance sensors as the distance sensors:
a leftward-direction distance sensor 12L that
measures an object distance in a leftward direction of
the automobile 10; and
a rightward-direction distance sensor 12R that
25 measures an object distance in a rightward direction of
the automobile 10.
Note that, as described above, these distance
sensors are not an essential configuration, and a
configuration that does not include any distance sensors
30 may be employed.
Even in a configuration that includes distance
17
SP371466WO00
sensors, it is sufficient if an inexpensive distance
zensor that uses, for example, a laser beam or infrared
light having a low output is used as each of the distance
sensors. It is sufficient if a distance sensor is used
5 that has, for example, a distance measurement range of
about 10 to 15 m at maximum.
[0030]
Examples of an image capturing range and a distance
measurement range of this automobile 10 that is attached
10 with the cameras 11L and 11R and the distance sensors 12L
and 12R are illustrated in Fig. 2.
Fig. 2 illustrates the following respective areas.
Leftward-direction camera imaging range 21L that is
an imaging area of the leftward-direction camera 11L,
15 Rightward-direction camera imaging range 21R that
is an imaging area of the rightward-direction camera 11R,
Leftward-direction distance sensor distance
measurement range 22L that is a distance measurement
range of the leftward-direction distance sensor 12L, and
20 Rightward-direction distance sensor distance
measurement range 22R that is a distance measurement
range of the rightward-direction distance sensor 12R.
[0031]
The leftward-direction distance sensor distance
25 measurement range 22L and the rightward-direction
distance sensor distance measurement range 22R fall, for
example, within about 10 m from the automobile 10.
[0032]
The cameras 11L and 11R can capture an image of an
30 object (a pedestrian) 31 or an object (a vehicle) 32 that
is illustrated in Fig. 2.
18
SP371466WO00
However, the distance sensors 12L and 12R fail to
directly measure a distance to the object (the
pedestrian) 31 or the object (the vehicle) 32 that is
described above.
5 [0033]
As illustrated in Fig. 2, when the automobile 10
goes out from an intersection of narrow alleys or goes
out from one of the parallel parking frames that are
occupied by vehicles, in some cases, a vehicle that is
10 approaching from a left-hand or right-hand side is
located in a blind spot of a direct field of view of a
driver, and the vehicle that is approaching from the
left-hand or right-hand side fails to be visually
recognized.
15 [0034]
A system has already been proposed that is mounted
with a wide-angle camera or a prism-type camera on front
and rear sides of a vehicle body and presents a camera
captured image to a driver. However, an image that has
20 been cut out from a wide-angle image includes distortion,
and a depth feeling is lost.
Therefore, there is a problem in which the driver
fails to accurately grasp a distance to an approaching
object, such as a vehicle or a person, that is included
25 in the image, even in a situation where the vehicle or
the person is approaching.
[0035]
In order for a driver to perform safe driving, it
is requested that distance information of an object be
30 accurately reported to the driver.
A device (a movement device or an information
19
SP371466WO00
processing device) in one example of the present
disclosure can estimate a distance to an object that is
present in a distant place on a left-hand or right-hand
side of the automobile 10 on the basis of only a captured
5 image of the camera 11L or 11R that images a leftward or
rightward direction of the automobile 10, and can report
an approach risk to a driver.
Furthermore, a device (a movement device or an
information processing device) in one example of the
10 present disclosure can estimate a distance to an object
that is present in a distant place on a left-hand or
right-hand side of the automobile 10 on the basis of a
captured image of the camera 11L or 11R that images a
leftward or rightward direction of the automobile 10, and
15 distance information of a close object (a reference point
object) that has been measured by an inexpensive distance
sensor 12L or 12R that can measure a distance only in a
close area, and can report an approach risk to a driver.
[0036]
20 An example of calculating an object distance is
described with reference to Fig. 3.
Fig. 3 is a diagram explaining an example of
calculating an object distance by using a central
projection image serving as a captured image of a camera.
25 Fig. 3 illustrates an example of processing for
estimating an object distance on the basis of geometric
information of a road or a peripheral environment object.
[0037]
Note that the description herein is provided under
30 the assumption that a real space projection image of a
captured image of a camera is converted into a central
20
SP371466WO00
projection image. In recent years, in many on-vehicle
cameras, a fisheye wide-angle camera has been widely used
due to a need for acquiring image information in a wider
range of a vehicle. In this case, in a scheme of
5 projection of a wide-angle real space to an imaging
plane, a conventional central projection image that is
often said to not include distortion is not obtained.
However, an image that corresponds to central projection
in an orientation specified according to a projection
10 scheme of a fisheye lens that is determined according to
physical optical design can be converted into a captured
image plane according to a virtual central projection
scheme, by converting an image that has been captured by
a fisheye lens into a central projection image in a
15 corresponding orientation that has been determined
according to a projection conversion function. Thus,
even in a case where the fisheye lens is used, processing
that is similar to the processing described herein can be
performed. Therefore, a description that uses a camera
20 system using a projection scheme of a fisheye lens is
omitted herein.
[0038]
Fig. 3 is a diagram explaining an example of
calculating an object distance by using a captured image
25 40 of the rightward-direction camera 11R.
It is assumed that the rightward-direction camera
11R has been installed at a certain height H from a flat
road surface.
The captured image (a central projection image) 40
30 of this camera 11R includes an object (a pedestrian) 31
serving as a distance measurement target.
21
SP371466WO00
[0039]
Note that the captured image (the central
projection image) 40 includes, as a captured image, a
reference point P 41 that corresponds to a close object
5 that is located in a position in which a distance can be
measured by the rightward-direction distance sensor 12R,
and a point at infinity O 42.
[0040]
A distance (a horizontal distance to a position of
10 an entrance pupil of a camera) L from the rightwarddirection
camera 11R to the object (the pedestrian) 31
can be uniquely calculated according to the calculation
formula described below (Formula 1), if an installation
height H and a focal length f of a camera 12 are fixed.
15 [0041]
L = Lref × (href/h) = f × (H/h) (Formula 1)
In this formula,
f: Focal length of camera,
H: Installation height (height from reference plane
20 (= road surface)) of camera,
h: Clearance (clearance in image upward or downward
direction) between object ground contact point (contact
point with reference plane (road surface)) and point at
infinity on captured image,
25 Lref: Distance (actual distance) to referent point
P (on reference plane (road surface)), and
href: Clearance (clearance on image in image upward
or downward direction) between reference point P and
point at infinity on captured image.
30 [0042]
As expressed by (Formula 1) described above, the
22
SP371466WO00
distance L to the object (the pedestrian) 21 can be
expressed to be in inverse proportion to a clearance on a
projected image between a road surface contact point of
the object and a point at horizontal infinity.
5 [0043]
As described above, also for a distant object to
which a distance fails to be measured by a distance
sensor 12, an object distance can be calculated according
to (Formula 1) described above by using an image captured
10 by a camera 11.
However, in order to calculate an object distance
by applying (Formula 1) described above, calculation
processing applying either (Formula 1a) or (Formula 1b)
described below is required.
15 L = Lref × (href/h) (Formula 1a)
L = f × (H/h) (Formula 1b)
[0044]
From among parameters included in (Formula 1)
described above, the following parameters are known:
20 f: Focal length of camera, and
H: Installation height (height from reference plane
(= road surface)) of camera.
[0045]
In this setting, in order to calculate an object
25 distance L by using (Formula 1a) described below:
L = Lref × (href/h) (Formula 1a),
values of the following respective parameters need
to be obtained.
Lref: Distance (actual distance) to referent point
30 P (on reference plane (road surface)),
h: Clearance (clearance on image in image upward or
23
SP371466WO00
downward direction) between object ground contact point
(contact point with reference plane (road surface)) and
point at infinity on captured image, and
href: Clearance (clearance on image in image upward
5 or downward direction) between reference point P and
point at infinity on captured image.
[0046]
In a case where (Formula 1a) described above is
used, first, a close object that is located in a position
10 where a distance can be measured by the rightwarddirection
distance sensor 12R is selected as a reference
point P, and a distance to the reference point P is
measured. Therefore, a distance Lref to the reference
point P is obtained.
15 Moreover, the parameters h and href need to be
obtained. Processing for obtaining these parameters h
and href is described next.
[0047]
Furthermore, in a case where (Formula 1b) described
20 below is used:
L = f × (H/h) (Formula 1b),
values of the following parameters need to be
obtained.
h: Clearance (clearance on image in image upward or
25 downward direction) between object ground contact point
(contact point with reference plane (road surface)) and
point at infinity on captured image
[0048]
Note that, in object distance calculation
30 processing using (Formula 1b) described above, a distance
to the reference point P does not need to be calculated,
24
SP371466WO00
and an automobile that does not include any distance
sensors can also calculate an object distance on the
basis of only a captured image.
[0049]
5 Specific examples of point-at-infinity calculation
processing and object distance calculation processing
that are required in processing for calculating the
parameters h and href are described below.
[0050]
10 [2. Specific examples of point-at-infinity
calculation processing and object distance calculation
processing]
Next, specific examples of point-at-infinity
calculation processing and object distance calculation
15 processing using an image captured by a camera 11 are
described.
[0051]
For example, as illustrated in Fig. 4, in a case
where the automobile 10 is about to go out from a parking
20 lot 50, it is assumed that an object (a vehicle) 32 that
is approaching from a lateral direction is present.
In such a case, a driver fails to visually
recognize the object (the vehicle) 32 at first. Only
after about half of the automobile 10 enters a road, the
25 driver can visually recognize the object (the vehicle)
32.
[0052]
In such a situation, it is requested that the
automobile 10 perform the following processes:
30 1. Move in a travelling direction at low speed; and
2. Check a situation on left-hand and right-hand
25
SP371466WO00
sides, and perform driving according to a checked
situation.
It is captured images of the leftward-direction
camera 11L and the rightward-direction camera 11R of the
5 automobile 10 that enable the situation on the left-hand
and right-hand sides to be checked in an earlier stage.
[0053]
A movement device according to the present
disclosure or an information processing device mounted in
10 the movement device uses captured images of the leftwarddirection
camera 11L and the rightward-direction camera
11R of the automobile 10, and calculates a distance to an
object (an object such as a vehicle or a person) that is
included in the captured images.
15 Alternatively, an object distance is calculated by
using the captured images of the leftward-direction
camera 11L and the rightward-direction camera 11R, and
measured distance information of the distance sensors 12L
and 12R that can measure a distance to an object only in
20 left-hand or right-hand close areas of the automobile 10.
[0054]
Specifically, as described above, the object
distance is calculated according to (Formula 1) described
below.
25 L = Lref × (href/h) = f × (H/h) (Formula 1)
In this formula,
f: Focal length of camera,
H: Installation height (height from reference plane
(= road surface)) of camera,
30 h: Clearance (clearance on image in image upward or
downward direction) between object ground contact point
26
SP371466WO00
(contact point with reference plane (road surface)) and
point at infinity on captured image,
Lref: Distance (actual distance) to referent point
P (on reference plane (road surface)), and
5 href: Clearance (clearance on image in image upward
or downward direction) between reference point P and
point at infinity on captured image.
[0055]
In (Formula 1) described above, the following
10 values are known:
f: Focal length of camera, and
H: Installation height (height from reference plane
(= road surface)) of camera.
Lref: Distance (actual distance) to reference point
15 P (on reference plane (road surface))
This distance to the reference point P can be
obtained by using a distance sensor 12.
[0056]
If the remaining parameters, that is, the
20 parameters described below, can be obtained:
h: Clearance (clearance on image in image upward or
downward direction) between object ground contact point
(contact point with reference plane (road surface)) and
point at infinity on captured image, and
25 href: Clearance (clearance on image in image upward
or downward direction) between reference point P and
point at infinity on captured image,
the object distance L can be calculated.
[0057]
30 A position of an object ground contact point (a
contact point with a reference plane (a road surface)) on
27
SP371466WO00
a captured image or a reference point P on the captured
image can be obtained from the captured image.
Accordingly, in order to calculate the parameters h
and href, it is sufficient if a position of a point at
5 infinity on the captured image is obtained.
[0058]
A plurality of examples of processing for detecting
the position of a point at infinity on a captured image
is described below.
10 The following two processing examples are
sequentially described.
(Processing Example 1) Example of processing for
detecting position of point at infinity by using a
plurality of parallel lines included in camera captured
15 image
(Processing Example 2) Example of processing for
detecting position of point at infinity by using one line
segment included in camera captured image
[0059]
20 [2-1. (Processing Example 1) Example of processing
for detecting position of point at infinity by using a
plurality of parallel lines included in camera captured
image]
First, as Processing Example 1, an example of
25 processing for detecting the position of a point at
infinity by using a plurality of parallel lines included
in a camera captured image is described with reference to
Fig. 5.
[0060]
30 Note that, in the description below of an example,
processing applying an image captured by the leftward28
SP371466WO00
direction camera 11L is described. However, processing
applying an image captured by the rightward-direction
camera 11R is also performed as similar processing.
[0061]
5 First, it is determined whether or not a parallel
line can be detected from a road surface included in a
captured image of the leftward-direction camera 11L.
Note that the parallel line is not a parallel line on an
image, but is a parallel line in the real world and is a
10 parallel line that extends in a direction away from a
camera position.
Specifically, examples include a median strip of a
road, a separation white line, a lane separation white
line, a separation block or a separation white line
15 between a road and a sidewalk, and the like.
In the example illustrated in Fig. 5 of a captured
image 60, four parallel lines, a parallel line a 61a, a
parallel line b 61b, a parallel line c 61c, and a
parallel line d 61d, are detected.
20 [0062]
In a case where a plurality of parallel lines
(parallel lines in the real world) has been detected from
the captured image 60, an estimated intersection point at
infinity on the extensions of the plurality of parallel
25 lines is determined to be a point at infinity O 62.
As described above, the point at infinity O 62 on
the captured image is detected.
[0063]
Note that, in the description of the present
30 disclosure, a point at infinity has been detected under
the assumption that a road surface is always located on
29
SP371466WO00
the same road surface as a road surface on which a
vehicle travels or all of the detected parallel lines are
parallel or quasi-parallel to a vehicle installation
surface.
5 When a parallel line segment caused by a road (such
as a lane, a road shoulder curb, a sidewalk, or a
sidewalk boundary line) has an inclination with respect
to a corresponding vehicle installation flat plane, an
orientation of a point at infinity of a group of parallel
10 lines is determined according to an orientation α of the
inclination. If an optical axis of central projection
matches infinity of a flat plane of interest, in a case
where each inclination α is formed with the same flat
plane, an image that corresponds to an orientation α
15 determined according to a focal length is projected to a
capturing imaging device to shift to a position of
f*tan(α). Therefore, correction of this needs to be
performed on a point at infinity of an inclined parallel
line viewed from the vehicle installation flat plane.
20 [0064]
On the other hand, in a case where the same vehicle
is always traveling on a flat road and as long as a
detected road is maintained on the same flat plane, a
point at infinity of a road parallel line segment in a
25 captured image does not move in horizontal and upward or
downward directions, and is always located on a
horizontal line including the same horizontal infinity.
Then, a position in a lateral orientation of the
horizontal line is determined according to an orientation
30 of the road flat plane. For convenience of description,
the description below is provided under the assumption
30
SP371466WO00
that a camera facing sideways is attached in an
orientation that is perpendicular to a horizontal plane
in a forward or backward movement orientation of a
vehicle and a parallel line segment has an angle β with
5 respect to the optical axis of the camera. Stated
another way, an orientation β of a detected line segment
in the case of measurement by using, as reference, a
right angle with respect to an orientation of
translational traveling of a vehicle hardly moves in a
10 small translational movement of the vehicle.
Furthermore, similarly, no movement is performed if the
world is completely configured by a flat plane. This is
because an object that has a determined orientation and
is located at a point at infinity maintains a certain
15 direction, unless the object itself rotates.
The optical axis of a camera directed in a lateral
orientation does not always need to be perpendicular to a
traveling direction. However, for convenience of
description, in a case where the optical axis is
20 perpendicular to the traveling direction, a group of
lines having a vanishing point in an orientation in which
an amount of shift in a lateral direction from the
optical axis of the camera is f*tan(β) is always line
segments having an inclination of π/2−β with respect to
25 the traveling direction of the vehicle.
[0065]
However, an intersection point within a singleframe
image captured as an image is only a “point at
infinity” that is an optical illusion, and line segments
30 in the real world are not always parallel lines. When
the estimation of a point at infinity is simply applied
31
SP371466WO00
to an intersection point of a plurality of line segments
in a screen that is imaged as a projection video of the
real world space, a visual “point at infinity” that is
only an illusion is treated as a point at infinity, and
5 this causes an illusion error in the calculation of a
distance.
Accordingly, in practice, a detected line is
present on a plane that is not located on a traveling
flat plane of a vehicle, and therefore exclusion
10 processing or calibration processing needs to be
appropriately performed. Moreover, even when an
extracted line segment is located on a road, in some
cases, one boundary line or both boundary lines have been
designed in such a way that a lane is narrowed as a
15 tapered road for convenience of road design, and as a
result, a road is viewed in which a pair of boundary
lines to be detected are partially inclined. When a
distance is calculated under the assumption that an
intersection point of these detected line segments is
20 always a point at infinity, a result including an
illusion error is obtained.
On a flat road, a structural change in a vehicle
body is a change, such as rolling, pitching, or the
sinking of suspension, in the vehicle body due to
25 loading, and if there is no change in the positions or
orientations of a camera and a road surface, an upward or
downward orientation at infinity that is detected by the
camera do not change.
[0066]
30 Furthermore, a horizontal orientation is determined
according to a traveling orientation of a traveling
32
SP371466WO00
vehicle of a corresponding vehicle with respect to a road
surface. Accordingly, normally, an influence of notparallel
lines on an intersection point of detected line
segments only changes when a road itself is curved or has
5 ups and downs or when a state where a camera is attached
to the local car has changed due to an accident, a
failure, or the like of the local car. Accordingly, a
comparison may be made on the basis of a history, and
every time there is a change, the displacement described
10 above may be evaluated in detail. A local dynamic map or
the like may be referred to, information relating to a
flat plane or a curve of a road in a field of view of a
camera may be combined and evaluated. There are a
variety of calibration means, such as inter-frame image
15 analysis based on a movement change or correction using
SLAM described later, in resources and resources allowed
as countermeasures.
Furthermore, in a case where a more inexpensive
system is used, and fails to perform stable and reliable
20 measurement, the system may be only used to call a
driver’s attention by giving a warning to the driver in a
case where there is a risk of a distance estimation
error, without providing a user with accurate distance
conversion information.
25 [0067]
As described above, a data processing unit of the
information processing device according to the present
disclosure detects, from a camera captured image, a
plurality of parallel lines on the real world that
30 extends in a direction away from a camera position, and
determines an intersection point on extended lines of the
33
SP371466WO00
detected plurality of parallel lines to be a point at
infinity. Alternatively, an intersection point on
respective extended lines of the straight lines in an
image frame unit is determined to be the point at
5 infinity, directions of the straight lines changing on
the captured image in accordance with a movement of the
camera.
[0068]
Note that all of the detected parallel line
10 segments on the real space cross each other in an
orientation at infinity inside a field of view or outside
the field of view, excluding a case where the parallel
line segments have been imaged from a vertical direction.
Stated another way, if an orientation that the parallel
15 lines face falls within a range of an imaging angle of
view of imaging in central projection, an intersection
point of corresponding parallel lines is drawn at a
projection point in the orientation within a field of
view. The present disclosure uses a feature in which
20 parallel lines cross each other at a point at infinity,
as a geometric property of projection of central
projection, and uses the feature to calculate a distance
by deriving a point at infinity of drawing of the
parallel lines and using the point at infinity.
25 However, when line segments cross each other on a
projection plane of an imaging plane, line segments on a
projection plane of central projection are not always
parallel lines in the real space. Line segments are
parallel to each other only in the case described later
30 where a single detected line segment has the same center
point of rotation on coordinates with a projection image
34
SP371466WO00
point that is located in an infinity direction as a
center point, in accordance with a translational movement
of an imaging camera. Stated another way, a detected
parallel line segment that has been captured by a camera
5 mounted on a movement device has an orientation at
infinity that does not changes according to movement, and
is fixed at infinity with respect to an orientation that
corresponds to an orientation of the parallel line
segment in a captured image that has been captured in
10 central projection. The parallel line segment rotates
with a point at infinity as a center. If a translational
distance of a vehicle is a close distance of about
several meters, it can be considered that a drawing
position in a central projection image in an orientation
15 of a distant part of a local road is almost constant.
[0069]
Even if a completely translational parallel
movement can be technically performed in a long distance
of several hundred meters or several kilometers, in
20 normal traveling, a point at infinity that is viewed in a
distant place is not tracked in a movement in such a wide
range, and it can be considered that the point at
infinity is not a target to be detected in the present
disclosure. A range to which the present disclosure can
25 be applied is a very short translational movement such as
a vehicle entering a crossroads or the like by a front
part of the vehicle, and the present disclosure is
applied to a range in which it can be considered that a
point at infinity is fixed and constant where parallel
30 segments of a detected peripheral road captured by an onvehicle
camera are converged.
35
SP371466WO00
[0070]
In other words, if a vehicle moves in a traveling
direction, a line segment of a parallel line that is
present on a plane that matches the same corresponding
5 road surface flat plane as a road flat place on which the
vehicle travels follows translation in images of frames
according to traveling, and a vanishing point at infinity
of the line segment maintains a constant position.
Moreover, a part that is close to the line segment moves
10 in a reversing direction according to the traveling of
the vehicle. Therefore, the line segment changes to
rotate around the vanishing point at infinity among the
frames. Details will be described later, but it is
sufficient if these substantial points at infinity are
15 appropriately calibrated, by using a point at which all
of the objects in a screen laterally move in inverse
proportion to a distance in accordance with a lateral
movement in an orientation that crosses an optical axis
of a central projection image. The information
20 processing device according to the present disclosure
place accurately estimates a point at infinity by
analyzing the coordinates of a line segment, as described
above.
[0071]
25 If a point at infinity O 62 on a captured image can
be detected, the parameters described below can be
calculated:
h: Clearance (clearance on image in image upward or
downward direction) between object ground contact point
30 (contact point with reference plane (road surface)) and
point at infinity on captured image; and
36
SP371466WO00
href: Clearance (clearance on image in image upward
or downward direction) between reference point P and
point at infinity on captured image.
[0072]
5 This processing is described with reference to Fig.
6.
Note that, in Fig. 6, it is assumed that an object
serving as a distance calculation target is an object (a
vehicle) 64.
10 As illustrated in Fig. 6,
the parameter h is a clearance (a clearance on an
image in an image upward or downward direction) between a
ground contact point (a contact point with a reference
plane (a road surface)) of the object (the vehicle) 64
15 and a point at infinity O 62 on a captured image, and
the parameter href is a clearance (a clearance on
the image in the image upward or downward direction)
between a reference point P 63 and the point at infinity
O 62 on the captured image.
20 These two clearances are clearances on the image,
and can be directly obtained from the image.
[0073]
As a result of this, an object distance can be
calculated according to (Formula 1) described below that
25 has been described above.
L = Lref × (href/h) = f × (H/h) (Formula 1)
[0074]
Note that, in order to calculate an object distance
by applying (Formula 1) described above, either (Formula
30 1a) or (Formula 1b) described below can be used, as
described above.
37
SP371466WO00
L = Lref × (href/h) (Formula 1a)
L = f × (H/h) (Formula 1b)
[0075]
From among parameters included in the respective
5 formulae described above, the following parameters are
known:
f: Focal length of camera; and
H: Installation height (height from reference plane
(= road surface)) of camera.
10 Note that, here, there is a possibility that H will
change according to a loading amount object or the like
of a vehicle. However, in a vehicle that has started
traveling, a change during correspondence is a temporary
name and a change that are caused by the working
15 conditions of suspension or the like. Therefore, selfcalibration
can be appropriately performed on a center
value of H via a reference measurement means, every time
traveling is started.
Furthermore, in a case where a lens is used that
20 projects and converts the real space into a captured
image plane in a projection scheme of normal central
projection, a focal length f has a fixed value based on
design. However, in performing calculation on the basis
of a virtual central projection image obtained by
25 performing image conversion in a specified orientation on
the basis of an image according to a projection scheme
that is different from central projection, such as
stereographic projection conversion using a fisheye lens,
it is sufficient if a conversion value of a small
30 distance that is assumed in a virtual projection scheme
of this time is substituted.
38
SP371466WO00
[0076]
In this setting, in order to calculate an object
distance L by using (Formula 1a) described below:
L = Lref × (href/h) (Formula 1a),
5 values of the following respective parameters need
to be obtained.
Lref: Distance (actual distance) to reference point
P (on reference plane (road surface))
h: Clearance (clearance on image in image upward or
10 downward direction) between object ground contact point
(contact point with reference plane (road surface)) and
point at infinity on captured image
href: Clearance (clearance on image in image upward
or downward direction) between reference point P and
15 point at infinity on captured image
[0077]
In a case where (Formula 1a) described above is
used, first, a close object that is located in a position
where a distance can be measured by the leftward20
direction distance sensor 12L is selected as a reference
point P, and a distance to the reference point P is
measured. Therefore, a distance Lref to the reference
point P is obtained.
Moreover, the parameters h and href are obtained on
25 the basis of the point at infinity O 62 that has been
detected in the processing described with reference to
Fig. 6.
By using these parameters, the object distance can
be calculated according to (Formula 1a) described below:
30 L = Lref × (href/h) (Formula 1a).
[0078]
39
SP371466WO00
Furthermore, in a case where (Formula 1b) described
below is used:
L = f × (H/h) (Formula 1b),
the parameter h is obtained on the basis of the
5 point at infinity O 62 that has been detected in the
processing described with reference to Fig. 6.
By using this parameter, the object distance can be
calculated according to (Formula 1b) described below:
L = f × (H/h) (Formula 1b).
10 [0079]
Note that, in object distance calculation
processing using (Formula 1b) described above, a distance
to the reference point P does not need to be calculated,
and an automobile that does not include any distance
15 sensors can also calculate an object distance on the
basis of only a captured image.
[0080]
Note that, in order to calculate a point at
infinity on the basis of a camera captured image in the
20 processing described above, it is preferable that the
data processing unit of the information processing device
according to the present disclosure have, for example, a
configuration that have the functions described below.
(Function 1) Function of detecting, from a camera
25 captured image, a plurality of lines on the real world
space that extends in a direction away from a camera
position, and analyzing a change in the coordinates of a
line segment among frames of a plurality of line segments
that has been detected to determine whether or not
30 corresponding line segments are a combination of parallel
lines in the real world space
40
SP371466WO00
(Function 2) Function of detecting, from a camera
captured image, a plurality of lines on the real world
space that extends in a direction away from a camera
position, and determining whether a plurality of line
5 segments that has been detected is parallel line segments
or non-parallel line segments
(Function 3) Function of detecting, from a camera
captured image, a plurality of lines on the real world
space that extends in a direction away from a camera
10 position, and detecting that the coordinates of a
crossing peak of a combination of detected line segments
sequentially move among frames in accordance with a
translational movement of a vehicle
[0081]
15 [2-2. (Processing Example 2) Example of processing
for detecting position of point at infinity by using one
line segment included in camera captured image]
Next, as Processing Example 2, an example of
processing for detecting the position of a point at
20 infinity by using one line segment included in a camera
captured image is described with reference to Fig. 7.
[0082]
In a case where a plurality of parallel lines has
failed to be detected from a captured image of a camera
25 11 of the automobile 10, a change in an individual
detected line segment that occurs according to a slight
translation of the automobile 10 is tracked, a peak of
the center of rotation is searched for, and a center
point of rotation is determined to be an (estimated)
30 point at infinity.
[0083]
41
SP371466WO00
A specific example is described with reference to
Fig. 7.
Image Frame (f(t1)) of Fig. 7(1) is a captured
image of the leftward-direction camera 11L at time t1.
5 Image Frame (f(t2)) of Fig. 7(2) is a captured
image of the leftward-direction camera 11L at time t2
after time t1.
Thereafter, a captured image of the leftwarddirection
camera 11L is captured at predetermined time
10 intervals, and
Image Frame (f(tn)) of Fig. 7(n) is a captured
image of the leftward-direction camera 11L at time tn
that follows.
These n captured images indicate n individual
15 detected lines 71(1), 71(2), ..., 71(n) that each
indicates the same subject.
Note that an individual detected line serving as a
detection target is a straight line that extends in a
direction away from a position (= a camera position) of
20 the automobile 10.
[0084]
According to the traveling of the automobile 10, a
direction of an individual detected line 71 slightly
changes.
25 An upper portion of Fig. 7 illustrates a composite
image 70 for calculating a point at infinity in which the
n individual detected lines 71(1), 71(2), ..., 71(n) are
displayed on the same image.
A point at which the n individual detected lines
30 71(1), 71(2), ..., 71(n) on this composite image are
extended and cross each other is determined to be a point
42
SP371466WO00
at infinity O 62.
[0085]
As described above, in a case where a plurality of
parallel lines fails to be detected from a captured
5 image, a change in an individual detected line segment
that occurs according to a slight translation of the
automobile 10 is tracked, a peak of the center of
rotation is searched for, and a center point of rotation
is determined to be an (estimated) point at infinity O
10 62.
As described above, the information processing
device in the movement device detects, from a captured
image, a straight line on the real world that extends in
a direction away from a camera position, detects an
15 intersection point on respective extended lines of
straight lines in an image frame unit for which a
direction changes on the captured image in accordance
with a movement of a camera, and determines the position
of this intersection point to be a point at infinity.
20 [0086]
If the point at infinity O 62 on the captured image
is detected, the parameters described below can be
obtained:
h: Clearance (clearance on image in image upward or
25 downward direction) between object ground contact point
(contact point with reference plane (road surface)) and
point at infinity on captured image; and
href: Clearance (clearance on image in image upward
or downward direction) between reference point P and
30 point at infinity on captured image.
If these parameters can be obtained, an object
43
SP371466WO00
distance can be calculated according to (Formula 1)
described below that has been described above.
L = Lref × (href/h) = f × (H/h) (Formula 1)
[0087]
5 Note that, in this processing described with
reference to Fig. 7, in a case where the automobile 10
has advanced straight, that is, in a case where the
automobile 10 has traveled in a direction that forms a
right angle with an individual detected line, no
10 particular correction is required. However, in a case
where the automobile 10 does not advance straight, that
is, in a case where the automobile 10 does not travel in
a direction that forms a right angle with an individual
detected line, for example, in a case where the
15 automobile 10 has travels to rotate by Δθ, yaw rotation
is generated by Δθ in each imaging frame. Accordingly,
if a line segment in a captured image faces an infinity
direction of an optical axis, an intersection point at a
point at infinity of the line segment horizontally and
20 laterally shifts by f × tan(Δθ), and therefore this needs
to be corrected.
[0088]
Note that the position of a point at infinity is
almost fixed in continuous captured image frames.
25 Fig. 8 is a diagram illustrating an example of
infinity O 62 that is detected in each of the continuous
captured image frames.
This is the point at infinity O 62 that has been
calculated according to the method described with
30 reference to Fig. 5, 6, or 7.
As illustrated, the infinity O 62 that is detected
44
SP371466WO00
in each of the continuous captured image frames is a
point that is fixed in almost the same position in each
of the images.
[0089]
5 An example of processing in a case where the image
illustrated in Fig. 8 is applied and a distance to a
pedestrian in this image is obtained is described with
reference to Fig. 9.
An image 80 illustrated in Fig. 9 is a captured
10 image of the leftward-direction camera 11L of the
automobile 10, that is, a leftward-direction camera
captured image 80.
This leftward-direction camera captured image 80
indicates a pedestrian that is walking toward the
15 automobile. It is assumed that this pedestrian is an
object 85 serving as a distance calculation target.
[0090]
The leftward-direction camera captured image 80
indicates a close object 82 to which a distance can be
20 measured by the leftward-direction distance sensor 12L of
the automobile 10. This close object 82 is used as a
reference point P. Stated another way, a distance from
the automobile 10 to the close object 82 is calculated by
the leftward-direction distance sensor 12L, and a
25 distance Lref of the reference point P is calculated.
[0091]
Moreover, a point at infinity O 81 is detected
according to the method described above with reference to
Fig. 5, 6, or 7.
30 When the point at infinity O 81 and the close
object (the reference point P) 82 to which a distance can
45
SP371466WO00
be measured are determined, an object distance L to a
pedestrian that is walking toward the automobile 10, that
is, an object distance L of the object (the pedestrian)
85 serving as a distance calculation target, can be
5 calculated according to (Formula 1) described below that
has been described above.
L = Lref × (href/h) = f × (H/h) (Formula 1)
[0092]
A specific example of processing is described with
10 reference to Fig. 10.
Respective parameters in (Formula 1) described
above are described below.
f: Focal length of camera,
H: Installation height (height from reference plane
15 (= road surface)) of camera,
h: Clearance (clearance on image in image upward or
downward direction) between object ground contact point
(contact point with reference plane (road surface)) and
point at infinity on captured image,
20 Lref: Distance (actual distance) to referent point
P (on reference plane (road surface)), and
href: Clearance (clearance on image in image upward
or downward direction) between reference point P and
point at infinity on captured image.
25 [0093]
From among these parameters, the following values
are known:
f: Focal length of camera; and
H: Installation height (height from reference plane
30 (= road surface)) of camera.
The distance described below to a reference point P
46
SP371466WO00
can be obtained by using a distance sensor 12.
Lref: Distance (actual distance) to reference point
P (on reference plane (road surface))
[0094]
5 The remaining parameters, that is, the parameters
described below, are set as illustrated in Fig. 10:
h: Clearance (clearance on image in image upward or
downward direction) between object ground contact point
(contact point with reference plane (road surface)) and
10 point at infinity on captured image; and
href: Clearance (clearance on image in image upward
or downward direction) between reference point P and
point at infinity on captured image.
[0095]
15 As described above, by detecting, from the image
80, the close object (the reference point P) 82 to which
a distance can be measured and the point at infinity O
81, the object distance L to the object (the pedestrian)
85 serving as a distance calculation target can be
20 calculated according to (Formula 1) described below.
L = Lref × (href/h) = f × (H/h) (Formula 1)
[0096]
Note that, as described above, in order to
calculate an object distance by applying (Formula 1)
25 described above, either (Formula 1a) or (Formula 1b)
described below can be used.
L = Lref × (href/h) (Formula 1a)
L = f × (H/h) (Formula 1b)
[0097]
30 From among parameters included in the respective
formulae described above, the following parameters are
47
SP371466WO00
known:
f: Focal length of camera; and
H: Installation height (height from reference plane
(= road surface)) of camera.
5 [0098]
In this setting, in order to calculate an object
distance L by using (Formula 1a) described below:
L = Lref × (href/h) (Formula 1a),
values of the following respective parameters need
10 to be obtained.
Lref: Distance (actual distance) to referent point
P (on reference plane (road surface)),
h: Clearance (clearance on image in image upward or
downward direction) between object ground contact point
15 (contact point with reference plane (road surface)) and
point at infinity on captured image, and
href: Clearance (clearance on image in image upward
or downward direction) between reference point P and
point at infinity on captured image.
20 [0099]
In a case where (Formula 1a) described above is
used, first, the close object 82 that is located in a
position where a distance can be measured by the
leftward-direction distance sensor 12L is selected as the
25 reference point P, and a distance to the reference point
P is measured. Therefore, a distance Lref to the
reference point P is obtained.
Moreover, the parameters h and href are obtained on
the basis of the point at infinity O 81 that has been
30 detected in the processing described with reference to
Fig. 5, 6, or 7.
48
SP371466WO00
By using these parameters, the object distance can
be calculated according to (Formula 1a) described below:
L = Lref × (href/h) (Formula 1a).
[0100]
5 Furthermore, in a case where (Formula 1b) described
below is used:
L = f × (H/h) (Formula 1b),
the parameter h is obtained on the basis of the
point at infinity O 81 that has been detected in the
10 processing described with reference to Fig. 5, 6, or 7.
By using this parameter, the object distance can be
calculated according to (Formula 1b) described below:
L = f × (H/h) (Formula 1b).
[0101]
15 Note that, in object distance calculation
processing using (Formula 1b) described above, a distance
to the reference point P does not need to be calculated,
and an automobile that does not include any distance
sensors can also calculate an object distance on the
20 basis of only a captured image.
[0102]
Note that, as is understood from the description
above, object distance calculation processing is
performed on the basis of a captured image of a camera
25 attached to an automobile, and an object distance at a
timing of capturing each captured image can be
calculated.
Image capturing processing performed by a camera is
performed as processing for capturing a moving image of a
30 predetermined frame rate, and an interval of capturing
each image is a specified time period. For example, in
49
SP371466WO00
the case of 60 fps, 60 frames of image per second are
captured.
[0103]
Accordingly, if an object distance is calculated in
5 each image frame unit, a movement distance of an object
at each frame interval can be calculated. Stated another
way, the movement speed of an object can be calculated.
The information processing device in the automobile
10 can calculate an object distance of each image frame
10 unit, as described above, and can also calculate the
movement speed of an object.
[0104]
[3. Other examples of object distance calculation
processing]
15 Next, other examples of object distance calculation
processing are described with reference to Fig. 11 and
the drawings that follow.
The following processing examples are sequentially
described.
20 (Processing Example 1) Example of object distance
calculation processing in case where image having certain
width, such as road, is included in respective positions
of distance calculation target object and reference point
object
25 (Processing Example 2) Example of object distance
calculation processing in case where distance calculation
target object includes component having known actual size
[0105]
[3-1. (Processing Example 1) Example of object
30 distance calculation processing in case where image
having certain width, such as road, is included in
50
SP371466WO00
respective positions of distance calculation target
object and reference point object]
First, as Processing Example 1, an example of
object distance calculation processing in a case where an
5 image having a certain width, such as a road, is included
in respective positions of a distance calculation target
object and a reference point object is described with
reference to Fig. 11.
[0106]
10 The image illustrated in Fig. 11 is an image that
has been captured by the leftward-direction camera 11L of
the automobile 10.
An object (a vehicle) 91 that is approaching on the
road has been imaged as a subject.
15 A distance to this object (the vehicle) 91 is
calculated.
[0107]
Moreover, the image indicates a close object 92 to
which a distance can be measured by the leftward20
direction distance sensor 12L of the automobile 10.
This close object 92 is used as a reference point
P. Stated another way, a distance from the automobile 10
to the close object 92 is measured by the leftwarddirection
distance sensor 12L and a distance Lref to the
25 reference point P is calculated.
Note that a starting point of an arrow of a line
segment indicating the distance Lref or a distance L is
illustrated as a painted-out circle in the drawing.
However, in a real camera image of central projection, an
30 orientation that is orthogonal to its optical axis is at
infinity, and therefore the orientation is not indicated.
51
SP371466WO00
Stated another way, this is because a camera of central
projection that faces a horizontal direction does not
form, as an image, a road part just below the camera.
The present disclosed drawing is provided to intuitively
5 illustrate a distance from a camera installation position
for convenience. This can be understood from a fact
that, in Fig. 3 in which a projection relation of central
projection is schematically illustrated, when it is
assumed that a lens is located at a starting point of an
10 arrow indicating a distance L, the starting point fails
to be formed as an image on a projection image plane.
[0108]
Moreover, the image includes a road, that is, a
road on which the object (the vehicle) 91 serving as a
15 distance calculation target object is traveling.
This road is also present in the position of the
close object (the reference point P) 92, and it can be
estimated that a road width in the real world is constant
over almost the entirety of this road.
20 Even if the road width is constant, in a situation
where the road is curved or undulates to have ups and
downs with respect to a road flat plane of the vehicle, a
point at infinity of detected parallel lines shifts
according to an orientation of the road, for example, as
25 illustrated in Fig. 12. In arrangement in a screen of an
image formed by a camera of a central projection scheme,
an orientation of an angle β is imaged to shift by a
captured image height f*tan(β) with respect to the center
of the optical axis of the camera. Therefore, a point at
30 infinity changes among sections to correspond to an
orientation of a section in each of the segments that
52
SP371466WO00
forms each parallel component of the road. In the
example illustrated in Fig. 12(1), it is assumed that a
line segment is parallel to the optical axis of a camera
that has been installed to be perpendicular to a vehicle
5 traveling direction within a distance Lturn from the
position of a lens of the camera and the line segment
forms an angle β with the optical axis of the camera in a
position beyond the distance Lturn. Furthermore, Fig.
12(2) illustrates a case where a road surface rises at an
10 inclination α in a position beyond the same distance
Lslope.
[0109]
On the other hand, in a case where boundary
division lines of a road are not parallel, if an attempt
15 is made to perform determination on the basis of a single
frame image, inconvenience is caused. For example, as
illustrated in Fig. 13, in a case where a road width is
tapered or in a case where a boundary line on one side of
a road has been designed in such a way that the road
20 becomes narrow in a forward direction, determination
based on a single frame image causes an optical illusion
of a point at infinity. Stated another way, the
assumption that an intersection point of line segments
that have been imaged and detected is a point at infinity
25 of parallel lines does not hold true.
[0110]
As in these examples illustrated in Figs. 12 and
13, in a case where the width of a road changes, in a
case where the road includes an area that is inclined
30 with respect to a flat plane on which the local vehicle
is traveling, or in another case, this intersection point
53
SP371466WO00
is present at a finite length, and therefore the position
of a point changes in a translational forward movement of
a vehicle. By using this property, a point at infinity
that can be applied to calculate a distance and an
5 “apparent” point at infinity in a single frame can be
distinguished from each other. Among frames of a
captured image of a line segment at a time when a vehicle
moves forward and translationally, a point at infinity of
parallel line segments of a combination of detected line
10 segments has a behavior that is different from a behavior
of an intersection point of line segments drawn as nonparallel
line segments. Here, attention is paid to a
fact that, in a camera image of central projection, in a
case where an orientation with respect to an optical axis
15 in the real space is γ, a drawing position of a point
from the center of an optical axis on a captured screen
is f*tan(γ) from a center point of an optical axis on its
drawing screen, and a vanishing point at infinity of each
of all of the imaged line segments is uniquely determined
20 according to a relationship between each of the imaged
line segments and the optical axis of a camera.
A section distance of each section can be estimated
from the point of a point at infinity of parallel line
segments that are detected in its section, and by
25 performing integration in each section that follows, in
principle, a distance to a more distant place can be
estimated even for a curved parallel road.
[0111]
Note that, in calibrating a point at infinity on a
30 road surface or estimating an object distance on the road
surface, in a situation where a plurality of road surface
54
SP371466WO00
features is detected, road surface feature points on an
imaging screen move in an inverse direction of a
traveling direction in accordance with a translational
movement of a vehicle, as illustrated in Fig. 14, and its
5 movement amount ΔM and a distance L have a relationship
of inverse proportion. By only selectively plotting
feature points on the road surface, a linear function can
be obtained. A distance may be estimated, for example,
according to a relational expression that has been fit
10 according to a least-squares method. However, in the
case of a road-surface three-dimensional structure,
application to calibration fails to be performed in a
position that is higher than the road surface of the
structure.
15 [0112]
However, on an image, the size of a subject
increases in a position that is closer to a camera.
Accordingly, the following two road width sizes on an
image are different from each other:
20 (a) Image size W of a road width in the position of
the object (the vehicle) 91 serving as a distance
calculation target object; and
(b) Image size Wref of a road width in the position
of the close object (the reference point P) 92.
25 [0113]
The size on an image of an object having the same
size in the real world is in inverse proportion to a
distance from a camera to a subject. Stated another way,
as a distance from a camera to a subject increases, a
30 size on an image decreases.
[0114]
55
SP371466WO00
For the close object (the reference point P) 92
illustrated in Fig. 11, a distance from the automobile 10
is measured by the leftward-direction distance sensor
12L. Stated another way, the following is established:
5 Reference point P distance = Lref.
Furthermore, a road width on an image in an image
lateral direction (a horizontal direction) in the
position of this close object (the reference point P) 92
is Wref. This road size Wref can be obtained from the
10 image.
Moreover, a road width on the image in the image
lateral direction (the horizontal direction) in the
position of the object (the vehicle) 91 serving as a
distance calculation target object is W. This road size
15 W can be obtained from the image.
[0115]
Accordingly, a distance L from the position of a
camera (= the position of the automobile 10) to the
object (the vehicle) 91 serving as a distance calculation
20 target object can be calculated according to (Formula 2)
described below.
L = Lref × (Wref/W) (Formula 2)
In this formula,
Lref: Distance (actual distance) to reference point
25 P (on reference plane (road surface)),
Wref: Width (length on image) of object (such as
road) in image lateral direction (horizontal direction)
of reference point P (on reference plane (road surface)),
and
30 W: Width (length on image) of object (such as road)
in image lateral direction (horizontal direction) of
56
SP371466WO00
distance calculation target object (on reference plane
(road surface)).
[0116]
[3-2. (Processing Example 2) Example of object
5 distance calculation processing in case where distance
calculation target object includes component having known
actual size]
Next, as Processing Example 2, an example of object
distance calculation processing in a case where a
10 distance calculation target object includes a component
having a known actual size is described with reference to
Fig. 15.
[0117]
The image illustrated in Fig. 15 is an image that
15 has been captured by the leftward-direction camera 11L of
the automobile 10.
An object (a vehicle) 91 that is approaching on the
road has been imaged as a subject.
A distance to this object (the vehicle) 91 is
20 calculated.
[0118]
The object (the vehicle) 91 serving as a distance
calculation target object includes an image of a license
plate.
25 The size of a license plate of an automobile
conforms to standards, and is the same in all of the
general passenger cars.
Stated another way, the actual size of the license
plate is known.
30 [0119]
It is assumed that this actual size (a width) that
57
SP371466WO00
is a known size of the license plate is Wrw.
Furthermore, it is assumed that an image size (a
width) of the license plate included in a captured image
is W.
5 Moreover, it is assumed that a focal length of a
camera is f. This f is known.
[0120]
In this case, a distance L from the position of a
camera (= the position of the automobile 10) to the
10 object (the vehicle) 91 serving as a distance calculation
target object can be calculated according to (Formula 3)
described below.
L = f × (Wrw/W) (Formula 3)
In this formula,
15 f: Focal length of camera,
Wrw: Actual size of component having known actual
size that is included in image of distance calculation
target object, and
W: Image size of component having known actual size
20 that is included in image of distance calculation target
object.
[0121]
In object distance calculation processing that has
been described with reference to Figs. 11 and 15, the
25 processing described above for detecting the position of
a point at infinity from an image is omitted.
Stated another way, even in a case where a point at
infinity fails to be detected from an image, an object
distance can be calculated.
30 Furthermore, in the processing described with
reference to Fig. 15, a distance to a reference point P
58
SP371466WO00
does not need to be calculated either, and an automobile
that does not include any distance sensors can also
calculate an object distance.
[0122]
5 [4. Sequence of processing performed by information
processing device]
Next, a sequence of processing performed by the
information processing device in the automobile 10 is
described with reference to the flowcharts illustrated in
10 Figs. 16 and 17.
Note that processing according to the flowcharts
illustrated in Figs. 16 and 17 can be performed, for
example, according to a program stored in a storage of
the information processing device.
15 The information processing device includes hardware
having a program execution function, such as a CPU.
The processes of respective steps in the flowcharts
are described below.
[0123]
20 (Step S101)
First, in step S101, it is determined whether or
not a distance calculation target object has been
detected in a camera captured image.
Note that a camera in this case is either the
25 leftward-direction camera 11L or the rightward-direction
camera 11R.
Furthermore, the distance calculation target object
may be any object that can be an obstacle against the
movement of the automobile 10, such as a vehicle, a
30 pedestrian, a cuardrail, or a side wall, or setting may
be performed in such a way that only an object that moves
59
SP371466WO00
is selected.
[0124]
(Step S102)
Next, in step S102, it is determined whether or not
5 a plurality of parallel lines (parallel lines in the real
world) that can be applied to the detection of a point at
infinity has been detected from the camera captured
image.
These parallel lines are lines that extend in a
10 direction away from a camera side. Stated another way,
these parallel lines are lines, such as the parallel
lines a to d or 61a to 61d, that have been described
above with reference to Fig. 5.
[0125]
15 In a case where it has been determined that a
plurality of parallel lines (parallel lines in the real
world) that can be applied to the detection of a point at
infinity has been detected from the camera captured
image, the processing moves on to step S104.
20 In contrast, in a case where it has been determined
that a plurality of parallel lines (parallel lines in the
real world) that can be applied to the detection of a
point at infinity fails to be detected from the camera
captured image, the processing moves on to step S103.
25 [0126]
(Step S103)
In step S102, in a case where it has been
determined that a plurality of parallel lines (parallel
lines in the real world) that can be applied to the
30 detection of a point at infinity fails to be detected
from the camera captured image, the processing moves on
60
SP371466WO00
to step S103.
In step S103, it is determined whether or not a
single line segment that can be applied to the detection
of a point at infinity has been detected from the camera
5 captured image.
This line segment is also a line that extends in a
direction away from a camera side. Stated another way,
this line segment is a line, such as the individual
detected line 71, that has been described above with
10 reference to Fig. 7.
[0127]
In a case where it has been determined that a
single line segment that can be applied to the detection
of a point at infinity has been detected from the camera
15 captured image, the processing moves on to step S104.
In contrast, in a case where it has been determined
that a single line segment that can be applied to the
detection of a point at infinity fails to be detected
from the camera captured image, the processing moves on
20 to step S201.
[0128]
(Step S104)
In step S102, in a case where it has been
determined that a plurality of parallel lines (parallel
25 lines in the real world) that can be applied to the
detection of a point at infinity has been detected from
the camera captured image, or
in step S103, in a case where it has been
determined that a single line segment that can be applied
30 to the detection of a point at infinity has been detected
from the camera captured image, the processing moves on
61
SP371466WO00
to step S104.
[0129]
In step S104, a point at infinity is detected from
the camera captured image.
5 In step S102, in a case where a plurality of
parallel lines (parallel lines in the real world) that
can be applied to the detection of a point at infinity
has been detected from the camera captured image, the
plurality of parallel lines is extended, and an
10 intersection point of the extended lines is detected, as
described above with reference to Fig. 5. This
intersection point is determined to be a point at
infinity.
[0130]
15 Furthermore, in step S103, in a case where a single
line segment that can be applied to the detection of a
point at infinity has been detected from the camera
captured image, a plurality of lines that is output in a
case where line segments that are included in a plurality
20 of captured image frames during a predetermined period
have been output on a single image are extended, and an
intersection point of the extended lines is detected, as
described above with reference to Fig. 7. This
intersection point is determined to be a point at
25 infinity.
[0131]
(Step S105)
In step S104, when the detection of a point at
infinity has been finished, the processing moves on to
30 step S105.
In step S105, an object distance is calculated
62
SP371466WO00
according to (Formula 1) described below.
L = Lref × (href/h) = f × (H/h) (Formula 1)
In this formula,
f: Focal length of camera,
5 H: Installation height (height from reference plane
(= road surface)) of camera,
h: Clearance (clearance on image in image upward or
downward direction) between object ground contact point
(contact point with reference plane (road surface)) and
10 point at infinity on captured image,
Lref: Distance (actual distance) to referent point
P (on reference plane (road surface)), and
href: Clearance (clearance on image in image upward
or downward direction) between reference point P and
15 point at infinity on captured image.
[0132]
Note that, in order to calculate an object distance
by applying (Formula 1) described above, either (Formula
1a) or (Formula 1b) described below can be used.
20 L = Lref × (href/h) (Formula 1a)
L = f × (H/h) (Formula 1b)
[0133]
From among parameters included in the respective
formulae described above, the following parameters are
25 known:
f: Focal length of camera; and
H: Installation height (height from reference plane
(= road surface)) of camera.
[0134]
30 In this setting, in a case where an object distance
L is calculated by using (Formula 1a) described below:
63
SP371466WO00
L = Lref × (href/h) (Formula 1a),
the parameters described below need to be obtained:
Lref: Distance (actual distance) to referent point
P (on reference plane (road surface));
5 h: Clearance (clearance on image in image upward or
downward direction) between object ground contact point
(contact point with reference plane (road surface)) and
point at infinity on captured image; and
href: Clearance (clearance on image in image upward
10 or downward direction) between reference point P and
point at infinity on captured image.
[0135]
In a case where (Formula 1a) described above is
used, first, a close object that is located in a position
15 where a distance can be measured by a distance sensor 12
is selected as a reference point P, a distance to the
reference point P is measured, and a distance Lref to the
reference point P is obtained.
Moreover, the parameters h and href are obtained on
20 the basis of the point at infinity that has been detected
in step S104.
By using these parameters, the object distance can
be calculated according to (Formula 1a) described below:
L = Lref × (href/h) (Formula 1a).
25 [0136]
Furthermore, in a case where (Formula 1b) described
below is used:
L = f × (H/h) (Formula 1b),
the parameter h is obtained on the basis of the
30 point at infinity that has been detected in step S104.
By using this parameter, the object distance can be
64
SP371466WO00
calculated according to (Formula 1b) described below:
L = f × (H/h) (Formula 1b).
[0137]
Note that, in object distance calculation
5 processing using (Formula 1b) described above, a distance
to the reference point P does not need to be calculated,
and an automobile that does not include any distance
sensors can also calculate an object distance on the
basis of only a captured image.
10 [0138]
(Step S201)
In step S102, in a case where it has been
determined that a plurality of parallel lines (parallel
lines in the real world) that can be applied to the
15 detection of a point at infinity fails to be detected
from the camera captured image, and
in step S103, in a case where it has been
determined that a single line segment that can be applied
to the detection of a point at infinity fails to be
20 detected from the camera captured image, the processing
moves on to step S201.
[0139]
In step S201, it is determined whether or not a
close object that can be a reference point and to which a
25 distance can be calculated has been detected from the
image.
In a case where the close object has been detected,
the processing moves on to step S202.
In a case where the close object fails to be
30 detected, the processing moves on to step S211.
[0140]
65
SP371466WO00
(Step S202)
In step S201, in a case where it has been
determined that a close object that can be a reference
point and to which a distance can be calculated has been
5 detected from the image, the processing moves on to step
S202.
In step S202, the close object is determined to be
a reference point P, and a distance to the reference
point P is calculated.
10 The distance is calculated by either of the
distance sensors 12L and 12R.
[0141]
(Step S203)
Next, an object distance L is calculated according
15 to (Formula 2) described below.
L = Lref × (Wref/W) (Formula 2)
In this formula,
Lref: Distance (actual distance) to reference point
P (on reference plane (road surface)),
20 Wref: Width (length on image) of object (such as
road) in image lateral direction (horizontal direction)
of reference point P (on reference plane (road surface)),
and
W: Width (length on image) of object (such as road)
25 in image lateral direction (horizontal direction) of
distance calculation target object (on reference plane
(road surface)).
[0142]
Object distance calculation processing using
30 (Formula 2) described above corresponds to the processing
described above with reference to Fig. 11.
66
SP371466WO00
[0143]
(Step S211)
In contrast, in step S201, in a case where it has
been determined that a close object that can be a
5 reference point and to which a distance can be calculated
fails to be detected from the image, the processing moves
on to step S211.
In step S211, it is determined whether or not a
component having a known actual size, such as a license
10 plate, is included an image of the distance calculation
target object.
[0144]
In a case where a component having a known actual
size is included, the processing moves on to step S212.
15 In a case where a component having a known actual
size is not included, the processing is terminated.
[0145]
(Step S212)
In step S211, in a case where it has been
20 determined that a component having a known actual size,
such as a license plate, is included an image of the
distance calculation target object, the processing moves
on to step S212.
[0146]
25 In step S212, an object distance L is calculated
according to (Formula 3) described below.
L = f × (Wrw/W) (Formula 3)
In this formula,
f: Focal length of camera,
30 Wrw: Actual size of component having known actual
size that is included in image of distance calculation
67
SP371466WO00
target object, and
W: Image size of component having known actual size
that is included in image of distance calculation target
object.
5 [0147]
Object distance calculation processing using
(Formula 3) described above corresponds to the processing
described above with reference to Fig. 15.
[0148]
10 [5. Example of configuration of movement device]
Next, an example of the configuration of a movement
device, such as the automobile 10, that performs
processing according to the present disclosure is
described with reference to Fig. 16.
15 Fig. 16 is a block diagram illustrating a schematic
functional configuration example of a vehicle control
system 100 that is one example of a control system of the
movement device, such as the automobile 10, that performs
the processing described above.
20 [0149]
Note that, hereinafter, in a case where a vehicle
that is provided with the vehicle control system 100 is
distinguished from another vehicle, the vehicle that is
provided with the vehicle control system 100 is referred
25 to as the local car or the local vehicle.
[0150]
The vehicle control system 100 includes an input
unit 101, a data acquisition unit 102, a communication
unit 103, an in-vehicle device 104, an output controller
30 105, an output unit 106, a drive system controller 107, a
drive system 108, a body system controller 109, a body
68
SP371466WO00
system 110, a storage 111, and an automatic driving
controller 112. The input unit 101, the data acquisition
unit 102, the communication unit 103, the output
controller 105, the drive system controller 107, the body
5 system controller 109, the storage 111, and the automatic
driving controller 112 are mutually connected via a
communication network 121. The communication network 121
includes, for example, an on-vehicle communication
network conforming to an arbitrary standard, such as a
10 controller area network (CAN), a local interconnect
network (LIN), a local area network (LAN), or FlexRay
(registered trademark), a bus, or the like. Note that,
in some cases, respective units of the vehicle control
system 100 are directly connected without the
15 communication network 121.
[0151]
Note that, hereinafter, in a case where the
respective units of the vehicle control system 100
perform communication via the communication network 121,
20 the description of the communication network 121 is
omitted. For example, in a case where the input unit 101
and the automatic driving controller 112 perform
communication via the communication network 121, it is
simply described that the input unit 101 and the
25 automatic driving controller 112 perform communication.
[0152]
The input unit 101 includes a device that a
passenger uses to input various types of data,
instructions, or the like. For example, the input unit
30 101 includes an operation device such as a touch panel, a
button, a microphone, a switch, or a lever, an operation
69
SP371466WO00
device on which an input operation can be performed by
using a method other than a manual operation, such as
sound or a gesture, and the like. Furthermore, for
example, the input unit 101 may be a remote control
5 device that uses infrared rays or other radio waves, or
an external connection device, such as a mobile device or
a wearable device, that corresponds to an operation of
the vehicle control system 100. The input unit 101
generates an input signal on the basis of data, an
10 instruction, or the like that has been input by a
passenger, and supplies the input signal to the
respective units of the vehicle control system 100.
[0153]
The data acquisition unit 102 includes a variety of
15 sensors or the like that acquire data to be used in
processing performed by the vehicle control system 100,
and supplies the acquired data to the respective units of
the vehicle control system 100.
[0154]
20 For example, the data acquisition unit 102 includes
a variety of sensors that detect a state of the local
car, or the like. Specifically, for example, the data
acquisition unit 102 includes a gyro sensor, an
acceleration sensor, an inertial measurement unit (IMU),
25 a sensor that detects an amount of an operation performed
on an accelerator pedal, an amount of an operation
performed on a brake pedal, a steering angle of a
steering wheel, engine speed, the rotational speed of a
motor, the rotational speed of a wheel, or the like, and
30 other sensors.
[0155]
70
SP371466WO00
Furthermore, for example, the data acquisition unit
102 includes a variety of sensors that detect information
relating to the outside of the local car. Specifically,
for example, the data acquisition unit 102 includes an
5 imaging device such as a time-of-flight (ToF) camera, a
stereo camera, a monocular camera, an infrared camera, or
another camera. Furthermore, for example, the data
acquisition unit 102 includes an environment sensor that
detects weather, meteorological phenomena, or the like,
10 and a peripheral information detection sensor that
detects an object around the local car. The environment
sensor includes, for example, a raindrop sensor, a fog
sensor, a sunshine sensor, a snow sensor, or the like.
The peripheral information detection sensor includes, for
15 example, an ultrasonic sensor, a radar, a light detection
and ranging or laser imaging detection and ranging
(LiDAR), a sonar, or the like.
[0156]
Moreover, for example, the data acquisition unit
20 102 includes a variety of sensors that detect a current
position of the local car. Specifically, for example,
the data acquisition unit 102 includes a global
navigation satellite system (GNSS) receiver that receives
a GNSS signal from a GNSS satellite, or the like.
25 [0157]
Furthermore, for example, the data acquisition unit
102 includes a variety of sensors that detect in-vehicle
information. Specifically, for example, the data
acquisition unit 102 includes an imaging device that
30 images a driver, a biosensor that detects biological
information of the driver, a microphone that collects
71
SP371466WO00
sound in a vehicle cabin, and the like. The biosensor is
provided, for example, on a seat surface, a steering
wheel, or the like, and detects biological information
relating to a passenger who is seated on a seat or a
5 driver who is holding the steering wheel.
[0158]
The communication unit 103 performs communication
with the in-vehicle device 104, and a variety of outsidevehicle
devices, a server, a base station, and the like,
10 and transmits data supplied from the respective units of
the vehicle control system 100 or supplies received data
to the respective units of the vehicle control system
100. Note that a communication protocol supported by the
communication unit 103 is not particularly limited, and
15 the communication unit 103 can also support plural types
of communication protocols.
[0159]
For example, the communication unit 103 performs
wireless communication with the in-vehicle device 104 by
20 using a wireless LAN, Bluetooth (registered trademark),
near field communication (NFC), a wireless USB (WUSB), or
the like. Furthermore, for example, the communication
unit 103 performs wired communication with the in-vehicle
device 104 via a not-illustrated connection terminal (and
25 a cable if necessary), by using a universal serial bus
(USB), a high-definition multimedia interface (HDMI)
(registered trademark), a mobile high-definition link
(MHL), or the like.
[0160]
30 Moreover, for example, the communication unit 103
performs communication with equipment (for example, an
72
SP371466WO00
application server or a control server) that is present
on an external network (for example, the Internet, a
cloud network, or a company-specific network) via the
base station or an access point. Furthermore, for
5 example, the communication unit 103 performs
communication with a terminal that is present near the
local car (for example, a terminal of a pedestrian or a
store, or a machine type communication (MTC) terminal) by
using a peer to peer (P2P) technology. Moreover, for
10 example, the communication unit 103 performs V2X
communication such as vehicle to vehicle communication,
vehicle to infrastructure communication, vehicle to home
communication, or vehicle to pedestrian communication.
Furthermore, for example, the communication unit 103
15 includes a beacon receiver, receives radio waves or
electromagnetic waves that have been transmitted from a
wireless station that is provided on a road, or the like,
and acquires information relating to a current position,
a traffic jam, traffic regulations, a required time, or
20 the like.
[0161]
The in-vehicle device 104 includes, for example, a
mobile device or a wearable device that is possessed by a
passenger, an information device that is carried in or
25 attached to the local car, a navigation device that
searches for a route to an arbitrary destination, and the
like.
[0162]
The output controller 105 controls an output of
30 various types of information to a passenger of the local
car or the outside of the vehicle. For example, the
73
SP371466WO00
output controller 105 controls an output of visual
information (for example, image data) and auditory
information (for example, sound data) from the output
unit 106 by generating an output signal including at
5 least one of the visual information or the auditory
information and supplying the output signal to the output
unit 106. Specifically, for example, the output
controller 105 combines pieces of image data that have
been captured by imaging devices that are different from
10 each other in the data acquisition unit 102, generates an
overhead image, a panoramic image, or the like, and
supplies an output signal including the generated image
to the output unit 106. Furthermore, for example, the
output controller 105 generates sound data including
15 warning sound, a warning message, or the like against
danger such as collision, contact, or entry into a danger
zone, and supplies, to the output unit 106, an output
signal including the generated sound data.
[0163]
20 The output unit 106 includes a device that can
output the visual information or the auditory information
to a passenger of the local car or the outside of the
vehicle. For example, the output unit 106 includes a
display device, an instrument panel, an audio speaker, a
25 headphone, a wearable device, such as an eyeglasses type
display, that a passenger wears, a projector, a lamp, or
the like. The display device included in the output unit
106 may be a device including a normal display, or may
be, for example, a device that displays auditory
30 information in a field of view of a driver, such as a
head-up display, a transmission type display, or a device
74
SP371466WO00
having an augmented reality (AR) display function.
[0164]
The drive system controller 107 controls the drive
system 108 by generating various control signals and
5 supplying the various control signals to the drive system
108. Furthermore, the drive system controller 107
supplies a control signal to respective units other than
the drive system 108, as needed, and gives notice or the
like of a control state of the drive system 108.
10 [0165]
The drive system 108 includes a variety of devices
that relate to a drive system of the local car. For
example, the drive system 108 includes a drive force
generation device that generates a drive force, such as
15 an internal combustion engine or a drive motor, a drive
force transmission mechanism that transmits a drive force
to wheels, a steering mechanism that adjusts a steering
angle, a braking device that generates a braking force,
an antilock brake system (ABS), an electronic stability
20 control (ESC), an electric power steering device, and the
like.
[0166]
The body system controller 109 controls the body
system 110 by generating various control signals and
25 supplying the various control signals to the body system
110. Furthermore, the body system controller 109
supplies a control signal to respective units other than
the body system 110, as needed, and gives notice or the
like of a control state of the body system 110.
30 [0167]
The body system 110 includes a variety of devices
75
SP371466WO00
of a body system equipped in a vehicle body. For
example, the body system 110 includes a keyless entry
system, a smart key system, a power window device, a
power seat, a steering wheel, an air conditioner, a
5 variety of lamps (for example, a headlamp, a back lamp, a
brake lamp, a turn signal, a fog lamp, and the like), or
the like.
[0168]
The storage 111 includes, for example, a read only
10 memory (ROM), a random access memory (RAM), a magnetic
storage device such as a hard disc drive (HDD), a
semiconductor storage device, an optical storage device,
a magneto-optical storage device, and the like. The
storage 111 stores various programs, data, or the like
15 that are used by the respective units of the vehicle
control system 100. For example, the storage 111 stores
map data such as a three-dimensional high-precision map,
e.g., a dynamic map, a global map that has a lower
precision and covers a wider area than the high-precision
20 map, or a local map including information relating to the
surroundings of the local car.
[0169]
The automatic driving controller 112 performs
control relating to automatic driving such as autonomous
25 traveling or driving support. Specifically, for example,
the automatic driving controller 112 performs cooperative
control aiming at implementing a function of an advanced
driver assistance system (ADAS) including collision
avoidance or impact mitigation of the local car, follow30
up traveling based on a distance between vehicles,
vehicle speed maintaining traveling, a warning against
76
SP371466WO00
the collision the local car, a warning against the lane
departure of the local car, or the like. Furthermore,
for example, the automatic driving controller 112
performs cooperative control aiming at automatic driving
5 or the like for autonomous traveling that is independent
of an operation performed by a driver. The automatic
driving controller 112 includes a detection unit 131, a
self-position estimation unit 132, a situation analyzer
133, a planning unit 134, and a motion controller 135.
10 [0170]
The detection unit 131 detects various types of
information required to control automatic driving. The
detection unit 131 includes an outside-vehicle
information detection unit 141, an in-vehicle information
15 detection unit 142, and a vehicle state detection unit
143.
[0171]
The outside-vehicle information detection unit 141
performs processing for detecting information relating to
20 the outside of the local car on the basis of data or a
signal from each of the units of the vehicle control
system 100. For example, the outside-vehicle information
detection unit 141 performs processing for detecting,
recognizing, and tracking an object around the local car,
25 and processing for detecting a distance to the object.
Examples of an object to be detected include a vehicle, a
person, an obstacle, a structure, a road, a traffic
light, a traffic sign, a road sign, and the like.
Furthermore, for example, the outside-vehicle information
30 detection unit 141 performs processing for detecting a
surrounding environment of the local car. Examples of a
77
SP371466WO00
surrounding environment to be detected include weather,
air temperature, humidity, brightness, a road surface
state, and the like. The outside-vehicle information
detection unit 141 supplies data indicating a result of
5 detection processing to the self-position estimation unit
132, a map analyzer 151, a traffic rule recognizer 152,
and a situation recognizer 153 of the situation analyzer
133, an emergency avoidance unit 171 of the motion
controller 135, and the like.
10 [0172]
The in-vehicle information detection unit 142
performs processing for detecting in-vehicle information
on the basis of data or a signal from each of the units
of the vehicle control system 100. For example, the in15
vehicle information detection unit 142 performs
processing for authenticating and recognizing a driver,
processing for detecting the driver’s state, processing
for detecting a passenger, processing for detecting an
in-vehicle environment, and the like. Examples of a
20 driver’s state to be detected include a physical
condition, a degree of awakening, a degree of
concentration, a degree of fatigue, a direction of a
line-of-sight, and the like. Examples of an in-vehicle
environment to be detected include air temperature,
25 humidity, brightness, an odor, and the like. The invehicle
information detection unit 142 supplies data
indicating a result of detection processing to the
situation recognizer 153 of the situation analyzer 133,
the emergency avoidance unit 171 of the motion controller
30 135, and the like.
[0173]
78
SP371466WO00
The vehicle state detection unit 143 performs
processing for detecting a state of the local car on the
basis of data or a signal from each of the units of the
vehicle control system 100. Examples of a state of the
5 local car that serves as a target to be detected include
speed, acceleration, a steering angle, the presence or
absence and content of abnormality, a state of a driving
operation, a position and an inclination of a power seat,
a state of door lock, a state of another on-vehicle
10 device, and the like. The vehicle state detection unit
143 supplies data indicating a result of detection
processing to the situation recognizer 153 of the
situation analyzer 133, the emergency avoidance unit 171
of the motion controller 135, and the like.
15 [0174]
The self-position estimation unit 132 performs
processing for estimating a position, an orientation, and
the like of the local car on the basis of data or a
signal from each of the units of the vehicle control
20 system 100, e.g., the outside-vehicle information
detection unit 141, the situation recognizer 153 of the
situation analyzer 133, and the like. Furthermore, the
self-position estimation unit 132 generates a local map
used to estimate a self-position (hereinafter referred to
25 as a self-position estimation map), as needed. It is
assumed, for example, that the self-position estimation
map is a high-precision map using a technology such as
simultaneous localization and mapping (SLAM). The selfposition
estimation unit 132 supplies data indicating a
30 result of estimation processing to the map analyzer 151,
the traffic rule recognizer 152, and the situation
79
SP371466WO00
recognizer 153 of the situation analyzer 133, and the
like. Furthermore, the self-position estimation unit 132
stores the self-position estimation map in the storage
111.
5 [0175]
The situation analyzer 133 performs processing for
analyzing situations of the local car and the
surroundings. The situation analyzer 133 includes the
map analyzer 151, the traffic rule recognizer 152, the
10 situation recognizer 153, and the situation prediction
unit 154.
[0176]
The map analyzer 151 performs processing for
analyzing various maps stored in the storage 111 by using
15 data or a signal from each of the units of the vehicle
control system 100, e.g., the self-position estimation
unit 132, the outside-vehicle information detection unit
141, and the like, as needed, and constructs a map
including information required for automatic driving
20 processing. The map analyzer 151 supplies the
constructed map to the traffic rule recognizer 152, the
situation recognizer 153, the situation prediction unit
154, a route planning unit 161, an action planning unit
162, and a motion planning unit 163 of the planning unit
25 134, and the like.
[0177]
The traffic rule recognizer 152 performs processing
for recognizing a traffic rule in the surroundings of the
local car on the basis of data or a signal from each of
30 the units of the vehicle control system 100, e.g., the
self-position estimation unit 132, the outside-vehicle
80
SP371466WO00
information detection unit 141, the map analyzer 151, and
the like. By performing this recognition processing, for
example, a position and a state of a traffic light around
the local car, the content of traffic regulations around
5 the local car, a travelable traffic lane, and the like
are recognized. The traffic rule recognizer 152 supplies
data indicating a result of recognition processing to the
situation prediction unit 154 or the like.
[0178]
10 The situation recognizer 153 performs processing
for recognizing a situation relating to the local car on
the basis of data or a signal from each of the units of
the vehicle control system 100, e.g., the self-position
estimation unit 132, the outside-vehicle information
15 detection unit 141, the in-vehicle information detection
unit 142, the vehicle state detection unit 143, the map
analyzer 151, and the like. For example, the situation
recognizer 153 performs processing for recognizing a
situation of the local car, a situation of the
20 surroundings of the local car, a situation of a driver of
the local car, and the like. Furthermore, the situation
recognizer 153 generates a local map used to recognize
the situation of the surroundings of the local car
(hereinafter referred to as a situation recognition map),
25 as needed. It is assumed, for example, that the
situation recognition map is an occupancy grid map.
[0179]
Examples of a situation of the local car that
serves as a target to be recognized include a position,
30 an orientation, a movement (for example, speed,
acceleration, a movement direction, or the like) of the
81
SP371466WO00
local car, the presence or absence and content of an
abnormality, and the like. Examples of a situation of
the surroundings of the local car that serves as a target
to be recognized include a type and a position of a
5 surrounding static object, a type, a position, and a
movement (for example, speed, acceleration, a movement
direction, and the like) of a surrounding moving object,
a configuration and a road surface state of a surrounding
road, weather, air temperature, humidity, and brightness
10 of the surroundings, and the like. Examples of a
driver’s state serving as a target to be recognized
include a physical condition, a degree of awakening, a
degree of concentration, a degree of fatigue, a movement
of a line-of-sight, a driving operation, and the like.
15 [0180]
The situation recognizer 153 supplies data
indicating a result of recognition processing (including
the situation recognition map, as needed) to the selfposition
estimation unit 132, the situation prediction
20 unit 154, and the like. Furthermore, the situation
recognizer 153 stores the situation recognition map in
the storage 111.
[0181]
The situation prediction unit 154 performs
25 processing for predicting a situation relating to the
local car on the basis of data or a signal from each of
the units of the vehicle control system 100, e.g., the
map analyzer 151, the traffic rule recognizer 152, the
situation recognizer 153, and the like. For example, the
30 situation prediction unit 154 performs processing for
predicting a situation of the local car, a situation of
82
SP371466WO00
the surroundings of the local car, a situation of a
driver, and the like.
[0182]
Examples of a situation of the local car that
5 serves as a target to be predicted include a behavior of
the local car, the occurrence of an abnormality, a
travelable distance, and the like. Examples of a
situation of the surroundings of the local car that
serves as a target to be predicted include a behavior of
10 a moving object around the local car, a change in a state
of a traffic light, a change in environment such as
weather, and the like. Examples of a situation of a
driver that serves as a target to be predicted include a
behavior, a physical condition, and the like of the
15 driver.
[0183]
The situation prediction unit 154 supplies data
indicating a result of prediction processing together
with data from the traffic rule recognizer 152 and the
20 situation recognizer 153, to the route planning unit 161,
the action planning unit 162, and the motion planning
unit 163 of the planning unit 134, and the like.
[0184]
The route planning unit 161 plans a route to a
25 destination on the basis of data or a signal from each of
the units of the vehicle control system 100, e.g., the
map analyzer 151, the situation prediction unit 154, and
the like. For example, the route planning unit 161 sets
a route from a current position to a specified
30 destination on the basis of a global map. Furthermore,
for example, the route planning unit 161 appropriately
83
SP371466WO00
changes a route on the basis of a traffic jam, an
accident, traffic regulations, a situation of a
construction work or the like, a physical condition of a
driver, and the like. The route planning unit 161
5 supplies data indicating the planned route to the action
planning unit 162 or the like.
[0185]
The action planning unit 162 plans an action of the
local car for safely traveling on the route planned by
10 the route planning unit 161 within a planned time, on the
basis of data or a signal from each of the units of the
vehicle control system 100, e.g., the map analyzer 151,
the situation prediction unit 154, and the like. For
example, the action planning unit 162 plans a start, a
15 stop, a traveling direction (for example, moving forward,
moving backward, turning left, turning right, changing
directions, and the like), a traveling lane, traveling
speed, passing, and the like. The action planning unit
162 supplies data indicating the planned action of the
20 local car to the motion planning unit 163 or the like.
[0186]
The motion planning unit 163 plans a motion of the
local car for achieving the action planned by the action
planning unit 162, on the basis of data or a signal from
25 each of the units of the vehicle control system 100,
e.g., the map analyzer 151, the situation prediction unit
154, and the like. For example, the motion planning unit
163 plans acceleration, deceleration, a traveling track,
and the like. The motion planning unit 163 supplies data
30 indicating the planned motion of the local car to an
acceleration or deceleration controller 172 and a
84
SP371466WO00
direction controller 173 of the motion controller 135,
and the like.
[0187]
The motion controller 135 controls a motion of the
5 local car. The motion controller 135 includes the
emergency avoidance unit 171, the acceleration or
deceleration controller 172, and the direction controller
173.
[0188]
10 The emergency avoidance unit 171 performs
processing for detecting emergency, such as collision,
contact, entry to a danger zone, an abnormality in a
driver, or an abnormality in a vehicle, on the basis of
detection results of the outside-vehicle information
15 detection unit 141, the in-vehicle information detection
unit 142, and the vehicle state detection unit 143. In a
case where the occurrence of emergency has been detected,
the emergency avoidance unit 171 plans a motion of the
local car for the avoidance of emergency, such as a
20 sudden stop or a sudden turn. The emergency avoidance
unit 171 supplies data indicating the planned motion of
the local car to the acceleration or deceleration
controller 172, the direction controller 173, and the
like.
25 [0189]
The acceleration or deceleration controller 172
controls acceleration or deceleration to achieve the
motion of the local car that has been planned by the
motion planning unit 163 or the emergency avoidance unit
30 171. For example, the acceleration or deceleration
controller 172 calculates a control target value of a
85
SP371466WO00
drive force generator or a braking device to achieve
acceleration, deceleration, or a sudden stop that has
been planned, and supplies a control command indicating
the calculated control target value to the drive system
5 controller 107.
[0190]
The direction controller 173 controls a direction
to achieve the motion of the local car that has been
planned by the motion planning unit 163 or the emergency
10 avoidance unit 171. For example, the direction
controller 173 calculates a control target value of a
steering mechanism to achieve a traveling track or a
sudden turn that has been planned by the motion planning
unit 163 or the emergency avoidance unit 171, and
15 supplies a control command indicating the calculated
control target value to the drive system controller 107.
[0191]
Note that the object distance detection processing
described in the example described above is performed,
20 for example, by applying the components described below.
Information acquired by a camera or a distance
sensor that is configured as the data acquisition unit
102 is input to the outside-vehicle information detection
unit 141 of the detection unit 131.
25 The outside-vehicle information detection unit 141
specifies an object serving as a distance detection
target by using the information acquired by the camera or
the distance sensor, and calculates a distance of the
object.
30 Alternatively, a configuration may be employed in
which the outside-vehicle information detection unit 141
86
SP371466WO00
specifies an object serving as a distance detection
target by using the information acquired by the camera or
the distance sensor, and outputs the specified
information to the situation recognizer 153 of the
5 situation analyzer 133, and the situation recognizer 153
calculates a distance of the object.
[0192]
The calculated object distance is output to the
planning unit 134 that determines a route of an
10 automobile, and a movement plan for safe traveling is
made.
Moreover, information relating to the route
determined by the planning unit 134 is input to the
motion controller 135 that controls a motion of the
15 automobile, and the motion controller 135 controls the
motion of the automobile.
[0193]
[6. Example of configuration of information
processing device]
20 Fig. 18 illustrates the configuration of the
vehicle control system 100 that can be attached in a
movement device that performs the processing described
above. In processing in the example described above,
information detected by a variety of sensors such as a
25 distance sensor or a camera can be input to an
information processing device such as a PC, data
processing can be performed, and a distance to an object,
or a size or a position of the object can be calculated.
A specific hardware configuration example of an
30 information processing device in this case is described
with reference to Fig. 19.
87
SP371466WO00
[0194]
Fig. 19 is a diagram illustrating a hardware
configuration example of an information processing device
such as a general PC.
5 A central processing unit (CPU) 301 functions as a
data processing unit that performs various types of
processing according to a program stored in a read only
memory (ROM) 302 or a storage 308. For example,
processing according to the sequence described in the
10 example described above is performed. A random access
memory (RAM) 303 stores a program executed by the CPU
301, data, or the like. The CPU 301, the ROM 302, and
the RAM 303 that are described above are mutually
connected via a bus 304.
15 [0195]
The CPU 301 is connected to an input/output
interface 305 via the bus 304, and the input/output
interface 305 is connected to an input unit 306 that
includes various switches, a keyboard, a touch panel, a
20 mouse, a microphone, a situation data acquisition unit
such as a sensor, a camera, or the GPS, and the like and
an output unit 307 that includes a display, a speaker, or
the like.
Note that input information from a sensor 321, such
25 as a distance sensor or a camera, is also input to the
input unit 306.
Furthermore, the output unit 307 also outputs a
distance to an object, positional information of the
object, or the like as information for the planning unit
30 322 such as a motion planning unit of a movement device.
[0196]
88
SP371466WO00
The CPU 301 receives, as an input, a command,
situation data, or the like that has been input from the
input unit 306, performs various types of processing, and
outputs a processing result, for example, to the output
5 unit 307.
The storage 308 that is connected to the
input/output interface 305 includes, for example, a hard
disk or the like, and stores a program executed by the
CPU 301 or various types of data. A communication unit
10 309 functions as a transmission/reception unit of data
communication via a network such as the Internet or a
local area network, and performs communication with an
external device.
[0197]
15 A drive 310 that is connected to the input/output
interface 305 drives a removable medium 311 such as a
magnetic disk, an optical disk, a magneto-optical disk,
or a semiconductor memory, e.g., a memory card, and
records or reads data.
20 [0198]
[7. Summary of configuration according to the
present disclosure]
Examples of the present disclosure have been
described in detail above with reference to specified
25 examples. However, it is obvious that those skilled in
the art could make modifications or substitutions to the
examples without departing from the gist of the present
disclosure. Stated another way, the present invention
has been disclosed in the form of an example, and are not
30 to be construed as limitations. In order to judge the
gist of the present disclosure, the claims should be
89
SP371466WO00
considered.
[0199]
Note that the technology disclosed herein can
employ the configuration described below.
5 (1) An information processing device including:
a data processing unit that calculates an object
distance on the basis of a captured image of a camera
that captures an image in a direction that is orthogonal
to a movement direction of a movement device or in a
10 direction having an orthogonal component,
in which the data processing unit
detects a point at infinity from the captured
image, and
calculates the object distance by using information
15 relating to a positional relationship between a position
of the point at infinity that has been detected and a
position of an object in the captured image.
[0200]
(2) The information processing device described
20 in (1),
in which the data processing unit has a function of
detecting, from the captured image, a plurality of lines
on a real world space that extends in a direction away
from a camera position, and analyzing a change in line
25 segment coordinates among frames of a plurality of line
segments that has been detected to determine that the
plurality of line segments is a combination of parallel
lines on the real world space, and estimates a point at
infinity of a group of the plurality of line segments.
30 [0201]
(3) The information processing device described
90
SP371466WO00
in (1) or (2),
in which the data processing unit has a function of
detecting, from the captured image, a plurality of lines
on a real world space that extends in a direction away
5 from a camera position, and determining whether a
plurality of line segments that has been detected is
parallel line segments or non-parallel line segments.
[0202]
(4) The information processing device described
10 in any of (1) to (3),
in which the data processing unit has a function of
detecting, from the captured image, a plurality of lines
on a real world space that extends in a direction away
from a camera position, and detecting that coordinates of
15 a crossing peak of a combination of line segments that
have been detected sequentially move among frames in
accordance with a translational movement of a vehicle.
[0203]
(5) The information processing device described
20 in any of (1) to (4),
in which the data processing unit
calibrates the position of the point at infinity
that has been detected, and
calculates the object distance by using the
25 information relating to the positional relationship
between the position that has been calibrated of the
point at infinity and the position of the object in the
captured image.
[0204]
30 (6) The information processing device described
in any of (1) to (5),
91
SP371466WO00
in which the data processing unit calculates the
object distance according to (Formula 1b) described
below:
L = f × (H/h) (Formula 1b),
5 when the object distance is L,
where
f: a focal length of the camera,
H: a height from a reference plane of the camera,
and
10 h: a clearance between a contact point with the
reference plane of the object and the point at infinity
on the captured image, the clearance being a clearance in
an image upward or downward direction.
[0205]
15 (7) The information processing device described
in any of (1) to (6),
in which the data processing unit calculates the
object distance by using the captured image and
information relating to a distance to a reference point P
20 that is located closer to the camera than the object.
[0206]
(8) The information processing device described
in (7),
in which the data processing unit calculates the
25 object distance according to (Formula 1a) described
below:
L = Lref × (href/h) (Formula 1a),
when the object distance is L,
where
30 Lref: an actual distance from the camera to the
reference point P,
92
SP371466WO00
href: a clearance between the reference point P and
the point at infinity on the captured image, the
clearance being a clearance in an image upward or
downward direction, and
5 h: a clearance between a contact point with a
reference plane of the object and the point at infinity
on the captured image, the clearance being a clearance in
the image upward or downward direction.
[0207]
10 (9) The information processing device described
in any of (1) to (8),
in which the data processing unit detects, from the
captured image, a plurality of parallel lines on a real
world that extends in a direction away from a camera
15 position, and determines an intersection point on
extended lines of the plurality of parallel lines that
has been detected to be the point at infinity.
[0208]
(10) The information processing device described
20 in any of (1) to (9),
in which the data processing unit detects, from the
captured image, a straight line on a real world that
extends in a direction away from a camera position, and
determines an intersection point on respective extended
25 lines of the straight lines in an image frame unit to be
the point at infinity, directions of the straight lines
changing on the captured image in accordance with a
movement of the camera.
[0209]
30 (11) The information processing device described
in any of (1) to (10),
93
SP371466WO00
in which the data processing unit calculates a
movement speed of the object on the basis of a plurality
of the object distances that corresponds to a plurality
of image frames that has been captured by the camera.
5 [0210]
(12) An information processing device including:
a data processing unit that calculates an object
distance on the basis of a captured image of a camera
that captures an image in a direction that is orthogonal
10 to a movement direction of a movement device or in a
direction having an orthogonal component,
in which the data processing unit calculates the
object distance by using the captured image and
information relating to a distance to a reference point P
15 that is located closer to the camera than the object, and
the data processing unit calculates the object
distance according to (Formula 2) described below:
L = Lref × (Wref/W) (Formula 2),
when the object distance is L,
20 where
Lref: an actual distance from the camera to the
reference point P,
Wref: a width on an image of a reference object
that is located in an image lateral direction of the
25 reference point P, and
W: a width on the image of the reference object
that is located in the image lateral direction of a
distance calculation target object.
[0211]
30 (13) An information processing device including:
a data processing unit that calculates an object
94
SP371466WO00
distance on the basis of a captured image of a camera
that captures an image in a direction that is orthogonal
to a movement direction of a movement device or in a
direction having an orthogonal component,
5 in which the data processing unit calculates the
object distance according to (Formula 3) described below:
L = f × (Wrw/W) (Formula 3),
when the object distance is L,
where
10 f: a focal length of the camera,
Wrw: an actual size of a component having a known
actual size, the component being included in an object
image, and
W: an image size of the component having the known
15 actual size, the component being included in the object
image.
[0212]
(14) A movement device including:
a camera that captures an image in a direction that
20 is orthogonal to a movement direction of the movement
device or in a direction having an orthogonal component;
and
a data processing unit that calculates an object
distance on the basis of a captured image of the camera,
25 in which the data processing unit
detects a point at infinity from the captured
image, and
calculates the object distance by using information
relating to a positional relationship between a position
30 of the point at infinity that has been detected and a
position of an object in the captured image, and
95
SP371466WO00
the data processing unit includes:
a planning unit that determines a route of the
movement device on the basis of the object distance that
has been calculated; and
5 a motion controller that controls a motion of the
movement device according to the route that has been
determined by the planning unit.
[0213]
(15) The movement device described in (14),
10 in which the data processing unit detects, from the
captured image, a plurality of parallel lines on a real
world that extends in a direction away from a camera
position, and determines an intersection point on
extended lines of the plurality of parallel lines that
15 has been detected to be the point at infinity.
[0214]
(16) The movement device described in (14),
in which the data processing unit detects, from the
captured image, a straight line on a real world that
20 extends in a direction away from a camera position, and
determines an intersection point on respective extended
lines of the straight lines in an image frame unit to be
the point at infinity, directions of the straight lines
changing on the captured image in accordance with a
25 movement of the camera.
[0215]
(17) An information processing method performed by
an information processing device,
in which the information processing device includes
30 a data processing unit that calculates an object distance
on the basis of a captured image of a camera that
96
SP371466WO00
captures an image in a direction that is orthogonal to a
movement direction of a movement device or in a direction
having an orthogonal component, and
the data processing unit
5 detects a point at infinity from the captured
image, and
calculates the object distance by using information
relating to a positional relationship between a position
of the point at infinity that has been detected and a
10 position of an object in the captured image.
[0216]
(18) An information processing method performed by
a movement device,
in which the movement device includes:
15 a camera that captures an image in a direction that
is orthogonal to a movement direction of the movement
device or in a direction having an orthogonal component;
and
a data processing unit that calculates an object
20 distance on the basis of a captured image of the camera,
the data processing unit
detects a point at infinity from the captured
image, and
calculates the object distance by using information
25 relating to a positional relationship between a position
of the point at infinity that has been detected and a
position of an object in the captured image,
a planning unit determines a route of the movement
device on the basis of the object distance that has been
30 calculated, and
a motion controller controls a motion of the
97
SP371466WO00
movement device according to the route that has been
determined by the planning unit.
[0217]
(19) A program that causes an information
5 processing device to perform information processing,
in which the information processing device includes
a data processing unit that calculates an object distance
on the basis of a captured image of a camera that
captures an image in a direction that is orthogonal to a
10 movement direction of a movement device or in a direction
having an orthogonal component, and
the program causes the data processing unit to:
detect a point at infinity from the captured image;
and
15 calculate the object distance by using information
relating to a positional relationship between a position
of the point at infinity that has been detected and a
position of an object in the captured image.
[0218]
20 Furthermore, a series of processes described in the
description can be performed by hardware or software, or
a composite configuration of hardware and software. In a
case where the series of processes is performed by
software, a program recording a processing sequence can
25 be installed in a memory within a computer that has been
incorporated into dedicated hardware and can be executed,
or the program can be installed in a general-purpose
computer that can perform various types of processing and
can be executed. For example, the program can be
30 recorded in a recording medium in advance. The program
can be installed in a computer from the recording medium,
98
SP371466WO00
or the program can be received via a network such as a
local area network (LAN) or the Internet, and can be
installed in a recording medium such as an incorporated
hard disk.
5 [0219]
Note that various types of processing described in
the description may be performed in series according to
the description, or may be performed in parallel or
individually according to a throughput of a device that
10 performs the various types of processing or as needed.
Furthermore, the system described herein is a logical
aggregative configuration of a plurality of devices, and
does not always include devices having respective
configurations in the same housing.
15
INDUSTRIAL APPLICABILITY
[0220]
As described above, by employing a configuration in
one example of the present disclosure, a configuration is
20 achieved that calculates an object distance on the basis
of a captured image of a camera that captures an image in
a direction that is orthogonal to a movement direction of
a movement device or in a direction having an orthogonal
component.
25 Specifically, for example, a data processing unit
detects a point at infinity from a captured image of a
camera that captures an image in a direction that is
orthogonal to a movement direction of a movement device
or in a direction having an orthogonal component, and
30 calculates an object direction by using information
relating to a positional relationship between a position
99
SP371466WO00
of the detected point at infinity and a position of an
object in the captured image. The data processing unit
detects, from the captured image, a plurality of parallel
lines on a real world that extends in a direction away
5 from a camera position, and determines an intersection
point on extended lines of the detected plurality of
parallel lines to be the point at infinity.
Alternatively, an intersection point on respective
extended lines of the straight lines in an image frame
10 unit is determined to be the point at infinity,
directions of the straight lines changing on the captured
image in accordance with a movement of the camera.
By employing the present configuration, a
configuration is achieved that calculates an object
15 distance on the basis of a captured image of a camera
that captures an image in a direction that is orthogonal
to a movement direction of a movement device.
REFERENCE SIGNS LIST
20 [0221]
10 Movement device
11L Leftward-direction camera
11R Rightward-direction camera
12L Leftward-direction distance sensor
25 12R Rightward-direction distance sensor
100 Vehicle control system
101 Input unit
102 Data acquisition unit
103 Communication unit
30 104 In-vehicle device
105 Output controller
100
SP371466WO00
106 Output unit
107 Drive system controller
108 Drive system
109 Body system controller
5 110 Body system
111 Storage
112 Automatic driving controller
121 Communication network
131 Detection unit
10 132 Self-position estimation unit
141 Outside-vehicle information detection unit
142 In-vehicle information detection unit
143 Vehicle state detection unit
151 Map analyzer
15 152 Traffic rule recognizer
153 Situation recognizer
154 Situation prediction unit
161 Route planning unit
162 Action planning unit
20 163 Motion planning unit
171 Emergency avoidance unit
172 Acceleration or deceleration controller
173 Direction controller
301 CPU
25 302 ROM
303 RAM
304 Bus
305 Input/output interface
306 Input unit
30 307 Output unit
308 Storage
101
SP371466WO00
309 Communication unit
310 Drive
311 Removable medium
321 Sensor
5 322 Planning unit
102
SP371466WO00
CLAIMS
1. An information processing device comprising:
a data processing unit that calculates an object
5 distance on a basis of a captured image of a camera that
captures an image in a direction that is orthogonal to a
movement direction of a movement device or in a direction
having an orthogonal component,
wherein the data processing unit
10 detects a point at infinity from the captured
image, and
calculates the object distance by using information
relating to a positional relationship between a position
of the point at infinity that has been detected and a
15 position of an object in the captured image.
2. The information processing device according to
claim 1,
wherein the data processing unit has a function of
20 detecting, from the captured image, a plurality of lines
on a real world space that extends in a direction away
from a camera position, and analyzing a change in line
segment coordinates among frames of a plurality of line
segments that has been detected to determine that the
25 plurality of line segments is a combination of parallel
lines on the real world space, and estimates a point at
infinity of a group of the plurality of line segments.
3. The information processing device according to
30 claim 1,
wherein the data processing unit has a function of
103
SP371466WO00
detecting, from the captured image, a plurality of lines
on a real world space that extends in a direction away
from a camera position, and determining whether a
plurality of line segments that has been detected is
5 parallel line segments or non-parallel line segments.
4. The information processing device according to
claim 1,
wherein the data processing unit has a function of
10 detecting, from the captured image, a plurality of lines
on a real world space that extends in a direction away
from a camera position, and detecting that coordinates of
a crossing peak of a combination of line segments that
have been detected sequentially move among frames in
15 accordance with a translational movement of a vehicle.
5. The information processing device according to
claim 1,
wherein the data processing unit
20 calibrates the position of the point at infinity
that has been detected, and
calculates the object distance by using the
information relating to the positional relationship
between the position that has been calibrated of the
25 point at infinity and the position of the object in the
captured image.
6. The information processing device according to
claim 1,
30 wherein the data processing unit calculates the
object distance according to (Formula 1b) described
104
SP371466WO00
below:
L = f × (H/h) (Formula 1b),
when the object distance is L,
where
5 f: a focal length of the camera,
H: a height from a reference plane of the camera,
and
h: a clearance between a contact point with the
reference plane of the object and the point at infinity
10 on the captured image, the clearance being a clearance in
an image upward or downward direction.
7. The information processing device according to
claim 1,
15 wherein the data processing unit calculates the
object distance by using the captured image and
information relating to a distance to a reference point P
that is located closer to the camera than the object.
20 8. The information processing device according to
claim 7,
wherein the data processing unit calculates the
object distance according to (Formula 1a) described
below:
25 L = Lref × (href/h) (Formula 1a),
when the object distance is L,
where
Lref: an actual distance from the camera to the
reference point P,
30 href: a clearance between the reference point P and
the point at infinity on the captured image, the
105
SP371466WO00
clearance being a clearance in an image upward or
downward direction, and
h: a clearance between a contact point with a
reference plane of the object and the point at infinity
5 on the captured image, the clearance being a clearance in
the image upward or downward direction.
9. The information processing device according to
claim 1,
10 wherein the data processing unit detects, from the
capture image, a plurality of parallel lines on a real
world that extends in a direction away from a camera
position, and determines an intersection point on
extended lines of the plurality of parallel lines that
15 has been detected to be the point at infinity.
10. The information processing device according to
claim 1,
wherein the data processing unit detects, from the
20 captured image, a straight line on a real world that
extends in a direction away from a camera position, and
determines an intersection point on respective extended
lines of the straight lines in an image frame unit to be
the point at infinity, directions of the straight lines
25 changing on the captured image in accordance with a
movement of the camera.
11. The information processing device according to
claim 1,
30 wherein the data processing unit calculates a
movement speed of the object on a basis of a plurality of
106
SP371466WO00
the object distances that corresponds to a plurality of
image frames that has been captured by the camera.
12. An information processing device comprising:
5 a data processing unit that calculates an object
distance on a basis of a captured image of a camera that
captures an image in a direction that is orthogonal to a
movement direction of a movement device or in a direction
having an orthogonal component,
10 wherein the data processing unit calculates the
object distance by using the captured image and
information relating to a distance to a reference point P
that is located closer to the camera than the object, and
the data processing unit calculates the object
15 distance according to (Formula 2) described below:
L = Lref × (Wref/W) (Formula 2),
when the object distance is L,
where
Lref: an actual distance from the camera to the
20 reference point P,
Wref: a width on an image of a reference object
that is located in an image lateral direction of the
reference point P, and
W: a width on the image of the reference object
25 that is located in the image lateral direction of a
distance calculation target object.
13. An information processing device comprising:
a data processing unit that calculates an object
30 distance on a basis of a captured image of a camera that
captures an image in a direction that is orthogonal to a
107
SP371466WO00
movement direction of a movement device or in a direction
having an orthogonal component,
wherein the data processing unit calculates the
object distance according to (Formula 3) described below:
5 L = f × (Wrw/W) (Formula 3),
when the object distance is L,
where
f: a focal length of the camera,
Wrw: an actual size of a component having a known
10 actual size, the component being included in an object
image, and
W: an image size of the component having the known
actual size, the component being included in the object
image.
15
14. A movement device comprising:
a camera that captures an image in a direction that
is orthogonal to a movement direction of the movement
device or in a direction having an orthogonal component;
20 and
a data processing unit that calculates an object
distance on a basis of a captured image of the camera,
wherein the data processing unit
detects a point at infinity from the captured
25 image, and
calculates the object distance by using information
relating to a positional relationship between a position
of the point at infinity that has been detected and a
position of an object in the captured image, and
30 the data processing unit includes:
a planning unit that determines a route of the
108
SP371466WO00
movement device on a basis of the object distance that
has been calculated; and
a motion controller that controls a motion of the
movement device according to the route that has been
5 determined by the planning unit.
15. The movement device according to claim 14,
wherein the data processing unit detects, from the
captured image, a plurality of parallel lines on a real
10 world that extends in a direction away from a camera
position, and determines an intersection point on
extended lines of the plurality of parallel lines that
has been detected to be the point at infinity.
15 16. The movement device according to claim 14,
wherein the data processing unit detects, from the
captured image, a straight line on a real world that
extends in a direction away from a camera position, and
determines an intersection point on respective extended
20 lines of the straight lines in an image frame unit to be
the point at infinity, directions of the straight lines
changing on the captured image in accordance with a
movement of the camera.
25 17. An information processing method performed by an
information processing device,
wherein the information processing device includes
a data processing unit that calculates an object distance
on a basis of a captured image of a camera that captures
30 an image in a direction that is orthogonal to a movement
direction of a movement device or in a direction having
109
SP371466WO00
an orthogonal component, and
the data processing unit
detects a point at infinity from the captured
image, and
5 calculates the object distance by using information
relating to a positional relationship between a position
of the point at infinity that has been detected and a
position of an object in the captured image.
10 18. An information processing method performed by a
movement device,
wherein the movement device includes:
a camera that captures an image in a direction that
is orthogonal to a movement direction of the movement
15 device or in a direction having an orthogonal component;
and
a data processing unit that calculates an object
distance on a basis of a captured image of the camera,
the data processing unit
20 detects a point at infinity from the captured
image, and
calculates the object distance by using information
relating to a positional relationship between a position
of the point at infinity that has been detected and a
25 position of an object in the captured image,
a planning unit determines a route of the movement
device on a basis of the object distance that has been
calculated, and
a motion controller controls a motion of the
30 movement device according to the route that has been
determined by the planning unit.
110
SP371466WO00
19. A program that causes an information processing
device to perform information processing,
wherein the information processing device includes
5 a data processing unit that calculates an object distance
on a basis of a captured image of a camera that captures
an image in a direction that is orthogonal to a movement
direction of a movement device or in a direction having
an orthogonal component, and
10 the program causes the data processing unit to:
detect a point at infinity from the captured image;
and
calculate the object distance by using information
relating to a positional relationship between a position
15 of the point at infinity that has been detected and a
position of an object in the captured image.