DESCRIPTION
IMAGE DISPLAY APPARATUS, IMAGE DISPLAY METHOD, MOBILE
APPARATUS, IMAGE DISPLAY SYSTEM, AND COMPUTER PROGRAM
5TECHNICAL FIELD
[0001]
The technology disclosed in this specification relates
to an image display apparatus that is mounted on the head or
the face of a user and is used for viewing an image, an image
display method, a mobile apparatus that captures images to
be viewed while moving, an image display system, and a
computer program. More particularly, the technology relates
to an image display apparatus, an image display method, a
mobile apparatus, an image display system, and a computer
program that are used for viewing an image captured by an
external camera, such as a whole-sky image, a fish-eye image,
and a panoramic image.
BACKGROUND ART
[0002]
An image display apparatus or a head mount display that
is mounted on the head or the face and is used for viewing
an image is well known. In a head mount display, a display
unit is provided for each of the right and left eyes, and an
25 enlarged virtual image of a display image is formed by a
virtual image optical system so that a user can view an image
with realistic sensation. If a head mount display is designed
to completely block the outside world when mounted on the head
of a user, the sense of immersion during the viewing is
30 increased. A head mount display can also display different
images for the right and left eyes, and can present a 3D image
3
by displaying images with parallaxes for the right and left
eyes.
[0003]
With a head mount display, a wide-angle image can be
viewed. For example, there is a disclosure 5 of a head mount
display with which a whole-space 360-degree image that
follows motion of the head of a user having a gyro sensor
attached to his/her head can be enjoyed (see Patent Document
1 and Patent Document 2, for example). The display region
is moved so as to offset motion of the head detected by the
gyro sensor. In this manner, an image that follows the motion
of the head of the user can be presented.
[0004]
Also, there is an application developed for enabling
a user to view a live image captured by an external camera
with a head mount display. For example, there is a disclosure
of an image display system in which an image actually captured
by an imaging device mounted on a moving object other than
a human being such as a radio-controlled model is displayed
on a display device a user is wearing (see Patent Document
3, for example).
[0005]
There is also the FPV (First Person Viewing) technique
known for enabling a user to control a radio-controlled model
of a helicopter or the like while looking at a first-person
viewpoint (pilot viewpoint) image captured by a wireless
camera mounted on the radio-controlled model.
[0006]
For example, there is a disclosure of an aerial imaging
system in which an all-direction camera that captures images
of the surroundings and a laser distance meter that measures
4
the altitude of the all-direction camera from the ground are
mounted on a small-size helicopter that has its flight
altitude and flight velocity controlled by a flight control
device, and aerial imaging is performed with the
all-direction camera at a predetermined altitude 5 based on the
altitude measured by the laser distance meter (see Patent
Document 4, for example). Images captured by the
all-direction camera can be transmitted to an external
computer via a communication network.
[0007]
Further, there is a disclosure of a network system in
which a radio-controlled model car equipped with a mid- and
long-range three-dimensional stereo camera and a short-range
three-dimensional stereo camera transmits a
15 three-dimensional combined image, and the three-dimensional
combined image is displayed on the side of the controller (see
Patent Document 5, for example).
[0008]
Also, there is a disclosure of a network system that
receives, on the side of the controller, an image generated
by a model apparatus capturing an image of the scenery in front
and information about the location and the orientation of the
model apparatus, and generates and displays a virtual image
in accordance with the location and the orientation (see
Patent Document 6, for example).
CITATION LIST
PATENT DOCUMENTS
[0009]
Patent Document 1: JP 9-106322 A
Patent Document 2: JP 2010-256534 A
5
Patent Document 3: JP 2008-147865 A, paragraphs [0024]
through [0026]
Patent Document 4: JP 2011-183824 A
Patent Document 5: JP 2012-151800 A
Patent Document 6: 5 JP 2012-143447 A
SUMMARY OF THE INVENTION
PROBLEMS TO BE SOLVED BY THE INVENTION
[0010]
The technology disclosed in this specification aims to
provide an image display apparatus, an image display method,
a mobile apparatus, an image display system, and a computer
program that are excellent and can be used in viewing an image
captured by an external camera, such as a whole-sky image,
a fish-eye image, or a panoramic image.
[0011]
The technology disclosed in this specification further
aims to provide an excellent image display apparatus capable
of appropriately displaying a first-person viewpoint image
captured by a camera mounted on a mobile apparatus such as
a radio-controlled model, and also provide an image display
method, a mobile apparatus, an image display system, and a
computer program.
SOLUTIONS TO PROBLEMS
[0012]
This application is made in view of the above problems,
and the technology disclosed in claim 1 is an image display
apparatus that includes:
a display unit mounted on the head of a user;
a posture detecting unit that detects posture of the
head; and
a display control unit that controls display of an image
on the display unit based on the posture of the head, the image
being captured by a mobile apparatus.
[0013]
According to the technology disclosed in claim 2 of this
application, the display control unit of the image display
apparatus of claim 1 clips a region corresponding to the
posture of the head from a wide-angle image captured by the
mobile apparatus, and displays the clipped region on the
display unit.
[0014]
According to the technology disclosed in claim 3 of this
application, when a plurality of viewpoint images captured
at a plurality of viewpoints by a deep focus parallel method
is displayed, the display control unit of the image display
apparatus of claim 1 adjusts the convergence point between
the viewpoint images based on the movement velocity of the
moving object.
[0015]
According to the technology disclosed in claim 4 of this
application, the display control unit of the image display
apparatus of claim 1 displays an image with an inter-viewpoint
distance corresponding to a zooming operation on the display
unit.
[0016]
According to the technology disclosed in claim 5 of this
application, the image display apparatus of claim 1 offsets
the direction of the line of sight of a camera unit of the
30 mobile apparatus in at least one of the directions of panning,
tilting, and rolling with respect to the posture of the head.
[0017]
According to the technology disclosed in claim 6 of this
application, the image display apparatus of claim 1 further
including
an operational feeling feedback unit 5 that feeds back
an operational feeling to the user through tactile sensation
or vibration,
wherein the feedback to the user is based on the
acceleration to which the mobile apparatus is subjected while
moving.
[0018]
According to the technology disclosed in claim 7 of this
application, the display control unit of the image display
apparatus of claim 1 superimposes an AR image on a real-world
image captured by the mobile apparatus, before displaying the
image.
[0019]
According to the technology disclosed in claim 8 of this
application, the display control unit of the image display
apparatus of claim 7 displays the AR image corresponding to
at least one of the current location of the mobile apparatus,
an object included in the captured image, and the state of
the mobile apparatus.
[0020]
According to the technology disclosed in claim 9 of this
application, the display control unit of the image display
apparatus of claim 1 displays location information about the
mobile apparatus and the user.
[0021]
30 According to the technology disclosed in claim 10 of
this application, the display control unit of the image
display apparatus of claim 1 further displays an image
captured by an automatic tracker that captures the image while
tracking the mobile apparatus.
[0022]
According to the technology disclosed 5 in claim 11 of
this application, the image display apparatus of claim 1
further includes a self-sight-line image acquiring unit that
acquires a self-sight-line image to be seen on the line of
sight of the user. The display control unit switches the
image being displayed between a moving-object sight-line
image captured by the mobile apparatus and the
self-sight-line image.
[0023]
According to the technology disclosed in claim 12 of
this application, the display control unit of the image
display apparatus of claim 1 switches the image being
displayed between images captured from a plurality of
viewpoint positions of the mobile apparatus in accordance
with the posture of the head.
[0024]
According to the technology disclosed in claim 13 of
this application, the display control unit of the image
display apparatus of claim 1 corrects shaking in a moving
image captured by the mobile apparatus, before displaying the
moving image.
[0025]
The technology disclosed in claim 14 of this
application is an image display method that includes:
a posture detecting step of detecting posture of the
head of a user; and
a display control step of controlling display of an
image based on the posture of the head, the image being
captured by a mobile apparatus.
[0026]
The technology disclosed in claim 15 of this
application is an image display system 5 that includes:
a mobile apparatus that captures an image while moving;
and
an image display apparatus that displays the image
captured by the mobile apparatus in accordance with posture
10 of the head of a user.
[0027]
It should be noted that the term “system” means a logical
assembly of devices (or functional modules that realize
specific functions), and the respective devices or functional
modules are not necessarily in a single housing.
[0028]
The technology disclosed claim 16 of this application
is a mobile apparatus that includes:
a camera unit;
20 a camera platform that controls the direction of the
line of sight of the camera unit;
a moving unit that moves the apparatus; and
a communication unit that communicates data including
an image captured by the camera unit,
25 wherein the camera unit includes a plurality of cameras
that perform imaging by a deep focus parallel method, the
cameras having different viewpoints.
[0029]
According to the technology disclosed in claim 17 of
30 this application, the mobile apparatus of claim 16 captures
a whole-sky image while changing the inter-viewpoint distance
10
between the cameras with different viewpoints.
[0030]
According to the technology disclosed in claim 18 of
this application, the mobile apparatus of claim 16
extrapolates images outside the viewpoints 5 of the cameras
from images captured by the cameras with different viewpoints
at a fixed inter-viewpoint distance from each other.
[0031]
The technology disclosed in claim 19 of this
10 application is a computer program written in a
computer-readable format,
the computer program causing a computer to function as:
a posture detecting unit that detects posture of the
head of a user; and
15 a display control unit that controls display of an image
based on the posture of the head, the image being captured
by a mobile apparatus.
[0032]
The computer program according to claim 19 of this
20 application defines a computer program written in a
computer-readable format so as to realize predetermined
processing in a computer. In other words, as the computer
program according to claim 19 of this application is installed
into a computer, cooperative actions are realized in the
25 computer, and the same effects as those of the image display
apparatus according to claim 1 of this application can be
achieved.
EFFECTS OF THE INVENTION
30 [0033]
The technology disclosed in this specification
11
provides an excellent image display apparatus that can
present an image with realistic sensation by displaying a
first-person viewpoint image captured by a camera mounted on
a mobile apparatus such as a radio-controlled model on a
screen placed in front of the eyes of a user, 5 and also provides
an image display method, an image display system, a mobile
apparatus, and a computer program.
[0034]
Other objects, features, and advantages of the
10 technology disclosed in this specification will be made
apparent by the embodiments described below and the detailed
descriptions with reference to the accompanying drawings.
BRIEF DESCRIPTION OF DRAWINGS
15 [0035]
Fig. 1 is a diagram schematically showing the
configuration of an image display system 100 according to an
embodiment of the technology disclosed in this specification.
Fig. 2 is a front view of a user wearing a head mount
20 display 110.
Fig. 3 is a top view of the user wearing the head mount
display 110 shown in Fig. 2.
Fig. 4 is a diagram showing an example internal
structure of the head mount display 110.
25 Fig. 5 is a diagram showing the system of coordinates
in the posture to be detected by the posture/location
detecting unit 404.
Fig. 6 is a diagram showing an example internal
structure of a mobile apparatus 120.
30 Fig. 7 is a diagram schematically showing the
functional structure of a control unit 401 for displaying an
12
image captured on the side of the mobile apparatus 120 with
the head mount display 110.
Fig. 8A is a diagram showing a situation where a camera
unit 605 is mounted so that a camera platform 606 can control
the line of sight in the respective directions 5 of rolling,
tilting, and panning.
Fig. 8B is a diagram showing a situation where the
distance between the left-eye camera and the right-eye camera
that constitute a stereo camera is variable.
10 Fig. 8C is a diagram showing a situation where a
principal camera 801 is mounted on the main frame of a mobile
apparatus 120-2 as an automobile, and a secondary camera 802
is mounted behind the principal camera 801.
Fig. 9 is a diagram showing a situation where the
15 position and the posture of a display field displayed on a
display panel 409 are moved in accordance with motion of the
head of a user.
Fig. 10 is a diagram showing an example sequence of
operation to display a wide-angle image such as a whole-sky
20 image captured on the side of a mobile apparatus 120, the
wide-angle image being made to follow the posture of the head
of the user on the side of the head mount display 110.
Fig. 11 is a diagram showing a modification of the
sequence of operation to display a wide-angle image such as
25 a whole-sky image captured on the side of a mobile apparatus
120, the wide-angle image being made to follow the posture
of the head of the user on the side of the head mount display
110.
Fig. 12A is a diagram for explaining a method of
30 controlling the convergence point when a three-dimensional
captured image is displayed on the side of the head mount
13
display 110 based on velocity information about a mobile
apparatus 120.
Fig. 12B is a diagram for explaining the method of
controlling the convergence point when a three-dimensional
captured image is displayed on the side 5 of the head mount
display 110 based on velocity information about a mobile
apparatus 120.
Fig. 12C is a diagram for explaining the method of
controlling the convergence point when a three-dimensional
10 captured image is displayed on the side of the head mount
display 110 based on velocity information about a mobile
apparatus 120.
Fig. 13A is a diagram for explaining a method of
capturing a three-dimensional image while varying an
15 inter-viewpoint distance d in accordance with a zooming
operation.
Fig. 13B is a diagram for explaining the method of
capturing a three-dimensional image while varying the
inter-viewpoint distance d in accordance with a zooming
20 operation.
Fig. 14 is a diagram showing a situation where the head
is tilted upward, and the camera unit 605 mounted on a mobile
apparatus 120-2 is also tilted upward.
Fig. 15 is a diagram showing a situation where the axis
25 of tilt of the camera unit 605 mounted on the mobile apparatus
120-2 is secured in a position that is rotated upward by Δθy
from the axis of tilt of the head of the user.
Fig. 16 is a diagram showing a situation where the
coordinate system of the camera unit 605 is offset in the
30 respective directions of panning, tilting, and rolling with
respect to the coordinate system of the head of the user.
14
Fig. 17 is a diagram showing a situation where location
information about a user is displayed as a small screen in
the screen displaying an image captured by a mobile apparatus
120.
Fig. 18 is a diagram showing a situation 5 where location
information about a user is displayed on a large screen, and
a captured image is displayed on a small screen.
Fig. 19 is a diagram showing an example of display of
location information about a user.
10 Fig. 20 is a diagram showing an example of display of
location information about a user.
Fig. 21 is a diagram showing a situation where an
automatic tracker 2100 equipped with a camera 2101 is tracking
the rear of a mobile apparatus 120-1 flying in the air.
15 Fig. 22 is a diagram showing an example of an image of
the mobile apparatus 120-1 captured from behind by the camera
2101 of the automatic tracker 2100.
Fig. 23 is a diagram showing an example of a
car-sight-line image captured by the mobile apparatus 120-2
20 as an automobile.
Fig. 24 is a diagram showing an example of a situation
where a user wearing the head mount display 110 follows a
running mobile apparatus 120-3 with his/her eyes.
Fig. 25 is a diagram showing an example of a
25 self-sight-line image of a user who is following the mobile
apparatus 120-3 with his/her eyes, the self-sight-line image
being captured by an external camera 413.
Fig. 26 is a diagram showing an example layout of images
captured by a mobile apparatus 120 in different viewpoint
30 positions.
Fig. 27 is a diagram showing a situation where observed
15
images have trapezoidal distortion in accordance with upward
and downward tilting of the head of a user.
Fig. 28 is a diagram showing a situation where AR images
of virtual obstacle 2801 and a prohibited area 2802 are
displayed on a car-sight-line image (the real 5 word) captured
by the mobile apparatus 120-2.
MODES FOR CARRYING OUT THE INVENTION
[0036]
10 The following is a detailed description of embodiments
of the technology disclosed in this specification, with
reference to the drawings.
[0037]
A. System Configuration
15 Fig. 1 schematically shows the configuration of an
image display system 100 according to an embodiment of the
technology disclosed in this specification. The image
display system 100 shown in the drawing includes an image
display apparatus (a head mount display) 110 mounted on the
20 head or the face of a user, mobile apparatuses 120-1, 120-2,
120-3, and others that are mobile models of an aircraft (or
a helicopter or some other flying object), an automobile, a
watercraft, and the like, and a controller 130 that wirelessly
controls the mobile apparatuses 120. Each of the mobile
25 apparatus 120-1, 120-2, 120-3, and others is equipped with
a wireless camera (not shown), and captures images of scenery
while moving. The controller 130 may be a multifunctional
information terminal such as a smartphone, and starts an
application for controlling the mobile apparatuses 120.
30 [0038]
The head mount display 110 and the mobile apparatuses
16
120, and the controller 130 and the mobile apparatuses 120
are wirelessly connected via a wireless network or infrared
communication or the like.
[0039]
Each of the mobile apparatuses 120 5 is equipped with a
wireless camera (not shown) via a camera platform that can
change its posture about the respective axes of panning,
tilting, and yawing. This wireless camera can capture
whole-sky images or celestial sphere images, or panoramic
10 images such as half-sky images and lesser images.
Alternatively, the wireless camera can perform wide-angle
imaging, using a fish-eye lens.
[0040]
The mobile apparatuses 120 transmit images captured
15 with the wireless cameras to the controller 130 and the head
mount display 110. In a case where captured images can be
transferred directly between the head mount display 110 and
the mobile apparatuses 120, however, the mobile apparatuses
120 do not need to transfer captured images to the controller
20 130, and wireless connection between the controller 130 and
the head mount display 110 is not necessary. In the
description below, the controller 130 will not be described
in detail on the assumption that data communication can be
performed directly between the head mount display 110 and the
25 mobile apparatuses 120.
[0041]
Although the user wearing the head mount display 110
is controlling the mobile apparatuses 120 with the controller
130 in the example shown in Fig. 1, a different person from
30 the user who enjoys captured images sent from the mobile
apparatuses 120 with the head mount display 110 may control
17
the mobile apparatuses 120 with the controller 130.
[0042]
Fig. 2 is a front view of the user wearing the head mount
display 110. The head mount display 110 shown in the drawing
is a structure having a shape similar to 5 that of a pair of
glasses, and is designed to directly cover the right and left
eyes of the user wearing the head mount display 110. At the
locations on the inner side of the main frame of the head mount
display 110 facing the right and left eyes, display panels
10 (not shown in Fig. 2) the user is observing are provided. The
display panels are formed with microdisplays such as organic
EL devices or liquid crystal displays.
[0043]
At almost the center of the front surface of the main
15 frame of the head mount display 110 having an eyeglass-like
shape, an external camera 413 for inputting surroundings
images (the user’s vision) is provided. Further,
microphones 201 and 202 are provided near the right and left
ends of the supporting member, respectively. As the two
20 microphones 201 and 202 are provided, only the voice localized
at the center (the user’s voice) is recognized and can be
separated from ambient noise and voices of the other people.
For example, an incorrect operation during a voice input
operation can be prevented.
25 [0044]
Fig. 3 is a top view of the user wearing the head mount
display 110 shown in Fig. 2. The head mount display 110 shown
in the drawing has display panels for the left eye and the
right eye on the side surfaces facing the face of the user.
30 The display panels are formed with microdisplays such as
organic EL devices or liquid crystal displays. Images
18
displayed on the right and left display panels are observed
as enlarged virtual images with the right and left eyes of
the user, having passed through the respective virtual image
optical units. Since the eye height and the interpupillary
distance vary among individuals, positioning 5 needs to be
performed between the respective right and left display
systems and the eyes of the user. In the example shown in
Fig. 3, an interpupillary adjustment mechanism is provided
between the display panel for the right eye and the display
10 panel for the left eye.
[0045]
Fig. 4 shows an example internal structure of the head
mount display 110. The following is a description of the
respective components.
15 [0046]
A control unit 401 includes a ROM (Read Only Memory)
401A and a RAM (Random Access Memory) 401B. The ROM 401A
stores program codes to be executed by the control unit 401,
and various kinds of data. By executing a program loaded into
20 the RAM 401B, the control unit 401 starts image display
control, and collectively controls operations of the entire
head mount display 11000. Examples of programs and data
stored in the ROM 401A include an image display control
program, a program for processing communication with external
25 devices such as the mobile apparatuses 120 and the controller
130, and identification information unique to this apparatus
110. The image display control program is for performing
display control on captured images received from the mobile
apparatuses 120, for example, but will be described later in
30 detail.
[0047]
19
An input operating unit 402 includes one or more
operation pieces with which the user performs an input
operation, such as keys, buttons, or switches. The input
operating unit 402 receives a user instruction via the
operation pieces, and outputs the user 5 instruction to the
control unit 401. The input operating unit 402 also receives
a user instruction formed with a remote control command
received by a remote control receiving unit 403, and also
outputs this user instruction to the control unit 401.
10 [0048]
A posture/location detecting unit 404 is a unit that
detects the posture of the head of the user wearing the head
mount display 110. The posture/location detecting unit 404
is formed with a gyro sensor, an acceleration sensor, a GPS
15 (Global Positioning System) sensor, or a geomagnetic sensor.
Alternatively, the posture/location detecting unit 404 is
formed with a combination of two or more of those sensors,
with the advantages and the disadvantages of the respective
sensors being taken into consideration.
20 [0049]
Fig. 5 shows the system of coordinates in the posture
to be detected by the posture/location detecting unit 404.
The depth direction of a display image (an enlarged virtual
image) is the z-axis, the horizontal direction is the y-axis,
25 the vertical direction is the x-axis, and the position of the
origin of the x-, y-, and z-axes is the viewpoint position.
Accordingly, the roll θz is equivalent to the motion of the
head of the user about the z-axis, the tilt θy is equivalent
to the motion of the head of the user about the y-axis, and
30 the pan θx is equivalent to the motion of the head of the user
about the x-axis. The posture/location detecting unit 404
20
detects the motions (θz, θy, and θx) in the respective
directions of rolling, tilting, and panning of the head of
the user, and the parallel motion of the head, and then outputs
the motions to the control unit 401. As will be described
later, when displaying an image captured by 5 a mobile apparatus
120 on the screen of a display panel 409, the control unit
401 can present an image that follows the motion of the head
of the user by moving the display field so as to offset the
motion of the head detected by the posture/location detecting
10 unit 404.
[0050]
A state detecting unit 411 acquires state information
about the state of the user wearing the head mount display
110, and outputs the state information to the control unit
15 401. For example, the operating state of the user (whether
the user is wearing the head mount display 110), the state
of action of the user (the motion state such as a resting state,
a walking state, or a running state, the opened/closed state
of the eyelids, or the direction of the line of sight), the
20 mental state (the level of excitement, consciousness, feeling,
or emotion, such as whether the user is absorbed in or
concentrates on observing the display image), and the
physiological state are obtained as the state information.
So as to acquire the state information from the user, the state
25 detecting unit 411 may include various kinds of state sensors
(not shown) such as an attachment sensor formed with a
mechanical switch or the like, a gyro sensor, an acceleration
sensor, a velocity sensor, a pressure sensor, a body
temperature sensor, a perspiration sensor, a myoelectric
30 potential sensor, an ocular potential sensor, and a
brain-wave sensor.
21
[0051]
An operational feeling feedback (FB) unit 412 includes
a vibration generator or the like, and gives operational
feeling feedback to the user wearing the head mount display
110 through tactile sensation 5 or vibration.
[0052]
The external camera 413 is placed at almost the center
of the front surface of the main frame of the head mount display
110 having an eyeglass-like shape or a hat-like shape (see
10 Fig. 2), and can capture images of the surroundings. Also,
posture control in the panning, tilting, and rolling
directions of the external camera 413 is performed in
accordance with the direction of the user’s line of sight
detected by the state information detecting unit 411, so that
15 an image on the level of the user’s line of sight can be
captured with the external camera 413. Furthermore, the
relative velocities of the mobile apparatuses 120 can be
measured with the external camera 413.
[0053]
20 A communication unit 405 performs a communication
process with external devices such as the mobile apparatuses
120 and the controller 130, and also performs
modulation/demodulation processes and encoding/decoding
processes on communication signals. For example, the
25 communication unit 405 receives images captured by wireless
cameras from the mobile apparatuses 120. Images received,
demodulated, and decoded by the communication unit 405, or
other received data is supplied to the control unit 401. The
control unit 401 also sends data to be transmitted to external
30 devices through the communication unit 405.
[0054]
22
The communication unit 405 can have any structure. For
example, the communication unit 405 can be designed in
accordance with the communication standard to be used in
transmitting/receiving operations with external devices
with which communication is to 5 be performed. The
communication standard may be for either cable communication
or wireless communication. Here, the communication standard
may be MHL (Mobile High-definition Link), USB (Universal
Serial Bus), HDMI (a registered trade name) (High Definition
10 Multimedia Interface), Wi-Fi (a registered trade name),
Bluetooth (a registered trade name) communication, or
infrared communication, for example.
[0055]
A storage unit 406 is a large-capacity storage that is
15 formed with an SSD (Solid State Drive) or the like. The
storage unit 406 stores the application program to be executed
by a control unit 701, and the data of whole-sky images,
fish-eye images, panoramic images, and the like captured by
the mobile apparatuses 120.
20 [0056]
An image processing unit 407 further performs signal
processing such as image quality correction on the image
signals output from the control unit 401, and converts the
resolution of the image signals to a resolution compatible
25 with the screen of the display panel 409. A display drive
unit 408 sequentially selects the pixels of the display panel
409 by the row, and performs line sequential scanning on the
pixels, to supply pixel signals based on the image signals
subjected to the signal processing.
30 [0057]
The display panel 409 is formed with a microdisplay such
23
as an organic EL (Electro-Luminescence) device or a liquid
crystal display. A virtual image optical unit 410 enlarges
the image displayed on the display panel 409, and projects
the image as an enlarged virtual image for the user to see.
5 [0058]
The virtual image optical unit 410 enlarges the image
displayed on the display panel 409 at a magnification of 1000
or more, for example, and forms, on the retina, an
approximately 750-inch virtual image 20 meters away from the
10 eyes of the user. This is equivalent to 45.09 degrees in the
horizontal angle of view of the displayed pixels.
[0059]
Fig. 6 shows an example internal structure of a mobile
apparatus 120. As shown in Fig. 1, the mobile apparatuses
15 120 are models of a moving object such as an aircraft, a
helicopter, an automobile, a watercraft, and the like, and
basically vary in a movement mechanism unit 603 while are
substantially identical in the other functional aspects.
[0060]
20 A storage unit 602 is realized by a memory device such
as a RAM or a ROM, and a large-capacity storage such as a hard
disk drive or an SSD. The storage unit 602 is used to store
the program to be executed by a CPU (Central Processing Unit)
601, and images captured by a camera. The CPU 601 controls
25 the respective components in the mobile apparatus 120 by
executing the program stored in the storage unit 602.
[0061]
The mobile apparatus 120 has the movement mechanism
unit 603 that varies with the type of the moving object such
30 as an aircraft, a helicopter, an automobile, or a watercraft,
and the movement mechanism unit 603 is activated in accordance
24
with a movement instruction from the CPU 601, and moves the
mobile apparatus 120.
[0062]
A location/posture/velocity detecting unit 604
includes a gyro sensor, an acceleration sensor, 5 a GPS sensor,
a geomagnetic sensor, and the like, and acquires information
about the current location and posture of the mobile apparatus
120, and information about the velocity of movement caused
by the movement mechanism unit 603. In the case of the mobile
10 apparatus 120-2, which is an automobile, the
location/posture/velocity detecting unit 604 can calculate
the movement velocity from the number of revolutions of the
motor that rotates the wheels, the gear ratio of the reducer,
and the diameter of the tires, for example. If these
15 numerical values are measured before shipment, and the data
is stored in the storage unit 602, the velocity can be
calculated simply by measuring the number of revolutions of
the motor at a time of usage.
[0063]
20 A camera unit 605 is formed with a stereo camera, for
example, and can capture three-dimensional images. The
camera unit 605 is mounted on the mobile apparatus 120 via
a camera platform 606. The camera unit 605 in the default
position faces forward with respect to the mobile apparatus
25 120 (or faces in the direction of movement caused by operation
of the movement mechanism unit 603), and primarily captures
first-person viewpoint (FPV) images.
[0064]
The camera platform 606 can operate in the respective
30 directions of rolling, tilting, and panning, and changes the
line of sight of the camera unit 605 in accordance with a
25
sight-line change instruction from the CPU 601. Accordingly,
the camera unit 605 can capture wide-angle images such as
panoramic images, and whole-sky images. Fig. 8A shows a
situation where the camera unit 605 is mounted so that the
camera platform 606 can control the line 5 of sight in the
respective directions of rolling, tilting, and panning.
[0065]
The camera unit 605 can adjust the inter-viewpoint
distance d between the two cameras forming the stereo camera.
10 Fig. 8B shows a situation where the inter-viewpoint distance
d between the left-eye camera and the right-eye camera that
constitute the stereo camera is variable. In this embodiment,
there is no convergence between the left-eye camera and the
right-eye camera, and the respective lines of sight are almost
15 parallel. More preferably, the left-eye camera and the
right-eye camera use deep focus, or perform imaging by
increasing the depth of the object.
[0066]
The camera unit 605 may be formed with two or more
20 cameras, such as a principal camera that captures
first-person viewpoint images, and a secondary camera that
performs imaging from behind the main frame of the mobile
apparatus 120. Fig. 8C shows a situation where a principal
camera 801 is mounted on the main frame of the mobile apparatus
25 120-2 as an automobile, and a secondary camera 802 is mounted
behind the principal camera 801.
[0067]
A communication unit 607 performs a communication
process with external devices such as the head mount display
30 110 and the controller 130, and also performs
modulation/demodulation processes and encoding/decoding
26
processes on communication signals. For example, when the
communication unit 607 receives a movement instruction from
the controller 130, the CPU 601 instructs the movement
mechanism unit 603 to move. The communication unit 607 also
transmits an image captured by the camera 5 unit 605, the
information about the location and the posture of the main
frame of the mobile apparatus 120 detected by the
location/posture detecting unit 604, to the head mount
display 110 and the controller 130.
10 [0068]
In the image display system 100 according to this
embodiment, an image captured by the camera unit 605 mounted
on the mobile apparatus 120 is transferred via the controller
130 or directly to the head mount display 110. Accordingly,
15 the user can enjoy the captured image sent from the mobile
apparatus 120 with the head mount display 110.
[0069]
Also, in the image display system 100 according to this
embodiment, when a wide-angle image such as a panoramic image
20 or a whole-sky image captured by the camera unit 605 mounted
on the mobile apparatus 120 is reproduced and displayed with
the head mount display 110, the CPU 601 moves the display field
so as to offset the motion of the head of the user detected
by the posture/location detecting unit 604. In this manner,
25 an image that follows the motion of the head of the user is
displayed. On the side of the head mount display 110, a
combined image generated by superimposing an AR (Augmented
Reality) image as virtual image information on the actual
image in the display field is displayed as necessary.
30 [0070]
Fig. 7 schematically shows the functional structure of
27
the control unit 401 for displaying an image captured on the
side of the mobile apparatus 120 with the head mount display
110. The functional structure shown in the drawing is
realized by the control unit 401 executing a predetermined
application program, 5 for example.
[0071]
The display field control unit 701 moves the position
and the posture of the display field (see Fig 5) displayed
on the display panel 409 in accordance with the motion of the
10 head of the user detected through the posture/location
detecting unit 404, and outputs the determined display field
to an image clipping unit 702.
[0072]
The image clipping unit 702 clips the image in the
15 display field determined by the display field control unit
701 from the image captured by the mobile apparatus 120 and
received by the communication unit 405, and outputs the
clipped image to an image combining unit 703.
[0073]
20 The image combining unit 703 generates a combined image
by superimposing an AR image on the actual image in the display
field as necessary, and outputs the combined image to the
image processing unit 407 in the later stage. For example,
when the current location of the user is acquired from the
25 posture/location detecting unit 404, or when the current
location of the mobile apparatus 120 is acquired, an AR image
of a guide or obstacle corresponding to the location
information is generated (described later). So as to give
the user feedback of an operational feeling corresponding to
30 the AR image such as collision with obstacle, the image
combining unit 703 instructs the operational feeling feedback
28
unit 412 to output feedback.
[0074]
The image combining unit 703 also performs correction
processing on distortion that appears when the image clipped
from the display field is output to and 5 displayed on the
display panel 409.
[0075]
Fig. 9 illustrates a situation where the position and
the posture of the display field displayed on the display
10 panel 409 are moved in accordance with motion of the head of
the user.
[0076]
When the posture/location detecting unit 404 detects
that the head or the line of sight of the user has moved in
15 a rightward panning direction, the display field control unit
701 moves the display field 910 in the direction of an arrow
denoted by reference numeral 901 in the drawing. As a result,
the image clipping unit 702 clips an image from the wide-angle
image, and the image displayed on the display panel 409 also
20 changes.
[0077]
When the posture/location detecting unit 404 detects
that the head or the line of sight of the user has moved in
a leftward panning direction, the display field control unit
25 701 moves the display field 910 in the direction of an arrow
denoted by reference numeral 902 in the drawing.
[0078]
When the posture/location detecting unit 404 detects
that the head or the line of sight of the user has moved in
30 an upward tilting direction, the display field control unit
701 moves the display field 910 in the direction of an arrow
29
denoted by reference numeral 903 in the drawing.
[0079]
When the posture/location detecting unit 404 detects
that the head or the line of sight of the user has moved in
a downward tilting direction, the display 5 field control unit
701 moves the display field 910 in the direction of an arrow
denoted by reference numeral 904 in the drawing.
[0080]
When the posture/location detecting unit 404 detects
10 that the head or the line of sight of the user has moved in
a rightward rolling direction, the display field control unit
701 moves the display field 910 in the direction of an arrow
denoted by reference numeral 905 in the drawing.
[0081]
15 When the posture/location detecting unit 404 detects
that the head or the line of sight of the user has moved in
a leftward rolling direction, the display field control unit
701 moves the display field 910 in the direction of an arrow
denoted by reference numeral 906 in the drawing.
20 [0082]
Fig. 10 shows an example sequence of operation to
display a wide-angle image such as a whole-sky image captured
on the side of a mobile apparatus 120, the wide-angle image
being made to follow the posture of the head of the user on
25 the side of the head mount display 110. In the operation
sequence shown in the drawing, data communication is
performed directly between the head mount display 110 and the
mobile apparatus 120, but some other apparatus such as the
controller 130 may intervene between the head mount display
30 110 and the mobile apparatus 120.
[0083]
30
The head mount display 110 transmits a captured image
request to the mobile apparatus 120 that is moving while being
remotely controlled by the controller 130 (SEQ1001).
[0084]
In response to the captured image request, 5 the mobile
apparatus 120 performs an image capturing process with the
camera unit 605 while driving the camera platform 606
(SEQ1002). After the captured image is processed, and a
wide-angle image such as a whole-sky image is generated, the
10 image is transmitted to the head mount display 110 (SEQ1003).
However, the mobile apparatus 120 may not perform imaging in
response to the request from the head mount display 110, but
may constantly perform an imaging process, and transmit a
wide-angle image such as a whole-sky image to the head mount
15 display 110 at a predetermined time.
[0085]
At the time of image transmission or some other time,
the mobile apparatus 120 may also transmit information about
the location, the posture, and the velocity of the main frame
20 of the mobile apparatus 120 measured by the
location/posture/velocity detecting unit 604.
[0086]
On the side of the head mount display 110, the
posture/location detecting unit 404 detects the motion of the
25 head of the user, or the direction of the line of sight
(SEQ1004). In accordance with the detected direction of the
line of sight, the position of the display field to be clipped
from the received captured image is controlled, and the
clipped image is displayed on the display panel 409 (SEQ1005).
30 [0087]
Fig. 11 shows a modification of the sequence of
31
operation to display a wide-angle image such as a whole-sky
image captured on the side of a mobile apparatus 120, the
wide-angle image being made to follow the posture of the head
of the user on the side of the head mount display 110. In
the operation sequence shown in 5 the drawing, data
communication is performed directly between the head mount
display 110 and the mobile apparatus 120, but some other
apparatus such as the controller 130 may intervene between
the head mount display 110 and the mobile apparatus 120.
10 [0088]
On the side of the head mount display 110, the
posture/location detecting unit 404 monitors motion of the
head of the user, or the direction of the line of sight
(SEQ1101).
15 [0089]
The head mount display 110 transmits a captured image
request to the mobile apparatus 120 that is moving while being
remotely controlled by the controller 130 (SEQ1102). In
doing so, the head mount display 110 also transmits sight-line
20 information about the user.
[0090]
In response to the captured image request, the mobile
apparatus 120 performs an image capturing process with the
camera unit 605 while driving the camera platform 606
25 (SEQ1103). After the captured image is processed, and a
wide-angle image such as a whole-sky image is generated, an
image in a display field in accordance with the line of sight
of the user is clipped (SEQ1104), and is transmitted to the
head mount display 110 (SEQ1105).
30 [0091]
At the time of image transmission, the mobile apparatus
32
120 may also transmit information about the location, the
posture, and the velocity of the main frame of the mobile
apparatus 120 measured by the location/posture/velocity
detecting unit 604. In the case of the mobile apparatus 120-2,
which is an automobile, the movement 5 velocity can be
calculated from the number of revolutions of the motor, the
gear ratio of the reducer, and the diameter of the tires, for
example. Alternatively, on the side of the head mount display
110, the relative velocity of the mobile apparatus 120 can
10 be measured with the external camera 413.
[0092]
The head mount display 110 then displays the received
image on the display panel 409 (SEQ1106).
[0093]
15 When a wide-angle image such as a whole-sky image
captured on the side of a mobile apparatus 120 is made to follow
the posture of the head of the user and is displayed on the
side of the head mount display 110, it is possible to execute
either the operation sequence in Fig. 10 or the operation
20 sequence in Fig. 11. For ease of explanation, the
descriptions below are based on the assumption that the
operation sequence shown in Fig. 10 is to be executed.
[0094]
B. Image Processing Application
25 The following is a detailed description of the process
to be performed when an image captured on the side of a mobile
apparatus 120 is displayed on the head mount display 110.
[0095]
B-1. Method of Displaying a Three-Dimensional
30 Whole-Sky Image
The camera unit 605 is formed with a stereo camera, as
33
shown in Fig. 8B. There is no convergence between the
left-eye camera and the right-eye camera, the respective
lines of sight are almost parallel, and deep focus imaging
is performed.
5 [0096]
Meanwhile, a human being tends to look at objects far
away when moving at a high speed, but tends to look at nearby
objects when moving at a low speed.
[0097]
10 In view of this, when the head mount display 110 displays
an image capture by a mobile apparatus 120, the convergence
point is adjusted based on the velocity information about the
mobile apparatus 120 in the process performed by the image
combining unit 703, for example.
15 [0098]
Where IL and IR represent images captured by a left-eye
camera and a right-eye camera using a deep focus parallel
method (see Fig. 12A), as the velocity of the mobile apparatus
120 becomes higher, the overlap between the right and left
20 images IL and IR is made smaller, so as to increase the distance
to the convergence point or the point where the lines of sight
of the right and left eyes intersect. In this manner, the
user is made to observe an image that looks clearer at a longer
distance (see Fig. 12B). In conjunction with the adjustment
25 of the convergence point, focus point adjustment is also
performed so that the regions other than the fixed screen
position (the convergence point) are blurred. In this manner,
the visual effect to make the user see faraway objects more
clearly is increased, and the user can experience more
30 realistic sensation of a journey on a high-speed moving
object.
34
[0099]
As the velocity of the mobile apparatus 120 becomes
lower, the overlap between the right and left images IL and
IR is made larger, so as to shorten the distance to the
convergence point or the point where the 5 lines of sight of
the right and left eyes. In this manner, the user is made
to observe an image that is clearer at a shorter distance (see
Fig. 12C). In conjunction with the adjustment of the
convergence point, focus point adjustment is also performed
10 so that the regions other than the fixed screen position are
blurred. In this manner, the visual effect to make the user
see nearby objects more clearly is increased, and the user
can experience more realistic sensation of a journey on a
low-speed moving object.
15 [0100]
In short, with velocity-derived motion of the viewpoint
including the focus point of the person being taken into
account, the person can enjoy natural viewing. When the
velocity of the mobile apparatus 120 becomes higher, the user
20 can see objects further away through the head mount display
110. When the velocity of the mobile apparatus 120 becomes
lower, the user can see nearby objects.
[0101]
B-2. Method of Capturing a Three-Dimensional Whole-Sky
25 Image
In a case where a three-dimensional image for whole-sky
display is to be captured, if partial zooming is performed,
a backdrop-like image is observed as if the user were using
a telescope. This is because the inter-viewpoint distance
30 is fixed. Therefore, on the side of the mobile apparatus 120,
imaging is performed while a stereo camera that can move the
35
inter-viewpoint distance between the right and left cameras
is made to move the inter-viewpoint distance in accordance
with the magnification of the zooming, so as to obtain a
natural three-dimensional whole-sky image. Alternatively,
in a case where the inter-viewpoint distance 5 between the
left-eye camera and the right-eye camera is fixed, the mobile
apparatus 120 or the head mount display 110 may perform image
processing so as to widen the viewpoint, such as extrapolating
the outside of the viewpoints of the cameras.
10 [0102]
Specifically, on the side of the mobile apparatus 120,
the stereo camera is designed to be capable of moving 50 to
100 mm, which is the inter-viewpoint distance d between the
left-eye camera and the right-eye camera (see Fig. 13A). The
15 inter-viewpoint distance is increased at a time of zooming-in,
but is reduced at a time of zooming-out. Where the stereo
camera is installed, this situation is created by making the
inter-viewpoint distance variable with zooming. Fig. 13B
shows a situation where the inter-viewpoint distance d
20 between the left-eye camera and the right-eye camera is
shortened to 50 mm when zooming-out is performed. In this
situation, the head mount display 110 or the controller 130
might directly control the camera unit 605 so that a
three-dimensional image is viewed in real time.
25 Alternatively, images captured with different
inter-viewpoint distances are recorded, and the
inter-viewpoint distance of the image to be reproduced is
changed in synchronization with the zooming operation at the
time of viewing. Also, at the time of viewing, an image with
30 an intermediate inter-viewpoint distance may be generated
through image processing in synchronization with zooming.
36
[0103]
In a case where the inter-viewpoint distance of the
stereo camera is fixed, viewpoint images with different
inter-viewpoint distances are combined through image
processing, and the image being reproduced 5 is switched to the
combined image in synchronization with zooming. In this
manner, zooming as natural as viewpoint motion can be
performed.
[0104]
10 When a three-dimensional whole-sky image captured on
the side of the mobile apparatus 120 is viewed with the head
mount display 110, the viewpoint motion effect can be achieved
by changing the inter-viewpoint distance between the right
and left images being displayed. However, a
15 three-dimensional motion parallax effect with viewpoint
motion is not achieved. So as to achieve a three-dimensional
motion parallax effect, parallaxes need to be generated by
some other means such as image processing, with motion
parallaxes being taken into consideration.
20 [0105]
B-3. Interaction Between the Posture of the Head of a
User and a Captured Image
In a case where the coordinate system of panning,
tilting, and rolling of the head of a user (see Fig. 5) is
25 matched with the coordinate system of panning, tilting, and
rolling of the camera unit 605 (see Fig. 8A), an image of the
display field that follows motion of the head of the user
facing forward shows a lower side while the mobile apparatus
120 is moving. For example, in the case of the mobile
30 apparatus 120-2 that is an automobile, only the ground surface
is shown. Therefore, so as to obtain an image of the scenery
37
ahead of the mobile apparatus 120-2, the user needs to tilt
his/her head in the upward direction and tilt the camera unit
605 mounted on the mobile apparatus 120-2 also in the upward
direction (see Fig. 14). However, if the user continues to
look upward while the mobile apparatus 120-5 2 is moving, the
neck of the user will be tired.
[0106]
To counter this problem, the axis of tilt of the camera
unit 605 mounted on the mobile apparatus 120-2 can be secured
10 in a position that is offset upward by Δθy from the axis of
tilt of the head of the user.
[0107]
For example, where the user is wearing the head mount
display 110 and is looking upward at a desired angle Δθy, Δθy
15 offsetting of the axis of tilt of the camera unit 605 may be
requested through the input operating unit 403. As shown in
Fig. 15, even if the user again faces forward after requesting
the Δθy offsetting, the posture of the camera unit 605 that
interacts with the head of the user remains secured in the
20 position that is offset in the upward tilting direction by
Δθy.
[0108]
Alternatively, a Δθy offsetting instruction for the
axis of tilt of the camera unit 605 may be transmitted from
25 the head mount display 110 to the mobile apparatus 120
directly or via the controller 130. In this case, on the side
of the mobile apparatus 120, a coordinate system offsetting
process is performed on the camera platform 606, for example.
With the axis of tilt being offset, a captured image that
30 follows motion of the head of the user is clipped and
displayed.
38
[0109]
Alternatively, instead of transmission of a coordinate
system offsetting instruction to the mobile apparatus 120,
internal processing in the head mount display 110 can realize
the same process as above. For example, in 5 response to a Δθy
offsetting instruction for the axis of tilt from the user,
the display field control unit 701 sets a display field that
is offset in the upward tilting direction by Δθy from the axis
of tilt of the line of sight of the user detected by the
10 posture/location detecting unit 404.
[0110]
The axis of tilt of the camera unit 605 can of course
be offset by Δθy not only in the upward direction but also
in the downward direction. Likewise, ±Δθx offsetting in the
15 panning direction or ±Δθz offsetting in the rolling direction
(see Fig. 16) can be performed so that a captured image that
follows motion of the head of the user is clipped and
displayed.
[0111]
20 In a case where a mobile apparatus 120 becomes unable
to operate the controller 130, such as when the mobile
apparatus 120-2 as a race car goes spinning out of control
while running, if an image captured by the camera unit 605
that is also spinning is displayed on the side of the head
25 mount display 110 in real time, the scenery being displayed
changes rapidly, and makes the user’s head spin, though the
user has not moved his/her line of sight. Therefore, the
coordinate system of the camera unit 605 is designed so that
the captured image does not change rapidly, regardless of
30 motion of the head of the user. Alternatively, an image
captured at some point (for example, immediately after a start
39
of spinning) is saved, and a display switching process that
follows motion of the line of sight of the user may be performed
on the basis of the coordinate system of the camera unit 605
at the time.
5 [0112]
B-4. Feedback to the User
As described above, the head mount display 110 includes
the operational feeling feedback unit 412 that gives
operational feeling feedback to the user through tactile
10 sensation or vibration. The head mount display 110 is a
device to be worn by a user, and can give tactile sensation
or vibration as effective feedback directly to the user
[0113]
Meanwhile, each mobile apparatus 120 is equipped with
15 an acceleration sensor as the location/posture/velocity
detecting unit 604, and can detect oscillation and impact to
which the apparatus is subjected while flying, running, or
sailing.
[0114]
20 In view of this, while displaying an image captured on
the side of the mobile apparatus 120, the head mount display
110 constantly monitors the value detected by the
acceleration sensor of the mobile apparatus 120, and
instructs the operational feeling feedback unit 412 to output
25 feedback in accordance with the oscillation and impact to
which the mobile apparatus 120 is subjected.
[0115]
When the mobile apparatus 120 collides with obstacle
that does not exist in the real word but is displayed in AR
30 by the image combining unit 703, the operational feeling
feedback unit 412 is also instructed to output feedback in
40
accordance with the virtual impact.
[0116]
B-5. AR Image Display
In conventional FPV, an image captured by a camera
mounted on a radio-controlled model is displayed 5 as it is.
In this embodiment, on the other hand, the head mount display
110 superimposes a non-existent AR image on an image captured
on the side of a mobile apparatus 120, and then displays the
superimposed image.
10 [0117]
In the case of the mobile apparatus 120-1 that is a
flying object such as an aircraft or a helicopter, an AR image
of a pylon or the like is superimposed on the captured image
to be displayed. In the case of the mobile apparatus 120-2
15 for traveling on land such as an automobile (a race car), an
AR image of a race course such as a circuit is superimposed
on the captured image to be displayed. In the case of the
mobile apparatus 120-3 that is a watercraft such as a yacht,
an AR image of a buoy is superimposed on the captured image
20 to be displayed.
[0118]
In a case where the site of an AR image is registered
in advance, the head mount display 110 constantly monitors
the current location of the mobile apparatus 120 while
25 displaying a captured image. When the mobile apparatus 120
reaches the site (or the site is shown in the captured image),
an AR image display process starts. Not only a guidepost for
movement of the mobile apparatus 120 such as a pylon or a buoy,
but also virtual obstacle that hinders movement of the mobile
30 apparatus 120 or a prohibited area such as an area dangerous
for the mobile apparatus 120 to pass through or the premises
41
of other people can be shown in an AR image. Fig. 28 shows
a situation where AR images of virtual obstacle 2801 and a
prohibited area 2802 are displayed on a car-sight-line image
(the real word) captured by the mobile apparatus 120-2.
5 [0119]
In a case where an object to be displayed in an AR image
is registered in advance, the head mount display 110
recognizes objects shown in a captured image sent from the
mobile apparatus 120. When the registered object is found,
10 the AR image display process is started. The registered
object may be a product, a person, a pet animal, a plant, a
building, or the like the user is looking for.
[0120]
An AR image of an object is displayed in a case where
15 a product or the like the user is looking for is emphasized,
or where the object (that should be hidden from the user) is
erased from the screen.
[0121]
In a case where an AR image display state is registered
20 in advance, the head mount display 110 monitors the current
state of the mobile apparatus 120. When the mobile apparatus
120 enters the state registered beforehand, the AR image
display process is started. An example of the state mentioned
above is a state where the radio waves received from the
25 controller 130 become weaker, and control cannot be performed.
In such a case, the area where the radio waves cannot be
received is set as a “prohibited area”, and guidance for
preventing the mobile apparatus 120 from entering the
prohibited area is displayed as an AR image. In an operation
30 of the image display system 100, it is of course possible to
provide actual guidance such as a signboard indicating the
42
prohibited area.
[0122]
Furthermore, if the mobile apparatuses 120 are located
in the same positions, are the same objects, or are in the
same state, different AR images may be displayed 5 in accordance
with the characteristics (age, nationality, sex, personality,
or the like) of the respective users or the skills of the
respective users operating the mobile apparatuses 120. For
example, it is expected that different AR images are required
10 for an adult user and a child user. Also, it is preferable
to display an AR image to a beginner sooner than to a
highly-skilled person.
[0123]
B-6. Display of Relative Location of the User
15 In the image display system 100 according to this
embodiment, a user can observe a captured image with the head
mount display 110 while moving a mobile apparatus 120 by
operating the controller 130. The user can enjoy the scenery
of a faraway place to which the mobile apparatus 120 has moved,
20 though the user has not moved at all.
[0124]
In doing so, the user can recognize the place of the
imaging by monitoring the location information that is output
from the location/posture/velocity detecting unit 604 of the
25 mobile apparatus 120. However, even if the user wishes to
go to the place, he/she cannot instantly decide which
direction to go simply by looking at a captured image.
[0125]
In view of this, the head mount display 110 can display
30 an AR image indicating the positional relationship between
the user and the mobile apparatus 120 in a captured image sent
43
from the mobile apparatus 120. As the head mount display 110
displays the location information about the user, the user
can obtain a navigation function as well as enjoying FPV.
[0126]
The display image of the positional 5 relationship can
be generated in the head mount display 110 by the image
combining unit 703 based on the location information about
the user detected by the location/posture detecting unit 404,
and the location information about the mobile apparatus 120
10 detected by the location/posture/velocity detecting unit 604
of the mobile apparatus 120.
[0127]
In a case where the location information about the user
is displayed, for example, the location information about the
15 user may be displayed as a small screen in the screen
displaying an image captured by the mobile apparatus 120, as
shown in Fig. 17. The small screen shows a map image, for
example, and respective icons indicating the current location
of the mobile apparatus 120 and the current location of the
20 user are shown in the image. Alternatively, depending on a
user operation through the input operating unit 402, the
location information about the user may be displayed on the
large screen, and a captured image may be displayed on the
small screen, as shown in Fig. 18.
25 [0128]
In a case where the location information about the user
is displayed, north is located on the upper side as shown in
Figs. 17 and 18 (North Up), the mobile apparatus 120 is moving
upward on the display as shown in Fig. 19, or the line of sight
30 of the user is in the upward direction on the display as shown
in Fig. 20. The moving direction of the mobile apparatus 120
44
can be detected by the location/posture/velocity detecting
unit 604. The direction of the line of sight of the user can
be detected by the posture/location detecting unit 404 or the
state detecting unit 411.
5 [0129]
In each of the examples shown in Figs. 17 through 20,
the user and the mobile apparatus 120 are placed on the map
screen, so that the respective absolute locations can be
displayed. However, the map screen may not be used, and the
10 relative locations of the user and the mobile apparatus 120
may be displayed.
[0130]
In a case where the mobile apparatus 120 is a submarine
or the like that travels under water, it might be difficult
15 to receive GPS signals. Therefore, absolute location
information may be acquired by a substitute means such as
underwater acoustic communication.
[0131]
B-7. Automatic Tracking of a Mobile Apparatus
20 In the above described embodiments, so-called FPV
(first-person viewpoint image) captured by a mobile apparatus
120 is displayed on the head mount display 110 and is enjoyed.
Some users wish to enjoy scenery including a mobile apparatus
120.
25 [0132]
In view of this, an automatic tracker that tracks the
rear (or a side) of a mobile apparatus 120 may be provided,
and an image captured by the automatic tracker may be
displayed on the head mount display 110.
30 [0133]
Fig. 21 shows a situation where an automatic tracker
45
2100 equipped with a camera 2101 is tracking the rear of the
mobile apparatus 120-1 flying in the air or the mobile
apparatus 120-2. Fig. 22 shows an example of an image of the
mobile apparatus 120-1 captured from behind by the camera 2101
of the automatic 5 tracker 2100.
[0134]
The automatic tracker 2100 may not fly as shown in Fig.
21, and lower costs can be realized by employing a secondary
camera that captures images of a mobile apparatus 120 from
10 behind as shown in Fig. 8C.
[0135]
B-8. Screen Switching
In the above described embodiments, an image of the
scenery of a remote place captured by a mobile apparatus 120
15 (or an image captured by the automatic tracker 2100) is
displayed on the head mount display 110 and is enjoyed.
[0136]
Meanwhile, the external camera 413 is placed at almost
the center of the front surface of the main frame of the head
20 mount display 110 having an eyeglass-like shape or a hat-like
shape (see Fig. 2), and can capture images of the surroundings.
Also, posture control in the panning, tilting, and rolling
directions of the external camera 413 is performed in
accordance with the direction of the user’s line of sight
25 detected by the state information detecting unit 411, so that
an image on the level of the user’s line of sight can be
captured with the external camera 413.
[0137]
In view of this, the screen of the display panel 409
30 may be switched between an image captured by a mobile
apparatus 120 and an image captured by the external camera
46
413 in accordance with a user operation or the like through
the input operating unit 402. Fig. 23 shows an example of
a car-sight-line image captured by the mobile apparatus 120-2
as an automobile. Fig. 24 shows an example of a situation
where a user wearing the head mount display 5 110 follows the
running mobile apparatus 120-3 with his/her eyes. Fig. 25
shows an example of a self-sight-line image of the user who
is following the mobile apparatus 120-3 with his/her eyes.
The self-sight-line image is captured by the external camera
10 413.
[0138]
The camera unit 605 mounted on a mobile apparatus 120
is at least a single camera, and basically performs imaging
from the first-person viewpoint of the operator. However,
15 the camera unit 605 may further include cameras that perform
imaging from other viewpoint positions.
[0139]
For example, the mobile apparatus 120-2 as an
automobile is equipped with cameras that include a camera that
20 captures an image on the line of sight of a car in front in
the traveling direction, a camera that captures images of
scenery reflected by the right and left sideview mirrors, a
camera that captures an image of scenery reflected by the
rearview mirror, and a camera that captures an image of
25 scenery reflected when the user looks at meters and gauges.
Alternatively, the mobile apparatus 120-2 may be equipped
with a single camera, but the camera platform 606 may move
the viewpoint position of the camera.
[0140]
30 In this case, images captured by the cameras of the
respective viewpoint positions are arranged as shown in Fig.
47
26, for example. In the example shown in the drawing, a
captured image on the line of sight of the car is placed at
the center, captured images on the lines of sight of the right
and left sideview mirrors are placed to the right and left
of the car-sight-line image, an image on 5 the line of sight
of the rearview mirror is placed above the car-sight-line
image, and a capture image on the line of sight of an eye
looking at the meters and gauges is placed below the
car-sight-line image. On the side of the head mount display
10 110, the display field is not moved in accordance with motion
of the head of the user, but the image displayed on the display
panel 409 is switched to the left-sideview-mirror sight-line
image when the head turns to the left, is switched to the
right-sideview-mirror sight-line image when the head turns
15 to the right, is switched to the rearview-mirror sight-line
image when the head tilts upward, and is switched to the meter
sight-line image when the head tilts downward. When the head
of the user again faces forward, the image displayed on the
display panel 409 is returned to the car-sight-line image.
20 [0141]
B-9. Image Correction
In a case where a wide-angle captured image such as a
whole-sky image is being viewed in a narrow display field,
when the user tilts his/her head, the display field control
25 unit 701 moves the display field upward or downward in
accordance with the tilting, and the image clipping unit 702
clips a display image in the moved display field (see Fig.
9). When the user tilts his/her head upward and moves the
display field upward, the distance from the projecting plane
30 is longer at the upper edge of the display field. Therefore,
the display image has a trapezoidal shape with a shorter upper
48
edge than the upper edge of the display frame of the display
panel 409. Likewise, when the user tilts his/her head
downward, the distance from the projecting plane is longer
at the lower edge of the display field. Therefore, the
display image has a trapezoidal shape with 5 a shorter lower
edge than the lower edge of the display frame of the display
panel 409 (see Fig. 27). A trapezoidal image causes a feeling
of strangeness in the user, who is viewing the image.
[0142]
10 In view of this, the image combining unit 703 performs
trapezoidal distortion correction so that the image being
viewed by the user always has a rectangular shape.
[0143]
B-10. Image Stabilizing Process
15 In the case of the mobile apparatus 120-2 as an
automobile, for example, a captured image might shake
vertically or horizontally due to the influence of uneven road
surface on which the mobile apparatus 120-2 is running. In
the case of the mobile apparatus 120-1 as a flying object such
20 as an aircraft or a helicopter, a captured image might shake
horizontally due to the influence of an airstream or the like.
In the case of the mobile apparatus 120-3 as a watercraft such
as a yacht, a captured image shakes due to the influence of
a stream or waves while sailing.
25 [0144]
When a person is watching an image that is shaking, the
person feels as if he/she were getting seasick. In general,
erratic shaking with slow waves and large amplitude easily
makes a person feel sick.
30 [0145]
When there is shaking in an image captured by a mobile
49
apparatus 120, the image may be displayed only after the
shaking is canceled on the side of the head mount display 110.
[0146]
When erratic shaking with slow waves and large
amplitude is detected by monitoring the value 5 detected by the
acceleration sensor on the side of the mobile apparatus 120,
for example, an image shaking correction process is started
on the side of the head mount display 110.
[0147]
10 For example, the display field control unit 701 inputs
the value detected by the acceleration sensor on the side of
the mobile apparatus 120, adds the detected value to the
posture information about the head of the user, determines
the position and the posture of the display field so as to
15 cancel the shaking in the mobile apparatus 120, and outputs
the position and the posture to the image clipping unit 702.
[0148]
Alternatively, the image clipping unit 702 inputs the
value detected by the acceleration sensor on the side of the
20 mobile apparatus 120, corrects the position and the posture
of the display field determined by the display field control
unit 701 based on the posture information about the head of
the user so as to cancel the shaking in the mobile apparatus
120, and clips an image in the display field from a captured
25 image.
[0149]
The image combining unit 703 or the like can perform
the shaking correction process by turning a motion image into
a slow-motion image or performing frame thinning. In a case
30 where a motion image is turned into a slow-motion image, only
the portions with motion in the frames may be turned into
50
slow-motion image portions.
[0150]
In the shaking correction process, shaking may be
prevented not by performing the above described image
processing but by reducing the screen size 5 only while shaking
is being detected in the mobile apparatus 120, with attention
being paid to the fact that a person easily feels sick with
a large screen. The virtual image optical unit 410 forms an
enlarged virtual image of an image displayed on the display
10 panel 409, as described above. The virtual image optical unit
410 may adjust magnification, or the display panel 409 may
reduce the display size.
[0151]
The optical system of the camera unit 605 may of course
15 be provided on the side of the mobile apparatus 120, or image
processing may of course have a shaking correcting function.
[0152]
C. Recapitulation
The technology disclosed in this specification may be
20 embodied in the structures described below.
(1) An image display apparatus including:
a display unit mounted on the head of a user;
a posture detecting unit that detects posture of the
head; and
25 a display control unit that controls display of an image
on the display unit based on the posture of the head, the image
being captured by a mobile apparatus.
(2) The image display apparatus of (1), wherein the
display control unit clips a region corresponding to the
30 posture of the head from a wide-angle image captured by the
mobile apparatus, and displays the clipped region on the
51
display unit.
(3) The image display apparatus of (1), wherein, when
displaying a plurality of viewpoint images captured at a
plurality of viewpoints by a deep focus parallel method, the
display control unit adjusts the convergence 5 point between
the viewpoint images based on the movement velocity of the
moving object.
(4) The image display apparatus of (1), wherein the
display control unit displays an image with an
10 inter-viewpoint distance corresponding to a zooming
operation on the display unit.
(5) The image display apparatus of (1), wherein the
direction of the line of sight of a camera unit of the mobile
apparatus is offset in at least one of the directions of
15 panning, tilting, and rolling with respect to the posture of
the head.
(6) The image display apparatus of (1), further
including
an operational feeling feedback unit that feeds back
20 an operational feeling to the user through tactile sensation
or vibration,
wherein the feedback to the user is based on the
acceleration to which the mobile apparatus is subjected while
moving.
25 (7) The image display apparatus of (1), wherein the
display control unit superimposes an AR image on a real-world
image captured by the mobile apparatus, before displaying the
image.
(8) The image display apparatus of (7), wherein the
30 display control unit displays the AR image corresponding to
at least one of the current location of the mobile apparatus,
52
an object included in the captured image, and the state of
the mobile apparatus.
(9) The image display apparatus of (1), wherein the
display control unit displays location information about the
mobile apparatus 5 and the user.
(10) The image display apparatus of (1), wherein the
display control unit further displays an image captured by
an automatic tracker that captures the image while tracking
the mobile apparatus.
10 (11) The image display apparatus of (1), further
including
a self-sight-line image acquiring unit that acquires
a self-sight-line image to be seen on the line of sight of
the user,
15 wherein the display control unit switches the image
being displayed between a moving-object sight-line image
captured by the mobile apparatus and the self-sight-line
image.
(12) The image display apparatus of (1), wherein the
20 display control unit switches the image being displayed
between images captured from a plurality of viewpoint
positions of the mobile apparatus in accordance with the
posture of the head.
(13) The image display apparatus of (1), wherein the
25 display control unit corrects shaking in a moving image
captured by the mobile apparatus, before displaying the
moving image.
(14) An image display method including:
a posture detecting step of detecting posture of the
30 head of a user; and
a display control step of controlling display of an
53
image based on the posture of the head, the image being
captured by a mobile apparatus.
(15) An image display system including:
a mobile apparatus that captures an image while moving;
5 and
an image display apparatus that displays the image
captured by the mobile apparatus in accordance with posture
of the head of a user.
(16) A mobile apparatus including:
10 a camera unit;
a camera platform that controls the direction of the
line of sight of the camera unit;
a moving unit that moves the apparatus; and
a communication unit that communicates data including
15 an image captured by the camera unit,
wherein the camera unit includes a plurality of cameras
that perform imaging by a deep focus parallel method, the
cameras having different viewpoints.
(17) The mobile apparatus of (16), which captures a
20 whole-sky image while changing the inter-viewpoint distance
between the cameras with different viewpoints.
(18) The mobile apparatus of (16), which extrapolates
images outside the viewpoints of the cameras from images
captured by the cameras with different viewpoints at a fixed
25 inter-viewpoint distance from each other.
(19) A computer program written in a computer-readable
format,
the computer program causing a computer to function as:
a posture detecting unit that detects posture of the
30 head of a user; and
a display control unit that controls display of an image
54
based on the posture of the head, the image being captured
by a mobile apparatus.
INDUSTRIAL APPLICABILITY
5 [0153]
The technology disclosed in this specification has been
described in detail, with reference to specific embodiments.
However, it is obvious that those skilled in the art can make
modifications to and substitutions of the embodiments without
10 departing from the scope of the technology disclosed in this
specification.
[0154]
In this specification, embodiments in which the
technology disclosed in this specification is applied to a
15 system formed with a head mount display to be mounted on the
head or the face of a user and radio-controlled mobile
apparatuses have been mainly described. However, the scope
of the technology disclosed in this specification is not
limited to the structure of a specific image display
20 apparatus.
[0155]
For example, the technology disclosed in this
specification can also be applied to various types of display
systems that display wide-angle images captured by cameras
25 mounted on various types of remotely-controlled mobile
apparatuses or cameras mounted on remotely-guided living
creatures such as birds, mammals, reptiles, amphibians, fish,
insects, in accordance with the direction of the line of sight
of a user.
30 [0156]
The mobile apparatuses are radio-controlled models of
55
an aircraft, a helicopter, an automobile, a yacht, and the
like, but may be actual mobile apparatuses such as an aircraft,
a helicopter, an automobile, and a yacht. As long as the
postures (panning, tilting, and rolling) of the mounted
cameras can be remotely controlled, the 5 mobile apparatuses
may not be remotely-operated or remotely-guided. The mobile
apparatuses may not be moving objects that are mechanical
devices, but may be living moving objects such as human beings
or animals.
10 [0157]
In short, the technology disclosed in this
specification has been described through examples, and the
descriptions in this specifications should not be interpreted
in a restrictive manner. The claims should be taken into
15 account in understanding the subject matter of the technology
disclosed in this specification.
REFERENCE SIGNS LIST
[0158]
20 100 Image display system
110 Head mount display
120 Mobile apparatus
130 Controller
201, 202 Microphone
25 401 Control unit
401A ROM
401B RAM
402 Input operating unit
403 Remote control receiving unit
30 404 Posture/location detecting unit
405 Communication unit
56
406 Storage unit
407 Image processing unit
408 Display drive unit
409 Display panel
410 Virtual 5 image optical unit
411 State detecting unit
412 Operational feeling feedback unit
413 External camera
601 Control unit
10 602 Storage unit
603 Movement mechanism unit
604 Location/posture/velocity detecting unit
605 Camera unit
606 Camera platform
15 607 Communication unit
701 Display field control unit
702 Image clipping unit
703 Image combining unit
57
CLAIMS
1. An image display apparatus comprising:
a display unit mounted on a head of a user;
a posture detecting unit configured 5 to detect posture
of the head; and
a display control unit configured to control display
of an image on the display unit based on the posture of the
head, the image being captured by a mobile apparatus.
10
2. The image display apparatus according to claim 1,
wherein the display control unit clips a region corresponding
to the posture of the head from a wide-angle image captured
by the mobile apparatus, and displays the clipped region on
15 the display unit.
3. The image display apparatus according to claim 1,
wherein, when displaying a plurality of viewpoint images
captured at a plurality of viewpoints by a deep focus parallel
20 method, the display control unit adjusts a convergence point
between the viewpoint images based on a movement velocity of
the moving object.
4. The image display apparatus according to claim 1,
25 wherein the display control unit displays an image with an
inter-viewpoint distance corresponding to a zooming
operation on the display unit.
5. The image display apparatus according to claim 1,
30 wherein a direction of a line of sight of a camera unit of
the mobile apparatus is offset in at least one of directions
58
of panning, tilting, and rolling with respect to the posture
of the head.
6. The image display apparatus according to claim 1,
5 further comprising
an operational feeling feedback unit configured to feed
back an operational feeling to the user through tactile
sensation or vibration,
wherein the feedback to the user is based on an
10 acceleration to which the mobile apparatus is subjected while
moving.
7. The image display apparatus according to claim 1,
wherein the display control unit superimposes an AR image on
15 a real-world image captured by the mobile apparatus, before
displaying the image.
8. The image display apparatus according to claim 7,
wherein the display control unit displays the AR image
20 corresponding to at least one of a current location of the
mobile apparatus, an object included in the captured image,
and a state of the mobile apparatus.
9. The image display apparatus according to claim 1,
25 wherein the display control unit displays location
information about the mobile apparatus and the user.
10. The image display apparatus according to claim 1,
wherein the display control unit further displays an image
30 captured by an automatic tracker that captures the image while
tracking the mobile apparatus.
59
11. The image display apparatus according to claim 1,
further comprising
a self-sight-line image acquiring unit configured to
acquire a self-sight-line image to be seen 5 on a line of sight
of the user,
wherein the display control unit switches the image
being displayed between a moving-object sight-line image
captured by the mobile apparatus and the self-sight-line
image.
12. The image display apparatus according to claim 1,
wherein the display control unit switches the image being
displayed between images captured from a plurality of
15 viewpoint positions of the mobile apparatus in accordance
with the posture of the head.
13. The image display apparatus according to claim 1,
wherein the display control unit corrects shaking in a moving
image captured by the mobile apparatus, before displaying the
moving image.
14. An image display method comprising:
a posture detecting step of detecting posture of a head
of a user; and
a display control step of controlling display of an
image based on the posture of the head, the image being
captured by a mobile apparatus.
30 15. An image display system comprising:
a mobile apparatus configured to capture an image while

moving; and
an image display apparatus configured to display the
image captured by the mobile apparatus in accordance with
posture of a head of a user.

16. A mobile apparatus comprising:
a camera unit;
a camera platform configured to control a direction of
a line of sight of the camera unit;
a moving unit configured to move the apparatus; and
a communication unit configured to communicate data
including an image captured by the camera unit,
wherein the camera unit includes a plurality of cameras
configured to perform imaging by a deep focus parallel method,
the cameras having different viewpoints.
17. The mobile apparatus according to claim 16, which
captures a whole-sky image while changing an inter-viewpoint
distance between the cameras with different viewpoints.

18. The mobile apparatus according to claim 16, which
extrapolates images outside the viewpoints of the cameras
from images captured by the cameras with different viewpoints
at a fixed inter-viewpoint distance from each other.

19. A computer program written in a computer-readable
format,
the computer program causing a computer to function as:
a posture detecting unit configured to detect posture
30 of a head of a user; and
a display control unit configured to control display

of an image based on the posture of the head, the image being
captured by a mobile apparatus.