HEAD-MOUNTED DISPLAY DEVICE,
CONTROL METHOD OF HEAD-MOUNTED DISPLAY DEVICE,
AND DISPLAY SYSTEM
Technical Field
[0001] The present invention relates to a head-mounted display device.
Background Art
[0002] A head-mounted display device is known which is mounted on a viewer's head for
use and forms a virtual image in the viewer's viewing region (see PTL 6). Such a headmounted
display device is also called head-mounted display (HMD) and is classified
into a non-transmissive head-mounted display device that blocks a user's viewing field
in a state where the user wears the head-mounted display device and a transmissive
head-mounted display device that does not block a user's viewing field in a state where
the user wears the head-mounted display device.
[0003] On the other hand, a technique called augmented reality (AR) is known which adds
information to a real environment using a computer. As a method of realizing the
augmented reality, a method based on image recognition and a method based on a
pass-through type are known. In the case of the method based on image recognition, a
virtual object is generated, for example, by recognizing an image of an external scene
imaged with a WEB camera. In the case of the method based on a pass-through type, a
virtual object is generated using current position information acquired, for example, by
a GPS, and azimuth information acquired, for example, by an electronic compass. In a
non-transmissive head-mounted display device, an image in which the image of the
external scene and the virtual object generated in the above-mentioned way overlap
with each other is displayed on a liquid crystal display. Accordingly, a user can feel
augmented reality. In a transmissive head-mounted display device, only the virtual
object generated in the above-mentioned way is displayed on a liquid crystal display. A
user can feel augmented reality by visually recognizing both of the virtual object
displayed as a virtual image on the liquid crystal display and the actual external scene
displayed via lenses in front of the eyes. Accordingly, in order to provide augmented
reality with a user's visual discomfort reduced in an optical transmissive head-mounted
display device, it is necessary to fit the virtual object displayed as a virtual image to the
external scene actually viewed by the user.
[0004] PTL 1 describes a technique of setting a distance from a user to a real object included
in an external scene and a distance from the user to a virtual object to the substantially
same distance in order to fit the virtual object to the external scene. In PTL 1, the
distance D between a user and a real object is calculated, a convergence angle theta of
the real object is determined from the distance D, the convergence angle of the virtual
object which is the convergence angle theta of the real object plus-minus 40 arc-minute
is computed, and right-eye image data and left-eye image data realizing the computed
convergence angle are generated. PTL 2 describes a technique of capable of adjusting a
convergence angle of a virtual object in order to fit the virtual object to an external
scene. PTL 3 describes a technique of adjusting a camera for capturing an image of an
external scene using imaging parameters measured in advance in order to provide
augmented reality with a user's visual discomfort reduced in a non-transmissive headmounted
display device.
[0005] PTL 4 and PTL 5 describe a technique of trimming a region of a specific color such
as a beige color from an image in a user's eyeline captured with a camera and masking
a virtual object in order to reduce discomfort when the virtual object overlaps with an
external scene.
Citation List
Patent Literature
[0006] PTL 1: Japanese Patent No. 37 17653
PTL 2: JP-A-5-328408
PTL 3: JP-A-2008-227865
PTL 4: JP-A-2005-346468
PTL 5: JP-A-2003-296759
PTL 6: JP-A-2012- 163637
Summary of Invention
Technical Problem
[0007] In order to stereoscopically display a virtual image of a virtual object with a target
convergence angle in a head-mounted display device, it is necessary to consider
various conditions of a display environment of a virtual image, such as the size of a
liquid crystal display, a distance between virtual images displayed in front of a user's
right and left eyes, and a user's interocular distance. However, in the techniques
described in PTL 1 and PTL 2, there is a problem in that this point is not sufficiently
considered. The technique described in PTL 3 is not directed to an optical transmissive
head-mounted display device. Accordingly, there is a need for an optical transmissive
head-mounted display device that can provide augmented reality with a user's visual
discomfort reduced by fitting a virtual object to an external scene in consideration of
various conditions on a display environment of a virtual image.
[0008] In the techniques described in PTL 4 and PTL 5, since a virtual object is processed
on the basis of an image captured with a camera, there is a problem in that processing
accuracy of a virtual object cannot be secured for plural objects having a complex an
teroposterior relationship or objects having similar colors, which can be viewed in a
real world. Accordingly, there is a need for an optical transmissive head-mounted
display device that can provide augmented reality with a user's visual discomfort
reduced by fitting a virtual object to an external scene with higher accuracy.
[0009] By employing only the configuration in which a virtual object is displayed for visual
recognition based on a coordinate system of a real space, there is a problem in that the
virtual object behaves in disregard of a real object. For example, when a virtual object
is present in a real space, the virtual object passes through a real object in a case where
the virtual object collides with the real object. Even in a case where the virtual object is
hidden by the real object from a user's point of view, the virtual object is recognized.
These phenomena serve as a reason for giving discomfort to a user and markedly
damaging a sense of reality.
[0010] Therefore, there is a need for an optical transmissive head-mounted display device
that can provide augmented reality with a user's visual discomfort reduced.
Solution to Problem
[001 1] An advantage of some aspects of the invention is to solve at least a part of the
problems described above and the invention can be implemented as the following
aspects.
[0012] (1) An aspect of the invention provides a head-mounted display device that enables a
user to visually recognize a virtual image and an external scene. The head-mounted
display device includes: an image display unit that causes the user to visually
recognize the virtual image; and an augmented reality processing unit that forms on the
image display unit the virtual image indicating a virtual object which is an object for
providing augmented reality to the user and of which the user's visual discomfort is
reduced. In the head-mounted display device according to this aspect, the image
display unit causes the user to visually recognize the virtual object, of which the user's
visual discomfort is reduced, as a virtual image. Accordingly, it is possible to provide
an optical transmissive head-mounted display device which can provide augmented
reality with a user's visual discomfort reduced.
[0013] (2) In the head-mounted display device according to the aspect, the augmented
reality processing unit may reduce the discomfort of the virtual object by fitting the
virtual object with an ambient environment of the user using three-dimensional in
formation in which the ambient environment is expressed as a virtual three-di
mensional object in a three-dimensional space. In the head-mounted display device
according to this aspect, the augmented reality processing unit fits the virtual object
with the ambient environment. Accordingly, the image display unit can cause the user
to visually recognize the virtual object, of which the user's visual discomfort is
reduced, as a virtual image. The augmented reality processing unit fits the virtual
object with the ambient environment using the three-dimensional information. Ac
cordingly, it is possible to improve fitting accuracy in comparison with a case where
the virtual object is fitted with the ambient environment on the basis of an image
captured with a camera.
[0014] (3) In the head-mounted display device according to the aspect, the augmented
reality processing unit may arrange the virtual object in the three-dimensional in
formation and may reduce the discomfort of the virtual object by adding a visual effect
based on the environment to at least one of the virtual object and the three-dimensional
information and then converting the virtual object into a two-dimensional image. In the
head-mounted display device according to this aspect, the augmented reality
processing unit can add the visual effect based on the ambient environment to the
virtual object using the three-dimensional information.
[0015] (4) In the head-mounted display device according to the aspect, the visual effect
based on the environment may include any one of: trimming a portion of the virtual
object which is a shadow of the three-dimensional object in the three-dimensional in
formation; lighting the virtual object based on the environment; and adjusting behavior
of the virtual object based on at least one of a restitution coefficient and a friction co
efficient of the three-dimensional object in the three-dimensional information set on
the basis of the environment. In the head-mounted display device according to this
aspect, the augmented reality processing unit can add at least one visual effect of the
trimming based on the environment, the lighting based on the environment, and the
adjusting of behavior based on the environment to the virtual object.
[0016] (5) The head-mounted display device according to the aspect may further include an
image acquiring unit that acquires an image in the user's viewing field in a state where
the user wears the head-mounted display device, and the augmented reality processing
unit may estimate the environment by recognizing the image in the viewing field
acquired by the image acquiring unit. In the head-mounted display device according to
this aspect, the augmented reality processing unit can automatically estimate the
ambient environment of the user by recognizing an image in the user's viewing field.
[0017] (6) In the head-mounted display device according to the aspect, the augmented
reality processing unit may consider a distance from the user's eyes to the user's point
of regard to convert the virtual object into a two-dimensional image. In the headmounted
display device according to this aspect, the augmented reality processing unit
considers the distance from the user's eyes to the user's point of regard at the time of
conversion of the virtual object into a two-dimensional image. Accordingly, it is
possible to add a defocus effect to, for example, a virtual object at a position deviated
from the user's point of regard.
[0018] (7) In the head- mounted display device according to the aspect, the augmented
reality processing unit may add a visual effect of enabling a stereoscopic view of the
virtual object to the virtual object to reduce the discomfort of the virtual object by
generating right-eye image data for a right eye and left-eye image data for a left eye
which indicate the virtual object, the image display unit may cause the user's right and
left eyes to visually recognize the different virtual images using the right-eye image
data and the left-eye image data, the head-mounted display device may further include
a pixel parallactic angle storage unit that stores a difference between a first con
vergence angle of the virtual image displayed on the basis of the right-eye image data
and the left-eye image data which are identical to each other and a second convergence
angle of the virtual image displayed on the basis of the right-eye image data and the
left-eye image data which are laterally deviated from each other, and the augmented
reality processing unit may generate the right-eye image data and the left-eye image
data for fitting the virtual object to the external scene using the difference stored in the
pixel parallactic angle storage unit. In the head-mounted display device according to
this aspect, the difference stored in the pixel parallactic angle storage unit is a
difference between the first convergence angle of the virtual image displayed on the
basis of the right-eye image data and the left-eye image data which are identical to
each other and the second convergence angle of the virtual image displayed on the
basis of the right-eye image data and the left-eye image data which are laterally
deviated from each other. Accordingly, the difference stored in the pixel parallactic
angle storage unit can be said to be a parallactic angle which is embodied by the image
data which is laterally deviated from each other and which is determined in con
sideration of various conditions of the display environment of the virtual image.
Therefore, the augmented reality processing unit can generate the right-eye image data
and the left-eye image data for fitting the virtual object to the external scene in con
sideration of various conditions of the display environment of the virtual image using
the difference stored in the pixel parallactic angle storage unit. In this way, since the
augmented reality processing unit fits the virtual object to the external scene in con
sideration of various conditions of the display environment of a virtual image at the
time of generating the image data, it is possible to provide a head-mounted display
device which can provide augmented reality with a user's visual discomfort reduced.
[0019] (8) In the head-mounted display device according to the aspect, the augmented
reality processing unit may determine a target distance at which the user is caused to
visually recognize the virtual object, calculate a target convergence angle, which is a
convergence angle at the target distance, from the determined target distance, calculate
a target parallactic angle which is a parallactic angle at the target distance using the
calculated target convergence angle and the first convergence angle, and generate the
right-eye image data and the left-eye image data from single image data indicating the
virtual object using the target parallactic angle and the difference stored in the pixel
parallactic angle storage unit. In the head-mounted display device according to this
aspect, the augmented reality processing unit determines the target distance at which
the user is caused to visually recognize the virtual object, calculates the target con
vergence angle which is a convergence angle at the target distance from the determined
target distance, and calculates the target parallactic angle which is a parallactic angle at
the target distance using the calculated target convergence angle and the first con
vergence angle. The augmented reality processing unit can generate the right-eye
image data and the left-eye image data which are deviated from each other from single
image data indicating the virtual object by using the calculated target parallactic angle
and the difference stored in the pixel parallactic angle storage unit.
[0020] (9) In the head-mounted display device according to the aspect, the augmented
reality processing unit may determine a target distance at which the user is caused to
visually recognize the virtual object, set two virtual cameras for acquiring a 2D
projection image in a 3D model space on the basis of the determined target distance,
calculate a target convergence angle, which is a convergence angle at the target
distance, from the determined target distance, calculate a target parallactic angle which
is a parallactic angle at the target distance using the calculated target convergence
angle and the first convergence angle, and generate the right-eye image data by scaling
the 2D projection image captured by one virtual camera and may generate the left-eye
image data by scaling the 2D projection image captured by the other virtual camera
using the target parallactic angle and the difference stored in the pixel parallactic angle
storage unit. In the head-mounted display device according to this aspect, the
augmented reality processing unit determines the target distance at which the user is
caused to visually recognize the virtual object, calculates the target convergence angle
which is a convergence angle at the target distance from the determined target distance,
and calculates the target parallactic angle which is a parallactic angle at the target
distance using the calculated target convergence angle and the first convergence angle.
The augmented reality processing unit can generate the right-eye image data and the
left-eye image data by respectively scaling the 2D projection images captured by the
two virtual cameras using the calculated target parallactic angle and the difference
stored in the pixel parallactic angle storage unit.
[0021] (10) The head- mounted display device according to the aspect may further include an
interocular distance storage unit that stores the user's interocular distance, and the
augmented reality processing unit may arrange a virtual viewpoint which is a virtual
point of view in the 3D model space on the basis of the determined target distance at
the time of setting the two virtual cameras, arrange one virtual camera at a position
separated by the interocular distance/2 from the arranged virtual viewpoint, and
arrange the other virtual camera at a position separated by the interocular distance/2
from the arranged virtual viewpoint. In the head-mounted display device according to
this aspect, the augmented reality processing unit first arranges the virtual viewpoint
on the basis of the target distance at which the user is caused to visually recognize the
virtual object at the time of setting the virtual cameras in the 3D model space. Then,
the augmented reality processing unit arranges one virtual camera at a position
separated by the interocular distance/2 from the arranged virtual viewpoint and
arranges the other virtual camera at the position separated by the interocular distance/2
from the arranged virtual viewpoint. As a result, the virtual cameras can acquire the 2D
projection images in consideration of both the target distance at which the user is
caused to visually recognize the virtual object and the user's interocular distance.
[0022] (11) The head-mounted display device according to the aspect may further include an
interpupillary distance measuring unit that measures the user's interpupillary distance,
and the measurement result by the interpupillary distance measuring unit may be stored
as the interocular distance in the interocular distance storage unit. In the head-mounted
display device according to this aspect, since the interpupillary distance is measured
for each user and the interocular distance storage unit is updated, it is possible to store
the interpupillary distance of each user in the interocular distance storage unit.
[0023] (12) The head-mounted display device according to the aspect may further include an
object detecting unit that detects a coordinate position and a shape of a real object in a
real space, and the augmented reality processing unit may reduce the discomfort of the
virtual object by controlling the virtual object, of which a coordinate position in the
real space is set, on the basis of the coordinate position and the shape of the real object.
In the head-mounted display device according to this aspect, since the virtual object
can be caused to behave in consideration of the coordinate position or the shape of a
real object, it is possible to solve the above-mentioned phenomena and to improve a
sense of reality. That is, in a case where a virtual object collides with a real object, the
virtual object can be caused to behave as if the virtual object collided with the real
object. In a case where a virtual object is hidden by a real object from the user's
viewpoint, the virtual object can be changed as if the virtual object were hidden by the
real object.
[0024] (13) In the head-mounted display device according to the aspect, the augmented
reality processing unit may control behavior of the virtual object on the basis of the co
ordinate position and the shape of the real object. In the head-mounted display device
according to this aspect, it is possible to provide contents coupled to real objects in a
transmissive display device by causing the virtual object to behave in consideration of
the coordinate position or the shape of the real object. For example, it is possible to
provide games or navigation services in which virtual objects are coupled to real
objects.
[0025] (14) In the head-mounted display device according to the aspect, the augmented
reality processing unit may change the virtual object on the basis of the detected co
ordinate position of the real object and the set coordinate position of the virtual object
so as to make an overlapping portion of the virtual object with the real object disappear
when the virtual object is located in the back of the real object from the user's
viewpoint.
[0026] (15) In the head-mounted display device according to the aspect, the augmented
reality processing unit may cause the virtual object to behave so as to avoid the real
object in a coordinate system of a real space.
[0027] (16) In the head-mounted display device according to the aspect, the augmented
reality processing unit may cause the virtual object to behave as if the virtual object
collided with the real object when the virtual object moves and reaches the real object
in the coordinate system of the real space.
[0028] (17) The head-mounted display device according to the aspect may further include an
eyeline detecting unit that detects the user's eyeline, and the augmented reality
processing unit may display the virtual object on the basis of the detected eyeline. In
the head-mounted display device according to this aspect, it is possible to fit the virtual
object with higher accuracy by displaying the virtual object in consideration of the
user's eyeline.
[0029] (18) The head-mounted display device according to the aspect may further include a
distance detecting unit that detects a distance from the user to the real object, and the
augmented reality processing unit may display the virtual object on the basis of the
detected distance. In the head-mounted display device according to this aspect, it is
possible to fit the virtual object with higher accuracy by displaying the virtual object in
consideration of the distance from the user to the real object.
[0030] (19) In the head-mounted display device according to the aspect, the image display
unit may include: an image light output unit that outputs image light of the image; and
a light guide section that guides the output image light to the user's eyes, and the light
guide section may transmit external light and cause the external light along with the
image light to be incident on the user's eyes. In the head-mounted display device
according to this aspect, it is possible to fit the virtual object with higher accuracy and
thus to improve a sense of reality by using a so-called virtual image projection type of
transmissive display device.
[0031] All the elements in the aspects of the invention are not essential. In order to solve all
or a part of the above-mentioned problems or to achieve all or a part of the effects
described in this specification, some elements may be appropriately subjected to
change, deletion, replacement with new elements, and partial deletion of defined
details. In order to solve all or a part of the above-mentioned problems or to achieve all
or a part of the effects described in this specification, all or a part of technical features
included in one aspect of the invention may be combined with all or a part of technical
features included in another aspect of the invention to constitute an independent aspect
of the invention.
[0032] For example, an aspect of the invention may be embodied as a device including one
or both of the augmented reality processing unit and the image display unit. That is,
this device may have or may not have the augmented reality processing unit. This
device may have or may not have the image display unit. This device can be embodied,
for example, as a head-mounted display device, but may be embodied as another
device other than the head-mounted display device. All or a part of the technical
features in the above-mentioned aspects of the head-mounted display device can be
applied to this device.
[0033] The invention can be embodied in various aspects and can be embodied, for example,
in aspects such as a head-mounted display device, a control method of a head-mounted
display device, a head-mounted display system, a computer program for performing
the method or the functions of the device or system, and a recording medium having
the computer program recorded thereon.
Brief Description of Drawings
[0034] [fig.l]Fig. 1 is a diagram schematically illustrating a configuration of a head-mounted
display device according to an embodiment of the invention.
[fig.2]Fig. 2 is a block diagram illustrating a functional configuration of the headmounted
display device.
[fig.3A]Fig. 3A is a diagram illustrating an example of a virtual image which is
visually recognized by a user.
[fig.3B]Fig. 3B is a diagram illustrating an example of a virtual image which is
visually recognized by a user.
[fig.4]Fig. 4 is a diagram illustrating an augmented reality processing procedure.
[fig.5]Fig. 5 is a diagram illustrating an interocular distance and a pixel parallactic
angle.
[fig.6A]Fig. 6A is a diagram illustrating a state in which an AR processing unit
generates right-eye image data and left-eye image data from image data.
[fig.6B]Fig. 6B is a diagram illustrating a state in which an AR processing unit
generates right-eye image data and left-eye image data from image data.
[fig.7]Fig. 7 is a flowchart illustrating an augmented reality processing procedure
according to a second embodiment.
[fig.8]Fig. 8 is a diagram illustrating details of the augmented reality processing
procedure according to the second embodiment.
[fig.9]Fig. 9 is a diagram illustrating a state in which an AR processing unit scales an
image acquired by projection using a 2D projection image of a virtual camera disposed
at a virtual viewpoint.
[fig.lO]Fig. 10 is a flowchart illustrating an augmented reality processing procedure
according to a third embodiment.
[fig. 1l]Fig. 11 is a block diagram illustrating a functional configuration of the headmounted
display device according to the second embodiment.
[fig.l2]Fig. 12 is a flowchart illustrating an augmented reality processing procedure
according to the second embodiment.
[fig.l3A]Fig. 13A is a diagram illustrating a mask object.
[fig.l3B]Fig. 13B is a diagram illustrating a mask object.
[fig.l4]Fig. 14 is a diagram illustrating an example of arrangement of a virtual object
relative to a mask object.
[fig.l5]Fig. 15 is a diagram illustrating a state of a virtual object having been subjected
to lighting.
[fig.l6]Fig. 16 is a diagram illustrating a state where behavior of a virtual object is
adjusted on the basis of a restitution coefficient and a friction coefficient.
[fig.l7]Fig. 17 is a diagram illustrating image data acquired by 2D projection of a
virtual camera.
[fig.l8]Fig. 18 is a diagram illustrating an example of a virtual image which is visually
recognized by a user in the augmented reality processing procedure according to the
second embodiment.
[fig. 19] Fig. 19 is a diagram illustrating an example of a virtual image and a real scene
which can be visually recognized by a user.
[fig.20]Fig. 20 is a block diagram illustrating a control configuration of a headmounted
display device.
[fig.21]Fig. 2 1 is a flowchart illustrating an augmented reality processing procedure
according to the third embodiment.
[fig.22]Fig. 22 is a transition diagram illustrating a first example of behavior control of
a virtual object according to the third embodiment.
[fig.23]Fig. 23 is a transition diagram illustrating a second example of the behavior
control of a virtual object according to the third embodiment.
[fig.24]Fig. 24 is a transition diagram illustrating a third example of the behavior
control of a virtual object according to the third embodiment.
[fig.25A]Fig. 25A is a diagram illustrating an outer configuration of a head-mounted
display according to a modification example.
[fig.25B]Fig. 25B is a diagram illustrating an outer configuration of a head-mounted
display according to a modification example.
Description of Embodiments
[0035] A. First Embodiment
A-l. Configuration of Head-mounted Display Device
Fig. 1 is a diagram schematically illustrating a configuration of a head-mounted
display device according to an embodiment of the invention. The head-mounted
display device 100 is a display device mounted on a head and is also referred to as a
head-mounted display (HMD). The head-mounted display 100 according to this em
bodiment is an optical transmissive head-mounted display device which enables a user
to directly visually recognize an external scene at the same time as visually rec
ognizing a virtual image.
[0036] The head-mounted display 100 includes an image display unit 20 that causes a user
to visually recognize a virtual image in a state where the user wears the head-mounted
display and a control unit (controller) 10 that controls the image display unit 20.
[0037] The image display unit 20 is a wearing member that is mounted on a user's head and
has a glasses-like shape in this embodiment. The image display unit 20 includes a right
support 21, a right display drive unit 22, a left support 23, a left display drive unit 24, a
right optical image display unit 26, a left optical image display unit 28, a camera 61,
and an interpupillary distance measuring unit 62. The right optical image display unit
26 and the left optical image display unit 28 are disposed to be located in front of a
user's right and left eyes, respectively, when the user wears the image display unit 20.
One end of the right optical image display unit 26 and one end of the left optical image
display unit 28 are connected to each other at a position between the user's eyebrows
when the user wears the image display unit 20.
[0038] The right support 2 1 is a member which extends from an end portion ER which is the
other end of the right optical image display unit 26 over a position corresponding to the
user's lateral head when the user wears the image display unit 20. Similarly, the left
support 23 is a member which extends from an end portion EL which is the other end
of the left optical image display unit 28 over a position corresponding to the user's
lateral head when the user wears the image display unit 20. The right support 2 1 and
the left support 23 support the image display unit 20 on the user's head like glass
temples (temples).
[0039] The right display drive unit 22 is disposed inside the right support 21, that is, on a
side facing the user's head when the user wears the image display unit 20. The left
display drive unit 24 is disposed inside the left support 23. In the following description,
the right support 2 1 and the left support 23 are generically simply referred to as a
"support", the right display drive unit 22 and the left display drive unit 24 are
generically simply referred to as a "display drive unit", and the right optical image
display unit 26 and the left optical image display unit 28 are generically simply
referred to as an "optical image display unit".
[0040] The display drive unit includes liquid crystal displays (hereinafter, referred to as
"LCD") 241 and 242 or projection optical systems 251 and 252 (see Fig. 2). The
detailed configuration of the display drive unit will be described later. The optical
image display unit as an optical member includes light guide plates 261 and 262 (see
Fig. 2) and light control plates. The light guide plates 261 and 262 are formed of a
light-transmitting resin material and guide image light output from the display drive
unit to the user's eyes. The light control plates are thin-plate optical elements and are
disposed to cover the front side (the side opposite to the user's eyes) of the image
display unit 20. The light control plates protect the light guide plates 261 and 262 and
suppress damage or contamination of the light guide plates 261 and 262. By adjusting
optical transmittance of the light control plates, it is possible to adjust an amount of
external light incident on the user's eyes so as to easily visually recognize a virtual
image. The light control plates may be skipped.
[0041] The camera 6 1 is disposed at a position between the user's eyebrows when the user
wears the image display unit 20. The camera 6 1 images an external scene (external
scene) on the front side of the image display unit 20, that is, in the user's viewing field
in a state where the user wears the head-mounted display 100, and acquires an external
scene image. The camera 6 1 is a so-called visible-light camera and the external scene
image acquired by the camera 6 1 is an image in which a shape of an object is displayed
by visible light emitted from the object. The camera 6 1 in this embodiment is a
monocular camera, but may be a stereoscopic camera. The camera 6 1 serves as the
"image acquiring unit".
[0042] The interpupillary distance measuring unit 62 measures a user's interpupillary
distance. The interpupillary distance is a distance between the center of the iris of the
user's right eye RE and the center of the iris of the user's left eye LE. As illustrated in
Fig. 1, the interpupillary distance measuring unit 62 includes two cameras that are
disposed on the inner surface of the image display unit 20 and that capture images of
the user's right eye RE and left eye LE, respectively, and a processing unit that
analyzes the captured images, for example, using a triangulation method and that
calculates the distance between the centers of the irises of the right and left eyes. The
interpupillary distance measuring unit 62 may measure the user's interpupillary
distance using ultrasonic waves or infrared rays instead of the camera. The inter
pupillary distance measuring unit 62 may measure the user's interpupillary distance by
combining the above-mentioned multiple methods.
[0043] The image display unit 20 includes a connecting section 40 that connects the image
display unit 20 to the control unit 10. The connecting section 40 includes a main cord
48 that is connected to the control unit 10, two cords of a right cord 42 and a left cord
44 into which the main cord 48 is branched, and a connecting member 46 that is
disposed at the branching point. The right cord 42 is inserted into a chassis of the right
support 2 1 from a tip portion AP in the extending direction of the right support 2 1 and
is connected to the right display drive unit 22. Similarly, the left cord 44 is inserted
into a chassis of the left support 23 from a tip portion AP in the extending direction of
the left support 23 and is connected to the left display drive unit 24. The connecting
member 46 is provided with a jack for connection to an earphone plug 30. A right
earphone 32 and a left earphone 34 extend from the earphone plug 30.
[0044] The image display unit 20 and the control unit 10 transmit various signals via the
connecting section 40. An end portion of the main cord 48 opposite to the connecting
section 46 and the control unit 10 are provided with connectors (not illustrated)
engaging with each other, respectively, and the control unit 10 and the image display
unit 20 are connected to or disconnected from each other by engagement or disen
gagement between the connector of the main cord 48 and the connector of the control
unit 10. The right cord 42, the left cord 44, and the main cord 48 can employ, for
example, a metal cable or an optical fiber.
[0045] The control unit 10 is a unit that controls the head-mounted display 100. The control
unit 10 includes a lighting unit 12, a touch pad 14, a cross key 16, and a power supply
switch 18. The lighting unit 12 notifies the operation state (for example, ON/OFF of a
power source) of the head-mounted display 100 using light-emitting forms thereof. The
lighting unit 12 can employ, for example, a light emitting diode (LED). The touch pad
14 detects a touch operation on an operation surface of the touch pad 14 and outputs a
signal corresponding to the detected details. The touch pad 14 can employ various
touch pads such as an electrostatic type, a pressure detection type, and an optical type.
The cross key 16 detects a pressing operation against keys corresponding to vertical
and horizontal directions and outputs a signal corresponding to the detected details.
The power supply switch 18 switches the power supply state of the head-mounted
display 100 by detecting a sliding operation on the switch.
[0046] Fig. 2 is a block diagram illustrating a functional configuration of the head-mounted
display 100. The control unit 10 includes an input information acquiring unit 110, a
storage unit 120, a power supply 130, a wireless communication unit 132, a GPS
module 134, a CPU 140, an interface 180, and transmitter units (Tx) 5 1 and 52. The
units are connected to each other via a bus not illustrated.
[0047] The input information acquiring unit 110 acquires a signal corresponding to an
operation input to, for example, the touch pad 14, the cross key 16, or the power supply
switch 18. The storage unit 120 is formed of a ROM, a RAM, a DRAM, a hard disk, or
the like. The storage unit 120 includes an interocular distance 122 and a pixel p ar
allactic angle 124. Details thereof will be described later. The power supply 130
supplies power to the respective units of the head-mounted display 100. For example, a
secondary battery can be used as the power supply 130. The wireless communication
unit 132 wirelessly communicates with other devices on the basis of a predetermined
wireless communication protocol such as a wireless LAN or a Bluetooth. The GPS
module 134 detects its current position by receiving a signal from GPS satellites.
[0048] The CPU 140 serves as an operating system (OS) 150, an image processing unit 160,
a sound processing unit 170, a display control unit 190, and an AR processing unit 142
by reading and executing computer programs stored in the storage unit 120. The AR
processing unit 142 performs a process (hereinafter, also referred to as "augmented
reality processing") of realizing augmented reality using a process start request from
the OS 150 or a specific application as a trigger. Details thereof will be described later.
The AR processing unit 142 corresponds to the "augmented reality processing unit" in
the appended claims.
[0049] The image processing unit 160 generates a signal on the basis of contents (image)
input via the interface 180 or the wireless communication unit 132. Then, the image
processing unit 160 supplies the generated signal to the image display unit 20 via the
connecting section 40. The signal to be supplied to the image display unit 20 has a
difference between an analog type and a digital type. In the case of the analog type, the
image processing unit 160 generates and transmits a clock signal PCLK, a vertical syn
chronization signal VSync, a horizontal synchronization signal HSync, and image data
Data. Specifically, the image processing unit 160 acquires an image signal included in
the contents. For example, in the case of a moving image, the acquired image signal is
generally an analog signal including 30 frame images per second. The image
processing unit 160 separates a synchronization signal such as the vertical synchro
nization signal VSync or the horizontal synchronization signal HSync from the
acquired image signal, and generates the clock signal PCLK through the use of a PLL
circuit or the like on the basis of the period of the synchronization signal. The image
processing unit 160 converts the analog image signal from which the synchronization
signal is separated into a digital image signal by the use of an A/D conversion circuit
or the like. The image processing unit 160 stores the converted digital image signal as
the image data Data of RGB data in the DRAM of the storage unit 120 for each frame.
On the other hand, in the case of the digital type, the image processing unit 160
generates and transmits the clock signal PCLK and the image data Data. Specifically,
when the contents are of a digital type, the clock signal PCLK is output in synchronization
with the image signal and thus the generation of the vertical synchronization
signal VSync and the horizontal synchronization signal HSync and the A/D conversion
of the analog image signal are not necessary. The image processing unit 160 may
perform various color correcting processes such as a resolution converting process and
adjustment of luminance and chroma and image processing such as a keystone
correcting process on the image data Data stored in the storage unit 120.
[0050] The image processing unit 160 transmits the generated clock signal PCLK, vertical
synchronization signal VSync, and horizontal synchronization signal HSync and the
image data Data stored in the DRAM of the storage unit 120 via the transmitter units
51 and 52. The image data Data transmitted via the transmitter unit 51 is also referred
to as "right-eye image data Datal" and the image data Data transmitted via the
transmitter unit 52 is also referred to as "left-eye image data Data2". The transmitter
units 5 1 and 52 serve as transceivers for serial transmission between the control unit 10
and the image display unit 20.
[0051] The display control unit 190 generates a control signal for controlling the right
display drive unit 22 and the left display drive unit 24. Specifically, the display control
unit 190 controls generation and emission of image light from the right display drive
unit 22 and the left display drive unit 24 by individually controlling drive ON/OFF of a
right LCD 241 by a right LCD control unit 211, drive ON/OFF of a right backlight 221
by a right backlight control unit 201, drive ON/OFF of a left LCD 242 by a left LCD
control unit 212, and drive ON/OFF of a left backlight 222 by a left backlight control
unit 202 using the control signal. For example, the display control unit 190 causes both
the right display drive unit 22 and the left display drive unit 24 to generate image light,
causes only any one thereof to generate image light, or causes both thereof not to
generate image light. The display control unit 190 transmits the control signals for the
right LCD control unit 211 and the left LCD control unit 212 via the transmitter units
5 1 and 52, respectively. The display control unit 190 transmits the control signals for
the right backlight control unit 201 and the left backlight control unit 202, respectively.
[0052] The sound processing unit 170 acquires a sound signal included in the contents,
amplifies the acquired sound signal, and supplies the amplified sound signal to a
speaker (not illustrated) in the right earphone 32 connected to the connecting member
46 and a speaker (not illustrated) in the left earphone 34 connected to the connecting
member. For example, when a Dolby (registered trademark) system is employed, the
sound signal is processed and different sounds having, for example, changed fre
quencies are output from the right earphone 32 and the left earphone 34.
[0053] The interface 180 is an interface for connecting various external devices OA as a
source of contents to the control unit 10. Examples of the external device OA include a
personal computer PC, a mobile terminal, and a game terminal. For example, a USB
interface, a micro USB interface, and a memory-card interface can be used as the
interface 180.
[0054] The image display unit 20 includes a right display drive unit 22, a left display drive
unit 24, a right light guide plate 261 as the right optical image display unit 26, a left
light guide plate 262 as the left optical image display unit 28, a camera 61, and a 9-axis
sensor 66.
[0055] The 9-axis sensor 66 is a motion sensor that detects accelerations (three axes),
angular velocities (three axes), and terrestrial magnetism (three axes). The 9-axis
sensor 66 is disposed in the image display unit 20. Accordingly, when the image
display unit 20 is mounted on a user's head, the 9-axis sensor serves as a movement
detecting unit that detects movement of the user's head. Here, the movement of a head
includes a velocity, an acceleration, an angular velocity, a direction, and a variation in
direction of the head.
[0056] The right display drive unit 22 includes a receiver unit (Rx) 53, a right backlight
(BL) control unit 201 and a right backlight (BL) 221 serving as a light source, a right
LCD control unit 211 and a right LCD 241 serving as a display element, and a right
projection optical system 251. The right backlight control unit 201, the right LCD
control unit 211, the right backlight 221, and the right LCD 241 are also generically
referred to as an "image light generating unit".
[0057] The receiver unit 53 serves as a receiver for serial transmission between the control
unit 10 and the image display unit 20. The right backlight control unit 201 drives the
right backlight 221 on the basis of the input control signal. The right backlight 221 is a
light-emitting member such as an LED or an electroluminescence (EL). The right LCD
control unit 211 drives the right LCD 241 on the basis of the clock signal PCLK, the
vertical synchronization signal VSync, the horizontal synchronization signal HSync,
and the right-eye image data Datal input via the receiver unit 53. The right LCD 241 is
a transmissive liquid crystal panel in which plural pixels are arranged in a matrix
shape. The right LCD 241 modulates illumination light emitted from the right
backlight 221 to effective image light indicating an image by driving the liquid crystal
at the pixel positions arranged in the matrix shape to change transmittance of light
passing through the right LCD 241. In this embodiment, the backlight type is
employed, but image light may be emitted using a front-light type or a reflection type.
[0058] The right projection optical system 251 includes a collimator lens that changes the
image light emitted from the right LCD 241 to a parallel light beam. The right light
guide plate 261 as the right optical image display unit 26 guides the image light output
from the right projection optical system 251 to the user's right eye RE while reflecting
the image light along a predetermined optical path. The optical image display unit may
employ any method as long as it can form a virtual image in front of the user's eyes
using image light. For example, a diffraction grating or a semi-transmissive film may
be used.
[0059] The left display drive unit 24 has the same configuration as the right display drive
unit 22. That is, the left display drive unit 24 includes a receiver unit (Rx) 54, a left
backlight (BL) control unit 202 and a left backlight (BL) 222 serving as a light source,
a left LCD control unit 212 and a left LCD 242 serving as a display element, and a left
projection optical system 252.
[0060] Figs. 3A and 3B are diagrams illustrating an example of a virtual image which is
visually recognized by a user. Fig. 3A illustrates a user's viewing field VR which is
subjected to a normal display process. As described above, the image light guided to
the user's eyes in the head-mounted display 100 is focused on the user's retinas,
whereby the user visually recognizes the virtual image VI. In the example illustrated in
Fig. 3A, the virtual image VI is a standby screen of the OS of the head-mounted
display 100. The user visually recognizes an external scene SC through the right
optical image display unit 26 and the left optical image display unit 28. In this way, the
user of the head- mounted display 100 according to this embodiment can view the
virtual image VI and the external scene SC in the back of the virtual image VI in a
portion of the viewing field VR in which the virtual image VI is displayed. In a portion
of the viewing field VR in which the virtual image VI is not displayed, the user can
directly view the external scene SC through the optical image display unit. In this spec
ification, "the head-mounted display 100 displays an image" includes causing the user
of the head- mounted display 100 to visually recognize a virtual image as described
above.
[0061] Fig. 3B illustrates a user's viewing field VR which is subjected to the augmented
reality processing. In the augmented reality processing, the AR processing unit 142 of
the head-mounted display 100 forms a virtual image indicating a virtual object, which
is used to augment an external scene SC recognized by a user and of which the user's
visual discomfort is reduced, on the image display unit 20. Specifically, the AR
processing unit 142 generates image data in which a visual effect capable of reducing
the user's visual discomfort is added to the virtual object and transmits the generated
image data to the image display unit 20. "To augment the external scene SC" means to
add, delete, emphasize, and attenuate information of a real environment, that is, an
external scene SC, which is viewed by the user. In the augmented reality processing
according to the first embodiment, in order to fit the virtual object to the external scene
SC, the AR processing unit 142 adds a visual effect enabling a stereoscopic view of the
virtual object by generating the right-eye image data Datal and the left-eye image data
Data 2 which are different from each other. "To fit a virtual object to an external
scene" means to display a virtual image VI which gives a user such a sense that the
virtual object is present at a position separated by a predetermined distance
(hereinafter, also referred to as a "target distance") from the user in the external scene
SC which is actually viewed by the user. In the example illustrated in Fig. 3B, an
image of an apple is displayed as a virtual image VI so as to overlap with a real road
included in the external scene SC. Accordingly, the user can feel such a sense that the
apple lies on the road having nothing. In the example illustrated in Fig. 3B, the
distance between the position of the user and the position at which the user feels "an
apple lies" corresponds to the target distance. In the example illustrated in Fig. 3B, the
apple corresponds to the virtual object OB.
[0062] A-2. Augmented Reality Processing (First Example)
The AR processing unit 142 performs an augmented reality processing procedure
using the following procedures al to a3.
(al) The target distance at which a virtual object is visually recognized is determined.
(a2) Image data indicating the virtual object is generated.
(a3) Right-eye image data Data 1 and left-eye image data Data2 are generated from
the image data.
[0063] Fig. 4 is a diagram illustrating the augmented reality processing procedure. When the
image processing unit 160 supplies the image display unit 20 with the right-eye image
data Datal and the left-eye image data Data2 which are identical to each other, the user
recognizes an object at a position COl separated by an initial imaging distance La
from the user. At this time, a convergence angle is referred to as an "initial con
vergence angle theta a". In the procedure al, the AR processing unit 142 determines
the target distance Lb at which the user is caused to visually recognize the virtual
object OB so as to enable displaying of the virtual object OB having a sense of per
spective. The AR processing unit 142 can determine the target distance Lb, for
example, using any method of the following methods.
- An external scene in the user's viewing field acquired by the camera 6 1 is analyzed.
- A current position coordinate of the user and movement of the user's head are
analyzed. In this case, the current position coordinate of the user is acquired from
position information of the control unit 10 detected by the GPS module 134. The
movement of the user's head is acquired from movement information detected by the
9-axis sensor 66.
[0064] In the procedure a2, the AR processing unit 142 generates image data indicating the
virtual object OB. In the augmented reality processing according to the first example,
the AR processing unit 142 acquires image data based on the analysis result of the
procedure al from plural image data pieces stored in advance in the storage unit 120.
[0065] In the procedure a3, the AR processing unit 142 generates the right-eye image data
Datal and the left-eye image data Data2 from the image data generated in the
procedure a2. At this time, the AR processing unit 142 uses the pixel parallactic angle
124 stored in the storage unit 120.
[0066] Fig. 5 is a diagram illustrating the interocular distance 122 and the pixel parallactic
angle 124. In this embodiment, the interocular distance 122 is considered as the
distance between the center of the right optical image display unit 26 and the center of
the left optical image display unit 28 as the distance DL between the user's right eye
RE and left eye LE. Accordingly, the distance (for example, 65 mm) between the
center of the right optical image display unit 26 and the center of the left optical image
display unit 28 based on the design value of the head- mounted display 100 is stored in
advance in the interocular distance 122. The actual distance DL between the user's
right eye RE and left eye LE may be stored in the interocular distance 122. Details
thereof will be described in a second example of the augmented reality processing. The
interocular distance 122 may be adjusted depending on a user's taste. The interocular
distance 122 corresponds to the "interocular distance storage unit" in the appended
claims.
[0067] The pixel parallactic angle 124 is a parallactic angle which is realized by image data
deviated laterally by one pixel and is expressed in theta pix(degree). The pixel p ar
allactic angle 124 is a difference between a convergence angel of a virtual image
displayed on the basis of the image data which are laterally identical to each other and
a convergence angel of a virtual image displayed on the basis of the image data which
are laterally deviated by one pixel from each other. Theta pix is calculated as follows
and is stored in advance in the pixel parallactic angle 124.
- The initial convergence angle theta a is measured when the right LCD 241 and the
left LCD 242 are driven on the basis of the right-eye image data Datal and the left-eye
image data Data2 which are identical to each other.
- The convergence angle theta c is measured when the right LCD 241 and the left
LCD 242 are driven on the basis of the right-eye image data Datal and the left-eye
image data Data2 which are deviated by one pixel.
- The difference between the initial convergence angle theta a and the convergence
angle theta c is calculated (theta c- theta a). In general, since theta c> theta a is
assumed, it is assumed herein that the initial convergence angle theta a is subtracted
from the convergence angle theta.
- In order to obtain a convergence angle difference for each eye, the difference
between the initial convergence angle theta a and the convergence angle theta c is
divided by 2.
[0068] That is, the pixel parallactic angle can be expressed by theta pix=(theta c-theta a)/2.
The initial convergence angle theta a corresponds to the "first convergence angle" in
the appended claims, the convergence angle theta c corresponds to the "second convergence
angle" in the appended claims, and the pixel parallactic angle 124 cor
responds to the "pixel parallactic angle storage unit" in the appended claims.
[0069] Referring to Fig. 4 again, the augmented reality processing procedure will be
described below. The target distance Lb (the distance at which the user can be caused
to visually recognize the virtual object OB) determined in the procedure al, the con
vergence angle theta b at that time, and the interocular distance 122 (DL/2) can be
expressed by Expression (1) using a trigonometrical function. Accordingly, the AR
processing unit 142 applies the target distance Lb determined in the procedure al and
the interocular distance 122 to Expression (1) to calculate the target convergence angle
theta b.
[Math.l]
t a n ( b/2 ) = ( D L 2 ) / L b
t a n ( b/2 ) = D L/ ( 2 X L b )
0 b 2 = a r c t a n | D L/ ( 2 x L b ) }
0 b = 2 X a r c t a n | D L/ ( 2 x L b ) } ( 1 )
[0070] The AR processing unit 142 calculates a target parallactic angle theta x which is a
parallactic angle at the target distance Lb (the distance at which the user is caused to
visually recognize the virtual object OB) using the calculated target convergence angle
theta b and the initial convergence angle theta a. Specifically, the AR processing unit
142 applies the values of the initial convergence angle theta a and the target con
vergence angle theta b to Expression (2) to calculate the target parallactic angle theta x.
[Math.2]
Q x = ( Q b - a ) 2 ( 2 )
In Expression (2), for example, when the set position is infinity, theta x is expressed
by theta x=- theta a/2.
[0071] The AR processing unit 142 processes the image data in the procedure a2 using the
calculated target parallactic angle theta x and the pixel parallactic angle 124 and
generates the right-eye image data Datal and the left-eye image data Data2.
Specifically, theta pix stored in the pixel parallactic angle 124 is a parallactic angle
which is realized by the image data laterally deviated by one pixel. Accordingly, in
order to generate the right-eye image data Datal for realizing the target parallactic
angle theta x, the basic image data can be deviated to the left by theta x/theta pix
pixels. Similarly, in order to generate the left-eye image data Data2 for realizing the
target parallactic angle theta x, the basic image data can be deviated to the right by
theta x/theta pix pixels.
[0072] Figs. 6A and 6B illustrate a state where the AR processing unit 142 generates the
right-eye image data Datal and the left-eye image data Data2 from the image data DT.
[0073] The AR processing unit 142 transmits the right-eye image data Datal and the left-eye
image data Data2 generated as described above to the image processing unit 160. The
image processing unit 160 transmits the received right-eye image data Datal to the
image display unit 20 via the transmitter unit 51. Similarly, the image processing unit
transmits the received left-eye image data Data2 to the image display unit 20 via the
transmitter unit 52. Thereafter, the display process described with reference to Fig. 2 is
performed. As illustrated in Fig. 3B, the user of the head-mounted display 100 can
visually recognize the stereoscopic virtual object OB in the viewing field VR.
[0074] The resolution of a person with an eyesight of 1.0 can be expressed by arctan(1.5
mm/5000 mm) and is about 0.017 degree. Therefore, when the error between the
virtual image VI displayed on the basis of the right-eye image data Datal and the lefteye
image data Data2 and the external scene SC is designed to be 0.017 degree or less,
it is possible to display the virtual image VI with an actual size based on the external
scene SC.
[0075] As described above, in the augmented reality processing according to the first
example, since the augmented reality processing unit (AR processing unit 142) fits the
virtual object OB to the external scene SC in consideration of various conditions (the
sizes of the right LCD 241 and the left LCD 242 and the distance between the virtual
images VI displayed in front of the user's right and left eyes) on the display en
vironment of the virtual image VI at the time of generating the image data (the righteye
image data Datal and the left-eye image data Data2), it is possible to implement a
head-mounted display device (head-mounted display 100) which can provide
augmented reality with a user's visual discomfort reduced. Specifically, the difference
(theta pix) stored in the pixel parallactic angle storage unit (pixel parallactic angle 124)
is a difference between the first convergence angle theta a (initial convergence angle
theta a) of the virtual image VI displayed on the basis of the right-eye image data
Datal and the left-eye image data Data2 which are identical to each other and the
second convergence angle theta c of the virtual image displayed on the basis of the
right-eye image data Datal and the left-eye image data Data2 which are laterally
deviated by one pixel. Accordingly, the difference (theta pix) stored in the pixel p ar
allactic angle storage unit (pixel parallactic angle 124) is a parallactic angle which is
realized by the image data laterally deviated by one pixel and is also a parallactic angle
determined in consideration of various conditions on the display environment of the
virtual image VI, that is, the conditions such as the sizes of the right LCD 241 and the
left LCD 242 and the distance between the virtual images VI displayed in front of the
user's right and left eyes. Therefore, the augmented reality processing unit (AR
processing unit 142) can generate the right-eye image data Datal and the left-eye
image data Data2 for fitting the virtual object OB to the external scene SC in con
sideration of various conditions on the display environment of the virtual image VI
using the pixel parallactic angle stored in the pixel parallactic angle storage unit (pixel
parallactic angle 124).
[0076] In the augmented reality processing according to the first example, the augmented
reality processing unit (AR processing unit 142) determines the target distance Lb at
which the user is caused to visually recognize the virtual object OB, calculates the
target convergence angle theta b which is a convergence angle at the target distance Lb
from the determined target distance Lb, and calculates the target parallactic angle theta
x which is a parallactic angle at the target distance Lb using the calculated target con
vergence angle theta b and the first convergence angle theta a (initial convergence
angle theta a). The augmented reality processing unit (AR processing unit 142) can
generate the right-eye image data Datal and the left-eye image data Data2 which are
deviated from each other from single image data DT (see Figs. 6A and 6B) indicating
the virtual object OB using the calculated target parallactic angle theta x and the
difference (theta pix) stored in the pixel parallactic angle storage unit (pixel parallactic
angle 124).
[0077] A-3. Augmented Reality Processing (Second Example)
In the augmented reality processing according to a second example, the AR
processing unit 142 generates image data indicating a virtual object based on a user's
movement from a virtual three-dimensional object in a three-dimensional (3D) model
space and generates the right-eye image data Datal and the left-eye image data Data2
from the generated image data. The augmented reality processing according to this
example is different from the augmented reality processing according to the first
example, in that the procedure illustrated in Fig. 7 is performed instead of the
procedures a2 and a3. The procedure al is the same as in the first example and thus
will not be repeatedly described.
[0078] Fig. 7 is a flowchart illustrating an augmented reality processing procedure according
to the second example. The AR processing unit 142 initially sets the interocular
distance 122 (step S102). Specifically, the AR processing unit 142 acquires a user's interpupillary
distance measured by the interpupillary distance measuring unit 62 (see
Fig. 2) and stores the acquired distance in the interocular distance 122. In this way,
according to step S102, the user's interpupillary distance can be stored in the in
terocular distance 122.
[0079] Fig. 8 is a diagram illustrating details of the augmented reality processing procedure
according to the second example. In step SI04 of Fig. 7, the AR processing unit 142
sets a virtual viewpoint at a predetermined position in the 3D model space.
Specifically, the AR processing unit 142 sets the virtual viewpoint IV at a position
separated by the distance Lb calculated in the procedure al from a virtual object OBI
in the 3D model space (Fig. 8). The virtual viewpoint IV is located as a position corre
sponding to the user's temple, and the direction of the virtual viewpoint IV (viewpoint
direction VD) is directed to the virtual object OBI. The position of the virtual
viewpoint IV is defined by a shift (x, y, z) with respect to the origin coordinate in the
3D model space. The viewpoint direction VD is defined by an angle (theta x, theta y,
theta z) with respect to the virtual viewpoint IV. (theta x, theta y, theta z) is expressed
by (roll angle, pitch angle, yaw angle)=(phi, theta, psi). After setting the virtual
viewpoint IV, the AR processing unit 142 performs the processes of steps SI 10 to
SI 16 and the processes of steps S120 to S126 in parallel.
[0080] After setting the virtual viewpoint IV, the AR processing unit 142 sets a virtual
camera CML at a position corresponding to the left eye on the basis of the position and
the direction of the virtual viewpoint IV (step SI 10). Specifically, the AR processing
unit 142 sets the virtual camera CML in the viewpoint direction VD at a position
separated to the left (by the interocular distance 122/2) from the virtual viewpoint IV.
Similarly, the AR processing unit 142 sets a virtual camera CMR at a position corre
sponding to the right eye on the basis of the position and the direction of the virtual
viewpoint IV (step S120). Specifically, the AR processing unit 142 sets the virtual
camera CMR in the viewpoint direction VD at a position separated to the right (by the
interocular distance 122/2) from the virtual viewpoint IV. Fig. 8 illustrates the virtual
cameras CML and CMR set as described above.
[0081] After setting the virtual cameras, the AR processing unit 142 performs twodimensional
(2D) projection of the virtual camera CML (step SI 12). Specifically, the
AR processing unit 142 converts three-dimensional objects (virtual objects OBI and
OB2) in the 3D model space into information of two-dimensional plane acquired by
the virtual camera CML and generates an image with a sense of depth. Similarly, the
AR processing unit 142 performs two-dimensional (2D) projection of the virtual
camera CMR (step S122). Specifically, the AR processing unit 142 converts the threedimensional
objects in the 3D model space into information of two-dimensional plane
acquired by the virtual camera CMR and generates an image with a sense of depth.
[0082] After performing the 2D projection of the virtual camera CML, the AR processing
unit 142 scales the image obtained by the projection (step SI 14). Specifically, the AR
processing unit 142 performs the following procedures b l to b3.
(bl) The AR processing unit 142 applies the target distance Lb determined in the
procedure al to Expression (1) to calculate the target convergence angle theta b.
(b2) The AR processing unit 142 applies the calculated target convergence angle theta
b and the initial convergence angle theta a to Expression (2) to calculate the target p ar
allactic angle theta x.
(b3) The 2D projection image of the virtual camera CML acquired in step SI 12 is
enlarged/reduced so that the 2D projection image of the virtual camera CML acquired
in step SI 12 and the 2D projection image of the virtual camera CM disposed at the
virtual viewpoint IV are deviated from each other by theta x/theta pix pixels. At this
time, the AR processing unit 142 considers the resolution of the right LCD 241.
[0083] Similarly, after performing the 2D projection of the virtual camera CMR, the AR
processing unit 142 scales the image obtained by the projection (step S124).
Specifically, the AR processing unit 142 performs the above procedures b l and b2 and
the following procedure b3.
(b3) The 2D projection image of the virtual camera CMR acquired in step S122 is
enlarged/reduced so that the 2D projection image of the virtual camera CMR acquired
in step S122 and the 2D projection image of the virtual camera CM disposed at the
virtual viewpoint IV are deviated from each other by theta x/theta pix pixels. At this
time, the AR processing unit 142 considers the resolution of the left LCD 242.
[0084] Fig. 9 illustrates a state where the AR processing unit 142 scales the image acquired
by the projection using the 2D projection image DT of the virtual camera CM disposed
at the virtual point IV as described above. In the above-mentioned example, it is
assumed that the virtual camera CMR is set at the position corresponding to the right
eye and the virtual camera CML is set at the position corresponding to the left eye.
However, the positions and the number of the virtual cameras to be set can be changed.
Thereafter, the AR processing unit 142 transmits the right-eye image data Datal and
the left-eye image data Data2 generated as described above to the image processing
unit 160 (steps SI 16 and S126). Thereafter, by performing the display process
described with reference to Fig. 2, the user of the head-mounted display 100 can
visually recognize the stereoscopic virtual object OB in the viewing field VR.
[0085] As described above, in the augmented reality processing according to the second
example, since the augmented reality processing unit (AR processing unit 142) fits the
virtual objects OBI and OB2 to the external scene SC in consideration of various
conditions (the sizes of the right LCD 241 and the left LCD 242 and the distance
between the virtual images VI displayed in front of the user's right and left eyes) on the
display environment of the virtual image VI at the time of generating the image data
(the right-eye image data Datal and the left-eye image data Data2), it is possible to
implement a head- mounted display device (head-mounted display 100) which can
provide augmented reality with a user's visual discomfort reduced.
[0086] In the augmented reality processing according to the second example, the augmented
reality processing unit (AR processing unit 142) determines the target distance Lb at
which the user is caused to visually recognize the virtual objects OBI and OB2,
calculates the target convergence angle theta b which is a convergence angle at the
target distance Lb from the determined target distance Lb, and calculates the target p ar
allactic angle theta x which is a parallactic angle at the target distance Lb using the
calculated target convergence angle theta b and the first convergence angle theta a
(initial convergence angle theta a). The augmented reality processing unit (AR
processing unit 142) can generate the right-eye image data Datal and the left-eye
image data Data2 by scaling the 2D projection images by the two virtual cameras
(virtual cameras CMR and CML) using the calculated target parallactic angle theta x
and the difference (theta pix) stored in the pixel parallactic angle storage unit (pixel
parallactic angle 124).
[0087] In the augmented reality processing according to the second example, the augmented
reality processing unit (AR processing unit 142) first arranges the virtual viewpoint IV
on the basis of the target distance Lb at which the user is caused to visually recognize a
virtual object at the time of setting the virtual cameras (virtual cameras CMR and
CML) in the 3D model space. Then, the augmented reality processing unit (AR
processing unit 142) arranges one virtual camera (virtual camera CMR) at a position
separated by the interocular distance/2 from the arranged virtual viewpoint IV and
arranges the other virtual camera (virtual camera CML) at a position separated by the
interocular distance/2 from the arranged virtual viewpoint. As a result, the virtual
cameras (virtual cameras CMR and CML) can acquire 2D projection images in con
sideration of both the target distance Lb at which the user is caused to visually
recognize the virtual objects OBI and OB2 and the user's interocular distance
(interocular distance 122).
[0088] A-4. Augmented Reality Processing (Third Example)
In the augmented reality processing according to a third example, the virtual objects
to be fitted to the external scene SC in the augmented reality processing according to
the second embodiment can be changed with movement of a user. Hereinafter, only a
procedure different from the augmented reality processing according to the second em
bodiment will be described. The same elements in the drawings as in the second
example will be referenced by the same reference signs and detailed description
thereof will not be repeated.
[0089] Fig. 10 is a flowchart illustrating the augmented reality processing according to the
third example. This example is different from the second example illustrated in Fig. 7,
in that steps S202 and S204 are performed after step SI04 and the process flow goes to
step S202 after steps SI 16 and S126 end.
[0090] In step S202, the AR processing unit 142 determines whether the user's viewpoint
moves. Specifically, the AR processing unit 142 determines that the user's viewpoint
moves when at least one of the current position coordinate of the user and the
movement of the user's head is changed. The current position coordinate of the user is
acquired from the position information of the control unit 10 detected by the GPS
module 134. The movement of the user's head is acquired from the movement in
formation detected by the 9-axis sensor 66. When the user's viewpoint does not move
(NO in step S202), the AR processing unit 142 causes the process flow proceed to
steps S110 and S120.
[0091] When the user's viewpoint moves (YES in step S202), the AR processing unit 142
sets the virtual viewpoint at a predetermined position in the 3D model space on the
basis of the current position coordinate of the user and the movement of the user's head
detected in step S202. Details thereof are the same as in step SI04 illustrated in Fig. 7.
[0092] As described above, in the augmented reality processing according to the third
example, it is possible to generate the right-eye image data Datal and the left-eye
image data Data2 by generating and scaling the 2D projection image of the virtual
camera CML and the 2D projection image of the virtual camera CMR with the
movement of the user's viewpoint, that is, with a variation of at least one of the current
position coordinate of the user and the movement of the user's head. As a result, in
addition to the effects of the second example, it is possible to change the virtual object
to be fitted to the external scene SC with the movement of the user's viewpoint.
[0093] In this way, in the head-mounted display device (head-mounted display 100)
according to the first embodiment, the image display unit 20 causes the user to visually
recognize the virtual object OB, to which the visual effect (that is, the visual effect for
enabling a stereoscopic view of the virtual object) for reducing the user's visual
discomfort is added, as a virtual image VI. Accordingly, it is possible to provide an
optical transmissive head- mounted display device which can provide augmented
reality with a user's visual discomfort reduced.
[0094] B. Second Embodiment
A second embodiment of the invention will describe a configuration in which a
"visual effect for fitting a virtual object to an ambient environment of a user" is
employed as the "visual effect for reducing a user's visual discomfort" instead of the
"visual effect for enabling a stereoscopic view of a virtual object". Only the con
figuration and the operation different from those of the first embodiment will be
described below. The same elements in the drawings as in the first embodiment will be
referenced by the same reference signs as described in the first embodiment and
detailed description thereof will not be repeated.
[0095] B-l. Configuration of Head-mounted Display Device
Fig. 11 is a block diagram illustrating a functional configuration of a head-mounted
display 100a according to the second embodiment. This embodiment is different from
the first embodiment illustrated in Fig. 2, in that a control unit 10a is provided instead
of the control unit 10. The storage unit 120 of the control unit 10a includes neither the
interocular distance 122 nor the pixel parallactic angle 124 (Fig. 2). The control unit
10a includes an AR processing unit 142a instead of the AR processing unit 142. The
AR processing unit 142a is different from the AR processing unit of the first em
bodiment in the process details of the "augmented reality processing" to be performed.
Specifically, the AR processing unit 142a performs augmented reality processing to be
described below instead of the augmented reality processing according to the first em
bodiment described in the first to third examples.
[0096] B-2. Augmented Reality Processing
Fig. 12 is a flowchart illustrating an augmented reality processing procedure
according to the second embodiment. In step S302, the AR processing unit 142a
acquires three-dimensional information in which an ambient environment of a user is
expressed as three-dimensional virtual objects in a three-dimensional space. The threedimensional
information is a so-called "3D model" expressing the ambient en
vironment of a user. The AR processing unit 142a can acquire the three-dimensional
information using any method of the following methods c l to c3.
[0097] (cl) The AR processing unit 142a acquires three-dimensional information from an
external device such as a ground server connected to the head-mounted display 100a.
(c2) When the three-dimensional information is stored in advance in the storage unit
120, the AR processing unit 142a acquires the three-dimensional information from the
storage unit 120.
(c3) The AR processing unit 142a generates three-dimensional information ex
pressing the ambient environment of a user using an external scene image captured by
the camera 6 1 or means for irradiating a user's eyes with infrared rays and acquires the
generated three-dimensional information.
[0098] In the methods c l and c2, the AR processing unit 142a can determine an acquisition
range of three-dimensional information on the basis of the current position coordinate
of the user detected by the GPS module 134. In the method c3, when the threedimensional
information is generated with the camera 61, the AR processing unit 142a
causes the camera 6 1 to acquire plural external scene images while continuously
changing the angle. Then, the AR processing unit 142a can generate the threedimensional
information on the basis of difference information between the n-th
(where n is an integer) captured external scene image and the (n+l)-th captured
external scene image. At this time, it is preferable that the camera 6 1 is plural cameras
which are arranged so that the imaging regions thereof cross each other. It is preferable
that at least one of the plural cameras is fixed to a position corresponding to the user's
eyeline. In order to cause the camera 6 1 to acquire the external scene images while
continuously changing the angle, the AR processing unit 142a may instruct the camera
6 1 to capture an external scene image when the movement of the user detected by the
9-axis sensor 66 is greater than a predetermined threshold value.
[0099] In step S304, the AR processing unit 142a specifies the position and the direction in
the three-dimensional information which correspond to the current position and the
eyeline of the user in a real space. In other words, in step S304, the AR processing unit
142a can be said to specify the current position and the eyeline of a virtual user in the
three-dimensional information. Specifically, the AR processing unit 142a uses any one
method of the following methods dl to d3.
[0100] (dl) The AR processing unit 142a specifies the current position and the eyeline of the
virtual user in the three-dimensional information by comparing the three-dimensional
information acquired in step S302 with the external scene image in the user's eyeline
acquired by the camera 6 1 at the time of step S304.
(d2) The AR processing unit 142a specifies the current position of the virtual user in
the three-dimensional information by comparing information of latitude and longitude
included in the three-dimensional information acquired in step S302 with the current
position coordinate (latitude and longitude) of the user acquired by the GPS module
134 at the time of step S304. The AR processing unit 142a specifies the eyeline of the
virtual user in the three-dimensional information using the movement of the user's head
acquired by the 9-axis sensor 66.
(d3) When plural signal generators are arranged in advance in a real space, the AR
processing unit 142a specifies the current position and the eyeline of the virtual user in
the three-dimensional information using intensity of radio waves or ultrasonic waves
received from the signal generators.
[0101] In step S306, the AR processing unit 142a generates a mask object. The "mask
object" is three-dimensional information viewed from the position and the direction
corresponding to the current position and the eyeline of the user in the real space, in
other words, three-dimensional information viewed from the current position and the
eyeline of the virtual user. The AR processing unit 142a generates the mask object
using the three-dimensional information obtained in step S302 and the positions and
the directions in the three-dimensional information specified in step S304.
[0102] Figs. 13A and 13B are diagrams illustrating an example of the mask object. Fig. 13A
illustrates an external scene SC viewed from the current position and the eyeline of the
user in the real space. The external scene SC is a scene of an office street in which
buildings are arranged in lines. Fig. 13B illustrates a mask object VM viewed from the
position and the direction in the three-dimensional information corresponding to the
current position and the eyeline of the user in the real space. In the mask object VM,
the ambient environment (external scene SC) of the user illustrated in Fig. 13A
includes three-dimensional objects such as "buildings", "roads", "trees", and "a
telephone pole", which are virtually expressed in a three-dimensional space.
[0103] The mask object is used to add a visual effect to a virtual object and is not displayed
as an image. In step S306 of Fig. 12, three-dimensional information may be acquired
from an external device such as a ground server connected to the head-mounted
display 100a on the basis of the position and the direction in the three-dimensional in
formation specified in step S304 and the mask object may be complemented.
[0104] In step S306 of Fig. 12, the AR processing unit 142a generates image data indicating
a virtual object which is displayed as a virtual image VI by the AR processing unit
142a in the augmented reality processing. Specifically, for example, the AR processing
unit 142a acquires arbitrary image data corresponding to the user's request or the
purpose of the processing from plural image data pieces stored in advance in the
storage unit 120.
[0105] In step S310, the AR processing unit 142a arranges the virtual object generated in
step S308 in the mask object generated in step S306. At the time of arrangement, the
AR processing unit 142a may consider the target distance at which the virtual object is
arranged in order to enable display of the virtual object OB having a sense of per
spective. The target distance can be determined, for example, using any one method of
the following methods.
- An external scene image in the user's eyeline acquired by the camera 6 1 is
analyzed.
- The current position coordinate of the user acquired by the GPS module 134 and
the movement of the user's head acquired by the 9-axis sensor 66 are analyzed.
[0106] Fig. 14 is a diagram illustrating an example of arrangement of a virtual object OB in
a mask object VM. The virtual object OB exemplified in Fig. 14 is a "bicycle". The
position at which the virtual object OB is arranged in the mask object VM is "one side
of a sidewalk in front of a tree". Here, when seen from the position and the direction
(the current position and the eyeline of a virtual user) in the three-dimensional in
formation specified in step S304, a "telephone pole" which is a part of the mask object
is arranged between the virtual object (bicycle) arranged in the mask object and the
user.
[0107] In step S312 of Fig. 12, the AR processing unit 142a adds an effect based on the
ambient environment to the mask object VM in which the virtual object OB is
arranged. In this embodiment, the "effect based on the ambient environment" means at
least one of the following effects e l to e3. The effects e l to e3 may be independently
employed or combinations thereof may be employed.
[0108] (el) The AR processing unit 142a trims a portion of the virtual object OB which is a
shadow of a three-dimensional object included in the mask object VM. For example, in
the example illustrated in Fig. 14, the AR processing unit 142a trims the portion of the
bicycle (virtual object OB) which is in the shadow of the telephone pole
(three-dimensional object included in the mask object VM), that is, the portion of the
bicycle which is not visible as the shadow of the telephone pole when seen from the
current position and the eyeline of a virtual user as illustrated in Fig. 14. According to
the effect el, the AR processing unit 142a can trim a part of the virtual object OB as if
the virtual object OB were in the shadow of a three-dimensional object (telephone pole
in the drawing) present in a real space. Accordingly, the image display unit 20 can
cause the user to visually recognize such a virtual image that the virtual object OB is
hidden by an object (telephone pole) present in a real space.
[0109] (e2) The AR processing unit 142a performs lighting based on the ambient en
vironment. Specifically, the AR processing unit 142a collects luminance information
of regions of the mask object VM by recognizing an external scene image captured by
the camera 61. The luminance information includes, for example, a shadow region,
specular reflection, and diffused reflection. The AR processing unit 142a determines a
type of light source to be added to the mask object VM, a color temperature of the light
source, intensity of the light source, and the like on the basis of the collected
luminance information. Here, examples of the type of light source include a point light,
a spot light, a directional light, an ambient light, a hemisphere light, and an imagebased
light (IBL). The AR processing unit 142a performs lighting of the virtual object
OB by arranging the light source having the determined type, color temperature, and
intensity in the mask object VM.
[01 10] Fig. 15 is a diagram illustrating a state of the virtual object OB subjected to lighting.
In the example illustrated in Fig. 15, since the virtual object OB is subjected to lighting
using the directional light DL, the shadow OBS of the virtual object OB is formed. In
this way, according to the effect e2, the AR processing unit 142a can correct the virtual
object OB so that the virtual object OB has brightness and darkness based on a real
space. Accordingly, the image display unit 20 can cause the user to visually recognize
a virtual image in which the virtual object OB is mixed with an object present in a real
space. The light source described in step S312 of Fig. 12 may be added alone or may
be added by combination. The AR processing unit 142a may collect the luminance in
formation of the regions in the mask object VM using a method (for example, com
putation based on the current time and orientation) other than the analysis of an
external scene image.
[0111] (e3) The AR processing unit 142a performs adjustment of behavior of the virtual
object OB based on at least one of a restitution coefficient and a friction coefficient of
each three-dimensional object included in the mask object VM. Specifically, the AR
processing unit 142a calculates at least one of the restitution coefficient and the friction
coefficient of the surface of each three-dimensional object included in the mask object
VM by recognizing the external scene image captured by the camera 61. At this time,
the AR processing unit 142a may calculate an average restitution coefficient or an
average friction coefficient of each three-dimensional object or the restitution co
efficient or the friction coefficient for each region of a three-dimensional object. The
AR processing unit 142a adjusts the behavior of the virtual object OB varying with the
lapse of time on the basis of the calculated restitution coefficient, the calculated
friction coefficient, or both thereof.
[01 12] Fig. 16 is a diagram illustrating a state where behavior of a virtual object OB is
adjusted on the basis of restitution coefficients and friction coefficients. In the example
illustrated in Fig. 16, a restitution coefficient XX (where X is an arbitrary number) and
a friction coefficient XX are calculated and stored for each region of a three-di
mensional object included in a mask object VM. The AR processing unit 142a changes
movement DI of a virtual object OB having a ball shape on the basis of the restitution
coefficients XX and the friction coefficients XX of the three-dimensional objects along
the movement DI. In the example illustrated in the drawing, the AR processing unit
142a adjusts the height with which the virtual object OB is bounded on the basis of the
restitution coefficient and the friction coefficient of the central lane of the road
including three lanes. In this way, according to the effect e3, the AR processing unit
142a can adjust the behavior of the virtual object OB on the basis of at least one of the
restitution coefficient and the friction coefficient of an object present in a real space.
Accordingly, the image display unit 20 can cause a user to visually recognize a virtual
image in which the virtual object OB moves with an influence of the object present in
the real space.
[01 13] In step S314 of Fig. 12, the AR processing unit 142a performs setting of a virtual
camera and 2D projection using the virtual camera. Specifically, the AR processing
unit 142a sets a virtual camera in the position and the direction (that is, the current
position and the eyeline of a virtual user in the mask object VM) in the mask object
VM corresponding to the current position and the eyeline of a user in a real space.
Then, the AR processing unit 142a converts the virtual object OB in the mask object
VM into information in a two-dimensional plane acquired by the virtual camera. Ac
cordingly, the AR processing unit 142a can acquire image data of the virtual object OB
having a visual effect added thereto by converting the virtual object OB in the mask
object VM into a two-dimensional image.
[01 14] Fig. 17 is a diagram illustrating image data acquired by 2D projection of a virtual
camera. In the example illustrated in Fig. 17, the image data DT includes a virtual
object (bicycle) in which a part of a front wheel is trimmed as the addition result of the
effect e l and a shadow OBS is added thereto as the addition result of the effect e2. At
the time of setting the virtual camera in step S314, it is preferable that the AR
processing unit 142a sets a depth of field of the virtual camera or the like on the basis
of the distance from the user's eyes to the user's point of regard. Then, it is possible to
add a defocus effect to the virtual object OB in the image data of the virtual object OB
acquired by the 2D projection of the virtual camera. The virtual camera set in step
S314 may be one camera, two virtual cameras corresponding to the user's right and left
eyes, and three or more virtual cameras.
[0115] Fig. 18 is a diagram illustrating an example of a virtual image VI which is visually
recognized by a user in the augmented reality processing according to the second em
bodiment. In step S316 of Fig. 12, the AR processing unit 142a transmits the image
data DT generated as described above to the image processing unit 160. Thereafter, by
performing the display process described with reference to Fig. 2, the user of the headmounted
display 100a can visually recognize the virtual object OB having the visual
effects (trimming and the shadow OBS) added thereto in the viewing field VR.
[0116] As described above, in the head-mounted display device (head-mounted display
100a) according to the second embodiment, the image display unit 20 causes the user
to visually recognize the virtual object OB with a user's visual discomfort reduced, that
is, the virtual object OB to which the visual effects (that is, visual effects for fitting the
virtual object to the ambient environment) for reducing the user's visual discomfort are
added, as a virtual image VI. Accordingly, it is possible to provide an optical
transmissive head-mounted display device which can provide augmented reality with a
user's visual discomfort reduced. The augmented reality processing unit (AR
processing unit 142a) fits the virtual object OB to the ambient environment by adding
the visual effect to the virtual object OB using the mask object VM generated from the
three-dimensional information. Accordingly, in comparison with an existing case in
which a virtual object is fitted to the ambient environment by adding a visual effect to
the virtual object on the basis of an image captured with a camera, it is possible to
further improve fitting accuracy.
[01 17] In the head-mounted display device (head-mounted display 100a) according to the
second embodiment, the augmented reality processing unit (AR processing unit 142a)
can add at least one visual effect of the trimming based on the environment (effect el),
the lighting based on the environment (effect e2), and the adjustment of behavior based
on the environment (effect e3) to the virtual object OB. The augmented reality
processing unit (AR processing unit 142a) can automatically estimate the ambient en
vironment of the user (specifically, luminance information of the effect e2 and the
restitution coefficients and the friction coefficients of the effect e3) by recognizing the
external scene image (an image in the user's viewing field) captured by the image
acquiring unit (camera 61).
[0118] C. Third Embodiment
A third embodiment of the invention describes a configuration in which a virtual
object of which the coordinate position in a real space is set is controlled on the basis
of the coordinate position and the shape of a real object in the real space. Only
elements having different configurations and operations from the first embodiment will
be described below. In the drawing, the same elements in the drawings as in the first
embodiment will be referenced by the same reference signs as described in the first
embodiment and detailed description thereof will not be repeated.
[0119] Fig. 19 is a diagram illustrating an example of a virtual image and a real scene which
can be visually recognized by a user. As illustrated in Fig. 19, a virtual image VI
generated by a head-mounted display 100b is displayed in the viewing field VR of a
user wearing the head-mounted display 100b according to this embodiment. Here, a
virtual object OB indicating a labyrinth navigator (guide) is displayed as the virtual
image VI. In the viewing field VR of the user, the user can view an external scene SC
through the right optical image display unit 26 and the left optical image display unit
28. Here, walls Al, A2, and A3 of a labyrinth as a real object A appear in the external
scene SC. That is, the user can simultaneously visually recognize the virtual image VI
and the external scene SC in the viewing field VR. Reference numeral 27 represents a
virtual image display region in which the virtual image VI is displayed (in which the
virtual image VI can be displayed) in the viewing field VR.
[0120] C-l. Configuration of Head-mounted Display Device
Fig. 20 is a block diagram illustrating a functional configuration of the head-mounted
display 100b according to the third embodiment. This embodiment is different from
the first embodiment illustrated in Fig. 2, in that a control unit 10b is provided instead
of the control unit 10 and an image display unit 20b is provided instead of the image
display unit 20.
[0121] The image display unit 20b is different from the image display unit of the first em
bodiment, in that it includes a distance sensor 67 and an eyeline sensor 69 instead of
the interpupiUary distance measuring unit 62. The distance sensor 67 detects a distance
from a user (strictly, the head-mounted display 100b) to a real object A, for example,
using an infrared ray method or an ultrasonic wave method. The eyeline sensor 69
detects a user's eyeline, for example, using an infrared ray method.
[0122] A storage unit 120b of the control unit 10b stores neither the interocular distance 122
nor the pixel parallactic angle 124 (Fig. 2), which is different from the first em
bodiment. The control unit 10b includes an AR processing unit 142b instead of the AR
processing unit 142 and includes an image analyzing unit 144 and an object detecting
unit 146, which is different from the first embodiment.
[0123] The image analyzing unit 144 acquires an image signal of an image captured by the
camera 6 1 and analyzes the acquired image signal. The image analyzing unit 144
recognizes an object recognition marker MK (see Fig. 19) disposed in a real object A
by the image analysis and recognizes the real object A on the basis of information
acquired from the marker MK. That is, the position of the real object A and the shape
of the real object A in the viewing field VR are recognized on the basis of information
of the position and the direction of the marker MK in the captured image, reading in
formation obtained by reading the marker MK, and information (the shape of the real
object A or the position of the marker MK in the real object A) of the real object A
stored in advance. In this embodiment, the marker MK is disposed in the real object A
and the real object A is recognized by recognizing the marker, but a configuration in
which a feature position (feature point) of the real object A is recognized without using
the marker and the real object A is recognized may be employed. The information of
the real object A may be acquired from an external server through the use of the
wireless communication unit 132 whenever an image is recognized.
[0124] The object detecting unit 146 detects the coordinate position and the shape of the real
object A in a real space on the basis of the recognition result of the real object A by the
image analyzing unit 144 and the detection result by a sensor. Specifically, the co
ordinate position of the real object A relative to a user in the real space and the shape
of the real object A are detected on the basis of the recognition result of the real object
A by the image analyzing unit 144, information of the distance from the user to the real
object A which is acquired from the distance sensor 67, and information of the
direction of the user's head acquired from the 9-axis sensor 66.
[0125] The AR processing unit 142b according to the third embodiment is different from the
AR processing unit according to the first embodiment, in process details of the
augmented reality processing to be performed. The AR processing unit 142b generates
display data to be displayed by the image display unit 20 and transmits the generated
display data to the image display unit 20 via the image processing unit 160. The AR
processing unit 142b sets the coordinate position in the real space for a virtual object
OB, and generates display data for displaying (so-called fitting) the virtual object OB
of which the coordinate position in the real space is set on the basis of a coordinate
system of the real space.
[0126] Specifically, the AR processing unit 142b sets the coordinate position relative to the
real object A in the real space for the virtual object OB on the basis of contents data.
Then, the AR processing unit 142b specifies the display position, the display size, and
the direction of the virtual object OB in the virtual image display region 27 which are
visually recognized on the basis of the coordinate system of the real space (which is
visually recognized in the coordinate system of the real space) on the basis of the
relative coordinate position of the virtual object OB, the detected coordinate position
of the real object A, information of the direction of a user's head, and information of
the user's eyeline acquired from the eyeline sensor 69. For example, when the virtual
object OB is present in the coordinate system of the real space, the display size is de
termined depending on the distance from the user to the virtual object OB. Ac
cordingly, the display size is specified by the distance from the user to the virtual
object OB which is calculated on the basis of the above-mentioned information. The
AR processing unit 142b enlarges/reduces, rotates, and arranges the virtual object OB
on the basis of the specified display position, display size, and direction to generate
display data including the virtual object OB.
[0127] When the virtual object OB is displayed as a stereoscopic image, display data in
which a parallax is added to the virtual object OB is generated as right-eye display data
and left-eye display data, the right-eye display data is transmitted to the right LCD
control unit 211 via the image processing unit 160, and the left-eye display data is
transmitted to the left LCD control unit 212 via the image processing unit 160. In this
case, an amount of parallax (sense of depth) of the virtual object OB is set on the basis
of the coordinate system of the real space. That is, similarly to the display size, since
the amount of parallax is determined depending on the distance from the user to the
virtual object OB, the distance from the user to the virtual object OB is calculated from
the above-mentioned information and the amount of parallax of the virtual object OB
is set on the basis of the calculated distance.
[0128] The AR processing unit 142b controls behavior of the virtual object OB on the basis
of the detected coordinate position and shape of the real object A. That is, it is
described above that the relative coordinate position in the real space is set on the basis
of the contents data, but strictly, the relative coordinate position is set on the basis of
the contents data and the detected coordinate position and shape of the real object A.
The virtual object OB is changed on the basis of the detected coordinate position and
shape of the real object A. As a result, the virtual object OB is caused to behave in con
sideration of the detected coordinate position and shape of the real object A.
[0129] C-2. Augmented Reality Processing
Fig. 2 1 is a flowchart illustrating an augmented reality processing procedure
according to the third embodiment. The augmented reality processing according to the
third embodiment is performed with the manipulation of starting an operation in the
control unit 10b.
[0130] As illustrated in Fig. 21, the head-mounted display 100b first captures an image of an
external scene SC by the use of the camera 6 1 (SI). The image analyzing unit 144
recognizes a marker MK in the image captured by the camera 6 1 and recognizes the
real object A (S2). In the example illustrated in Fig. 2, the walls Al and A2 for rec
ognizing the marker MK are recognized from the captured image.
[0131] When the real object A is recognized, the distance from the user to the real object A
is detected by the distance sensor 67 (S3) and the direction of the user's head is
detected by the 9-axis sensor 66 (S4). Then, the coordinate position (coordinate
position relative to the user) and the shape of the real object A are detected by the
object detecting unit 146 on the basis of the recognition result of the real object A by
the image analyzing unit 144, information of the detected distance from the user to the
real object A, and information of the detected direction of the head (S5).
[0132] When the coordinate position and the shape of the real object A are detected, the
user's eyeline is detected by the eyeline sensor 69 (S6). The viewing field VR (the
direction thereof) of the user is estimated by the AR processing unit 142b on the basis
of information of the detected eyeline and information of the detected direction of the
head (S7).
[0133] Thereafter, the AR processing unit 142b sets the coordinate position (coordinate
position relative to the real object A) of the virtual object OB in the real space on the
basis of the contents data and the detected coordinate position and shape of the real
object A (S8). The AR processing unit determines the display position, the display
size, and the direction of the virtual object OB so as to visually recognize the virtual
object in the coordinate system of the real space on the basis of the estimated viewing
field VR, the set coordinate position (coordinate position relative to the real object A)
of the virtual object OB, and the detected coordinate position (coordinate position
relative to the user) of the real object A (S9), and generates display data including the
virtual object OB (S10). Thereafter, the AR processing unit transmits the generated
display data to the image display unit 20 via the image processing unit 160 (SI 1). The
"display control step" mentioned in the appended claims corresponds to steps S6 to
S10.
[0134] The image display unit 20 displays the display data including the virtual object OB
(projects image light to a retina) (S12, display step). Then, the display operation ends.
[0135] Behavior control of the virtual object OB by the AR processing unit 142b will be
described below with reference to Figs. 22 to 24. As described above, the AR
processing unit 142b controls the behavior of the virtual object OB on the basis of the
detected coordinate position and shape of the real object A. Fig. 22 is a transition
diagram illustrating a first example of the behavior control of the virtual object OB by
the AR processing unit 142b. In the example illustrated in Fig. 22, the AR processing
unit 142b changes the virtual object OB so that the overlapping portion OBa of the
virtual object OB with the real object A disappears when (it is estimated that) the
virtual object OB is located in the back of the real object A from the viewpoint of the
user on the basis of the detected coordinate position of the real object A and the set co
ordinate position of the virtual object OB. Specifically, as illustrated in the drawing,
when the virtual object OB is caused to move to the back of the wall Al from the
viewpoint of the user, the virtual object OB is changed so that the overlapping portion
OBa with the wall Al disappears from the viewpoint of the user. In this way, the
virtual object OB is displayed as if the virtual object is hidden by the wall Al from the
viewpoint of the user.
[0136] Fig. 23 is a transition diagram illustrating a second example of the behavior control
of the virtual object OB by the AR processing unit 142b. In the example illustrated in
Fig. 23, the AR processing unit 142b causes the virtual object OB to behave so as to
avoid the real object A in the coordinate system of the real space. Specifically, as i l
lustrated in the drawing, when the virtual object OB moves to the vicinity of a pillar
A4 as an obstacle, the trajectory of the virtual object OB is changed and the virtual
object is displayed as if it avoided the real object A.
[0137] Fig. 24 is a transition diagram illustrating a third example of the behavior control of
the virtual object OB by the AR processing unit 142b. In the example illustrated in Fig.
24, when the virtual object OB moves and reaches the real object A in the coordinate
system of the real space, the AR processing unit 142b causes the virtual object OB to
behave as if it collided with the real object A. Specifically, as illustrated in the
drawing, when the virtual object OB moves until it reaches the wall A2, the trajectory
of the virtual object OB is changed to rebound from the wall A2 and the virtual object
is displayed as if it collided with the real object A. At the time point at which the
virtual object OB reaches the wall Al, the virtual object OB is changed (the size is
reduced in the collision direction in the example illustrated in the drawing) as if the
virtual object OB collides with the real object A.
[0138] In the augmented reality processing according to the third embodiment, since the
virtual object OB can be caused to behave in consideration of the coordinate position
or the shape of the real object A, it is possible to improve a sense of reality. Ac
cordingly, it is possible to provide an optical transmissive head-mounted display
device (head-mounted display 100b) which can provide augmented reality with a user's
visual discomfort reduced. By causing the virtual object OB to behave in consideration
of the coordinate position or the shape of the real object A, a transmissive headmounted
display device can provide contents coupled to a real object A. Therefore, it is
possible to provide a game or navigation service in which a virtual object OB is
coupled to a real object A. In an example of the contents of the navigation service, it is
thought that a guide of a labyrinth is displayed as a virtual object OB and the guide
shows a path while avoiding the walls Al, A2, and A3 in consideration of the walls
Al, A2, and A3 of the labyrinth. In an example of the contents of the game, it is
thought that a ball is displayed as a virtual object OB and the thrown ball reaches the
walls Al, A2, and A3 and rebounds therefrom.
[0139] By detecting an eyeline by the use of the eyeline sensor 69 and displaying a virtual
object OB in consideration of a user's eyeline, it is possible to fit the virtual object OB
(to determine the display position, the display size, and the direction of the virtual
object OB) with high accuracy.
[0140] By detecting a distance from a user to a real object A by the use of the distance
sensor 67 and displaying a virtual object OB in consideration of the distance, it is
possible to fit the virtual object OB with high accuracy.
[0141] This embodiment describes the configuration in which the coordinate position
relative to the real object A is set as the coordinate position of the virtual object OB in
the real space, but a configuration in which the coordinate position relative to a user is
set as the coordinate position of the virtual object OB in the real space or a con
figuration in which the absolute position is set as the coordinate position of the virtual
object OB in the real space may be employed.
[0142] This embodiment describes the configuration in which the coordinate position
relative to a user is detected as the coordinate position of a real object A in the real
space, but a configuration in which the absolute position is detected as the coordinate
position of the real object A in the real space may be employed.
[0143] This embodiment describes the configuration in which a user's eyeline is detected
and the user's viewing field is detected on the basis of the user's eyeline and the
direction of the user's head, but a configuration in which the user's eyeline is de
termined to always face the front side and the user's viewing field is detected on the
basis of only the direction of the user's head may be employed.
[0144] In this embodiment, the marker MK for recognizing an object may be an AR marker
or a two-dimensional code (such as a QR code) or a barcode.
[0145] This embodiment describes the configuration in which an image of a real scene is
captured by the camera 6 1 and the coordinate position and the shape of a real object A
are detected by recognition of the captured image, but a configuration in which map
data including the coordinate positions and the shapes of all real objects A within a
predetermined range in the real space is stored in advance and the coordinate position
and the shape of the real object A are detected on the basis of the map data may be
employed.
[0146] This embodiment may employ a configuration in which the interocular distance, or
the right and left parallactic angle, the convergence angle, the dominant eye, the
parallax with an image, a user's relative speed to a real object, or the like is detected to
enhance fitting accuracy of a virtual object OB and the virtual object OB is fitted in
consideration of the detected data.
[0147] It is preferable that a light source environment of an external scene SC is estimated
and the color and the brightness of a virtual object OB are adjusted depending on the
light source environment. It is also preferable that the color and the brightness of a
virtual object OB are determined in consideration of a distance from a user to a real
object A and a variation in visual environment of a user (for example, dark adaptation
or light adaptation).
[0148] This embodiment may employ a configuration in which a distance from a user to a
virtual object OB in the coordinate system of the real space is calculated and the focal
distance of the virtual object OB is changed.
[0149] This embodiment may also employ a configuration in which a user operation in
contents is performed by a controller not illustrated in the drawings or a configuration
in which a part of a user's body (such as a finger or a hand) and behavior of the part of
the body is received as the user operation. For example, a user's hand, finger, arm, or
the like is recognized and the AR processing unit 142b displays a virtual object OB
such as a bat, a racket, a trump, or a pen as if they were carried with the user's hand.
The user moves the hand, the finger, the arm, or the like to operate the virtual object
OB and this operation may be received as a user operation. In this case, when the
virtual object OB is located in the back of the user's hand, finger, arm, or the like as a
real object A from the viewpoint of the user, the virtual object OB is changed so that
the overlapping portion of the virtual object OB with the real object A is deleted. The
real object A (such as the bat, the racket, the trump, or the pen) which can be handled
by the user is recognized and behavior of the real object A may be received as the user
operation. In addition, the movement (for example, the direction) of a user's head or a
user's eyeline is detected and the detected data may be received as the user operation.
[0150] The virtual object OB to be displayed is not limited to the above-mentioned objects,
but a virtual object OB such as an arrow or a mark may be displayed. A part of an
object or a human body (a finger tip, an arm, or the like) in a real space may be imaged
and received and the captured image may be displayed as a virtual object OB.
[0151] D. Modification Example
In the above-mentioned embodiments, a part of elements embodied by hardware may
be replaced with software, or a part of elements embodied by software may be replaced
with hardware. In addition, the following modifications can be considered.
[0152] Modification Example 1
The above-mentioned embodiments describe the configuration of the head-mounted
display. However, the configuration of the head-mounted display can be arbitrarily de
termined without departing from the gist of the invention, and for example, the
elements may be added, deleted, changed, and the like.
[0153] In the above-mentioned embodiments, allocation to the control unit and the image
display unit is only an example and various aspects can be employed. For example, the
following aspects may be considered: (i) an aspect in which the control unit is provided
with processing functions of a CPU or a memory and the image display unit is
provided with only a display function; (ii) an aspect in which both the control unit and
the image display unit are provided with the processing functions of the CPU or the
memory; (iii) an aspect in which the control unit and the image display unit are in
corporated into a single unit (for example, an aspect in which the image display unit
includes the control unit and the whole configuration serves as a glasses-like wearable
computer); (iv) an aspect in which a smart phone or a portable game machine is used
instead of the control unit; and (v) an aspect in which the control unit and the image
display unit are configured to wirelessly communicate with each other or to be
supplied with power in a wireless manner and the connecting section (cord) is made
unnecessary.
[0154] The above-mentioned embodiments describe the configuration in which the control
unit includes the transmitter unit and the image display unit includes the receiver unit,
for the purpose of convenience of explanation. However, the transmitter unit and the
receiver unit in the above-mentioned embodiments may have a bidirectional commu
nication function and may serve as a transceiver unit. For example, the control unit il
lustrated in Fig. 2 is connected to the image display unit via a wired signal
transmission line. However, the control unit and the image display unit may be
connected to each other via a wireless signal transmission line such as a wireless LAN,
infrared communication, or Bluetooth (registered trademark).
[0155] For example, the configurations of the control unit and the image display unit i l
lustrated in Fig. 2 can be arbitrarily modified. Specifically, for example, the touch pad
may be removed from the control unit and the operation may be carried out using only
the cross key. The control unit may be provided with another operation interface such
as an operational stick. The control unit may be configured to be connected to a device
such as a keyboard or a mouse and may receive an input from the keyboard or the
mouse. For example, an operation input using a foot switch (a switch operated with a
user's foot) may be acquired in addition to the operation inputs using the touch pad and
the cross key. For example, the image display unit may be provided with an eyeline
detecting unit such as an infrared sensor, a user's eyeline may be detected, and an
operation input based on a command correlated with the movement of the eyeline may
be acquired. For example, a user's gesture may be detected using a camera and an
operation input based on a command correlated with the detected gesture may be
acquired. At the time of detecting the gesture, a user's finger tip, a ring worn by a user's
finger, a medical instrument grasped by a user's hand, or the like may be moved and
may be used as a mark for detection. When the operation input using the foot switch or
the eyeline can be acquired, the input information acquiring unit can acquire an
operation input from a user even in work in which it is difficult to unhand.
[0156] For example, the head-mounted display is described above as a binocular type
transmissive head-mounted display, but a monocular type head-mounted display may
be employed. A non-transmissive head-mounted display that blocks an external scene
in a state where a user wears the head-mounted display may be employed.
[0157] Figs. 25A and 25B are diagrams illustrating an outer configuration of a headmounted
display according to a modification example. In the example illustrated in
Fig. 25A, the head-mounted display is different from the head-mounted display 100 il
lustrated in Fig. 1, in that an image display unit 20a includes a right optical image
display unit 26a instead of the right optical image display unit 26 and includes a left
optical image display unit 28a instead of the left optical image display unit 28. The
right optical image display unit 26a is formed with a size smaller than the optical
member according to the first embodiment and is disposed on the oblique upper side of
a user's right eye in a state where the user wears the head-mounted display. Similarly,
the left optical image display unit 28a is formed with a size smaller than the optical
member according to the first embodiment and is disposed on the oblique upper side of
a user's left eye in a state where the user wears the head-mounted display. In the
example illustrated in Fig. 25B, the head-mounted display is different from the headmounted
display 100 illustrated in Fig. 1, in that an image display unit 20b includes a
right optical image display unit 26b instead of the right optical image display unit 26
and includes a left optical image display unit 28b instead of the left optical image
display unit 28. The right optical image display unit 26b is formed with a size smaller
than the optical member according to the first embodiment and is disposed on the
oblique lower side of a user's right eye in a state where the user wears the headmounted
display. Similarly, the left optical image display unit 28b is formed with a
size smaller than the optical member according to the first embodiment and is disposed
on the oblique lower side of a user's left eye in a state where the user wears the headmounted
display. In this way, the optical image display unit has only to be disposed in
the vicinity of the user's eyes. The size of the optical member constituting the optical
image display unit is not limited, and a head-mounted display of an aspect in which the
optical image display unit covers a part of the user's eye, that is, an aspect in which the
optical image display unit does not fully cover the user's eye, may be embodied.
[0158] For example, it is described above that the functional units such as the image
processing unit, the display control unit, the AR processing unit, and the sound
processing unit are embodied by causing the CPU to develop a computer program
stored in a ROM or a hard disk into a RAM and to execute the developed computer
program. However, the functional units may be embodied by an application specific integrated
circuit (ASIC) designed to perform the functions.
[0159] For example, the above-mentioned embodiments describe that the image display unit
is a head-mounted display which is mounted like a pair of glasses, but the image
display unit may be a general panel display device (such as a liquid crystal display
device, a plasma display device, and an organic EL display device). In this case, the
control unit and the image display unit may be connected to each other via a wired
signal transmission line or via a wireless signal transmission line. By employing this
configuration, the control unit may be used as a remote controller of a general panel
display device.
[0160] An image display unit having a different shape such as an image display unit which
is mounted like a hat may be employed as the image display unit instead of the image
display unit which is mounted like a pair of glasses. The earphone may be of an ear
hook type or of a head band type or may be skipped. For example, the image display
unit may be embodied as a head-up display (HUD) mounted on a vehicle such as an
automobile or an airplane. For example, the image display unit may be embodied as a
head-mounted display built into a body protecting instrument such as a helmet.
[0161] For example, the above-mentioned embodiments describe that a secondary battery is
used as the power supply, but the power supply is not limited to the secondary battery
and various batteries may be used. For example, a primary battery, a fuel cell battery, a
solar cell battery, and a thermal battery may be used.
[0162] For example, the above-mentioned embodiments describe that the image light
generating unit includes the backlight, the backlight control unit, the LCD, and the
LCD control unit. However, this configuration is only an example. The image light
generating unit may include a constituent unit for embodying another scheme in
addition to the constituent units or instead of the constituent units. For example, the
image light generating unit may include an organic electroluminescence (EL) display
and an organic EL control unit. For example, the image light generating unit may
include a digital micromirror device instead of the LCD. For example, the invention
can be applied to a laser retina projection type head-mounted display device.
[0163] Modification Example 2
The first embodiment describes an example of the augmented reality processing
procedure. However, the augmented reality processing procedure is only an example
and can be modified in various forms. For example, some steps may be skipped or
another step may be added thereto. The order of the steps to be performed may be
changed.
[0164] The above-mentioned embodiments describe that the pixel parallactic angle is
defined as a difference between the convergence angle of a virtual image displayed on
the basis of image data which are laterally identical to each other and the convergence
angle of a virtual image displayed on the basis of image data which are laterally
deviated by one pixel from each other. However, the pixel parallactic angle has only to
be a difference between the convergence angle of a virtual image displayed on the
basis of image data which are laterally identical to each other and the convergence
angle of a virtual image displayed on the basis of image data which are laterally
deviated from each other, and the amount of deviation of the right and left image data
is not limited to one pixel.
[0165] For example, in the augmented reality processing procedure, when the calculated
target parallactic angle theta x is greater than the parallelism (that is, theta x<-theta a/
2), the AR processing unit can suppress generation of image data for embodying the
target parallactic angle theta x. Accordingly, it is possible to reduce a user's discomfort.
[0166] When the parallactic angle is excessively great, a user can easily feel discomfort. Ac
cordingly, for example, when the calculated target parallactic angle theta x is equal to
or greater than a predetermined threshold value in the augmented reality processing
procedure (that is, theta x>theta lim), the AR processing unit can suppress generation
of image data for embodying the target parallactic angle theta x. The predetermined
threshold value (theta lim) is preferably set to a "boundary between the parallax at
which an object appears dual and the parallax at which an object appears singly for the
user". The predetermined threshold value (theta lim) may be adjusted depending on a
user's taste.
[0167] The AR processing unit may adjust chroma, brightness, and contrast of the right-eye
image data and the left-eye image data depending on the determined magnitude of the
target distance Lb or the calculated magnitude of the target parallactic angle theta x.
For example, by lowering at least one of chroma, brightness, and contrast of image
data indicating a virtual object to be displayed distantly, it is possible to emphasize a
user's feeling of perspective.
[0168] The parallactic angle theta pix stored in the pixel parallactic angle may be adjusted
depending on a user's taste. A user can adjust the parallactic angle theta pix to be
smaller when the user wants to emphasize the sense of perspective, and can adjust the
parallactic angle theta pix to be greater when the user wants to alleviate the sense of
perspective.
[0169] Modification Example 3
In the first embodiment, the AR processing unit may perform the augmented reality
processing procedure by matching an external scene image in a user's viewing field
acquired by the camera with patterns using the pixel parallactic angle. Specifically, the
image display unit includes a right-eye camera and a left-eye camera. The right-eye
camera is a camera which is disposed at a position of the image display unit corre
sponding to a user's right eye and which can capture an image of an external scene on
the front side of the image display unit. The left-eye camera is a camera which is
disposed at a position of the image display unit corresponding to a user's left eye and
which can capture an image of an external scene on the front side of the image display
unit. The AR processing unit may calculate an amount of deviation between a target
object (an object in which a virtual object is displayed in the vicinity thereof) included
in the image captured by the right-eye camera and a target object included in the image
captured by the left-eye camera and may determine the "target distance" in the
augmented reality processing procedure using the amount of deviation and the pixel
parallactic angle.
[0170] Modification Example 4
In the first embodiment, the AR processing unit may perform the augmented reality
processing procedure only when a predetermined condition is satisfied. For example,
the image display unit may be provided with a configuration for detecting a user's
eyeline and the AR processing unit may perform the augmented reality processing
procedure only when the detected eyeline satisfies any one of the following conditions.
- A range of viewing angle of about 200 degree horizontally and about 125 degree
vertically (for example, 75 degree downward and 50 degree upward)
- A range of viewing angle of about 30 degree horizontally and about 20 degree
vertically as an effective viewing field in which information acceptance capability is
excellent
- A range of viewing angle of about 60 degree to 90 degree horizontally and about 45
degree to 70 degree vertically as a stable field of fixation in which a point of regard is
rapidly and stably viewed
- A range of viewing angle of from about 20 degree horizontally at which self-motion
perception (vection) induced in an image starts to about 110 degree at which the selfmotion
perception is saturated
[0171] Modification Example 5
The second embodiment describes an example of the augmented reality processing
procedure. However, the augmented reality processing procedure is only an example
and can be modified in various forms. For example, some steps may be skipped or
another step may be added thereto. The order of the steps to be performed may be
changed.
[0172] The augmented reality processing procedure according to the second embodiment is
described above as an alternative procedure of the augmented reality processing
procedure according to the first embodiment. However, the augmented reality
processing procedure according to the second embodiment may be performed in
parallel with the augmented reality processing procedure according to the first em
bodiment.
[0173] In step S302 of the augmented reality processing procedure according to the second
embodiment, the methods c l to c3 of acquiring three-dimensional information are ex
emplified. However, the methods are only examples, and the AR processing unit may
acquire the three-dimensional information using methods other than the methods c l to
c3. The AR processing unit may correct the three-dimensional information acquired
using one method (for example, the method c3) on the basis of information acquired
using another method (for example, the method cl).
[0174] In the augmented reality processing procedure according to the second embodiment,
in order to adjust balance of the size of the real space with the size of the virtual object,
a process of artificially matching (calibrating) the size of a region which can be imaged
with the camera and the size of a region which is projected as a virtual image onto a
user's eyes may be performed.
[0175] The effects e l to e3 in step S312 of the augmented reality processing procedure
according to the second embodiment are only examples and the effects to be added by
the AR processing unit can be arbitrarily determined. For example, the AR processing
unit may add the following effect instead of the effects e l to e3 or in addition to the
effects e l to e3. Specifically, the AR processing unit adds an effect for providing a
sense of perspective to a virtual object depending on the target distance at which the
virtual object is arranged. As the effect for providing a sense of perspective, a fog
effect can be employed when the target distance is great, and a sharpness effect can be
employed when the target distance is small.
[0176] Modification Example 6
The invention is not limited to the above-mentioned embodiments, examples, and
modification examples, but can be modified in various forms without departing from
the gist of the invention. For example, the technical features of the embodiments, the
examples, and the modification examples corresponding to the technical features of the
aspects described in the summary of invention can be appropriately exchanged or
combined in order to solve all or a part of the above-mentioned problems or to achieve
all or a part of the above-mentioned effects. The technical features can be appropriately
deleted when the technical features are not described to be essential in the speci
fication.
Reference Signs List
[0177] 10: control unit (controller)
12: lighting unit
14: touch pad
16: cross key
18: power supply switch
: image display unit
: right support
: right display drive unit
: left support
: left display drive unit
: right optical image display unit
: left optical image display unit
: earphone plug
: right earphone
: left earphone
: connecting section
: right cord
: left cord
: connecting member
: main cord
: transmitter unit
: transmitter unit
: receiver unit
: receiver unit
: camera
: distance sensor
: eyeline sensor
: interpupillary distance measuring unit
0: input information acquiring unit
0, 100a, 100b: head-mounted display (head-mounted display device)
0: storage unit
2: interocular distance (interocular distance storage unit)
4: pixel parallactic angle (pixel parallactic angle storage unit)
0: power supply
0: CPU
2, 142a, 142b: AR processing unit (augmented reality processing unit)
4: image analyzing unit
6: object detecting unit
0: image processing unit
0: sound processing unit
0: interface
0: display control unit
1: right backlight control unit
202: left backlight control unit
2 11: right LCD control unit
212: left LCD control unit
221: right backlight
222: left backlight
241: right LCD
242: left LCD
251: right projection optical system
252: left projection optical system
261: right light guide plate
262: left light guide plate
PCLK: clock signal
VSync: vertical synchronization signal
HSync: horizontal synchronization signal
Data: image data
Datal: right-eye image data
Data2: left-eye image data
OA: external device
PC: personal computer
SC: external scene
RE: right eye
LE: left eye
VI: virtual image
ER: end portion
EL: end portion
AP: tip portion
VR: viewing field
VD: viewpoint direction
IV: virtual viewpoint
OB: virtual object
OBI: virtual object
OB2: virtual object
OBS: shadow of virtual object
OBa : overlapping portion
CM: virtual camera
CML: virtual camera
CMR: virtual camera
DI: movement of virtual object
VM: mask object
theta a : initial convergence angle (first convergence angle)
theta b : target convergence angle
theta c : convergence angle (second convergence angle theta)
theta x: target parallactic angle
A: object
MK: marker

Claims
[Claim 1] A head-mounted display device that enables a user to visually
recognize a virtual image and an external scene, comprising:
an image display unit that causes the user to visually recognize the
virtual image; and
an augmented reality processing unit that forms on the image display
unit the virtual image indicating a virtual object which is an object for
providing augmented reality to the user and of which the user's visual
discomfort is reduced.
[Claim 2] The head-mounted display device according to claim 1, wherein the
augmented reality processing unit reduces the discomfort of the virtual
object by fitting the virtual object with an ambient environment of the
user using three-dimensional information in which the ambient en
vironment is expressed as a virtual three-dimensional object in a threedimensional
space.
[Claim 3] The head-mounted display device according to claim 2, wherein the
augmented reality processing unit arranges the virtual object in the
three-dimensional information and reduces the discomfort of the virtual
object by adding a visual effect based on the environment to at least
one of the virtual object and the three-dimensional information and
then converting the virtual object into a two-dimensional image.
[Claim 4] The head-mounted display device according to claim 3, wherein the
visual effect based on the environment includes any one of:
trimming a portion of the virtual object which is a shadow of the threedimensional
object in the three-dimensional information;
lighting the virtual object based on the environment; and
adjusting behavior of the virtual object based on at least one of a
restitution coefficient and a friction coefficient of the three-dimensional
object in the three-dimensional information set on the basis of the en
vironment.
[Claim 5] The head-mounted display device according to claim 3, further
comprising an image acquiring unit that acquires an image in the user's
viewing field in a state where the user wears the head-mounted display
device,
wherein the augmented reality processing unit estimates the en
vironment by recognizing the image in the viewing field acquired by
the image acquiring unit.
WO 2014/156033 PCT/JP2014/001464
[Claim 6] The head- mounted display device according to claim 3, wherein the
augmented reality processing unit considers a distance from the user's
eyes to the user's point of regard to convert the virtual object into a
two-dimensional image.
[Claim 7] The head- mounted display device according to claim 1, wherein the
augmented reality processing unit adds a visual effect of enabling a
stereoscopic view of the virtual object to the virtual object to reduce the
discomfort of the virtual object by generating right-eye image data for a
right eye and left-eye image data for a left eye which indicate the
virtual object,
wherein the image display unit causes the user's right and left eyes to
visually recognize the different virtual images using the right-eye
image data and the left-eye image data,
wherein the head-mounted display device further comprises a pixel p ar
allactic angle storage unit that stores a difference between a first con
vergence angle of the virtual image displayed on the basis of the righteye
image data and the left-eye image data which are identical to each
other and a second convergence angle of the virtual image displayed on
the basis of the right-eye image data and the left-eye image data which
are laterally deviated from each other, and
wherein the augmented reality processing unit generates the right-eye
image data and the left-eye image data for fitting the virtual object to
the external scene using the difference stored in the pixel parallactic
angle storage unit.
[Claim 8] The head-mounted display device according to claim 7, wherein the
augmented reality processing unit
determines a target distance at which the user is caused to visually
recognize the virtual object,
calculates a target convergence angle, which is a convergence angle at
the target distance, from the determined target distance,
calculates a target parallactic angle which is a parallactic angle at the
target distance using the calculated target convergence angle and the
first convergence angle, and
generates the right-eye image data and the left-eye image data from
single image data indicating the virtual object using the target p ar
allactic angle and the difference stored in the pixel parallactic angle
storage unit.
[Claim 9] The head- mounted display device according to claim 7, wherein the
WO 2014/156033 PCT/JP2014/001464
augmented reality processing unit
determines a target distance at which the user is caused to visually
recognize the virtual object,
sets two virtual cameras for acquiring a 2D projection image in a 3D
model space on the basis of the determined target distance,
calculates a target convergence angle, which is a convergence angle at
the target distance, from the determined target distance,
calculates a target parallactic angle which is a parallactic angle at the
target distance using the calculated target convergence angle and the
first convergence angle, and
generates the right-eye image data by scaling the 2D projection image
captured by one virtual camera and generates the left-eye image data by
scaling the 2D projection image captured by the other virtual camera
using the target parallactic angle and the difference stored in the pixel
parallactic angle storage unit.
[Claim 10] The head-mounted display device according to claim 9, further
comprising an interocular distance storage unit that stores the user's interocular
distance,
wherein the augmented reality processing unit arranges a virtual
viewpoint which is a virtual point of view in the 3D model space on the
basis of the determined target distance at the time of setting the two
virtual cameras, arranges one virtual camera at a position separated by
the interocular distance/2 from the arranged virtual viewpoint, and
arranges the other virtual camera at a position separated by the in
terocular distance/2 from the arranged virtual viewpoint.
[Claim 11] The head-mounted display device according to claim 10, further
comprising an interpupiUary distance measuring unit that measures the
user's interpupiUary distance,
wherein the measurement result by the interpupiUary distance
measuring unit is stored as the interocular distance in the interocular
distance storage unit.
[Claim 12] The head-mounted display device according to claim 1, further
comprising an object detecting unit that detects a coordinate position
and a shape of a real object in a real space,
wherein the augmented reality processing unit reduces the discomfort
of the virtual object by controlling the virtual object, of which a co
ordinate position in the real space is set, on the basis of the coordinate
position and the shape of the real object.
PCT/JP2014/001464
The head-mounted display device according to claim 12, wherein the
augmented reality processing unit controls behavior of the virtual
object on the basis of the coordinate position and the shape of the real
object.
The head-mounted display device according to claim 13, wherein the
augmented reality processing unit changes the virtual object on the
basis of the detected coordinate position of the real object and the set
coordinate position of the virtual object so as to make an overlapping
portion of the virtual object with the real object disappear when the
virtual object is located in the back of the real object from the user's
viewpoint.
The head-mounted display device according to claim 13, wherein the
augmented reality processing unit causes the virtual object to behave so
as to avoid the real object in a coordinate system of a real space.
The head-mounted display device according to claim 13, wherein the
augmented reality processing unit causes the virtual object to behave as
if the virtual object collided with the real object when the virtual object
moves and reaches the real object in the coordinate system of the real
space.
The head-mounted display device according to claim 12, further
comprising an eyeline detecting unit that detects the user's eyeline,
wherein the augmented reality processing unit displays the virtual
object on the basis of the detected eyeline.
The head-mounted display device according to claim 12, further
comprising a distance detecting unit that detects a distance from the
user to the real object,
wherein the augmented reality processing unit displays the virtual
object on the basis of the detected distance.
The head-mounted display device according to claim 12, wherein the
image display unit includes:
an image light output unit that outputs image light of the image; and
a light guide section that guides the output image light to the user's
eyes,
wherein the light guide section transmits external light and causes the
external light along with the image light to be incident on the user's
eyes.
A transmissive display system that causes a user to visually recognize a
display image as a virtual image, comprising:
WO 2014/156033 PCT/JP2014/001464
an object detecting unit that detects a coordinate position and a shape of
a real object in a real space;
an augmented reality processing unit that displays a virtual object, of
which a coordinate position in the real space is set, so as to visually
recognize the virtual object on the basis of a coordinate system of the
real space; and
an image display unit that displays the display image,
wherein the augmented reality processing unit controls the virtual
object on the basis of the coordinate position and the shape of the real
object.
[Claim 21] A control method of a head-mounted display device that enables a user
to visually recognize a virtual image and an external scene, comprising:
causing the user to visually recognize the virtual image; and
forming the virtual image indicating a virtual object which is an object
for providing augmented reality to the user and of which the user's
visual discomfort is reduced in the causing of the user to visually
recognize the virtual image.
[Claim 22] The control method of a head-mounted display device according to
claim 21, wherein the forming of the virtual image on the image display
unit includes reducing the discomfort of the virtual object by fitting the
virtual object with an ambient environment of the user using threedimensional
information in which the ambient environment is
expressed as a virtual three-dimensional object in a three-dimensional
space.
[Claim 23] The control method of a head-mounted display device according to
claim 21, wherein the forming of the virtual image on the image display
unit includes adding a visual effect of enabling a stereoscopic view of
the virtual object to the virtual object to reduce the discomfort of the
virtual object by generating right-eye image data for a right eye and
left-eye image data for a left eye which indicate the virtual object,
wherein the causing of the user to visually recognize the virtual image
includes causing the user's right and left eyes to visually recognize the
different virtual images using the right-eye image data and the left-eye
image data, and
wherein the forming of the virtual image on the image display unit
includes generating the right-eye image data and the left-eye image
data for fitting the virtual object to the external scene using a difference
between a first convergence angle of the virtual image displayed on the
WO 2014/156033 PCT/JP2014/001464
basis of the right-eye image data and the left-eye image data which are
identical to each other and a second convergence angle of the virtual
image displayed on the basis of the right-eye image data and the lefteye
image data which are laterally deviated from each other.
[Claim 24] The control method of a head-mounted display device according to
claim 21, further comprising detecting a coordinate position and a
shape of a real object in the real space,
wherein the forming of the virtual image on the image display unit
includes reducing the discomfort of the virtual object by controlling the
virtual object, of which a coordinate position in the real space is set, on
the basis of the coordinate position and the shape of the real object.