Technical Field
[0001] The present invention relates to a head mounted display.
Background Art
[0002] A head mounted display (HMD) which is a display mounted on the head is known.
The HMD generates image light representing an image by using, for example, a liquid
crystal display and a light source, and guides the generated image light to a user's eyes
by using a projection optical system or a light guiding plate, thereby allowing the user
to visually recognize a virtual image.
[0003] PTL 1 discloses a surgical imaging system and a surgical robot using such a head
mounted display. In the technique disclosed in PTL 1, a user (surgeon) of the head
mounted display operates a surgical manipulator while visually recognizing, as a
virtual image, an image with an endoscope which is inserted into a body cavity of a
patient, thereby performing surgery. PTL 2 discloses a face fitting image display
device for medical use (head mounted display). In the technique disclosed in PTL 2,
when a user of the face fitting display device visually recognizes an endoscope image,
the user directs his/her visual line in a horizontal direction and views an LCD disposed
in front of a main body. In addition, when the user visually recognizes external
scenery, the user directs his/her visual line in a vertical direction and views an ob
servation window disposed under the main body.
Citation List
Patent Literature
[0004] PTL 1: JP-A-2012-223363
PTL 2: Japanese Patent No. 3599854
Summary of Invention
Technical Problem
[0005] In the technique disclosed in PTL 1 and PTL 2, the user of the head mounted display
can visually recognize an endoscope image during surgery. However, in the technique
disclosed in PTL 1, there is a problem in that the user of the head mounted display
cannot directly view a surgical site (a patient's body). In addition, in the technique
disclosed in PTL 2, there is a problem in that the user of the head mounted display is
required to move his/her visual line in the vertical direction in order to directly view
the surgical site.
[0006] In addition, these problems are not limited to an endoscope image, and are problems
common to all cases where information which is not shown in an appearance of an
object is provided to a user of the head mounted display, such as, for example, a
computed tomography (CT) image, a magnetic resonance imaging (MRI) image, and
an X-ray image. In addition, these problems are not limited to a medical head mounted
display, and are problems common to head mounted displays for use in other ap
plications (for example, a head mounted display which is used in a construction site
and provides information which is not shown in an appearance of a building to a user).
[0007] For this reason, a head mounted display is desirable which enables a real image of an
object in a visual field direction and information which is not shown in an appearance
of the object to be visually recognized by a user together without the user moving his/
her visual line.
Solution to Problem
[0008] An advantage of some aspects of the invention is to solve at least a part of the
problems described above, and the invention can be implemented as the following
aspects.
[0009] (1) An aspect of the invention provides a head mounted display which allows a user
to visually recognize a virtual image and external scenery. The head mounted display
includes an image display unit that forms the virtual image which is visually
recognized by the user; and a superimposition processing unit that causes the image
display unit to form the virtual image based on superimposition information for super
imposing invisible information which is not shown in an appearance of an object on
the object included in the external scenery. According to the head mounted display, the
image display unit allows the user to visually recognize the virtual image based on the
superimposition information which is information for superimposing the invisible in
formation of the object on the object included in the external scenery. The head
mounted display is a head mounted display which allows the user to visually recognize
a virtual image and external scenery. For this reason, the user of the head mounted
display can visually recognize an object in the visual field direction, that is, a real
image of the object included in the external scenery, and the information
(superimposition information) which is not shown in the appearance of the object
together, without moving the visual line of the user.
[0010] (2) In the head mounted display, the superimposition processing unit may cause the
image display unit to repeatedly form the virtual image based on the superimposition
information corresponding to the object included in the external scenery every prede
termined time. According to the head mounted display, the superimposition processing
unit causes the image display unit to repeatedly form the virtual image based on the superimposition
information corresponding to the object included in the external scenery
every predetermined time. Thus, the user can visually recognize the virtual image
based on the superimposition information which tracks a motion of the head of the user
over time.
[001 1] (3) The head mounted display may further include a storage unit that stores the
invisible information; and an image acquisition unit that acquires an image of external
scenery in a visual field direction of the user when the user wears the head mounted
display. The superimposition processing unit may extract a feature of the object
included in the external scenery through image recognition on the image in the visual
field direction acquired by the image acquisition unit, and may correct the invisible in
formation of the object stored in the storage unit on the basis of the extracted feature so
as to generate the superimposition information. According to the head mounted
display, the superimposition processing unit can generate the superimposition in
formation by using the invisible information of the object stored in the storage unit. In
addition, the image acquired by the image acquisition unit is an image obtained by
capturing a real image of the object (in other words, an object included in the external
scenery) in the visual field direction of the user. The superimposition processing unit
extracts a feature of the object included in the external scenery from the external
scenery image acquired by the image acquisition unit, and generates superimposition
information on the basis of the extracted feature. For this reason, the superimposition
processing unit can generate the superimposition information which matches the
feature of the object (the object included in the external scenery) in the visual field
direction of the user.
[0012] (4) In the head mounted display, the invisible information may be an image in
dicating a structure of the object. According to the head mounted display, the invisible
information is an image indicating a structure of the object. For this reason, the user of
the head mounted display can visually recognize a real image of the object in the visual
field direction and the image indicating a structure of the object together, without
moving the visual line of the user.
[0013] (5) In the head mounted display, the feature may be an edge of the object. According
to the head mounted display, the superimposition processing unit can extract a contour
of the object included in the external scenery by detecting an edge of the object
through image recognition. As a result, the superimposition processing unit can
generate the superimposition information by correcting the invisible information of the
object on the basis of the extracted contour of the object.
[0014] (6) In the head mounted display, the feature may be a marker attached to the object.
According to the head mounted display, the superimposition processing unit can
extract a position of the object included in the external scenery by detecting markers
attached to the object through the image recognition. As a result, the superimposition
processing unit can generate the superimposition information by correcting the
invisible information of the object on the basis of the extracted position of the object.
[0015] (7) In the head mounted display, the correction may be performed by performing at
least one of enlargement, reduction, rotation, inversion, trimming, distortion, and noise
removal on an image indicating the invisible information. According to the head
mounted display, the superimposition processing unit performs correction by
performing at least one of enlargement, reduction, rotation, inversion, trimming,
distortion, and noise removal on an image indicating the invisible information of the
object stored in the storage unit, so as to generate the superimposition information.
[0016] (8) In the head mounted display, the correction is performed by dividing an image in
dicating the invisible information into a plurality of regions and performing at least one
of enlargement, reduction, trimming, distortion, and noise removal on each of the
plurality of regions. According to the head mounted display, the superimposition
processing unit performs correction by dividing an image indicating the invisible in
formation of the object stored in the storage unit into a plurality of regions and
performing at least one of enlargement, reduction, trimming, distortion, and noise
removal on each of the plurality of regions. As above, the superimposition processing
unit divides the image indicating the invisible information into a plurality of regions
and then corrects each region. Therefore, it is possible to improve accuracy of the
correction of the invisible information.
[0017] (9) In the head mounted display, the superimposition processing unit may measure a
distance between the user and each part of the object by using triangulation, and may
perform the division on the basis of a result of the measurement. According to the head
mounted display, the superimposition processing unit can measure a distance between
the user and each part of the object by using triangulation, and can divide the region of
the invisible information efficiently and in light of the actual situation on the basis of a
result of the measurement.
[0018] (10) In the head mounted display, the superimposition processing unit may change
accuracy of the correction on the basis of an information accepting characteristic in
dicating an extent of visual capability within a visual field of the user. According to the
head mounted display, the superimposition processing unit changes accuracy of the
correction of the invisible information on the basis of the information accepting char
acteristic within the visual field of the user. As a result, the superimposition processing
unit can efficiently correct the invisible information.
[0019] ( 11) The head mounted display may further include a visual line detection unit that
detects a visual line of the user. Here, the image display unit may form both the virtual
image corresponding to the right eye of the user and the virtual image corresponding to
the left eye of the user. In addition, the superimposition processing unit may further
obtain a steady gaze point distance which is a distance between the user and a steady
gaze point at which the user steadily gazes, from the visual line detected by the visual
line detection unit, and may generate the superimposition information for the right eye
and the superimposition information for the left eye to which a parallax corresponding
to the obtained steady gaze point distance is given. According to the head mounted
display, the superimposition processing unit generates the superimposition information
for the right eye and the superimposition information for the left eye to which the
parallax corresponding to a distance (steady gaze point distance) between the user and
the steady gaze point at which the user gazes steadily is given, and the image display
unit allows the user to visually recognize the virtual image based on this superim
position information. As a result, the user can visually recognize the superimposition
information at a position of the same distance as the steady gaze point.
[0020] (12) The head mounted display may further include parallax information that
correlates a plurality of distances between the user and the virtual image visually
recognized by the user with parallaxes which respectively correspond to the plurality
of distances. Here, the superimposition processing unit may give a parallax corre
sponding to the steady gaze point distance by using the parallax information.
According to the head mounted display, the superimposition processing unit can easily
generate the superimposition information for the right eye and the superimposition in
formation for the left eye by using the parallax information which is prepared in
advance.
[0021] (13) In the head mounted display, the parallax information may be provided in a
plurality of pieces, and, here, the superimposition processing unit may select one
parallax information piece from the plurality of parallax information pieces in ac
cordance with the user. According to the head mounted display, the superimposition
processing unit selects one parallax information piece from a plurality of parallax in
formation pieces in accordance with the user. For this reason, the superimposition
processing unit can select parallax information matching the characteristics (for
example, the dominant eye or the interocular distance) of the user, thereby reducing
visual discomfort which the user feels.
[0022] (14) The head mounted display may further include a position detection unit that
detects a position of the user relative to the object, and, here, the storage unit may store
a plurality of invisible information pieces regarding a single object. In addition, the su
perimposition processing unit may select one invisible information piece corre
sponding to the position detected by the position detection unit from the plurality of
invisible information pieces stored in the storage unit, and may correct the selected
invisible information. According to the head mounted display, the superimposition
processing unit can select one invisible information piece corresponding to the position
of the user relative to the object from the plurality of invisible information pieces
stored in the storage unit, and can correct the selected invisible information. For this
reason, the superimposition processing unit can generate the superimposition in
formation by using appropriate invisible information on the basis of the positional rela
tionship between the user and the object.
[0023] (15) In the head mounted display, the superimposition processing unit may further
generate enlargement information in which at least a part of the object included in the
external scenery is extracted and enlarged, from the image in the visual field direction
acquired by the image acquisition unit. In addition, the image display unit may allow
the user to visually recognize the virtual image based on the superimposition in
formation and the enlargement information. According to the head mounted display,
the user of the head mounted display can simultaneously visually recognize the en
largement information of at least a part of the object in the visual field direction, in
addition to the real image of the object (in other words, the object included in the
external scenery) in the visual field direction and the information (the superimposition
information) which is not shown in the appearance of the object.
[0024] (16) In the head mounted display, the superimposition processing unit may not
display the virtual image in response to a request from the user. According to the head
mounted display, the user of the head mounted display can cause the virtual image not
to be displayed at a user's own will. For this reason, it is possible to improve con
venience to the user of the head mounted display.
[0025] (17) The head mounted display may further include a procedure management unit
that generates procedure information which is information regarding content which is
to be performed by the user in a current procedure, and, here, the image display unit
may allow the user to visually recognize the virtual image based on the superim
position information and the procedure information. According to the head mounted
display, the image display unit allows the user to visually recognize the virtual image
based on the procedure information which indicates content to be performed by the
user in the current procedure. For this reason, it is possible to support work performed
by the user.
[0026] (18) In the head mounted display, the procedure information may include at least one
of an image of a scene in which a third party exemplarily performs the procedure and
an image indicating content to be performed by the user when the individual specific
procedure is performed. According to the head mounted display, it is possible to
visually show content to be performed by the user, and thus, it is possible to more in
telligibly support work performed by the user. In addition, if the image indicating
content to be performed by the user when the individual specific procedure is
performed is shown, it is possible to support work performed by the user more indi
vidually and specifically, and thus it is possible to improve accuracy and efficiency of
the work.
[0027] (19) In the head mounted display, the superimposition processing unit may further
generate auxiliary information which is additional information for assisting work of the
user. In addition, the image display unit may allow the user to visually recognize the
virtual image based on the superimposition information and the auxiliary information.
According to the head mounted display, the user visually recognizes the virtual image
based on the information for assisting work of the user. For this reason, it is possible to
improve efficiency of work using the head mounted display.
[0028] (20) In the head mounted display, the superimposition processing unit may further
perform a color conversion process for improving visibility of the generated superim
position information on the basis of the image in the visual field direction acquired by
the image acquisition unit. According to the head mounted display, the superim
position processing unit performs a color conversion process based on the image in the
visual field direction of the user on the superimposition information. For this reason, it
is possible to further improve visibility of the virtual image for the user of the head
mounted display.
[0029] (21) In the head mounted display, the superimposition processing unit may withdraw
the superimposition information which is displayed as the virtual image to a position
which does not hinder the visual field of the user, in response to a request from the
user. According to the head mounted display, the user of the head mounted display can
withdraw the superimposition information which is displayed as the virtual image at a
user's own will. For this reason, it is possible to improve convenience to the user of the
head mounted display.
[0030] (22) In the head mounted display, the superimposition processing unit may generate
the superimposition information corresponding to an overlap region in which a display
region of the virtual image overlaps with an acquisition region of the image acquisition
unit in the image acquired by the image acquisition unit. According to the head
mounted display, the superimposition processing unit generates the superimposition in
formation corresponding to an overlap region in which a display region of the virtual
image overlaps with an acquisition region of the image acquisition unit. For this
reason, it is possible to reduce the occurrence of "mismatch between an image which is
directly viewed by the user in his/her visual field and a virtual image which is allowed
to be visually recognized by the image display unit", which is a problem occurring in
the transmission type head mounted display. As a result, when the user visually
recognizes a virtual image based on the superimposition information, it is possible to
reduce the discomfort which the user feels.
[0031] (23) In the head mounted display, the procedure management unit may generate the
procedure information corresponding to an overlap region in which a display region of
the virtual image overlaps with an acquisition region of the image acquisition unit in
the image acquired by the image acquisition unit. According to the head mounted
display, the procedure management unit generates the procedure information corre
sponding to an overlap region in which a display region of the virtual image overlaps
with an acquisition region of the image acquisition unit. For this reason, it is possible
to reduce the occurrence of "mismatch between an image which is directly viewed by
the user in his/her visual field and a virtual image which is allowed to be visually
recognized by the image display unit", which is a problem occurring in the
transmission type head mounted display. As a result, when the user visually recognizes
a virtual image based on the procedure information, it is possible to reduce the
discomfort which the user feels.
[0032] All of the plurality of constituent elements in the respective aspects of the invention
described above are not essential, and some of the plurality of constituent elements
may be changed, deleted, exchanged with other new constituent elements, and partially
deleted from limited content thereof, as appropriate, in order to solve some or all of the
above-described problems or in order to achieve some or all of the effects described in
the present specification. In addition, in order to solve some or all of the abovedescribed
problems or in order to achieve some or all of the effects described in the
present specification, some or all of the technical features included in one aspect of the
invention described above may be combined with some or all of the technical features
included in another aspect of the invention described above, and as a result may be
treated as an independent aspect of the invention.
[0033] For example, one aspect of the invention may be realized as a device which includes
either one or both of the two constituent elements, the superimposition processing unit
and the image display unit. In other words, this device may or may not include the su
perimposition processing unit. Further, the device may or may not include the image
display unit. This device may be realized as, for example, a head mounted display, but
may be realized as devices other than the head mounted display. Some or all of the
above-described technical features of each aspect of the head mounted display are ap
plicable to the device.
[0034] In addition, the invention may be realized in various aspects. For example, the
invention may be realized in aspects such as a head mounted display and a control
method for the head mounted display, a medical work supporting device and a control
method for the medical work supporting device, a medical work supporting system, a
computer program for realizing functions of the methods, the devices, or the system,
and a recording medium recording the computer program thereon.
Brief Description of Drawings
[fig. 1]Fig. 1 is a diagram illustrating a schematic configuration of a head mounted
display according to an embodiment of the invention.
[fig.2]Fig. 2 is a functional block diagram illustrating a configuration of a head
mounted display.
[fig.3]Fig. 3 is a diagram illustrating a state in which image light is emitted by an
image light generation portion.
[fig.4]Fig. 4 is a diagram illustrating an example of a virtual image recognized by a
user.
[fig.5]Fig. 5 is a flowchart illustrating procedures of a superimposition information
display process according to a first embodiment.
[fig.6]Fig. 6 is a diagram illustrating an example of invisible information.
[fig.7]Fig. 7 is a diagram illustrating an example of appearance information.
[fig.8]Fig. 8 is a flowchart illustrating procedures of a superimposition information
generation process.
[fig.9]Fig. 9 is a diagram illustrating a feature (edge) of a target object extracted from
an external scenery image by using a method i.
[fig.lO]Fig. 10 is a diagram illustrating a feature (edge) of the target object extracted
from invisible information by using the method i.
[fig. 1l]Fig. 11 is a diagram illustrating an example of corrected invisible information.
[fig.l2A]Fig. 12A is a diagram illustrating another example of a virtual image
recognized by the user.
[fig.l2B]Fig. 12B is a diagram illustrating another examples of a virtual image
recognized by the user.
[fig.l3]Fig. 13 is a functional block diagram illustrating a configuration of a head
mounted display according to a second embodiment
[fig. 14] Fig. 14 is a diagram illustrating an example of a procedure table.
[fig.l5]Fig. 15 is a flowchart illustrating procedures of a superimposition information
display process according to the second embodiment.
[fig. 16] Fig. 16 is a diagram illustrating an example of procedure information.
[fig.l7]Fig. 17 is a diagram illustrating an example of superimposition information to
which the procedure information is added.
[fig.l8]Fig. 18 is a diagram illustrating an example of a virtual image visually
recognized by a user due to procedures el to e9.
[fig. 19] Fig. 19 is a diagram illustrating a schematic configuration of a head mounted
display according to a third embodiment.
[fig.20]Fig. 20 is a functional block diagram illustrating a configuration of a head
mounted display according to the third embodiment.
[fig.21]Fig. 2 1 is a diagram for explaining parallax information.
[fig.22]Fig. 22 is a flowchart illustrating procedures of a superimposition information
display process according to the third embodiment.
[fig.23]Fig. 23 is a diagram illustrating an example of invisible information.
[fig.24]Fig. 24 is a flowchart illustrating procedures of a first superimposition in
formation generation process.
[fig.25]Fig. 25 is a diagram for explaining step S404 of the first superimposition in
formation generation process.
[fig.26]Fig. 26 is a flowchart illustrating procedures of a second superimposition in
formation generation process.
[fig.27]Fig. 27 is a diagram for explaining step S504 of the second superimposition in
formation generation process.
[fig.28A]Fig. 28A is a diagram illustrating an exterior configuration of a head mounted
display in a modification example.
[fig.28B]Fig. 28B is a diagram illustrating an exterior configuration of a head mounted
displays in a modification example.
Description of Embodiments
[0036] A. First embodiment
A-l. Configuration of head mounted display
Fig. 1 is a diagram illustrating a schematic configuration of a head mounted display
(HMD) according to an embodiment of the invention. The head mounted display 100 is
a display mounted on the head. The head mounted display 100 according to the present
embodiment is an optical transmission type head mounted display which allows a user
to visually recognize a virtual image and also to directly visually recognize external
scenery. In addition, in the present embodiment, a type of head mounted display 100
which allows the user to visually recognize an image as a virtual image is exemplified.
[0037] The head mounted display 100 includes an image display section 20 which allows
the user to visually recognize a virtual image in a state of being mounted on the head
of the user, and a control section (a controller ) 10 which controls the image display
section 20.
[0038] The image display section 20 is a mounting body which is mounted on the head of
the user, and has a glasses shape in the present embodiment. The image display section
20 includes a right holding unit 21, a right display driving unit 22, a left holding unit
23, a left display driving unit 24, a right optical image display unit 26, a left optical
image display unit 28, and a camera 61. The right optical image display unit 26 and the
left optical image display unit 28 are respectively disposed so as to be located in front
of the right and left eyes of the user when the user wears the image display section 20.
One end of the right optical image display unit 26 and one end of the left optical image
display unit 28 are connected to each other at the position corresponding to the glabella
of the user when the user wears the image display section 20.
[0039] The right holding unit 2 1 is a member which is provided so as to extend over a
position corresponding to the temporal region of the user when the user wears the
image display section 20, from an end part ER which is the other end of the right
optical image display unit 26. Similarly, the left holding unit 23 is a member which is
provided so as to extend over a position corresponding to the temporal region of the
user when the user wears the image display section 20, from an end part EL which is
the other end of the left optical image display unit 28. The right holding unit 2 1 and the
left holding unit 23 hold the image display section 20 on the head of the user in the
same manner as temples of glasses.
[0040] The right display driving unit 22 is disposed inside the right holding unit 21, that is,
on a side opposing the head of the user when the user wears the image display section
20. In addition, the left display driving unit 24 is disposed inside the left holding unit
23. Further, hereinafter, the right holding unit 2 1 and the left holding unit 23 are also
collectively referred to as "holding units", the right display driving unit 22 and the left
display driving unit 24 are also collectively referred to as "display driving units", and
the right optical image display unit 26 and the left optical image display unit 28 are
also collectively referred to as "optical image display units".
[0041] The display driving units 22 and 24 include liquid crystal displays (hereinafter,
referred to as "LCDs") 241 and 242, projection optical systems 251 and 252, and the
like (refer to Fig. 2). Details of configurations of the display driving units will be
described. The optical image display units as optical members include light guiding
plates 261 and 262 (refer to Fig. 2) and dimming plates. The light guiding plates 261
and 262 are made of a light-transmitting resin material or the like, and guide image
light output from the display driving units 22 and 24 to the eyes of the user. The
dimming plates are thin plate-shaped optical elements, and are disposed so as to cover
a surface side of the image display section 20 (a side opposite the eye sides of the
user). The dimming plates protect the light guiding plates 261 and 262 so as to prevent
the light guiding plates 261 and 262 from being damaged, polluted, or the like. In
addition, light transmittance of the dimming plates is adjusted so as to adjust an
amount of external light entering the eyes of the user, thereby controlling an extent of
visually recognizing a virtual image. Further, the dimming plates may be omitted.
[0042] The camera 6 1 is disposed at the position corresponding to the glabella of the user
when the user wears the image display section 20. The camera 6 1 captures an image of
external scenery in the surface side direction of the image display section 20, that is, in
a visual field direction of the user when the user wears the head mounted display 100,
thereby obtaining an external scenery image. The camera 6 1 is a so-called visible light
camera, and an external scenery image acquired by the camera 6 1 is an image which
represents a shape of an object from visible light radiated from the object. The camera
6 1 in the present embodiment is a monocular camera, but may be stereo camera. The
camera 6 1 corresponds to an "image acquisition unit" in the appended claims, and the
external scenery image acquired by the camera 6 1 corresponds to an "image in a visual
field direction of a user" in the appended claims.
[0043] The image display section 20 further includes a connection unit 40 which connects
the image display section 20 to the control section 10. The connection unit 40 includes
a main body cord 48 connected to the control section 10, a right cord 42, a left cord 44,
and a connection member 46. The right cord 42 and the left cord 44 are two cords into
which the main body cord 48 branches out, and the connection member 46 is provided
at the branch point. The right cord 42 is inserted into a casing of the right holding unit
2 1 from an apex AP in the extending direction of the right holding unit 21, and is
connected to the right display driving unit 22. Similarly, the left cord 44 is inserted into
a casing of the left holding unit 23 from an apex AP in the extending direction of the
left holding unit 23, and is connected to the left display driving unit 24. The connection
member 46 has a jack for connection of an earphone plug 30. A right earphone 32 and
a left earphone 34 extend from the earphone plug 30.
[0044] The image display section 20 and the control section 10 transmit various signals via
the connection unit 40. An end part of the main body cord 48 on a side opposite the
connection member 46, and the control section 10 are respectively provided with
connectors (not illustrated) fitted into each other. The connector of the main body cord
48 and the connector of the control section 10 are fitted into or released from each
other, and thus the control section 10 is connected to or disconnected from the image
display section 20. For example, a metal cable or an optical fiber may be used as the
right cord 42, the left cord 44, and the main body cord 48.
[0045] The control section 10 is a device used to control the head mounted display 100. The
control section 10 includes a lighting unit 12, a touch pad 14, a cross key 16, and a
power switch 18. The lighting unit 12 indicates an operation state (for example, ON
and OFF of power, or the like) of the head mounted display 100 by using a light
emitting state thereof. For example, a light emitting diode (LED) may be used as the
lighting unit 12. The touch pad 14 detects a contact operation on an operation surface
of the touch pad 14 so as to output a signal based on detected content. Various touch
pads of a capacitance type, a pressure detection type, and an optical type may be
employed as the touch pad 14. The cross key 16 detects a pushing operation on keys
corresponding to vertical and horizontal directions so as to output a signal based on
detected content. The power switch 18 detects a sliding operation of the switch so as to
change a power supply state of the head mounted display 100.
[0046] Fig. 2 is a functional block diagram illustrating a configuration of the head mounted
display 100. The control section 10 includes an input information acquisition unit 110,
a storage unit 120, a power supply 130, a wireless communication unit 132, a CPU
140, an interface 180, transmission units (Tx) 5 1 and 52, and the above-described con
stituent elements are connected to each other via a bus (not illustrated).
[0047] The input information acquisition unit 110 acquires a signal based on an operation
input which is performed on, for example, the touch pad 14, the cross key 16, the
power switch 18, or the like. In addition, the input information acquisition unit 110
may acquire an operation input from the user by using various methods. For example,
not only an operation input using the touch pad 14 or the cross key 16 illustrated in
Fig. 2 but also an operation input using a foot switch (a switch operated by the leg of
the user) may be acquired. In addition, for example, a visual line detection unit such as
an infrared sensor may be provided in the image display section 20, then a visual line
of the user is detected, and an operation input based on a command correlated with a
motion of the visual line may be acquired. Further, for example, a gesture of the user
may be detected using the camera 61, and an operation input based on a command
correlated with the gesture may be acquired. When the gesture is detected, the fingertip
of the user, a ring attached to the hand of the user, or a medical instrument held by the
user may be used as a mark for detecting a motion. If an operation input can be
acquired using a foot switch or on the basis of a visual line of the user, the input in
formation acquisition unit 110 can acquire an operation input from the user even in
work (for example, surgery) in which it is difficult for the user to release the user's
hand.
[0048] The storage unit 120 is constituted by a ROM, a RAM, a DRAM, a hard disk, or the
like. The storage unit 120 includes an invisible information storage portion 122 and an
appearance information storage portion 124. The invisible information storage portion
122 is a storage region which preliminarily stores invisible information of an object
which is a target of a superimposition information display process. The "invisible in
formation" indicates information which is not shown in an appearance of an object.
The invisible information in the present embodiment is an image indicating a structure
of an object. In a case where an object is a living body, the invisible information is, for
example, a computed tomography (CT) image, a magnetic resonance imaging (MRI)
image, an X-ray image, a fluoroscopic image, an ultrasonic diagnostic image, an
endoscope image, a thermography image, or the like. In addition, in a case where an
object is an artifact, the invisible information is, for example, a design drawing, a
circuit diagram, an X-ray image, a fluoroscopic image, an infrared image, a thermography
image, or the like.
[0049] The appearance information storage portion 124 is a storage region which pre
liminarily stores appearance information of an object which is a target of the superimposition
information display process. The "appearance information" in the present em
bodiment is an image which indicates a state (appearance) of an object viewed from
the outside, and is a picture or the like obtained by a visible light camera.
[0050] The power supply 130 supplies power to the respective units of the head mounted
display 100. For example, a secondary battery may be used as the power supply 130.
The wireless communication unit 132 performs wireless communication with other ap
paratuses in accordance with a predetermined wireless communication standard such
as a wireless LAN or Bluetooth.
[0051] The CPU 140 reads and executes the computer programs stored in the storage unit
120 so as to function as an operating system (OS) 150, an image processing unit 160, a
sound processing unit 170, a display control unit 190, a superimposition processing
unit 142. The superimposition processing unit 142 is an application program which
executes a superimposition information display process. The superimposition in
formation display process is a process of superimposing information which is not
shown in an appearance of an object on a real image of the object in a visual field
direction of the user for display.
[0052] The image processing unit 160 generates signals on the basis of content (video)
which is input via the interface 180 or the wireless communication unit 132. In
addition, the image processing unit 160 supplies the generated signals to the image
display section 20 via the connection unit 40. The signals supplied to the image display
section 20 are different in cases of an analog format and a digital format. In a case of
the analog format, the image processing unit 160 generates and transmits a clock signal
PCLK, a vertical synchronization signal VSync, a horizontal synchronization signal
HSync, and image data Data. Specifically, the image processing unit 160 acquires an
image signal included in the content. The acquired image signal is an analog signal
which is generally formed by thirty frame images per second, for example, in a case of
a moving image. The image processing unit 160 separates synchronization signals such
as the vertical synchronization signal VSync and the horizontal synchronization signal
HSync from the acquired image signal, and generates the clock signal PCLK by using
a phase locked loop (PLL) circuit or the like (not illustrated) on the basis of a cycle of
the separated vertical synchronization signal VSync or horizontal synchronization
signal HSync. The image processing unit 160 converts an analog image signal from
which the synchronization signals are separated into a digital image signal by using an
A/D conversion circuit or the like. Next, the image processing unit 160 stores the
converted digital image signal in a DRAM of the storage unit 120 for each frame as
image data Data of RGB data. On the other hand, in a case of the digital format, the
image processing unit 160 generates and transmits the clock signal PCLK and the
image data Data. Specifically, in a case where the content has the digital format, since
the clock signal PCLK is output in synchronization with the image signal, the
generation of the vertical synchronization signal VSync, the horizontal synchronization
signal HSync, and the A/D conversion of the analog image signal are not necessary.
Further, the image processing unit 160 may perform, on the image data Data stored in
the storage unit 120, image processes including a resolution conversion process,
various color tone correction processes such as adjustment of luminance and color
saturation, a keystone correction process, and the like.
[0053] The image processing unit 160 transmits each of the generated clock signal PCLK,
vertical synchronization signal VSync and horizontal synchronization signal HSync,
and the image data Data stored in the DRAM of the storage unit 120, via the
transmission units 5 1 and 52. Here, the image data Data which is transmitted via the
transmission unit 5 1 is referred to as "right eye image data", and the image data Data
which is transmitted via the transmission unit 52 is referred to as "left eye image data".
The transmission units 5 1 and 52 function as a transceiver for serial transmission
between the control section 10 and the image display section 20.
[0054] The display control unit 190 generates control signals for control of the right display
driving unit 22 and the left display driving unit 24. Specifically, by using the control
signals, the display control unit 190 separately controls a right LCD control portion
2 11 to turn on and off driving of a right LCD 241, a right backlight control portion 201
to turn on and off driving of a right backlight 221, a left LCD control portion 212 to
turn on and off driving of a left LCD 242, and a left backlight control portion 202 to
turn on and off driving of a left backlight 222. Accordingly, the display control unit
190 controls each of the right display driving unit 22 and the left display driving unit
24 to generate and emit image light. For example, the display control unit 190 causes
both of the right display driving unit 22 and the left display driving unit 24 to generate
image light, causes either thereof to generate image light, or causes neither thereof to
generate image light. In addition, the display control unit 190 transmits control signals
for the right LCD control portion 211 and the left LCD control portion 212 thereto via
the transmission units 5 1 and 52, respectively. Further, the display control unit 190
transmits control signals for the right backlight control portion 201 and the left
backlight control portion 202 thereto, respectively.
[0055] The sound processing unit 170 acquires an audio signal included in the content so as
to amplify the acquired audio signal, and supplies the amplified audio signal to a
speaker (not illustrated) of the right earphone 32 connected to the connection member
46 and a speaker (not illustrated) of the left earphone 34 connected thereto. In addition,
for example, in a case where a Dolby (registered trademark) system is employed, the
audio signal is processed, and, thus, for example, different sounds of which frequencies
are changed are respectively output from the right earphone 32 and the left earphone
34.
[0056] The interface 180 is an interface which connects various external apparatuses OA
which are content supply sources to the control section 10. The external apparatuses
OA include, for example, a personal computer (PC), a mobile phone terminal, a
gaming terminal, and the like. For example, an USB interface, a micro-USB interface,
a memory card interface, and the like may be used as the interface 180.
[0057] The image display section 20 includes the right display driving unit 22, the left
display driving unit 24, the right light guiding plate 261 as the right optical image
display unit 26, the left light guiding plate 262 as the left optical image display unit 28,
and the camera 61.
[0058] The right display driving unit 22 includes a reception portion (Rx) 53, the right
backlight (BL) control portion 201 and the right backlight (BL) 221 which function as
a light source, the right LCD control portion 211 and the right LCD 241 which
function as a display element, and a right projection optical system 251. In addition,
the right backlight control portion 201, the right LCD control portion 211, the right
backlight 221, and the right LCD 241 are collectively referred to as an "image light
generation portion".
[0059] The reception portion 53 functions as a receiver for serial transmission between the
control section 10 and the image display section 20. The right backlight control portion
201 drives the right backlight 221 on the basis of an input control signal. The right
backlight 221 is a light emitting body such as, for example, an LED or an electrolu
minescent element (EL). The right LCD control portion 211 drives the right LCD 241
on the basis of the clock signal PCLK, the vertical synchronization signal VSync, the
horizontal synchronization signal HSync, and the right eye image data Datal, which
are input via the reception portion 53. The right LCD 241 is a transmissive liquid
crystal panel in which a plurality of pixels are disposed in a matrix.
[0060] The right projection optical system 251 is constituted by a collimator lens which
converts image light emitted from the right LCD 241 into parallel beams of light flux.
The right light guiding plate 261 as the right optical image display unit 26 reflects
image light output from the right projection optical system 251 along a predetermined
light path, so as to guide the image light to the right eye RE of the user. The optical
image display unit may employ any type as long as a virtual image is formed in front
of the eyes of the user by using image light, and, for example, a diffraction grating may
be used. In addition, a semi-transmissive reflective film may be used.
[0061] The left display driving unit 24 has the same configuration as the right display
driving unit 22. In other words, the left display driving unit 24 includes a reception
portion (Rx) 54, the left backlight (BL) control portion 202 and the left backlight (BL)
222 which function as a light source, the left LCD control portion 212 and the left
LCD 242 which function as a display element, and a left projection optical system 252.
Detailed description thereof will be omitted.
[0062] Fig. 3 is a diagram illustrating a state in which image light is emitted by the image
light generation portion. The right LCD 241 drives liquid crystal at respective pixel
positions disposed in a matrix so as to change a transmittance of light transmitted
through the right LCD 241, thereby modulating illumination light IL applied from the
right backlight 221 into effective image light PL representing an image. This is also the
same for the left side. In addition, as in Fig. 3, the backlight type is employed in the
present embodiment, but there may be a configuration in which image light is emitted
using a front light type or a reflective type.
[0063] Fig. 4 is a diagram illustrating an example of a virtual image recognized by the user.
Fig. 4 exemplifies a visual field VR of the user. As described above, the image light
which is guided to both eyes of the user of the head mounted display 100 forms an
image on the retinas of the user, and thus the user visually recognizes a virtual image
VI. In the example illustrated in Fig. 4, the virtual image VI is an X-ray image of a
patient's hand. In addition, the user visually recognizes an external scenery SC which is
transmitted through the right optical image display unit 26 and the left optical image
display unit 28. In the example illustrated in Fig. 4, the external scenery SC is a state in
which a cutting edge of a scalpel held by a surgeon is in contact with the second joint
of the little finger of the patient's right hand. As above, in the head mounted display
100, the user can visually recognize the external scenery SC on the rear side of the
virtual image VI not only in a part which does not overlap with the virtual image VI,
but also in a part which overlaps with the virtual image VI.
[0064] In addition, the head mounted display according to the present embodiment may
allow the user to visually recognize a three-dimensional virtual image by using a sideby-
side type. In a mode (hereinafter, also referred to as a "three-dimensional display
mode") in which the user is allowed to visually recognize a three-dimensional virtual
image, the LCD is equally divided into the left and right, the right half is used for
display for the right eye, and the left half is used for display for the left eye. In
addition, instead of the side-by-side type, other types such as, for example, a frame
packing type and a top-and-bottom type may be used to realize the three-dimensional
display mode.
[0065] A-2. Superimposition information display process
Fig. 5 is a flowchart illustrating procedures of a superimposition information display
process according to a first embodiment. Steps SI02 and SI04 of Fig. 5 indicate
procedures of a preprocess of the superimposition information display process. The
procedures from step S106 of Fig. 5 indicate procedures of the superimposition in
formation display process. The superimposition information display process is a
process of superimposing information which is no shown in an appearance of an object
on a real image of the object in a visual field direction of the user for display.
[0066] The superimposition information display process includes a method of extracting a
feature of an object (hereinafter, referred to as a "target object") which is a target of the
superimposition information display process through edge detection, and a method of
extracting a feature of the target object through maker detection (details thereof will be
described later). Of the two, in a case of employing the method of extracting a feature
of a target object through marker detection, markers are preliminarily attached to the
target object in steps S102, SI04 and S108. In addition, as the marker attached to a
target object, various types of markers may be used. For example, a tape, a seal, a
magic marker, a laser marker, a Magic Tape (registered trademark), and the like may
be used. Further, any number of markers attached to a target object may be set.
[0067] In step S102 of Fig. 5, the user stores invisible information of an object which is a
target of the superimposition information display process in the invisible information
storage portion 122. Specifically, the user performs image capturing on an object
which is a target of the superimposition information display process by using image
capturing apparatuses (a CT apparatus, an MRI apparatus, an X-ray apparatus, an
endoscope, a thermography apparatus, and the like), so as to acquire an image
(invisible information) indicating a structure of the target object. In addition, the user
stores the acquired image in the invisible information storage portion 122. Further, the
invisible information acquired in step S102 is displayed to overlap with a real image in
the external scenery SC which is viewed through the right optical image display unit
26 and the left optical image display unit 28 in the superimposition information display
process in and after step S106. For this reason, in step S102 of the superimposition in
formation display process, preferably, an image matching a direction of the target
object, expected in the superimposition information display process in and after step
S106, is acquired and stored in the invisible information storage portion 122. In
addition, in step S102, invisible information received from an external device may be
stored in the invisible information storage portion 122 instead of capturing an image of
the target object.
[0068] Fig. 6 is a diagram illustrating an example of invisible information. The invisible in
formation SI exemplified in Fig. 6 is an X-ray image of a person's right hand. In this
case, the target object is a "person's right hand". It can be seen from the invisible in
formation SI of Fig. 6 that the second joint of the little finger of the person's right hand
is deformed due to rheumatoid arthritis.
[0069] In step S104 of Fig. 5, the user stores appearance information of the object which is a
target of the superimposition information display process in the appearance in
formation storage portion 124. Specifically, the user performs image capturing on the
target object by using the camera 6 1 or a digital camera, thereby acquiring an image of
an appearance of the target object. In addition, the user stores the acquired image in the
appearance information storage portion 124.
[0070] Fig. 7 is a diagram illustrating an example of the appearance information. The ap
pearance information FI exemplified in Fig. 7 is a picture image of a person's right
hand. The target object is the same as in Fig. 6.
[0071] In step S106 of Fig. 5, the superimposition processing unit 142 determines whether
or not there is an instruction for starting the superimposition information display
process. Specifically, in a case where an instruction for starting the superimposition in
formation display process is received from the input information acquisition unit 110,
the superimposition processing unit 142 determines that there is a starting instruction
(step S106: YES), and in a case where the instruction is not received, the superim
position processing unit determines that there is no starting instruction (step S106:
NO). If there is no starting instruction (step S106: NO), the superimposition processing
unit 142 causes the process to transition to step S106 so as to wait for a starting in
struction to be received. On the other hand, if there is a starting instruction (step S106:
YES), the superimposition processing unit 142 generates superimposition information
through a superimposition information generation process described next (step S108).
[0072] A-2-1. Superimposition information generation process
Fig. 8 is a flowchart illustrating procedures of the superimposition information
generation process. The superimposition processing unit 142 acquires an external
scenery image (step S202). Specifically, the superimposition processing unit 142
activates the camera 6 1 so as to instruct the camera 6 1 to capture an image, and
acquires an external scenery image captured by the camera 61.
[0073] After the external scenery image is acquired, the superimposition processing unit 142
performs image recognition on the external scenery image so as to extract a feature of
the target object (step S204). Specifically, the superimposition processing unit 142
extracts a feature of the target object included in the external scenery image by using
image recognition methods such as the following methods i and ii. In addition, the
methods i and ii may be combined with each other.
[0074] Method i) The superimposition processing unit 142 detects an edge of the target
object. In this case, the detected edge corresponds to a "feature of the target object".
Method ii) The superimposition processing unit 142 detects markers attached to the
target object. In this case, the detected markers correspond to a "feature of the target
object".
[0075] Fig. 9 illustrates a feature (edge) of the target object extracted from the external
scenery image by using the method i.
[0076] In step S206 of Fig. 8, the superimposition processing unit 142 performs image
recognition on invisible information SI so as to extract a feature of the target object.
Specifically, the superimposition processing unit 142 reads invisible information SI
corresponding to the target object from the invisible information storage portion 122.
The superimposition processing unit 142 extracts a feature of the target object included
in the read invisible information SI by using the image recognition methods described
in the above methods i and ii. In addition, in order to improve correction accuracy in
step S208, image recognition methods used in steps S204 and S206 are preferably
unified into one.
[0077] Fig. 10 illustrates a feature (edge) of the target object extracted from the invisible in
formation SI by using the method i.
[0078] In step S208 of Fig. 8, the superimposition processing unit 142 corrects the invisible
information SI so that the feature of the external scenery image extracted in step S204
and the feature of the invisible information SI extracted in step S206 match each other.
Specifically, in a case where the method i is used in steps S204 and S206, the superim
position processing unit 142 performs at least one processing of enlargement,
reduction, rotation, inversion, trimming, distortion, and removal of noise, on the image
indicating the invisible information SI so that the feature (that is, the edge detected in
step S204) of the external scenery image and the feature (that is, the edge detected in
step S206) of the invisible information SI match each other in contours or charac
teristic sites thereof. Here, the characteristic sites are the joints, tips of the limbs, the
blood vessels, the bones, and the like in a case where the target object is a living body.
The characteristic sites are protrusions, corners, and the like in a case where the target
object is an artifact. On the other hand, in a case where the method ii is used in steps
S204 and S206, the superimposition processing unit 142 performs at least one
processing of enlargement, reduction, rotation, inversion, trimming, distortion, and
removal of noise, on the image indicating the invisible information SI so that positions
of the feature (that is, the marker detected in step S204) of the external scenery image
and the feature (that is, the marker detected in step S206) of the invisible information
SI match each other.
[0079] In the examples of Figs. 9 and 10, the superimposition processing unit 142 performs
correction on the invisible information SI so that the contours and the characteristic
sites (the joints, the fingertips, the blood vessels, and the bones) of the hand match
each other in the edge (Fig. 9) detected in step S204 and the edge (Fig. 10) detected in
step S206.
[0080] Fig. 11 is a diagram illustrating an example of the corrected invisible information SI.
In corrected superimposition information CP1, the invisible information SI (Fig. 6) is
trimmed so as to match a range of the target object (that is, the patient's right hand)
included in the external scenery image. Similarly, in the corrected superimposition in
formation CP1, the invisible information SI (Fig. 6) is reduced so as to match the size
of the target object included in the external scenery image, and the invisible in
formation SI (Fig. 6) is rotated and moved so as to match the position of the target
object included in the external scenery image. In addition, the invisible information SI
corrected in step S208 corresponds to "superimposition information" in the appended
claims.
[0081] As above, in the superimposition information generation process, the superim
position processing unit 142 can generate the superimposition information by using the
invisible information SI of the object stored in the invisible information storage portion
122. The external scenery image acquired by the camera 6 1 (image acquisition unit) in
step S202 is an image obtained by capturing a real image of the target object (in other
words, an object included in the external scenery) in the visual field direction of the
user. The superimposition processing unit 142 extracts an edge or markers (a feature of
the target object) of the target object included in the external scenery from the external
scenery image acquired by the camera 61, and generates superimposition information
on the basis of the extracted feature. For this reason, the superimposition processing
unit 142 can generate the superimposition information which matches the feature of the
target object (the object included in the external scenery) in the visual field direction of
the user.
[0082] In addition, according to the method i, the superimposition processing unit 142 can
extract a contour of the target object included in the external scenery by detecting an
edge of the target object through the image recognition. As a result, the superim
position processing unit 142 can generate the superimposition information CP1 by
correcting the invisible information SI of the target object on the basis of the extracted
contour of the target object.
[0083] Further, according to the method ii, the superimposition processing unit 142 can
extract a position of the target object included in the external scenery by detecting
markers attached to the target object through the image recognition. As a result, the su
perimposition processing unit 142 can generate the superimposition information CP1
by correcting the invisible information SI of the target object on the basis of the
extracted position of the target object.
[0084] After the superimposition information generation process is completed, the superim
position processing unit 142 displays the superimposition information CP1 in step
SI 10 of Fig. 5. Specifically, the superimposition processing unit 142 transmits the su
perimposition information CP1 generated due to the superimposition information
generation process to the image processing unit 160. The image processing unit 160
having received the superimposition information CPl performs the display process
described in Fig. 2 on the superimposition information CPl. As a result, as illustrated
in Fig. 4, the superimposition information CPl is displayed as the virtual image VI in
the visual field VR of the user of the head mounted display 100. The head mounted
display 100 according to the present embodiment is an optical transmission type head
mounted display which allows the user to visually recognize a virtual image and also to
visually recognize external scenery. For this reason, the user of the head mounted
display 100 can visually recognize the real image (a state in which the cutting edge of
the scalpel held by the surgeon is in contact with the second joint of the little finger of
the patient's right hand) in the external scenery SC which is viewed through the right
optical image display unit 26 and the left optical image display unit 28 and the superimposition
information CPl which is viewed as the virtual image VI, in a superim
position state.
[0085] In step SI 12 of Fig. 5, the superimposition processing unit 142 determines whether
or not there is an instruction for changing display, and determines what is content of a
changing instruction. In a case where there is no display changing instruction, the su
perimposition processing unit 142 causes the process to proceed to step S108 after
waiting for a predetermined time (not illustrated). Accordingly, since capturing of an
external scenery image and generation and display of the superimposition information
CPl are repeatedly performed for each predetermined time, it is possible to display the
superimposition information CPl which tracks a motion of the head of the user over
time, as the virtual image VI. In addition, the predetermined time may be arbitrarily
set.
[0086] When an "image changing" instruction for changing the display of the superim
position information CPl which is currently displayed is received from the input in
formation acquisition unit 110 (step SI 12: image changing), the superimposition
processing unit 142 causes the process to proceed to step SI 14. The superimposition
processing unit 142 generates another superimposition information piece (step SI 14).
Specifically, the superimposition processing unit 142 performs again the superim
position information generation process described in Fig. 8 so as to generate another
superimposition information piece. At this time, the superimposition processing unit
142 uses the "appearance information FI stored in the appearance information storage
portion 124" as a processing target instead of the "invisible information SI stored in the
invisible information storage portion 122" in the description of Fig. 8. For this reason,
another superimposition information piece is information indicating an appearance of
the target object. After generating another superimposition information piece, the su
perimposition processing unit 142 displays another superimposition information piece
(step SI 16). Specifically, the superimposition processing unit 142 transmits another su
perimposition information piece to the image processing unit 160. The image
processing unit 160 having received another superimposition information piece
performs the display process described in Fig. 2 on another superimposition in
formation piece. As a result, instead of the superimposition information CPl, another
superimposition information piece is displayed as the virtual image VI in the visual
field VR of the user of the head mounted display 100. Subsequently, the superim
position processing unit 142 causes the process to proceed to step SI 12.
[0087] As described above, according to steps SI 14 and SI 16, the user of the head mounted
display 100 can change the display of superimposition information generated on the
basis of the invisible information SI indicating a structure of the target object and the
display of another superimposition information piece generated on the basis of the ap
pearance information FI indicating an appearance of the target object, at user's own
will. For this reason, it is possible to improve convenience to the user of the head
mounted display 100.
[0088] In Fig. 5, when a "non-display" instruction for not displaying the virtual image which
is currently displayed is received from the input information acquisition unit 110 (step
SI 12: non-display), the superimposition processing unit 142 causes the process to
proceed to step SI 18. The superimposition processing unit 142 causes the virtual
image VI (the superimposition information CPl or another superimposition in
formation piece) which is currently displayed not to be displayed (step SI 18).
Specifically, the superimposition processing unit 142 stops transmission of the superimposition
information CPl or another superimposition information piece to the image
processing unit 160. Accordingly, image data to be displayed is removed, and thus the
virtual image VI is not displayed. In addition, instead of stopping of transmission of
the superimposition information CPl or another superimposition information piece, the
superimposition processing unit 142 may transmit a request for not displaying an
image to the display control unit 190. In this case, the display control unit 190 may
transmit a control signal for causing the LCD control portions to turn off driving of the
LCDs or a control signal for causing the backlight control portions to turn off driving
of the backlights, thereby not displaying the virtual image VI.
[0089] As above, according to step SI 18, the user of the head mounted display 100 can
cause the virtual image VI which is currently displayed not to be displayed at user's
own will. For example, in a case where the user wants to keep an eye on a real image
which is viewed through the optical image display units, the user can erase the virtual
image VI on purpose. For this reason, it is possible to improve convenience to the user
of the head mounted display 100.
[0090] In Fig. 5, when an "enlargement" instruction for enlarging and displaying the real
image in the visual field direction is received from the input information acquisition
unit 110 (step SI 12: enlargement), the superimposition processing unit 142 causes the
process to proceed to step S120. The superimposition processing unit 142 acquires an
external scenery image (step S120). Specifically, the superimposition processing unit
142 activates the camera 6 1 so as to instruct the camera 6 1 to capture an image, and
acquires an external scenery image captured by the camera 61. After the external
scenery image is acquired, the superimposition processing unit 142 performs image
recognition on the external scenery image acquired in step S120, and trims a region in
a predetermined range centering on a point which is designated along with the en
largement instruction so as to enlarge the trimmed image (step S122). Next, the superimposition
processing unit 142 generates enlargement information in which the
enlarged image is disposed in an end part, and black dummy data is disposed in other
parts. After generating the enlargement information, the superimposition processing
unit 142 displays the superimposition information CP1 and the enlargement in
formation. Specifically, the superimposition processing unit 142 transmits the superim
position information CP1 and the enlargement information to the image processing
unit 160. The image processing unit 160 having received the superimposition in
formation CP1 and the enlargement information generates a composite image which
has the superimposition information CP1 and the enlargement information as r e
spective layers, and performs the display process described in Fig. 2 on the generated
composite image.
[0091] Figs. 12A and 12B are diagrams illustrating other examples of a virtual image
recognized by the user. Fig. 12A illustrates a virtual image recognized by the user due
to step S122 of Fig. 5. As above, as a result of executing steps S120 and S122, the su
perimposition information CP1 and the enlargement information CP2 are displayed as
the virtual image VI in the visual field VR of the user of the head mounted display 100.
As described above, according to steps S120 and S122, the user of the head mounted
display 100 can simultaneously visually recognize the enlargement information CP2 of
the real image centering on the designated point, that is, the enlargement information
CP2 of at least a part of the object in the visual field direction, in addition to the real
image of the target object (in other words, the object included in the external scenery)
in the visual field direction and the information (the superimposition information CP1)
which is not shown in the appearance of the target object, at user's own will. As a
result, it is possible to improve convenience to the user of the head mounted display
100.
[0092] As described above, in the superimposition information display process according to
the first embodiment, the image display section 20 allows the user to visually
recognize a virtual image VI (video) which is based on the superimposition information
CP1 for superimposing the invisible information SI (in the above-described
example, the X-ray image of the person's right hand) of the target object on the target
object (in the above-described example, the person's right hand) included in the
external scenery SC. The head mounted display 100 according to the present em
bodiment is a head mounted display which allows the virtual image VI (video) and the
external scenery SC to be visually recognized together. For this reason, the user of the
head mounted display 100 can visually recognize the real image of the target object (in
other words, the target object included in the external scenery) in the visual field
direction and the information (the superimposition information CP1) which is not
shown in the appearance of the target object together, without moving the visual line of
the user.
[0093] In addition, in the first embodiment, the invisible information SI stored in the
invisible information storage portion 122 is an image indicating a structure of the
target object. For this reason, the user of the head mounted display 100 can visually
recognize the real image of the target object in the visual field direction and the image
indicating the structure of the target object together without moving his/her visual line.
For this reason, the head mounted display 100 according to the present embodiment is
very useful for minimally invasive surgery. In addition, the minimally invasive surgery
is a surgical method of keeping an incision site caused by a scalpel to a minimum in
order to minimize a patient's pain or a scar from surgery due to the surgery.
[0094] A-3. Additional processes of superimposition information display process
In addition, the following additional processes may be further performed in the su
perimposition information display process (Fig. 5). The additional processes may be
added singly or added in combination.
[0095] A-3-1. Additional process 1
In the additional process 1, the superimposition processing unit 142 further displays
auxiliary information CP3 as a virtual image VI in addition to the superimposition in
formation CP1. The auxiliary information CP3 may be any information as long as the
information is additional information for assisting work of the user of the head
mounted display 100. The auxiliary information CP3 may be, for example, the
following information in the medical head mounted display 100.
[0096] * Information which displays a measurement value of a medical instrument (an elec
trocardiogram, a heart rate meter, a sphygmomanometer, a pulse oximeter, a blood
glucose meter, or the like) attached to a patient
* Information which displays a caution or a warning based on a measurement value
of a medical instrument attached to a patient
[0097] Fig. 12B illustrates an example in which the auxiliary information CP3, which
displays a heart rate (HR), a blood pressure (SYS/DIA), and arterial oxygen saturation
(Sp02), is displayed as the virtual image VI. In the example of Fig. 12B, the auxiliary
information CP3 is represented by characters. However, as the auxiliary information
CP3, a picture, a pattern, a figure, and a combination thereof may be employed in
addition to characters. In addition, the auxiliary information CP3 is preferably
disposed, for example, in an end part so as not to overlap with the superimposition in
formation CP1. A method of displaying the superimposition information CP1 and the
auxiliary information CP3 together is the same as in step S122.
[0098] Further, Fig. 12B illustrates an example of the auxiliary information CP3 in the
medical head mounted display 100. However, the auxiliary information CP3 can be
displayed as the virtual image VI even in the head mounted display 100 used for other
applications. For example, the auxiliary information CP3 in the head mounted display
100 used in a construction site may be information which displays information of a
building (an age of the building, information regarding the ground on which the
building is currently constructed, information regarding a construction method, in
formation regarding materials used in the construction, and the like). In addition,
various pieces of auxiliary information may be employed on the basis of purpose of
use of the head mounted display 100.
[0099] As above, according to the additional process 1, the user of the head mounted display
100 can visually recognize the auxiliary information CP3 which is information for
assisting work as the virtual image VI (video). For this reason, it is possible to improve
efficiency of work using the head mounted display 100.
[0100] A-3-2. Additional process 2
In the additional process 2, a color conversion process is performed on at least one of
the superimposition information CP1 generated in step S108, another superimposition
information piece generated in step SI 14, and the enlargement information CP2
generated in step S122. The color conversion process is preferably a color conversion
process for improving visibility of each piece of information (the superimposition in
formation CP1, another superimposition information piece, and the enlargement in
formation CP2) generated by the superimposition processing unit 142, on the basis of
brightness, contrast, color saturation, and a color of the external scenery image
captured in step S202. For example, the color conversion process may be performed so
that a color of each piece of information generated by the superimposition processing
unit 142 is converted into a color complementary to a color of the external scenery
image captured in step S202. In addition, the color conversion process may be
performed by the superimposition processing unit 142 before each piece of information
is transmitted to the image processing unit 160, and may be performed by the image
processing unit 160 having received each piece of information.
[0101] As described above, according to the additional process 2, it is possible to further
improve visibility of the virtual image VI (video) for the user of the head mounted
display 100.
[0102] A-3-3. Additional process 3
In the additional process 3, the superimposition processing unit 142 withdraws the
information (the superimposition information CPl, another superimposition in
formation piece, the enlargement information CP2, and the auxiliary information CP3)
which is displayed as the virtual image VI, to a position which does not hinder the
visual field of the user. A case where the virtual image VI which is currently displayed
is the superimposition information CPl will be described as an example. The superim
position processing unit 142 reduces the superimposition information CPl. Then, the
superimposition processing unit 142 generates reduction information in which the
reduced image is disposed in an end part and black dummy data is disposed in other
parts. After generating the reduction information, the superimposition processing unit
142 displays the reduction information. Specifically, the superimposition processing
unit 142 transmits the reduction information to the image processing unit 160. The
image processing unit 160 having received the reduction information performs the
display process described in Fig. 2 on the reduction information. As a result, the
reduction information in which the superimposition information CPl is reduced and
disposed in the end part is displayed as the virtual image VI instead of the superim
position information CPl in the visual field VR of the user of the head mounted
display 100. In addition, the additional process 3 may be performed instead of step
SI 18 of the superimposition information display process (Fig. 5).
[0103] As described above, according to the additional process 3, the user of the head
mounted display 100 can withdraw the information (the superimposition information
CPl, another superimposition information piece, the enlargement information CP2,
and the auxiliary information CP3) which is displayed as the virtual image VI (video),
to a position which does not hinder the visual field at user's own will. For example, in a
case where the user wants to keep an eye on the real image which is viewed through
the optical image display units, the user may withdraw the information which is
displayed as the virtual image VI (video) on purpose. For this reason, it is possible to
improve convenience to the user of the head mounted display 100.
[0104] Further, according to the additional process 3, since a variety of reduced information
(the superimposition information CPl, another superimposition information piece, the
enlargement information CP2, and the auxiliary information CP3) is disposed in the
end part, it is possible to allow the user of the head mounted display 100 to pay
attention to the presence of a variety of information. In other words, a variety of
reduced information may be used in the same manner as an icon. For example, when
the reduced superimposition information CPl is selected by the user, the superimposition
information CPl may be returned to original position and size thereof. In this
way, it is possible to further improve convenience to the user of the head mounted
display 100.
[0105] A-3-4. Additional process 4
In the additional process 4, the superimposition processing unit 142 realizes a socalled
stabilizer function of suppressing shaking of the virtual image VI due to slight
shifting of the head of the user. In a case where there is no display changing instruction
in step SI 12 of the superimposition information display process (Fig. 5), the superim
position processing unit 142 causes the process to proceed to step S108 after waiting
for a predetermined time. Accordingly, since capturing of an external scenery image
and generation and display of the superimposition information CPl are repeatedly
performed for each predetermined time, it is possible to display the superimposition in
formation CPl which tracks a motion of the head of the user over time, as the virtual
image VI. However, if the superimposition information CPl is often changed by
tracking a slight motion of the head or breathing of the user, on the contrary, there is a
concern that eyestrain of the user is caused, or there is a concern that concentration of
the user is hindered. Therefore, the superimposition processing unit 142 may perform
processes indicated in procedures al and a2 between steps S202 and S204 of the superimposition
information generation process (Fig. 8).
[0106] Procedure al) A change amount of an RGB histogram is obtained between an
external scenery image captured in step S202 and an external scenery image captured
in step S202 of the previously performed superimposition information generation
process.
Procedure a2) In a case where the change amount obtained in the procedure al is
larger than a predetermined threshold value, the processes in and after step S204 are
successively performed. In a case where the change amount obtained in the procedure
a l is equal to or smaller than the predetermined threshold value, the superimposition
information CPl generated due to the previously performed superimposition in
formation generation process is set as a processing result of the currently performed
superimposition information generation process.
[0107] As described above, according to the additional process 4, in a case where a change
amount between an external scenery image (external scenery in the visual field
direction of the user) in the previously performed superimposition information
generation process and an external scenery image in the currently performed superim
position information generation process is small, that is, a motion of the head of the
user is slight, the superimposition processing unit 142 does not generate new superim
position information CPl, and sets the superimposition information CPl generated due
to the previously performed superimposition information generation process as a
processing result of the currently performed superimposition information generation
process. As a result, it is possible to prevent the superimposition information CPl from
being often changed by tracking a slight motion of the head or breathing of the user of
the head mounted display 100. Therefore, it is possible to prevent eyestrain or
reduction in concentration of the user. In addition, there may be a configuration in
which the user can change turning-on and off of the stabilizer function.
[0108] A-3-5. Additional process 5
In the additional process 5, the superimposition processing unit 142 changes a size at
which the enlargement information CP2 and the auxiliary information CP3 are
displayed as the virtual image VI in accordance with a distance between the optical
image display units 26 and 28 of the image display section 20 and an object present in
the visual field direction of the user. Specifically, the superimposition processing unit
142 may perform processes indicated in procedures b l to b3 in the superimposition in
formation display process (Fig. 5). In addition, in a case where the additional process 5
is performed, a range finding sensor is further provided in the head mounted display
100. The range finding sensor is a sensor which acquires a distance between an object
present in the visual field direction of the user and the image display section 20 by
using reflected light, and may be disposed, for example, around the camera 6 1 of Fig.
1.
[0109] Procedure bl) The superimposition processing unit 142 periodically acquires a mea
surement value obtained by the range finding sensor.
Procedure b2) The superimposition processing unit 142 changes an enlargement ratio
in accordance with the acquired measurement value of the range finding sensor, during
the enlargement process in step S122 of the superimposition information display
process (Fig. 5).
Procedure b3) The superimposition processing unit 142 changes a size of the
auxiliary information CP3 in the additional process 1 in accordance with the mea
surement value of the range finding sensor.
[0110] As described above, according to the additional process 5, the superimposition
processing unit 142 changes a size at which the enlargement information CP2 and the
auxiliary information CP3 are displayed as the virtual image VI (video) in accordance
with a distance between an object present in the visual field direction of the user and
the image display section 20. For this reason, it is possible to improve convenience to
the user of the head mounted display 100.
[0111] A-3-6. Additional process 6
In the additional process 6, the wireless communication unit 132 transmits external
scenery images which are acquired in the middle of the above-described process to an
external device. The external scenery images include, specifically, the external scenery
image in step S202 of the superimposition information generation process (Fig. 8) and
the external scenery image acquired in step SI20 of the superimposition information
display process (Fig. 5). The external device is, for example, another head mounted
display 100, a server, a monitor, or the like. The wireless communication unit 132
transmits these external scenery images to the external device through wireless
connection. The external device which has received the external scenery images may
utilize the received external scenery images in various aspects. For example, the
external scenery images may be displayed on another head mounted display 100 or
monitor. In addition, the external scenery images may be stored in a server so as to be
preserved as a log of surgery (work).
[0112] In the aspect in which the external scenery images are displayed on another head
mounted display 100 or monitor, the external device may acquire advice given by a
surgeon (worker) who views the displayed images. In this case, the external device
transmits the acquired advice to the head mounted display 100. The wireless commu
nication unit 132 of the head mounted display 100, having received the advice,
transfers the received advice. Specifically, in a case where the received advice is sound
information, the wireless communication unit 132 transfers the sound information to
the sound processing unit 170. In a case where the received advice is character in
formation or image information, the wireless communication unit 132 transfers the
character information or the image information to the superimposition processing unit
142. The sound processing unit 170 having received the sound information amplifies
an audio signal included in the sound information so as to supply the amplified audio
signal to speakers built in the right earphone 32 and the left earphone 34. Accordingly,
the user of the head mounted display 100 can hear the advice with sounds. On the other
hand, the superimposition processing unit 142 having received the character in
formation or the image information further displays the received character information
or image information as the virtual image VI in addition to the superimposition in
formation CP1. A method of displaying the character information or the image in
formation is the same as in the auxiliary information CP3 described in the additional
process 1. Accordingly, the user of the head mounted display 100 can visually
recognize the advice indicated by text or an image as the virtual image VI.
[01 13] As described above, according to the additional process 6, the external scenery image
acquired in the middle of the process can also be viewed and thus can be preserved in
the external device. In addition, according to the additional process 6, the user of the
head mounted display 100 can obtain advice from a person who has viewed the
external scenery image acquired in the middle of the process. For this reason, it is
possible to improve efficiency and accuracy of work using the head mounted display
100.
[01 14] A-3-7. Additional process 7
In the additional process 7, the superimposition processing unit 142 performs cal
ibration of a display region of the virtual image VI and an image capturing region of
the camera 61. The superimposition processing unit 142 performs processes indicated
in the following procedures c l to c7 prior to the superimposition information display
process (Fig. 5).
[0115] Procedure cl) Image data of a guide image (hereinafter, referred to as "guide image
data") for calibration is stored in the storage unit 120 in advance. The guide image is
an image for showing an end part of a display region of a virtual image to the user of
the head mounted display 100. For example, the guide image may be a rectangular
image which has the same aspect ratio as the right LCD 241 and the left LCD 242, has
a totally white background, and has circular guide marks in four corners thereof and a
part located at an intersection of diagonal lines.
Procedure c2) The superimposition processing unit 142 reads the guide image data
stored in the storage unit 120, and transmits the guide image data to the image
processing unit 160. The process described in Fig. 2 is performed in the image
processing unit 160 having received the guide image data. As a result, the guide image
is displayed as the virtual image VI in the visual field VR of the user of the head
mounted display 100.
[01 16] Procedure c3) The superimposition processing unit 142 guides the user so as to se
quentially point at the guide marks of the guide image displayed as the virtual image
VI. As a method of guiding the user, a message using a dialog box may be used, the
guiding may be performed using sounds. The guiding using sounds is preferable in that
the guiding can be performed without obstructing the guide image whose display is in
progress.
Procedure c4) The superimposition processing unit 142 causes the camera 6 1 to
capture an external scenery image.
Procedure c5) The superimposition processing unit 142 performs image recognition
on the external scenery image captured by the camera 6 1 so as to specify a position of
a human fingertip. The superimposition processing unit 142 acquires coordinates of the
fingertip position in relation to each guide mark so as to preserve the coordinates
thereof in the storage unit 120. For example, the coordinates of the fingertip position
may be defined as a shift amount in the X direction and Y direction when the
uppermost left part of the external scenery image is set to (0,0).
[01 17] Procedure c6) The superimposition processing unit 142 repeatedly performs a series
of processes including acquisition of an external scenery image, acquisition of co
ordinates of a fingertip position, and preservation of the coordinates, in relation to all
the guide marks. Subsequently, the superimposition processing unit 142 stops the
display of the guide image. In this way, the superimposition processing unit 142 pre
liminarily acquires a region where an image capturing region of the camera 6 1
overlaps with a display region of the virtual image VI. In addition, the region where
the image capturing region (acquisition region) of the camera 6 1 overlaps with the
display region of the virtual image VI is also referred to as an "overlap region".
Procedure c7) The superimposition processing unit 142 extracts a feature of an object
included in an image of the overlap region from the external scenery image captured
by the camera 6 1 in step S204 of the superimposition information generation process
(Fig. 8). In addition, in step S208, superimposition information CPl corresponding to
the overlap region is generated.
[0118] As described above, according to the additional process 7, the superimposition
processing unit 142 can generate the superimposition information CPl in relation to
the overlap region where the display region of the virtual image VI (video) overlaps
with the acquisition region of the camera 6 1 (image acquisition unit). For this reason, it
is possible to reduce the occurrence of "mismatch between an image which is directly
viewed by the user in his/her visual field and a virtual image VI (video) which is
allowed to be visually recognized by the image display section 20", which is a problem
occurring in the transmissive head mounted display 100. As a result, when the user
visually recognizes a virtual image based on the superimposition information CPl, it is
possible to reduce the discomfort which the user feels.
[0119] B. Second embodiment
In the second embodiment of the invention, a description will be made of a con
figuration in which a head mounted display can further support work performed by a
user. Hereinafter, only parts having configurations and operations different from the
first embodiment will be described. In addition, in the drawings, the same constituent
elements as in the first embodiment are given the same reference numeral as in the first
embodiment, and detailed description thereof will be omitted.
[0120] B-l. Configuration of head mounted display
Fig. 13 is a functional block diagram illustrating a configuration of a head mounted
display 100a according to the second embodiment. A difference from the first em
bodiment illustrated in Fig. 2 is that a control section 10a is provided instead of the
control section 10. The control section 10a includes a storage unit 120a instead of the
storage unit 120, and a CPU 140a instead of the CPU 140.
[0121] The storage unit 120a includes a procedure table 126 in addition to the respective
storage portions illustrated in Fig. 2. The CPU 140a includes a superimposition
processing unit 142a instead of the superimposition processing unit 142 illustrated in
Fig. 2, and further includes a procedure management unit 144 in addition to the re
spective processing units illustrated in Fig. 2. The superimposition processing unit
142a and the procedure management unit 144 perform a superimposition information
display process in the second embodiment in cooperation with each other.
[0122] Fig. 14 is a diagram illustrating an example of the procedure table 126. The
procedure table 126 is a table used in the superimposition information display process
in the second embodiment. The procedure table 126 is stored in the storage unit 120a
in advance. The procedure table 126 includes fields of a work number, a procedure
number, a procedure name, instruction content, and an image.
[0123] An identifier for uniquely identifying work is stored in the work number. For
example, in the medical head mounted display 100, an identifier indicating a technique
of surgery is stored in the work number. The identifier is formed by a character string,
and may be formed by, for example, a combination of alphanumeric characters.
[0124] Various information pieces regarding a plurality of procedures included in the work
specified by the work number are stored in the respective fields of the procedure
number and thereafter. An identifier for uniquely identifying a procedure is stored in
the procedure number. The identifier is formed by a character string, and may be
formed by, for example, a combination of alphanumeric characters. In the example of
Fig. 14, it is assumed that a plurality of procedures in the work progress in an
ascending order of procedure numbers. The name of a corresponding procedure is
stored in the procedure name. A character string indicating content to be performed by
the user in the corresponding procedure is stored in the instruction content. In addition,
in the example of Fig. 14, the instruction content is represented in a character string
with a guide sentence form for the user. An image indicating content to be performed
by the user is stored in the image field. The image is a picture or the like of a scene in
which, for example, a third party exemplarily performs a corresponding procedure.
Further, in addition to the picture, the image may be drawings (an explanatory
diagram, an assembly diagram, a layout drawing, a circuit diagram, and the like), and
may be combinations thereof.
[0125] In the example of Fig. 14, it can be seen that the procedure table 126 is a table which
stores procedures of work (that is, surgery) performed by a surgeon in the medical
head mounted display 100. In addition, it can be seen that the work identified by the
work number 1 is artificial finger joint replacement surgery. Further, it can be seen that
the content to be performed by the user in the procedure identified by the procedure
number 1 is "preparing for articles necessary in the surgery", and "what necessary
articles are". Furthermore, an image indicating content to be performed by the user is
not correlated with the procedure identified by the procedure number 1.
[0126] B-2. Superimposition information display process
Fig. 15 is a flowchart illustrating procedures of the superimposition information
display process according to the second embodiment. A difference from the first embodiment
illustrated in Fig. 5 is that steps S302, S304 and S306 are added.
[0127] After there is an instruction for starting the superimposition information display
process, the procedure management unit 144 estimates a current procedure in step
S302. Specifically, the procedure management unit 144 activates the camera 6 1 so as
to instruct the camera 6 1 to capture an image, and acquires an external scenery image
captured by the camera 61. The procedure management unit 144 performs image
recognition on the acquired external scenery image so as to estimate a current
procedure from a plurality of procedures stored in the procedure table 126. In this way,
the procedure management unit 144 can automatically estimate a current procedure on
the basis of the image (external scenery image) in the visual field direction of the user,
acquired by the camera 6 1 (image acquisition unit).
[0128] After the current procedure is estimated, the superimposition processing unit 142a
performs a superimposition information generation process in step S108. Details
thereof are the same as in the first embodiment described in Fig. 8.
[0129] After the superimposition information generation process is performed, the procedure
management unit 144 generates procedure information in step S304. Specifically, the
procedure management unit 144 performs processes indicated in the following
procedures dl to d4.
[0130] Procedure dl) The procedure management unit 144 searches the procedure table 126,
and extracts entries correlated with the current procedure estimated in step S302.
Procedure d2) The procedure management unit 144 determines whether or not an
image (an image indicating content to be performed by the user) is stored in the image
field among the extracted entries. The procedure management unit 144 performs the
procedure d3 if an image is stored, and performs the procedure d4 if an image is not
stored.
Procedure d3) The procedure management unit 144 generates procedure information
in which the image stored in the image field is disposed in an upper end part, the
character string stored in the instruction content field is disposed in a lower end part,
and black dummy data is disposed in other parts. In addition, the character string
preferably has a color with high visibility in consideration of a color of the external
scenery image.
Procedure d4) The procedure management unit 144 generates procedure information
in which the character string stored in the instruction content field is disposed in a
lower end part, and black dummy data is disposed in other parts.
[0131] In addition, the arrangement of the image and the character string in the procedures
d3 and d4 may be arbitrarily changed. However, in order to inform the user of content
to be performed in an aspect of not overly hindering the visual field of the user, the
image and the character string are preferably disposed around the end part (in other
words, around the four sides thereof) of the procedure information CP4.
[0132] Fig. 16 is a diagram illustrating an example of the procedure information. In the
procedure information CP4, a character string MG4, "Insert the artificial finger joint",
indicating content to be performed by the user, is disposed in the lower end part. In
addition, an image IM4 indicating a model in which an artificial finger joint is inserted
is disposed in the upper right end part.
[0133] After the procedure information CP4 is generated, the superimposition processing
unit 142a and the procedure management unit 144 display the superimposition in
formation CPl and the procedure information in step S306. Specifically, the superim
position processing unit 142a transmits the superimposition information CPl to the
image processing unit 160, and the procedure management unit 144 transmits the
procedure information CP4 to the image processing unit 160. The image processing
unit 160 which has received the superimposition information CPl and the procedure
information CP4 generates a composite image which has the superimposition in
formation CPl and the procedure information CP4 as respective layers, and performs
the display process described in Fig. 2 on the generated composite image. As a result,
the superimposition information CPl and the procedure information CP4 are displayed
as the virtual image VI in the visual field VR of the user of the head mounted display
100a.
[0134] As described above, in the superimposition information display process according to
the second embodiment, the image display section 20 allows the user to visually
recognize, as the virtual image VI (video), the procedure information CP4 indicating
content to be performed by the user of the head mounted display 100a in the current
procedure. For this reason, in addition to the effects of the first embodiment, it is
possible to visually show content to be performed by the user of the head mounted
display 100a using the image. Therefore, it is possible to more intelligibly support
work performed by the user.
[0135] In addition, in the second embodiment, a description has been made of a case where
a picture or the like of a scene in which a third party exemplarily performs a corre
sponding procedure is stored in the image field of the procedure table 126, as an
example. However, when an actual individual specific procedure is performed, an
image indicating content to be performed by the user may be stored in the image field
of the procedure table 126. Specifically, for example, an image indicating an incision
site of the skin of a patient who actually undergoes surgery, or an image indicating a
site through which a medical instrument (for example, a medical screw, a pacemaker, a
catheter, an artificial joint, or the like) is inserted into a body of a patient who actually
undergoes surgery, may be stored in the image field of the procedure table 126. In this
case, the procedure management unit 144 performs processes indicated in the
following procedures e l to e9 instead of steps S108, S304 and S306 of the superim
position information display process (Fig. 15).
[0136] Procedure el) The procedure management unit 144 searches the procedure table 126,
and extracts entries correlated with the current procedure estimated in step S302.
Procedure e2) The procedure management unit 144 determines whether or not an
image is stored in the image field among the extracted entries. The procedure
management unit 144 performs the procedures e3 to e7 and e9 if an image is stored,
and performs the procedures e8 and e9 if an image is not stored.
[0137] Procedure e3) The superimposition processing unit 142a acquires an external scenery
image. Details thereof are the same as in step S202 of the superimposition information
generation process (Fig. 8).
Procedure e4) The superimposition processing unit 142a performs image recognition
on the external scenery image so as to extract a feature of the target object. Details
thereof are the same as in step S204 of the superimposition information generation
process.
Procedure e5) The superimposition processing unit 142a performs image recognition
on the image stored in the image field so as to extract a feature of the target object.
Details thereof are the same as in step S206 of the superimposition information
generation process. However, instead of the "invisible information" in step S206, the
"image stored in the image field" is read.
Procedure e6) The superimposition processing unit 142a corrects the image stored in
the image field so that the feature of the external scenery image extracted in the
procedure e4 matches the feature of the image stored in the image field, extracted in
the procedure e5, so as to generate superimposition information CP5. Details thereof
are the same as in step S208 of the superimposition information generation process.
However, instead of the "invisible information" in step S208, the "image stored in the
image field" is read.
Procedure e7) The procedure management unit 144 adds a layer indicating the
procedure information to the superimposition information CP5 generated in the
procedure e6. Specifically, the procedure management unit 144 disposes the character
string stored in the instruction content field of the entry extracted in the procedure el,
in a lower end part of the superimposition information CP5.
[0138] Fig. 17 is a diagram illustrating an example of superimposition information to which
the procedure information is added. In the superimposition information CP5, an image
IM5, which indicates an incision site of the skin of a patient who actually undergoes
surgery, is disposed at an angle, a position and a size matching the feature of the target
object (the object included in the external scenery) in the visual field direction of the
user. In addition, in the superimposition information CP5, a character string MG5,
"Incise the skin of the surgical site using a scalpel with a circular edge to expose the
finger joint", indicating content to be performed by the user, is disposed in a lower end
part.
[0139] Procedure e8) The procedure management unit 144 generates procedure information
CP4 in which the character string stored in the instruction content field of the entry
extracted in the procedure e l is disposed in a lower end part, and black dummy data is
disposed in other parts.
Procedure e9) The procedure management unit 144 displays the superimposition in
formation CP5 generated in the procedure e7 or the procedure information CP4
generated in the procedure e8. Specifically, the procedure management unit 144
transmits the generated superimposition information CP5 or procedure information
CP4 to the image processing unit 160. The image processing unit 160 having received
the superimposition information CP5 or the procedure information CP4 performs the
display process described in Fig. 2 thereon. As a result, the superimposition in
formation CP5 or the procedure information CP4 is displayed as the virtual image VI
in the visual field VR of the user of the head mounted display 100a.
[0140] Fig. 18 is a diagram illustrating an example of a virtual image which is visually
recognized by the user due to the procedures e l to e9. As illustrated in Fig. 18, the su
perimposition information CP5 is displayed as the virtual image VI in the visual field
VR of the user of the head mounted display 100a. As above, the user of the head
mounted display 100a according to the present embodiment can visually recognize the
real image (a state in which the cutting edge of the scalpel held by the surgeon is in
contact with the second joint of the little finger of the patient's right hand) in the
external scenery SC which is viewed through the right optical image display unit 26
and the left optical image display unit 28 and the superimposition information CP5
(that is, the image indicating the incision site of the skin of the patient who actually
undergoes the surgery, and the character string indicating the content to be performed
by the user) which is viewed as the virtual image VI, in a superimposition state. In this
way, it is possible to support work performed by the user of the head mounted display
100a more individually and specifically, and thus it is possible to improve accuracy
and efficiency of the work.
[0141] B-3. Additional processes of superimposition information display process
In addition, in the superimposition information display process (Fig. 15) according to
the second embodiment, the additional processes 1 to 7 described in "A-3. Additional
processes of superimposition information display process" may be further performed.
The additional processes may be added singly or added in combination. Further, the
additional process 7 is mainly performed by the procedure management unit 144, and
may be performed on procedure information generated by the procedure management
unit 144.
[0142] C. Third embodiment
In the third embodiment of the invention, a description will be made of a con
figuration capable of further improving accuracy of superimposition of "information
(superimposition information) which is not shown in an appearance of an object" on a
"real image of the object which is a target of the superimposition information display
process" in the superimposition information display process in the first embodiment.
Hereinafter, only parts having configurations and operations different from the first
embodiment will be described. In addition, in the drawings, the same constituent
elements as in the first embodiment are given the same reference numeral as in the first
embodiment, and detailed description thereof will be omitted.
[0143] C-l. Configuration of head mounted display
Fig. 19 is a diagram illustrating a schematic configuration of a head mounted display
100b according to the third embodiment. Fig. 20 is a functional block diagram i l
lustrating a configuration of the head mounted display 100b according to the third em
bodiment. A difference from the first embodiment illustrated in Figs. 1 and 2 is that a
control section 10b is provided instead of the control section 10, and an image display
section 20b is provided instead of the image display section 20.
[0144] As illustrated in Fig. 20, the control section 10b includes a storage unit 120b instead
of the storage unit 120, and a CPU 140b instead of the CPU 140. The storage unit 120b
further includes parallax information 128 in addition to the invisible information
storage portion 122 and the appearance information storage portion 124 described in
the first embodiment.
[0145] Fig. 2 1 is a diagram for explaining the parallax information 128. The parallax in
formation 128 in the present embodiment is a table in which a plurality of distances Lx
between the user and virtual images visually recognized by the user are correlated with
parallaxes Px which respectively correspond to the plurality of distances Lx. In
addition, the parallax indicates a disparity (difference) between images formed on the
retinas of both eyes. In the example of Fig. 21, a distance LI between the user and a
virtual image VII visually recognized by the user is correlated with a parallax of 2
mm. In addition, a distance L2 between the user and a virtual image VI2 visually
recognized by the user is correlated with a parallax of 6 mm, and a distance L3
between the user and a virtual image VI3 visually recognized by the user is correlated
with a parallax of 10 mm. Further, the human depth interval dramatically decreases as
a distance between the user and a virtual image visually recognized by the user
becomes larger. For this reason, a larger volume of data may be prepared in the
parallax information 128 as the distance Lx becomes smaller.
[0146] This correlation is created in advance by measuring distances between positions of
virtual images (that is, image formation positions where a virtual image visually
recognized by the right eye and a virtual image visually recognized by the left eye)
visually recognized by both eyes of the user, and both eyes of the user, as a result of
deviating a display position of a virtual image for the right eye and a display position
of a virtual image for the left eye. In addition, in a case where the virtual image for the
right eye and the virtual image for the left eye are assumed to be the same virtual
image, a position of a virtual image visually recognized by both eyes of the user is
defined as a "reference plane". In this case, an image formation position when creating
the parallax information 128 may be located on the front side of the reference plane
when viewed from the user, and may be located on the rear side of the reference plane
when viewed from the user. An image formation position may be located both on the
front side and on the rear side of the reference plane. In addition, in the former case, a
virtual image appears to protrude toward the user, and, in the latter case, a virtual
image appears to be depressed from the user.
[0147] The above-described parallaxes of 2 mm, 6 mm and 10 mm are parallaxes of a
central part of a virtual image corresponding to a discriminative visual field and an
effective visual field. Therefore, for more accuracy, parallaxes (1.9 mm, 5 mm, and 9
mm) of parts corresponding to a steady-gaze stable field of the user, parallaxes (1.8
mm, 4 mm, and 8 mm) of parts corresponding to an induced visual field of the user,
and parallaxes (1.7 mm, 3 mm, and 7 mm) of parts corresponding to an auxiliary visual
field of the user, and the like may be differentiated from each other and be stored in the
parallax information 128. Furthermore, the specific numerical values in Fig. 2 1 are
only an example.
[0148] In addition, the discriminative visual field is a range in which, among the human
visual fields, a visual function such as vision or color discrimination is good, and thus
high accuracy information can be accepted. The discriminative visual field is typically
within several degrees from the center. The effective visual field is a range in which,
among the human visual fields, information is gazed steadily at only with the eye
movement, and specific information can be instantaneously accepted. The effective
visual field is typically within about 15 degrees to both the left and the right, within
about 8 degrees upwardly, and within about 12 degrees downwardly. The steady-gaze
stable field is a range in which, among the human visual fields, a steady gaze is
possible comfortably in a state in which the head movement assists the eye movement.
The steady-gaze stable field is typically within 30 to 45 degrees to both the left and the
right, within 20 to 30 degrees upwardly, and within 25 to 40 degrees downwardly. The
induced visual field is a range in which, among the human visual fields, there is only
discrimination capability that is a degree of being capable of determining the presence
of presented information, but the human spatial coordinate sense is influenced. The
induced visual field is typically within 30 to 100 degrees horizontally, and within 20 to
85 degrees vertically. The auxiliary visual field is a range in which, among the human
visual fields, information acceptance is notably decreased, and an auxiliary function
that is a degree of inducing a steady gaze operation due to a strong stimulus or the like
is performed. The auxiliary visual field is typically within 100 to 200 degrees hori
zontally, and within 85 to 135 degrees vertically.
[0149] The CPU 140b of Fig. 20 includes a superimposition processing unit 142b instead of
the superimposition processing unit 142. The superimposition processing unit 142b is
different from the first embodiment in content of the performed superimposition in
formation display process. Details thereof will be described later.
[0150] As illustrated in Fig. 20, the image display section 20b further includes pupil image
capturing cameras 62 and 63, near-infrared-ray receiving and emitting units 64 and 65,
a nine-axis sensor 66, and a depth sensor 67, in addition to the respective units
described in the first embodiment.
[0151] The pupil image capturing camera 62 is disposed near the end part ER (Fig. 1) of the
image display section 20. The pupil image capturing camera 62 captures an image of
the pupil of the right eye RE of the user when the user wears the head mounted display
100b, in the rear side direction of the image display section 20. Similarly, the pupil
image capturing camera 63 is disposed near the EL (Fig. 1) of the image display
section 20. The pupil image capturing camera 63 captures an image of the pupil of the
left eye LE of the user when the user wears the head mounted display 100b.
[0152] The near- infrared-ray receiving and emitting unit 64 is disposed near the end part ER
(Fig. 1) of the image display section 20. The near-infrared-ray receiving and emitting
unit 64 applies infrared rays toward the right eye RE of the user and acquires reflected
light in the rear side direction of the image display section 20. Similarly, the nearinfrared-
ray receiving and emitting unit 65 is disposed near the end part EL (Fig. 1) of
the image display section 20. The near-infrared-ray receiving and emitting unit 65
applies infrared rays toward the left eye LE of the user and acquires reflected light in
the rear side direction of the image display section 20.
[0153] The pupil image capturing cameras 62 and 63 and the near-infrared-ray receiving and
emitting units 64 and 65 function as a "visual line detection unit" which detects a
visual line of the user in cooperation with each other. In addition, some of the pupil
image capturing cameras 62 and 63 and the near-infrared-ray receiving and emitting
units 64 and 65 may be omitted. For example, the near-infrared-ray receiving and
emitting units 64 and 65 may be omitted, and the pupil image capturing camera 63 and
the near-infrared-ray receiving and emitting unit 65 may be omitted.
[0154] The nine-axis sensor 66 is disposed at the position corresponding to the glabella of
the user when the user wears the image display section 20. The nine-axis sensor 66 is a
motion sensor which detects acceleration (in three axes), angular velocity (in three
axes), and geomagnetism (in three axes). The nine-axis sensor 66 is provided in the
image display section 20, and thus functions as a "motion detection unit" which detects
a motion of the head of the user when the image display section 20 is mounted on the
head of the user. Here, the motion of the head includes velocity, acceleration, angular
velocity, a direction, and a change in a direction of the head.
[0155] The depth sensor 67 is disposed at the position corresponding to the glabella of the
user when the user wears the image display section 20. The depth sensor 67 includes
an infrared-ray irradiation portion and an infrared-ray sensor, and measures a depth of
each point located in front of the depth sensor 67. The depth sensor 67 functions as a
"position detection unit" which detects a position of the user relative to an object
(target object) which is a target of the superimposition information display process.
[0156] C-2. Superimposition information display process
Fig. 22 is a flowchart illustrating procedures of the superimposition information
display process according to the third embodiment. A difference from the first em
bodiment illustrated in Fig. 5 is that: step S602 is provided instead of step S102; step
S604 is provided instead of step S108; step S606 is provided instead of step SI 10; and
step S608 is further provided after step SI 10. In addition, the superimposition in
formation display process according to the third embodiment is performed in a threedimensional
display mode.
[0157] In step S602, the user stores a plurality of invisible information pieces regarding a
certain object which is a target of the superimposition information display process in
the invisible information storage portion 122. Here, the invisible information is an
image indicating a structure of a target object acquired by using image capturing ap
paratuses such as a CT apparatus, an MRI apparatus, an X-ray apparatus, an
endoscope, and a thermography apparatus, in the same manner as in step S102 of Fig.
5. In the superimposition information display process of the third embodiment, one
invisible information piece is selected from a plurality of invisible information pieces
in accordance with a positional relationship between the target object and the user. For
this reason, in step S602, preferably, invisible information regarding the target object is
acquired from various angles and is stored in the invisible information storage portion
122.
[0158] Fig. 23 is a diagram illustrating an example of the invisible information. As i l
lustrated in Fig. 23, the invisible information storage portion 122 of the third em
bodiment stores a plurality of coronary plane CT images SI1 which are any planar
images that divide the body of a patient PA into a front side and a rear side; a plurality
of transverse cross-sectional CT images SI2 which are any planar images that divide
the body of the patient PA into planes perpendicular to the longitudinal direction; and
an X-ray image SI3 of a surgical site of the patient PA. Further, the invisible in
formation illustrated in Fig. 23 is only an example, and, for example, a sagittal plane
CT image may be included, and an MRI image or an endoscope image may be
included instead of the CT image.
[0159] In step S604 of Fig. 22, the superimposition processing unit 142b generates superim
position information by performing either of "C-2-1. First superimposition information
generation process" and "C-2-2. Second superimposition information generation
process" described below. A difference between the first superimposition information
generation process and the second superimposition information generation process is
accuracy with which the superimposition processing unit 142b corrects invisible in
formation. Specifically, correction is performed in the second superimposition in
formation generation process with higher accuracy than in the first superimposition in
formation generation process.
[0160] Which one of the first superimposition information generation process and the
second superimposition information generation process is performed may be
predefined, and may be designated by the user. In addition, for example, which one of
the two is performed may be automatically determined depending on an amount of a
motion of the head of the user detected by the nine-axis sensor, and may be auto
matically determined depending on the type or a technique of surgery performed using
the head mounted display 100b.
[0161] C-2- 1. First superimposition information generation process
Fig. 24 is a flowchart illustrating procedures of the first superimposition information
generation process. In step S402, the superimposition processing unit 142b acquires an
external scenery image. Specifically, the superimposition processing unit 142b
activates the camera 6 1 so as to instruct the camera 6 1 to capture an image, and
acquires an external scenery image captured by the camera 61.
[0162] Fig. 25 is a diagram for explaining step S404 of the first superimposition information
generation process. The superimposition processing unit 142b acquires a visual line
and a position of the user in step S404 of Fig. 24. Specifically, the superimposition
processing unit 142b acquires a visual line of the user from detection values of the
pupil image capturing cameras 62 and 63 and the near-infrared-ray receiving and
emitting units 64 and 65. Subsequently, the superimposition processing unit 142b
obtains a steady gaze point distance which is a distance between the user and a point
PP (in Fig. 25, a steady gaze point PP) at which the user gazes steadily, from the
acquired visual line of the user. In addition, the steady gaze point is a point where the
central foveae (locations where vision is most acute in the retinas) of both eyes of the
user intersect each other on extension lines thereof. Further, the superimposition
processing unit 142b acquires positions POl to P03 (Fig. 25) of the user relative to the
target object from a detection value of the depth sensor 67. In this case, the superimposition
processing unit 142b may use, for example, a depth DP (Fig. 25) for features
Ml to M4 (in Fig. 25, edges or markers) of the target object as a reference for
acquiring a position.
[0163] In step S406, the superimposition processing unit 142b selects one invisible in
formation piece corresponding to the position of the user relative to the target object
from the plurality of invisible information pieces stored in the invisible information
storage portion 122. Specifically, the superimposition processing unit 142b selects
several invisible information pieces which are considered to be optimum on the basis
of the positional relationship between the user and the target object, acquired in step
S404. For example, in a case where the user is located at a position where the user
looks down at the patient PA (target object) illustrated in Fig. 23, the superimposition
processing unit 142b selects a plurality of coronary plane CT images SI1. Next, the su
perimposition processing unit 142b selects one invisible information piece corre
sponding to the depth acquired in step S404 from the selected several invisible in
formation pieces. In addition, in step S406, the superimposition processing unit 142b
may present the selected several invisible information pieces to the user as candidates,
and may receive a selection of the invisible information from the user.
[0164] In step S408, the superimposition processing unit 142b performs image recognition
on the external scenery image so as to extract a feature of the target object. Details
thereof are the same as in step S204 of Fig. 8 described in the first embodiment.
[0165] In step S410, the superimposition processing unit 142b performs image recognition
on the invisible information selected in step S408 so as to extract a feature of the target
object. Details thereof are the same as in step S206 of Fig. 8 described in the first em
bodiment.
[0166] In step S412, the superimposition processing unit 142b corrects the invisible in
formation selected in step S408 so that the feature of the external scenery image
extracted in step S408 and the feature of the invisible information extracted in step
S410 match each other, thereby generating superimposition information. Details
thereof are the same as in step S208 of Fig. 8 described in the first embodiment.
[0167] In step S414, the superimposition processing unit 142b gives a parallax to the superimposition
information. Specifically, the superimposition processing unit 142b
searches the parallax information 128 by using the steady gaze point distance obtained
in step S404 as a keyword, and acquires a parallax stipulated in the parallax in
formation 128. The superimposition processing unit 142b gives the parallax corre
sponding to the steady gaze point distance acquired from the parallax information 128
to the superimposition information generated in step S412, so as to generate superim
position information for the right eye and superimposition information for the left eye.
As described above, according to step S414, the superimposition processing unit 142b
can easily generate the superimposition information for the right eye and the superim
position information for the left eye by using the parallax information 128 which is
prepared in advance.
[0168] After the first superimposition information generation process is completed, the su
perimposition processing unit 142b displays the superimposition information in step
S606 of Fig. 22. Specifically, the superimposition processing unit 142b transmits the
superimposition information for the right eye and the superimposition information for
the left eye generated due to the superimposition information generation process to the
image processing unit 160. The image processing unit 160 which has received the su
perimposition information for the right eye and the superimposition information for the
left eye performs the display process described in Fig. 2 on the respective pieces of the
received superimposition information. As a result, a virtual image based on the superimposition
information is displayed in the visual field of the user of the head mounted
display 100b. Here, the parallax corresponding to the steady gaze point distance is
given to the superimposition information for the right eye and the superimposition in
formation for the left eye. For this reason, the virtual image visually recognized by the
user of the head mounted display 100b is formed around the steady gaze point of the
user and is displayed in a three-dimensional manner.
[0169] In step S608 of Fig. 22, the superimposition processing unit 142b determines whether
or not the visual line and the position of the user are changed. The visual line and the
position of the user may be acquired in the same method as in step S404 of the first su
perimposition information generation process (Fig. 25). In a case where the position
and the visual line of the user are changed so as to exceed a predetermined threshold
value (step S608: YES), the superimposition processing unit 142b causes the process
to proceed to step S604, and repeatedly performs the first or second superimposition
information generation process. In a case where the position and the visual line of the
user are changed so as not to exceed the predetermined threshold value (step S608:
NO), the superimposition processing unit 142b causes the process to proceed to step
SI 12. In addition, the predetermined threshold value may be arbitrarily defined.
Further, a threshold value related to a change amount of a position and a threshold
value related to a change amount of a visual line may be different from each other.
[0170] As described above, according to the first superimposition information generation
process, the superimposition processing unit 142b generates the superimposition in
formation for the right eye and the superimposition information for the left eye to
which the parallax corresponding to a distance (steady gaze point distance) between
the user and the steady gaze point at which the user gazes steadily is given, and the
image processing unit 160 of the image display section 20 allows the user to visually
recognize the virtual image based on this superimposition information. As a result, the
user can visually recognize the superimposition information at a position of the same
distance as the steady gaze point.
[0171] In addition, according to the first superimposition information generation process, the
superimposition processing unit 142b can select one invisible information piece corre
sponding to a position of the user relative to the target object from a plurality of
invisible information pieces stored in the invisible information storage portion 122 of
the storage unit 120b, so as to correct the selected invisible information. For this
reason, the superimposition processing unit 142b can generate superimposition in
formation by using appropriately invisible information on the basis of a positional rela
tionship between the user and the target object.
[0172] C-2-2. Second superimposition information generation process
Fig. 26 is a flowchart illustrating procedures of the second superimposition in
formation generation process. A difference from the first superimposition information
generation process illustrated in Fig. 24 is that step S502 is provided instead of step
S404, and step S504 is provided instead of step S412.
[0173] In step S502, the superimposition processing unit 142b acquires a visual line, a
position, an angle of the head of the user. The visual line and the position of the user
are the same as in step S404 of the first superimposition information generation
process (Fig. 25). In addition, the superimposition processing unit 142b acquires the
angle of the head of the user from a detection value of the nine-axis sensor 66.
[0174] Fig. 27 is a diagram for explaining step S504 of the second superimposition in
formation generation process. In step S504 of Fig. 26, the superimposition processing
unit 142b corrects the invisible information. Specifically, the superimposition
processing unit 142b corrects the invisible information selected in step S408 by using a
method described in the following procedures f l to f3.
[0175] Procedure fl) The superimposition processing unit 142b divides an image indicating
the invisible information SI into a plurality of regions Al to An (Fig. 27). As for this
division, the superimposition processing unit 142b may measure distances between the
user and respective parts of the target object by using triangulation, and may divide the
image indicating the invisible information SI into regions on the basis of the mea
surement result. Specifically, for example, the superimposition processing unit 142b
may group the respective parts of the target object for each measured distance from the
user, and may collect invisible information pieces SI corresponding to the grouped re
spective parts as a single region. In addition, as for this grouping, a predetermined
allowable range (threshold value) may be set to the distance. In this way, the superim
position processing unit 142b can divide the region of the image indicating the
invisible information SI into a plurality of regions efficiently and in light of the actual
situation.
[0176] Procedure f2) The superimposition processing unit 142b performs at least one
processing of enlargement, reduction, trimming, distortion, and removal of noise, on
each of the plurality of regions divided in the procedure fl, so as to generate superim
position information. During this processing, the superimposition processing unit 142b
performs the processing so that the feature of the external scenery image and the
feature of the invisible information SI, detected in steps S408 and S410, match each
other.
[0177] Procedure f3) The superimposition processing unit 142b may change processing
(correction) accuracy on the basis of an information accepting characteristic within the
visual field of the user when the processing in the procedure f2 is performed. Here, the
information accepting characteristic indicates an extent of visual capability. The
human visual fields may be divided into (1) a discriminative visual field, (2) an
effective visual field, (3) a steady-gaze stable field, (4) an induced visual field, and (5)
an auxiliary visual field, in an order of a superior information accepting characteristic
within the visual field (Fig. 21). Therefore, in a case where, among the plurality of
regions divided in the procedure fl, a processing target region belongs to a visual field
with a superior information accepting characteristic (for example, the above (1) and
(2)), the superimposition processing unit 142b may increase accuracy of the
processing, and, in a case where the processing target region belongs to a visual field
(for example, the above (4) and (5)) with an inferior information accepting charac
teristic, the superimposition processing unit 142b may reduce processing accuracy.
This is because, if accuracy of superimposition of the invisible information SI and the
target object is increased in the visual field with an inferior information accepting char
acteristic, there is a concern of uselessness due to a limit of perception capability of the
user.
[0178] After the second superimposition information generation process is completed, the
superimposition processing unit 142b displays the superimposition information in step
S606 of Fig. 22. As a result, in the same manner as in the first superimposition in
formation generation process, a virtual image based on the superimposition in
formation is displayed in the visual field of the user of the head mounted display 100b.
In addition, the virtual image visually recognized by the user is formed around the
steady gaze point of the user and is displayed in a three-dimensional manner.
[0179] As described above, according to the second superimposition information generation
process, the superimposition processing unit 142b divides the image indicating the
invisible information SI of the object stored in the invisible information storage portion
122 of the storage unit 120b into a plurality of regions Al to An. Then, the superim
position processing unit 142b performs at least one processing of enlargement,
reduction, trimming, distortion, and removal of noise, on each of the plurality of
regions, so as to correct the regions. As above, the superimposition processing unit
142b divides the image indicating the invisible information SI into a plurality of
regions Al to An, and then corrects each region. Therefore, it is possible to improve
accuracy of the correction of the image indicating the invisible information SI.
[0180] Further, according to the second superimposition information generation process, the
superimposition processing unit 142b changes accuracy of processing (correction) on
the invisible information SI on the basis of the information accepting characteristic
within the visual field of the user. As a result, the superimposition processing unit 142b
can efficiently correct the invisible information SI.
[0181] C-3. Additional processes of superimposition information display process
In addition, in the superimposition information display process (Fig. 22), the ad
ditional processes 1 to 7 described in "A-3. Additional processes of superimposition in
formation display process" may be further performed. The additional processes may be
added singly or added in combination.
[0 182] D. Modification Example
In the above-described embodiments, some of the constituent elements realized in
hardware may be realized in software, and, conversely, some of the configurations
realized in software may be realized in hardware. In addition, the following modi
fications may also occur.
[0183] Modification Example 1
In the above-described embodiments, a configuration of the head mounted display
has been exemplified. However, any configuration of the head mounted display may be
defined within the scope without departing from the spirit of the invention, and, for
example, each configuration unit may be added, deleted, changed, or the like.
[0184] In the above-described embodiments, the allocation of the constituent elements to the
control section and the image display section are only an example, and may employ
various aspects. For example, the following aspects may be employed: (i) an aspect in
which a processing function such as a CPU and a memory is mounted in the control
section, and only a display function is mounted in the image display section; (ii) an
aspect in which a processing function such as a CPU and a memory is mounted in both
the control section and the image display section; (iii) an aspect in which the control
section and the image display section are integrally formed (for example, an aspect in
which the image display section includes the control section and functions as a glasses
type wearable computer); (iv) an aspect in which a smart phone or a portable game
machine is used instead of the control section; and (v) an aspect in which the control
section and the image display section are configured to communicate by wireless with
each other and to be supplied with power in a wireless manner so as to remove the
connection unit (cords).
[0185] In the above-described embodiments, for convenience of description, the control
section is provided with the transmission unit, and the image display section is
provided with the reception unit. However, both of the transmission unit and the
reception unit of the above-described embodiments have a bidirectional commu
nication function, and thus can function as a transmission and reception unit. In
addition, for example, the control section illustrated in Fig. 2 is connected to the image
display section via the wired signal transmission path. However, the control section
and the image display section may be connected to each other via a wireless signal
transmission path such as a wireless LAN, infrared communication, or Bluetooth
(registered trademark).
[0186] For example, configurations of the control section and the image display section i l
lustrated in Fig. 2 may be arbitrarily changed. Specifically, for example, the touch pad
may be omitted from the control section, and an operation may be performed using
only the cross key. In addition, other operation interfaces such as an operation stick
may be provided in the control section. Further, there may be a configuration in which
the control section is connectable to a device such as a keyboard or a mouse so as to
receive an input from the keyboard or the mouse.
[0187] For example, in the above-described embodiments, the head mounted display is a
binocular transmission type head mounted display, but may be a monocular head
mounted display. In addition, the head mounted display may be a non-transmissive
head mounted display through which external scenery is blocked from being
transmitted in a state in which the user wears the head mounted display.
[0188] Figs. 28A and 28B are diagrams illustrating exterior configurations of head mounted
displays in a modification example. In a case of an example of Fig. 28A, a difference
from the head mounted display 100 illustrated in Fig. 1 is that an image display section
20c includes a right optical image display unit 26c instead of the right optical image
display unit 26 and a left optical image display unit 28c instead of the left optical
image display unit 28. The right optical image display unit 26c is formed to be smaller
than the optical member of the first embodiment, and is disposed on the obliquely
upper side of the right eye of the user when the head mounted display is mounted.
Similarly, the left optical image display unit 28c is formed to be smaller than the
optical member of the above-described embodiments, and is disposed on the obliquely
upper side of the left eye of the user when the head mounted display is mounted. In a
case of an example of Fig. 28B, a difference from the head mounted display 100 i l
lustrated in Fig. 1 is that an image display section 20d includes a right optical image
display unit 26d instead of the right optical image display unit 26 and a left optical
image display unit 28d instead of the left optical image display unit 28. The right
optical image display unit 26d is formed to be smaller than the optical member of the
first embodiment, and is disposed on the obliquely lower side of the right eye of the
user when the head mounted display is mounted. Similarly, the left optical image
display unit 28d is formed to be smaller than the optical member of the first em
bodiment, and is disposed on the obliquely lower side of the left eye of the user when
the head mounted display is mounted. As above, the optical image display units have
only to be disposed near the eyes of the user. Any size of the optical member forming
the optical image display units may be used, and the head mounted display may be im
plemented in an aspect in which the optical image display units cover only a part of the
eyes of the user; in other words, the optical image display units do not completely
cover the eyes of the user.
[0189] For example, a description has been made that the function units such as the image
processing unit, the display control unit, the superimposition processing unit, the
procedure management unit, and the sound processing unit are realized by the CPU de
veloping a computer program stored in the ROM or the hard disk on the RAM and
executing the program. However, these function units may be configured using an ap
plication specific integrated circuit (ASIC) which is designed for realizing each of the
corresponding functions.
[0190] For example, in the above-described embodiments, the image display section of the
head mounted display is worn as glasses, but the image display section may be a
typical flat display device (a liquid crystal display device, a plasma display device, an
organic EL display device, or the like). Also in this case, the control section and the
image display section may be connected to each other via a wired signal transmission
path, and may be connected to each other via a wireless signal transmission path. In
this way, the control section may be used as a remote control of a typical flat display
device.
[0191] Further, as an image display section, instead of the image display section which is
worn as glasses, other types of image display sections such as an image display section
which is worn as, for example, a cap, may be employed. In addition, the earphone may
employ an ear-mounted type or a head band type, or may be omitted. Further, for
example, a head-up display (HUD) may be configured to be mounted in a vehicle such
as an automobile or an airplane. Furthermore, for example, the head mounted display
may be configured to be built in a body protection tool such as a helmet.
[0192] For example, in the above-described embodiments, a secondary battery is used as the
power supply, but the power supply is not limited to the secondary battery and may use
various batteries. For example, a primary battery, a fuel cell, a solar cell, a thermal cell,
and the like may be used.
[0193] For example, in the above-described embodiments, the image light generation
portion is configured using the backlight, the backlight control portion, the LCD, and
the LCD control portion. However, the above aspect is only an example. The image
light generation portion may include a configuration portion for realizing other types
along with this configuration portion or instead of this configuration portion. For
example, the image light generation portion may include an organic electroluminescent
(EL) display and an organic EL controller. In addition, for example, instead of the
LCD, a digital micromirror device may be used as the image light generation portion.
Further, for example, the invention is applicable to a laser retinal projective head
mounted display.
[0194] For example, the head mounted display may include a range finding sensor or a di s
placement sensor instead of the camera. In this case, the superimposition processing
unit may extract a feature of a target object by using the range finding sensor or the
displacement sensor so as to perform the same process as in the above-described em
bodiments.
[0195] For example, the head mounted display may be provided with an optical filter which
suppresses transmission of light with the same color as a color which is frequently used
in invisible information. In addition, the head mounted display may be provided with
an optical filter with a color complementary to a color which is frequently used in
invisible information. In this way, it is possible to increase visibility of the invisible in
formation.
[0196] For example, any one of the camera, the pupil image capturing cameras, the nearinfrared-
ray receiving and emitting units, the nine-axis sensor, and the depth sensor
may be omitted. For example, the pupil image capturing cameras and the nearinfrared-
ray receiving and emitting units may be omitted, and a camera provided on a
ceiling of a surgical room may be substituted, thereby detecting a visual line of the
user. In addition, in a case where the above-described constituent elements are not
omitted, the camera provided on the ceiling of the surgical room may be used together,
thereby improving processing accuracy of invisible information.
[0197] Modification Example 2
In the above-described embodiment, an example of the superimposition information
display process (Figs. 5 and 22) has been described. However, the procedures of the
process illustrated in Figs. 5 and 22 are only an example and may have various modi
fications. For example, some steps may be omitted, and other steps may be further
added. In addition, order of steps to be performed may be changed.
[0198] For example, in step S102, the user stores the invisible information of the object
which is a target of the superimposition information display process in the invisible in
formation storage portion. The invisible information of the above-described em
bodiments is assumed to be an image which is captured by an image capturing
apparatus (a CT apparatus, an MRI apparatus, an X-ray apparatus, an endoscope, a
thermography apparatus, or the like) and is represented in a two-dimensional manner.
However, invisible information which is represented in a three-dimensional manner
may be stored in step S102. The invisible information which is represented in a threedimensional
manner is a three-dimensional model which is created from a plurality of
images captured by the image capturing apparatus. In a case of using the invisible in
formation which is represented in a three-dimensional manner, the superimposition
processing unit performs rendering of a three-dimensional model on the basis of the
external scenery image acquired in step S202, between steps S202 and S204 of the su
perimposition information generation process (Fig. 8). In addition, the subsequent
processes are continuously performed using the rendered images as "invisible in
formation". In this way, it is possible to visually recognize information which is not
shown in the appearance of the target object from various directions. Further, the
invisible information stored in step S102 is not limited to a still image and may be a
moving image.
[0199] For example, in steps SI 14 and SI 16, another superimposition information piece is
generated on the basis of the appearance information. However, various information
pieces may be used as a source of another superimposition information piece. For
example, another invisible information piece may be used which is captured using the
image capturing apparatus and is different from the invisible information used when
generating the superimposition information CP1.
[0200] For example, the processes in and after step SI 12 may be omitted.
[0201] Modification Example 3
In the second embodiment, an example of the superimposition information display
process (Fig. 15) has been described. However, the procedures of the process i l
lustrated in Fig. 15 are only an example and may have various modifications. For
example, some steps may be omitted, and other steps may be further added. In
addition, order of steps to be performed may be changed.
[0202] For example, the procedure management unit may add information for arousing
user's attention to the procedure information as in the following example.
* In the procedure, matters to which the user should pay particular attention are
displayed in a pop-up form.
* An image or a character string in the procedure information is displayed in a
blinking state.
* A message or an icon indicating the degree of importance, the degree of difficulty,
a frequency of accidents, and the like of a corresponding procedure is displayed in the
procedure information.
[0203] Modification Example 4
In the above-described embodiments, an example of the procedure table has been
described. However, the details of the procedure table are only an example and may
have various modifications. For example, a field may be added, deleted, or changed. In
addition, the procedure table may be divided into a plurality of tables so as to be
normalized.
[0204] Modification Example 5
In the second embodiment, a description has been made of the configuration in which
the head mounted display is provided with the procedure table and the procedure
management unit, and can support work performed by the user without requiring other
devices. However, also in a work supporting system including the head mounted
display and an externally provided server, the same process may be performed.
[0205] For example, the head mounted display is configured to have the procedure
management unit, and the server is configured to have the procedure table. In this case,
the procedure management unit of the head mounted display accesses the procedure
table stored in the server so as to refer to data of the procedure table. In this way, the
information of the procedure table stored in a single server can be shared by a plurality
of head mounted displays. As a result, the necessary labor spent updating the
procedure table is reduced compared with a configuration in which the procedure table
is provided in each head mounted display.
[0206] Modification Example 6
In the third embodiment, an example of the first and second superimposition in
formation generation processes (Figs. 24 and 26) has been described. However, the
procedures of the process illustrated in Figs. 24 and 26 are only an example and may
have various modifications. For example, some steps may be omitted, and other steps
may be further added. In addition, order of steps to be performed may be changed.
[0207] For example, a plurality of parallax information pieces may be stored in the storage
unit. The plurality of parallax information pieces may be, for example, parallax in
formation when a dominant eye is the right eye and parallax information when a
dominant eye is the left eye, and may be parallax information based on an interocular
distance. In this case, in step S414 of the first and second superimposition information
generation processes, one parallax information piece to be used in the process in step
S414 may be selected from a plurality of parallax information pieces stored in the
storage unit on the basis of characteristics (for example, the interocular distance or the
dominant eye) of the user of the head mounted display. In this way, the superim
position processing unit can select parallax information matching the characteristics
(for example, the dominant eye or the interocular distance) of the user, thereby
reducing visual discomfort which the user feels.
[0208] For example, in the first and second superimposition information generation
processes, an edge emphasis process or a binarization process may be used together,
thereby further improving accuracy of correction of invisible information.
[0209] Modification Example 7
The invention is not limited to the above-described embodiments or modification
examples, and may be implemented using various configurations within the scope
without departing from the spirit thereof. For example, the embodiments, the
examples, and the modification examples corresponding to technical features of the r e
spective aspects described in Summary of Invention and the technical features in the
modification examples may be exchanged or combined as appropriate in order to solve
some or all of the above-described problems, or in order to achieve some or all of the
above-described effects. In addition, if the technical feature is not described as an
essential feature in the present specification, the technical feature may be deleted as ap
propriate.
Reference Signs List
[0210] 10 Control Section
12 Lighting unit
14 Touch pad
16 Cross key
18 Power switch
20 Image display section
2 1 Right holding unit
22 Right display driving unit
23 Left holding unit
24 Left display driving unit
26 Right optical image display unit
28 Left optical image display unit
30 Earphone plug
32 Right earphone
34 Left earphone
40 Connection unit
42 Right cord
44 Left cord
46 Connection member
48 Main body cord
5 1 Transmission unit
52 Transmission unit
53 Reception portion
4 Reception portion
1 Camera (image acquisition unit)
2 Pupil image capturing camera (visual line detection unit)
3 Pupil image capturing camera (visual line detection unit)
64 Near-infrared-ray receiving and emitting unit (visual line detection unit)
65 Near-infrared-ray receiving and emitting unit (visual line detection unit)
66 Nine-axis sensor (motion detection unit)
67 Depth sensor (position detection unit)
110 Input information acquisition unit
100 Head mounted display
120 Storage unit
122 Invisible information storage portion
124 Appearance information storage portion
126 Procedure table
128 Parallax information
130 Power supply
140 CPU
142 Superimposition processing unit
144 Procedure management unit
160 Image processing unit
170 Sound processing unit
180 Interface
190 Display control unit
201 Right backlight control portion
202 Left backlight control portion
211 Right LCD control portion
212 Left LCD control portion
221 Right backlight
222 Left backlight
241 Right LCD
242 Left LCD
251 Right projection optical system
252 Left projection optical system
261 Right light guiding plate
262 Left light guiding plate
PCLK Clock signal
VSync Vertical synchronization signal
HSync Horizontal synchronization signal
Data Image data
Datal Right eye image data
Data2 Left eye image data
OA External apparatus
PC Personal Computer
SC External scenery
RE Right eye
VI Virtual image
EL End part
IL Illumination light
PL Image light
AP Apex
ER End part
VR Visual field
CPl Superimposition information
CP2 Enlargement information
CP3 Auxiliary information
CP4 Procedure information
CP5 Superimposition information
DP Depth
Lx Distance
PP Steady gaze point
Px Parallax
SI Invisible information
511 Coronary plane CT image
512 Transverse cross- sectional CT
513 X-ray image

Claims
A head mounted display which allows a user to visually recognize a
virtual image and external scenery, comprising:
an image display unit that forms the virtual image which is visually
recognized by the user; and
a superimposition processing unit that causes the image display unit to
form the virtual image based on superimposition information for super
imposing invisible information which is not shown in an appearance of
an object on the object included in the external scenery.
The head mounted display according to claim 1,
wherein the superimposition processing unit causes the image display
unit to repeatedly form the virtual image based on the superimposition
information corresponding to the object included in the external
scenery every predetermined time.
The head mounted display according to claim 1, further comprising:
a storage unit that stores the invisible information; and
an image acquisition unit that acquires an image of external scenery in
a visual field direction of the user when the user wears the head
mounted display,
wherein the superimposition processing unit extracts a feature of the
object included in the external scenery through image recognition on
the image in the visual field direction acquired by the image acquisition
unit, and corrects the invisible information of the object stored in the
storage unit on the basis of the extracted feature so as to generate the
superimposition information.
The head mounted display according to claim 3,
wherein the invisible information is an image indicating a structure of
the object.
The head mounted display according to claim 3,
wherein the feature is an edge of the object.
The head mounted display according to claim 3,
wherein the feature is a marker attached to the object.
The head mounted display according to claim 3,
wherein the correction is performed by performing at least one of en
largement, reduction, rotation, inversion, trimming, distortion, and
noise removal on an image indicating the invisible information.
The head mounted display according to claim 3,
WO 2014/125789 PCT7JP2014/000595
wherein the correction is performed by dividing an image indicating the
invisible information into a plurality of regions and performing at least
one of enlargement, reduction, trimming, distortion, and noise removal
on each of the plurality of regions.
[Claim 9] The head mounted display according to claim 8,
wherein the superimposition processing unit measures a distance
between the user and each part of the object by using triangulation, and
performs the division on the basis of a result of the measurement.
[Claim 10] The head mounted display according to claim 3,
wherein the superimposition processing unit changes accuracy of the
correction on the basis of an information accepting characteristic in
dicating an extent of visual capability within a visual field of the user.
[Claim 11] The head mounted display according to claim 3, further comprising:
a visual line detection unit that detects a visual line of the user,
wherein the image display unit forms both the virtual image corre
sponding to the right eye of the user and the virtual image corre
sponding to the left eye of the user,
wherein the superimposition processing unit further obtains a steady
gaze point distance which is a distance between the user and a steady
gaze point at which the user steadily gazes, from the visual line
detected by the visual line detection unit, and generates the superim
position information for the right eye and the superimposition in
formation for the left eye to which a parallax corresponding to the
obtained steady gaze point distance is given.
[Claim 12] The head mounted display according to claim 11, further comprising:
parallax information that correlates a plurality of distances between the
user and the virtual image visually recognized by the user with
parallaxes which respectively correspond to the plurality of distances,
wherein the superimposition processing unit gives a parallax corre
sponding to the steady gaze point distance by using the parallax in
formation.
[Claim 13] The head mounted display according to claim 12,
wherein the parallax information is provided in a plurality of pieces,
and
wherein the superimposition processing unit selects one parallax in
formation piece from the plurality of parallax information pieces in ac
cordance with the user.
[Claim 14] The head mounted display according to claim 3, further comprising:
WO 2014/125789 PCT7JP2014/000595
a position detection unit that detects a position of the user relative to the
object,
wherein the storage unit stores a plurality of invisible information
pieces regarding a single object,
wherein the superimposition processing unit selects one invisible in
formation piece corresponding to the position detected by the position
detection unit from the plurality of invisible information pieces stored
in the storage unit, and corrects the selected invisible information.
[Claim 15] The head mounted display according to claim 3,
wherein the superimposition processing unit further generates en
largement information in which at least a part of the object included in
the external scenery is extracted and enlarged, from the image in the
visual field direction acquired by the image acquisition unit, and
wherein the image display unit allows the user to visually recognize the
virtual image based on the superimposition information and the en
largement information.
[Claim 16] The head mounted display according to claim 1,
wherein the superimposition processing unit does not display the virtual
image in response to a request from the user.
[Claim 17] The head mounted display according to claim 1, further comprising:
a procedure management unit that generates procedure information
which is information regarding content which is to be performed by the
user in a current procedure,
wherein the image display unit allows the user to visually recognize the
virtual image based on the superimposition information and the
procedure information.
[Claim 18] The head mounted display according to claim 17,
wherein the procedure information includes at least one of an image of
a scene in which a third party exemplarily performs the procedure and
an image indicating content to be performed by the user when the in
dividual specific procedure is performed.
[Claim 19] The head mounted display according to claim 1,
wherein the superimposition processing unit further generates auxiliary
information which is additional information for assisting work of the
user, and
wherein the image display unit allows the user to visually recognize the
virtual image based on the superimposition information and the
auxiliary information.
PCT7JP2014/000595
The head mounted display according to claim 3,
wherein the superimposition processing unit further performs a color
conversion process for improving visibility of the generated superim
position information on the basis of the image in the visual field
direction acquired by the image acquisition unit.
The head mounted display according to claim 1,
wherein the superimposition processing unit withdraws the superim
position information which is displayed as the virtual image to a
position which does not hinder the visual field of the user, in response
to a request from the user.
The head mounted display according to claim 3,
wherein the superimposition processing unit generates the superim
position information corresponding to an overlap region in which a
display region of the virtual image overlaps with an acquisition region
of the image acquisition unit in the image acquired by the image ac
quisition unit.
The head mounted display according to claim 16,
wherein the procedure management unit generates the procedure in
formation corresponding to an overlap region in which a display region
of the virtual image overlaps with an acquisition region of the image
acquisition unit in the image acquired by the image acquisition unit.
A control method for a head mounted display which allows a user to
visually recognize a virtual image and external scenery, the method
comprising:
(a) forming the virtual image which is visually recognized by the user;
and
(b) forming the virtual image based on superimposition information for
superimposing invisible information which is not shown in an ap
pearance of an object on the object included in the external scenery.