Technical Field
[0001] The present invention relates to an information processing apparatus.
Background Art
[0002] There is known a head mounted display apparatus (a head mounted display (HMD))
which is a display apparatus mounted on the head. For example, the head mounted
display apparatus generates image light for an image using a liquid crystal display and
a light source, and leads the generated image light to the eyes of a user using a
projection optical system, a light guide plate, or the like, and thus the user is allowed to
recognize a virtual image.
[0003] PTL 1 discloses a system in which the head mounted display apparatus is used,
which improves display of information in a medical environment. In the technology
disclosed in PTL 1, information is acquired from a plurality of information sources
based on a query. The acquired information undergoes a filtering process based on
rules and is projected on eyepieces of a headset-type eyeglass viewer. In the
technology disclosed in PTL 1, a control right and a priority right for display of the in
formation on a fixed large display are determined based on a visual line of a person
that wears the head mounted display apparatus. PTL 2 discloses a face mounted video
display apparatus for medical use (a head mounted display apparatus). In the
technology disclosed in PTL 2, operation state information of peripheral devices of an
endoscope, which are used with the endoscope for an endoscopic examination or an
endoscopic surgery, is displayed on the head mounted display apparatus. PTL 3 and
PTL 4 disclose a method in which an augmented reality technology is applied to
display information of a dental surgery in the head mounted display apparatus. In the
technology disclosed in PTL 3 and PTL 4, display of a preoperative image, a simulated
image, or the like for an operated part is superimposed on an actual image of an
operated part.
Citation List
Patent Literature
[0004] PTL 1: JP-A-2006- 102495
PTL 2: JP-A-2001- 104331
PTL 3: JP-A-2010-259497
PTL 4: JP-A-20 13-34764
Summary of Invention
Technical Problem
[0005] Progresses in Medicine and electronic devices have brought about rapid progress in
current medical locations. In the medical locations, various apparatuses for medical use
(hereinafter, referred to as "medical apparatuses") such as an electrocardiogram, a heart
rate meter, a blood pressure meter, a pulse oximeter, a blood glucose meter, an X-ray,
and an endoscope are used. It is a great burden for those in the medical locations to un
derstand information of the various medical apparatuses and to perform tasks. For this
reason, there has been a demand for a system in which all the information of the
various medical apparatuses can be unified. In the technology disclosed in PTL 1, in
formation can be acquired from a plurality of information sources (medical ap
paratuses), but there is no mention of how the acquired information is displayed on a
plurality of the head mounted display apparatuses. In the technologies disclosed in
PTL 2 to PTL 4, targeted medical apparatuses are limited, and thus there is a problem
in that the information of the various medical apparatuses cannot be unified.
[0006] The problem is not limited to the head mounted display apparatus, but is common to
all information processing apparatuses that generate information to be displayed on an
image display apparatus such as the head mounted display apparatus. As such, there is
a demand for an information processing apparatus that can unify the information of the
various medical apparatuses, and that can control a display of the information on a
plurality of image display apparatuses. In addition, there are various demands for the
information processing apparatus as follows: a versatility improvement, a convenience
improvement, a manufacturing cost reduction, and a reliability improvement.
Solution to Problem
[0007] An advantage of some aspects of the invention is to solve at least a part of the
problems described above, and the invention can be implemented as the following
aspects.
[0008] (1) An aspect of the invention provides an information processing apparatus. The in
formation processing apparatus includes an acquisition unit that acquires plural pieces
of medical apparatus information which are plural pieces of information acquired from
a plurality of medical apparatuses; a generation unit that generates plural pieces of pre
sentation information which respectively contain at least a part of the plural pieces of
medical apparatus information acquired by the acquisition unit; and a presentation unit
that outputs the generated presentation information to a plurality of image display ap
paratuses. In the information processing apparatus according to the aspect, the
generation unit generates the presentation information using the plural pieces of
medical apparatus information acquired by the acquisition unit, and thus the in
formation (the medical apparatus information) of various medical apparatuses which
are connected to the information processing apparatus can be unified. The presentation
unit outputs the generated presentation information to the plurality of image display
apparatuses, and thus the generation unit can control display on the plurality of image
display apparatuses. As a result, it is possible to unify the information of various
medical apparatuses, and it is possible to realize the information processing apparatus
that can control the display on the plurality of image display apparatuses.
[0009] (2) The information processing apparatus according to the aspect described above
may be configured such that the generation unit has a first mode in which the pre
sentation information common to the plurality of image display apparatuses is
generated, and a second mode in which the plural pieces of presentation information
are generated to respectively correspond to the plurality of image display apparatuses.
In the information processing apparatus according to this aspect, the generation unit
has the first mode in which the presentation information common to the plurality of
image display apparatuses is generated using the unified medical apparatus in
formation, and the second mode in which the plural pieces of presentation information
are generated to respectively correspond to the plurality of image display apparatuses.
The generation unit uses the modes separately, and thus the information processing
apparatus can control the display on the plurality of image display apparatuses.
[0010] (3) The information processing apparatus according to the aspect described above
may be configured such that, in the first mode, the generation unit generates the
common presentation information that contains at least the medical apparatus in
formation selected by a user of a specific image display apparatus. In the information
processing apparatus according to this aspect, the generation unit generates the
common presentation information that contains at least the medical apparatus in
formation selected by the user of the specific image display apparatus, and the pre
sentation unit outputs the generated common presentation information to the plurality
of image display apparatuses. As a result, the information processing apparatus can
control display on the plurality of image display apparatuses based on the selection of
the user of the specific image display apparatus.
[001 1] (4) The information processing apparatus according to the aspect described above
may be configured such that the information processing apparatus further includes an
inter-apparatus relationship storage unit that stores information for individually
identifying the plurality of image display apparatuses in association with information
for indicating whether the image display apparatus is the specific image display
apparatus, and in the first mode, the generation unit identifies the specific image
display apparatus using the inter-apparatus relationship storage unit. In the information
processing apparatus according to this aspect, the generation unit can simply identify
the specific image display apparatus from the plurality of image display apparatuses
using the inter-apparatus relationship storage unit.
[0012] (5) The information processing apparatus according to the aspect described above
may be configured such that, in the second mode, the generation unit generates the
plural pieces of presentation information that respectively contain at least the pieces of
acquired medical apparatus information which are respectively selected by users of the
plurality of image display apparatuses. In the information processing apparatus
according to this aspect, the generation unit generates the plural pieces of presentation
information that respectively contain at least the pieces of medical apparatus in
formation which are respectively selected by the users of the plurality of image display
apparatuses. The presentation unit outputs the plural pieces of generated presentation
information to the plurality of image display apparatuses. As a result, the information
processing apparatus can individually control display on the plurality of image display
apparatuses based on a selection of each user of the image display apparatus.
[0013] (6) The information processing apparatus according to the aspect described above
may be configured such that, in the second mode, the generation unit generates the
plural pieces of presentation information that respectively contain at least the pieces of
medical apparatus information acquired from the medical apparatuses based on the role
of each user. In the information processing apparatus according to this aspect, the
generation unit generates the plural pieces of presentation information that respectively
contain at least the pieces of medical apparatus information which are acquired from
the medical apparatuses based on the role of each user. The presentation unit outputs
the plural pieces of generated presentation information to the plurality of image display
apparatuses. As a result, the information processing apparatus can individually control
display on the plurality of image display apparatuses based on the role of each user of
the image display apparatus, and it is possible to improve convenience of each user of
the image display apparatus.
[0014] (7) The information processing apparatus according to the aspect described above
may be configured such that the information processing apparatus further includes a
role storage unit that stores the information for individually identifying the plurality of
image display apparatuses in association with a role of a user of the image display
apparatus, and in the second mode, the generation unit identifies the role of the user
using the role storage unit. In the information processing apparatus according to this
aspect, the generation unit can simply identify a role of each user using the role storage
unit.
[0015] (8) The information processing apparatus according to the aspect described above
may be configured such that, in the second mode, the generation unit further generates
combined information in which the plural pieces of generated presentation information
are combined together, and the presentation unit outputs the combined information that
is generated for the specific image display apparatus. In the information processing
apparatus according to this aspect, the generation unit generates the combined in
formation in which the plural pieces of generated presentation information are
combined together, and the presentation unit outputs the combined information that is
generated for the specific image display apparatus. As a result, the user of the specific
image display apparatus can understand what presentation information is displayed on
the plurality of image display apparatuses connected to the information processing
apparatus. As a result, it is possible to improve convenience of the user of the specific
image display apparatus.
[0016] (9) The information processing apparatus according to the aspect described above
may be configured such that, when the generation unit generates the presentation in
formation, the generation unit enlarges or reduce an image while maintaining an aspect
ratio of the image that is contained in the medical apparatus information. In the in
formation processing apparatus according to this aspect, the generation unit enlarges or
reduces the image while maintaining the aspect ratio of the image that is contained in
the medical apparatus information, and thus it is possible to suppress unexpected
distortion of the medical apparatus information acquired by the medical apparatus
when the presentation information is generated.
[0017] (10) The information processing apparatus according to the aspect described above
may be configured such that the information processing apparatus is connected to a
head mounted display apparatus as the image display apparatus, on which a user can
see a virtual image and an outside scene. In the information processing apparatus
according to this aspect, it is possible to unify the information of various medical ap
paratuses, and it is possible to realize the information processing apparatus that can
control display on the plurality of head mounted display apparatuses.
[0018] (11) The information processing apparatus according to the aspect described above
may be configured such that the presentation unit further acquires from the head
mounted display apparatus an image in a direction of a visual field of a user that wears
the head mounted display apparatus, the presentation unit further extracts char ac
teristics of an object contained in the outside scene by recognizing the image acquired
in the direction of the visual field, and the presentation unit further compensates the
presentation information generated by the generation unit based on the extracted char
acteristics, and outputs the compensated presentation information. In the information
processing apparatus according to this aspect, the presentation unit extracts the char ac
teristics of the object contained in the outer scene by recognizing the image in the
direction of the visual field of the user of the head mounted display apparatus, and the
presentation unit compensates the generated presentation information based on the
extracted characteristics. For this reason, when the presentation information contains
invisible part information such as a CT image, an MRI image, or an X-ray image, it is
possible to align the invisible part information with the object (for example, the body
of a patient) contained in the outside scene. As a result, it is possible to improve con
venience of the user of the head mounted display apparatus. Furthermore, since the
head mounted display apparatus can receive the compensated presentation information,
the head mounted display apparatus may display the received presentation information
as it is. As a result, it is possible to simplify a configuration of the head mounted
display apparatus.
[0019] (12) The information processing apparatus according to the aspect described above
may be configured such that the information processing apparatus further includes a
step management unit that transmits information on a predetermined step to the
generation unit, and that changes the step to be transmitted based on the plural pieces
of medical apparatus information acquired by the acquisition unit, and the generation
unit generates the presentation information that contains the received step. In the in
formation processing apparatus according to this aspect, the step management unit
transmits the information on the predetermined step to the generation unit, and the
generation unit generates the presentation information that contains the received step.
For this reason, the information processing apparatus can display the predetermined
step on the plurality of image display apparatuses connected to the information
processing apparatus. The step management unit changes the step to be transmitted
based on the plural pieces of medical apparatus information that are acquired by the ac
quisition unit. For this reason, the information processing apparatus can change the
step to be displayed on the plurality of image display apparatuses based on the plural
pieces of medical apparatus information that are acquired by the plurality of medical
apparatuses. As a result, it is possible to provide the information processing apparatus
that can flexibly change the predetermined step based on the medical apparatus in
formation, and can guide each user of the image display apparatus through the step.
[0020] (13) The information processing apparatus according to the aspect described above
may be configured such that the step management unit repeatedly acquires the plural
pieces of medical apparatus information, and changes the step to be transmitted to the
generation unit based on the plural pieces of latest medical apparatus information. In
the information processing apparatus according to this aspect, the step management
unit repeatedly acquires the plural pieces of medical apparatus information, and
changes the step to be transmitted to the generation unit based on the plural pieces of
latest medical apparatus information. For this reason, the information processing
apparatus can change the step to be displayed on the plurality of image display apparatuses
in real time based on the plural pieces of medical apparatus information that
are acquired from the plurality of medical apparatuses.
[0021] (14) The information processing apparatus according to the aspect described above
may be configured such that the step management unit transmits to the generation unit
information on a series of predetermined steps which are formed of a plurality of steps,
and the step management unit changes the steps by performing at least one of the
following procedures: addition of a new step to the series of steps, removal of at least a
part of steps from the series of steps, and replacement of at least a part of steps in the
series of steps. In the information processing apparatus according to this aspect, the
step management unit may change the steps by performing at least one of the following
procedures: addition of a new step to the series of steps, removal of at least a part of
steps from the series of steps, and replacement of at least a part of steps in the series of
steps.
[0022] (15) The information processing apparatus according to this aspect described above
may be configured such that the step management unit sequentially transmits to the
generation unit the series of predetermined steps one by one, which are formed of the
plurality of steps and the step management unit changes the steps by stopping the
transmission of the single step. In the information processing apparatus according to
this aspect, the step management unit can change the steps by stopping the
transmission of the step.
[0023] (16) The information processing apparatus according to the aspect described above
may be configured such that, in the first mode, the step management unit transmits the
steps for the user of the specific image display apparatus to the generation unit. In the
information processing apparatus according to this aspect, in the first mode, the step
management unit transmits the steps for the user of the specific image display
apparatus to the generation unit. For this reason, the information processing apparatus
can display common "steps for the user of the specific image display apparatus" on the
plurality of image display apparatuses.
[0024] (17) The information processing apparatus according to the aspect described above
may be configured such that, in the second mode, the step management unit transmits
to the generation unit the steps that respectively correspond to the roles of the users. In
the information processing apparatus according to this aspect, in the second mode, the
step management unit transmits to the generation unit the steps that respectively
correspond to the roles of the users. For this reason, the information processing
apparatus can individually display the "steps which respectively correspond to the roles
of the users of the image display apparatuses" on the plurality of image display ap
paratuses.
[0025] (18) Another aspect of the invention provides an information processing system. The
information processing system includes a plurality of medical apparatuses; an in
formation processing apparatus; and a plurality of head mounted display apparatuses,
on each of which a user can see a virtual image and an outside scene. The information
processing apparatus includes an acquisition unit that acquires plural pieces of medical
apparatus information that are acquired from the plurality of medical apparatuses; a
generation unit that generates presentation information which contains at least a part of
the plural pieces of medical apparatus information acquired by the acquisition unit; and
a presentation unit that outputs the generated presentation information to the plurality
of head mounted display apparatuses. The head mounted display apparatus includes a
presentation information acquisition unit that acquires the presentation information
from the information processing apparatus; and an image display unit on which a user
of the head mounted display apparatus sees the acquired presentation information as
the virtual image. In the information processing system according to the aspect, it is
possible to unify information of various medical apparatus (medical apparatus in
formation) using the information processing apparatus, and it is possible to realize the
information processing system in which display on the plurality of head mounted
display apparatuses can be controlled.
[0026] Not the entirety of a plurality of configuration elements of each aspect of the
invention is essential. It is possible to appropriately change, remove, replace a part of
the plurality of configuration elements with new configuration elements, or to remove
a part of limited content so as to solve a part or the entirety of the problems, or to
achieve a part or the entirety of the advantages described in this specification. A part or
the entirety of technical characteristics contained in one aspect of the invention can be
combined with a part or the entirety of technical characteristics contained in the other
aspects of the invention, or can be an independent aspect of the invention so as to solve
a part or the entirety of the problems, or to achieve a part or the entirety of the ad
vantages described in this specification.
[0027] For example, an aspect of the invention can be realized as an apparatus that includes
a part or the entirety of three configuration elements which are the acquisition unit, the
generation unit, and the presentation unit. That is, the apparatus may or may not
include the acquisition unit. The apparatus may or may not include the generation unit.
The apparatus may or may not include the presentation unit. For example, the
apparatus can be realized as the information processing apparatus, but the apparatus
can also be realized as apparatuses other than the information processing apparatus. A
part or the entirety of the technical characteristics of each aspect of the information
processing apparatus can be applied to any apparatus.
[0028] The invention can be realized in various forms. For example, it is possible to realize
the invention in the following forms: the information processing apparatus; the information
processing method; the information processing system; the head mounted
display apparatus; a method of controlling the head mounted display apparatus; a head
mounted display system; a computer program for the realization of functions of the
methods, the apparatuses, and the systems, a record medium on which the computer
program is recorded, and the like.
Brief Description of Drawings
[fig. 1]Fig. 1 is a diagram illustrating a schematic configuration of an information
processing system according to an embodiment of the invention.
[fig.2]Fig. 2 is a block diagram illustrating a functional configuration of an information
processing apparatus.
[fig.3]Fig. 3 is a table illustrating an example of a common video specification.
[fig.4]Fig. 4 is a table illustrating an example of a management table.
[fig.5]Fig. 5 is a block diagram illustrating a functional configuration of a head
mounted display.
[fig.6]Fig. 6 is a flow chart illustrating procedures of presentation information
generating process.
[fig.7A]Fig. 7A illustrates a view describing a method of generating the presentation
information.
[fig.7B]Fig. 7B illustrates a view describing a method of generating the presentation
information.
[fig.8]Fig. 8 is a view describing the method of generating the presentation in
formation.
[fig.9A]Fig. 9A illustrates a view showing a first example of the presentation in
formation.
[fig.9B]Fig. 9B illustrates a view showing a first example of the presentation in
formation.
[fig.l0A]Fig. 10A illustrates a view showing a second example of the presentation in
formation.
[fig.l0B]Fig. 10B illustrates a view showing a second example of the presentation in
formation.
[fig. 1l]Fig. 11 is a view illustrating a third example of the presentation information.
[fig.l2]Fig. 12 is a view illustrating a state in which the presentation information is
displayed as a virtual image on a plurality of head mounted displays.
[fig.l3]Fig. 13 is a view illustrating a state in which the presentation information is
displayed as a virtual image on the plurality of head mounted displays that are p o
sitioned in different places.
[fig.l4]Fig. 14 is a view illustrating a state in which the presentation information is
displayed as a virtual image on the plurality of head mounted displays after steps S122
to S134 are executed two times.
[fig.l5]Fig. 15 is a block diagram illustrating a functional configuration of an in
formation processing apparatus according to Second Embodiment.
[fig.l6]Fig. 16 is a flow chart illustrating procedures of combined information
generating process.
[fig.l7]Fig. 17 is a view describing a method of generating the combined information.
[fig.l8]Fig. 18 is a view illustrating a state in which the combined information is
displayed as a virtual image on the head mounted display.
[fig.l9]Fig. 19 is a block diagram illustrating a functional configuration of an in
formation processing apparatus according to Third Embodiment.
[fig.20]Fig. 20 is a schematic diagram illustrating an example of step information.
[fig.21]Fig. 2 1 is a table illustrating an example of emergency response information.
[fig.22]Fig. 22 is a flow chart illustrating procedures of a step management process.
[fig.23A]Fig. 23A illustrates a view showing an external configuration of each head
mounted display in modification examples.
[fig.23B]Fig. 23B illustrates a view showing an external configuration of each head
mounted display in modification examples.
Description of Embodiments
[0030] A. First Embodiment:
A-l. Configuration of Information Processing System:
Fig. 1 is a diagram illustrating a schematic configuration of an information
processing system 1000 according to an embodiment of the invention. The information
processing system 1000 includes a plurality of medical apparatuses 501 to 50n, an in
formation processing apparatus 300, and an image display apparatus 100. The in
formation processing system 1000 is a system used in a medical location. The in
formation processing apparatus 300 acquires information from the plurality of medical
apparatuses 501 to 50n, unifies the information, and generates presentation information
to be displayed on the image display apparatus 100. The medical apparatuses 501 to
50n and the image display apparatus 100 are connected to the information processing
apparatus 300 by a wired connection or a wireless connection. Hereinafter, "n" is used
as a letter indicating an "arbitrary plural" n is an integer equal to or greater than 1.
[0031] In the embodiment, an operation site in an operation room is illustrated as the
medical location. However, the information processing system 1000 may be used in a
consulting room, a treatment room, a curing room, a delivery room, and the like. In the
embodiment, a head mounted display apparatus is illustrated as the image display
apparatus 100. However, the image display apparatus 100 may be a stationary display,
a portable display, a smart phone, and the like. Hereinafter, the head mounted display
apparatus 100 is also referred to as the "head mounted display (HMD) 100". In Fig. 1,
a single head mounted display 100 is illustrated. However, as illustrated in Fig. 2 to be
described below, a plurality of head mounted displays 100a to 100h may be connected
to the information processing apparatus 300.
[0032] In the following description, when the head mounted displays 100a to 100h are not
particularly specified and are generally described, the head mounted displays 100a to
100h are simply referred to as the "head mounted display 100". Similarly, when the
medical apparatuses 501 to 50n are not particularly specified and are generally
described, the medical apparatuses 501 to 50n are simply referred to as a "medical
apparatus 500".
[0033] The medical apparatuses 501 to 50n are various apparatuses for a medical use. For
example, the following can be used as the medical apparatuses 501 to 50n: a visible
light camera for photographing indoor and outdoor still images and indoor and outdoor
moving images, a visible light camera for photographing still and moving images of an
operated part, an electrocardiogram examination apparatus, a timer for surgery, a b i
ological monitor, a heart rate meter, a pulse rate meter, a blood pressure meter, an u l
trasonic examination apparatus, a pulse oximeter, a blood glucose meter, a CT
(Computed Tomography) image photographing apparatus, an MRI (Magnetic
Resonance Imaging) image photographing apparatus, an X-ray image photographing
apparatus, a fluoroscopic image photographing apparatus, an endoscopic image pho
tographing apparatus, a thermography image photographing apparatus, and the like.
[0034] A-2. Configuration of Information Processing Apparatus:
Fig. 2 is a block diagram illustrating a functional configuration of the information
processing apparatus 300. The information processing apparatus 300 includes a first
input/output unit 310, a ROM 320, a RAM 330, a communication interface 340, a CPU
350, and an external storage apparatus 380. The units are connected to each other via a
bus.
[0035] The first input/output unit 310 as an acquisition unit includes a plurality of first
input/output units 3 11 to 3In. For example, the first input/output units 311 to 3In are
the following input/output interfaces conforming to a communication standard: a USB
(Universal Serial Bus), an HDMI (High Definition Multimedia Interface), a DVI
(Digital Visual Interface), a VGA (Video Graphics Array), a composite, an RS-232C
(Recommended Standard 232), an infrared ray, a short-range wireless communication
(for example, Bluetooth (registered trademark)), a wireless LAN (for example,
IEEE802.1 1), a wired LAN (for example, IEEE802.3), and the like. The first input/
output units 311 to 3In respectively acquire information input from the medical ap
paratuses 501 to 50n that are respectively connected thereto. The information input
from the medical apparatus is information measured by or photographed by the
medical apparatus. Hereinafter, the information is also referred to as "medical
apparatus information".
[0036] In the embodiment, the first input/output unit 3 11 is an input/output interface
conforming to the DVI. The first input/output unit 312 is an input/output interface
conforming to the HDMI. The first input/output unit 313 is an input/output interface
conforming to the HDMI. The first input/output unit 3In is an input/output interface
conforming to the wireless LAN. In Fig. 2, the medical apparatus 501 and a stationary
display 700 are connected to the first input/output unit 311. Similarly, the medical
apparatus 502 is connected to the first input/output unit 312, the medical apparatus 503
is connected to the first input/output unit 313, and the medical apparatus 50n is
connected to the first input/output unit 3In. In this way, the first input/output unit 310
is configured to have the first input/output units 3 11 to 3 1n conforming to the different
types of communication standards. For this reason, it is possible to connect the medical
apparatuses conforming to various communication standards and the first input/output
unit 310, and it is possible to improve versatility of the information processing
apparatus 300. The first input/output unit 310 can also output to the display apparatus
such as the display 700 an input signal as it is.
[0037] The communication interface (I/F) 340 is a wired LAN interface. The commu
nication interface 340 communicatively connects a doctor PR in an external facility
and the information processing apparatus 300 via an Internet INT line. The commu
nication interface 340 communicatively connects a database OD connected to an inhospital
LAN and the information processing apparatus 300 via an intranet. For
example, the doctor PR is assumed to be a doctor who supervises a user of the head
mounted display 100 that is connected to the information processing apparatus 300, or
is assumed to be an outside expert. The database OD stores procedure information il
lustrating a procedure of a treatment such as an operation, an electronic medical record
system, electronic academic books, and the like.
[0038] The CPU 350 deploys and executes a computer program stored in the ROM 320 on
the RAM 330 to thereby control the information processing apparatus 300. The CPU
350 realizes each function of a generation unit 360 and a second input/output unit 370.
The generation unit 360 generates presentation information to be displayed on the head
mounted display 100 by executing a presentation information generating process to be
described below. The generation unit 360 of the embodiment has the following two
modes as processing modes in the presentation information generating process.
(1) First mode in which common presentation information is generated
(2) Second mode in which plural pieces of presentation information are generated to
respectively correspond to the head mounted displays 100a to 100h
[0039] The second input/output unit 370 as a presentation unit includes a plurality of second
input/output units 371 to 37n. Similarly to the first input/output unit 310, the second
input/output units 37 1 to 37n are input/output interfaces conforming to various com
munication standards. Since the detail is the same as that of the first input/output unit
310, the descriptions thereof will be omitted.
[0040] In the embodiment, the second input/output unit 37 1 and the second input/output unit
372 are input/output interfaces conforming to the wireless LAN. The second input/
output unit 373 is an input/output interface conforming to the USB. The second input/
output unit 37n is an input/output interface conforming to the DVI. In Fig. 2, the head
mounted display 100a as the image display apparatus is connected to the second input/
output unit 371. Similarly, the head mounted display 100b is connected to the second
input/output unit 372, the head mounted display 100c is connected to the second input/
output unit 373, and the head mounted display 100h is connected to the second input/
output unit 37n. In this way, the second input/output unit 370 is configured to have the
second input/output units 37 1 to 37n conforming to the different types of commu
nication standards. For this reason, it is possible to connect the head mounted displays
conforming to various communication standards and the second input/output unit 370,
and it is possible to improve the versatility of the information processing apparatus
300.
[0041] The external storage apparatus 380 is configured to have a ROM, a RAM, a DRAM,
a hard disk, a flash memory, and the like which are not illustrated. The external storage
apparatus 380 stores a common video specification 381, a management table 382,
invisible part information 383 as an invisible part information storage unit, an
operation record 384, and an operation mode 385.
[0042] Fig. 3 is a table illustrating an example of the common video specification 381. The
common video specification 381 is a table that defines a standard specification
concerning a video signal of the presentation information. The common video speci
fication 381 is used when the generation unit 360 generates the presentation in
formation in the presentation information generating process.
[0043] The common video specification 381 of the embodiment includes the number of
pixels, an aspect ratio, and a resolution of an image contained in the video signal, an
image compression method, and an image encoding method as the specification of the
video signal. The common video specification 381 is determined in advance, and is
stored in the external storage apparatus 380. When the common video specification
381 is determined, the video signal specifications, which are adopted by the head
mounted displays 100a to 100h connected to the second input/output unit 370, are
preferably taken into consideration.
[0044] Fig. 4 is a table illustrating an example of the management table 382. The
management table 382 as a role storage unit and an inter-apparatus relationship storage
unit is a table for managing the information of the head mounted display 100
connected to the second input/output unit 370, more specifically, for managing the in
formation of the head mounted displays 100a to 100h that are respectively connected
to the second input/output units 371 to 37n. The management table 382 is used when
the generation unit 360 generates the presentation information in the presentation in
formation generating process, and is also used when the second input/output units 371
to 37n transmit the presentation information to the head mounted displays 100a to
100h, respectively.
[0045] An identifier, an inter-apparatus relationship, a role, and the video specification are
stored in the management table 382 of the embodiment in association with each other.
The "identifier" stores each unique identifier for individually identifying the head
mounted displays 100 connected to the second input/output units 370. The "interapparatus
relationship" stores a string of letters indicating master/slave relationships
between the plurality of head mounted displays 100a to 100h which are connected to
the second input/output units 370. In the embodiment, one head mounted display 100
is set to be a master, and the other head mounted displays 100 are set to be slaves. The
"role" stores a string of letters indicating a role that the user of the head mounted
display 100 identified by the identifier takes in a medical location. The "video speci
fication" stores a string of letters indicating a pattern of the specifications (the number
of pixels, the aspect ratio, the resolution of the image contained in the video signal, the
image compression method, the image encoding method, and the like) of the video
signal in the head mounted display 100 that is identified by the identifier.
[0046] In the example illustrated in Fig. 4, the user of the head mounted display 100
identified by an identifier "1" is an "operating surgeon", and the video specification of
the head mounted display 100 is a "specification A" (an entry E01). Similarly, the user
of the head mounted display 100 identified by an identifier "n" is a "nurse" and the
video specification of the head mounted display 100 is a "specification E" (an entry
EOn). In the example illustrated in Fig. 4, the head mounted display 100 identified by
the identifier "1" is a master, and the head mounted displays 100 identified by the other
identifiers are slaves (entries E01 to EOn).
[0047] The management table 382 may be determined in advance and be stored in the
external storage apparatus 380. Alternatively, the management table 382 may be auto
matically generated in an initial process when the head mounted display 100 is
connected to the information processing apparatus 300. Specifically, for example,
when the generation unit 360 detects the connection of the head mounted display 100
and the second input/output unit 370, the generation unit 360 gives a unique identifier
to the head mounted display 100, and acquires a designation of master/slave, a role of
the user, and a video specification from the head mounted display 100. The generation
unit 360 stores the given identifier, the designation of master/slave, the acquired role,
and the acquired video specification in the management table 382. The initial process
is performed whenever the head mounted displays 100a to 100h are respectively
connected to the second input/output units 370, and thus the generation unit 360 can
automatically generate the management table 382.
[0048] As described above, the "inter-apparatus relationship" in the management table 382
is acquired from the head mounted display 100. In addition, for example, the "interapparatus
relationship" may be determined by the following methods.
* The "inter-apparatus relationship" is determined based on the role that is acquired
from the head mounted display 100. For example, the generation unit 360 sets the head
mounted display 100 of the operating surgeon to a master, and sets the head mounted
displays 100 of users other than the operating surgeon to slaves.
* The "inter-apparatus relationship" is determined based on acquisition of current
position information of the head mounted display 100 and a positional relationship
between the head mounted displays. For example, the generation unit 360 acquires the
current position information of the head mounted display 100. The generation unit 360
sets the head mounted display 100 positioned closest to a center of an operating table
to a master, and sets other head mounted displays 100 to slaves.
* The "inter-apparatus relationship" is determined based on differences of hardware
and software configurations of the head mounted display 100. For example, the
generation unit 360 acquires a model number of the head mounted display 100. The
generation unit 360 sets the head mounted display 100 with a specific model number to
a master, and sets other head mounted displays 100 to slaves.
[0049] As described above, the "role" in the management table 382 is acquired from the
head mounted display 100. In addition, for example, the "role" may be determined by
the following methods.
* The "role" is determined based on current position information of the head
mounted display 100. For example, the external storage apparatus 380 of the in
formation processing apparatus 300 pre-stores layout information that contains layout
of equipment (the operating table and the medical apparatus 500) in the operation
room, and layout of standing positions of staffs based on the roles in the operation
room. The generation unit 360 acquires the current position information of the head
mounted display 100, and compares the acquired current position information with the
layout information, and thus the role of the user of the head mounted display 100 is de
termined.
* The "role" is determined based on an image (an outside scene image) in a direction
of a visual field of the user of the head mounted display 100. The generation unit 360
acquires an outside scene image from the head mounted display 100, and recognizes
the outside scene image. Accordingly, the generation unit 360 specifies the type of the
medical apparatus 500, the type of the equipment in the operation room, a part of the
patient body, and the like which the user sees. The generation unit 360 determines the
role of the user of the head mounted display 100 based on the specified information.
[0050] The invisible part information 383 (the invisible part information storage unit) i l
lustrated in Fig. 2 stores invisible part information that does not appear in an external
appearance of an object. The invisible part information is image information or letter
information illustrating a structure of the object. When the object is a living body, the
invisible part information is, for example, a CT image, an MRI image, an X-ray image,
a fluoroscopic image, an ultrasonic diagnosis image, an endoscopic image, or a ther
mographic image, and letter information incident to the image. The invisible part in
formation is photographed in advance, and is stored in the invisible part information
383.
[0051] The operation record 384 illustrated in Fig. 2 is a recording unit that records the
medical apparatus information acquired by the information processing apparatus 300
and the presentation information generated by the information processing apparatus
300 in association with a date and time when the information is acquired or generated.
The medical apparatus information acquired by the information processing apparatus
300 indicates the medical apparatus information that the information processing
apparatus 300 acquires from the plurality of medical apparatuses 501 to 50n which are
respectively connected to the first input/output units 3 11 to 3In. The presentation in
formation generated by the information processing apparatus 300 indicates the entire
presentation information that the generation unit 360 generates in the presentation in
formation generating process. The record of the information acquired and generated by
the information processing apparatus 300 can be regarded to be the substantially same
as a log of an operation (a diagnosis, a treatment, or a curing). Accordingly, in this
manner, after the operation or the like is ended, it is possible to use the log of the
operation in the analysis of content of the operation, training and academic purpose,
and the like.
[0052] The operation mode 385 illustrated in Fig. 2 stores a processing mode (that is, an
operation mode of the generation unit 360) in the presentation information generating
process of the generation unit 360. Content of the operation mode 385 can be changed
by an input device which the information processing apparatus 300 includes and which
is not illustrated. The content of the operation mode 385 can be also changed by a
change instruction from the head mounted display 100.
[0053] A-3. Configuration of Head Mounted Display Apparatus:
Fig. 5 is a block diagram illustrating a functional configuration of the head mounted
display 100a. Hereinafter, the head mounted display 100a is illustrated and described
as an example of a configuration of the head mounted display 100. The head mounted
display 100a is an optical transmission type head mounted display apparatus on which
a user can see a virtual image and concurrently, can directly see an outside scene. The
presentation information acquired from the information processing apparatus 300 is
displayed as a virtual image on the head mounted display 100a. The head mounted
display 100a includes an image display unit 20 on which the user sees the virtual
image in a state where the user wears the head mounted display 100a on the head, and
a control unit 10 that controls the image display unit 20. Various signals are
transmitted between the image display unit 20 and the control unit 10 via a cable 40.
[0054] A-3-1. Configuration of Control Unit:
The control unit 10 includes the following units: an input information acquisition unit
110 that acquires an operation input to an input device such as a touch pad, a cross key,
a foot switch, a gesture, or a microphone; a storage unit 120 configured to have a
ROM, a hard disk, and the like; a power supply 130 that supplies an electrical power to
each unit of the head mounted display 100a; a wireless communication unit 132 that
wirelessly communicates with other devices conforming to a predetermined wireless
communication standard such as the wireless LAN, Bluetooth, or the like; a current
position acquisition unit 134; a CPU 140; an interface 180 that connects the in
formation processing apparatus 300 and various external devices OA; and transmitting
units (Tx) 51 and 52. The units are connected to each other via a bus which is not i l
lustrated. The current position acquisition unit 134 detects a current position of the user
of the head mounted display 100a by receiving a signal from a GPS satellite, and
generates current position information indicating the current position of the user. The
current position acquisition unit 134 may generate the current position information by
receiving radio waves from a plurality of base stations and by determining strength of
the received radio waves. For example, the current position information can be realized
by a coordinate indicating a latitude/longitude. Similar to the first input/output unit 310
of the information processing apparatus 300, the interface 180 can conform to various
communication standards.
[0055] The CPU 140 reads out and executes a computer program stored in the storage unit
120, and thus functions as the following units: an operating system (OS) 150; an image
processing unit 160; a voice processing unit 170 that supplies voice signals to speakers
of a right earphone 32 and a left earphone 34; and a display control unit 190. The
image processing unit 160 generates a signal based on a video signal of the pre
sentation information that is input via the interface 180 as the presentation information
acquisition unit or the wireless communication unit 132. The image processing unit
160 supplies the generated signal to the image display unit 20 via the transmitting units
51 and 52 and the cable 40, and thus controls the display on the image display unit 20.
The signal supplied to the image display unit 20 can be a analog signal or a digital
signal which are widely known. The display control unit 190 generates a control signal
to control a right display driving unit 22 and a left display driving unit 24 of the image
display unit 20, and the control signal is transmitted via the transmitting units 51 and
52. The control signal is an individual signal through which a right LCD control unit
211 turns on or off the driving of a right LCD 241, through which a right backlight
control unit 201 turns on or off the driving of a right backlight 221, through which a
left LCD control unit 212 turns on or off the driving of a left LCD 242, and through
which a left backlight control unit 202 turns on or off the driving of a left backlight
222.
[0056] A-3-2. Configuration of Image Display Unit:
The image display unit 20 is a mounting body that the user wears on the head. In the
embodiment, the image display unit 20 has an eyeglass shape. The image display unit
20 includes the right display driving unit 22 and the left display driving unit 24 that
generate and emit image light for an image; a right optical image display unit 26 and a
left optical image display unit 28 (refer to Fig. 2) that guide the image light to both
eyes of the user; a camera 6 1 as the image acquisition unit, and a 9-axis sensor 66.
[0057] The right display driving unit 22 includes a receiving unit (Rx) 53; the right backlight
(BL) control unit 201 and the right backlight (BL) 221 which function as light sources;
the right LCD (Liquid Crystal Display) control unit 211 and the right LCD 241 which
function as a display element; and a right projection optical system 251. The right
backlight control unit 201 drives the right backlight 221 based on a control signal that
is input via the receiving unit 53. The right backlight 221 is a light emitting body such
as an LED or an electroluminescence (EL). The right LCD control unit 211 drives the
right LCD 241 based on an input video signal. The right LCD 241 is a transmissive
liquid crystal panel on which a plurality of pixels are arranged in a matrix shape. The
right LCD 241 drives liquid crystal at a position of each of the pixels that are arranged
in a matrix shape. Accordingly, the right LCD 241 changes transmittance of light that
transmits through the right LCD 241, and modulates illumination light irradiated from
the right backlight 221 to effective image light for the image. The right projection
optical system 251 is configured to have a collimate lens that converts the image light
emitted from the right LCD 241 into parallel light fluxes. The left display driving unit
24 has the same configuration as that of the right display driving unit 22 and operates
similar to the right display driving unit 22.
[0058] The right optical image display unit 26 and the left optical image display unit 28 are
arranged to be positioned in front of the right and left eyes of the user when the user
wears the image display unit 20 (refer to Fig. 2). The right optical image display unit
26 includes a right light guide plate 261 formed of a light transmissive resin material or
the like. The right light guide plate 261 reflects the image light along a predetermined
optical path to guide the image light output from the right display driving unit 22 to a
right eye RE of the user. A diffraction grating or a semi-transmissive reflective film
may be used in the right light guide plate 261. The left optical image display unit 28
has the same configuration as that of the right optical image display unit 26, and
operates similar to the right optical image display unit 26. In this way, the image light
guided to both eyes of the user of the head mounted display 100a is focused on the
retinas of the user, and thus the user can see the virtual image. The user of the head
mounted display 100a of the embodiment can concurrently see the virtual image and
the outside scene in the background of the virtual image in a portion of the visual field
where the virtual image is displayed. In a portion of the visual field where the virtual
image is not displayed, the user can directly see the outside scene through the right
optical image display unit 26 and the left optical image display unit 28.
[0059] The 9-axis sensor 66 is a motion sensor that detects acceleration (3 axes), angular ve
locities (3 axes), and terrestrial magnetism (3 axes). When the user wears the image
display unit 20 on the head, the 9-axis sensor 66 functions as a movement detecting
unit that detects a movement of the head of the user of the head mounted display 100a.
Here, the movement of the head includes changes in velocity, acceleration, angular
velocity, direction, and orientation of the head. The orientation of the head is also
referred to as "orientation information". The camera 6 1 as a visual field image ac
quisition unit is arranged at a position that corresponds to the middle of the forehead of
the user when the user wears the image display unit 20. The camera 6 1 photographs an
image of the outside scene in a front direction of the image display unit 20, in other
words, in the direction of the visual field of the user in a state where the user wears the
head mounted display 100a, and the camera 6 1 acquires the outside scene image. The
camera 6 1 is a so-called visible light camera. The camera 6 1 of the embodiment is a
monocular camera, but a stereo camera may be adopted.
[0060] A-4. Presentation Information Generating Process:
Fig. 6 is a flow chart illustrating procedures of the presentation information
generating process. The presentation information generating process is a process of
generating the presentation information to be displayed on the head mounted display
100, and is executed by the generation unit 360 of the information processing
apparatus 300 (refer to Fig. 2). The presentation information generating process is
divided into the first mode in which the common presentation information is generated,
and the second mode in which the plural pieces of presentation information are
generated to respectively correspond to the head mounted displays 100a to 100h. The
process executed in the second mode can be roughly divided into an initial process of
steps S120 to S134 and an individual process of steps S136 to S142.
[0061] The generation unit 360 confirms an operation mode of the generation unit 360 with
reference to content that is stored in the operation mode 385 (step S100).
[0062] A-4-1. First Mode:
When the operation mode is the "first mode" (step S100: the first mode), the
generation unit 360 retrieves the master head mounted display 100 from the
management table 382 (step SI 10). Specifically, the generation unit 360 retrieves an
entry, the "inter-apparatus relationship" of which is stored as a master, with reference
to the management table 382, and the generation unit 360 acquires an "identifier" of
the retrieved entry.
[0063] The generation unit 360 acquires a request for an information source selection from
the master head mounted display 100 (step SI 12). Specifically, the generation unit 360
requests the head mounted display 100 with the identifier acquired in step SI 10 to
transmit an request for the information source selection. Here, for example, the request
for an information source selection indicates any one of requests listed in the following
al to a6.
(al) A request for reading the medical apparatus information acquired by the medical
apparatus 500
(a2) A request for placing a telephone call to the doctor PR in the external facility
(a3) A request for reading data recorded in the in-hospital database OD
(a4) A request for reading data recorded in the invisible part information 383
(a5) A request for reading data recorded in the operation record 384
(a6) A request for reading the medical apparatus information of the medical
apparatus 500, the display unit of which the user of the head mounted display 100 can
hardly see
[0064] The information source in the request al is the medical apparatus 500. Similarly, the
information source in the request a2 is the doctor PR; the information source in the
request a3 is the database OD; the information source in the request a4 is the invisible
part information 383; the information source in the request a5 is the operation record
384; and the information source in the request a6 is the medical apparatus 500, the
display unit of which the user can hardly see. Here, for example, "the medical
apparatus 500, the display unit of which the user can hardly see" is a general term of
the following medical apparatus 500.
* The medical apparatus 500, the display unit (a display or the like) of which the user
cannot see or can hardly see because the medical apparatus 500 is present at a blind
spot of the user.
* The medical apparatus 500, the display unit of which the user cannot see or can
hardly see because the medical apparatus 500 is present in a place remote from the
user.
* The medical apparatus 500, the display unit of which the user cannot see or can
hardly see because the medical apparatus 500 is not present at a blind spot of the user
and is present in a place close to the user, but a visual line of the user faces the display
unit.
[0065] The requests al to a6 may be individually issued, and may be combined together.
However, in the embodiment, a request for information source acquisition includes at
least the request al. The request for an information source selection is appropriately
issued based on an operation of the user of the head mounted display 100.
[0066] The head mounted display 100 can transmit the request for an information source
selection to the information processing apparatus 300, for example, as per the
following procedures b l to b5.
(bl) The user of the head mounted display 100 inputs a desired information source
and information for specifying information in the information source via the input
device (a touch pad, a cross key, a foot switch, a gesture, a microphone, or the like) of
the head mounted display 100. If a voice input via the microphone is adopted as an
input method, it is possible to considerably improve convenience of the user when it is
difficult for the user to operate the head mounted display 100 by the hands in a medical
location. In the requests al and a6, it is not necessary to input the "information for
specifying information in the information source".
(b2) The input information acquisition unit 110 of the head mounted display 100
acquires a signal based on an operation input of the user and transmits the signal to the
OS 150.
(b3) The OS 150 of the head mounted display 100 generates a request for an in
formation source selection. Specifically, in the request al, the OS 150 generates the
request for an information source selection, which includes an identifier of the head
mounted display 100 and an identifier for specifying the information source designated
in the procedure bl. In the requests a2 to a5, the OS 150 generates a request for an in
formation source selection, which includes the identifier of the head mounted display
100, the identifier for specifying the information source designated in the procedure
bl, and the information for specifying the information in the information source. In the
request a6, the OS 150 generates a request for an information source selection which
includes the identifier of the head mounted display 100, and current position in
formation and orientation information of the head mounted display 100. The current
position information and the orientation information are used to specify "the medical
apparatus 500, the display unit of which the user can hardly see" in the request a6.
(b4) The OS 150 of the head mounted display 100 transmits the request for an in
formation source selection generated in the procedure b3 to the information processing
apparatus 300 to which the OS 150 is connected via the interface 180 and the wireless
communication unit 132.
(b5) The second input/output unit 370 of the information processing apparatus 300
transmits the received request for an information source selection to the generation unit
360.
[0067] The generation unit 360 acquires the information for specifying the information
source and the information for specifying the information in the information source,
which are contained in the request for an information source selection in step SI 14 of
in the presentation information generating process (refer to Fig. 6). Furthermore, when
the current position information and the orientation information are inserted as the in
formation for specifying the information source, the generation unit 360 compares the
current position information and the orientation information with position information
acquired from the entirety of other medical apparatuses 500, and the generation unit
360 specifies "the medical apparatus 500, the display unit of which the user can hardly
see". In addition, in step SI 14, the generation unit 360 may specify "the medical
apparatus 500, the display unit of which the user can hardly see" by combining the
current position information and the orientation information with a movement (changes
in velocity, acceleration, angular velocity, direction, and orientation) of the head of the
user that is obtained by the 9-axis sensor 66 of the head mounted display 100.
[0068] The generation unit 360 accesses the designated (specified) information source to
acquire the designated information. The generation unit 360 generates the common
presentation information supplied to the entirety of the head mounted displays 100
connected to the second input/output units 370 using the acquired information. The
common presentation information is referred to as "presentation information PN*".
The generation unit 360 stores the generated presentation information PN* in the
operation record 384 (step SI 16).
[0069] Figs. 7A and 7B illustrate views describing a method of generating the presentation
information. When single information is acquired in step SI 16, the generation unit 360
generates presentation information PN by converting an image contained in acquired
information CP1 as per the video signal specification defined in the common video
specification 381 (refer to Fig. 7A). In particular, when an aspect ratio of an image
contained in the acquired information CP1 is different from the aspect ratio of an
image defined in the common video specification 381 during the conversion, the
generation unit 360 enlarges or reduces the image while maintaining the aspect ratio of
an image contained in the acquired information CP1. Then, a margin (indicated by a
hatched line in Fig. 7A) is formed in a longitudinal direction or a lateral direction of
the presentation information PN.
[0070] The generation unit 360 executes any one of the following procedures c l and c2 with
respect to the margin. Accordingly, the margin is displayed as a virtual image on the
head mounted display 100, and thus it is possible to suppress the blockage of the visual
field of the user.
(cl) The generation unit 360 inserts black dummy dot data in the margin of the pre
sentation information PN.
(c2) The generation unit 360 sets a signal (an enable signal) to a Lo value (an invalid
value) with respect the margin of the presentation information PN, and the signal is
intended to perform the switching between validity and invalidity of generation of
image light in the right display driving unit 22 and the left display driving unit 24 of
the head mounted display 100.
[0071] In this manner, the aspect ratio of an image contained in the presentation information
PN does not change. It is preferable to strictly maintain an aspect ratio of an image
photographed by the medical apparatus 500 in such a manner that the user see an
actual unspoiled size of the image (for example, a still image and a moving image, a
CT image, an MRI image, or an X-ray image of an operated part) in a medical location
where the information processing apparatus 300 is used. In this regard, if the aspect
ratio of an image which is contained in the medical apparatus information CP1
acquired in step SI 16 is maintained, and the information source is the medical
apparatus 500, it is possible to suppress unexpected distortion of the medical apparatus
information measured or photographed by the medical apparatus 500. Fig. 7B i l
lustrates presentation information PNx that is distorted because the aspect ratio of an
image contained in the acquired information CP1 is not maintained, and the image is
enlarged to the aspect ratio of an image defined in the common video specification
381.
[0072] Fig. 8 is a view describing the method of generating the presentation information.
When plural pieces of information are acquired in step SI 16, the generation unit 360
generates a first image in which the image contained in the acquired entire information
(in examples illustrated in Figs. 10A and 10B, medical apparatus information CP2 and
CP3) is arranged. At this time, the generation unit 360 maintains the aspect ratio of an
image contained in the acquired information. The image contained in the acquired in
formation is preferably arranged in an end portion of the first image. The generation
unit 360 generates the presentation information PN by converting the generated first
image as per the video signal specification defined in the common video specification
381 (refer to Fig. 8). The generation unit 360 executes any procedure of the methods
c l and c2 with respect to a margin (indicated by a hatched line in Fig. 8) in which the
image is not arranged, in the first image. Accordingly, the margin is displayed as a
virtual image on the head mounted display 100, and it is possible to suppress the
blockage of the visual field of the user.
[0073] Figs. 9A and 9B illustrate views showing a first example of the presentation in
formation. Fig. 9A is an example of the presentation information PN generated by the
generation unit 360 when an X-ray image (CP4) of a patient stored in the invisible part
information 383 is designated in the request for an information source selection. The
same presentation information as that illustrated in Fig. 9A is generated when the
medical apparatus 500 (an X-ray image photographing apparatus) from which the
invisible part information can be acquired is designated in the request for an in
formation source selection.
[0074] Fig. 9B is an example of the presentation information PN generated by the generation
unit 360 when a timer for surgery is designated as the medical apparatus 500, a pulse
rate meter is designated as the medical apparatus 500, and a biological monitor is
designated as the medical apparatus 500 in the request for an information source
selection. The following is arranged in each end portion of the presentation in
formation PN: present time CP5 acquired from the timer for surgery, time CP7 elapsed
from the start of surgery, a pulse image CP6 acquired from the pulse rate meter, and an
image CP8 of a heart rate (HR), a blood pressure (SYS/DIA), and arterial oxygen
saturation (Sp02) acquired from the biological monitor.
[0075] Figs. 10A and 10B illustrates views showing a second example of the presentation
information. Fig. 10A is an example of the presentation information PN generated by
the generation unit 360 when an academic book (CP9) stored in the in-hospital
database OD and operation procedure information (CP10) stored in the in-hospital
database OD are designated in the request for an information source selection. Fig.
10B is an example of the presentation information PN generated by the generation unit
360 when a visible light camera for photographing a still image and a moving image of
an operated part is designated as the medical apparatus 500 and a telephone call with
the doctor PR in the external facility is designated in the request for an information
source selection.
[0076] Fig. 11 is a view illustrating a third example of the presentation information. Fig. 11
is an example of the presentation information PN generated by the generation unit 360
when the biological monitor is designated as the medical apparatus 500 in the request
for an information source selection. The biological monitor illustrated in Fig. 11 does
not include an interface that can be connected to the first input/output unit 310.
However, when the information processing apparatus 300 is configured to include the
visible light camera, and the visible light camera photographs a screen (CP13) of the
biological monitor, as illustrated in Fig. 11, the generation unit 360 can generate the
presentation information PN using the medical apparatus information from the medical
apparatus that does not include an interface that can be connected to the first input/
output unit 310.
[0077] When the presentation information PN is generated, the head mounted display 100
may directly or indirectly designate "arrangement", a "size" and "transmittance" of the
information of each information source in the request for an information source
selection. Here, the "direct designation" means that the head mounted display 100
designates numerical values indicating a coordinate at which each information is
arranged, and a size and transmittance of each information. The "indirect designation"
means that the head mounted display 100 designates a desired mode from display
modes that are prepared in advance, for example, from an "outside scene priority
mode" in which an outside scene is prioritized, an "image priority mode" in which an
image is prioritized, and the like. For example, when the outside scene priority mode is
designated, as illustrated in Fig. 8, the generation unit 360 generates the presentation
information PN in such a manner that the information is arranged in a small size at an
end portion of a screen so as not to interfere with the visual field of the user at the
center of the screen. When the image priority mode is designated, as illustrated in Fig.
9A, the generation unit 360 generates the presentation information PN in such a
manner that the information is arranged in a large size at the center of the screen.
[0078] In step SI 18 of the presentation information generating process (refer to Fig. 6), the
generation unit 360 transmits the generated presentation information PN* to the
entirety of the second input/output units 370 (the second input/output units 37 1 to
37n). The second input/output unit 371 converts the image contained in the received
presentation information PN* as per the video signal specification of the head mounted
display 100a. At this time, the second input/output unit 371 may refer to the "video
specification" from the management table 382. After the conversion is performed as
per the video specification, the second input/output unit 37 1 transmits the presentation
information PN* to the head mounted display 100a. Similarly, the second input/output
units 372 to 37n convert images contained in the received presentation information
PN* as per the video signal specifications of the head mounted displays 100b to 100h,
respectively. The second input/output units 372 to 37n transmit the converted pre
sentation information PN* to the head mounted displays 100b to 100h, respectively.
[0079] Fig. 12 is a view illustrating a state in which the presentation information PN* is re
spectively displayed as a virtual image on the plurality of head mounted displays 100a
and 100b. The aforementioned display processing is performed by the image
processing unit 160 of the head mounted display 100a that receives the presentation in
formation PN* transmitted from the second input/output unit 371 via the interface 180
or the wireless communication unit 132. As a result, image light guided to both eyes of
the user of the head mounted display 100a is focused on the retinas of the user, and
thus the user of the head mounted display 100a can see the presentation information
PN* as a virtual image VI in a visual field VR. The user can directly see a patient lying
on the operating table (an outside scene SC) through the right optical image display
unit 26 and the left optical image display unit 28. Similarly, the aforementioned
display processing is performed by the image processing unit 160 of the head mounted
display 100b that receives the presentation information PN* transmitted from the
second input/output unit 372, and the user of the head mounted display 100b can see
the presentation information PN* as the virtual image VI in the visual field VR. In the
example illustrated in Fig. 12, the medical apparatus information CP acquired from the
medical apparatus 502 for photographing a still image and a moving image of an
operated part is displayed in the virtual image VI (PN*) seen by the user of the head
mounted display 100a and the user of the head mounted display 100b.
[0080] Fig. 13 is a view illustrating a state in which the presentation information PN* is
displayed as virtual images on the plurality of head mounted displays 100a and 100b
that are positioned in different places. As illustrated in Fig. 13, the second input/output
units 370 (the second input/output units 371 to 37n) of the information processing
apparatus 300 may perform a fitting process in such a manner that an orientation and a
size of the image contained in the presentation information coincide with an orientation
and a size of a target object contained in the outside scene SC. For example, the target
object is the body of a patient or a part of the body in a medical location. Hereinafter,
the fitting process will be described with reference to the second input/output unit 371.
In step SI 18 of the presentation information generating process (refer to Fig. 6), the
second input/output unit 37 1 converts the image contained in the presentation in
formation PN* as per the video signal specification of the head mounted display 100a,
and then performs the following procedures d l to d5.
[0081] (dl) The second input/output unit 371 instructs the head mounted display 100a to
photograph an outside scene image using the camera 61, and acquires the pho
tographed outside scene image.
(d2) The second input/output unit 371 extracts characteristics of the target object
contained in the outside scene image using the following image recognition methods 1
and 2. The methods 1 and 2 may be combined.
1. An edge (characteristics) of the target object is detected
2. A marker (characteristics) which is attached to the target object in advance is
detected. Various types of markers can be used as the marker attached to the object.
For example, a tape, a seal, a magic marker, a laser marker, and a magic tape
(registered trademark) can be used. The number of markers attached to the target
object is arbitrarily adjusted.
(d3) The second input/output unit 371 extracts the characteristics of the target object
by recognizing the image contained in the received presentation information. At this
time, the second input/output unit 37 1 can use the same image recognition methods as
the aforementioned methods 1 and 2.
[0082] (d4) The second input/output unit 371 compensates the image CP contained in the
presentation information PN* in such a manner that the characteristics of the outside
scene image extracted in the procedure d2 are positioned in the same place in which
the characteristics of the presentation information extracted in the procedure d3 are p o
sitioned. Specifically, when the method 1 is used in the procedures d2 and d3, the
second input/output unit 371 at least enlarges, reduces, rotates, reverses, trims, or
distorts the image CP contained in the presentation information PN*, or removes noise
from the image CP in such a manner that a contour and a characteristic portion of the
edge of the outside scene image are positionally aligned with a contour and a charac
teristic portion of the edge of the presentation information. Here, when the target
object is a living body, the characteristic portion is a contour of an incised skin, a
contour of an internal organ, a joint, distal ends of limbs, a blood vessel, a bone, and
the like. In contrast, when the method 2 is used in the procedures d2 and d3, the second
input/output unit 371 at least enlarges, reduces, rotates, reverses, trims, or distorts the
image CP contained in the presentation information PN*, or removes noise from the
image CP in such a manner that the marker of the outside scene image is positionally
aligned with the marker of the presentation information.
[0083] In this manner, the second input/output units 370 (the second input/output units 371
to 37n) recognize the image in the direction of the visual field of the user of the head
mounted display 100 (the head mounted display apparatus), and thus the second input/
output units 370 extract the characteristics of the object contained in the outside scene
and compensates the presentation information generated by the generation unit 360
based on the extracted characteristics. For this reason, when the presentation in
formation contains the invisible part information such as a CT image, an MRI image,
or an X-ray image, it is possible to align the invisible part information with the object
(for example, the body of a patient) contained in the outside scene. As a result, it is
possible to improve convenience of the user of the head mounted display 100. Since
the fitting processing is executed in the second input/output unit 370 connected to the
specific head mounted display 100, a processing load is not applied to the second
input/output units 370 connected to the other head mounted displays 100. Furthermore,
since the head mounted display 100 can receive the presentation information that
undergoes the fitting processing, the image processing unit 160 of the head mounted
display 100 may display the received presentation information as it is. As a result, it is
possible to simplify the configuration of the head mounted display 100.
[0084] As described above, when the operation mode of the generation unit 360 is the "first
mode", the generation unit 360 generates the common presentation information PN*
that contains at least the medical apparatus information selected by the user of the head
mounted display 100 (that is, the specific image display apparatus) in which the interapparatus
relationship of the management table 382 is a "master". The second input/
output units 370 (presentation units) output the generated common presentation in
formation PN* to the plurality of head mounted displays 100a to 100h (the image
display apparatuses) which are respectively connected to the second input/output units
371 to 37n. As a result, the information processing apparatus 300 can control display
on the plurality of head mounted displays 100a to 100h based on the selection of the
user of the specific head mounted display 100.
[0085] A-4-2. Second Mode (Initial Process):
When the operation mode is the "second mode" in step SI00 of the presentation in
formation generating process (refer to Fig. 6) (step S100: the second mode), the
generation unit 360 sets a variable i used in the presentation information generating
process to "1" (step S120). Thereafter, the generation unit 360 refers to an i-th entry of
the management table 382 (step S122). The generation unit 360 acquires an "identifier"
and a "role" which are stored in the i-th entry of the management table 382 (step S124).
[0086] Hereinafter, the head mounted display 100 having the identifier acquired in step
S124 is also referred to as a "head mounted display lOOi". The head mounted display
lOOi indicates the head mounted display 100 that is currently under the processing in
the initial process. For example, when the variable i is 1, the head mounted display
lOOi indicates the head mounted display 100a of the operating surgeon, which has the
identifier of 1 and is stored in the first entry E01 of the management table 382.
[0087] In step S126 of the presentation information generating process (refer to Fig. 6), the
generation unit 360 specifies the medical apparatus 500 based on the role of the user of
the head mounted display lOOi. Specifically, when the generation unit 360 carries out
the "role" acquired in step S124, the generation unit 360 specifies the medical
apparatus 500 that the user of the head mounted display lOOi has to check. The medical
apparatus 500 is specified based on a practice in a medical location. For example,
when the role is an operating surgeon, an apparatus which acquires the invisible part
information such as a CT image or an X-ray image is specified as the medical
apparatus 500. When the role is an anesthesiologist, each of a blood pressure meter, an
electrocardiogram, an airway pressure meter, a ventilation meter, a capnogram and a
pulse oximeter is specified as the medical apparatus 500. The information processing
apparatus 300 may store a table in the external storage apparatus 380 in advance, in
which the role and the medical apparatus 500 necessary for the role are stored in a s
sociation with each other.
[0088] After the medical apparatus 500 is specified, the generation unit 360 generates the
presentation information PNi to be transmitted to the head mounted display lOOi using
the medical apparatus information that is acquired from the medical apparatus 500
specified in step S126 among the medical apparatus information of the medical ap
paratuses 501 to 50n, which the first input/output units 311 to 3In acquire in step
S128. The generation unit 360 generates the presentation information in the same
method as described with reference to Figs. 7A, 7B, and 8. Thereafter, the generation
unit 360 stores the generated presentation information PNi in the operation record 384.
[0089] In step SI 30, the generation unit 360 transmits the presentation information PNi
generated in step SI24 to a second input/output unit 370i to which the head mounted
display lOOi is connected. The second input/output unit 370i converts an image
contained in the received presentation information PNi as per a video signal speci
fication of the head mounted display lOOi. At this time, the second input/output unit
370i may execute the fitting process. The detail is the same as described in step SI 18.
[0090] Fig. 14 is a view illustrating a state in which the presentation information PNi is
displayed as virtual images on the plurality of head mounted displays 100a and 100b
after steps S122 to S134 are executed two times. The aforementioned display
processing is performed by the image processing unit 160 of the head mounted display
lOOi that receives the presentation information PNi via the interface 180 or the wireless
communication unit 132. As a result, the user of the head mounted display lOOi can see
the presentation information PNi as a visual image, which contains the medical
apparatus information acquired from the medical apparatus 500 based on the role of
the user. In an example illustrated in Fig. 14, the medical apparatus information CP4
acquired from the medical apparatus 500, which an operating surgeon has to check, is
displayed as a virtual image VI (i) that the user of the head mounted display 100a sees.
The medical apparatus information CP2 and CP3 acquired from the medical apparatus
500, which a first assistant has to check, is displayed as the virtual image VI (i) that the
user of the head mounted display 100b sees.
[0091] The generation unit 360 increments the variable i in step SI32 of the presentation in
formation generating process (refer to Fig. 6). Thereafter, the generation unit 360 de
termines whether the variable i is equal to or greater than n (step S134). The fact that
the variable i is smaller than n (step S134: NO) means that a part of the head mounted
displays 100 do not end the initial process among the head mounted displays 100
which are respectively connected to the second input/output units 370. For this reason,
the generation unit 360 transfers the process to step S122 to continuously execute the
initial process. In contrast, the fact that the variable i is equal to or greater than n (step
S134: YES) means that the initial processes for the entirety of the head mounted
displays 100 are ended. For this reason, the generation unit 360 transfers the process to
an individual process in step SI 36.
[0092] As described above, when the operation mode of the generation unit 360 is the
"second mode", the generation unit 360 repeatedly executes the steps S122 to S134 in
the initial process. Accordingly, the generation unit 360 generates plural pieces of the
presentation information PNi that contain at least the medical apparatus information
which is acquired from the medical apparatuses 500 based on the respective roles of
the users of the plurality of head mounted displays lOOi (100a to 100h). In addition, the
second input/output units 370 (the presentation units) sequentially output the plural
pieces of generated presentation information PNi to the plurality of head mounted
displays 100a to 100h (the image display apparatuses) which are respectively
connected to the second input/output units 37 1 to 37n. As a result, the information
processing apparatus 300 can individually control display on the plurality of head
mounted displays 100a to 100h based on the role of each user of the head mounted
display 100, and it is possible to improve convenience of each user of the head
mounted display 100.
[0093] A-4-3. Second Mode (Individual Process):
In the individual process, based on a request from the head mounted display 100, the
generation unit 360 generates the presentation information that contains the medical
apparatus information acquired from the requested medical apparatus 500 and the in
formation acquired from information sources other than the requested medical
apparatus 500.
[0094] The generation unit 360 determines whether a request for an information source
selection is received from the head mounted display 100 in step S136 of the pre
sentation information generating process (refer to Fig. 6). The request for an in
formation source selection is the same as the requests a l to a6 which are described in
step SI 12. When the request for an information source selection is not received (step
S136: not received), the generation unit 360 transfers the process to the step S136, and
waits for the request for an information source selection.
[0095] In contrast, when the request for an information source selection is received (step
S136: received), The generation unit 360 acquires an identifier of the head mounted
display lOOi contained in the request for an information source selection, information
for specifying the information source, and information for specifying the information
in the information source (step S128). Furthermore, when the current position in
formation and the orientation information are inserted as the information for specifying
the information source, the generation unit 360 compares the current position in
formation and the orientation information with position information acquired from the
entirety of other medical apparatuses 500, and the generation unit 360 specifies "the
medical apparatus 500, the display unit of which the user can hardly see". Here,
similarly to in step SI 14, the generation unit 360 may combine the current position in
formation and the orientation information with a movement of the head of the user of
the head mounted display 100. The head mounted display lOOi indicates the head
mounted display 100 (that is, the head mounted display 100 that is currently under the
processing) which is a source of issuing the request for an information source
selection.
[0096] The generation unit 360 accesses the designated information source, and acquires the
designated information in step S140. The generation unit 360 generates the pre
sentation information PNi for the head mounted display lOOi using the acquired in
formation, and stores the generated presentation information PNi in the operation
record 384. The generation unit 360 generates the presentation information in the same
method as described with reference to Figs. 7A, 7B, and 8. Thereafter, the generation
unit 360 stores the generated presentation information PNi in the operation record 384.
[0097] In step S142, the generation unit 360 transmits the presentation information PNi
generated in step SI40 to the second input/output unit 370i to which the head mounted
display lOOi is connected. The second input/output unit 370i converts an image
contained in the received presentation information PNi as per the video signal speci
fication of the head mounted display lOOi. At this time, the second input/output unit
370i may execute the fitting process. The detail is the same as described in step SI 18.
[0098] The aforementioned display processing is performed by the image processing unit
160 of the head mounted display lOOi that receives the presentation information PNi
via the interface 180 or the wireless communication unit 132. As a result, the user of
the head mounted display lOOi, which transmits the request for an information source
selection, can see a virtual image of the presentation information PNi that includes the
information acquired from the information source designated in the request for an in
formation source selection.
[0099] As described above, when the operation mode of the generation unit 360 is the
"second mode", the generation unit 360 executes the individual process. Accordingly,
the generation unit 360 generates the plural pieces of presentation information PNi that
contain at least the medical apparatus information which is selected by each user of the
plurality of head mounted displays lOOi (100a to 100h). The second input/output units
370 (the presentation units) sequentially output the plural pieces of generated pre
sentation information PNi to the plurality of head mounted displays 100a to 100h (the
image display apparatuses) which are respectively connected to the second input/
output units 371 to 37n. As a result, the information processing apparatus 300 can indi
vidually control display on the plurality of head mounted displays 100a to 100h based
on the role of each user of the head mounted display 100.
[0100] As described above, in the information processing apparatus 300 of First Em
bodiment, the generation unit 360 generates the presentation information PN using the
plural pieces of medical apparatus information acquired by the first input/output units
310 (acquisition units and the first input/output units 3 11 to 3In), and thus the information
processing apparatus 300 can unify the information (the medical apparatus
information) of various medical apparatuses 500 (the medical apparatuses 501 to 50n)
which are connected to the first input/output units 310 of the information processing
apparatus 300. The second input/output units 370 (the presentation units) output the
generated presentation information PN to the plurality of head mounted displays 100a
to 100h (the image display apparatuses), respectively, and thus the second input/output
units 370 can control display on the plurality of head mounted displays 100a to 100h,
respectively. As a result, it is possible to unify the information of various medical ap
paratuses 500, and it is possible to realize the information processing apparatus 300
that can control the display on the plurality of head mounted displays 100a to 100h.
[0101] Furthermore, the generation unit 360 has the first mode in which the common pre
sentation information PN* is generated from the unified medical apparatus in
formation, which is common to the plurality of head mounted displays 100a to 100h
(the image display apparatuses), and the second mode in which the plural pieces of
presentation information PNi are generated to respectively correspond to the plurality
of head mounted displays 100a to 100h. For this reason, the generation unit 360 uses
the modes separately, and thus the information processing apparatus 300 can control
the display on the plurality of head mounted displays 100a to 100h.
[0102] As described above, the information processing system 1000 of First Embodiment
can unify the information (the medical apparatus information) of various medical ap
paratuses 500 (the medical apparatuses 501 to 50n) in the information processing
apparatus 300, and it is possible to realize the information processing system 1000 that
can control the display on the plurality of head mounted displays 100a to 100h (the
head mounted display apparatuses).
[0103] A-5. Additional Process:
In the presentation information generating process (refer to Fig. 6), the following ad
ditional processes may be further executed. The additional processes may be added in
dependently or may be combined together.
[0104] A-5-1. Additional Process 1:
In an additional process 1, the information processing apparatus 300 performs a color
conversion process for the presentation information. Specifically, the second input/
output unit 370, which receives the presentation information from the generation unit
360, causes the camera 6 1 of the head mounted display 100 to acquire an outside scene
image. The second input/output unit 370 carries out the color conversion process so as
to improve visibility of the image contained in the presentation information based on
brightness, contrast, chroma, and color of the acquired outside scene image. For
example, the second input/output unit 370 can perform the color conversion process in
which the color of the image contained in the presentation information is converted
into a complementary color of the color of the outside scene image.
[0105] In this manner, the additional process 1 can further improve visibility of the virtual
image VI for the user of the head mounted display 100.
[0106] A-5-2. Additional Process 2:
An additional process 2 enables the information processing apparatus 300 to realize a
so-called stabilizer function that suppresses the flickering of the virtual image VI
incident to a small shake of the head of the user of the head mounted display 100.
Specifically, the second input/output unit 370, which receives the presentation in
formation from the generation unit 360, repeatedly performs the fitting process
described in the procedures d l to d4, and the transmission of the presentation in
formation described in steps SI 18, SI 30, and SI42 of the presentation information
generating process at every predetermined time. Accordingly, the following is r e
peatedly performed at every predetermined time: the photographing of the outside
scene image, and the compensation and the display of the image contained in the pre
sentation information. As a result, the presentation information can be displayed as the
virtual image VI so as to trace the movement over time of the head of the user.
[0107] However, when the presentation information traces a slight movement or a
fluctuation of the head of the user of the head mounted display 100, and thus the pre
sentation information is frequently changed, there is a problem in that eyestrain of the
user is induced and concentration of the user is disturbed. Therefore, the second input/
output unit 370 may perform processes described in the following procedures e l and
e2 between the procedure d l and the procedure d2.
[0108] (el) The second input/output unit 370 calculates the amount of change in RGB
histogram between the outside scene image photographed in the procedure c l and the
outside scene image photographed in the procedure d l in the fitting process that is
executed at the previous cycle.
(e2) When the amount of change calculated in the procedure f 1 exceeds a prede
termined threshold value, the second input/output unit 370 continuously performs the
processes subsequent to the procedure d2.
[0109] In this manner, in the additional process 2, when there is a small amount of change
between the outside scene image in the fitting process executed at the previous cycle
and the outside scene image in the fitting process currently being executed, that is,
when the movement of the head of the user of the head mounted display 100 is very
small, the second input/output unit 370 does not compensate the image contained in the
presentation information. As a result, since the presentation information can be
prevented from tracing a very small movement or a fluctuation of the head of the user
of the head mounted display 100, and can be prevented from frequently being changed,
it is possible to suppress eyestrain or deterioration of concentration of the user. A configuration

Claims
An information processing apparatus comprising:
an acquisition unit that acquires plural pieces of medical apparatus in
formation which are plural pieces of information acquired from a
plurality of medical apparatuses;
a generation unit that generates plural pieces of presentation in
formation which respectively contain at least a part of the plural pieces
of medical apparatus information acquired by the acquisition unit; and
a presentation unit that outputs the generated presentation information
to a plurality of image display apparatuses.
The information processing apparatus according to claim 1,
wherein the generation unit has a first mode in which the presentation
information common to the plurality of image display apparatuses is
generated, and a second mode in which the plural pieces of presentation
information are generated to respectively correspond to the plurality of
image display apparatuses.
The information processing apparatus according to claim 2,
wherein in the first mode, the generation unit generates the common
presentation information that contains at least the medical apparatus in
formation selected by a user of a specific image display apparatus.
The information processing apparatus according to claim 3, further
comprising:
an inter-apparatus relationship storage unit that stores information for
individually identifying the plurality of image display apparatuses in
association with information for indicating whether the image display
apparatus is the specific image display apparatus,
wherein in the first mode, the generation unit identifies the specific
image display apparatus using the inter-apparatus relationship storage
unit.
The information processing apparatus according to claim 2,
wherein in the second mode, the generation unit generates the plural
pieces of presentation information that respectively contain at least the
pieces of medical apparatus information which are respectively selected
by users of the plurality of image display apparatuses.
The information processing apparatus according to claim 2,
wherein in the second mode, the generation unit generates the plural
pieces of presentation information that respectively contain at least the
WO 2015/008470 PCT/JP2014/003697
pieces of medical apparatus information acquired from the medical ap
paratuses based on the role of each user.
[Claim 7] The information processing apparatus according to claim 6, further
comprising:
a role storage unit that stores the information for individually
identifying the plurality of image display apparatuses in association
with a role of a user of the image display apparatus,
wherein in the second mode, the generation unit identifies the role of
the user using the role storage unit.
[Claim 8] The information processing apparatus according to claim 2,
wherein in the second mode, the generation unit further generates
combined information in which the plural pieces of generated pre
sentation information are combined together, and
wherein the presentation unit outputs the combined information that is
generated for the specific image display apparatus.
[Claim 9] The information processing apparatus according to claim 1,
wherein when the generation unit generates the presentation in
formation, the generation unit enlarges or reduces an image while
maintaining an aspect ratio of the image that is contained in the medical
apparatus information.
[Claim 10] The information processing apparatus according to claim 1,
wherein the information processing apparatus is connected to a head
mounted display apparatus as the image display apparatus, on which a
user can see a virtual image and an outside scene.
[Claim 11] The information processing apparatus according to claim 10,
wherein the presentation unit further acquires from the head mounted
display apparatus an image in a direction of a visual field of the user
that wears the head mounted display apparatus,
wherein the presentation unit further extracts characteristics of an
object contained in the outside scene by recognizing the image acquired
in the direction of the visual field, and
wherein the presentation unit further compensates the presentation in
formation generated by the generation unit based on the extracted char
acteristics, and outputs the compensated presentation information.
[Claim 12] The information processing apparatus according to claim 1, further
comprising:
a step management unit that transmits information on a predetermined
step to the generation unit, and that changes the step to be transmitted
WO 2015/008470 PCT/JP2014/003697
based on the plural pieces of medical apparatus information acquired
by the acquisition unit,
wherein the generation unit generates the presentation information that
contains the received step.
[Claim 13] The information processing apparatus according to claim 12,
wherein the step management unit repeatedly acquires the plural pieces
of medical apparatus information, and changes the step to be
transmitted to the generation unit based on the plural pieces of latest
medical apparatus information.
[Claim 14] The information processing apparatus according to claim 12,
wherein the step management unit transmits to the generation unit in
formation on a series of predetermined steps which are formed of a
plurality of the steps and
wherein the step management unit changes the steps by performing at
least one of the following procedures; addition of a new step to the
series of steps, removal of at least a part of steps from the series of
steps, and replacement of at least a part of steps in the series of steps.
[Claim 15] The information processing apparatus according to claim 12,
wherein the step management unit sequentially transmits to the
generation unit the series of predetermined steps one by one, which are
formed of the plurality of steps, and
wherein the step management unit changes the steps by stopping the
transmission of the single step.
[Claim 16] The information processing apparatus according to claim 12 which is
dependent of claim 2,
wherein in the first mode, the step management unit transmits the steps
for the user of the specific image display apparatus to the generation
unit.
[Claim 17] The information processing apparatus according to claim 12 which is
dependent of claim 2,
wherein in the second mode, the step management unit transmits to the
generation unit the steps that respectively correspond to the roles of the
users.
[Claim 18] An information processing method comprising:
(a) acquiring plural pieces of medical apparatus information that are
plural pieces of information acquired from a plurality of medical ap
paratuses;
(b) generating presentation information that contains at least a part of
WO 2015/008470 PCT/JP2014/003697
the plural pieces of acquired medical apparatus information; and
(c) outputting the generated presentation information to a plurality of
image display apparatuses.
[Claim 19] An information processing system comprising:
a plurality of medical apparatuses;
an information processing apparatus; and
a plurality of head mounted display apparatuses, on each of which a
user can see a virtual image and an outside scene,
wherein the information processing apparatus includes
an acquisition unit that acquires plural pieces of medical apparatus in
formation which are plural pieces of information that are acquired from
the plurality of medical apparatuses;
a generation unit that generates presentation information which
contains at least a part of the plural pieces of medical apparatus in
formation acquired by the acquisition unit; and
a presentation unit that outputs the generated presentation information
to the plurality of head mounted display apparatuses, and
wherein the head mounted display apparatus includes
a presentation information acquisition unit that acquires the pre
sentation information from the information processing apparatus; and
an image display unit on which a user of the head mounted display
apparatus sees the acquired presentation information as the virtual
image.