Description
Title of Invention: INFORMATION PROCESSING APPARATUS,
INFORMATION PROCESSING METHOD, AND PROGRAM
Technical Field
[0001] The present disclosure relates to an information processing apparatus, an information
processing method, and a program encoded on a non-transitory computer readable
medium.
Background Art
[0002] In recent years, a technique called augmented reality (AR) in which additional in
formation is superimposed on a real image so as to be presented to a user has attracted
attention. In the AR technique, information presented to a user is also called annotation
and the information is visualized as various types of virtual objects such as, for
example, texts, icons or animation. As an example of the AR technique, Patent
Literature 1 discloses a technique in which an image of a virtual object imitating a real
object such as furniture is superimposed on a real image so as to be displayed, thereby
easily attempting to arrange furniture and so on.
Citation List
Patent Literature
[0003] PTL 1: JP 2003-256876A
Summary
Technical Problem
[0004] The above-described AR technique is realized, for example, by capturing a real
image using a device held by the user and by superimposing the display of a virtual
object related to a real object reflected in the image. In this case, for example, postures
of the reflected real object are variously changed depending on the movement of the
user or changes in methods of holding the device. That is to say, even in the same real
object, a size or angle in the image is changed depending on the circumstances. Thus,
even if a captured image of the same real object is displayed, there are cases where ap
propriate display methods of additional information are different.
[0005] Therefore, it is desirable to provide novel and improved information processing
apparatus, information processing method, and program encoded on a non-transitory
computer readable medium, capable of appropriately displaying additional information
so as to be suitable for a posture of a real object.
Solution to Problem
[0006] In one embodiment, the present invention includes apparatus including an image
processor configured to receive a video including an object, determine a positional re
lationship between the apparatus and the object, and change a positional relationship
between an image superimposed on the video and the object when the positional rela
tionship between the apparatus and the object changes.
Advantageous Effects of Invention
[0007] As described above, according to embodiments of the present disclosure, it is
possible to appropriately display additional information so as to be suitable for a
posture of a real object.
Brief Description of Drawings
[0008] [fig. 1]FIG. 1 is a diagram illustrating a reference technique regarding a first em
bodiment of the present disclosure;
[fig.2]FIG. 2 is a block diagram illustrating an example of the functional configuration
of the information processing apparatus according to the first embodiment of the
present disclosure;
[fig.3]FIG. 3 is a diagram illustrating a first state where a virtual object is displayed by
the information processing apparatus according to the first embodiment of the present
disclosure;
[fig.4]FIG. 4 is a diagram illustrating a second state where a virtual object is displayed
by the information processing apparatus according to the first embodiment of the
present disclosure;
[fig.5]FIG. 5 is a diagram illustrating a change in display of a virtual object by the in
formation processing apparatus according to the first embodiment of the present
disclosure;
[fig.6]FIG. 6 is a flowchart illustrating an example of the process flow of image
display process in the information processing apparatus according to the first em
bodiment of the present disclosure;
[fig.7]FIG. 7 is a flowchart illustrating an example of the process flow of display target
setting process in the information processing apparatus according to the first em
bodiment of the present disclosure;
[fig.8]FIG. 8 is a diagram illustrating an example of the alternative selection and the
message display according to the first embodiment of the present disclosure;
[fig.9]FIG. 9 is a diagram illustrating a first state where a virtual object is displayed by
an information processing apparatus according to a second embodiment of the present
disclosure;
[fig.l0]FIG. 10 is a diagram illustrating a second state where a virtual object is
displayed by the information processing apparatus according to the second em
bodiment of the present disclosure;
[fig. 11]FIG. 11 is a diagram illustrating a change in display of a virtual object by the
information processing apparatus according to the second embodiment of the present
disclosure;
[fig. 12]FIG. 12 is a diagram illustrating a positional relationship between a detected
real object and a device according to a third embodiment of the present disclosure;
[fig.l3]FIG. 13 is a diagram illustrating a first state where a virtual object is displayed
by the information processing apparatus according to the third embodiment of the
present disclosure;
[fig.l4]FIG. 14 is a diagram illustrating a second state where a virtual object is
displayed by the information processing apparatus according to the third embodiment
of the present disclosure;
[fig. 15]FIG. 15 is a diagram illustrating a change in display of a virtual object by the
information processing apparatus according to the first embodiment of the present
disclosure;
[fig.l6]FIG. 16 is a flowchart illustrating an example of the process flow of display
target setting process in the information processing apparatus according to the third
embodiment of the present disclosure;
[fig.l7]FIG. 17 is a block diagram illustrating another example of the functional con
figuration of an information processing apparatus according to an embodiment of the
present disclosure;
[fig.l8]FIG. 18 is a block diagram illustrating further another example of the
functional configuration of the information processing apparatus according to an em
bodiment of the present disclosure; and
[fig. 19]FIG. 19 is a block diagram illustrating a hardware configuration of the in
formation processing apparatus.
Description of Embodiments
[0009] Hereinafter, preferred embodiments of the present disclosure will be described in
detail with reference to the appended drawings. Note that, in this specification and the
appended drawings, structural elements that have substantially the same function and
structure are denoted with the same reference numerals, and repeated explanation of
these structural elements is omitted.
[0010] In addition, the description will be made in the following order.
1. First Embodiment (an example of changing a display position relative to a real
object)
2. Second Embodiment (an example of changing an amount of displayed in
formation)
3. Third Embodiment (an example of changing a content of displayed information)
4. Other Embodiments regarding Apparatus Configuration
5. Supplement
[0011] 1. First Embodiment
The first embodiment of the present disclosure will be described with reference to
FIGS. 1 to 8. In the first embodiment of the present disclosure, a display position of a
virtual object relative to a real object is changed depending on a relative positional r e
lationship between the real object reflected in an image and a device capturing the
image.
[0012] 1-1. Reference Technique
First, a description will be made of a reference technique for better understanding of
the present embodiment. FIG. 1 is a diagram illustrating the reference technique
regarding the first embodiment of the present disclosure.
[0013] An information processing apparatus 10 related to the reference technique is shown
in FIG. 1. The information processing apparatus 10 includes an image pickup unit (not
shown) and a display unit 12, and the information processing apparatus 10 executes an
AR application.
[0014] Here, the display unit 12 displays an image where the information processing
apparatus 10 superimposes a virtual object obj_v related to a real object obj_r on an
image in which the real object obj_r captured by the image pickup unit is reflected. In
the shown example, in a case where the real object obj_r (a book) is captured from the
front side as shown in (a) of FIG. 1, and in a case where the real object obj_r is
captured obliquely as shown in (b) of FIG. 1, the virtual object obj_v (an image
regarding a content of the book) is transformed into different forms and is displayed. In
other words, the virtual object obj_v is transformed and displayed to be suitable for the
change in the appearance due to the change in a posture of the real object obj_r.
[0015] As such, in the AR application, there are cases where a virtual object is transformed
to be suitable for a posture of a real object and is disposed in an AR space. Thereby,
for example, a user recognizes the virtual object as if the virtual object is reflected on a
surface of a real object and exists therein, and thus can easily understand a relationship
between the real object and the virtual object.
[0016] However, as shown in (b) of FIG. 1 of the illustrated example, there is a case where it
is difficult for the transformed virtual object obj_v to be visually recognized according
to a posture of the real object obj_r. In this case, it is difficult to say that the virtual
object obj_v is displayed in an appropriate manner.
[0017] Therefore, in the present embodiment, by changing a display position of a virtual
object relative to a real object depending on a positional relationship between the real
object reflected in an image and a device capturing the image, even in the case, it is
possible to display the virtual object obj_v in an appropriate manner.
[0018] 1-2. Apparatus Configuration
Next, with reference to FIG. 2, a functional configuration according to the present
disclosure will be described. FIG. 2 is a block diagram illustrating an example of the
functional configuration of the information processing apparatus according to the first
embodiment of the present disclosure.
[0019] Referring to FIG. 2, an information processing apparatus 100 according to the
present embodiment includes an image pickup unit 102, an image recognition unit 104,
a model DB 106, a positional relationship detection unit 108, a selection unit 110, an
object DB 112, a display target information output unit 114, a message information
output unit 116, a display control unit 118, and a display unit 120.
[0020] In the present embodiment, the information processing apparatus 100 will be
described as a terminal apparatus into which the respective units including the image
pickup unit and the display unit, such as a smart phone, a tablet type PC (Personal
Computer), PDA (Personal Digital Assistant), a portable gaming machine, or a
portable music player are integrated. However, the information processing apparatus
100 may be other information processing apparatuses such as a notebook type or
desktop type PC. In this case, for example, the image pickup unit or the display unit
may be provided separately from the other units. In addition, in a case where the in
formation processing apparatus 100 is divided into a plurality of units, the "information
processing apparatus 100" in the disclosure such as a "positional relationship between
the information processing apparatus 100 and a real object" or an "angle of a real
object relative to the information processing apparatus 100" indicates a portion at
which the image pickup unit of the information processing apparatus 100 is provided
unless otherwise specifically defined herein.
[0021] Among the above-described functional configurations, the image pickup unit 102 is
realized by, for example, an image pickup device included in the information
processing apparatus 100. In addition, the image recognition unit 104, the positional
relationship detection unit 108, the selection unit 110, the display target information
output unit 114, the message information output unit 116, and the display control unit
118 are realized by, for example, a CPU (Central Processing Unit), a RAM (Random
Access Memory), and a ROM (Read Only Memory) of the information processing
apparatus 100 being operated according to a program. The model DB 106 and the
object DB 112 are realized by, for example, an internal or external storage device of
the information processing apparatus 100. The display unit 120 is realized by, for
example, a variety of displays which are included in the information processing
apparatus 100 as output devices or are connected to the information processing
apparatus 100 as external connection devices.
[0022]
The image pickup unit 102 captures a real space to generate an image. The image
pickup unit 102 may capture moving images. The moving images are configured by a
plurality of images forming consecutive frames. The image pickup unit 102 provides
the generated image to both the image recognition unit 104 and the display control unit
118. At this time, the generated image may be temporarily stored in, for example, the
RAM or the storage device.
[0023]
The image recognition unit 104 performs object recognition using data stored in the
model DB 106. The model DB 106 is a database which stores in advance model data
regarding shapes or appearances of objects which are targets recognized by the in
formation processing apparatus 100. In the present embodiment, targets recognized by
the information processing apparatus 100 are any objects in a real space. The model
data includes data defining a shape of each object, image data such as a predetermined
symbol mark or a text label added to each object, data of a feature amount set extracted
from an existing image for each object, and the like.
[0024] More specifically, the image recognition unit 104 uses an image captured by the
image pickup unit 102 as an input image, thereby recognizing which object is reflected
in the input image. The image recognition unit 104 compares, for example, a feature
point set extracted from the input image with a shape of an object defined by the model
data. In addition, the image recognition unit 104 may compare image data such as a
symbol mark or a text label defined by the model data with the input image. Fur
thermore, the image recognition unit 104 may compare a feature amount of an image
of an existing object defined by the model data with a feature amount extracted from
the input image.
[0025]
The positional relationship detection unit 108 detects a positional relationship
between the object recognized by the image recognition unit 104 and the information
processing apparatus 100 (more specifically, the image pickup unit 102). In the present
embodiment, an angle of the object relative to the information processing apparatus
100 is used as a positional relationship between the object and the information
processing apparatus 100. This angle is expressed as, for example, a tilt from a
reference posture which is set for an object in advance. In other words, in the present
embodiment, the positional relationship detection unit 108 detects a posture rela
tionship between the object and the information processing apparatus 100 as the p o
sitional relationship between the object and the information processing apparatus 100.
[0026] More specifically, the positional relationship detection unit 108 uses an image
captured by the image pickup unit 102 as an input image, thereby detecting a posture
of an object reflected in the input image. Here, in the present embodiment, a posture of
the object is collectively expressed by a 4*4 homogeneous transformation matrix in
dicating transformation between a model coordinate system in the model data stored in
the model DB 106 and a coordinate system of the object reflected in the input image.
The positional relationship detection unit 108 extracts an angle of the object relative to
the information processing apparatus 100 from the homogeneous transformation
matrix.
[0027] In addition, detailed examples of the positional relationship acquired by the p o
sitional relationship detection unit 108 will be described later. Since a process
performed by the positional relationship detection unit 108 is common to a process
performed by the image recognition unit 104 such as referring to the model DB 106,
the process may be performed along with the process performed by the image
recognition unit 104.
[0028]
The selection unit 110 selects a display target object which is superimposed on an
image, from a virtual object group including a plurality of kinds of virtual objects
related to the real object, according to the positional relationship detected by the p o
sitional relationship detection unit 108. More specifically, the selection unit 110 selects
a virtual object from a virtual object group including a plurality of virtual objects of
which the display positions relative to the real object are different from each other,
according to the angle of the real object relative to the information processing
apparatus 100. The virtual object is displayed at a position which is easily visually
recognized by a user. The angle of the real object relative to the information processing
apparatus 100 is acquired by the positional relationship detection unit 108. Such a
virtual object and an example of the selection of a virtual object by the selection unit
110 will be described later.
[0029] In the present embodiment, data of the virtual object group is stored in the object DB
112. The object DB 112 may be a database which stores in advance data of a virtual
object group corresponding to each real object. In addition, the object DB 112 may be
a database which temporarily stores a virtual object group which corresponds to a real
object recognized by the image recognition unit 104 and is downloaded from an
external device.
[0030] Furthermore, the object DB 112 may be a database which temporarily stores a virtual
object which is obtained by processing a part of or the overall real object reflected in
an image captured by the image pickup unit 102. The processing of the real object is
performed by a processing unit (not shown). The processing unit may generate a
virtual object by processing an image according to a result of a selection made by the
selection unit 110. In this case, for example, the selection unit 110 refers to in
formation indicating "what kind of virtual object can be generated" stored in the object
DB 112. The selection unit 110 also selects the kind of virtual object to be displayed
and instructs the processing unit to generate the virtual object.
[0031] In addition, as described later as an example of the alternative selection and the
message display, in a case where a selected virtual object cannot be displayed in an
image region, the selection unit 110 may make an alternative selection of a virtual
object which can be displayed in the image region regardless of the selection result.
[0032] In addition, the selection unit 110 may adjust a selection result such that the same
kind of virtual object is selected for a plurality of real objects related to each other. In
this case, for example, the selection unit 110 may average results detected by the p o
sitional relationship detection unit 108 for the respective real objects. The selection
unit 110 also may use the averaged result as a positional relationship common to the
respective real objects. In addition, the selection unit 110 may count the results
detected by the positional relationship detection unit 108 for the respective real objects
and may use a detection result detected from more real objects as a positional rela
tionship common to the respective real objects. Thereby, for example, the kinds of
virtual objects displayed for a plurality of real objects related to each other can be
unified, and the unification of the virtual objects for the real objects can be shown to a
user.
[0033]
The display target information output unit 114 outputs information of the display
target object selected by the selection unit 110 to the display control unit 118 such that
the information is superimposed on the image captured by the image pickup unit 102
so as to be displayed. Here, the output information of the display target object may be
image data itself of a virtual object or may be identification information such as an ID
for separately referring to the image data of the virtual object. Here, in a case of
outputting the image data of the virtual object, the display target information output
unit 114 may generate a superimposed image of the virtual object, for example,
according to a positional relationship between the real object and the information
processing apparatus 100. The positional relationship is detected by the positional rela
tionship detection unit 108. The display target information output unit 114 also may
output information of the superimposed image to the display control unit 118. For this
process, the display target information output unit 114 may refer to the object DB 112.
[0034]
The message information output unit 116 is additionally provided. The message in
formation output unit 116 generates information of a message which is superimposed
on an image and displayed, and the message information output unit 116 outputs the
information to the display control unit 118. For example, in a case where an alternative
display target object is selected on account of restriction of a region of an image
captured by the image pickup unit 102, as described later, the message information
output unit 116 generates information of a message prompting a user to change a
region of the image by moving the image pickup unit 102, and the message in
formation output unit 116 outputs the information to the display control unit 118.
[0035]
The display control unit 118 superimposes the virtual object which is a display target
object on the image captured by the image pickup unit 102. The superimposition uses
the information outputted from the display target information output unit 114. The
display control unit 118 also causes the display unit 120 to display the superimposed
result as an AR application image. At this time, the display control unit 118 may
generate a superimposed image where the display target object is disposed and
transformed to be suitable for the position, posture and size of the real object reflected
in the image, and the display control unit 118 may superimpose the superimposed
image on the image. As described above, the superimposed image may be generated by
the display target information output unit 114. When the superimposed image is
generated, the display control unit 118 may use, for example, information indicating
the positional relationship between the real object and the information processing
apparatus 100. The positional relationship is detected by the positional relationship
detection unit 108. In a case where the message information output unit 116 is
provided, the display control unit 118 superimposes a message on the image by further
using information output from the message information output unit 116.
[0036]
The display unit 120 presents the AR application image to a user. The AR ap
plication image is generated by the display control unit 118. As described above, the
display unit 120 is realized by various display devices which are formed integrally
with or separately from the information processing apparatus 100. The display unit 120
may be, for example, a display screen using an LCD (Liquid Crystal Display), an
OLED (Organic Light-Emitting Diode) or a PDP (Plasma Display Panel), or a seethrough
type or a non-see-through type HMD (Head Mounted Display) which is
installed by a user.
[0037] As described above, in the information processing apparatus 100 according to the
present embodiment, a display target object is selected according to a positional rela
tionship between a real object and the information processing apparatus 100. Also, the
display target object is selected from a virtual object group including a plurality of
kinds of virtual objects related to the real object. In addition, here, the "plurality of
kinds of virtual objects" are virtual objects where a superimposed image is generated
according to a plurality of pieces of image data (in other words, original data) which
are different from each other. The plurality of pieces of image data are stored in the
object DB 112, such as, for example, an "image which looks as if it exists in an inner
space of a real object" or an "image which looks as if it protrudes from a real object".
In addition, the "virtual object" is an object indicating additional information
(annotation) for a real object as described above.
[0038] 1-3. Example of Display Change
Next, an example of the display change according to the present embodiment will be
described with reference to FIGS. 3 to 5. FIG. 3 is a diagram illustrating a first state
where a virtual object is displayed by the information processing apparatus according
to the first embodiment of the present disclosure. FIG. 4 is a diagram illustrating a
second state where a virtual object is displayed by the information processing
apparatus according to the first embodiment of the present disclosure. FIG. 5 is a
diagram illustrating a change in display of a virtual object by the information
processing apparatus according to the first embodiment of the present disclosure.
[0039] FIG. 3 shows the first state where a virtual object is displayed by the information
processing apparatus 100 according to the present embodiment. As described above, in
the present embodiment, the kind of virtual object which is a display target is selected
according to an angle of a real object relative to the information processing apparatus
100. In the shown example, in the first state, since an angle Al of the real object obj_r
(a book) relative to the information processing apparatus 100 is relatively small, a
virtual object obj_vl (an image regarding a content of the book) which looks as if it
exists in an inner space of the real object obj_r is displayed. In the following de
scription, display of the virtual object when the real object obj_r is reflected in the
image in a state close to front view is referred to as "front view display".
[0040] Here, an angle A of the real object obj_r relative to the information processing
apparatus 100 may be expressed as a tilt of a posture of the real object obj_r from a
reference posture when a line of sight direction from the image pickup unit 102 of the
information processing apparatus 100 is used as a reference. The reference posture of a
real object is, for example, a posture which is defined in advance for each real object,
and may be a posture of a real object according to the above-described model co
ordinate system. In the real object obj_r (a book) in the shown example, a posture
where the cover thereof faces the image pickup unit 102 is set as the reference posture.
Here, since the real object obj_r has a plate shape, an angle A is defined as an angle
formed by a line perpendicular to the surface and a line of sight from the image pickup
unit 102. In addition, a shape of the real object obj_r is not limited to the plate shape,
and may be any shape. Therefore, the angle A of the real object obj_r relative to the in
formation processing apparatus 100 may be defined variously in addition to the shown
example.
[0041] FIG. 4 shows the second state where a virtual object is displayed by the information
processing apparatus 100 according to the present embodiment. In the shown example,
in the second state, since an angle A2 of the real object obj_r (a book) relative to the
information processing apparatus 100 is relatively large, a virtual object obj_v2 (an
image regarding a content of the book) which looks as if it protrudes from the real
object obj_r is displayed. In the following description, the display of virtual object
when the real object obj_r is reflected in the image in a state of oblique view is referred
to as "oblique view display".
[0042] As such, in the present embodiment, the virtual object obj_vl and the virtual object
obj_v2 of which display positions relative to the real object obj_r are different from
each other are selectively displayed according to a positional relationship between the
information processing apparatus 100 and the real object obj_r. In the shown example,
the virtual object obj_vl and the virtual object obj_v2 have different positions in the
depth direction when the surface of the real object obj_r is used as a reference surface.
More specifically, in a case where the real object obj_r is reflected in the image in a
state close to front view, the virtual object obj_vl which looks as if it is reflected in the
real object obj_r and exists therein is displayed. On the other hand, in a case where the
real object obj_r is reflected in the image in a state of oblique view, the virtual object
obj_v2 which looks as if it protrudes from the surface of the real object obj_r is
displayed.
[0043] Thereby, a user can be presented the virtual object obj_v which is easily visually
recognized and where a relationship with the real object obj_r is also easily u n
derstood, so as to be suitable for each posture even in a case where a posture of the real
object obj_r reflected on an image is variously changed. In addition, here, although an
example where a display target object is selected from a virtual object group including
the virtual object obj_vl in the front view display and the virtual object obj_v2 in the
oblique view display has been described, virtual objects included in the virtual object
group are not limited to these two kinds. For example, more kinds of virtual objects
may be included in the virtual object group, and display target objects may be changed
in more steps according to a posture relationship between a real object and the in
formation processing apparatus 100. Further, the changes of the positional relationship
between the virtual and real object may be performed linearly or non-linearly.
[0044] FIG. 5 shows an example of the change in display of a virtual object by the in
formation processing apparatus 100 according to the present embodiment. As
described above, in the present embodiment, display of a virtual object by the in
formation processing apparatus 100 is varied depending on an angle of a real object
relative to the information processing apparatus 100. In addition, in the example
described here, an angle A of a plate-shaped real object obj_r relative to the in
formation processing apparatus 100 is defined as an angle formed by a line perpendicular
to the surface of the real object obj_r and a line of sight from the image
pickup unit 102 of the information processing apparatus 100. FIG. 5 is a diagram i l
lustrating a positional relationship between the information processing apparatus 100
and the real object obj_r through simplification only in one direction on the twodimensional
plane in relationships between the angle A and the front view display and
the oblique view display of the virtual object. As will be readily apparent to a person
skilled in the art, in practice, similar relationship is established for any direction on a
three-dimensional space.
[0045] In the shown example, when the information processing apparatus 100 faces the real
object obj_r, the angle A is 0(deg). In a state of A=0(deg), display of the virtual object
is the front view display as in a case of the angle A=A1 shown in FIG. 3. If the in
formation processing apparatus 100 is gradually tilted with respect to the real object
obj_r from the state and the angle A exceeds a threshold value Tl, the display of the
virtual object is changed to the oblique view display as in a case of the angle A=A2
shown in FIG. 4. On the other hand, if the information processing apparatus 100
gradually comes close to the front surface of the real object obj_r from the state where
the display of the virtual object is the oblique view display and the angle A becomes
smaller than a threshold value T2, then the display of the virtual object is changed to
the front view display.
[0046] As such, in the present embodiment, the kind of virtual object which becomes a
display target object is selected using a threshold value set as an index value (the angle
A) regarding a positional relationship between the information processing apparatus
100 and the real object obj_r. In a case where the image pickup unit 102 captures
moving images and the kind of virtual object is selected for each of images forming
consecutive frames, a different threshold value may be used depending on a virtual
object which has been displayed until then, that is, the kind of virtual object selected in
an image of the previous frame, such as the above-described threshold values Tl and
T2. As such, by giving hysteresis to the selection of a virtual object, it is possible to
prevent reduction in visibility occurring since display of the virtual object is frequently
changed due to vibration of the angle A around a threshold value caused by camera
shake, etc.
[0047] 1-4. Example of Process Flow
Next, with reference to FIGS. 6 and 7, a description will be made of an example of
the process flow performed by the information processing apparatus according to the
present embodiment. FIG. 6 is a flowchart illustrating an example of the process flow
of image display process in the information processing apparatus according to the first
embodiment of the present disclosure. FIG. 7 is a flowchart illustrating an example of
the process flow of display target setting process in the information processing
apparatus according to the first embodiment of the present disclosure.
[0048] FIG. 6 is a flowchart illustrating an image display process according to the present
embodiment. The image display process described here is a series of processes
performed until a virtual object is superimposed on the image so as to be displayed
after the information processing apparatus 100 acquires an image for a real space. In
the shown example, the image pickup unit 102 captures moving images, and thus the
image display process is repeatedly performed for the respective images forming the
consecutive frames.
[0049] First, in step S101, the image pickup unit 102 acquires an image. Here, the acquired
image is an image which is obtained by capturing a real space and is reflecting a real
object obj_r therein.
[0050] Next, in step S103, the image recognition unit 104 performs object recognition for
the acquired image. Here, the image recognition unit 104 recognizes the real object
obj_r reflected in the image by the use of the model data stored in the model DB 106.
[0051] Next, in step S105, the positional relationship detection unit 108 detects a positional
relationship between the information processing apparatus 100 and the real object
obj_r. Here, the positional relationship detection unit 108 detects an angle A of the real
object obj_r relative to the information processing apparatus 100.
[0052] Next, in step SI07, the selection unit 110 determines whether or not the positional re
lationship between the information processing apparatus 100 and the real object obj_r
is varied as compared with an image of the previous frame. The positional relationship
is indicated by the angle A. If the positional relationship is varied, then the selection
unit 110, in step S109, selects a display target object on the basis of the positional rela
tionship after being varied. On the other hand, if the positional relationship is not
varied, since the kind of virtual object selected as a display target object for the image
of the previous frame is continuously used, the selection unit 110 skips step S109.
[0053] In addition, a display target selection process in step S109 will be described later
with reference to FIG. 7.
[0054] Next, in step Sill, the display target information output unit 114 outputs information
related to the display target object selected by the selection unit 110 to the display
control unit 118.
[0055] Next, in step SI 13, the display control unit 118 superimposes the virtual object on
the image so as to be displayed on the display unit 120 according to the information of
the display target object outputted from the display target information output unit 114.
Thereby, the image on which the virtual object appropriately selected by the selection
unit 110 is superimposed is presented to a user as an AR application image.
[0056] FIG. 7 is a flowchart illustrating a display target setting process according to the
present embodiment. The display target setting process described here is a process
performed in step S109 of the image display process as described above. In the shown
example, the image pickup unit 102 captures moving images, and thus the display
target setting process may be also performed for the respective images forming the
consecutive frames.
[0057] First, in step S201, the selection unit 110 determines whether or not the display of
virtual object in the previous frame is the front view display. As described above, the
front view display is a display of the virtual object obj_vl which looks as if it exists in
an inner space of the real object obj_r when the real object obj_r is reflected in the
image in a state close to front view.
[0058] Here, if the display of the virtual object in the previous frame is the front view
display, then the selection unit 110, in step S203, sets a threshold value Tl as a
threshold value of the angle A used to determine a continuous display target. On the
other hand, if the display of the virtual object in the previous frame is not the front
view display, that is, the display is the oblique view display, the selection unit 110, in
step S205, sets a threshold value T2 as the threshold value. As described with reference
to FIG. 5, the threshold values Tl and T2 of the angle A in the present embodiment has
a relationship of T2
FIG. 17 is a block diagram illustrating another example of the functional con
figuration of an information processing apparatus according to an embodiment of the
present disclosure. Referring to FIG. 17, an information processing apparatus 200
includes a sensor unit 222 as well as an image pickup unit 102, an image recognition
unit 104, a model DB 106, a positional relationship detection unit 108, a selection unit
110, an object DB 112, a display target information output unit 114, a message in
formation output unit 116, a display control unit 118, and a display unit 120.
[0106] The sensor unit 222 includes a motion sensor group which measures a motion of the
image pickup unit 102. The motion sensor may be, for example, an acceleration sensor
which measures acceleration of the image pickup unit 102, a gyro sensor which
measures a tilt angle, a geomagnetic sensor which measures the azimuth to which the
image pickup unit 102 is oriented, or the like. In addition, the sensor unit 222 may
include a GPS (Global Positioning System) sensor which receives a GPS signal and
measures latitude, longitude and altitude of the apparatus.
[0107] In the shown example, the positional relationship detection unit 108 detects a p o
sitional relationship between a real object and the information processing apparatus
200 (more specifically, the image pickup unit 102) on the basis of the motion of the
image pickup unit 102 acquired by the sensor unit 222. For example, the positional r e
lationship detection unit 108 detects a posture of the information processing apparatus
200 on the basis of information of acceleration or tilt angle of the image pickup unit
102. If there is information of a position of a real object, whether or not the real object
is on a horizontal plane such as the ground, or whether or not the real object is on a
vertical plane such as a wall, it is possible to obtain a positional relationship with the
real object by detecting a posture of the information processing apparatus 200. In
addition, the positional relationship detection unit 108, as shown in the figure, may
detect a positional relationship using information from the sensor unit 222, and may
detect a positional relationship using information from the image recognition unit 104
together.
[0108]
FIG. 18 is a block diagram illustrating still another example of the functional con
figuration of an information processing apparatus according to an embodiment of the
present disclosure. Referring to FIG. 18, an information processing apparatus 300
includes an image recognition unit 104, a model DB 106, a positional relationship
detection unit 108, a selection unit 110, an object DB 112, a display target information
output unit 114, a message information output unit 116, and a communication unit 324.
[0109] In the shown example, the information processing apparatus 300 does not include the
image pickup unit 102, the display control unit 118, and the display unit 120. The in
formation processing apparatus 300 is, for example, a server installed on a network,
and receives data of an image captured by the image pickup unit of an apparatus which
is a client as an input image via the communication unit 324. In addition, the in
formation processing apparatus 300 transmits information of a display target object
output from the display target information output unit 114 and further information of a
message output from the message information output unit 116, to the client via the
communication unit 324. An image on which a virtual object is superimposed is
generated, and the image is displayed in the display control unit of the terminal
apparatus using the transmitted information.
[0110] As such, the information processing apparatus according to the embodiments of the
present disclosure may not necessarily include the image pickup unit 102 or the display
unit 120, and has various configurations. For example, in the above-described in
formation processing apparatus 300, the display control unit 118 may be provided, and
an image on which a virtual object is superimposed may be generated and transmitted
to the client via the communication unit 324. Further, the information processing
apparatus may not include the image pickup unit 102 but include the display unit 120,
and superimpose a virtual object on an image captured by an external device so as to
be displayed. Furthermore, the information processing apparatus may not include the
image pickup unit 102 but include the display unit 120, and transmit an image where a
virtual object is superposed on a captured image to an external device.
[0111] 5. Supplement
Finally, with reference to FIG. 19, a description will be made of a hardware con
figuration of the information processing apparatus 900 capable of realizing the information
processing apparatus according to the embodiments of the present
disclosure. FIG. 19 is a block diagram illustrating a hardware configuration of the in
formation processing apparatus.
[01 12] The information processing device 900 includes a CPU (Central Processing Unit)
901, ROM (Read Only Memory) 903, and RAM (Random Access Memory) 905.
Further, the information processing device 900 may include a host bus 907, a bridge
909, an external bus 9 11, an interface 913, an input device 915, an output device 917, a
storage device 919, a drive 921, a connection port 923, and a communication device
925. The information processing device 900 may include a processing circuit such as
DSP (Digital Signal Processor) in addition to or instead of the CPU 901.
[0113] The CPU 901 functions as an arithmetic processing unit and a control unit, and
controls the entire operation within the information processing device 900 or a part
thereof in accordance with various programs recorded on the ROM 903, the RAM 905,
the storage 919, or the removable recording medium 927. The ROM 903 stores
programs, operation parameters, and the like used by the CPU 901. The RAM 905
temporarily stores programs used in the execution of the CPU 901, parameters that
change as appropriate during the execution, and the like. The CPU 901, the ROM 903,
and the RAM 905 are mutually connected by a host bus 907 constructed from an
internal bus such as a CPU bus. Further, the host bus 907 is connected to the external
bus 911 such as a PCI (Peripheral Component Interconnect/Interface) via the bridge
909.
[01 14] The input device 915 is a device used by a user such as, for example, a mouse, a
keyboard, a touch panel, a button, a switch, or a lever. The input device 915 may be,
for example, a remote control device that uses infrared rays or other radio waves, or an
external connection device 929 such as a portable phone corresponding to the
operation of the information processing device 900. The input device 915 includes an
input control circuit that generates an input signal based on information input by a user
and outputs the input signal to the CPU 901. The user can, by operating the input
device 915, input various data to the information processing device 900 or instruct the
information processing device 900 to perform a processing operation.
[0115] The output device 917 includes a device that can visually or audibly inform a user of
the acquired information. The output device 917 can be, for example, a display device
such as an LCD (liquid crystal display), a PDP (Plasma Display Panel,) an organic EL
(Electro-Luminescence) display; an audio output device such as a speaker or
headphones; or a printer device. The output device 917 outputs the result obtained
through the processing of the information processing device 900 as text or video such
as an image or as sound such as voice or audio.
[0116] The storage device 919 is a device for storing data, constructed as an example of a
storage unit of the information processing device 900. The storage device 919 includes,
for example, a magnetic storage device such as HDD (Hard Disk Drive), a semi
conductor storage device, an optical storage device, or a magneto-optical storage
device. This storage device 929 includes, for example, programs or various data
executed by the CPU 901 or various data acquired from the outside.
[0117] The drive 921 is a reader/writer for a removable recording medium 927 such as a
magnetic disk, an optical disc, a magneto-optical disk, or semiconductor memory, and
is incorporated in or externally attached to the information processing device 900. The
drive 921 reads information recorded on a removable recording medium 927 that is
mounted, and outputs the information to the RAM 905. The drive 921 also writes in
formation to the removable recording medium 927 that is mounted.
[0118] The connection port 923 is a port for directly connecting a device to the information
processing device 900. The connection port 923 can be, for example, a USB (Universal
Serial Bus) port, an IEEE 1394 port, or a SCSI (Small Computer System Interface)
port. In addition, the connection port 923 may be an RS-232 port, an optical audio
terminal, or a HDMI (High-Definition Multimedia Interface) port. When the external
connection device 929 is connected to the connection port 923, the information
processing device 900 and the external connection device 929 can exchange various
data.
[01 19] The communication device 925 is, for example, a communication interface including
a communication device or the like for connection to a communications network 931.
The communication device 925 can be, for example, a wired or wireless LAN (Local
Area Network) or a communication card for Bluetooth (registered trademark) or
WUSB (Wireless USB). Alternatively, the communication device 925 may be a router
for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line),
or a modem for various communication. The communication device 925 transmits or
receives signals or the like via the Internet or to/from other communication devices, for
example, using a predetermined protocol such as TCP/IP. In addition, the commu
nications network 931 connected to the communication device 925 is a network
connected by wire or wirelessly, and is, for example, the Internet, a home LAN,
infrared communication, radio wave communication, or satellite communication.
[0120] The image pickup device 933 is, for example, an apparatus which captures a real
world and generates a captured image by using image sensor such as CCD (Charge
Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) and various
components such as lens for picking up a subject image to the image sensor. The image
device 933 may be configured to pick up still images or moving images.
[0121] The sensor 935 may be various types of sensors such as an acceleration sensor, a
gyro sensor, a geomagnetic sensor, an optical sensor, and an acoustic sensor. The
sensor 935 acquires information related to the state of an information processing
apparatus 900 such as the shape of housing of the information processing apparatus
900 and information related to surrounding environment of the information processing
apparatus 900 such as brightness or noise in surroundings of the information
processing apparatus 900. Moreover, the sensor 935 may include a GPS (Global Po
sitioning System) sensor which receives a GPS signal and measures latitude, longitude
and altitude of the apparatus.
[0122] An example of the hardware configuration of the information processing apparatus
900 has been described. The respective components described above may be
configured using general purpose elements, and may be configured by hardware spe
cialized to the function of the respective components. Such configurations can be ap
propriately changed according to the technical level at the time of implementing the
present embodiments.
[0123] Although preferred embodiments of the present disclosure are described in detail
above with reference to the appended drawings, the disclosure is not limited thereto. It
should be understood by those skilled in the art that various modifications, com
binations, sub-combinations and alterations may occur depending on design re
quirements and other factors insofar as they are within the scope of the appended
claims or the equivalents thereof.
[0124] Additionally, the following configurations are also within the technical scope of the
present disclosure.
(1) An apparatus including:
an image processor configured to
receive a video including an object,
determine a positional relationship between the apparatus and the object,
change a positional relationship between an image superimposed on the video and
the object when the positional relationship between the apparatus and the object
changes.
(2) The apparatus according to (1), wherein the image processor determines an angle
between a direction of the object and a direction of the apparatus and changes the p o
sitional relationship between the image superimposed on the video and the object
based on the angle.
(3) The apparatus according to (2), wherein the image processor determines that the
positional relationship between the image superimposed on the video and the object is
a front view relationship when the angle is below a first threshold angle.
(4) The apparatus according to (3), wherein the image processor changes the p o
sitional relationship between the image superimposed on the video and the object from
the front view relationship to an oblique view relationship when the angle exceeds the
first threshold angle.
(5) The apparatus according to (4), wherein the image processor changes the positional
relationship between the image superimposed on the video and the object from the
oblique relationship to the front view relationship when the angle drops below a
second threshold angle.
(6) The apparatus according to (5), wherein the second threshold angle is lower than
the first threshold angle.
(7) The apparatus according to (4), wherein the image processor superimposes the
image on the video in a plane of the object for the front view relationship and su
perimposes the image on the video above the object for the oblique view relationship.
(8) The apparatus according to (1) to (7), wherein the image processor superimposes a
text message on the video with the image.
(9) The apparatus according to (8), wherein the image processor superimposes a text
message on the video with the image when the image is modified to fit on a display
screen.
(10) The apparatus according to (9), wherein the image processor superimposes a text
message providing instructions to eliminate modification of the image to fit on the
display screen.
(11) The apparatus according to (1), wherein the image processor determines a
distance between the object and the apparatus and changes the positional relationship
between the image superimposed on the video and the object based on the distance.
(12) The apparatus according to ( 11), wherein the image processor changes the image
superimposed on the video based on the distance.
(13) The apparatus according to (12), wherein the image processor changes the image
superimposed on the video based on the distance such that additional detail is
displayed when the distance is less than a first threshold distance.
(14) The apparatus according to (13), wherein the image processor changes the image
superimposed on the video based on the distance such that additional detail is removed
when the distance exceeds a second threshold distance.
(15) The apparatus according to (14), wherein the second threshold distance is shorter
than the first threshold distance.
(16) The apparatus according to (1) to (15), further comprising:
a receiver configured to receive the image from an external device.
(17) The apparatus according to (1) to (16), further comprising:
a display configured to display the image superimposed on the video.
(18) The apparatus according to (1) to (17), further comprising:
a sensor configured measure positional information of the apparatus.
(19) A method including:
receiving a video including an object,
determining a positional relationship between the apparatus and the object,
changing a positional relationship between an image superimposed on the video and
the object when the positional relationship between the apparatus and the object
changes.
(20) A non-transitory computer readable medium encoded with a program that, when
loaded on a processor, causes the processor to perform a method including:
receiving a video including an object,
determining a positional relationship between the apparatus and the object,
changing a positional relationship between an image superimposed on the video and
the object when the positional relationship between the apparatus and the object
changes.
Reference Signs List
100, 200, 300 Information processing apparatus
102 Image pickup unit
104 Image recognition unit
108 Positional relationship detection unit
110 Selection unit
114 Display target information output unit
116 Message information output unit
118 Display control unit
120 Display unit
222 Sensor unit
324 Communication unit
obj_r Real object
obj_v Virtual object
PCT7JP2012/005904
Claims
[Claims 1] An apparatus comprising:
an image processor configured to
receive a video including an object,
determine a positional relationship between the apparatus and the
object,
change a positional relationship between an image superimposed on the
video and the object when the positional relationship between the
apparatus and the object changes.
[Claims 2] The apparatus according to claim 1, wherein the image processor de
termines an angle between a direction of the object and a direction of
the apparatus and changes the positional relationship between the
image superimposed on the video and the object based on the angle.
[Claims 3] The apparatus according to claim 2, wherein the image processor de
termines that the positional relationship between the image su
perimposed on the video and the object is a front view relationship
when the angle is below a first threshold angle.
[Claims 4] The apparatus according to claim 3, wherein the image processor
changes the positional relationship between the image superimposed on
the video and the object from the front view relationship to an oblique
view relationship when the angle exceeds the first threshold angle.
[Claims 5] The apparatus according to claim 4, wherein the image processor
changes the positional relationship between the image superimposed on
the video and the object from the oblique relationship to the front view
relationship when the angle drops below a second threshold angle.
[Claims 6] The apparatus according to claim 5, wherein the second threshold angle
is lower than the first threshold angle.
[Claims 7] The apparatus according to claim 4, wherein the image processor su
perimposes the image on the video in a plane of the object for the front
view relationship and superimposes the image on the video above the
object for the oblique view relationship.
[Claims 8] The apparatus according to claim 1, wherein the image processor su
perimposes a text message on the video with the image.
[Claims 9] The apparatus according to claim 8, wherein the image processor su
perimposes a text message on the video with the image when the image
is modified to fit on a display screen.
[Claims 10] The apparatus according to claim 9, wherein the image processor superimposes a text message providing instructions to eliminate modi
fication of the image to fit on the display screen.
[Claims 11] The apparatus according to claim 1, wherein the image processor de
termines a distance between the object and the apparatus and changes
the positional relationship between the image superimposed on the
video and the object based on the distance.
[Claims 12] The apparatus according to claim 11, wherein the image processor
changes the image superimposed on the video based on the distance.
[Claims 13] The apparatus according to claim 12, wherein the image processor
changes the image superimposed on the video based on the distance
such that additional detail is displayed when the distance is less than a
first threshold distance.
[Claims 14] The apparatus according to claim 13, wherein the image processor
changes the image superimposed on the video based on the distance
such that additional detail is removed when the distance exceeds a
second threshold distance.
[Claims 15] The apparatus according to claim 14, wherein the second threshold
distance is shorter than the first threshold distance.
[Claims 16] The apparatus according to claim 1, further comprising:
a receiver configured to receive the image from an external device.
[Claims 17] The apparatus according to claim 1, further comprising:
a display configured to display the image superimposed on the video.
[Claims 18] The apparatus according to claim 1, further comprising:
a sensor configured measure positional information of the apparatus.
[Claims 19] A method comprising:
receiving a video including an object,
determining a positional relationship between the apparatus and the
object,
changing a positional relationship between an image superimposed on
the video and the object when the positional relationship between the
apparatus and the object changes.
[Claims 20] A non-transitory computer readable medium encoded with a program
that, when loaded on a processor, causes the processor to perform a
method comprising:
receiving a video including an object,
determining a positional relationship between the apparatus and the
object,
changing a positional relationship between an image superimposed on
PCT/JP2012/005904
the video and the object when the positional relationship between the
apparatus and the object changes.