Description
Title of Invention: IMAGE PROCESSING APPARATUS, IMAGE
PROCESSING METHOD, AND PROGRAM
Technical Field
[0001] The present disclosure relates to an image processing apparatus, an image processing
method, and a program.
Background Art
[0002] In recent years, attention has been focused on a technology called augmented reality
(AR) that presents virtual content to the user by overlaying such content onto a real
space. The content presented to the user by AR technology may be visualized in a
variety of forms, such as text, icons, and animations.
[0003] In AR technology, content to be overlaid on an image may be selected according to a
variety of criteria. One of such criteria is recognition of an object associated in advance
with content. As one example, JP2010-170316A discloses a technique that detects a
marker, which is an object on which a specified pattern is drawn, in an image and
overlays content associated with the detected marker at the detected position of the
marker.
Citation List
Patent Literature
[0004] PTL 1: JP 2010-170316A
Summary
Technical Problem
[0005] However, with an AR technique based on the detection of markers as described
above, it is normally difficult to continue the displaying of AR content once a marker
has been lost from the image. Also, even if the displaying of AR content were
continued after a marker was lost from the image, there would be a tendency for the
displaying of AR content to not reflect the state of the real space and therefore appear
unnatural.
[0006] Accordingly, it would be desirable to realize an arrangement capable of continuing
the displaying of AR content in a natural state even after an object that acts as a marker
has been lost from the image.
Solution to Problem
[0007] According to an embodiment of the present disclosure, there is provided an in
formation processing system comprising: one or more processing units that: acquire
video data captured by an image pickup unit; detect an object from the video data;
detect a condition corresponding to the image pickup unit; and control a display to
display content associated with the object at a position other than a detected position of
the object based on the condition corresponding to the image pickup unit.
[0008] According to another embodiment of the present disclosure, there is provided an in
formation processing method performed by an information processing system, the
method comprising: acquiring video data captured by an image pickup unit; detecting
an object from the video data; detecting a condition corresponding to the image pickup
unit; and controlling a display to display content associated with the object at a
position other than a detected position of the object based on the condition corre
sponding to the image pickup unit.
[0009] According to still another embodiment of the present disclosure, there is provided a
non-transitory computer-readable medium including computer program instructions,
which when executed by an information processing system, cause the information
processing system to perform a method, the method comprising: acquiring video data
captured by an image pickup unit; detecting an object from the video data; detecting a
condition corresponding to the image pickup unit; and controlling a display to display
content associated with the object at a position other than a detected position of the
object based on the condition corresponding to the image pickup unit.
Advantageous Effects of Invention
[0010] According to the above embodiments of the present disclosure, an arrangement
capable of continuing the displaying of AR content in a natural state even after an
object that acts as a marker has been lost from the image is realized.
Brief Description of Drawings
[001 1] [fig. 1]Fig. 1 is a diagram useful in explaining an overview of an information
processing apparatus according to an embodiment of the present disclosure.
[fig.2A]Fig. 2A is a diagram useful in explaining one example of a marker that can be
detected by the present embodiment.
[fig.2B]Fig. 2B is a diagram useful in explaining another example of a marker that can
be detected by the present embodiment.
[fig.3]Fig. 3 is a block diagram showing one example of the hardware configuration of
the information processing apparatus according to the present embodiment.
[fig.4]Fig. 4 is a block diagram showing one example of the logical functional con
figuration of the information processing apparatus according to the present em
bodiment.
[fig.5]Fig. 5 is a flowchart showing one example of the flow of an analyzing process
according to an analyzing unit illustrated in Fig. 4.
[fig.6]Fig. 6 is a diagram useful in explaining one example of the configuration of
feature point information.
[fig.7]Fig. 7 is a diagram useful in explaining one example of the configuration of
camera position/posture information.
[fig.8]Fig. 8 is a diagram useful in explaining one example of the configuration of
marker basic information.
[fig.9]Fig. 9 is a diagram useful in explaining one example of the configuration of
marker detection information.
[fig. 10] Fig. 10 is a diagram useful in explaining one example of the configuration of
content information.
[fig. 1l]Fig. 11 is a diagram useful in explaining a first example of a removal condition
of AR content.
[fig. 12] Fig. 12 is a diagram useful in explaining a second example of a removal
condition of AR content.
[fig.l3A]Fig. 13A is a diagram useful in explaining a first example of displaying of
AR content according to the present embodiment.
[fig.l3B]Fig. 13B is a diagram useful in explaining a second example of displaying of
AR content according to the present embodiment.
[fig.l3C]Fig. 13C is a diagram useful in explaining a third example of displaying of
AR content according to the present embodiment.
[fig.l3D]Fig. 13D is a diagram useful in explaining a fourth example of displaying of
AR content according to the present embodiment.
[fig.l4]Fig. 14 is a flowchart showing one example of a flow of image processing
according to the same embodiment.
Description of Embodiments
[0012] Hereinafter, preferred embodiments of the present disclosure will be described in
detail with reference to the appended drawings. Note that, in this specification and the
appended drawings, structural elements that have substantially the same function and
structure are denoted with the same reference numerals, and repeated explanation of
these structural elements is omitted.
[0013] The following description is given in the order indicated below.
1. Overview
2. Example Configuration of Image Processing Apparatus According to Embodiment
of the Present Disclosure
2-1. Hardware Configuration
2-2. Functional Configuration
2-3. Example Displaying of AR Content
2-4. Flow of Processing
3. Conclusion
[0014] 1. Overview
First, an overview of an image processing apparatus according to an embodiment of
the present disclosure will be described with reference to Figs. 1 to 2B.
[0015] Fig. 1 is a diagram useful in explaining an overview of an image processing
apparatus 100 according to an embodiment of the present disclosure. Fig. 1 shows the
image processing apparatus 100 in the possession of the user Ua. The image processing
apparatus 100 is equipped with an image pickup unit 102 (hereinafter sometimes
referred to simply as the "camera") that picks up images of a real space 1 and a display
unit 110. In the example in Fig. 1, a table 11, a coffee cup 12, a book 13, and a poster
14 are present in the real space 1. The image pickup unit 102 of the image processing
apparatus 100 picks up a series of images that compose video images produced by
image pickup of the real space 1. The image processing apparatus 100 then carries out
image processing with an image picked up by the image pickup unit 102 as an input
image to generate an output image. In the present embodiment, the output image is
typically generated by overlaying virtual content (hereinafter referred to as "AR
content") for augmented reality (AR) onto the input image. The display unit 110 of the
image processing apparatus 100 successively displays the generated output images.
Note that the real space 1 shown in Fig. 1 is merely one example. The input image
processed by the image processing apparatus 100 may be any image in which a real
space appears.
[0016] The provision of AR content by the image processing apparatus 100 may start with
detection of a marker appearing in an input image as a trigger. In this specification, the
term "marker" typically refers to any kind of object present in the real space that has a
known pattern. That is, the term "marker" may include a shape, symbol, character
string or design shown on a real object, part of a real object, or the surface of a real
object, or an image or the like displayed on a display. Although there are cases where
as a narrow definition, the term "marker" refers to a special object provided for some
kind of application, the technology according to the present disclosure is not limited to
such a definition.
[0017] Note that in Fig. 1, a smartphone is shown as one example of the image processing
apparatus 100. However, the image processing apparatus 100 is not limited to this
example. As examples, the image processing apparatus 100 may be a PC (Personal
Computer), a PDA (Personal Digital Assistant), a game console, a PND (Portable
Navigation Device), a content player, or a digital home appliance.
[0018] Fig. 2A is a diagram useful in explaining one example of a marker that can be
detected in the present embodiment. Fig. 2A shows an input image ImOl as one
example that can be acquired by the image processing apparatus 100 illustrated in Fig.
1. The table 11, the coffee cup 12, and the poster 14 appear in the input image ImOl. A
marker 20a that is a known design is printed on the poster 14. If the image processing
apparatus 100 detects such a marker 20a in the input image ImOl, content associated
with the marker 20a may be overlaid on the input image ImOl.
[0019] Fig. 2B is a diagram useful in explaining another example of a marker that may be
detected in the present embodiment. Fig. 2B shows an input image Im02. The table 11
and the book 13 appear in the input image Im02. A marker 20b that is a known design
is printed on the book 13. If the image processing apparatus 100 detects such a marker
20b in the input image Im02, content associated with the marker 20b may be overlaid
on the input image Im02. In place of the marker 20b illustrated in Fig. 2B, the image
processing apparatus 100 may use a marker 20c that is a known character string.
[0020] After a marker has been detected in the input image as described above, in some
cases the marker will stop being detected from the input image due to the camera
moving or the posture of the camera changing. In such case, with typical AR
technology that is based on the detection of markers, it is difficult to continue
displaying the AR content. If the displaying of AR content is continued even after a
marker has been lost, the display will become unnatural, such as by having AR content
displayed that is unrelated to the position or posture of the marker.
[0021] For this reason, in the present embodiment, to eliminate or reduce the unnatural
displaying of AR content, the image processing apparatus 100 tracks the position and
posture of the camera in the three-dimensional real space and manages the positions
and postures of the detected markers using a database. As described in detail later, the
image processing apparatus 100 then controls the behavior of AR content based on at
least one of the position and posture of the camera relative to the markers.
[0022] 2. Example Configuration of Image Processing Apparatus According to an Em
bodiment
2-1. Hardware Configuration
Fig. 3 is a block diagram showing one example of the hardware configuration of the
image processing apparatus 100 according to the present embodiment. As shown in
Fig. 3, the image processing apparatus 100 includes the image pickup unit 102, a
sensor unit 104, an input unit 106, a storage unit 108, a display unit 110, a commu
nication unit 112, a bus 116, and a control unit 118.
[0023] (1) Image Pickup Unit
The image pickup unit 102 is a camera module that picks up an image. The image
pickup unit 102 picks up images of a real space using an image pickup element such as
a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semi
conductor) to generate a picked-up image. A series of the picked-up images generated
by the image pickup unit 102 compose video images in which the real space appears.
Note that the image pickup unit 102 does not need to be part of the image processing
apparatus 100. As one example, an image pickup apparatus connected to the image
processing apparatus 100 wirelessly or using wires may be treated as the image pickup
unit 102.
[0024] (2) Sensor Unit
The sensor unit 104 may include a variety of sensors such as a positioning sensor, an
acceleration sensor, and a gyrosensor. The position, posture, or movement of the image
processing apparatus 100 that can be measured by the sensor unit 104 may be used for
a variety of applications such as supporting recognition of the position and posture of a
camera, described later, acquisition of data that specifies a global position, or
recognition of instructions from the user. Note that the sensor unit 104 may be omitted
from the configuration of the image processing apparatus 100.
[0025] (3) Input Unit
The input unit 106 is an input device used by the user to operate the image
processing apparatus 100 or to input information into the image processing apparatus
100. As one example, the input unit 106 may include a touch sensor that detects
touches made by the user on the screen of the display unit 110. In place of (or in
addition to) this, the input unit 106 may include a pointing device such as a mouse or a
touch pad. In addition, the input unit 106 may include another type of input device
such as a keyboard, a keypad, a button or buttons, or a switch or switches.
[0026] (4) Storage Unit
The storage unit 108 is constructed of a storage medium such as a semiconductor
memory or a hard disk drive and stores programs and data for processing by the image
processing apparatus 100. The data stored by the storage unit 108 may include pickedup
image data, sensor data, and data in a variety of databases (DB), described later.
Note that instead of being stored in the storage unit 108, some of the programs and
data described in the present specification may be acquired from an external data
source (as examples, a data server, network storage, or an external memory).
[0027] (5) Display Unit
The display unit 110 is a display module including a display such as an LCD (Liquid
Crystal Display), an OLED (Organic Light-Emitting Diode), or a CRT (Cathode Ray
Tube). As one example, the display unit 110 is used to display an output image
generated by the image processing apparatus 100. Note that the display unit 110 also
does not need to be part of the image processing apparatus 100. As one example, a
display apparatus connected to the image processing apparatus 100 wirelessly or using
wires may be treated as the display unit 110.
[0028] (6) Communication Unit
The communication unit 112 is a communication interface that serves as a mediator
for communication by the image processing apparatus 100 with other apparatuses. The
communication unit 112 supports an arbitrary wireless communication protocol or
wired communication protocol and establishes a communication connection with other
apparatuses.
[0029] (7) Bus
The bus 116 connects the image pickup unit 102, the sensor unit 104, the input unit
106, the storage unit 108, the display unit 110, the communication unit 112, and the
control unit 118 to one another.
[0030] (8) Control Unit
The control unit 118 corresponds to a processor such as a CPU (Central Processing
Unit) or a DSP (Digital Signal Processor). By executing a program stored in the
storage unit 108 or another storage medium, the control unit 118 causes the image
processing apparatus 100 to function in a variety of ways as described later.
[0031] 2-2. Functional Configuration
Fig. 4 is a block diagram showing one example of a logical functional configuration
realized by the storage unit 108 and the control unit 118 of the image processing
apparatus 100 shown in Fig. 3. As shown in Fig. 4, the image processing apparatus 100
includes an image acquiring unit 120, an analyzing unit 125, a three-dimensional (3D)
structure database (DB) 130, a marker DB 135, a marker detecting unit 140, a marker
managing unit 145, a content DB 150, a content control unit 155, and a display control
unit 160.
[0032] (1) Image Acquiring Unit
The image acquiring unit 120 acquires the picked-up image generated by the image
pickup unit 102 as an input image. The input image acquired by the image acquiring
unit 120 may be an individual frame that composes video images produced by image
pickup of a real space. The image acquiring unit 120 outputs the acquired input image
to the analyzing unit 125, the marker detecting unit 140, and the display control unit
160.
[0033] (2) Analyzing Unit
The analyzing unit 125 analyzes the input image inputted from the image acquiring
unit 120 to recognize the three-dimensional position and posture in the real space of
the apparatus that picked up the input image. The analyzing unit 125 also recognizes
the three-dimensional structure of the peripheral environment of the image processing
apparatus 100 and stores the recognized three-dimensional structure in the 3D structure
DB 130. In the present embodiment the analyzing process performed by the analyzing
unit 125 is carried out according to SLAM (Simultaneous Localization And Mapping).
The fundamental principles of SLAM are disclosed in "Real-Time Simultaneous Lo
calization and Mapping with a Single Camera" (Andrew J . Davison, Proceedings of the
9th IEEE International Conference on Computer Vision Volume 2, 2003,
pp. 1403- 1410). Note that the present disclosure is not limited to this example and the
analyzing unit 125 may analyze the input image using any other three-dimensional en
vironment recognition technique.
[0034] One characteristic of SLAM is that it is possible to dynamically recognize the threedimensional
structure of a real space appearing in an input image from a single
(monocular) camera in parallel with the position and posture of such camera. Fig. 5
shows one example of the analyzing process carried out by the analyzing unit 125.
[0035] In Fig. 5, the analyzing unit 125 first initializes state variables (step S101). The state
variables initialized here include at least the position and posture (rotational angle) of
the camera and the movement speed and angular velocity of the camera, with the threedimensional
position of at least one feature point appearing in the input image also
being added to the state variables. Input images acquired by the image acquiring unit
120 are successively inputted into the analyzing unit 125 (step S102). The processing
in step SI03 to step S105 may be repeated for each input image (that is, for each
frame).
[0036] In step S103, the analyzing unit 125 tracks the feature points appearing in the input
image. For example, the analyzing unit 125 matches a patch (for example, a small
image composed of nine pixels in a 3 by 3 grid centered on a feature point) for each
feature point included in the state variables against a new input image. The analyzing
unit 125 then detects the position of each patch in the input image, that is, the positions
of the feature points. The positions of the feature points detected here are used when
subsequently updating the state variables.
[0037] In step S104, the analyzing unit 125 generates predicted values of the state variables
of the next frame, for example, based on a specified prediction model. In step S105,
the analyzing unit 125 uses the predicted values of the state variables generated in step
SI04 and observed values in keeping with the positions of the feature points detected
in step S103 to update the state variables. The analyzing unit 125 carries out the
processing in step S104 and S105 based on the principles of an extended Kalman filter.
Note that such processing is described in detail in JP201 1-159163A, for example.
[0038] By carrying out such analyzing process, parameters included in the state variables are
updated in each frame. The number of feature points included in the state variables
may increase or decrease in each frame. That is, if the field of view of the camera
changes, parameters of feature points in a region that has newly entered the frame may
be added to the state variables and parameters of feature points in a region that has left
the frame may be deleted from the state variables.
[0039] The analyzing unit 125 stores the position and posture of the camera that are updated
in this way for each frame in a time series in the 3D structure DB 130. The analyzing
unit 125 also stores the three-dimensional positions of the feature points included in
the state variables for SLAM in the 3D structure DB 130. Information on the feature
points is gradually accumulated in the 3D structure DB 130 in keeping with movement
of the field of view of the camera.
[0040] Note that an example where the analyzing unit 125 uses SLAM to recognize both the
position and the posture of the image pickup unit 102 is described here. However, the
present disclosure is not limited to this example and it is also possible to recognize the
position or the posture of the image pickup unit 102 based on sensor data from the
sensor unit 104, for example.
[0041] (3) 3D Structure DB
The 3D structure DB 130 is a database storing feature point information 131 used in
the analyzing process by the analyzing unit 125 and camera position/posture in
formation 132 recognized as the result of the analyzing process.
[0042] Fig. 6 is a diagram useful in explaining one example configuration of the feature
point information 131. As shown in Fig. 6, the feature point information 131 includes
four data items, namely "feature point ID", "position", "patch", and "updated time".
The "feature point ID" is an identifier for uniquely identifying each feature point. The
"position" is a three-dimensional vector expressing the position of each feature point in
the real space. The "patch" is image data of a small image used to detect each feature
point in an input image. The "updated time" expresses the time when each record was
updated. In the example in Fig. 6, information on the two feature points FP01 and
FP02 is shown. However, in reality, information on a larger number of feature points
may be stored by the 3D structure DB 130 as the feature point information 131.
[0043] Fig. 7 is a diagram useful in explaining one example of the configuration of the
camera position/posture information 132. As shown in Fig. 7, the camera position/
posture information 132 includes the following three data items "time", "camera
position", and "camera posture". The "time" expresses the time at which each record
was stored. The "camera position" is a three-dimensional vector showing the position
of the camera recognized at each time as the result of the analyzing process. The
"camera posture" is a rotational angle vector showing the posture of the camera
recognized at each time as the result of the analyzing process. The camera position and
posture tracked in this way are used by the content control unit 155, described later, to
control behavior of AR content and by the display control unit 160 to control the
displaying of AR content.
[0044] (4) Marker DB
The marker DB 135 is a database storing information on at least one marker a s
sociated with content disposed in the AR space. In the present embodiment, the in
formation stored by the marker DB 135 includes marker basic information 136 and
marker detection information 137.
[0045] Fig. 8 is a diagram useful in explaining one example of the configuration of the
marker basic information 136. As shown in Fig. 8, the marker basic information 136
includes three data items, namely "marker ID", "related content ID" and "size", in
addition to a "marker image". The "marker ID" is an identifier for uniquely identifying
each marker. The "related content ID" is an identifier for identifying content associated
with each marker. The "marker image" is image data of a known marker image used to
detect each marker in an input image. Note that in place of a marker image, it is also
possible to use a set of feature amounts extracted from each marker image to detect
each marker. In the example in Fig. 8, an image in which a lion is drawn is used as the
marker image of the marker M01 and an image in which an elephant is drawn is used
as the marker image of the marker M02. The "size" expresses the assumed size of each
marker image in the real space. Such marker basic information 136 may be stored in
advance by the marker DB 135. As an alternative, the marker basic information 136
may be stored in advance by an external server and selectively downloaded to the
marker DB 135 in keeping with the position of the image processing apparatus 100 or
the object of the provided AR application, for example.
[0046] (5) Marker Detecting Unit
The marker detecting unit 140 detects markers present in the real space from the
input image. As a specific example, the marker detecting unit 140 extracts feature
amounts of the input image and feature amounts of the respective marker images
included in the marker basic information 136 in accordance with some kind of feature
amount extraction algorithm. The marker detecting unit 140 then matches the extracted
feature amounts of the input image against the feature amounts of each marker image.
When a marker appears in the input image, this is indicated by a high matching score
for the region in which such marker appears. By doing so, the marker detecting unit
140 is capable of detecting a marker that is present in the real space and appears in the
input image. As examples, the feature amount extraction algorithm used by the marker
detecting unit 140 may be Random Ferns described in "Fast Keypoint Recognition
using Random Ferns" (Mustafa Oezuysal, IEEE Transactions on Pattern Analysis and
Machine Intelligence, Vol.32, Nr.3, pp.448-461, March 2010) or SURF described in
"SURF: Speeded Up Robust Features" (H. Bay, A. Ess, T. Tuytelaars and L.V. Gool,
Computer Vision and Image Understanding (CVIU), Vol.110, No.3, pp.346-359,
2008).
[0047] In addition, the marker detecting unit 140 estimates the three-dimensional position
and posture of a marker in the real space based on the position of the detected marker
in the input image (i.e., the two-dimensional position on the image pickup plane) and
the marker size and form in the input image. The estimation carried out here may be
part of the matching process for feature points described above. The marker detecting
unit 140 then outputs the marker ID of the detected marker and also the estimated
three-dimensional position and posture of the marker to the marker managing unit 145.
[0048] (6) Marker Managing Unit
When a new marker appearing in the input image has been detected by the marker
detecting unit 140, the marker managing unit 145 stores the marker ID, the position
and posture in the real space, and the detection time of the new marker in the marker
DB 135. Also, if a marker that has previously been detected is lost from the input
image (due to a reason such as movement that places the marker outside the field of
view or the marker being blocked by an obstacle), the marker managing unit 145 may
also store a lost time of the marker that has been lost in the marker DB 135.
[0049] Fig. 9 is a diagram useful in explaining one example of the configuration of the
marker detection information 137 stored by the marker DB 135. As shown in Fig. 9,
the marker detection information 137 has five data items, namely "marker ID",
"position", "posture", "detection time", and "lost time". The "marker ID" corresponds
to the marker ID in the marker basic information 136 illustrated in Fig. 8. The
"position" is a three-dimensional vector expressing the estimated position in the real
space of each marker. The "posture" is a rotational angle vector expressing the
estimated posture of each marker. The "detection time" expresses the time at which
each marker was detected. The "lost time" expresses the time at which a marker that
has previously been detected stops being detected. In the example in Fig. 9, the lost
time LI is stored for the marker M01. This means that after the marker M01 was
detected, the marker M01 was lost from the input image at time LI. Meanwhile,
although a detection time D2 is stored for the marker M02, no lost time is stored. This
means that after being detected at time D2, the marker M02 continues to appear in the
input image as before. The parameters for respective markers that are managed in this
way are used by the content control unit 155, described later, to control the behavior of
AR content.
[0050] (7) Content DB
The content DB 150 is a database storing content information 151 used to control and
display at least one AR content item associated with the markers described above.
[0051] Fig. 10 is a diagram useful in explaining one example of the configuration of the
content information 151. As shown in Fig. 10, the content information 151 includes a
content ID and attributes, and also drawing data. The "content ID" is an identifier that
uniquely identifies each AR content item. In the example in Fig. 10, "type" and
"control parameter set" are shown as the attributes of the AR content. The "type" is an
attribute used to classify the AR content. The AR content may be classified according
to a variety of viewpoints, such as the type of associated marker, the type of character
expressed by the AR content, or the type of application providing the AR content. The
"control parameter set" may include at least one control parameter used to control the
behavior of AR content, described later.
[0052] In the example in Fig. 10, two types of drawing data, namely "near" and "remote",
are defined for each AR content item. As one example, such drawing data is CG
(Computer Graphics) data for modeling AR content. The two types of drawing data
differ in terms of display resolution. The display control unit 160, described later,
switches between which of such drawing data is to be used to display the AR content
based on the position or posture of the camera relative to the detected marker.
[0053] The content information 151 may be stored in advance in the content DB 150. As an
alternative, in the same way as the marker basic information 136 described earlier, the
content information 151 may be stored in advance in an external server and selectively
downloaded to the content DB 150 in keeping with the position of the image
processing apparatus 100 or the object of the provided AR application, for example.
[0054] (8) Content Control Unit
The content control unit 155 controls the behavior of AR content associated with a
detected marker in the AR space based on at least one of the camera position and the
camera posture relative to the detected marker that is being tracked using the marker
detection information 137 described above. In the present specification, the expression
"behavior of AR content" includes the appearance and removal (disappearance) of AR
content in the AR space and movement of the AR content.
[0055] (8- 1) Appearance of AR Content
When a new marker appearing in the input image has been detected by the marker
detecting unit 140 for example, the content control unit 155 has the AR content a s
sociated with such new marker in the marker basic information 136 appear in the AR
space. The AR content may appear instantly in keeping with detection of the associated
marker or may appear when a specified appearance condition has also been satisfied.
As one example, the expression "specified appearance condition" may refer to a
condition that a distance from the marker to the present camera position is below a
specified distance threshold. In such case, even if a marker appears in the input image,
the AR content will not appear if the distance from such marker to the camera position
is far and the AR content will only appear when the camera position moves closer to
the marker. Such distance threshold may be commonly defined for a plurality of AR
content items or may be defined as a control parameter for each AR content item.
[0056] (8-2) Movement of AR Content
The content control unit 155 moves the AR content in the AR space in accordance
with a change in at least one of the position and posture of the camera. For example,
the content control unit 155 recognizes an operation such as panning or tilting of the
camera by the user from a change in the camera posture (for example, a change in the
angle of the optical axis that exceeds a specified amount of change.) As examples, the
content control unit 155 may then change the orientation of the AR content in keeping
with the panning and move the AR content forward or backward in keeping with the
tilting. Note that the mapping between such types of operation and the movement of
the AR content is not limited to this example.
[0057] If a detected marker has moved outside the field of view of the input image, the
content control unit 155 may move the AR content associated with such marker in the
AR space so that the AR content is kept within the field of view of the new input
image. The three-dimensional position to which the AR content is moved may be
decided from the feature point positions stored by the 3D structure DB 130.
[0058] If the AR content is an image of a character capable of expressing a line of sight (i.e.,
looking in a certain direction) such as those illustrated in Fig. 10, it is possible for the
content control unit 155 to point the line of sight of the character toward the camera
based on the position of the camera relative to the position of the character in the AR
space.
[0059] (8-3) Removal of AR Content
In the present embodiment, as described earlier, the AR content is not necessarily
removed (i.e., does not necessarily disappear) when the associated marker has moved
out of the field of view of the input image. However, if AR content endlessly
continued to be displayed regardless of the position and posture of the camera, this
would conversely appear unnatural to the user. For this reason, in the present em
bodiment, the content control unit 155 removes AR content if at least one of the
camera position and camera posture relative to a detected marker satisfies a specified
removal condition. As examples, any of the following conditions A to D or a com
bination thereof may be used as the specified removal condition.
Condition A: the distance from the marker to the camera position exceeds a specified
distance threshold.
Condition B: the angle made between the optical axis of the camera and the direction
from the camera to the marker exceeds a specified angle threshold.
Condition C: the time elapsed since the detection time of the marker exceeds a
specified time threshold.
Condition D: the time elapsed since the lost time of the marker exceeds a specified
time threshold.
The distance threshold, angle threshold, and time thresholds referred to here may be
commonly defined for a plurality of AR content items or may be defined as control p a
rameters for each AR content item.
[0060] Fig. 11 is a diagram useful in explaining the removal condition A for AR content. In
Fig. 11, the real space 1 described earlier is shown again. In Fig. 11, a point PI is the
detection position of a marker 20a and a broken line DLl shows a boundary where the
distance from the point PI is equal to the distance threshold d^. The distance of an
image processing apparatus 100a from the marker 20a is below the distance threshold
dt l . In this case, the content control unit 155 of the image processing apparatus 100a
does not remove the AR content 32a associated with the marker 20a and moves the AR
content 32a within a field of view 30a of the image processing apparatus 100a. After
this, assume that the apparatus has moved for example from the position of the image
processing apparatus 100a to the position of the image processing apparatus 100b. The
distance from the marker 20a to an image processing apparatus 100b exceeds the
distance threshold d^. In this case, the content control unit 155 removes the AR
content 32a associated with the marker 20a. That is, the AR content 32a does not
appear in a field of view 30b of the image processing apparatus 100b.
[0061] Fig. 12 is a diagram useful in explaining one example of the removal condition B for
AR content. In Fig. 12, the real space 1 described earlier is shown again. In Fig. 12,
point PI shows a detection position of the marker 20a. The distance from the marker
20a to an image processing apparatus 100c is assumed to be shorter than the specified
distance threshold. However, the angle ropt between the optical axis Vopt of the image
pickup unit 102 of the image processing apparatus 100c and the direction Vm k from
the image pickup unit 102 to the marker 20a exceeds a specified angle threshold (not
shown). In this case, the content control unit 155 of the image processing apparatus
100c removes the AR content 32a associated with the marker 20a.
[0062] Note that regardless of these removal conditions A and B, the content control unit
155 may remove the AR content associated with a marker when, as shown in removal
conditions C and D given above, the time elapsed from the detection time of the
marker or the time elapsed from the lost time of the marker exceeds a specified time
threshold. Also, the AR content associated with a marker may be removed when
removal condition A or B is satisfied and the time elapsed from the detection time of
the marker or the time elapsed from the lost time of the marker exceeds a specified
time threshold.
[0063] By controlling the behavior of AR content in this way, an unnatural state where AR
content endlessly continues to be displayed regardless of the position and posture of
the camera is prevented. Overcrowding of AR content due to the displaying of a large
number of AR content items is also avoided. In particular, in the present embodiment,
the removal of AR content is controlled in keeping with the position or posture of the
camera relative to a marker. This means that it is possible to remove AR content if the
user has stopped being interested in such content (for example, if the user has moved
away from the marker or is now picking up images in a completely different direction
to the marker). That is, the life cycle from appearance to removal of AR content can be
appropriately managed in keeping with the state of the user.
[0064] (8-4) Coexistence of AR Content
The content control unit 155 may control the coexistence of a plurality of AR content
items associated with different markers based on the camera position or posture
relative to such markers. For example, the content control unit 155 may select one of
the two following control options when a second marker is newly detected in a state
where a first AR content item associated with the first marker is already disposed in
the AR space.
Option A: dispose the second AR content item associated with the second marker in
the AR space in addition to the first AR content item.
Option B: dispose the second AR content item associated with the second marker in
the AR space in place of the first AR content item.
[0065] As one example, the content control unit 155 may select Option A if the distance
from the first marker to the camera position is below a specified distance threshold
when the second marker is detected and may select Option B if such distance is above
the distance threshold. If Option A is selected, the first and second AR content items
will coexist in the AR space. By doing so, as one example it is also possible to express
interaction between the AR content items. In particular, in the present embodiment,
since the displaying of an AR content item continues even after a marker has been lost
from the image, even if a plurality of markers do not simultaneously appear in the
input image, it is still possible to gradually add AR content items to the AR space. In
this case, it is possible to avoid the coexistence of an excessive number of AR content
items in the AR space and to have AR content items coexist in more natural conditions.
[0066] Note that the content control unit 155 may control the coexistence of a plurality of
AR content items based on the types (for example, the "types" illustrated in Fig. 10) of
the first and second AR content items. For example, the content control unit 155 may
select Option A described above only when the first and second AR content items are
the same type. The expression "AR content items of the same type" may refer for
example to AR content items associated with the same type of marker, to AR content
items that express the same type of character, or AR content items for an application
with the same object. By doing so, it is possible to avoid having a mixture of AR
content items that are incapable of interaction coexist with one another.
[0067] (8-5) Output of Control Results
By controlling the behavior of AR content in this way, the content control unit 155
selects the AR content to be overlaid on the input image. The content control unit 155
then decides the three-dimensional display position and display posture in the AR
space of the selected AR content. The display position and display posture of the AR
content are typically decided using the recognition results of the peripheral en
vironment of the image processing apparatus 100 produced by the analyzing unit 125.
That is, the content control unit 155 decides the display position and display posture of
the AR content using the feature point information 131 and the camera position/posture
information 132 stored by the 3D structure DB 130. The display position and display
posture of the AR content may be decided so that the AR content is within the field of
view of the camera and the respective AR content items stand on an object or on the
ground in the field of view. If there is a sudden change in field of view, the display
position(s) of the AR content may be decided so that the AR content moves slowly
without completely tracking the change in the field of view. Note that the method of
deciding the display position and display posture of the AR content is not limited to
this example. The content control unit 155 then outputs drawing data, display
positions, display postures, and other control parameters for the AR content to be
overlaid on the input image to the display control unit 160.
[0068] The control parameters additionally outputted from the content control unit 155 to
the display control unit 160 may include parameters including the line of sight of an
AR content item, for example. Also, the control parameters may include a transparency
parameter relating to the fading out of AR content. For example, during the deter
mination of the removal condition A described earlier, the content control unit 155
may set the transparency of an AR content item higher as the distance from the marker
to the camera position approaches the specified distance threshold. In the same way,
during the determination of the removal condition B described earlier, the content
control unit 155 may set the transparency of an AR content item higher as the angle
between the optical axis of the camera and the direction from the camera to the marker
approaches the specified angle threshold. By setting the transparency in this way, it is
possible to have an AR content item gradually fade out before the AR content
disappears. The content control unit 155 may also output a control parameter to the
display control unit 160 indicating that a graphic indicating is to be displayed when the
AR content is about to disappear from the display when one of the removal conditions
is satisfied. This control parameter may cause the display to display a graphic indicia
instructing a user to adjust the camera position such that a removal condition may no
longer be satisfied. This instruction may, for example, be an arrow instructing the user
to adjust a position of the camera and/or an instruction to move the camera closer to
the marker. The graphic indicia may also simply be a warning indicating that the AR
content is about to disappear from the display.
[0069] (9) Display Control Unit
The display control unit 160 generates an output image by overlaying the AR content
associated with the marker(s) detected by the marker detecting unit 140 on the input
image inputted from the image acquiring unit 120. The display control unit 160 then
displays the generated output image on the screen of the display unit 110.
[0070] More specifically, the drawing data, the display positions, the display posture, and
the other control parameters for the AR content to be displayed are inputted from the
content control unit 155 into the display control unit 160. The display control unit 160
also acquires the present camera position and posture from the 3D structure DB 130.
The display control unit 160 then overlays the AR content at a rendering position on
the image pickup plane based on the display position and display posture of the AR
content and the present camera position and posture.
[0071] The drawing data used for displaying by the display control unit 160 may be
switched between the two types of drawing data illustrated in Fig. 10 based on the
camera position and camera posture relative to the marker. By doing so, in a state
where the user is close to a marker or is picking up images of the periphery of a
marker, for example, the content associated with such marker may be displayed with a
high display resolution. Also, the display control unit 160 may change the transparency
of the AR content in keeping with a control parameter from the content control unit
155.
[0072] In the present embodiment, as described earlier, since the display position and
display posture of the AR content are decided using the recognition result for the p e
ripheral environment of the image processing apparatus 100, the display control unit
160 is capable, even after a marker that was previously detected has moved out of the
field of view of the input image, of overlaying AR content associated with such marker
on the input image in a natural way. Also, since the recognition results for the p e
ripheral environment are stored by the 3D structure DB 130, even if recognition of the
environment fails for a certain frame, for example, it is possible to continue
recognition based on the previous recognition result without having to restart
recognition of the environment from the beginning. Therefore, according to the present
embodiment, it is possible to continue displaying AR content even if a marker no
longer appears in the input image and recognition has temporarily failed. This means
that the user can move the camera freely without having to worry about whether
markers appear in the input image or whether the peripheral environment is being
properly recognized.
[0073] 2-3. Example Displaying of AR Content
Fig. 13A is a diagram useful in explaining a first example of the displaying of AR
content according to the present embodiment. Fig. 13A shows an output image Iml 1 as
one example. The table 11, the coffee cup 12, and the poster 14 appear in the output
image IM1 1. Based on the positions of feature points of such real objects, the
analyzing unit 125 of the image processing apparatus 100 recognizes the camera
position and camera posture in three dimensions and also the three-dimensional
structure (that is, the three-dimensional positions of such feature points) of the en
vironment in accordance with SLAM. The marker 20a is printed on the poster 14. The
marker 20a is detected by the marker detecting unit 140 and the AR content 34a a s
sociated with the marker 20a is disposed in the AR space by the content control unit
155. As a result, the AR content 34a is displayed in the output image Iml 1.
[0074] Fig. 13B is a diagram useful in explaining a second example of displaying of AR
content according to the present embodiment. An output image Iml 2 shown in Fig.
13B is an image that may be displayed following the output image Iml 1 described
above. Only part of the poster 14 appears in the output image Iml 2, so that the marker
20a cannot be detected by the marker detecting unit 140. However, it is assumed that
the camera position and camera posture relative to the marker 20a do not satisfy the
removal condition described above. The content control unit 155 moves the AR
content 34a within the field of view of the output image Iml2. The display control unit
160 then overlays the AR content 34a at a position decided based on the camera
position/posture information 132 stored in the 3D structure DB 130. After this, if for
example the image processing apparatus 100 moves further in a direction away from
the marker 20a, the AR content 34a may fade out and finally disappear.
[0075] Fig. 13C is a diagram useful in explaining a third example of displaying of AR
content according to the present embodiment. Fig. 13C shows an output image Im21 as
one example. The table 11 and the book 13 appear in the output image Im21. The
analyzing unit 125 of the image processing apparatus 100 recognizes the camera
position and camera posture in three dimensions and also the three-dimensional
structure of the environment based on the positions of the feature points of such real
objects according to SLAM described earlier. The marker 20b is printed on the book
13. The marker 20b is detected by the marker detecting unit 140 and the AR content
34b associated with the marker 20b is disposed in the AR space by the content control
unit 155. As a result, the AR content 34b is displayed in the output image Im21.
[0076] Fig. 13D is a diagram useful in explaining a fourth example of displaying of AR
content according to the present embodiment. An output image Im22 shown in Fig.
13D is an image that may be displayed following the output image Im21 described
above. Although the marker 20b does not appear in the output image Im22, displaying
of the AR content 34b is continued. The marker 20a additionally appears in the output
image Im22. The marker 20a is detected by the marker detecting unit 140. In the state
in Fig. 13D, since the distance from the marker 20b to the camera position is below the
specified distance threshold, Option A described above is selected. As a result, the
content control unit 155 disposes the AR content 34a associated with the newly
detected marker 20a in the AR space in addition to the AR content 34b.
[0077] 2-4. Flow of Processing
Fig. 14 is a flowchart showing one example of the flow of image processing by the
image processing apparatus 100 according to the present embodiment.
[0078] As shown in Fig. 14, the image acquiring unit 120 first acquires a picked-up image
generated by the image pickup unit 102 as an input image (step SI 10). The image
acquiring unit 120 then outputs the acquired input image to the analyzing unit 125, the
marker detecting unit 140, and the display control unit 160.
[0079] Next, the analyzing unit 125 executes the analyzing process described above on the
input image inputted from the image acquiring unit 120 (step S120). The analyzing
process executed here may for example correspond to one frame out of the SLAM
computation process described with reference to Fig. 5. As a result, the latest threedimensional
camera position and posture and the three-dimensional positions of the
new feature points appearing in the input image are stored by the 3D structure DB 130.
[0080] After this, the marker detecting unit 140 searches the input image for a marker
defined in the marker basic information 136 (step S130). If a new marker has been
detected in the input image by the marker detecting unit 140 (step S135), the marker
managing unit 145 stores the three-dimensional position and posture and detection
time of the new marker in the marker DB 135 (step S140).
[0081] Next, the content control unit 155 selects the AR content to be displayed (step S150).
The AR content selected here may be markers that do not satisfy the removal condition
described earlier out of the markers that have been detected and whose detection times
are stored in the marker detection information 137. The process hereafter branches in
step SI50 according to whether AR content selected by the content control unit 155 is
present (step S155).
[0082] If no AR content has been selected by the content control unit 155, that is, if there is
no AR content to be displayed, the display control unit 160 sets the input image as it is
as the output image (step SI 60). Meanwhile, if there is AR content to be displayed, the
content control unit 155 decides the three-dimensional display position and display
posture in the AR space of the selected AR content and the other control parameters
(for example, the transparency) (step S165). The display control unit 160 then
generates the output image by overlaying the AR content on the input image using the
decided parameters and the position and posture of the camera (step S170).
[0083] The display control unit 160 then displays the generated output image (which may be
the same as the input image) on the screen of the display unit 110 (step SI 80). After
this, the processing returns to step SI 10 and the processing described above may be
repeated for the next frame.
[0084] 3. Conclusion
The image processing apparatus 100 according to an embodiment of the present
disclosure has been described in detail above with reference to Figs. 1 to 14. According
to the above embodiment, markers associated with AR content to be disposed in an AR
space are detected in an input image and information on the position and posture in the
real space of each detected marker is managed using a storage medium. The position
and posture of the camera relative to the detected marker (s) are tracked and the
behavior of the AR content associated with such markers is controlled based on at least
one of such position and posture. The laying out of AR content is carried out based on
analysis results for the input image using an environment recognition technique such as
SLAM. Accordingly, it is possible to continue displaying AR content even after a
marker has been lost from the image and to maintain a natural displaying of AR
content associated with markers. Note that it is not necessary to manage both the
position and posture of detected markers in the real space and only one (for example,
only the position) may be managed in a database.
[0085] Note that some of the logical functions of the image processing apparatus 100
described earlier may be implemented at an apparatus present in a cloud computing en
vironment instead of being implemented at the image processing apparatus itself. In
this case, the information exchanged between the logical functions may be transmitted
or received between apparatuses via the communication unit 112 illustrated in Fig. 3.
[0086] The series of control processes carried out by the image processing apparatus 100
described in the present specification may be realized by software, hardware, or a com
bination of software and hardware. Programs that compose such software may be
stored in advance for example on a storage medium provided inside or outside the
image processing apparatus 100. As one example, during execution, such programs are
written into RAM (Random Access Memory) and executed by a processor such as a
CPU.
[0087] Although a preferred embodiment of the present disclosure has been described above
with reference to the attached drawings, the technical scope of the present disclosure is
not limited to such embodiment. It should be understood by those skilled in the art that
various modifications, combinations, sub-combinations and alterations may occur
depending on design requirements and other factors insofar as they are within the
scope of the appended claims or the equivalents thereof.
[0088] Additionally, the present technology may also be configured as below.
(1) An information processing system comprising: one or more processing units that:
acquire video data captured by an image pickup unit; detect an object from the video
data; detect a condition corresponding to the image pickup unit; and control a display
to display content associated with the object at a position other than a detected position
of the object based on the condition corresponding to the image pickup unit.
(2) The information processing system of (1), wherein the object detected from the
video data is at least one of a shape, symbol, character string, design, object, part of an
object and an image displayed on a display.
(3) The information processing system of (1) or (2), wherein detecting a condition cor
responding to the image pickup unit includes detecting at least one of a position and
posture of the image pickup unit.
(4) The information processing system of (3), wherein the one or more processing units
detect at least one of a position and a posture of the detected object.
(5) The information processing system of (4), wherein the one or more processors
control the position on the display at which the content is displayed based on the at
least one of a position and posture of the image pickup unit and the at least one of a
position and a posture of the detected object.
(6) The information processing system of any of (1) to (5), further comprising: a sensor
unit that detects the condition corresponding to the image pickup unit.
(7) The information processing system of (6), wherein the sensor unit includes at least
one of a positioning sensor, an acceleration sensor and a gyrosensor.
(8) The information processing system of any of (1) to (7), further comprising: a
memory that stores information corresponding to at least one object in association with
information corresponding to the content.
(9) The information processing system of (8), wherein the information corresponding
to the content includes at least one of a marker identifier, a related content identifier
and a size corresponding to the content to be displayed.
(10) The information processing system of (8), wherein the one or more processing
units acquire the content based on a comparison between the detected object and the
information corresponding to the at least one object stored in the memory.
(11) The information processing system of any of (1) to (10), further comprising: a
memory that stores at least one of identification information, attribute information and
an image corresponding to the content.
(12) The information processing system of ( 11), wherein the one or more processing
units acquire the image corresponding to the content the memory based on the detected
object.
(13) The information processing system of any of (1) to (12), further comprising: a
memory that stores information corresponding to the detection of the object.
(14) The information processing system of (13), wherein the information corre
sponding to the detection of the object includes at least one of an object identifier, an
object position, an object posture, an object detection time and a time corresponding to
when the object was no longer detected in the acquired video data.
(15) The information processing system of (14), wherein the one or more processing
units control the position on the display at which the content is displayed based on the
condition corresponding to the image pickup unit and the information corresponding to
the detection of the object.
(16) The information processing system of any of (1) to (15), wherein the one or more
processing units control the display to move the content on the display based on a
change in at least one of a position and a posture of the image pickup unit.
(17) The information processing system of any of (1) to (16), wherein the one or more
processing units control the display to display the content on the display when the
object is no longer detected in the acquired video data.
(18) The information processing system of any of (1) to (17), wherein the detected
condition corresponding to the image pickup unit includes at least one of a position and
a posture of the image pickup unit, and the one or more processing units control the
display to stop displaying the content when at least one of the position and the posture
of the image pickup unit satisfies a predetermined condition.
(19) The information processing system of (18), wherein the one or more processing
units controls the display to display an indication that the display is to stop displaying
the content when at least one of the position and the posture of the image pickup unit
satisfies the predetermined condition.
(20) The information processing system of (18), wherein the predetermined condition
corresponds to a distance between the image pickup unit and the detected object, and
the one or more processing units control the display to stop displaying the content
when the distance between image pickup unit and the detected object exceeds a prede
termined threshold value.
(21) The information processing system of (18), wherein the predetermined condition
corresponds to a difference in an angle between an optical axis of the image pickup
unit and an axis extending from the image pickup unit to the detected object, and the
one or more processing units control the display to stop displaying the content when
the difference in angle exceeds a predetermined threshold value.
(22) The information processing system of (18), wherein the predetermined condition
corresponds to a time elapsed from when the object was first detected in the acquired
video data, and the one or more processing units control the display to stop displaying
the content when time elapsed exceeds a predetermined threshold value.
(23) The information processing system of (18), wherein the predetermined condition
corresponds to a time elapsed from when the object was last detected in the acquired
video data, and the one or more processing units control the display to stop displaying
the content when time elapsed exceeds a predetermined threshold value.
(24) An information processing method performed by an information processing
system, the method comprising: acquiring video data captured by an image pickup
unit; detecting an object from the video data; detecting a condition corresponding to
the image pickup unit; and controlling a display to display content associated with the
object at a position other than a detected position of the object based on the condition
corresponding to the image pickup unit.
(25) A non-transitory computer-readable medium including computer program in
structions, which when executed by an information processing system, cause the in
formation processing system to perform a method, the method comprising: acquiring
video data captured by an image pickup unit; detecting an object from the video data;
detecting a condition corresponding to the image pickup unit; and controlling a display
to display content associated with the object at a position other than a detected position
of the object based on the condition corresponding to the image pickup unit.

Claims
An information processing system comprising:
one or more processing units that:
acquire video data captured by an image pickup unit;
detect an object from the video data;
detect a condition corresponding to the image pickup unit; and
control a display to display content associated with the object at a
position other than a detected position of the object based on the
condition corresponding to the image pickup unit.
The information processing system of claim 1, wherein
the object detected from the video data is at least one of a shape,
symbol, character string, design, object, part of an object and an image
displayed on a display.
The information processing system of claim 1, wherein
detecting a condition corresponding to the image pickup unit includes
detecting at least one of a position and posture of the image pickup
unit.
The information processing system of claim 3, wherein
the one or more processing units detect at least one of a position and a
posture of the detected object.
The information processing system of claim 4, wherein
the one or more processors control the position on the display at which
the content is displayed based on the at least one of a position and
posture of the image pickup unit and the at least one of a position and a
posture of the detected object.
The information processing system of claim 1, further comprising:
a sensor unit that detects the condition corresponding to the image
pickup unit.
The information processing system of claim 6, wherein
the sensor unit includes at least one of a positioning sensor, an ac
celeration sensor and a gyrosensor.
The information processing system of claim 1, further comprising:
a memory that stores information corresponding to at least one object in
association with information corresponding to the content, wherein
the information corresponding to the content includes at least one of a
marker identifier, a related content identifier and a size corresponding
to the content to be displayed.
PCT/JP2012/005582
The information processing system of claim 1, further comprising:
a memory that stores information corresponding to the detection of the
object, wherein
the information corresponding to the detection of the object includes at
least one of an object identifier, an object position, an object posture, an
object detection time and a time corresponding to when the object was
no longer detected in the acquired video data.
The information processing system of claim 9, wherein
the one or more processing units control the position on the display at
which the content is displayed based on the condition corresponding to
the image pickup unit and the information corresponding to the
detection of the object.
The information processing system of claim 1, wherein
the one or more processing units control the display to move the
content on the display based on a change in at least one of a position
and a posture of the image pickup unit.
The information processing system of claim 1, wherein
the one or more processing units control the display to display the
content on the display when the object is no longer detected in the
acquired video data.
The information processing system of claim 1, wherein
the detected condition corresponding to the image pickup unit includes
at least one of a position and a posture of the image pickup unit, and
the one or more processing units control the display to stop displaying
the content when at least one of the position and the posture of the
image pickup unit satisfies a predetermined condition.
The information processing system of claim 13, wherein
the one or more processing units controls the display to display an in
dication that the display is to stop displaying the content when at least
one of the position and the posture of the image pickup unit satisfies the
predetermined condition.
The information processing system of claim 13, wherein
the predetermined condition corresponds to a distance between the
image pickup unit and the detected object, and
the one or more processing units control the display to stop displaying
the content when the distance between image pickup unit and the
detected object exceeds a predetermined threshold value.
The information processing system of claim 13, wherein
PCT/JP2012/005582
the predetermined condition corresponds to a difference in an angle
between an optical axis of the image pickup unit and an axis extending
from the image pickup unit to the detected object, and
the one or more processing units control the display to stop displaying
the content when the difference in angle exceeds a predetermined
threshold value.
The information processing system of claim 13, wherein
the predetermined condition corresponds to a time elapsed from when
the object was first detected in the acquired video data, and
the one or more processing units control the display to stop displaying
the content when time elapsed exceeds a predetermined threshold
value.
The information processing system of claim 13, wherein
the predetermined condition corresponds to a time elapsed from when
the object was last detected in the acquired video data, and
the one or more processing units control the display to stop displaying
the content when time elapsed exceeds a predetermined threshold
value.
An information processing method performed by an information
processing system, the method comprising:
acquiring video data captured by an image pickup unit;
detecting an object from the video data;
detecting a condition corresponding to the image pickup unit; and
controlling a display to display content associated with the object at a
position other than a detected position of the object based on the
condition corresponding to the image pickup unit.
A non-transitory computer-readable medium including computer
program instructions, which when executed by an information
processing system, cause the information processing system to perform
a method, the method comprising:
acquiring video data captured by an image pickup unit;
detecting an object from the video data;
detecting a condition corresponding to the image pickup unit; and
controlling a display to display content associated with the object at a
position other than a detected position of the object based on the
condition corresponding to the image pickup unit.