INFORMATION PROCESSING DEVICE; INFORMATION PROCESSING,
5 METHOD, AND PROGRAM
Technical Field [0001]
The present disclosure relates to an information processing device, an 10 information pT-oceaSing method, and a program. In particular, the present disclosure relates to an information processing device, an information processing method, and a program, whicH perform an augmented reality (AR) display.
Background Art 15 [0002]
For QKample, an image, in wliich a "virlnal image other tiian a captured image is superimposed on a captured image of a camera, is called an augmented reality (AR.) image and has recently been used in various fields.
In particular, in recent years, portable terminals, such as a sniartphone 20 Laving a camera iunction and a display as well as a communication fiinction, have been widespread, and applications that apply an^AR im^e to such a smarlphone are wide^used. [0003]
As one esamplcof use of the AR image, there are the following examples.
2fi - As an example of processing, when a certain object (subject) is captured by
using a camera fiinction of a portable terminal such as a smar^hone, object
information is transmitted to a server, and information related to tile object is
received firom the server and is displayed on a display section of the sniar^hone
together with a captured object.
30 Image information transmitted fiom the server to the smartphone is various.
For example, there is an image of a virtual character rel^d to acaptmed object, or a

SPS46fi95WO00
2/72
pre-captured person or landscape.
Incidentally, as an example of related art disclosing die generation ^d
display processing of an AR image, diere is Patent Literature I (JP 20 3 2-5 8 S3 8 A),
[O004]
6 However, in a case where a virtual image to be transmitted &om a server is a
moving image in which a real person is captured, when the moving image is superimposedly dis.played on a captured camera image of a portable termmal of a user, the captured image and the virtual image show different motions, "nius, the person as the virtual image is merely stuck on the captured image, and it does not 10 look, as if the person exists in the captured image.
This is a main cause that tiie lEve-action image captured using the portable temiinal by die user is changed according to a change in the angle or the like of the portable terminal, whereas the virtual image provided by the server is moved regardless of the change in the angle or the like of the portable termmal 15 [0005]
III order to solve such problems, it seems effective to transmit a plurality of
. moving images captured ifrom every angle from the server to the user terminal and
selectively display, an image according to die angle of the terminal at tJie user
termmal side. However, if such processing is pcrfomied, a communication load or
20 a load of a User teiininal becomes enormous, and a commiinication delay ot a
processing delay is caused.
Citation List Paletit Literature 25 [0006]
Patent Literamrel: JP 2012-5 8 S3 8 A
Summary of Invention Technical Problem 30 [0007]
Ihe present disclosure has been made m view of, for example, the above

SP34G695WO00 3/72

problems, and provides ah infoimallon processing device^ an information processing method, and a program. According to the present disclosure, in a configuration that superimposedly displays a virtual image on a captured image captured by a camera □fa portable tmminal held by a user and displayed'on a display section, the virtual 5 image is changed accorduig to the motion of the camera, so that an AR image can be displayed as if the virtual image exislB in the captured image.
Solution to Problem [0008]
10 According to a first aspect of the present disclosure^ there is provided an
information processing device including:
an acc^uiaition section configured to acquire space position information of a teraiinal including a display section that displays an image in which a virtual image is superimposed on a eaptui^d image acquired by an imaging section^ arid
15 a specifying section configured to specif display information for displaying
the virtual image on the display section according to the space position information of the terminal. [0009]
According to an embodiment of the preseiit disclosure, the iiifomiation
20 processing device iurther includes an imaging section configured to capture an'image,
a display section configured to displ^ a c-qitured image of the imaging section, and
a data processing section configured to receive the virtual image fiom a server and
superimposedly display the received virtual image, on the captured image displayed
■ on the displ^ section. The data processing section estimates the position of the
; '-.-l-.i——-
2fi. imaging section after predetermined time according to motion infomiation of the information processing device, transmits the estimated position information to tiie server, receives the virtual image captured ftom the estimated position or ihe depth map and the image for virtual image generation fixwn the server, and displays the received image or the image generated based on the received image on the display
30 section. [0010]

SP346695WO00 4/72
According to an embodiment of the present disclosure, the data processing section estiipates the capturing direction of the imaging section afler time T + At considering network delay time At that is round trip time of a communication with a server by applying posillon/orientation mfbrmation of flie imaging section and 5 motion infontiation at cUEient time T, and transmits the estimated direction to the server as the estimated position informalion. [OOllJ
According to an embodiment of the present disclosure, the data processing
. section calculates proliabiH^ of the capturing direction of the imaging section afler
10 time T + At considering network delay time At that is round trip time of a
communication with a server by applying position/orieritation information of tiie
imaging section and motion information at current time.T» selects an angle range of a
range with high probability, and receives, form the server, virtual images of mullEple
viewpoints included in the selected angle range or the depth map and the image for
IB virtual image generation,
[0012]
Accordmg to an embodiment of tiie present disclosure, the data processing section receives, from the server, image data in wliich an image with a relatively high probability is set as a high-resolution image and an image with a relatively low 20 pmhabili^ is set as a.low-resolution image in the selected angle range. [0013]
Accordmg to m embodiment of the present disclosure, when there is no great difference iri the probability in the selected angle range, ^e data processing section receives, from the server, the virtual images of the multiple viewpoints 25 included in the selected angle range as images having a same resolution. [0014]
According to an embodiment of the present disclosure, the data processing
section selects a virtual image of a viewpoint close to the capturing direction'of the
imaging section at a display timing of the virtual image from the virtual images of
30 the multiple viewpoints received from the server, or generates a virtual image
captiired from a viewpoint close to the c^mring direction of the imaging Section at

SP34GG95WO00
5/72

the display timing of the virtual image by applying the depth map and the image for virtual image generation, anddisplays the virtual image on the display section. [0015]
According to a second aspect of the present disclosure, ^ere is provided an 6 infonnatioii prgcessing device as a server including:
a communication section configured to perfoim cfimmunication with a client; and
a data processing section configured to select and transmit a virtual image to be superimposedly displayed on a captured image of a client imaging section which
10 is displayed on a display section of the chent,
wherein the data processing section selects a virtual image captured ^oni the position of the chent capturing section a^er predetermined time estimated according to the motion information of ^e chent or a depth map and aa image for' virtual image generation us data to be transmitted to tire chent, and transmits the
16 selected virtual image or the deplh map and the image to the client. [0016]
According to an embodiment of the present disclosure, the data processing section estimates the capturing direction of the client imaging section after time T4-At considering netwoil: delay time At that is round trip time of a communication
20 hetween the server and tiie client by applying position/orientation information of the client imaging section and motion information at'currenl time T, and selects the virtual imagd captured in the estimated direction or the depth map and the image for virtual image generation as data to be transmitted to the client. [0017J
25 According to an embodiment of fhe present disclosme, the data processing
section calculates probabili^ of the-capturing direction of the client imaging section after time T + At considering network delay time At that is round trip time of a" communication between the server and tiie client by applying poshion/orientation information of the client imaging section and motion information at current time T,
30 selects an angle range of a range with high probabili^, and selects the virtual images of the multiple viewpoints included in the selected angle range as data to be

I -V .- '^

SP34GG95WO00 6/72
transmitted tO the elient. [OOIS]
Accordrng to an embodiment of the present disdosure, the data processing
section generates image data to; be transmitted in which an image with a relatively
B , high probability is set as a high-resolntion image and an image vrith a relatively low
probability is set as a low-resolution image in the selected angle range, and transmits
tiie image data to the client
[0019] ■
According lo a third embodiment of tiie present disclosure^ there is provided 10 an information processing device including:
an imaging section configured to capture an image; . a display section configured to display a captured image, of the imaging sectioui
a stor^e section configured to store virtual images corresponding lo. 16 multiple viewpoints thai are virtual images to be snperimposedly displayed on a captured imagt displayed on the display section and are obtained by capturing an object trom different muitiple viewpoints; and
a data processing section configmed to acquhe the virtual image fmm the storage section and superimposedly display the acquired virtual image on the 20 captared image displayed on tiie display section,
wherein the data processing section estimates the position of the imaging section, after predetennined time according to motion information of the information processing device^ and then selects the virtual image captured frpm the estimated position from the storage unit and displays the virtual image, or generates tiie virtual 25 image based on the virtual image acquired fl-om tiia storage section and displays the virtual image on the display section, [0020] .
According to an embodiment of the present disclosure, Ihe storage section stores the depth map and the virtual im^e corresponding to multiple viewpoints 30 oblain^by capturing an object fiom different multiple viewpoints^ and
wherein the data processing section generates and displays the virtual image

SP34GG95WO00 7/72
captured fiom the estimated position on the display section by applying the depth map and the virtual image acquired from Ihe storage section.
[002]]
According to an. embodiment of the present disclosure, the data processing 5 section calculates probability of the capturing direction of the imaging section alter predetermined time according to motion infocmation of the information processing device, selects a virtual image wifli a hi^ piobabili^ from the storage section or, generates a virtual image based on the virtual image acquired from the storage section, and displays the virtual image on the display section. 10 [0022]
According to a fourth aspect of the present- disclosure, an information processing method^ which is performed by an infbitnation processing device, wherein the information processing device includes:
a communication section configured to peifann communication 15 with achen^ and
. a'dataprocessing*section configured to select and transmit a virEua} image to be sopcrimpbsediy displayed on a captured image of a client imaging section which is dispLayedonadisplay section of the client, and
wherein the data processing section selects a virtual image captured from
20 the position of the client capturing section after predetennined time estimated
according to the motion infomiation of the Uient or a depLh map and ^n image for
virtual image generation as data to be transmitted to the client, aud transmits the
selected image to the client.
[0023] ^ .
2^ According to a fifth aspect of (he present disclosure, an information
processing method, which is perfonned by an information processing device,
wherein a data processing section acquires space position infoimation of a
terminal including a display section that dispiays an image in which a vhtual image
is superimposed on a captured image acquired by an imagbg section, and^
30 wherein the data processing section specifies display information for
displaying the virtual image on the display section Recording to the space position

SP346695WO00 8/72
■ infbrmalionof theterminal.
[0024]
According to a sixth aspect of tbe present disclosure, an infonnation
'. processing method, which is performed by an information processing deWce,
6 wherein the information processing device includes:
an imaging section configured to capture an image; . .
a. display section configured to display a captured image of the
imaging section;
a storage section configured to store virtual images corresponding
10 to multiple viewpoints that are virtual images to be superimposed ly displayed .on a
captured image displayed on the display section and that have a captured object &om
differentmulfiplc viewpoints; and
a data processing section configured to acquire the virtual image
item the storage section and superimposed iy display the acquired virtual image on
1&, ttie captured image displayed on the display section,
wh^ein the data processing section estimates tlJe position of the imaging
■ ■ ■ - section after tii^detennmed time accotdingto motion information of the information
processing device, and tiiCn selects th.^ virtual image .captured from ttie estimated
position from the storage section and displays ttie virtual image, or generates the
20 virtual image based on the virtual image acquired from the storage section, and
displays the virtual image oixthe display section.
[0025]
According to a seventh aspect of the present disclosure, a program for
causing an infonnafion processing device to execute information processing,
26 wherein the information processing device includes:
a communication section configured to perform communication
with a clienti and
a data processing section configured to select and transmit a virtual
image to be SQperimposedly displayed on a. captured image of a client iinaging
30 sectionwhichisdisplayed onadisplay section oftheclient,
wherein the program causes the data processing section to select a virtual

SP3 46695 WOOD
9/72 ' .
image captured from the position of the client capturmg section alter predeteimined time estimated according to the motion infonnation of the client yr a depth map and an image for virtual image generation as data to be transmitted to the client, and transmit the selected image to the chent. 5 [0026]
AccDiding to an eighth aspect of the present disclosure, a program for causing ati infoimatiDn processing device to execnte infonnation processing of:
causing a daLa processing section to acquire space position information of a terminal including a display section that displays an image in which a virtual image 10 is superfmpoaed on a captured image captured by an imaging section; and
causing the data processing section to specif display information for
displaying .the virtual image on the display section according to the space position
" information of thetemiinaL
[(i027]
15 According to a ninth aspect of tiie present disclosure, a program for causing
an information processing device to execute infonnation processing, ^
whereinthe infonnation processing device includes:
an imaging section configured to capture an image; a display section configured to display a captured image of the 20 imaging seclion;
a storage section configured to store virtual images corresponding
to multiple viewpoints that are virtual images to be superunposedly displayed on a
captured image displayed on the display section and that have c^tured a object irom
different multiple viewpoints; and
25 .3 data processing section configured to acquire the virtual image
from the storage section and stiperimposedly displi^ the acquhed virtual image on the captured image displayed on the display section,
. wherein the program causes the data processing seclion to estimate the
position of the imaging section after predetermined time accordmg to motion
30 infonnation of the information processing device, and then selects the virtual image
captured irom the estimated position itom the storage section and displays the virtual

SP34e695WO00 10/72
image, or generates the Virtual image based on the virtual image acquired from the storage section, and display the virtual image on the display section.
■
[002B]
& Note ttialthe program according to,the present disclosure is apro^nm that
can be provided in a storage medium or communication luedium that is provided in a computer-readable fbiru for an infbnnation processing'device OF a computer system that is capable of executing various types of program code, for example. Providing this sort of program in a computer-readable form makes it possible to implement the
10 processing according to die program in the inforfliation processing device or the computer system. [0029]
. The object, feamres, and advantages of the present disclosure will be made clear later by a more detailed explanation that is based on the emhodinients of tiie
IG present disclosure and the appended drawings. Futttieimore, the system in this specilication is not^ limited to being a conlLguration that logically aggregates a plurahty of devices, all of which ^e contained within the same housing, .
Advantageous Effects of Invention
20 [0030]
According to the configuration of one embodimentof the present disclosure,
the virtual im^e, which is natural according to flie capturing dhection of 4e camera,
can be superimposedly displayed on the c^tOred cEUnera image displayed on tiie
display section. ' . ' - ,
25 Specifically, ttie configuration includes the server that transmits the virtual
image to the client, and the client that superimposedly displays the captured camera image and the virtual image transmitted from the server. The server frausmits, to die client, the virtual image captured from the client position after a predetermined time estimated according to the motion of the client (camera), hi one of the server
30 and the client, by ^plying the client position and orientation information at the current trme Tand the motion i^omiation, the capturing direction of ttie client after

.SP34669BWO00
11/72
tiie time T + At considciir^ the rietwork delay time At that is the round trip time of comiiiunicatioTi between the server and the client is estiiiiated, and "die virtual image eapmred in the estbnated direction is transmitted la the client
According to &uch configurations, the virtual image, which is natural fi according to the capturing direction of the camera, can be superimposedly displayed on the c^tured camera image displayed on the display section of the client.
Brief Description of Drawings
[0031] 10 [FIG 1] FIQ- 1 IS a diagram describing an example of use of processing of the
present disclosure.
[FIG 2] FIG 2 is a diagram describing an overview of processing of the present
disclosure. - " "
[FIG 3] FIG 3 is a diagram dcseribing an overview of processing of the present
15 disclosure. ■
[FIG 4] FIG 4 is a diagram describing an -overview of a processing sequence
between a server and a client.
[FIG 5] FIG 5 is a diagram illustrating a flowchart describing a processing sequence
of a first embodjhient of the present disclosure.' ' 2D [FIG 6] FIG 6 is a diagram describing estunation processing of a motion mode of a
client (camera).
[FIG 7] FIG 7 is a diagram illustratmg a flowchart describing details of.processmg
of estimating an angle of a client (camera) and determming a transmission image.
[FIG 8] FIG S is a diagram describing an example of information to be applied to 25 proeessingofestimaling an angle of a client (camera) and determining a transmission
image. . '
[FIG 9] FIG 9 is a diagram describing an example of information to be applied, to
processing of estimating an angle of a client (camera) and detcimining a transmission
image. , . . ""
30 [FIG ,10] FIG 10 is a diagram describing an example of information to be applied to
processing of estimating an ariglc of a client (camera) and determining a tiansmfssioti

SPa4GflB5WO00 12/72
image.
jTIG 11] FIG 11 is a diagram describing an example of infomiadon to be applied to
processing of esdmating an angle of a client (camera) and determining a transmission
image. - '
5 [HQ 12] FIG 12 is a diagram describing an example of information to be appUed to
processing of estimating an angle of a client (camera) and deteimining a transmission
image.
[PiG 13] FIG 13 is a diagram describing an example of transmission data to a client.
[FIG 14] FIG 14 is a diagram describing an example of transmission data to a client 10 P'IG.ISJ FIG 15 is a diagtam describing a configuration example of a server and a
dient in a fu'st embodiment of the present disclosure. .[FIG 16] FIG. 16 is a diagram describing an example of ai^ image provided &om a
server to a client in a fust embodiment of the present disclosure. ' [FIG 17] FIG 17 is a diagram describing an example of an image provided from a IB server to a client in a second embodiment of the present disclosure.
jFIG 18] FIG 18 is a diagram illustrating a flowchart describing a processing
sequence of a second embodiment of the present disclosure,
[FIG 19] FIG 19 is a diagram describing an example of transmission data to a client
[FlG 20] FIG 20 is a diagram describing an example of transmission data to a client 2D |F1G 21] FIG 21 isa diagram describing an example of transmission data to a client,
[FIG 22] FIG 22 is a diagram describing a configuration example of a server and a
client in a second embodiment of the present disclosure.
[FIG 23] FIG 23 is a diagram describing a configuration example of a server and a
client in a third embodiment of the present disclosure.
35 ■ " /
Description of Embodiments
[0032]
Hereinatter, an information processing device, an information processing
method, andaprogram accordingto the present disclosure will be described in detail 30 with reference to the appended drawings. Note that description will be provided in
the following order. ■ ■

13/72
1, Overview of Processing of Present Disclosure
2. First Embodiment of Pr&sentbiselosure
,3, Second Embodiment of Present Disclosure
4. Third Embodiment of Present Disclosure
5. Other Embodiments
6. Summary of Configuration of Present Disclosmft

-SP346695WOQ0

[0033]
[1. Overview of Processing of Present Disclosure]
First, the overview of processing of the present disclosure is described with 10 reference to FlCt 1
FIG, 1 is a diagram illustrating an example of use of processing of the
present disclosure, A user A 21 stands in front of a poster 10, and a user B 22
captures ai^ imsge, includmg the poster 10 and the user A 21, by using a portable
tenninal (client) 30 such as a smar^hone With a camera function. Incidentalty, the
1& captured image is» for example, a moving image.
ffl034] . " , .
TTie portable lenninal (client) 30 transmits capturing information to a server
through a communication section of the chent (portable terminal) 30. TTie server
recognizes that the client (portable terminal) 30 captures the image including the
-20 poster 10, based on the capturing information, received fixfm the chent (portable
terminal) 30, acquires content stored in a content database inside the server, for
example, a moving image content generated hy c!q>turing a person printed on the
poster 10, and transmits the acijuiredcontelitto the portable terminal (client) 30.
[0035]
25 ' TTie portable terminal (client) 30 superimposedly displays the content
(moving image) received from the server on the image being captured. A virtual
image (live-action image) 31 is an image displayed on the client (portable terminal)
30. .
As illustrated in FIG. 1, the portable temiinal (client) 30 displays an
30 augmented reality (AR) image in which the captured image actually captured by the
portable terminal (client) 30 is displayed together with the virtual image received

SP34669SWO00 14/72
from the server, [0036J
The virtual image 31 displayed on the portable tenninal (client) 30 is an image oif a person who does not exist at a place where the user A and the user B are b located.
However, on a screen of the client (portable'termmal) 30, tiie image is . displayed as if the person stands next to the user A 21.
The client (portable tenninal) 30 can c^ture and record the captured image, includiilgthevirtual image 31, as a moving image oi a still image. 10 [0037]
However, the user B 22 who performs capture processing by nsmg the portable tenninal (client) 30 captures an image white moving tiie portable terminal (client) 30. At this time, a capturing direction is changed endfije image is changed according to a change in the capturing direction, 15 [003K]
For example, as illustrated in FIG 2, a position of a camera 31 provided in the portable tenninal (client) 30 is changed to various positions, such as (a), (b), and . (c),by theuser.
At tiiis time, the capturing direction ofthc camera 31 is variously set to, for 20 example, (direction a), (dno^tion b), and (direction c). [0039]
In order to display the image as if die subject person of theviriual image 31 displayed on the portable terminal (client) 30 exists at that capturing spot the virtual image 31 needs to be changed to an image conesponding to each capturing dnection, 2& that is, (direction a), (direction b), and (direction c). [00401
For example, as illastrated in FIQ 2, it is assumed tiiat a virtual image
subject assumption position 32 is set in front of the poster 10. At this time, as
illustrated in FIG 3, ttic virtual image 31 displayed on the portable terminal (chent)
30 30 is changed according to the motion of the portable tenninal (client) 30 and is set
as if the person as the virtual image displayed on the portable terminal (client) 30

SP34fie95WO00 15/72
exists at that place. [0041]
FIG 3 iliualrates ideal display image examples of the virtual image
corresponding to the follovi'ing five capturing duections.
5 FTG 3(a) is a capturing state from a front side of the virtual image subject
assumption, position 32.
FIG 3(R1) is a capturing state from aright oblique direction of the virtual image subject assumption position 32.
FIG 3(R2) is a cflpturing state from a light direction of the virtual image 10 subject assumption position 32.
FIG 3(L1) is a capturing state from a left oblique direction of the virtual image subject assumption position 32.
FIG 3(L2) is a capturing state from a left direction of the viiTual image subject assumption position 32. 16 [0042]
» As illustrated in FIG 3, the virtual image corresponding to the motion of the
portable terminal is displayed on a display section of the client (portable terminal) 30,
and a moving im^e can be viewed as if the person correspnjnding to the virtual
image exists at that place attd can be captured and recorded.
20 [0043J ...
Incidentally, the client that perfomis the image capture and display is not
limited to the sraartphone illustrated in FIG 1 and can be realized by various
information processing devices, for example, a PC or a glass-type AR gldss.
[0044]
25 By applying the processing of the present disclosure, the virtual image can
be changed according to the change in the image to be superimposed, when the
virtual image such as a person is sijperimposedly displayed on the image being
displayed on the chent such as the smartphone or ttie AR glass, and the virmal image
can be displayed as if it exists in a base image to be superimposed.
30 Specifically, for example, free-viewpOint iivc-action moving image content
observed from various vie\vpoints is transmitted from the server to the client and is

SP346695WO00 16/72
superimposedly displayed on a captured camera image displayed on the display
section of the client In this processing, a virtual image with a natural motion
correspondingto die motion of the client (camera) is realized.
[0045]
5 ' Regarding an embodhnent that performs the processing of the present
disclosure^ the following two embodiments will be sequentially described,
(1) Bmbodiment 1: embodiment in which contents of different-viewpoints,
which can be displayed in the Client, are transmitted irom the server
(2) Embodiment 2: embodiment in which contents of discrete muhiple
10 vie\vpoints and depth information (depth map) corresponding to the contents are
transmitted to the client, and the image of an flrbilrary viewpoint to be displayed on
the clietit is generated on the client side.
[0046J
In embodiment 1, the server delivers moving im^;e content captured trom a 1& plurality of different viewpoints to the client. Hie client selects displ^ content
from the received moving image content of each viewpoint according to capturing
angle information of the imaging section (camera) of the client, and performs
decoding and display processingonlhe selected moving image content.
Inthe client, it is only necessary to decode the moving image, and thus, a, 20 ' load is low,
However, since it is necessary to transmit moving images of more
viewpoints at a tune, a communication load is increased.
[0047]
In embodiment 2, the server deUvers moving image content from 25 discontinuous, that is, discrete different viewpoints. Furthermore,.embodiment 2 is
an embodiment that also provides depth information (depth ra^) according to the
moving content to be provided to the client Incidentally, the depth map is
configured by data in which distance information from the camera to the subject is
set by. a pixel unit..
30 On the client side, the moving image content from discontinuous different
viewpoints is used to generate necessary display content, that is, a virtual image

SP34e695WO00 17/72
observed ftom a desired direction. For example', an image of a middle viewpoint of
The directions A and B is generated and displayed by using the real-action images of
[he direction A and the direction B and the deptfi map corresponding to each image.
[n04B]
5 In this processing, transmission data to be trairsmitted from the server to the
client can be reduced. It is only necessary to transmit the Images of at least two viewpoints (three or more viewpoints according to an angle or a position).
However^ there may be a need to generate an image of a middle viewpoint on die chent side, and the load of tiie dient side is increased
10 [0049J
Incidentally, in ether of embodiments 1 and 2, the client superimposes the virtual image, which is generated or selected according to the position/angle of the camera provided in the cEient, on the captured camera image tiiat is the base image being displayed in the client. Incidentally, when the virtual image, which is
16 provided by the server and is to be superimposed on the client is, for example, a live-
aclion person image, &K permeability of pbitl portions ottier ttian the person is
■I maximized and the virtual image is moving image content with a channel
infrirmation of pixel unit, called permeability information or mask information such
as setting the permeability of a person region to 0.
20 By superimposing content witii such a channel infonnation on the captured
image, only the person region included in the virtual image is superimposed on "flie
captured image of the client and the cs^tured image is directly displayed on Qic
image region other than Ihe person.
[0050]
25 Also, although described In detail in the following embodiment, a network
, delay occurs in communication between the server and the clicnL Therefor^, the
content provided fi'om Ihe server to the client needs to be content in whidi the
network delay is considered. That is, for example, the capmring angle of the client
after a predetermined time is predicted based on the motion of the client, and the 30 moving image content from a viewpoint within a predetermined range, with the
predicted angle as a center, is adaptively delivered.

SP34e69fiWO00 18/72
Due to this processing, a neeessaiy bandwidth can be considerably reduced as compared with the deliveiy of the moving images of ail viewpoints. Hic processing «f tiie present disclosure can realize, for example, real-time rendering of the virtual image corresponding to the motion of tiie client,-which has been described B with reference to FIG 3. [0051]
Incidentally, the motion prediction of the client can be performed as, for example, the processing using the marker included in the captured image of tlic client. That is, camera position information or camera moving direction, and camera 10 moving speed are acquired from the capturing position of the marker, the motion of the cUent is predicted &om such information, and the camera position and orien^ion after a predetermined time are estimatedr Also, it m^ be configured to use sensor information of acceleration/gyro sensors provided in the client,
Hereinafter, specific embodrments oflhe processing according to the present 15 disclosure will be described.
[0052J " [2. First Embodiment of Present Disclosure]
Afirst embodiment of the present disclosure will be described.
As described with reference to FIGS. 1 to 3^ the processing oflhe present 20 disclosure generiites and displays the AR image in which the virtual image is superimposed together wilh the captured image on the display section of the client.
An example in which the virtual iniage is provided from the server to the client will be described. Incidentally, although described in the latter part, the virtual image can be acquired from ttie storage section of the client, without being 25 provided from the server, . [0053]
A basic communication sequence between the server and the client wiii be described with reference to FIG 4.
First in step Si I, a client 100, which performs an image capture, transmits 30 [^hiring information to a server.
The capturing infoimation may be, for c?tample, information that can

SP346696WO00 19/72
confirm that the poster 10 illustrated in FIG 1 is captured, and may be ideDiification
information such as a poster image or a two-dimensional bar code snch as a cyber-
code set to in the poster 10,
[0054] .
5 When the capturing information is received from Lhe client, the server 200
acquires moving image content, which is to be provided to the client based Ofl the
capturing information, from a database, and transmits the acquired moving image
content.
However, in the moving image transmission, the server selects and transmits 10 an image of a viewpoint determined according to the motion of the client 100, tiiat Js,
a virtual image observed ttom a direction estimated as a direction in which the client
captures an image after a predetcrmiiied lime. Details of the moving image content
transmission will be described below.
[0055J
15 Subsequently, in step S13, ^c client 100 sup^imposedly displays the
moving image contentreceiVed&om the server 200 on the capmred image. J
The flow of die basic processing of the server and the client is performed as
described above.
[O056]
20 _ Furthermore, the detailed processing sequence will be described with
reference to a flowchart illustrated in FIG 5.
The flow illustrated in FIG 5 is a flow including the processing of the client
that pertorms the processing of displaying the AR image in which the virtual image
is superimposed on the c^tured image» and the processing of the server that provides 2B the virtual image.
The processing of steps SlOl to SI 05 is processing that is performed by the
client,
the processing of steps SlOri and SI 07 is processing that is performed hy
one of the server and the client,
30 the processing of step S103 is processing tiiat is performed by the server, .
and . "

SP346fl95WO00 20/72
the processmg of steps S109 to S114 is processing thai is performed by the client, [0057]
hicidentally, the processing illustrated in the flow is performed under tiie
5 control of a data processing section of the server or the c|ientj that is, a data
processing section having a CPU or the like with a program execution function/
according to a„ program stored in, for example, a storage section of the server or the
chent.
Hereinafter, details of the processing of die respective steps will be 10 sequentially described. [005S]
(Step SI 01)
First, in step SlOl, the client captures an image. For example, the poster as illustrated in FIQ 1 is captured, 15 [O05?J
(Step 3102)
Subsequently, in step S102^ the client determines whether information
acquisition for calculating position and angle information of the camera, which is
necessaiy at a latter stage, is successful.
20 An example of the information for caleulatuig Ihe position and angle
infoimation of the camera is a marker included in the caphired image. The marker is a two-dimensional bar code such as, for example, a cyber-code printed in advance on the poster TO illustrated in FIG 1. When such a marker is recognized, the position or the angle of the camera can be calculated from an angle of a marker 25 reflected on a camera image. [0060]
Incidentally, ftie information for calculating tiie position and angle
infbrmatiop of the camera is not Ihnited to the maiker such as the cyber-code, and
may be an object itself such as aposter or a CD jacket.
30 What is used as Ihe information for calculating the position and angle
information of Ihe Camera is different according to Ihe camera position and angle

SP34G69fiWOnn 21/72
calculation processing algorithm performed by the cjient, and the application
inlbiraation can be variously set. '
[0061J
For example, just like the recognitionof a three-dimensional petition of the 5 camera by extracting and tracking a feature point i&am an image captured by the camera of the client, position identitication processing lo which a simultaneous localization and mapping (SLAM) technology is applied may be performed. Licidentally, the processing to ^vhich the SLAM is apphed is described iti^ for example, JP2011-43419A, filed by the same applicant as the applicant of the present 10 application. [0062]
Also, in addition, as the information for calculating the posilion and angle'
information of the camera, information of a sensor included in the client device may
be applied.
15 As such, instep S102, it is detennined whether tiie acquisition of the camera
position and angle information, which is performed by the client, is successful.
When the information is not acquired, it is determined that subsequent AR image generation processing is impossible, and the processing is ended,
When the information is acquired, die processing proceeds to step S103. ao [0063]
(StepS103)
Subsequently, in step S103, ttie client calculates a current position and
orientation of the chent (camera) by applying tiic information acquired in step SI 02.
[0064]
25 (StepS104)
Subsequently, in step SI 04, the client acquires output values of an
acceleration sensor and a gyro sensor provided in the chent.
[0065] ■ .
CStepS105)
30 Subsequently, in step S105, ttie client calculates a moving speed and a
moving direction of the. clleiit (camera).

SP34669&WO00 22/72
,Thi6 processing is performed by applymg, for example, the current position
and orientation of the client (camera), which is calculated in step S103, and the
sensor information^ which is acquired in step SI 04,
[0066] )
5 , A speciHc processing example of step SI 05 will be desctibed with reference
to FIG 6.
. FIQ 6 illustrates the client 100 that displays the captured imagfe and the AR.
image, and the virtual image subject assumption position 32. The client 100 can be
&eely moved by the user,
10 Instep S103, a cu^ent position C = (Cx, Cy, Cz) of Wie client is calculated.
Instep Si 04, a speed vector VI of the client is calculated.
Also, the virtual image subject assumption position T is T = (tx, ty, iz).
[0067] ■
A vector (C - T) having the virtual.image subject assumption position T = 15 (tx, ty, tz) as a start point and the current position C = (Cx, Cy, Cz) ofthe client as an end point is set as a radius vector r.
TTje purpose of tiie processing of step S105 is to calculate an angular speed
(D iltus&ated in FTG. 6, that is, an angular speed (fl of a viewpoint switching direction
with respect to the virtual image subject
20 The angular speed oi illustrated in FiQ 6 is an angular speed corresponding
to the motion of the client 100 al the current time. [0068]
First, a speed vector v2 ofthe client 100 in a direction perpendicular to the
ladius vector r is calculaled based on the speed vector vl.
26 The speed vector V2 is a speed Vector in the viewpoint swilcbing direction
correspondmg to the virtual image subject
The angular speed p in the viewpoint switching direction corresponding to the virtual image subject c^h be calculated by the following formula, based on the radius vector r and the speed vector v2 of the client in the dhection perpendicular to 30 the radius vector r.
« = |rxv2|/r^

SPa4fifi95WOQ0 23/72
The angnlar speed at m tiie viewpoint switching direction cocresponding to the viitual image subject is calculated based on the above formula. [0069]
(SlepS106)
5 The processing of steps S106 and S107 is processing that is performed by
one of the cheni and the server.
In step Si 06, for example, a ping command is transmitted and received '
between the client and the server, and a round liip time (RTT) is calculated.
[0070]
10 (StepSlO?) ,
Subsequently, the capturing, angle of the client (camera) alter the
predetermined time is estimated based on the moving direction and speed of the
client (camera) obtained in step S105 and the network delay tune obtained in step
S106, ananglerangeof the virtual image content to be transmitted to the client is set,
IR /and die transmission image is determined.
[0071] . ■.
. Details of ttie processing of step SI 07 will be. described with reference to FIG 7 and thereafter.
The processing of step S107 is performed by processing of steps S201 to 20 S203ofaflowinustratedinFIG7, [0072]
(StepSZOl)
First, t3ie center viewpoint of the client (camera) after a network delay time
^t is calculated.
25 The current tune is T/
the network delay time is At,
the'cameraviewpointof the client at the cuircnftimeTisOT, and
the angular speed in the viewpoint switching direction of the client at the
current time T is tiVf.
30 Incidentally, the network delay time At is a round trip time (latency) from
the client to the server.

■ SF3 46695 WOOO
24/72
,[0073]
Incidentelly, as the operation of the client by the user» there are operations
such as ^om in/zoom, out, as well as the motion of the camera, but these can be
supported t)y the processing of changing a size of the subject. Hcr^, in order to
fi acquire information for selecting the moving image corresponding to the client
viewpoint from the moving image conterit of each viewpoint coiresponding to the
virtual inxage, the processing using the angular speed o>r in the viewpoint switching
direction is performed,
[007'f]
10 At the current time T, data required to be transmitted to the client by the
server is a virtual ittiage IhaC is observed &om an estimated position ofthe client after the network delay time At from the current time t, that is, at the tune T + At.
TTiat is, tiie Server transmits a plurality of virtual image contents that is
captured fr^m a yiewpouit that is not in a predetermined angle range, with the client
15 viewpoint 9T + &i at tiie time T + At as a center. The client selects the image
a corresponding to the actual posilion of the client from the plurali^ of images and
displiiys the selected image.
[0075]
hi this way, in the processing ofthe present disclosure, the motion ofthe 20 client is predicted, and the image of the viewpoint corresponding to the predicted posilion ofthe chent is selected and provided from the server to the client
This processing reduces an ainouut of transmission data, reduces the processing load of tiie client side, and realizes the display ofthe AR ima£e efficicntiy and with highly accurately. 25 [0076J
The client viewpoint Sji-fli at the time T + At can be simply calculated based on the following formula, 9i+i[ = &T + <^At
hi step S20I, the client viewpoint er^itat tiie rime T + At is calculated 30 based on the above formula. That is, the center viewpoint of the client (camera) afler the network delay tune At is calculated,

SP34G695WO0Q 25/72
[0077]
Incidentally, the chentviewpoint Oi-i^it at dielime T-i- At may he calculated considering an angular acceleration of the chent and an ai^lar speed at the time T + ! At. 5 . [0078] ^
(Step S202)
Subsequently, a viewpoint range of the image to be transmitted from tiie. server to the client is determined..
The client viewpoint QT ♦ AI obtained in stq> S201 is one viewpoint 10 corresponding to the position of the client after At fiom the current time T, which is estimated binder the assumption of the data ac^^aired at the time T. [0079]
However, in practice, the user does not necessarily move the client according to the assumption of the dataacquhedatihetimeT, and the moving speed 15 or direction of the client (camera) can be sequentially changed. Also, the network delay time is inconstantly changed by a cnnununication state of the network.
"nierefore, the client viewpoint ST + M obtamed in step S201 is not
utuiecessarily matched to the actual client viewpoint at the time T + At and is merely
one viewpoint position predicted to occur with high probabitily.
20 [0080] ■
M a case where the server transmits only one virtual image content corresponding to the client viewpoint 8T ♦ at to the cheni, when the actual client viewpoint is matched with the viewpoint 0T^-AIJ it is only necessary to display ftie transmission image, but an erroneous image is displayed if the actual client 25 . viewpoint is deviated. [OOBl]
In order to prevent tiie display of such an erroneous image, the server
transmits an image of a viewpoint around that viewpoint to the client, as well as the
virtual image correspondingto the client viewpoint Br-tii estimated in step S201. ,
30 In step S202, a Viewpoint langc correspondmg to a plmali^ of image
contents of different viewpoints" to be IransHiittcd to the client is determined.

SP34e695WO00 26/72
[00a2]
For the determination of the viewpoint range, various melhods can be used,
For example, the following processing can be performed: the angular speed OJT-I-AI of the chent at fiic time T-I- At is predicted atthe timepoint of the current time 5 T, a log of a difference ratio of the predicted angular apeed to the actual angular speed [Oitit of the client at the time T 4- At is collected, and ±2a of ttie standard deviation, of the error ratio is set to an angle range to be transmitted. [00S3]
Also, instead of usmg such a log, an angle rangein degrees may be set in 10 advance, and an image of a range of client viewpoint &T+ii± " degrees estimated in step S201 may be set as the transmission image.-
FUttiiemiore, the image of the range of client viewpoint &r4jii± n degrees may be set as the transmission image,
a function fl^OTj, At) having values of the client angular speed OST of the time 15 T and the network delay time At as parameters may be set, and the angle range of the transmission image can be determined by the setting of gradually increasing the ^gle range if the iunction f^tur, At) e?(ceeds a preset threshold value. [0084J
In this way, in step S202, for example, the moving image content of the 20 following multiple viewpoints are deteraiined as the content to be transmitted to the client
(1) Moving image content 1 in which the subject (for example, the person)
corresponding to the virtual image is captured from the client viewpoint BT + M
calculated in step S201
25 (2) Moving image content 2 in which the subject (for example, tiic person)
corresponding to the virtual image is capmred from the viewpoint of client vic«point flr-hii + fcdegrees
(3) Moving image content 3 in Which the subject (for example, the person) corresponding to the virtual image is captured &om the viewpoint of client viewpoint 30 "^GT4Ai-kdegrees
{4) Moving image content 4 in which the Subject (for example, the person)

SP34fiB95WO00
27/72
corresponding lo the virtual image is caphired fixun the viewpoint of client viewpoint 6rtAi-i-2kdegrees
(5) Moving image contents in which the subject (for example, the person) corresponding to the virtual image is captured from the viewpoint of client viewpoint 5 6T-I-AI—2k: degrees
For example, the five moving image contents aic determined as tiie contents to be transmitted to the client. [O0B5]
(SiepS203)
10 hi step S202, the moving images of the multiple viewpoints within the
predetermined range includmgthe clitait viewpoint Br-tai estimated instep S201 are determined as the images to be transmitted to ftie client
For example, the determined moving images are the above moving image
contents (1) to <5). - ...
15 [00S6J
However, the viewpoint positions of the plurality of moving image contents
are viewpoint positions deteimined based on the motion prediction of the client
(camera), and it is estimated that the probability that the client (camera) will be set to
the viewpoints corresponding to the above images (l)to(5) is different
SO That is, the client viewpoint BT^-M estimated in step S201 is a viewpoint at
which it is estimated that it will occur with the highest probabili^ as the viewpoint of
the client at the tune T + At^ and it is estimated that the occurrence probability will be
reduced as the distance ftorafiiat viewpoint is increased.
[0087]
2& When the entire image contents of the multiple viewpoints included in the
viewpoint range set in step S202 are transmitted as high-resolution images, the amount of data is increased and it is highly likely that the delay of the communication time will occur and tiie processing efficiency will be, lowered.
In order to prevent the occurrence of such problems, the image contents of
30 the multiple viewpoints included in the viewpoint range set in step S202 are
transmitted as the following setting, , .

SP34e695WO00 28/72
ITiat is, the image of the viewpoint estimated to be high probabihty as the
viewpoint of the client at the time T + At is transmitted as a high-resolution image,
and the image in the range estimated to be low probability is transmitted as a low-
resolution image.
S Wiienthe image transmission is performed in sach a setting, the amount of
the transmission data is reduced. [0OK8]
hi step S203, in order to enable the image transmission in Such a setting, the probabili^ ihat the client (camera) Will be set 10 each viewpoint position included in 10 the" viewpoint range set in step S202 is estimated, [0089]
Specifically, for example, a normal distribution is set such that the client
viewpoint GT * at at Ihe time T -I- At calculated in step S201 has the maximum
probability, and the probability that the client (camera) will be set to each viewpoint
15 posilion is estimated by ^plying a probability density flmction according to the
normal distribution. »
[0090] . ^
Specifically, the probability calculation algorithm-for each viewpoint will be
described with reference tp FIG S and thereafter,
20 FIG. 8 illustrates the normal disMbution in which the client viewpoint 9H-AI
at the time T + At calculated instep S201 has the maximum probability
A horizontal axis represaits the viewpoint and a vertical axis fcpresents the occurrence probabili^.
The center (0) of the viewpoint axis corresponds to the client vIewp^*"itGT+ 25 fli at the time T -H At calculated in step S201.
[0091] .
Normal distribution data illustrated in FIQ S is a viewpoint probabih^ density fiinction madenp of normal distribution datawilh average = 0 and standard deviation c, with the client viewpoint Si + ai at the time T + At calculated in step 30 S201 asareference, based on the following parameters. The client angular speed ofthetimeTis atx,

SP34669BWO00 29/72
thenetworkdelay timeis At,.
standard deviation is Q = a|(DT|+ At,
where «is a preset coefticient and p is a preset positive integer.
[0092] - , ! ■
5 As an example, FIG S illustrates an example m which the network delay
time At =100 ms and the angular speed G) of the client (camera) are set to the following three cases.
at = Odeg/s,
CQ = 2deg/s,
10 oi = 10deg/s,
10093]
As the angular speed at of the client (camera) is lower, the occurrence
probabili^ of the client viewpoint ef^-ii at the time T-l-At calculated in step S201 is
higher as compared with the occurrence probahili^ of the surrounding viewpoints.
15 On the other hand, as the angular speed w of the client (camera) is higher, a
difference between the occmrence probability of the client viewpoint Oj + ai at the
time T + At calculated instep S201 and the occuirence probability of the surrounding
viewpoints is reduced.
[0094]
20 B^usingthe normal distribution data illustrated itiFlQ 8, that is, the normal
distribution data in which the client viewpoint GT+AL 3* ^^ lirii^ T + At calculated in step S201 has the maximum probability, the probability that the client (camera) >vill be set to each viewpoint position included in the viewpoint range set in step S202 is estimated. 25 [0095J
Incidentally, in the normal dishibution data illustrated in FIG 8, the center (0) of the viewpoint axis is set to the client viewpoint QT-^M at the time T + At calculated in step S20I.
The probability density function indicating the occurrence probability 30 corresponding to each angle when the center (0) of the viewpoint axis is set to the actual client viewpoint BT at the time T is set as illostraled in FIQ 9.

' SP34669&WO00 30/72
[0096]
As an example, FIG 9 iUustrates an example in which the network delay
time At = 100 ms and the angular speed o> of the client (camera) are set to the
tbllowing five cases.
5 ffi = Odeg/s,
(0 = -2deg/s,
ei = + 2deg/s,
o = -10deg/s,
ai = -MOdeg/s, LO [0097]
The probahility densiljf function illustrated in FIG 9 indicates the
probabili^ of the client position estimated when the center (0) of the viewpoint axis
Is set to the actual client viewpoint Qj at tiie time T and is located at the position after
the time At wilh respect to the angnl^ speed o> of the client (camera).
15 This is the setting in whicih it is highly likely that it will be deviated tO a left
direction when the angutar S|>eed (a is (-) and will be deviated to a right dhecfion when the angular speed 0,
&>CU0O|
30 -p=NOaMDIST (9,0100^0)
0 = S107, for example, as illustrated in FIG 12, flie following data (1) to (5) are determined as the transmission images, and the probability of each image is 5 calculated,
(1) Moving image content 1 (View I)
Moving image content 1 in which the subject (for example, the person)
corresponding to the virtual image is c^tured &om the client viewpoint ST* it
calculated in step S201,
10 Probability that fiie client will be set to ti]e position corresponding to the
image viewpoint of the ipoving image content 1 = 0.40.
(2) Moving image content 2 (View 2)
Moving image content 2 in which the subject (for example, the person) corresponding to the virtual image is captured &om the client viewpoint &T-I-M + k 15 degrees
Probabili^ that the client will be set to the position corresponding to the image viewpoint of t3ie moving image content 2 = 0.20. [0109]
(3) Moving image content 3 (View 3)
20 Moving image content 3 in which the subject (for example, the person)
corresponding to the virtual image is captured from ftie client viewpoint QT+M- k degrees
Probabili^ that the client will be set to the position corresponding to the
image viewpoint of the moving image contents = 0.20.
25 (4) Moving image content 4 (View 4) .
Moving image content 4 jn which the subject (for example, the person)
corresponding to the virtual image is captured &om the client viewpoint ST+JI + 2k
degrees .
Probability that the client will be set to the posilion corresponding to the 30 image viewpoint of the moving image content 4 = 0.05r (5) Moving image content 5 (View 5)

SP346695WO00 34/72'
Moving image content 5 in which the subject (for example, the p^son) coiTesponding to the viitual image is captured from the client viewpoint Ox-^ai- 2k degrees
Probabili^ that the client will be set to tbe position corresponding to the 5 image viewpoint of tlie moving image contents = 0.05.
In step S107, for example, the probability of determining and generating
sueh transmission images is determined,
[OilOJ - '
(Step S108)
10 Next, the processing of step SiOS of the flowchart illustrated in FIG 5 will
be ^escribed.
The processing of step S108 is processing Ehatis performed by the server.
The server generates a stream to be traiismitted to the client by using the data determined in the processing of step S107, that is, for example, the probabiUty 15 data and the images to be transmitted to the client as iliuslraied in FIG 12.
[0111] -
•f
A stream generation example is illustrated in FIG, 13 ,-
For example, the server generates a stream including a plurality of moving
'' image contents illustrated inFIQ 13(A) andtransinits the stream to-the client.
20 FIG 13(B) is a graph of the probabilily density fiinction described aboye
with reference to FIG E and illustrates the probability of the client (camera) set to
each angle, with the estimated angle at the time'T + At of the chent (camera) as the
center (0) of a horizontal axis.
This probability corresponds to the prohabihty of the data illustrated in FIG
2fi 12,
[0112]
As illustrated in FIG 13(A), the server generates stream data of multiple
images in which tire image df the viewpoint with high probability is configured as
the high-resolution image (high-bit-rate uuage) and the image of the viewpoint with
30 low probabilily is coniigured as fiic low-resolution image (low-bit-rate image), as the
image 1 (View 1), the image 2 (View 2),the image 3 (View 3),^,,, the image 7 (View

SP34G695WO00 35/72
7) in descending order of probability, and transmits the stream data to the client [0113]
The example illustrated in FIG 13(A-) is a setlhig example of the transmission image stream when the probability distribution illustrated in ¥10. 13(B) 5 is acquired.
In this regard, for example, as illustrated in FIG 14(B], when the probability
distribution is smooth, the image stream data in which images of different viewpoints
are alt set as the data of the same resolution may be generated and transmitted as
iflustrated in FIG 14(A).
.10 [t)114] . .
Incidentally, the image of each viewpoint may be transmitted as a separate, stream, but it may be set such that multiple images of multiple viewpoints are packed into one stream and then transmitted.
Also, in order to suppress the load of the server, it is desirable that the image.
15 s.Iream of each viewpoint is generated in advance for each resolution and bit rate and
is stored in a database, instead o^real-time encoding.
[0115] ^
Tncidentally, the image data to be transmitted froni the server to the client is data that is encoded by, ibr example, a predetermined encoding algorithm such as 20 MPE<4 and it is necessary to perform decoding processing on the client side. In the client] when the display is switched to the image slream of different viewpoints, it is necessary to select different moving image streams and newly start decoding. Due to the delay at the time of switchirig the stream, it is ^prehended that smooth image switching will become impossible. 25 [0116]
hi order to mmimize the delay of the decodingprocessing, the image stream
to be provided to the client is set to two ^es of image streams: (a) moving image
sequence in which all frames are key frames^ and (b) general moving image sequence.
[0117]
30 The moving image sequence in which all frames are key frames ts, for
example, encoded data that ^B configured by image frames capable of decoding each

SP34G695WO00 36/72
unage frame by itself. The general moving image sequence is, for. example, encoded data that need to be decoded with refeT^nce to the previous or nest image frame.
The chent can tirst read the first fi-ame ^om the ^'moving image sequence in 5 which all &am&5 are key frames" at (he time of switching the ftieam, perform fest decoding processing to display the frrst frame on the display section and contmue the display by reading the second aud subsequent frames from the ''general moving image sequence" and decoding the second and subsequent fi'ames. Thus, 1he delay atthe time of display switching can be reduced. 10 [0118]
Also, inordertoreduce theloadof the content decoding on the client side, it
may be configured such that the stream to deliver the image of the maximum
probabiEity is set as a fixed stream, and partial i^al-time encoding is performed to
deliver contents of different viewpoints changed according to the motion of l^e client
15 by using the same fixed stream.
[0119] " ■ .
In this way, by transmitting the image of the viewpoint with high probability as a high^ bit rate and resolution, data, high-quali^ images can be displayed with high probability. Also, since ftie image of the viewpoint with low probability is also 20 transmitted, it is possible to reduce the probabiiily that the image of the corresponding viewpoint will not be displayed, even when the client is suddenly moved. [0120]
(StcpSI09)
25 Returning to the flow of FIG 5, the description about the processing
sequence of embodiment I will be continued,
When die image transmission is perfbmied by the server in step S108, the client receives the image stream of the virtual image instep SI09.
For example, llie image stream is a stream made up of moving image data of 30 the images of multiple viewpoints described with reference to FIG 13(A) or 14(A). [0121]

SP346695WO00 37/72
(StepSllO)
Subsequently, the client calculates llie camera position and angle upon
reception of the stream, selects the virtual image to be displayed on the display
section of the client aud determines the supeiimposition position of the selected
5 virtual image. The processing of calculating the camera position and augle is the
same processing of steps S102 and SI 03 as described above and detennines the
processing of selecting the virtual image and the superimposition position by
applying the calculation result.
[0122]
10 (Step Sill)
Subsequenfly, in step Sill, the client performs the decoding processing on
h
the virtual image stream selec1&] as the image to be displayed. -[0123]
(StepSlU)
15 Subsequently, in step SI 12, the client superimposes tlie decoded content on
thecapturedcsmeraimagebeingdisplayedon the display section of the-clieTM.
[0124] ''
(StepSin)
Subsequently, in step S113, the client outputs an AR image in which the
20 virtual image is superimposed on the c^luied image as the final result on the display
section (display) of the client. . .
(Step SIM)
In step SU4, it is determined whether a predetermined fuiish condition, such as the finish of the image capture processing or the finish of the application, 25 occurs, and non-processing is ended when Qie finish condition occurs. When the finish condition does not occur, the processing returns to step SlOl and the same processing is repealed. [0125] ■
In Ibis way» tiie virtual image transmitted from the server ■is superimposedly 30 displayed on the captured image that is displayed on the disptay section of the client
Incidentally, in tiie flowchart illustrated in FIG 5, it may be conflgurpd such

SP34G695WO00 3S/72
that the processing of Steps SIOS to S107 is omitted, and, after the acquisition of the camera position/orientation information in step S103 and the calculation of the moving dhection and speed of the camera in step S]t|4, the virtual image to he displayed on the display section is generated or selected and. then on^ut io the 5 display section, based on sach acquhed. information, fliat is, space position infonnation such as the position, orientation, or motion of the camera. [0126]
FIG. 15 illustrates a configuration example of the server 200 and the client
100 that perform Ihe present embodiment 1,
10 Data transmission and reception is performed between a communication
section 108 of the client 100 and a communication section 207 of the server 200,
Incidentally, in FIG 15, a network delay estimation section 106 and an angle
range estimation section 107, which arc illustrated in the configuration of the client
100, maybe set as elements of the server 200, instead of the chent 100.
15 [0127] ,
The chent 100 and the server 200, which are illustrated in FIQ 15, perform the above-described processing according to the flow illustialed above in FIQ 5.
The processing, which is performed by die respective sections illustrated in FIQ 15, will be described in association widi the processing of the respective steps
20 of the flow of FIG 5-[012SJ
- An imaging section (camera) 101 of the chent 100 performs the camera image acquisition processing of step SlOl of the flow ihuslrated in FIQ 5.
An image recognition section 102 performs the camera position/orientation 2fi Calculation information acquisition processing of step S102 of the flow of FIG 5, Specifically, for example, the processing of recognizing the marker set to the subject such as the poster illustrated in FIG 1 is performed. [0129]
A cU^t (camera) position/orientation calculation section 103 performs the
30 camera posifiDn/orientation calculation processing of step S103 of the flow of FIG 5.
A sensor (acceleration/gyro sensor, etc.) 104 perfonns ttiee sensor

SP34GB95WO00 39/72
information acquisition processing of step S104 of the flow of FIG 5.
A plient moving speed/direction calculation section 105 perfonns tiie client
(camera) moving speed/direetion calculation processing of step S105 of the flow of
FIG 5. : .
5 [0130]
A Tietworii delay esthnation section 106 perfbnns the network delay estimation processing ofstq>S106 of the flow of FIG 5.'
All angle range estimalion section I07perfonns the processing of step Si 07 of the flow of FIG 5, that is, the transmission image determination processing and 10 the" probabili^ estimation processmg, based on the cUent - (camera) pusition - estimation.
Incidentally, the network delay estimation section 106 and the angle range estimation section 107 may be set to the server 200 side and performed as the . processingoftheserver200,. 16 [0131].
The angle range estimation section 107 determines the transraission image'
based on the client (camera) position estimation and calculates the probabilily
corresponding to each transmission image, that is, the probability that the client will
be set to the viewpoint position corresponding to. each image. Specificany, for
20 example, the data described above with reference to FIG 12 is generated,
TTie data is input to a transmission image data generation section 206 of the server 200 through a communicalion seclion. [0132J
The server 200 previously generates the irnage content (moving image 25 stream) of multiple viewpoints to be provided lo the client, and holds the image content as the content 205 corresponding to each viewpoint
The server generates the content 205 corresponding to each viewpoint by
performing the following processing as pre-processing.
[0133]
30 That is, a deplh informalion (depth map) generation section 202 acqnires
content stored in a content database 201, that is, image content of discrete multiple

SP346695WOU0 40/72.
viewpoints, and geneiates depth information (depth map) 203 corresponding to the image.
Furdiermbre, a middle viewpoint image generation section 204 generates image content of a middle viewpoint, which is not included in the content stored in 5 the content database 201, by applying the content stored in the content database 201 and the depth infonnation (deptii map) 203, [0134J
For example, the server 200 previously generates the image of each
viewpoint of 3fiO degrees with respect to a certain sobject, which includes the content
10 stored in the content database 201 and the image cont^t of the middle viewpoint that
is not included ia the content stored in the content database 201, as the content 205
corresponding to each viewpoint.
[0135] - . ' .
The transmission image data generation section 206 of ftie server 200
15 pcTibrms the processing of generating the transmission stream fiom ^e content 205
coiresponding to each viewpoint to the chent, based on the image content to be
transmitted to the client. That is, the process of step S108 of the flow illustrated in
FIG 5 is peifonned.
For example, the server generates the stream to be transmitted to the client
20 by using the probability data and the images to be transmitted to the client as
illustrated in FIG 12, " :
Specifically, for example, the stream is combined data of ttic image contents
of the multiple viewpoints as illustrated in FIGS. 13(A) and 14(A).
[0136]
2R The transmission data generated by the tiansmission image data genemtion
section 206 is transmitted to the communication section 108 ofthe client 100 through the communication section 207. ■ .
The communication section 108 ofthe client 100 performs the vhtuai image stream reception processing of step S109 ofthe fiow of FIG 5. 30 [0137]
A content sclectioii and superimposition position calculation section 109 of

SP34fiG9&WO00 41/72
the client 100 perfbnns tiie processing of step SllO of the flow of FIG 5, that is. The
processing of selecting the virhial image and determining the superimposilion
position. That is, the cam^a position and angle are calculated upon reception of the
stream, the virtual image to be displayed on the display section of the client is
5 selected, and the superimpositioil position of the selected virtual image is determined.
[013S] . . ^
A decoding Eeclioii 110 performs the processing of step Sill of the flow of FIG 5, that is, the processing of decoding the selected virtual image stream.
A display conirol section 111 performs the processing of steps S112 and 10 SI13 of the flow of FIG 5, that is, outputs an AR image in which the decoded content is. superimposed on the captured camera image that is displayed on the display section 112 of the client 100. [01^9]
In this way, the virtual fmage transmitted &om the server is superimpoaedly 15 displayed on the captured image that is displayed on the display section of the client. Incidentally, the configuration illusfrated in FIQ 15 is a diagram illustrating the main configuration of the server and ihe client. Besides the configuration illustrated in the drawing, ttie server and die'client include, for example, the control section having the CPU or the like that controls the processing described with 20 reference to FIG 6, or the storage section that stores the program to be performed by . the control section. [0140]
[3. Second Embodiment of Present DigclosuieJ
Next, a second embodiment of the present disclosure will be described.
25 The second embodiment is an embodiment in which contents of discrete
multiple viewpoints and depth inform ation (depth map) corresponding to tiie contents are transmitted from the server to the client, and the image of an arbitrary viewpoint to he displayed on the client is generated on the client side.
That iSj the server transmits fiic depth map and the image for the virtual ^0 image generation for tise purpose of generating the virtual image to be displayed to the chenl.

SP34e695WO00 4^72
[0141]
The above-described processing of tiie iirSt embodiment and the case of the second embodiment will be described with reference to FIGS. 16 and 17.
' In ihd first embodiment, the moving image content of a viewpoint of a
■ 5 certain angle range, for example, n moving image contents &om the moving image 1 to the moving image n a^ illustrated in FIG 16> are transmitted to theelient and the cUent performs the processing of selecting the content suitable for die angle or the like of the cUent from the n contents and displaying the selected content, [0142]
10 In ttie second embodiment described below, as illustrated in FIG 17, the
. images of several discrete viewpoints, for example, the moving images, of only three
viewpoints as described in FIG 17 among Ihe moving image contents of the multiple
viewpoints of a Certain angle range, are transmitted to the, client. Furthermore,
depth information (depth map) corresponding to.tiie three moving images to be
15 transmitted is also transmitted to the client. [0143]
TTie client generates the image oflhe middle viewpoint from the viewpoint between the moving image 1 and the moving image 2 by applying the images of discrete viewpoints and the depth map, and generates and displays tiie image to be
ao . displayed on the Client.
In the configuration of the second embodhnent, there is an advantage that can reduce the data amount of the image data to be transmitted from the seivcr to the client.
[0144] - :
25 The processuig sequence of the second embodiment will be described, with
reference to a flowchart illustrated in FIG 18.
The flow illustrated in FIG. 18 includes the processing common to the flow of FIG 5 described above as the processing sequence of the first embodhnent.
The same processing as embodiment 1 is the processing of steps SlOl to 30 S106, the processing of step S109, and the processuig of steps 3112 to S114.
TTie processuig of step S107b, step SlOSb, step SUOb, and step Slllb,

SP346695WO00
43/72
which are iUustrated in the flow of PIQ 18, is different from the processiDg of
embodiment L
[0145J
Hereinafter, fiie processing common to embodiment 1 will be briefly & described, and different parts ^ill be mainly described.
Incidentally, similarly to embodiment 1, the flow illustrated in FlQ 18 is a
flow inclnding the processing of the client that peiferms the processing of displaying
the AR image in which thevirEiral image is superimposed on the captured image, and
the processing ofthe server that provides the virtual image,
10 The processing of steps SlOl to S105 is process tiiat is performed by the
client,
theprocessing,ofstepsS106 and S1(J7 is process that is performed by one of the server and flie cUent,
the processing of step SI OS is process that is performed by the server, and
15 tiie processing of steps S109 to S114 is process that is performed by the
client.
hicidentally, the processing illustrated in the flow is performed under the
control of Ihe data processing section of the server or the client, that is, the data
20 processmg section having a CPU or the like with the program execulion function,
according to ftie program stored in, for example, the storage section of the server or
the client,
Hereinafter, details of the processing of the respectcve steps will be sequentially described, 25 [0147]
(Step SlOl)
First, in step SIOI, the client captures ah image. For example, the poster
as illuslrated in FIG 1 is captured.
(StepSl02)
30 Subsequently, in step S102, the client determines whether hjfbimation
acquisition for calculating position a^d angle infbimation ofthe camera, which is

SP346695WO0Q 44/72
necessary at a later stage, is successfiil.
An example of the information for calculating the position and angle information of Qis camera is a matter included in the captured image. Hie marker is a t^o-dimensional bar code such as, for example, a c^ber-code printed in advance 5 on the poster 10 illustrated in FlQ 1. [014B]
{Step SI 03)
Subsequently, in step S103, the ehcnt. caleulates a eoncnt position and
orientation of the dient (camera) by applying the information acquired in step SI 02.
10 - 2L

SP346696WO00 48/72
[0I63J
(SlepSllOb)
Subsequently, the client calculates the camera position and angle upon
reception of the stream, determines the viewpoint position of the virtual image to be
5 displayed on the display section of the client, and detennines the superimposition
position of the virtual image corresponding to the determined viewpoint. The
processing of calculatmg the'camera positicin and angle is the same processing of
steps S102 and Si03 as described above and determines the vie^vpoint position and
the superimposition position of the virtual-iLiiage by applying the calculation result. '
10 [0164] . "
(StepSlllb)
Subsequently, in step S11 lb, the client performs.the decoding processing on
the image stream necessary for generating the image corresponding to the determined
viewpoint position.
15 For example, when the image Lllustrated. in FIG 20(B) is received and the
iihage of the middle viewpoint between (View 3) and (View 7) is determined as the display image, the images of (View 3) and (View 7) are decoded, and the image of the middle viewpoint is generated by applying the depth map of these images.
20 (Step S112)
Subsequently, in step S112, the client superimposes the generated image content on the captured camera image that is displayed on the display section of the client.
[0166J ^ . -
26 (StepSlB)
Subsequently, in step S113, tiie cHent outputs an AR image in which the
virtual image is superimposed on the captured image as the final result on Ihe display
section (display) of the client.
^[0167]
30 " (StepSn4)
In step S114, it is determined whether a predetermined finish condition,

SP34669fiWO00 49/72
such as the finish of Ihe image capture processing or the finish of the application, occurs, and non-processing is ended when the finish condition occurs. When tiie finish condition does not occur, the processing returns to step SlOl and tiie same processing is repeated. ' 5 [016S]
In this way, the virtual image transmitted from the server is superimposedly displayed onthc captured image liiat is displayed on the display^section of the client. [0169]
FIG. 22 illustrates a configuration example of the server 200 and the client 10 100 [hat perform the present embodiment 2.
Data .transmission and reception is performed between a commumcalion section 108 of the client 100 and a commuriication section 207 of the server 200.
Ineidenlally, in FIG 22, a nelwork delay estimation section 106 and an angle range estimafion section 107, which are illustrated in the configuration of the client. 15 100, may be set as elements of the server 200, instead of the client 100. [0170]
Ihe chent 100 and the server 200, which are illustrated in FIQ 22, perform the above-described processing according to the flow illustrated above in FIG. 19.
The processing, which is performed by the respective sections illustrated in 20 FIG 22, will be described in associafion wifti the processing of the respective steps of,LheflowofFIGl9.
- [0171] . \ .. - . '
An imagmg section (camera) 101 of the client 100 performs the camera
image acquisition processing of step SlOl of the flow illustrated in FIQ 19,
26 An image recognition section. 102 performs the cam^a position/orientafion
calculation information acquisition processing of step SI 02 of the flow of FIG 19.
Specifically, for example, the processing of reeognii^ing the marker set to Ihe subject
suchastheposIerillustratedinFIG 1 isperformed,
[0172]
30 A client (camera) pasition/orientalion calculation section 103 performs the
camera position/orientation calculation processing of step SI 03 of the flow of FIQ

SP346695WO00 50/72
19.
A sensor (acceleration/gyro sensor) 104 performs thee saisor information acquisition processing of step Sl04of the flowofFlQ 19.
IA client moving speed/direction calculation section 105 per^Dims Qis client 5 (camera) moving speed/direction calculation processing of step S105 of the flow of FIG 19. [0173]
A network delay estimation section 106 performs the network delay
eslimation processing of step S1II6 oftheflowofFlG 19.
10 An angle range estimation section 107 performs the processing of step
S107b of flie flow, of FIG. 19, that is, the transmission image determination
- processing and the probahility estimation processing, based on ^e cHent (camera)
posidon.estim ation.
Incidentally, the network delay estimation section 106 and die angle range 15 estimation section 107 may be set to the server 200 side and performed as the processing ofthe server 200, [0174J
The angle range estimation section 107 determines the transmission image
based on the client (camera) position estimation anil calculates the probability
20 corresponding to each transmission image, diat is, the probabili^ that the client will
be set to the viewpoint position corresponding to each image. Specifically, for
example, the data described above with reference to FIQ 12 is generated.
The data is input to a transmission image data generation section 206 of the server 200 through a communication section. 25 [0175J '
The server 200 holds the image conlent (moving imSge stream) of discrete . mulliple viewpoints to be provided to the client in a content database 201.
In the server, as pre-processing, a depth information generation section 202 acquires content stored in a content database 201, that is». image content of discrete 30 multiple viewpoints, and generates deptii information (dcpfti map) 203 corresponding to the image.

SP346fi95WO00 51/72
[01761
The transmission image d^ta generation section 206 of the server 200
performs the processing of generating the transmission stream &om the content 205
corr&sponding to each viewpoint, based on the image content to be transmitted to the
6 client; Tliat is, tiie process of step SlOKb of the flow illustrated in PIQ 19 is
performed.
For example, as described above with reference to FIGS. 19 to 21, the server sets the data, in "which the images of discrete multiple viewpoint and tiie depfii map are comljined, as the data lo be provided to the client, 10 [0177]
Tlie transmission data generated hy the transmission image data generation section 206 is transmitted to the communication section 108 of the client 100 throng. the communication section 207,'
The communication section 108 of the client 100 p^forms the virtual image
Ifi stream reception processing of step S109 of the flow of FIG 19.
[0173J -
A content selection and supcrimposition position calculation section 109 of
the client 100 performs fiie processing of step SllOb of tiie flow of FIG 19, that is,
the processing of determining the viewpoint posifion of the virtual image to be
20 displayed and the superimposition position. Tliat is, fiie camera position and angle
are calculated upon reception of the stream, the viewpoint position of the virtual
image to be displayed on the display section of tbe'chent is determined, and the
superimposition position of the determined virtual image is determined.
[0179J
25 A decoding section 110 performs the processing of step Sill of the flow of
HG. 19, that is^ the processing .of decoding the image stream necessaiy" for generating the image corresponding to the determined viewpoint position.
For example, when the image illustrated in FIG 20(B) is received and the im^e of the middle viewpoint between (View 3) and (View 7) is determined as the 30 display image, the images of (View 3) and (View 7) are decoded, and the image of the middle viewpoint is generated by applying the depth map of these images.

SP346695WO00
' 52/72
[OlSOl
A display control section HI performs the processing of steps S112 and SI13 of the flow of FIQ 19, that is, outputs an AR im^e in which flie decoded content is superimposed on the caphired camera image that is displayed on the ft display section 112 of tiie cHenI 100. [0181]
hithis way, the virtual image transmitted from the server is superimposedly
displayed on the captured imege that is displayed on flie display section of the client.
[0182]
10 The processing of the second embodiment has the following advantages.
(Advantage 1) The bandwidth of the network can be reduced. Alternatively, the image quality is improved while maintaining the bandwidtii.
This is because it is better to the image in which the niimhet of viewpoints
to be transmitted is at least two viewpoints.
1& (Advantage 2) It is more robust to the abrupt motion of the client terminal.
4 In the stream to be transmitted, only the view of both ends can be
■1 ' ' -
transmitted by skipping the original video in the middle of the viewpoint with low
probabili^. In the interval of the viewpoint with low probabili^, the number of the
streams to he transmitted can be reduced by increasing a.parallax between
20 viewpoints to be interpolated. TTierefore, as compared with embodiment 1, it is
easj' to widen the angle range that can be transmitted.
[0183]
[4. Third Embodiment of Present Disclosure]
Next, a third embodiment of the present disclosure will be described,
2fl The third embodiment is an embodiment in which the first embodiment and
the second embodiment described above arc combined,
In the first emfcodbnent, the moving'image content of a viewpoint of a
■ certain angle range, for example, n moving image contents from the moving image 1
to the moving image n in FIG 16, is transmitted to the client, and the client performs
SO the processing of selecting the content suitable for the angle or the like of the client
&om the n contents and displaying the selected content ■

SP346695WO0a
53/72

hi the second emhodiment, as illustrated in FIG 17, tiie images of several discrete viewpoints, for example, tiie moving images of only three viewpoints as described in FIG 17 among the moving image contents of the multiple viewpoints of a certain angle range, are transmitted to the client Furthermore, depth information 5 (depth map) correspondir^ to the three moving images to be transmitted is also transmitted to the client. [OIUJ
In the processing of the first embodiment, a more network band is required and the image quality is relatively low, but the processing in liie client is a low load 10 and real-time rendering canbe easily performed.
On the other hand, in the processing of the second embodunent, the
processing in the client is a high load, real-time rendering is difficult, and a battery is
more consumed, but a network band necessary for high hnage quality is small,
[0185J
15 By adaptively switching the two melhods, il is possible to realize the
1 The combined method will be described as the thhd embodiment.
ccmfignration that can deliver the moving image further optimal to the situation.
The CO]
[0136]
Specifically, the third embodiment performs the processing of switching
20 whether to perform the optbnal rnethod having (a) rendering speed and calculation
"performance of the client and (b) network bandwidth as parameters, thai is, one of
theprocessingof the first embodiment and the processing of the second embodiment.
£0187]
Furthermore, the processing may be set to be switched for each content to 2& be superimposed.
For example, content infoimation of (a) content that leqLiires higher image
, quality but is good even when the frame rate is lowered and (b) content that requires
30-:5's rendering but is good even at low image quality is held as metadata
cotrespondirig to the content, and the processing of switching whether to perfomi the
30 ■ optimal method, that is, one of the processing of the first embodiment and the
processing of the second embodiment is performed according to Ihe meladata,

SP346 695 WOOD
54/72
[01S8]
FIG 23 illusft^tes a configuration example of the server 200 and the chent 100 that perfoim the processing of the third embodiment,.
TTie eonfigutafion illustrated in FIG 2S is a configuration in which the 5 following elements are added to the configuration illustrated above in FIG 15 described as the s^Yer/elietit configuration of embodiment 1. [01S9]
Anetworkdelaybandwidlhaequisition section 115 and a chent performance acquisition section 116 of the elienl 100, and a content meta database 20R and a 10 stream switching section 2d9 of die server 200 are added., [0190]
Ineidenlally, the network delay bandwidth acquisition section 115 of the
client 100 m^ be set on the server 200 side.
[0I9IJ
15 The network delay bandwidth acquisition section 115 of the client 100
measures the bandwidth of the commumcati«)n network between the server and the client.
The chent performance acquisition section 116 of the client 100 identifies
the rendering speed performance and the operation performance of the client.
20 The measured data or the identified data are notified to the stream switching
sectLon209oft3]eserver200. ■ "
[0192]
The sQ^am switching section 209 of the server 200 perfomis the processing of switching whedier to perform the optimal method according to input infomiation, 2.5 flial is, one of the processing of the first embodiment and the processing of the second embodiment [0193]
Also, the content meta database 203 stores metadata corresponding to content, for example, metadata corresponding to (a) content that requires high image 30 quality but is good even when the fiame rdte is lowered and (b) content that requires aO-i^'s rendering but is good even at low image qnali^.

55/72 ■ .
[0194] . . ■
The stream switching section 209 of the server 200 acquires metadata
correspoLidmg to transmission content &om the content meta database 208 and
performs ttie processing of switchmg whether to perform the optimal method
S "according to the metadata, that is, one of the processing of the first embodiment and
the processing of the second embodiment
The processing of the remaining configuration is the same processing as
described in embodnuenl 1 and embodiment 2.
. [0155]
10 The present embodiment realizes the configuration tiiat can deliver the
moving image optimal to the situation b^ adaptiveiy switching the processing described as the first embodiment' and the processing described as the second embodiment according to the performance of the client, the available bandwiddi in the network; orthe metadata as the content inforrnaliou. 15 [0196]
[5, OiherEmbodunenfs] »
(a) Configuration usmg no Server
In the respective embodiments described above, the embodiment in which the virtual image is held by tHc server and is provided from the server to the client 20 has been deEcribed.
However, the content made" up of the virtual image or the virtual image
depth map may be stored in a media of the client, for example, a disk-tjfpe media
such as a hard disk, a DVD, or a BD, or a storage section such as a flash memory,
and the client data processing Section may read the content from the storage section,
2& select the virtual image corresponding to the viewpoint, or generate the virtual image
by applying the depth map, and display the virtual image on the display section of
the client. ■ - ^
[0197]
In this case, the processing of the server side in ftie foregoing embodiment is 30 all perfoimed in the client.
Also, the network delay described in ttie foregoing embodiment need not be

SF346695WO00 56/72
considered, and the processing of measuring ttie network delay or the setting of the
angle range considering the network delay is not required.
[019K]
(b) Configuration thai reduces the processing of the cUent
& Furlhetmore, as the configuration that reduces the processing of the client,
for example, in the flowchart illustrated in FIG 5 described as the piocessmg
sequence of the first embodiment, the client may perform only the image acquisition
processing of step SIOl and tiie processing of steps 3113 and SI 14 and the server
may perfoim &e remaining processing.
LO Similarly, in the flowchart illustrated in FIG 18 described as the processing
sequence of the second erabodunent, the chent may perfomi only the image
acquisition processing of step SI 01 and the processing of steps S113 and Si 14 and
the server maj' perform the remaining processing,
[0199]
15 That is, the client transmits the image captured in the client to the server,
and tiie secver performs flie processing of steps SI 02 to S112 based on the
transmission image. The server transmits only the image of one viewpoint to be
displayed in the client to the client, and the chent superimposedly displays the
received image on the captured image.
20 iTiis processing configuration greatly reduces the processing load of the
chent.
[0200]
[6. Summary of Configuration of Present Disclosure]
Tlie embodiments of the present disclosure have been described in detaiJ 25 wilh reference to specific embodunenls. However, it is obvious that modifications
or substitutions of the cmboditnenis can be made by those skilled in the art, without
departmg from the scope of the present disclosure. That is, the present invention is
disclosed in the form of examples and should not be construed as JimiCed thereto.
The scope of the present disclosure should be detcnnined wilh reference to the SO appended cEaims.
[0201]

SF346S95WO00 57/72
Additionally^ the present technology may also be coniigured as below.
(J)
An informationprocessing device including:
an acquisition section conTigured to acquire space position infoimation of a R tenuLiid including a display section that displays an image in vrhich a virtual image is superimposed on a captured im^ge acquired by an imaging section; and
a specifying section configured to specify display infomiation for displaying the virtual image on the displ^ section according to the space position information of the terminal, 10 [0202]
(2) , ' ■' ",' "
Ilie infoimation processing device according to (1)^ tuither including; iUi imaging section configured to capture an image;
a display section configured to display a captured image of the imaging IB section; and
a dala processing section configured lo receive the virtual image frpm a server and superimposedly display the received virtual image on the captured image displayed on the display section,
wherein the data processing section estimates the position of the imaging
20 section afier predetermined time according to motion infoimation of the infomiatian
processing device, transmits the estimated position information to the server, receives
the virtual image captured from the estimated position fiumthe server, and displays
the received image or the image generated based on the received image on the
display section. " - - .
25 [0203]
(3)
The information pirocessing device according to(2),
Whctcin the data processing section estimates the capluring direction of tiie
imaging section after time T + At considering network delay time At that is round trip
30 time of a communication witii, a server by applying position/orientation information
of the imaging section and motion information at current time T, and transmits the

SP34e695WO00 .58/72
estimated direction to the server as the estunated position information,
(4)
The information processing device according to any one of (1) to (3), wherein the da^a processing section calculates probabili^ of the capturing fi direction of file imaging section after time T + At considering network delay time At that is round trip time of a communication with a server by applying position/orientation.infomiation of tiie imaging section and motion infonnation at cnirent time T, selects an angle range of a range with high probability, and receives^ form the sctvcr, virtual images of multiple viewpoints included in the selected angle 10 range or the depth map and the image for virtual image generation. [0204]"
Tlie infonnation processing device according 1o (4), .
wherein the dalaproce^sing section receives, 5^m the server, image data in 15 which an image with a relatively high probability is set as a high-resolution image and an image with a relatively low probability is set as a low-resolution image in the selected angle range.
The information processing device according to (4), ■
20 wherein, when there is no great difference in the probability in the selected
angle range, the data processing section receives, from the server, the virtual images
of the multiple viewpoints included in the selected angle range as images having a
same resolution.
(7) ■
25 The informalion processing device according to any one of (4) to (7),
wherein the data processing section selects a virtual image of a viewpoint close to the capturing direction of the imaging section at a display timing of the virtual image from the virtual images of the multiple viewpoints received from fiie server, or generates a virtual image captured ftom a viewpoint close to the capturing
3D direction of the imaging section at the display timing of the virtual image by applying the depth map and the image for virtual image generation, and displays the

SP340fi9SWOOQ 59/72
virtual image on the display aeelion.
[0205]
. (8) .
An Itifbmiation processing device as a server includmg:
5 a communication section configured to perform communication with a
cli^t; and .
a data processing section conligured to select and transmit a virtual image to
be superiniposedly displayed on a captured image of a client imaging section which
is displayedonadisplay section of the client
10 wherein the data processing section selects a virtual image captured ftom
the position of the client capturing section after predetermined time estimated according to the motion information of the client or a depth map and an image for virtual image generation, as data to be transmitted to the client, and transmits the selected virtual image or the depth map and the image to the client. -1& [0206J
The information processing device accordmg to (8),
wherein the data processing section estimates Ihe capturing direction of the
client imaging section after time T + At considering network delay time At that is
20 ^ round trip time of a communication between the server and the client by applying
' position/orientation iiifbimation of the client imaging section and motion information
at cuirent time T, and selects the virtual image captured in the estimated dhection or
the deptti map and the image for virtual image generation a& data to be transmitted to
the client
25 (10)
The informatioti processing device accordmg to (8) or (9),
wherein the data processing section calculates probability of Ihe capmring
direction of ttie client imaging section after time T + At considering network delay
time AC that is round trip time of a communication between the server and the client
30 ' by applying position/orienmtion infbiniation of the client imaging section and motion
infonnation at current time T, selects an angle range of a range with high probability,

SP34fi635WO00 60/72
and selects tlie virtual images of tiie multiple viewpoints included in the selected angle range as data to be trEmsmitted to the client
01)
; The infomiation processing device according to (10),
5 wherein the data processing section generates image data to be transmitted
in which an image vrith a relatively high probability is set as a high-resolution image and an image wilii a relatively low probability is set as a low-resolution image in the selected angle range, and transmits the image data to tbe client. [0207] 10 (12)
An information processing device including: an imaging section configured to capture an image;
a display section configured to display a captured image of the imaging
section;
15 a storage section configured to store virtual images corresponding to
multiple viewpoints that are virtual images to be superimposedly displayed on a captured image displayed on the display section and are obtained by capturing an objectfrom differentmultipleviewpoints; and
a data processing section configured to acquire the virtual image firom the 20 storage section and superimposedly display the.acquired virtual image oil the captured image displayed on the displ^ section,
wherein the data processing section estimates the position of the imaging section after predetemiined tune according to motion inftirmation of the information processing device, and then selects the virtual image captured fiom the estimated 25 position from ^ storage unit and displays the virtual im^e, or generates the virtual image based on the virtual image acquired trom the storage section and displays tiie virtual image on the display section.
[020S] , ' ' .
(13) / ■
30 The infomiation processing device accordingto (12),
wherein the storage section stores the depth map and the virtual image

SF34669fiWO00
61/72
corresponding to multiple viewpoints obtained by capturing an object from different multiple viewpoints, and
wherein the datapreccBsingGecIion generates and displays ^ virtual image captured from the estimated position on the display section by applying tiie depth S map and the virtual image acquired from the storage section. (14)
The intbrmation processing device according to (12) or (13), wherein Ihe data processing section calculates probability of the capturing direction of the imaging section after predetermined tune according to motion
10 information of the inft>rmation processing device, selects a virtual image with a high probability from the storage section or generates a virtual image based on the viitual image acquired from the storage section, and displays fhe virtual image on the display section. [0209]
15 Furtiier, a method of .processing performed in the above apparatus and
system and a program to execute the processing are included in the configuration of the present disclosure. [0210]
■- Furthermore, the processing sequence that is explained in the specification
20 ■ can be implemented by hardware, by software aijd by a con^guratipn that combines hardware and softw^e. In a case where the processing is implemented by software, it is possible to'install in memory within a computer that is incorporated into dedicated hardware a program in which the processing sequence is encoded and to execute the program. It is also possible to install a program in a general-purpose
25 -computer that is capable ofperformingvarious types of processing and to execute the program. For example, the ,program can he installed in advance in a storage medium, hi addition to being installed in a computer ftrtm the storage medium, tiie program can also be received through a network, such as a local area network (LAN) or the Intemef, and can be installed in a storage medium such as a hard disk or the
3D like thai is built into the compuler. [0211]

SP34G695WO00 62/72
Note tiiat the various types of processing that are described in diis specification may not only be performed in a temporal sequence as has heen described, but may also be performed in parallel or individually, in accordance with the processing capacity of the device that performs the processing or as needed. 5 Furthermore, the system in this specification is not limited to being a configuration that logically aggregates a plurality of devices, all of which are contained wifliin the same housmg
Industrial Applicabihty 10 [0212]
As described atiove. according to the configuration of one embodiment of
the present disclosure, the virtual image, which is natural according to the capturing
direction of the camera, can be superimposedly displayed on the captured camera
image displayed on the display section.
IE Specifically, the configuration includes the servef, that transmits the virtual
image to the chent, and Ih^ client that superimposedly displays the captured camera
■I image and the virtual image transmitted fixtm the server: The server transmits, to
the client, the virtual image capmred from the client position after a predetermined
time estimated according to the motion of the client (camera). In one of the server
20 and the client, by applying the client position/orientation information at the curreTit time T and the motion infoimation, the capturing direction of the client after the time T + At considering the network delay time At that is the round trip time of communication between the server and the client is estimated, and the virtual image captured in fke estimated direction is transmitted to the client
25 According to suth coniigmalions, the virtual image, which is natural
according to the capturing direction of the camera, can he sUperimposedly displayed on the captured camera image displayed on the display section of the client.
Reference Signs List 30 [0313]
10 poster

SP34fi63fiWOflO 63/72
II marker
21,22 nser
30 client (portable terminal)
31 virtual image
5 100 client
101 imaging section (camera)
102 image recognition section
103 client (camera) position/orientation catculation section
104 sensor (acceleration/gyro sensor)
ID 105 client motion speed/direction calculation secljon
1 OS network delay estimation section
107 angle range estimation sectioTJ
108 communication section
109 content selection and superimposition position calculation section
IB 110 decoding section
III display control"section "
112 display section
J15 network delay bandwidth acquisition section
116 client performance acquisition settioh
20 200 server
201 content database
202 depth information (depth map) generation section
203 content correspondence depth information (depth map)
204 middle viewpoint image generation 'section -
2& 205 content according to each viewpoint
206 transmission image data generation section
207 coimnunication section
208 content meta database
209 stream switching section 30

WE CLAIMS:-
An information processing device comprising:
an acquisition section configured to acquire Space position information of a terminal including a display section fiiat displays an image in which a Virtual image is superimposed on a c<^tured image acquired by an imaging section; and
a specifying section configured to specif disjil^ information for displaying the virtual image on the display section according to the space position information of the terminal.
10
Claim 2
The inibrmation processing device according to claim 1, further comprising; an imaging section configured to capture an image; . a display section configured to display a captured image of the imaging 15 secdon; and
a data processing section configured to receive the virtual image from a server and supcfimpOBcdly display the received virtual image on the captured image displayed on the display section,
wherein the data pr<)cessing section estimates the position of the imaging
20 section after predetermined time according to motion-information of the information
processing device, transmits the estimated position infoimation to the server, receives
the virtual image captured trom the estimated position or the depth map and the
image for virtual image generation from the server, and displays the received image
or the image generated based on the received image on die display section.
25 ' "
Claun3
The informationprocessing device according to claim 2,
wherein the data processing section estimates ttic capturing direction of the
imaging section after lime T +"At considering network delay time it that is round trip
30 time of a communication with a server by applying position/orientation information
of the imaging section and motion informafion at current time T» and transmits the

SP346G9BWO00 65/72
eslimated direction to the server as the estimated position information.
Claim 4
The infcimation processing device according to claim 2, ..
5 wherein the data processing section calcidales probability of the capturing
direotion of the imaging section after time T + At considering network delay time At that is round trip time of a communication with a server by applying position/orientation inibrmation of the imaging section and motion information Eit cutienl time T, selects an angle range of a range with high probability, and receives, IW form the server^ virtual images of multiple viewpoints included in the selected angle range or tiie depth map and the image for virtual image generation.
Claims ■
The information processihg device accordingto claim 4,
15 wherein the data processing section receives, &oni the s^er, image data in
Which an image with a relatively high probability is set as a higt-iesolution image and an image with a relatively low probability is set as a low-resolution image in the selected angle range.
20 Ciaim 6
, The information processing device according to claim. 4, wherein, when there is no great difference in the probability in the selected angle range, the data processing section receives, from, the server, the virtual images of the multiple viewpoints included in the selected angle range as images having a 25 same resolution.
Claim 7
The information processing device according to claim 4, wherein the data processing section selects a virtual image of a viewpoint 30 close to the capturing direction of the imaging section at a display "Eimiiig of the
virtual image ftom the virtual images of the multiple viewpoints received &om the

SP34GB3fiWO0D 66/72
server, or generates a virtual image captured from a viewpoint ctosc to the capturing
direction of the imaging section at the display timing of the virtual image by
applying the depth m^ and the image for virtual image generation, and displays the
virtual image on Lhe display section.
5 . '
Chhn 8
An information processing device as a server comprising:
a communication section configured to perform communication with a
client and
10 a data processing section configured to select and transmit a virtual image to
be superimposedly displayed on a captured image of a client imaging section which
is displayed on a display section of the client,
wherein Che data processing section selects a virtual image captured &om
the position of tiie client capturing section aft^ predetermined time estunated 15 according to' the motion information of the client or g depth map and an image for
virtual image generation as data to be transmitted to the client, and transmits tiic
selected virtual image or the depth map and the image to the client.
Clami9
20 The information processing device accof dmg to claim S,
wherein the data processing section estunates the capturing direction of the cHent imaging section after time T + At considering network delay time At iJiat is round trip time of a communication between the server and the client by ^plying positiorf/orientation information of the client imaging section and motion information 25 at current time T, and selects the virhial image captured in the eslimated direction or the depth map and the image for virtual image generation as data to be transmitted to , the client
Claim 10 . .
30 The information processing device accordinglo claim S,
. whereiti the data processing section calculates probahihty of the capturing

SP346636WO00 67/72
direction of the dienl imaging section alter time T -I- At considering network delay "time at tiiat is round trip time of a commULiication between the server and the client by applying position/orientation information of the client imaging section and motion information at current time T, selects an angle range of a range with high probabihty, 6 and selects the virtAial images of the multiple viewpoints included in the selected angle range as data to be transmitted to the cHent
Claun 11
The information processing device according to claim 10, .
10 wherein the data processing section generates iniage data to be transmitted
in which an image wilh a relatively high probability is set as a high-resolution image ' and an image with a relatively low probability is set as a low-resolution image in the selected angle range, and transmits tiie image data to the client.
15 Claim 12
An information processing device comprising:
an imaging section configured to capture an image;
. a display section configured to display a captured image of the imaging
section; ...
20 a stoiage section configured to store virtual images corresponding to
multiple viewpoints that are virtual irnages to be superimposedLy displayed on, a captured image displayed on the display" section and are obtained by capturing an objectfrom different multiple viewpoints; and
a data processing section configured to acquire the virtual image fiom the 26 storage section and. superimposedly display the acquired virtual image on the captured image displayed on the display section,
wherein the data processing section estimates the position of the imaging
section after ptedetemiined time according to motion infomiation of the information
processing device, ^id then Selects the virtual image Captured fiom the estimated
30 position from the storage unit and displays the virtual image, or generates the virtual
image based on the virtual image acquired Irom the storage section and displays the

SP346fi95WO00 6a/72
virtual image rai the display section.
Claim 13
The infonnalion processing device according to claim 12,
5 wherein the storage section stores the depth map and the virtual image
corresponding to multiple viewpoints obtained by capturing an object from different tnultiple viewpoints, and
wherein the data processing section generates and displays the virtual image captured from the estimated position on the display section by applying the depth 10 map and the virtual image acquired from the storage section.
Claim 14
The information processing device according to claim 12, wherein the data processing section calculates probabili^ of the capturing 15 direction of flie imaguig section afrer piedeteniiined time according to motion information of the infrimiation processing device, stletts a virtual image with a high ptobability from the storage section or generates a virtual image based on the virtual image acquired from the stor^ section, and displays the virtual image on the display section. 20
Claim 15 ^ ^
An information processing method^ which is per&jmed by an infonnation proceasmg device,
whereintheinfomiation processing device comprises: -_ .
25 a communication section configured to perform communication
with a client; and
a data processing section configured to select and transmit a virtu^
image to be superimposedly disp^ayed on a captured image of a chent iniagmg
section which is displayed on a display section of the client, and
30 wherein the data processing section selects a virtual image captured from
the position of the client capturing seclion after predetermined tune estimated

aP346695WO00 69/72
according to the motion iiiformetion of the client or a depth map and an image for virtual image generation as data to be transmitted to the client, and transmits the selected image to the client
5 Claunie
I An ijifbmiation processing mettiod, which is perfonneil by an information
processing device,
wherein a data processing section acquires space position information of a terminal including a display section that displays an image in which a virtual image 10 is superimposed on a cjqjtured image acquired by an imaging section, and '
wherein the data processing section speciiies display information for displaying the virtual image on the display section according to the space position information of the lerminaL
15 aaim 17 ■ - .
4 . An information processing method, which is performed by an infomialion
processing device,
wherein the information processing device comprises:
an imagmg section configured to capture an image;
20 a display section configured to display a captm^ed image of the
imaging sectioui
a storage section configured to store virtual images corresponding to multiple viewpoints thai are virtual images to be superimptisedly displayed oi^ a captured image displayed on the display section and fliat have a captured object irom 25 different multiple viewpoints^ and
a data processing section configured to acquire the virtual image fiom the storage section arid superimposedly display fiie acquired virtual image on the captured image displayed onfiie display section, ■
wherein, the data processing section estimates the position of the imaging
30 section a:[ter predetermined time according to motion information of the information
processing device, and then selects the virtual image captured, from-the estimated

SP34fi695Wonn
70/72

position from the storage section and displays the virtual image, or generates the virtual image Ijased on the virtual image acquired from the storage section, and displays the virtue image on the display section.
5 Claim IS
A program for causing an information processing device to execute
information processing, - - -
wherein the information processing device comprises:
a communicalion section configtired to perform cutnmunication 10 with a client; and
a data processing section configLircd to select and transmit a virtual ' linage to be superimposedly displayed on a captured >niage of a client imaging section which is displayed on a display section of the client,
wherein tire program causes the data processing section io select a virtual 15 , image captured from the position of the client capturing section after predetermined
an image fijr virtual image generation ss data to be transmitted to the chent, and
time estimated according to the motion information of the client or a depth map and an image fijr virtual image "generation transmit the selected image to the chent
20 Claim 19
A program for causing an information processing device to execute information processing of:
causing a data processing section to acquire space position information of a terminal including a display section that displays an image in which a virtual image 25 ' is superimposed on a c^tured image captured by an imaging section; and
causing the data processing section to specif display information for displaying the virtual image on ttie display section, accordmg to the space position infomiation'of the terminal.
30 . Claim 20 .
A program for causing an information processing device to execute

spa4sefi5wooo
71/72
infoimation pro cessuLg,;
wherein the infonnation processing device comprises: . '
an imaging section configured to capture an image;
a display section configured to display a captured image of the
&- imaging section; - ,
, a storage section configured to store virtual unages coitcsponiling
to multiple Viewpoints tiiat are virtual images to.be superimposedly displayed on a
captured image displayed on the display section and that have captured "an object
fiom different multiple viewpoints; and
10 ' - a data processing section configured to acquire the virtual image
&om the storage section and superimposedly display the acquired virtual image on the cE^tuied image displayed on the display section,
wherein the program causes the data processing section to estimate the position of the unaging section after predetermined time according to motion 15 information of the information processing device^ and then selects the virtual image captured firomthe estintatedposilion&omthe storage section and displays the virtual image, or generates the virtual image based on the virtual image acquired fiom the storage section, and display the virtual image on the display section.