[Title of Document] REQUEST FOR PATENT
[Reference number] 308490JP00
[Filing Date] 7 October 2010
[Address] JPO Commissioner
5 [Int.Cl.] H04N 7/00
[Inventor]
[Domicile or Residence] c/o SONY CORPORATION,
1-7-1 Konan, Minato-ku, Tokyo, Japan
[Name] SAKAI, Yusuke
10 [Inventor]
[Domicile or Residence] c/o SONY CORPORATION,
1-7-1 Konan, Minato-ku, Tokyo, Japan
[Name] KONDO, Masao
[Applicant for Patent]
15 [Identification Number] 000002185
[Name] SONY CORPORATION
[Agent]
[Identification Number] 100095957
[Patent Attorney]
20 [Name] KAMEYA, Yoshiaki
[Telephone Number] 03-5919-3808
[Appointed Agent]
[Identification Number] 1000963 89
[Patent Attorney]
25 [Name] KANEMOTO, Tetsuo
[Telephone Number] 03-3226-6631
[Appointed Agent]
[Identification Number] 100101557
[Patent Attorney]
30 [Name] HAGIWARA, Yasushi
[Telephone Number] 03-3226-6631
1/54
[Appointed Agent]
[Identification Number] 100128587
[Patent Attorney]
[Name] MATSUMOTO, Kazunori
5 [Telephone Number] 03-5919-3808
[Fee]
[Prepayment Ledger Number] 005094
[Amount] JPY 15,000-
[List of Attached Documents]
10 [Title of Document] Specification 1
[Title of Document] Claims 1
[Title of Document] Abstract 1
[Title of Document] Drawings 1
[Number of General Power of Attorney] 1003295
2/54
[Title of Document] SPECIFICATION
[Title of the Invention] INFORMATION PROCESSING DEVICE AND
INFORMATION PROCESSING METHOD
[Technical Field]
5 [0001]
The present disclosure relates to an information processing device and an
information processing method.
[Background Art]
[0002]
10 Telephones, so-called videophones, video conferencing systems and the like are used
as devices for interaction between users at remote locations. Further, a method of
performing text chat or video chat with video and speech through Internet connection
using personal computers and the like is also used.
[0003]
15 In such a background, devices such as electronic equipment are becoming more
sophisticated and multifunctional, and, a television receiver, for example, has a
network communication function so that it can not only receive and display video
and speech contents of a program from a broadcasting station but also exchange
various kinds of information with another receiver.
20 [0004]
For example. Patent Literature 1 discloses a technique that, when a user views a
program content of television broadcasting with a television receiver, displays
information related to other users registered (e.g., videos of other users, thumbnail
images of other users, names, channels or videos of contents viewed by other users
25 etc.) concurrently with the program content.
[Prior Art Document]
[Patent Literature]
[0005]
[Patent Document 1] JP 2006-50370A
30 [Summary of the Invention]
[Object to be Achieved by the Invention]
3/54
[0006]
In the technique disclosed in Patent Literature 1 described above, the television
receiver receives a video or speech from the other receiver(s) in one location or a
plurality of locations and plays the received video or speech concurrently with the
5 display of the program content, for example, to thereby enable the user to share the
same program content and communicate face to face with a user at a remote location.
[0007]
However, in the case where there are a plurality of users of the other receiver in one
location, when any one of the plurality of users of the other receiver performs some
10 act of expression such as speech, for example, it is difficult for a user of the
television receiver to be aware of which user of the plurality of users of the other
receiver has performed the expression from the playback of the video or speech
received from the other receiver.
[0008]
15 In light of the foregoing, it is desirable to provide novel and improved information
processing device and information processing method that enable a user to be aware
of which user of another information processing device has performed an expression.
[Solution for Achieving Object]
[0009]
20 To solve the above problem, according to one aspect of the present disclosure, there
is provided an information processing device comprising communication unit that
receives a predetermined signal related to an expression of a user of a
communication target device from the communication target device, a recognition
unit that detects a face from a captured image in the predetermined signal received
25 by the communication unit, and recognizes a user of the communication target device
based on a detection result of the face, and a confrol unit that displays a display
image where a user image representing the user of the communication target device
recognized by the recognition unit is displayed for each user on a display screen,
wherein the recognition unit further recognizes a user having performed the
30 expression and a detail of the expression from the predetermined signal received by
the communication unit, and the control unit displays a display image where the user
4/54
image representing the user having performed the expression recognized by the
recognition unit and the detail of the expression recognized by the recognition unit
are further displayed in association with each other in the display image on the
display screen.
5 [0010]
The recognition unit may recognize a user whose facial expression has changed as
the user having performed the expression and a facial expression as the detail of the
expression based on the detection resuh of the face.
[0011]
10 The recognition unit may recognize a speaker as the user having performed the
expression and a presence of a speech as the detail of the expression from a speech in
the predetermined signal received by the communication unit.
[0012]
The recognition unit may further recognize contents of a speech by the speaker as the
15 detail of the expression from the speech received by the communication unit.
[0013]
The recognition unit may recognize an operator as the user having performed the
expression and a specific operation as the detail of the expression from an operation
signal in the predetermined signal received by the communication unit.
20 [0014]
The information processing device further comprises an input unit that allows input
of a predetermined signal related to an expression of a user of the information
processing device, wherein the recognition unit may detect a face from a captured
image in the predetermined signal input to the mput unit, and recognize a user of the
25 information processing device based on a detection result of the face, the control unit
may display a display image where a user image representing the user of the
information processing device recognized by the recognition unit is further displayed
for each user on the display screen, the recognition unit may further recognize a user
having performed the expression and a detail of the expression from the
30 predetermined signal input to the input unit, and the control unit may display a
display image where the user image representing the user having performed the
5/54
expression recognized by the recognition unit and the detail of the expression
recognized by the recognition unit are further displayed in association with each
other in the display image on the display screen.
[0015]
5 The recognition unit may detect a face from a captured image m the predetermined
signal input to the input unit, and recognize a user whose facial expression has
changed as the user having performed the expression and a facial expression as the
detail of the expression based on the detection result of the face.
[0016]
10 The recognition unit may recognize a speaker as the user having performed the
expression and a presence of a speech as the detail of the expression from a speech in
the predetermined signal input to the input unit.
[0017]
The recognition unit may further recognize contents of a speech by the speaker as the
15 detail of the expression from the speech input to the input unit.
[0018]
The recognition unit may recognize an operator as the user having performed the
expression and a specific operation as the detail of the expression from an operation
signal in the predetermined signal input to the input unit.
20 [0019]
To solve the above problem, according to another aspect of the present disclosure,
there is provided an information processing method comprising receiving a
predetermined signal related to an expression of a user of a communication target
device from the communication target device, detecting a face from a captured image
25 in the received predetermined signal, recognizing a user of the communication target
device based on a detection result of the face, and further recognizing a user having
performed the expression and a detail of the expression from the received
predetermined signal, and displaying a display image where a user unage
representing the recognized user of the communication target device is displayed for
30 each user on a display screen, wherein in the display image, the user image
representing the recognized user havmg performed the expression and the recognized
6/54
detail of the expression are displayed in association with each other.
[Advantage of the Invention]
[0020]
As described above, the present disclosure enables a user to be aware of which user
5 of another information processing device has performed an expression.
[Brief Description of Drawings]
[0021]
[FIG. 1]
Fig. 1 is an explanatory view illustrating an example of a display system according to
10 an embodiment of the disclosure.
[FIG. 2]
Fig. 2 is an explanatory view illustrating an example of a functional configuration of
an information processing device according to a first embodiment of the disclosure.
[FIG. 3]
15 Fig. 3 is an explanatory view illustrating an example of a hardware configuration of
the information processing device in Fig. 2.
[FIG. 4]
Fig. 4 is a flowchart of a display process executed by the information processing
device in Fig. 2.
20 [FIG. 5]
Fig. 5 is a flowchart of user display processing performed in Step S114 in the display
process in Fig. 4.
[FIG. 6] (A) to (D) are explanatory views illustrating an example 1 of the disclosure.
[FIG. 7] (A) to (D) are explanatory views illustrating the example 1 of the disclosure.
25 [FIG. 8] (A) to (D) are explanatory views illustrating an example 2 of the disclosure.
[FIG. 9] (A) to (D) are explanatory views illustrating an example 3 of the disclosure.
[FIG. 10] (A) and (B) are explanatory views illustrating an example 4 of the
disclosure.
[FIG. 11] (A) to (E) are explanatory views illustrating an example 5 of the
30 disclosure.
[FIG. 12] (A) and (B) are explanatory views illustrating an example 6 of the
7/54
disclosure.
[FIG. 13] (A) to (L) are explanatory views illustrating an example 7 of the
disclosure.
[FIG. 14]
5 Fig. 14 is an explanatory view illustrating an example of a functional configuration
of an information processing device according to a second embodiment of the
disclosure.
[Mode for Carrying out the Invention]
[0022]
10 Hereinafter, preferred embodiments of the present disclosure will be described in
detail with reference to the appended drawings. Note that, in this specification and
the appended drawings, structural elements that have substantially the same fimction
and structure are denoted with the same reference numerals, and repeated
explanation of these structural elements is omitted.
15 [0023]
Note that the description will be provided in the following order.
1. Display System
2. Functional Configuration of Information Processing Device (First Embodiment)
3. Hardware Configuration of Information Processing Device
20 4. Display Process
5. Examples
6. Functional Configuration of Information Processing Device (Second Embodiment)
[0024]
(1. Display System)
25 A display system according to an embodiment of the disclosure is described firstly.
Fig. 1 is an explanatory view illustrating an example of a display system according to
the embodiment. Fig. 1 is a fi"ont view where the display system is viewed from the
front.
[0025]
30 In Fig. 1, a display system 100 includes a display device 102 and an imaging device
104, for example.
8/54
[0026]
The display device 102 can display still images or moving images based on drive
signals. For example, the display device 102 displays still images or moving
images by liquid crystals. Note that the display device 102 may display still images
5 or moving images using a self-luminous display device such as an organic EL
(Electro-Luminescence).
[0027]
The imaging device 104 is placed at the center of the upper part of the display device
102, and it can capture a subject image in the display direction of the display device
10 102. The imaging device 104 may capture a still image or a moving image using a
CCD (Charge Coupled Device) image sensor or using a CMOS (Complementary
Metal Oxide Semiconductor) image sensor.
[0028]
Note that, although the imaging device 104 is mounted at the center of the upper part
15 of the display device 102 in this embodiment, the place where the imaging device
104 is mounted is not limited to the center of the upper part of the display device
102. For example, the imaging device 104 may be mounted at the center of the
lower part of the display device 102. Further, although one imaging device 104 is
mounted in this embodiment, the number of imaging devices 104 is not limited to
20 one. For example, two or more imaging devices 104 may be moimted.
Furthermore, although the display device 102 and the imaging device 104 are
integrated in a single piece in this embodiment, the display device 102 and the
imaging device 104 may be separated into different pieces.
[0029]
25 Note that, in the display system 100, a sensor (not shown) that detects the presence or
absence of a user in front of the display device 102, a signal receiving unit (not
shown) that is capable of receiving a control signal by infrared ray or radio wave
from a remote controller (not shown) may be mounted. The control signal is an
example of an operation signal of the disclosure. Further, the sensor may be
30 capable of detecting the distance between the display device 102 and a user in front
of the display device 102. Furthermore, a soimd acquisition means that is capable
9/54
of acquiring the surrounding sound, such as a microphone, for example, may be
mounted in the display system 100.
[0030]
The display device 102 according to the embodiment can display contents of still
5 images or moving images, and display images generated by an information
processing device 200 in a display process of Fig. 4, as described later.
[0031]
Further, the imaging device 104 according to the embodiment can capture still
images or moving images of users A and B looking at the display screen of the
10 display device 102 in Fig. 2.
[0032]
(2. Functional Configuration of Information Processing Device (First Embodiment))
Next, a flinctional configuration of an information processing device according to a
first embodiment of the disclosure is described. Fig. 2 is an explanatory view
15 illustrating an example of a functional configuration of an information processing
device according to the embodiment. Note that Fig. 2 shows the display system 100
that transmits a captured image to the information processing device 200, which is an
information processing device according to the embodiment, and receives a signal
for driving the display device 102 fi-om the information processing device 200, and
20 users A and B, which are users of the display system 100 and the information
processing device 200. Fig. 2 further shows a communication network 800 to
which the information processing device 200 can connect, a communication server
300 and other information processing devices 500 and 700 that can connect to the
communication network 800, another display system 400 that transmits a captured
25 image to the information processing device 500 and receives a signal from the
information processing device 500, users C and D, which are users of the display
system 400 and the information processing device 500, another display system 600
that transmits a captured image to the information processing device 700 and
receives a signal from the information processing device 700, and users E and F,
30 which are users of the display system 600 and the information processing device 700.
The other display systems 400 and 600 have the same configuration as the display
10/54
user, and comparing the computed user face information with user face information
223 for each user which is previously computed and stored in the storage unit 222.
The act of appearing in a captured image is an example of the expression in the
disclosure. Note that the face recognition unit 204 may further recognize the
5 gender, age or the like of users appearing in the captured unage. Then, the face
recognition unit 204 transmits a result of recognition of the users of the information
processing device 500, 700 to the display image generation unit 206. Further, the
face recognition unit 204 may recognize users of the information processing device
200, which are the users A and B, by receiving a captured image from the image
10 input unit 202 and recognizing the faces of users of the information processing
device 200 from the received captured image. The captured image from the image
input unit 202 is an example of a predetermined signal related to the expression of
users of the information processing device according to the disclosure. The face
recognition unit 204 may then transmit a result of recognition of the users of the
15 information processing device 200 to the display image generation unit 206 or the
communication unit 212.
[0036]
Further, the face recognition unit 204 may recognize facial expressions of the users
of the other information processing device 500, 700 from the captured image from
20 the other information processing device 500, 700. The act of changing facial
expressions is an example of the expression in the disclosure. Then, the face
recognition unit 204 may fransmit a result of recognition of the facial expressions of
the users of the other information processing device 500, 700 to the display image
generation unit 206. Note that the face recognition unit 204 may recognize facial
25 expressions of the users of the information processing device 200 from the captured
image from the image input unit 202. Then, the face recognition unit 204 may
transmit a result of recognition of facial expressions of the users of the information
processing device 200 to the display image generation unit 206 or the
communication unit 212.
30 [0037]
The display image generation unit 206, which is an example of a confrol unit in the
12/54
disclosure, receives a result of recognition of the users of the other information
processing device 500, 700 from the face recognition unit 204 and fiirther receives a
captured image captured in the other display system 400 or a captured image
captured in the other display system 600 which is received by the communication
5 unit 212. Then, based on the received result of recognition of the users of the other
information processing device 500, 700, the display image generation unit 206
generates a display image in which user images representing the users of the other
information processing device 500, 700 are displayed for each user. The detail of
the display image is described later. Note that the display image generation unit
10 206 may generate the user images in the display image by extracting the users' facial
parts from the received captured image captured in the other display system 400 or
the received captured image captured in the other display system 600. Further, the
display image generation unit 206 may use an avatar image representing each user,
for example, as the user images in the display image. Then, the display image
15 generation unit 206 transmits the generated display image to the display control unit
210.
[0038]
Further, when the display image generation unit 206 receives a captured image
captured in the display system 100 from the image input unit 202, receives a resuh of
20 recognition of the users of the information processing device 200 and a result of
recognition of the users of the other information processing device 500, 700 from the
face recognition unit 204, and receives a captured image captured in the other display
system 400 or a captured image captured in the other display system 600 which is
received by the communication unit 212, the display image generation unit 206
25 generates a display image in which user images representing the users of the
information processing device 200 and the other information processing device 500,
700 are displayed for each user based on the received result of recognition of the
users of the information processing device 200 and users of the other information
processing device 500, 700.
30 [0039]
Further, when the communication unit 212 receives a result of recognition of users
13/54
system 100, and detailed explanation thereof is omitted. Further, the other
information processing devices 500 and 700 have the same configuration as the
information processing device 200, and detailed explanation thereof is omitted. The
other information processing devices 500 and 700 are examples of a communication
5 target device in the disclosure.
[0033]
In Fig. 2, the information processing device 200 includes an image input unit 202, a
face recognition unit 204, a display image generation unit 206, an output unit 208, a
display control unit 210, a communication unit 212, a speech input unit 214, a speech
10 recognition unit 216, an operation signal input unit 218, an action recognition unit
220, and a storage unit 222, for example.
[0034]
The image input unit 202, which is an example of an input unit in the disclosure,
receives a captured image generated by imagmg in the imaging device 104. Then,
15 the image input unit 202 transmits the received (input) captured image to the
communication unit 212. Note that the image input unit 202 may transmit the
received captured image to the face recognition unit 204 or the display image
generation unit 206.
[0035]
20 The face recognition unit 204, which is an example of a recognition unit in the
disclosure, receives a captured image fi*om the other information processing device
500, 700 which is received by the communication unit 212, detects faces of users of
the information processing device 500, 700 fi-om the received captured image, and
recognizes users of the other information processing device 500, 700, i.e., the users C
25 and D or the users E and F, based on the detection result of faces. The captured
image fi-om the other mformation processing device 500, 700 is an example of a
predetermined signal related to the expression of users of the communication target
device according to the disclosure. For example, the face recognition unit 204
recognizes users appearing in the captured image by detecting a face part of each
30 user from the received capture image, computes information about the user's face
(hereinafter referred to as "user face information") fi-om the detected face part of the
11/54
from a captured image captured in the other display system 400 or a result of
recognition of users from a captured image captured in the other display system 600,
the display image generation unit 206 may receive the result of recognition and
generate a display image based on the result of recognition received from the
5 communication unit 212 in stead of a result of recognition by tiie face recognition
unit 204. In this case, the above-described face recognition unit 204 may skip the
recognition of users from the captured image from the other information processing
device 500, 700.
[0040]
10 Further, when the display image generation unit 206 further receives a result of
recognition of facial expressions of the users of the other information processing
device 500, 700 from the face recognition unit 204, the display image generation unit
206 generates a display image in which a user image representing a user whose facial
expression has changed and the facial expression are displayed in association with
15 each other in the above-described display image based ftirther on the received result
of recognition of facial expressions of the users of the other information processing
device 500, 700. For example, when the display image generation unit 206
recognizes that a facial expression of a certain user has changed to smiling, the
display image generation unit 206 generates a display image in which a user image
20 representing the user is displayed like jumping.
[0041]
Further, when the display image generation unit 206 fiirther receives a result of
recognition of facial expressions of the users of the information processing device
200 from the face recognition unit 204, the display image generation imit 206
25 generates a display image in which a user image representing a user whose facial
expression has changed and the facial expression are displayed in association with
each other in the above-described display image based fiirther on the received result
of recognition of facial expressions of the users of the information processing device
200.
30 [0042]
Further, when the communication unit 212 receives a result of recognition of facial
14/54
expressions of users from a captured image captured in the other display system 400
or a result of recognition of facial expressions of users from a captured image
captured in the other display system 600, the display image generation unit 206 may
receive the result of recognition and generate a display image based on the result of
5 recognition received from the communication unit 212 in stead of a result of
recognition by the face recognition unit 204.
[0043]
Further, when the display image generation unit 206 receives a result of recognition
of a speaker among the users of the other information processing device 500, 700
10 from the speech recognition unit 216, the display image generation unit 206
generates a display image in which a user image representing the speaker and the
presence of a speech are displayed in association with each other in the abovedescribed
display image based fiirther on the received result of recognition of the
speaker among the users of the other mformation processing device 500, 700. For
15 example, the display image generation unit 206 generates a display image in which a
user image representing the speaker is displayed in a magnified form.
[0044]
Further, when the display image generation unit 206 fiirther receives a result of
recognition of a speaker among the users of the information processing device 200
20 from the speech recognition unit 216, the display image generation unit 206
generates a display image in which a user image representing the speaker and the
presence of a speech are displayed in association with each other based fiirther on the
received resuh of recognition of the speaker among the users of the information
processing device 200.
25 [0045]
Further, when the communication unit 212 receives a result of recognition of a
speaker from a speech input to the other information processing device 500 or a
resuh of recognition of a speaker from a speech input to the other information
processing device 700, the display image generation unit 206 may receive the result
30 of recognition and generate a display image based on the result of recognition
received from the communication unit 212 in stead of a result of recognition by the
15/54
speech recognition unit 216.
[0046]
Further, when the display image generation unit 206 further receives a result of
recognition of contents of a speech by a speaker among the users of the other
5 information processing device 500, 700 from the speech recognition unit 216, the
display image generation unit 206 generates a display image in which a user image
representing the speaker, the presence of a speech, and the contents of the speech are
displayed in association with one another based further on the received resuh of
recognition of the contents of the speech by the speaker among the users of the other
10 information processing device 500, 700. For example, the display image generation
unit 206 generates a display image in which a character image representing the
contents of the speech is displayed within a dialogue balloon from a user image
representing the speaker.
[0047]
15 Further, when the display image generation unit 206 further receives a result of
recognition of contents of a speech by a speaker among the users of the information
processing device 200 from the speech recognition unit 216, the display image
generation unit 206 generates a display image in which a user image representing the
speaker, the presence of a speech, and the contents of the speech are displayed in
20 association with one another based further on the received result of recognition of the
contents of the speech by the speaker among the users of the information processing
device 200.
[0048]
Further, when the communication unit 212 receives a result of recognition of
25 contents of a speech by a speaker from a speech input to the other information
processing device 500 or a resuh of recognition of contents of a speech by a speaker
from a speech input to the other information processing device 700, the display
image generation unit 206 may receive the result of recognition and generate a
display image based on the result of recognition received from the communication
30 unit 212 in stead of a result of recognition by the speech recognition unit 216.
[0049]
16/54
Further, when the display image generation unit 206 receives a result of recognition
of actions of the users of the other information processing device 500, 700 from the
action recognition unit 220, the display image generation unit 206 generates a display
image in which a user image representing an operator user and a specific operation
5 are displayed in association with each other based fiirther on the received result of
recognition of actions of the users of the other information processing device 500,
700. For example, the display image generation unit 206 generates a display image
in which a content image which the operator user intends to display is displayed
within a dialogue balloon from a user image representing the operator user.
10 [0050]
Further, when the display image generation unit 206 receives a result of recognition
of actions of the users of the information processing device 200 from the action
recognition unit 220, the display image generation unit 206 generates a display image
in which a user image representing an operator user and a specific operation are
15 displayed in association with each other based fiirther on the received result of
recognition of actions of the users of the information processing device 200.
[0051]
Further, when the communication unit 212 receives a result of recognition of actions
of users from an operation signal input to the other information processing device
20 500 or a result of recognition of actions of users from an operation signal input to the
other information processing device 700, the display image generation unit 206 may
receive the result of recognition and generate a display image based on the result of
recognition received from the communication unit 212 in stead of a resuh of
recognition by the action recognition unit 220.
25 [0052]
The output unit 208 receives a signal for driving the display device 102 from the
display control unit 210, and transmits the received signal to the display device 102.
Further, when the output unit 208 receives a speech input to the other information
processing device 500, 700 which is received by the communication unit 212, the
30 output unit 208 may transmit the received speech to a speaker unit (not shown) or the
like of the display system 100.
17/54
[0053]
The display control unit 210, which is an example of a control in the disclosure,
receives a display image from the display image generation unit 206. Further, a
content of a still image or a moving image is supplied to the display control unit 210.
5 Then, the display control unit 210 transmits a signal for displaying the display unage
generated by the display image generation unit 206 on the display screen of the
display device 102 to the output unit 208. Further, the display control unit 210 may
transmit a signal for displaying the content on the display screen of the display
device 102 to the output unit 208.
10 [0054]
The communication unit 212 receives the captured image captured in the other
display system 400, and the speech and the operation signal input to the other
information processing device 500 from the communication server 300 through the
communication network 800. Further, the communication unit 212 receives the
15 captured image captured in the other display system 600 and the speech and the
operation signal input to the other information processing device 700 from the
communication server 300 through the communication network 800. Note that the
communication unit 212 may receive the captured image captured in the other
display system 400 and the speech and the operation signal input to the other
20 information processing device 500 directly from other information processing device
500 through the communication network 800. Likewise, the communication unit
212 may receive the captured image captured in the other display system 600 and the
speech and the operation signal input to the other information processing device 700
directly from other information processing device 700 through the communication
25 network 800.
[0055]
Further, the communication unit 212 may receive a result of recognition of users
from the captured image captured in the other display system 400, a result of
recognition from the speech or a result of recognition from the operation signal input
30 to the other information processing device 500, a result of recognition of users from
the captured image captured in the other display system 600, or a result of
18/54
recognition from the speech or a result of recognition from the operation signal input
to the other information processing device 700 from the communication server 300
through the communication network 800. Note that the communication unit 212
may receive a result of recognition of users from the captured image captured in the
5 other display system 400 and a result of recognition from the speech or a result of
recognition from the operation signal input to the other information processing
device 500 directly from the other information processing device 500 through the
communication network 800. Likewise, the communication unit 212 may receive a
result of recognition of users from the captured image captured in the other display
10 system 600 and a result of recognition from the speech or a result of recognition from
the operation signal input to the other information processing device 700 directly
from the other information processing device 700 through the communication
network 800.
[0056]
15 Further, the communication unit 212 may receive the captured image from the image
input unit 202, the speech from the speech input unit 214, and the operation signal
from the operation signal input unit 218, and transmit the received captured image,
speech and operation signal to the communication server 300 through the
communication network 800. Further, the communication unit 212 may receive a
20 result of recognition of users from the captured image captured in the display system
100 from the face recognition unit 204, a result of recognition from the speech input
to the information processing device 200 from the speech recognition unit 216, and a
result of recognition from the operation signal input to the information processing
device 200 from the action recognition unit 220, and fransmit the received results of
25 recognition to the communication server 300 through the communication network
800. Note that the communication unit 212 may transmit the received captured
image, speech and operation signal, the received result of recognition of users from
the captured image, result of recognition from the speech and result of recognition
from the operation signal directly to the other information processing device 500,
30 700 through the communication network 800.
[0057]
19/54
The speech input unit 214, which is an example of an input unit in the disclosure,
receives a speech from the users A and B, which are users of the information
processing device 200, for example. Then, the speech input unit 214 transmits the
received (input) speech to the communication unit 212. Further, the speech input
5 unit 214 may transmit the received speech to the speech recognition unit 216.
[0058]
The speech recognition unit 216, which is an example of a recognition unit in the
disclosure, receives a speech from the other information processing device 500, 700
which is received by the communication unit 212, and recognizes a speaker among
10 the users of the other information processing device 500, 700 from the received
speech. The act of speaking in an example of the expression in the disclosure.
The speech from the other information processing device 500, 700 is an example of a
predetermined signal related to the expression of users of the communication target
device according to the disclosure. For example, the speech recognition unit 216
15 recognizes a speaker by extracting a feature from the received speech, generating
information about the user's speech (hereinafter referred to as "user speech
information"), and comparing the generated user speech information with user
speech information 224 for each user which is previously generated and stored in the
storage unit 222. Then, the speech recognition unit 216 fransmits a result of
20 recognition of a speaker among the users of the information processing device 500,
700 to the display image generation unit 206. Further, the speech recognition unit
216 may receive a speech from the speech input unit 214 and recognize a speaker
among the users of the information processing device 200 from the received speech.
The speech from the speech input unit 214 is an example of a predetermined signal
25 related to the expression of users of the information processing device according to
the disclosure. The speech recognition unit 216 may then transmit a result of
recognition of a speaker among the users of the information processing device 200 to
the display image generation unit 206 or the communication unit 212.
[0059]
30 Further, the speech recognition unit 216 may recognize contents of a speech by a
speaker among the users of the other information processing device 500, 700 from
20/54
the speech from the other information processing device 500, 700. For example,
the speech recognition unit 216 analyzes the received speech and thereby extracts
and recognizes the contents of the speech by the user as character data. Then, the
speech recognition unit 216 may transmit a result of recognition of the contents of
5 the speech by the speaker among the users of the other information processing device
500, 700 to the display image generation unit 206. Further, the display image
generation unit 206 may recognize contents of a speech by a speaker among the users
of the information processing device 200 from the speech from the speech input unit
214. Then, the speech recognition unit 216 may transmit a result of recognition of
10 the contents of the speech by the speaker among the users of the information
processing device 200 to the display image generation unit 206 or the
communication unit 212.
[0060]
The operation signal input unit 218, which is an example of an input unit in the
15 disclosure, receives an operation signal by the user A or B, the user of the
information processing device 200, for example, which is transmitted using a remote
controller (not shown), for example. Then, the operation signal input unit 218
transmits the received (input) operation signal to the communication unit 212.
Further, the operation signal input unit 218 may transmit the received operation
20 signal to the action recognition unit 220.
[0061]
The action recognition unit 220, which is an example of a recognition unit in the
disclosure, receives an operation signal from the other information processing device
500, 700 which is received by the communication unit 212, and recognizes actions of
25 the users of the other information processing device 500, 700 from the received
operation signal. The act of taking action is an example of the expression m the
disclosure. The operation signal from the other information processing device 500,
700 is an example of a predetermined signal related to the expression of users of the
communication target device according to the disclosure. For example, the action
30 recognition unit 220 recognizes a user as an operator by extracting information about
equipment of an operator user (hereinafter referred to as "user equipment
21/54
information") that is contained in the operation signal from the received operation
signal, and comparmg the extracted user equipment information with user equipment
information 226 for each user which is previously generated and stored in the storage
unit 222. Further, the action recognition unit 220 recognizes a specific operation of
5 the operator by extracting information about a specific operation which is contained
in the operation signal from the received operation signal, for example. For
example, the action recognition unit 220 recognizes a content which the operator user
intends to display on the display screen by comparing information about a content
which the operator user intends to display on the display screen (hereinafter referred
10 to as "user content information") that is contained in the information about a specific
operation with user content information 225 that is previously stored in the storage
unit 222. Then, the action recognition unit 220 transmits a result of recognition of
actions of the users of the other information processing device 500, 700 to the
display image generation unit 206. Further, the action recognition unit 220 may
15 receive an operation signal from the operation signal input unit 218 and recognize
actions of the users of the information processing device 200 from the received
operation signal. The operation signal from the operation signal input unit 218 is an
example of a predetermined signal related to the expression of users of the
information processing device according to the disclosure. Then, the action
20 recognition unit 220 may transmit a result of recognition of actions of the users of the
information processing device 200 to the display image generation unit 206 or the
communication unit 212.
[0062]
The storage unit 222 stores user face information 223 that is information about the
25 faces of the users of the other information processing device 500, 700 or the
information processing device 200, user speech information 224 that is information
about the speeches of the users of the other information processing device 500, 700
or the information processing device 200, user content information 225 that is
information about the contents of the users of the other information processing
30 device 500, 700 or the information processing device 200, and user equipment
information 226 that is information about the equipments used by the users of the
22/54
other information processing device 500, 700 or the information processing device
200.
[0063]
It should be noted that the predetermined signal related to the expression of users of
5 the communication target device according to the disclosure is not limited to the
captured image, the speech and the operation signal from the other information
processing device 500, 700 described above, and it may be any signal from the other
information processing device 500, 700 which is generated as a result that a user of
the other information processing device 500, 700 performs some acts. Further, the
10 predetermined signal related to the expression of users of the information processing
device according to the disclosure is not limited to the captured image, the speech
and the operation signal which are input to the information processing device 200
described above, and it may be any signal which is generated as a result that a user of
the information processing device 200 performs some acts.
15 [0064]
(3. Hardware Configuration of hiformation Processing Device)
A hardware configuration of the information processing device 200 in Fig. 2 is
described hereinbelow. Fig. 3 is an explanatory view illusfrating an example of a
hardware configuration of the information processing device 200 in Fig. 2.
20 [0065]
In Fig. 3, the information processing device 200 includes an MPU 230, a ROM 232,
a RAM 234, a recording medium 236, an input/output interface 238, an operation
input device 240, a display device 242, and a communication interface 244, for
example. Further, in the information processing device 200, the components are
25 connected through a bus 246 as a data transmission line, for example.
[0066]
The MPU 230 is configured as an MPU (Micro Processing Unit), an integrated
ckcuit in which a plurality of circuits for implementing various ftinctions such as
image processing are integrated or the like, and fiinctions as a control unit (not
30 shown) that controls the information processing device 200 as a whole. Further, the
MPU 230 acts as the face recognition unit 204, the display image generation unit
23/54
206, the display control unit 210, the speech recognition unit 216 and the action
recognition unit 220 m the information processing device 200.
[0067]
The ROM 232 stores programs to be used by the MPU 230, control data such as
5 operation parameters and the like. The RAM 234 temporarily stores a program
executed by the MPU 230, for example.
[0068]
The recording medium 236 functions as the storage unit 222, and stores various
kinds of data (or database) such as the user face information, the user speech
10 information, the user content information and the user equipment information,
applications, and the like. The recording medium 236 may be a magnetic recording
medium such as hard disk, nonvolatile memory such as EEPROM (Electrically
Erasable and Programmable Read Only Memory), flash memory, MRAM
(Magnetoresistive Random Access Memory), FeRAM (Ferroelectric Random Access
15 Memory), PRAM Phase change Random Access Memory), and the like. Further,
the information processing device 200 may include a recording medium 236 which is
removable from the information processing device 200.
[0069]
The input/output interface 238 connects to the operation input device 240 and the
20 display device 242, for example. Further, the input/output interface 238 may act as
the image input unit 202, the output unit 208, the speech input unit 214 and the
operation signal input unit 218. The operation input device 240 flinctions as an
operation unit (not shown), and the display device 242 functions as a display unit 254
in Fig. 15, which is described later. The input/output interface 238 may be a USB
25 (Universal Serial Bus) terminal, a DVI (Digital Visual Interface) terminal, an HDMI
(High-Defmition Multimedia Interface) terminal, or various processing circuits, for
example. Further, the operation mput device 240 is mounted on the information
processing device 200 and connected with the input/output interface 238 inside the
information processing device 200, for example. The operation input device 240
30 may be a button, a cursor key, a rotating selector such as Jog Dial, or a combination
of those, for example. Furthermore, the display device 242 is mounted on the
24/54
information processing device 200 and connected with the mput/output interface 238
inside the information processing device 200, for example. The display device 242
may be a liquid crystal display (LCD), or an organic EL display (organic
ElectroLuminescence display. Also called an OLED display (Organic Light
5 Emitting Diode display)) and the like, for example. Note that the input/output
interface 238 can also connect to external devices of the information processing
device 200, such as an operation input device (e.g. a keyboard, a mouse), a display
device (e.g. an external display device such as the display device 102) and an
imaging device (e.g. the imaging device 104). Further, the display device 242 may
10 be a device capable of display and user operation, such as a touch screen, for
example.
[0070]
The communication interface 244 is a communication means included in the
information processing device 200, and functions as the communication unit 212 for
15 performing wireless/wired communication with external devices such as the
communication server 300 and the other information processing device 500, 700, for
example, through the communication network 800 (or directly). The
communication interface 244 may be a communication antenna and an RF circuit
(wireless communication), IEEE802.15.1 port and a transmitting and receiving
20 circuit (wireless communication), IEEE802.1 lb port and a transmitting and receiving
circuit (wireless communication), or a LAN terminal and a transmitting and
receiving circuit (wired communication) and the like, for example.
[0071]
It should be noted that the hardware configuration of the information processing
25 device 200 according to the embodiment is not limited to the configuration shown in
Fig. 3. For example, the information processing device 200 may include a speech
output device composed of a DSP (Digital Signal Processor), an amplifier, a speaker
and the like, which acts as a speech output unit (not shown).
[0072]
30 Further, the information processing device 200 may include an imaging device
composed of a lens/image pickup device and a signal processing circuit, which acts
25/54
as an imaging unit 252 in Fig. 15, for example. In this case, the information
processing device 200 can process the captured image generated in its own device.
The lens/image pickup device is made up of an optical lens and an image sensor
composed of a plurality of image pickup devices such as CCD (Charge Coupled
5 Device) or CMOS (Complementary Metal Oxide Semiconductor), for example.
The signal processing circuit includes an AGC (Automatic Gain Control) circuit and
an ADC (Analog to Digital Converter), for example, and converts an analog signal
generated by the image pickup device into a digital signal (image data) and performs
various signal processing. The signal processing that is performed by the signal
10 processing circuit may be White Balance correction, interpolation, color tone
correction, gamma correction, YCbCr conversion, edge enhancement, coding and the
like, for example.
[0073]
Further, the information processing device 200 may have a configuration in which
15 the operation input device 240 and the display device 242 shown in Fig. 3 are not
included, for example.
[0074]
(4. Display Process)
A display process that is executed by the information processing device 200 in Fig. 2
20 is described hereinbelow. Fig. 4 is a flowchart of the display process executed by
the information processing device 200 in Fig. 2.
[0075]
Referring to Fig. 4, when an instruction for network connection with the users C and
D and the users E and F is given from the user A, for example, in the information
25 processing device 200, the communication unit 212 becomes able to communicate
with the communication server 300 through the communication network 800 (Step
SI00). Note that, in Step SI00, the communication unit 212 may become able to
communicate with the other information processing device 500, 700 directly through
the communication network 800.
30 [0076]
Next, the communication unit 212 transmits a captured unage which is generated by
26/54
imaging in the imaging device 104 in the display system 100 and received via the
image input unit 202 to the communication server 300 through the communication
network 800 (Step S102). Note that, in Step S102, the communication unit 212
may transmit a result of recognition of the users of the information processing device
5 200 and a result of recognition of facial expressions of the users of the information
processing device 200 which are received from the face recognition unit 204 to the
communication server 300 through the communication network 800. Further, in
Step SI02, the communication unit 212 may transmit the captured image, a result of
recognition of the users, and a result of recognition of facial expressions of the users
10 to the other information processing device 500, 700 directly through the
communication network 800.
[0077]
Then, the communication unit 212 transmits a speech which has been input to the
information processing device 200 and received via the speech input unit 214 to the
15 communication server 300 through the communication network 800 (Step SI04).
Note that, in Step SI04, the communication unit 212 may transmit a result of
recognition of a speaker among the users of the information processing device 200
and a result of recognition of contents of a speech by a speaker among the users of
the information processing device 200 which are received from the speech
20 recognition unit 216 to the communication server 300 through the communication
network 800. Further, in Step SI04, the communication unit 212 may transmit the
speech, a result of recognition of a speaker among the users, a result of recognition of
contents of a speech by a speaker among the users to the other information
processing device 500, 700 directly through the communication network 800.
25 [0078]
Then, the communication unit 212 transmits an operation signal which has been
input to the information processing device 200 and received via the operation signal
input unit 218 to the communication server 300 through the communication network
800 (Step SI06). Note that, in Step SI06, the communication unit 212 may
30 transmit a result of recognition of actions of the users of the information processing
device 200 which is received from the action recognition unit 220 to the
27/54
communication server 300 through the communication network 800. Further, in
Step SI06, the communication unit 212 may transmit the operation signal and a
result of recognition of actions of the users to the other mformation processing
device 500, 700 directly through the communication network 800.
5 [0079]
After that, the communication unit 212 receives a captured image which has been
captured in the display system 400 and transmitted from the other information
processing device 500 from the communication server 300. Further, the
communication unit 212 receives a captured image which has been captured in the
10 display system 600 and fransmitted from the other information processmg device 700
from the communication server 300 (Step S108). Note that, in Step S108, the
communication unit 212 may receive a result of recognition of users and a result of
recognition of facial expressions of users from the captured image which has been
captured in the display system 400 and transmitted from the other information
15 processing device 500, and a result of recognition of users and a resuh of recognition
of facial expressions of users from the captured image which has been captured in
the display system 600 and transmitted from the other information processing device
700 from the communication server 300. Further, in Step SI08, the communication
unit 212 may receive the captured image, a result of recognition of users, and a result
20 of recognition of facial expressions of the users from the other information
processing device 500, 700 directly through the communication network 800.
[0080]
Then, the communication unit 212 receives a speech which has been input to the
other information processing device 500 and transmitted from the other information
25 processing device 500 from the communication server 300. Further, the
communication unit 212 receives a speech which has been input to the other
information processing device 700 and fransmitted from the other information
processing device 700 from the communication server 300 (Step SI 10). Note that,
in Step SI 10, the communication unit 212 may receive a result of recognition of a
30 speaker among users and a result of recognition of contents of a speech by a speaker
among users from the speech which has been input to the other information
28/54
processing device 500 and transmitted from the other information processing device
500, and a result of recognition of a speaker among users and a result of recognition
of contents of a speech by a speaker among users from the speech v^hich has been
input to the other information processing device 700 and transmitted from the other
5 information processing device 700 from the communication server 300. Further, in
Step SllO, the communication unit 212 may receive the speech, a result of
recognition of a speaker among the users, a result of recognition of contents of a
speech by a speaker among the users from the other information processing device
500, 700 directly through the communication network 800.
10 [0081]
Then, the communication unit 212 receives an operation signal which has been input
to the other information processing device 500 and transmitted from the other
information processing device 500 from the communication server 300. Further,
the communication unit 212 receives an operation signal which has been input to the
15 other information processing device 700 and transmitted from the other information
processing device 700 from the communication server 300 (Step SI 12). Note that,
in Step S112, the communication unit 212 may receive a result of recognition of
actions of users from the operation signal which has been input to the other
information processing device 500 and fransmitted from the other information
20 processing device 500, and a result of recognition of actions of users from the
operation signal which has been input to the other information processing device 700
and transmitted from the other information processing device 700 from the
communication server 300. Further, in Step SI 12, the commimication unit 212 may
receive the operation signal and a result of recognition of actions of the users from
25 the other information processing device 500, 700 directly through the communication
network 800.
[0082]
After that, the information processing device 200 performs user display processing in
Fig. 5, which is described later (Step S114) so that user images representing the users
30 of the information processing device 200 and the other information processing device
500, 700 are displayed for each user, and a user image representing a user who has
29/54
performed an expression and a detail of the expression are displayed in association
with each other on the display screen of the display device 102, for example.
[0083]
Fig. 5 is a flowchart of the user display processing that is performed in Step S114 in
5 the display process of Fig. 4.
[0084]
Referring to Fig. 5, the face recognition unit 204 first receives a captured image from
the other information processing device 500, 700 which is received by the
communication unit 212, detects faces of the users of the information processing
10 device 500, 700 from the received captured image, and recognizes users of the
information processing device 500, 700, i.e., the users C and D or the users E and F,
based on the detection result of faces (Step S200). Further, in Step S200, the face
recognition unit 204 may detect faces of the users of the information processing
device 500, 700 from the received captured image, and recognize facial expression of
15 the users of the information processing device 500, 700, i.e., the users C and D or the
users E and F, based on the detection result of faces. Furthermore, in Step S200,
the face recognition unit 204 may receive a captured image from the image input unit
202, detect faces of the users of the mformation processing device 200 from the
received captured image, and recognize users of the information processing device
20 200, i.e., the users A and B or recognize facial expression of the users of the
information processing device 200, i.e., the users A and B, based on the detection
result of faces.
[0085]
Next, the speech recognition unit 216 receives a speech from the other information
25 processing device 500, 700 which is received by the communication unit 212, and
recognizes a speaker among the users of the information processing device 500, 700
from the received speech (Step S202). Further, in Step S202, the speech
recognition unit 216 may receive a speech from the other information processing
device 500, 700 which is received by the communication unit 212, and recognizes
30 contents of a speech by a speaker among the users of the information processing
device 500, 700 from the received speech. Further, in Step S202, the speech
30/54
recognition unit 216 may receive a speech from the speech input unit 214 and
recognize a speaker among the users of the information processing device 200 or
recognize contents of a speech by a speaker among the users from the received
speech.
5 [0086]
Then, the action recognition unit 220 receives an operation signal from the other
information processing device 500, 700 which is received by the communication unit
212, and recognizes actions of the users of the information processing device 500,
700 from the received operation signal (Step S204). Further, in Step S204, the
10 action recognition unit 220 may receive an operation signal from the operation signal
input unit 218 and recognize actions of the users of the information processing
device 200 from the received operation signal.
[0087]
After that, the display image generation unit 206 receives a result of recognition of
15 the users of the other information processing device 500, 700 from the face
recognition unit 204, and receives the captured image captured m the other display
system 400 or the captured image captured in the other display system 600 which is
received by the communication unit 212. Then, based on the received result of
recognition of the users of the other information processing device 500, 700, the
20 display image generation unit 206 generates a display image in which user images
representing the users of the other information processing device 500, 700 are
displayed for each user (Step S206).
[0088]
Further, in Step S206, when the display image generation unit 206 receives a
25 captured image captured in the display system 100 from the image input unit 202,
receives a result of recognition of the users of the information processing device 200
and a result of recognition of the users of the other information processing device
500, 700 from the face recognition unit 204, and receives a captured image captured
in the other display system 400 or a captured image captured in the other display
30 system 600 which is received by the communication unit 212, the display image
generation unit 206 generates a display image in which user images representing the
31/54
users of the information processing device 200 and the other information processing
device 500, 700 are displayed for each user based on the received resuhs of
recognition of the users of the information processing device 200 and the users of the
other information processing device 500, 700.
5 [0089]
Further, in Step S206, when the communication unit 212 receives a result of
recognition of users from a captured image captured in the other display system 400
or a result of recognition of users from a captured image captured in the other display
system 600, the display image generation unit 206 may receive the result of
10 recognition and generate a display image based on the result of recognition received
from the communication unit 212 in stead of a result of recognition by the face
recognition unit 204.
[0090]
Further, in Step S206, when the display image generation unit 206 ftirther receives a
15 resuh of recognition of facial expressions of the users of the other information
processing device 500, 700 from the face recognition unit 204, the display image
generation unit 206 generates a display image in which a user image representing a
user whose facial expression has changed and the facial expression are displayed in
association with each other in the above-described display image based further on the
20 received result of recognition of facial expressions of the users of the other
information processing device 500, 700.
[0091]
Further, in Step S206, when the display image generation unit 206 ftirther receives a
result of recognition of facial expressions of the users of the information processing
25 device 200 from the face recognition unit 204, the display image generation unit 206
generates a display image in which a user image representing a user whose facial
expression has changed and the facial expression are displayed m association with
each other based further on the received result of recognition of facial expressions of
the users of the information processing device 200.
30 [0092]
Further, in Step S206, when the communication unit 212 receives a result of
32/54
recognition of facial expressions of users from a captured image captured in the other
display system 400 or a result of recognition of facial expressions of users from a
captured image captured in the other display system 600, the display image
generation unit 206 may receive the result of recognition and generate a display
5 image based on the result of recognition received from the communication unit 212
in stead of a result of recognition by the face recognition unit 204.
[0093]
Further, in Step S206, when the display image generation unit 206 receives a result
of recognition of a speaker among the users of the other information processing
10 device 500, 700 from the speech recognition unit 216, the display image generation
unit 206 generates a display image in which a user image representing the speaker
and the presence of a speech are displayed in association with each other in the
above-described display image based further on the received result of recognition of
the speaker among the users of the other information processing device 500, 700.
15 [0094]
Further, in Step S206, when the display image generation unit 206 flirther receives a
result of recognition of a speaker among the users of the information processing
device 200 from the speech recognition unit 216, the display image generation unit
206 generates a display image in which a user image representing the speaker and the
20 presence of a speech are displayed in association with each other based ftirther on the
received result of recognition of the speaker among the users of the information
processing device 200.
[0095]
Further, in Step S206, when the communication unit 212 receives a result of
25 recognition of a speaker from a speech input to the other information processing
device 500 or a result of recognition of a speaker from a speech input to the other
information processing device 700, the display image generation unit 206 may
receive the result of recognition and generate a display image based on the result of
recognition received from the communication unit 212 in stead of a result of
3 0 recognition by the speech recognition unit 216.
[0096]
33/54
Further, in Step S206, when the display image generation unit 206 further receives a
result of recognition of contents of a speech by a speaker among the users of the
other information processing device 500, 700 from the speech recognition unit 216,
the display image generation unit 206 generates a display image in which a user
5 image representing the speaker, the presence of a speech, and the contents of the
speech are displayed m association with one another based further on the received
result of recognition of the contents of the speech by the speaker among the users of
the other information processing device 500,700.
[0097]
10 Further, in Step S206, when the display image generation unit 206 further receives a
result of recognition of contents of a speech by a speaker among the users of the
information processing device 200 from the speech recognition unit 216, the display
image generation unit 206 generates a display image in which a user image
representing the speaker, the presence of a speech, and the contents of the speech are
15 displayed in association with one another based fiarther on the received result of
recognition of the contents of the speech by the speaker among the users of the
information processing device 200.
[0098]
Further, in Step S206, when the communication unit 212 receives a result of
20 recognition of contents of a speech by a speaker from a speech input to the other
information processing device 500 or a result of recognition of contents of a speech
by a speaker from a speech input to the other information processing device 700, the
display image generation unit 206 may receive the result of recognition and generate
a display image based on the result of recognition received from the communication
25 unit 212 in stead of a result of recognition by the speech recognition unit 216.
[0099]
Further, in Step S206, when the display image generation unit 206 receives a result
of recognition of actions of the users of the other information processing device 500,
700 from the action recognition unit 220, the display image generation unit 206
30 generates a display image in which a user image representing an operator user and a
specific operation are displayed in association with each other based further on the
34/54
received result of recognition of actions of the users of the other information
processing device 500, 700.
[0100]
Further, in Step S206, when the display image generation unit 206 receives a result
5 of recognition of actions of the users of the information processing device 200 from
the action recognition unit 220, the display image generation unit 206 generates a
display image in which a user image representing an operator user and a specific
operation are displayed in association with each other based further on the received
result of recognition of actions of the users of the information processing device 200.
10 [0101]
Further, in Step S206, when the communication unit 212 receives a result of
recognition of actions of users from an operation signal input to the other information
processing device 500 or a result of recognition of actions of users from an operation
signal input to the other information processing device 700, the display image
15 generation unit 206 may receive the result of recognition and generate a display
image based on the result of recognition received from the communication unit 212
in stead of a result of recognition by the action recognition unit 220.
[0102]
After that, the display control unit 210 receives a display image from the display
20 image generation unit 206, and transmits a signal for displaying the received display
image on the display screen of the display device 102 to the output unit 208. Then,
the output unit 208 fransmits the signal for displaying the received display image to
the display device 102. As a result, a display image in which user images
representing the users of the other information processing device 500, 700 are
25 displayed for each user, a display image in which user images representing the users
of the information processing device 200 and the other information processing device
500, 700 are displayed for each user, a display image in which user images
representing the users of the other information processing device 500, 700 are
displayed for each user and a user image representing a user who has performed an
30 expression and a detail of the expression are displayed in association with each other,
and a display image in which user images representing the users of the information
35/54
processing device 200 and the other information processing device 500, 700 are
displayed for each user and a user image representing a user who has performed an
expression and a detail of the expression are displayed in association with each other
are displayed on the display screen of the display device 102 (Step S208). The
5 processing thereby ends.
[0103]
According to the display process of Fig. 4, a display image in which user images
representing the users of the other information processing device 500, 700 are
displayed for each user is displayed on the display screen of the display device 102.
10 Thus, even when there are a plurality of users of the other information processing
device 500, for example, a user of the information processing device 200 can be
surely aware of the existence of any user of the other information processing device
500 because the user images representing the respective users of the other
information processing device 500 are displayed for each of the users on the display
15 screen of the display device 102.
[0104]
Further, according to the display process of Fig. 4, a display image in which user
images representing the users of the information processing device 200 and the other
information processing device 500, 700 are displayed for each user is displayed on
20 the display screen of the display device 102. Thus, a user of the information
processing device 200 can be surely aware of the existence of any user of the other
information processing device 500 and can feel a sense of realism, a sense of
togetherness, a sense of closeness and the like as if commimicating with them in the
same place.
25 [0105]
Further, according to the display process of Fig. 4, a display image in which user
images representing the users of the other information processing device 500, 700 are
displayed for each user, and a user image representing a user who has performed an
expression and a detail of the expression are displayed in association with each other
30 is displayed on the display screen of the display device 102. Thus, a user of the
information processing device 200 can be surely aware of the existence of any user
36/54
of the other infonnation processing device 500 and, even when there are a plurality
of users of the other information processing device 500, for example, can be surely
aware of which user of the other information processing device 500 has performed an
expression because a user image representing a user who has performed an
5 expression and a detail of the expression are displayed in association with each other
on the display screen of the display device 102.
[0106]
Further, according to the display process of Fig. 4, a display image m which user
images representing the users of the infonnation processing device 200 and the other
10 information processing device 500, 700 are displayed for each user, and a user image
representing a user who has performed an expression and a detail of the expression
are displayed in association with each other is displayed on the display screen of the
display device 102. Thus, a user of the information processing device 200 can be
surely aware of the existence of any user of the other information processing device
15 500, can feel a sense of realism, a sense of togetherness, a sense of closeness and the
like as if communicating with them in the same place, and fiirther can be surely
aware of which user of the other information processing device 500 has performed an
expression.
[0107]
20 (5. Examples)
Examples of the disclosure are described hereinbelow. Figs. 6A to 6D and Figs. 7 A
to 7D are explanatory views illustrating an example 1 of the disclosure. In the
example 1, a case where an instruction for network coimection with the users C and
D is given by the user A, for example, in the infonnation processing device 200 of
25 Fig. 2, and a display image in which user images representing the users of the
information processing device 200 and the other information processing device 500,
700 are displayed for each user, and a user image representing a user who has
performed an expression and a detail of the expression are displayed in association
with each other is displayed on the display screen of the display device 102 in Fig. 2
30 and the display screen in the display system 400 is described by way of illustration.
[0108]
37/54
First, the inforaiation processing device 200 receives a captured image shown in Fig.
6A and a captured image shown in Fig. 6B, and recognizes the users A and B of the
information processing device 200 and the users C and D of the other information
processing device 500 by the above-described face recognition. Then, as shown in
5 Fig. 6C, the information processing device 200 displays a display image in which a
user image 110 representing the user A of the information processing device 200, a
user image 112 representing the user B of the information processing device 200, a
user image 114 representing the user C of the other information processing device
500, and a user image 116 representing the user D of the other information
10 processing device 500 are displayed for each user on the display screen of the display
device 102. Likewise, as shown in Fig. 6D, the other information processing device
500 displays a display image in which a user image 410 representing the user A of
the information processing device 200, a user image 412 representing the user B of
the information processing device 200, a user image 414 representing the user C of
15 the other information processing device 500, and a user unage 416 representing the
user D of the other information processing device 500 are displayed for each user on
the display screen of a display device 402 in the display system 400.
[0109]
Next, as shown in Fig. 7A, when a speech saying "I want to see Mt. Fuji", for
20 example, is made as an expression by the user D, the other information processing
device 500 receives the speech and recognizes the user D as a speaker among the
users of the other information processing device 500 and the contents of the speech
by the user D by the above-described speech recognition. Then, as shown in Fig.
7B, the other information processing device 500 displays a display image in which a
25 character image representing the contents of the speech, which is the image "I want
to see Mt. Fuji", is displayed within a dialogue balloon 418 from the user unage 416
representing the user D as a speaker. Likewise, the information processing device
200 displays a display image in which a character image representing the contents of
the speech, which is the image "I want to see Mt. Fuji", is displayed within a
30 dialogue balloon (not shown) from the user image 116 representing the user D as a
speaker on the display screen of the display device 102, although not shown.
38/54
[0110]
Further, as shown in Fig. 7A, when transmission of the character data "I want to see
Mt. Fuji" is performed using a remote controller, for example, as an expression by
the user D, the other information processing device 500 receives the operation signal
5 and recognizes the user D as an operator among the users of the other information
processing device 500 and a specific operation performed by the operator by the
above-described action recognition. Then, as shown in Fig. 7B, the other
information processing device 500 displays a display image in which a character
image representing the specific operation by the operator, which is the image "I want
10 to see Mt. Fuji", is displayed within a dialogue balloon 418 from the user image 416
representing the user D as a speaker on the display screen of the display device 402.
Likewise, the information processing device 200 displays a display image in which a
character image representing the specific operation by the operator, which is the
image "I want to see Mt. Fuji", is displayed within a dialogue balloon (not shown)
15 from the user image 116 representing the operator user D on the display screen of the
display device 102, although not shown.
[0111]
Then, as shown in Fig. 7C, when transmission of a request for display of "the content
of Mt. Fuji" is performed using a remote controller, for example, as an expression by
20 the user B, the information processing device 200 receives the operation signal and
recognizes the user B as an operator among the users of the information processing
device 200 and a specific operation performed by the operator by the abovedescribed
action recognition. Then, as shown in Fig. 7D, the information
processing device 200 displays a display image in which a content image which the
25 operator user B intends to display, which is the image of "the content of Mt. Fuji", is
displayed within a dialogue balloon 118 from the user image 112 representing the
operator user B on the display screen of the display device 102. Likewise, the other
information processing device 500 displays a display image in which a content image
which the operator user B intends to display, which is the image of "the content of
30 Mt. Fuji", is displayed within a dialogue balloon (not shown) from the user image
412 representing the operator user B on the display screen of the display device 402,
39/54
although not shown. At this time, the other information processing device 500 may
receive a content image which the operator user B intends to display, which is the
image of "the content of Mt. Fuji", from the information processing device 200 or
receive the content image from another device through the communication network
5 800, for example.
[0112]
According to the example 1, a user of the information processing device 200 and the
other information processing device 500 can be surely aware of the existence of any
user of the information processing device 200 and the other information processing
10 device 500, can feel a sense of realism, a sense of togetherness, a sense of closeness
and the like as if communicating with them in the same place, and further can be
surely aware of which user of the information processing device 200 and the other
information processing device 500 has performed an expression.
[0113]
15 Figs. 8(A) to 8(D) are explanatory views illustrating an example 2 of the disclosure.
In the example 2, a case where an instruction for network connection with the users
C and D is given by the user A, for example, in the information processing device
200 of Fig. 2, and the user A goes away during network connection with the users C
and D is described by way of illustration.
20 [0114]
When the user A has gone away during network connection with the users C and D
as shown in Figs. 8(A) and 8(B), a display image in which a user image 120
representing the user A of the information processing device 200 is deleted at the
same time when the user A goes away is displayed on the display screen of the
25 display device 102. Likewise, a display image in which a user image (not shown)
representing the user A of the information processing device 200 is deleted at the
same time when the user A goes away is displayed on the display screen of the
display device 402 in the display system 400, although not shown.
[0115]
30 According to the example 2, because the user image 120 representing the user A of
the information processing device 200 is deleted at the same time when the user A
40/54
goes away in the display image which is displayed on the display screen of the
display device 402, a user of the other information processing device 500 can be
surely aware that the user A of the information processing device 200 has gone away.
[0116]
5 Figs. 9(A) to 9(D) are explanatory views illustrating an example 3 of the disclosure.
In the example 3, a case where an instruction for network connection with the users
C and D is given by the user A, for example, in the information processing device
200 of Fig. 2, and the facial expression of the user A has changed to smiling during
network connection with the users C and D is described by way of illustration.
10 [0117]
When the facial expression of the user A has changed to smiling during network
connection with the users C and D as shown in Figs. 9(A) to 9(D), a display image in
which a user image 130 representing the user A whose facial expression has changed
to smiling is displayed like jumping is displayed on the display screen of the display
15 device 102. Likewise, a display image in which a user image (not shown)
representing the user A whose facial expression has changed to smiling is displayed
like jumping is displayed on the display screen of the display device 402 in the
display system 400, although not shown.
[0118]
20 According to the example 3, because the user image 130 representing the user A
whose facial expression has changed to smiling is displayed like jumping in the
display image which is displayed on the display screen of the display device 402, a
user of the other information processing device 500 can be surely aware that the
facial expression of the user A of the information processing device 200 has changed
25 to smiling.
[0119]
Figs. 10(A) and 10(B) are explanatory views illustrating an example 4 of the
disclosure. In the example 4, a case where an instruction for network connection
with the users C and D is given by the user A, for example, in the information
30 processing device 200 of Fig. 2, and a display image in which user images
representing the users of the information processing device 200 and the other
41/54
information processing device 500, 700 are displayed for each user, and a user image
representing a user who has performed an expression and a detail of the expression
are displayed in association with each other is displayed on the display screen of the
display device 102 in Fig. 2 and the display screen in the display system 400 is
5 described by way of illustration.
[0120]
As shown m Figs. 10(A) and 10(B), the information processing device 200
recognizes the users A and B of the information processing device 200 and the users
C and D of the other information processing device 500 by the above-described face
10 recognition and further recognizes the gender of the users A and B of the information
processing device 200 and the users C and D of the other information processing
device 500, and then displays a display image in which a user image 140
representing the user A of the information processing device 200, a user image 144
representing the user B of the information processing device 200, a user image 142
15 representing the user C of the other information processing device 500, and a user
image 146 representing the user D of the other information processing device 500 are
displayed for each user and for each gender on the display screen of the display
device 102. Likewise, though not shown, the other information processing device
500 recognizes the users A and B of the information processing device 200 and the
20 users C and D of the other information processing device 500 by the above-described
face recognition and further recognizes the gender of the users A and B of the
information processing device 200 and the users C and D of the other information
processing device 500, and then displays a display image in which a user image (not
shown) representing the user A of the information processing device 200, a user
25 image (not shown) representing the user B of the information processing device 200,
a user image (not shown) representing the user C of the other information processing
device 500, and a user image (not shown) representing the user D of the other
information processing device 500 are displayed for each user and for each gender on
the display screen of the display device 402.
30 [0121]
According to the example 4, a user of the information processing device 200 and the
42/54
other information processing device 500 can be surely aware of the gender of any
user of the information processing device 200 and the other information processing
device 500.
[0122]
5 Figs. 11(A) to 11(E) are explanatory views illustrating an example 5 of the
disclosure. In the example 5, a case where an instruction for network connection
with the users A and B is given by the user C, for example, in the other mformation
processing device 500 of Fig. 2, and a content image 430 is newly displayed on the
display screen in the display system 400 in Fig. 2 when a display image 420 which is
10 generated by a display image generation unit (not shown) of the other information
processing device 500 is being displayed on the display screen in the display system
400 is described by way of illustration.
[0123]
As shown in Figs. 11(A) to 11(E), in the case where the content image 430 is
15 displayed on the display screen in the display system 400 when the display image
420 is being displayed thereon, the display image 420 is gradually shifted to the right
to fade out of the screen, and the content image 430 is shifted from the left to the
right to appear in the screen, thereby displaying the content image 430. Then, as
shown in Fig. HE, a display 440 indicating that it is under network connection is
20 displayed on the display screen in the display system 400.
[0124]
According to the example 5, a user of the other information processing device 500
can be surely aware that it is under network connection with the users A and B.
[0125]
25 Figs. 12(A) and 12(B) are explanatory views illustrating an example 6 of the
disclosure. In the example 6, a case where an instruction for network connection
with the user C is given by the user A, for example, in the information processing
device 200 of Fig. 2, and a content image and a display image generated by the
display image generation unit 206 are displayed in superposition on the display
30 screen of the display device 102 is described by way of illustration.
[0126]
43/54
As shown in Figs. 12(A) and 12(B), in the case where a content image and a display
image generated by the display image generation unit 206 are displayed in
superposition on the display screen of the display device 102, user images 150 and
152 are gradually moved offscreen. Then, as shown in Fig. 12(B), when a user
5 represented by the user image 150, 152 performs some act of expression such as
speech, wave-like expression, for example, is made from the position corresponding
to where the user image 150, 152 has been moved.
[0127]
According to the example 6, a user of the information processing device 200 can
10 view the entire content image and can be surely aware that it is under network
connection with the user C.
[0128]
Figs. 13A to 13L are explanatory views illustrating an example 7 of the disclosure.
In the example 7, a case where an instruction for network connection with the users
15 A and B and the users C and D is given by the user E, for example, in the
information processing device 700 of Fig. 2, and a display image in which user
images representing the users of the information processing device 200 and the other
information processing device 500, 700 are displayed for each user, and a user image
representing a user who has performed an expression and a detail of the expression
20 are displayed in association with each other is displayed on the display screen in the
display system 600 of Fig. 2 is described by way of illustration.
[0129]
As shown in Figs. 13(A) to 13(L), on the display screen in the display system 600, a
common content area Al as an area where the users of the information processing
25 device 200 and the other information processing device 500, 700 share contents, and
a user-specific content area A2 as an area which represents contents of the users E
and F of the other information processing device 700 are set. Then, when a content
image 602 is selected as a content to be displayed in the common content area Al by
the user E, for example, a content image 604 corresponding to the content image 602
30 is displayed and gradually scaled up in a lower part as a position corresponding to the
user E in the common content area Al (Figs. 13(B) to 13(D)). Further, when a
44/54
content image (not shown) is selected as a content to be displayed in the common
content area Al by the user A, for example, a user image representing the user A is
scaled up, and a content image 606 corresponding to the content image (not shown)
which is selected by the user A is displayed and gradually scaled up in a lower right
5 part as a position corresponding to the user A in the common content area Al (Figs.
13(F) to 13(H)). Further, when a content image (not shown) is selected as a content
to be displayed in the common content area Al by the user C, for example, a user
image representing the user C is scaled up, and a content image 608 corresponding to
the content image (not shown) which is selected by the user C is displayed and
10 gradually scaled up in an upper left part as a position corresponding to the user C in
the common content area Al (Figs. 13(J) to 13(L)). At this time, the content image
606 selected by the user A may fade out as shown in Fig. 13(K).
[0130]
According to the example 7, a user of the other information processing device 700
15 can be surely aware of the existence of any user of the information processing device
200 and the other information processing device 500, can feel a sense of realism, a
sense of togetherness, a sense of closeness and the like as if communicating with
them in the same place, and fiirther can be surely aware of which user of the
information processing device 200 and the other information processing device 500
20 has performed an expression.
[0131]
(6. Functional Configuration of Information Processing Device (Second
Embodiment))
A flinctional configuration of an information processing device according to a second
25 embodiment of the disclosure is described. Fig. 14 is an explanatory view
illustrating an example of a functional configuration of an information processing
device according to the embodiment. An information processing device 250 as the
information processing device according to the embodiment is different from the
information processing device according to the first embodiment described above
30 only in that it includes an imaging unit 252 and a display imit 254, and therefore
explanation of redundant elements and functions is omitted, and explanation of
45/54
different elements and functions is provided below.
[0132]
Note that Fig. 14 shows users A and B, which are users of the information processing
device 250, a communication network 800 to which the information processing
5 device 250 can connect, a communication server 300 and other information
processing devices 550 and 750 that can connect to the communication network 800,
users C and D, which are users of the other information processing device 550, and
users E and F, which are users of the other information processing device 750. The
other information processing devices 550 and 750 have the same configuration as the
10 information processing device 250, and detailed explanation thereof is omitted. The
other information processing devices 550 and 750 are examples of a communication
target device in the disclosure.
[0133]
In Fig. 13, the information processing device 250 includes an imaging unit 252, a
15 face recognition unit 204, a display image generation unit 206, a display unit 254, a
display control unit 210, a communication unit 212, a speech input unit 214, a speech
recognition unit 216, an operation signal input unit 218, an action recognition unit
220, and a storage unit 222, for example.
[0134]
20 The imaging unit 252 can capture still images or moving images of the users A and B
looking at the display screen of the display unit 254. Then, the imaging unit 252
can transmit a captured image generated by imaging to the communication unit 212,
the face recognition unit 204 and the display image generation unit 206.
[0135]
25 In the display unit 254, a display image in which user images representing the users
of the information processing device 250 and the other information processing device
550, 750 are displayed for each user, and a user image representing a user who has
performed an expression and a detail of the expression are displayed in association
with each other can be displayed, for example.
30 [0136]
According to the embodiment, the information processing device 250 executes the
46/54
display process of Fig. 4 which is described earlier, thereby having the same effect as
described earlier in the first embodiment. Further, according to the embodiment,
the need for separately preparing the display system 100 shown in Fig. 2 can be
eliminated, thereby allowing easier implementation.
5 [0137]
Further, an object of the disclosure may be accomplished also by providing a storage
medium storing program code of software that implements the functions of each of
the embodiments described above to a system or a device, and reading and executing
the program code stored in the storage medium by a computer (or CPU, MPU etc.) of
10 the system or the device.
[0138]
In this case, the program code which is read from the storage medium implements
the functions of each of the embodiments described above, and the program code and
the storage medium storing the program code constitute the disclosure.
15 [0139]
Further, the storage medium for providing the program code may be floppy
(registered trademark) disk, hard disk, magneto-optical disk, optical disk such as CDROM,
CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW or DVD+RW, magnetic
tape, nonvolatile memory card, ROM or the like, for example. Alternatively, the
20 program code may be downloaded over a network.
[0140]
Further, the disclosure is not limited to the case where a computer executes the read
program code to thereby implement the ftinctions of each of the embodiments
described above, and the case where an OS (Operating System) or the like running
25 on a computer performs a part or the whole of actual processing based on
instructions of the program code to thereby implement the functions of each of the
embodiments described above is also included.
[0141]
Further, the case where the program code read from the storage medium is written
30 into a memory in a function expansion board inserted to a computer or a function
expansion unit connected to a computer, and then a CPU or the like in the expansion
47/54
board or the expansion unit performs a part or the whole of actual processing for the
expanded functions based on instructions of the program code to thereby implement
the functions of each of the embodiments described above is also included.
[0142]
5 It should be understood by those skilled in the art that various modifications,
combinations, sub-combinations and alterations may occur depending on design
requirements and other factors insofar as they are within the scope of the appended
claims or the equivalents thereof.
[0143]
10 For example, the communication server 300 may recognize users or facial expression
of users of the other information processing device 500, 700 from the captured image
from the other information processing device 500, 700. Further, the communication
server 300 may recognize a speaker among users of the other information processing
device 500, 700 and contents of a speech by a speaker among the users from the
15 speech from the other information processmg device 500, 700. Furthermore, the
communication server 300 may recognize actions of the users of the other
information processing device 500, 700 from the operation signal from the other
information processing device 500, 700.
[0144]
20 Further, the information processing device 200 may transmit contents to the other
information processing device 500, 700 to share the contents among users.
[Description of Reference Numerals]
[0145]
100 Display system
25 102 Display device
104 Imaging device
200 Information processing device
202 Image input unit
204 Face recognition unit
30 206 Display image generation unit
208 Output unit
48/54
210 Display control unit
212 Communication unit
214 Speech input unit
216 Speech recognition unit
5 218 Operation signal input unit
220 Action recognition unit
222 Storage unit
300 Communication server
400, 600 Other communication systems
10 500, 700 Other information processing devices
800 Communication network

[Title of Document] CLAIMS
[Claim 1]
An infonnation processing device comprising:
a communication unit that receives a predetermined signal related to an
5 expression of a user of a communication target device from the communication
target device;
a recognition unit that detects a face from a captured image in the
predetermined signal received by the communication unit, and recognizes a user of
the communication target device based on a detection resuh of the face; and
10 a control unit that displays a display image where a user image representing
the user of the communication target device recognized by the recognition unit is
displayed for each user on a display screen,
wherein
the recognition unit further recognizes a user having performed the
15 expression and a detail of the expression from the predetermined signal received by
the communication unit, and
the control unit displays a display image where the user image representing
the user having performed the expression recognized by the recognition unit and the
detail of the expression recognized by the recognition unit are further displayed in
20 association with each other m the display image on the display screen.
[Claim 2]
The information processing device according to Claim 1, wherein
the recognition unit recognizes a user whose facial expression has changed
as the user having performed the expression and a facial expression as the detail of
25 the expression based on the detection result of the face.
[Claim 3]
The information processing device according to Claim 1, wherein
the recognition unit recognizes a speaker as the user having performed the
expression and a presence of a speech as the detail of the expression from a speech in
30 the predetermined signal received by the communication unit.
[Claim 4]
50/54
The information processing device according to Claim 3, wherein
the recognition unit further recognizes contents of a speech by the speaker
as the detail of the expression from the speech received by the communication unit.
[Claim 5]
5 The information processing device according to Claim 1, wherein
the recognition unit recognizes an operator as the user having performed the
expression and a specific operation as the detail of the expression from an operation
signal in the predetermined signal received by the communication unit.
[Claim 6]
10 The information processing device according to Claim 1, further comprising:
an input unit that allows input of a predetermined signal related to an
expression of a user of the information processing device,
wherein
the recognition unit detects a face from a captured image in the
15 predetermined signal input to the input unit, and recognizes a user of the information
processing device based on a detection result of the face,
the control unit displays a display image where a user image representing
the user of the information processing device recognized by the recognition unit is
fiirther displayed for each user on the display screen,
20 the recognition unit further recognizes a user having performed the
expression and a detail of the expression from the predetermined signal input to the
input unit, and
the control unit displays a display image where the user image representing
the user having performed the expression recognized by the recognition unit and the
25 detail of the expression recognized by the recognition unit are further displayed in
association with each other in the display image on the display screen.
[Claim 7]
The information processing device according to Claim 6, wherein
the recognition unit detects a face from a captured image in the
30 predetermined signal input to the input unit, and recognizes a user whose facial
expression has changed as the user having performed the expression and a facial
51/54
expression as the detail of the expression based on the detection result of the face.
[Claim 8]
The information processing device according to Claim 6, wherein
the recognition unit recognizes a speaker as the user having performed the
5 expression and a presence of a speech as the detail of the expression from a speech in
the predetermined signal input to the input unit.
[Claim 9]
The information processing device according to Claim 8, wherein
the recognition unit further recognizes contents of a speech by the speaker
10 as the detail of the expression from the speech input to the input unit.
[Claim 10]
The information processing device according to Claim 6, wherein
the recognition unit recognizes an operator as the user having performed the
expression and a specific operation as the detail of the expression from an operation
15 signal in the predetermined signal input to the input unit.
[Claim 11]
An information processing method comprising:
a communication step for receiving a predetermined signal related to an
expression of a user of a communication target device from the communication
20 target device;
a detection step for detecting a face from a captured image in the received
predetermined signal in the communication step, and recognizing a user of the
communication target device based on a detection result of the face; and
a control step for displaying a display image where a user image
25 representing the user of the communication target device who is recognized in the
detection step is displayed for each user on a display screen,
wherein
in the recognition step, a user having performed the expression from the
predetermined signal received in the communication step and the detail of the
30 expression are further recognized; and
52/54
in the control step, further in the display image, the user image representing
the user having performed the expression who is recognized in the recognition step
and the detail of the expression recognized in the recognition step are displayed in
the display screen in association with each other.
5
53/54
36
WO 2012/046425 PCT/JP2011/005535
including information of a recognized user and a recognized behavior
of the recognized user; and
controlling display on a display screen of an image including a user
image coiTesponding to the information of the recognized user and a
visual representation corresponding to the information of the recognized behavior.