18
WO 2012/046432 PCT/JP2011/005567
position information.
[Claim 10] The apparatus of claim 9, wherein the determined image is' a cursor
image.
[Claim 11] The apparatus of claim 7, wherein the pointing information indicates a
shape of the pointing object.
[Claim 12] The apparatus of claim 11, wherein the pointing image is in accordance
with the shape of the pointing object.
[Claim 13] The apparatus of claim 11, wherein the pointing image indicates a user
corresponding to the pointing object based on the shape of the pointing
object. y-^
Dated this March 28, 2013 ^^
[SHRIMANTI SINGH]
OF REMFRY «s/ SAGAR
[Title of Document] REQUEST FOR PATENT
[Reference number] 308700JP00
[Filing Date] 4 October 2010
[Address] JPO Commissioner
[Int. CI.] G06F 3/01
[Inventor]
[Domicile or Residence]c/o SONY CORPORATION,
1-7-1 Konan, Minato-ku, Tokyo, Japan
[Name] SAKAI, Yusuke
[Inventor]
[Domicile or Residence]c/o SONY CORPORATION,
1-7-1 Konan, Minato-ku, Tokyo, Japan
[Name] KONDO, Masao
[Applicant for Patent]
[Identification Number] 000002185
[Name] SONY CORPORATION
[Agent]
[Identification Number] 100095957
[Patent Attorney]
[Name] KAMEYA, Yoshiaki
[Telephone Number] 03-5919-3808
[Appointed Agent]
[Identification Number] 100096389
[Patent Attorney]
[Name] KANEMOTO, Tetsuo
[Telephone Number] 03-3226-6631
[Appointed Agent]
[Identification Number] 100101557
[Patent Attorney]
1
[Name] HAGIWARA, Yasushi
[Telephone Number] 03-3226-6631
[Appointed Agent]
[Identification Number] 100128587
[Patent Attorney]
[Name] MATSUMOTO, Kazunori
[Telephone Number] 03-5919-3808
[Fee]
[Prepayment Ledger Number] 005094
[Amount] JPY 15,000-
[List of Attached Documents]
[Title of Document] Specification 1
[Title of Document] Claims 1
[Title of Document] Abstract 1
[Title of Document] Drawings 1
[Number of General Power of Attorney] 1003295
2
[Title of Document] SPECIFICATION
[Title of the Invention] INFORMATION PROCESSING APPARATUS, INFORMATION
PROCESSING SYSTEM AND INFORMATION PROCESSING METHOD
[Technical Field]
[0001]
The present disclosure relates to an information processing apparatus, an
information processing system and an information processing method.
[Background Art]
[0002]
In the related art, a television conference system is known which transmits and
receives images and audio sounds between a plurality of distant user terminals. By the
way, there are some occasions in television conferences that images and audio sounds
are transmitted and received referring to common image information displayed on a
plurality of user terminals.
[Summary of the Invention]
[Object to be Achieved by the Invention]
[0003]
When a user points to an arbitrary position on image information for other users,
a user terminal is operated to display a marker on a pointing position or the pointing
position is notified by an audio sound. Therefore, there are cases where the user takes
labor to point to an arbitrary position on image information for other users.
[0004]
Accordingly, the disclosure provides an information processing apparatus, an
information processing system and an information processing method which can readily
point to an arbitrary position on image information for other users.
[Solution for Achieving Object]
[0005]
According to an aspect of the disclosure, an information processing apparatus is
provided which includes a captured image input unit which receives from an image
3
capturing apparatus, an input of a captured image capturing a user positioned in front of
a display apparatus, a detecting unit which detects a face of the user from the captured
image, and generates position information indicating a position of the face, a pointing
image generating unit which generates a pointing image including information which
enables a position which the user points to by means of a pointing object to be specified,
and information which enables the user to be identified, a superimposed image
generating unit which generates a superimposed image in which the pointing image is
superimposed on a content image displayed on the display apparatus based on the
position information, a display control unit which makes the display apparatus display
the superimposed image, and a transmitting unit which transmits at least the captured
image to another information processing apparatus to display the superimposed image by
means of the another information processing apparatus connected through a
communication line.
[0006]
In addition, the detecting unit may detect the pointing object and further
generate position information indicating a position of the pointing object, and the
pointing image generating unit may generate a pointing image including information
indicating the position of the pointing object specified based on the position information,
and information which enables the user to be identified.
[0007]
In addition, the superimposed image generating unit may match a predetermined
range which the user can point to in the captured image, with a range of the content
image, and generate a superimposed image in which the pointing image is superimposed
on the content image based on the position information.
[0008]
In addition, the display control unit may match a range of the captured image
with a range of the content image, and generate a superimposed image in which the
pointing image is superimposed on the content image based on the position information.
[0009]
4
In addition, the pointing image may be an image specified from a mirror image
of the captured image based on the position information to include the pointing object
and the face of the user.
[0010]
In addition, the pointing image may be an image in which a background image is
removed from the specified image.
[0011]
In addition, the information processing apparatus may further include a
recognizing unit which recognizes the user from the detected face of the user, and the
pointing image may be an image including information indicating a position of the
pointing object specified based on the position information, and information indicating
the recognized user.
[0012]
In addition, the information processing apparatus may further include a
recognizing unit which recognizes a shape of the pointing object, and the pointing image
is an image which becomes different when a first shape is recognized and when a second
shape is recognized.
[0013]
In addition, when detecting pointing objects and faces of a plurality of users, the
detecting unit may detect the pointing objects and the faces of the plurality of users.
[0014]
In addition, when detecting the pointing objects and the faces of the plurality of
users in the captured image, the detecting unit may detect the pointing object and the
face of a user having a maximum area of the detected face.
[0015]
In addition, the information processing apparatus may further include a
receiving unit which receives a captured image capturing another user, from the another
information processing apparatus, and the pointing image generating unit may generate a
pointing image including information which enables a position which the another user
5
points to by means of a pointing object to be specified, and information which enables
the another user to be identified, and the superimposed image generating unit may
generate a superimposed image in which at least the pointing image of the another user
is superimposed on the content image displayed on the display apparatus, based on
position information indicating a position of a face of the another user.
[0016]
In addition, the superimposed image generating unit may generate a
superimposed image in which the pointing image of the user and the pointing image of
another user are superimposed on the content image displayed on the display apparatus,
based on the position information.
[0017]
In addition, when the pointing image of the user and the pointing image of the
another user are displayed overlapping on the content image, the superimposed image
generating unit may increase display transmittance of at least one pointing image and
generate a superimposed image in which the two pointing images are displayed
distinguishably.
[0018]
In addition, the information processing apparatus may further include the
display apparatus and the image capturing apparatus.
[0019]
According to another aspect of the disclosure, an information processing system
is provided which includes a first information processing apparatus, and a second
information processing apparatus connected to the first information processing apparatus
through a communication line. The first information processing apparatus includes a
captured image input unit which receives an input of a captured image capturing a user
positioned in front of a display apparatus, a detecting unit which detects a face of the
user from the captured image, and generates position information indicating a position
of the face, a pointing image generating unit which generates a pointing image including
information which enables a position which the user points to by means of a pointing
6
object to be specified, and information which enables the user to be identified, a
superimposed image generating unit which generates a superimposed image in which the
pointing image is superimposed on a content image displayed on the display apparatus
based on the position information, a display control unit which makes the display
apparatus display the superimposed image, and a transmitting unit which transmits at
least the captured image to the second information processing apparatus. The second
information processing apparatus includes a receiving unit which receives the captured
image capturing the user, a pointing image generating unit which generates a pointing
image including information which enables a position which the user points to by means
of a pointing object to be specified, and information which enables the user to be
identified, a superimposed image generating unit which generates a superimposed image
in which the pointing image is superimposed on the content image based on position
information indicating the position of the face of the user, and a display control unit
which makes a display apparatus connected to the second information processing
apparatus display the superimposed image.
[0020]
According to another aspect of the disclosure, an information processing method
performed by an information processing apparatus is provided, which includes receiving
an input of a captured image capturing a user positioned in front of a display apparatus,
detecting a face of the user from the captured image, and generating position
information indicating a position of the face, generating a pointing image including
information which enables a position which the user points to by means of a pointing
object to be specified, and information which enables the user to be identified,
generating a superimposed image in which the pointing image is superimposed on a
content image displayed on the display apparatus based on the position information,
making the display apparatus display the superimposed image, and transmitting at least
the captured image to another information processing apparatus to display the
superimposed image by means of the another information processing apparatus
connected with the information processing apparatus through a communication line.
7
[Advantage of the Invention]
[0021]
The disclosure can provide an information processing apparatus, an information
processing system and an information processing method which can readily point to an
arbitrary position on image information for other users.
[Brief Description of Drawings]
[0022]
[Fig. 1] Fig. 1 is a block diagram illustrating a configuration of a television
conference system according to an embodiment of the disclosure.
[Fig. 2] Fig. 2 is a view illustrating a configuration of a display system.
[Fig. 3] Fig. 3 is a block diagram illustrating a hardware configuration of an
information processing apparatus.
[Fig. 4] Fig. 4 is a flowchart (1/2) illustrating a procedure of a basic operation of
a television conference system.
[Fig. 5] Fig. 5 is a flowchart (2/2) illustrating a procedure of a basic
configuration of a television conference system.
[Fig. 6] Fig. 6 is a view illustrating an example of generating a superimposed
image including a pointing image of a user.
[Fig. 7] Fig. 7 is a view illustrating a relationship between a user's position in
front of a display apparatus and a superimposing position of a pointing image.
[Fig. 8] Fig. 8 is a view (1/4) illustrating an example of generating a
superimposed image.
[Fig. 9] Fig. 9 is a view (2/4) illustrating an example of generating a
superimposed image.
[Fig. 10] Fig. 10 is a view (3/4) illustrating an example of generating a
superimposed image.
[Fig. 11] Fig. 11 is a view (4/4) illustrating an example of generating a
superimposed image.
[Fig. 12] Fig. 12 is a view illustrating an example of generating a superimposed
8
image including pointing images of a plurality of users.
[Fig. 13] Fig. 13 is a view illustrating a modified example of generating a
superimposed image including pointing images of a plurality of users.
[Fig. 14] Fig. 14 is a view illustrating a setting example of a pointable range.
[Fig. 15] Fig. 15 is a view illustrating an example of a pointing operation based
on a pointable range.
[Fig. 16] Fig. 16 is a view illustrating another example of a pointing operation
based on a pointable range.
[Fig. 17] Fig. 17 is a view (1/2) illustrating an example of a superimposed image
including a pointing image of a different mode.
[Fig. 18] Fig. 18 is a view (2/2) illustrating an example of a superimposed image
including a pointing image of a different mode.
[Fig. 19] Fig. 19 is a block diagram illustrating another configuration of a
television conference system.
[Mode for Carrying out the Invention]
[0023]
Hereinafter, preferred embodiments of the present disclosure will be described
in detail with reference to the appended drawings. Note that, in this specification and
the appended drawings, structural elements that have substantially the same function and
structure are denoted with the same reference numerals, and repeated explanation of
these structural elements is omitted.
[0024]
Fig. 1 is a block diagram illustrating a configuration of a television conference
system (information processing system) according to an embodiment of the disclosure.
As illustrated in Fig. 1, the television conference system includes a plurality of user
terminals including a display system 100 and an information processing apparatus 200
connected to the display system 100. As illustrated in Fig. 2, the display system 100
includes a display apparatus 102 such as a display or projector, an image capturing
apparatus 104 such as a video camera or still camera, and a speaker and microphone or
9
the like (not illustrated).
[0025]
For example, a first user terminal utilized by a user A includes the display
system 100 and information processing apparatus 200. A second user terminal utilized
by a plurality of other users include a display system 400 and an information processing
apparatus 500. A third user terminal utilized by another user includes a display system
600 and an information processing apparatus 700. A plurality of user terminals are
connected through a communication line 800 including a communication server 300.
[0026]
The display apparatus 102 displays a content image IC, and the image capturing
apparatus 104 captures an image of the user positioned in front of the display apparatus
102. The content image IC is an arbitrary image including a text, figure, symbol,
image or a combination of these, and is displayed on an arbitrary image on a display
screen of the display apparatus 102. The front of the display apparatus 102 means the
front of the display screen of the display apparatus 102.
[0027]
The information processing apparatus 200 includes a captured image input unit
202, a detecting unit 204, a recognizing unit 206, a mirror image generating unit 208, a
communication unit 210, a pointing image generating unit 212, a superimposed image
generating unit 214, a display control unit 216 and a storage unit 218.
[0028]
The captured image input unit 202 receives an input of a captured image II
capturing the user, from the image capturing apparatus 104. The captured image input
unit 202 supplies the captured image II to the detecting unit 204, mirror image
generating unit 208 and communication unit 210. The captured image II is an image
capturing an object positioned in a predetermined range in front of the display apparatus
102, that is, in a range of an angle of view of the image capturing apparatus 104.
Although the captured image II is a moving image, the captured image II may be a still
image depending on cases. The user generally looks at the content image IC displayed
10
on the display apparatus 102. Hence, the captured image II generally includes an
image capturing a user's face F from nearly the front.
[0029]
The detecting unit 204 detects the user's face F from the captured image II, and
generates position information indicating the position of the face F in the captured image
II from the detection result. The detecting unit 204 selectively detects a pointing
object H which the user uses to point to an arbitrary position, from the captured image II,
and generates position information indicating the position of the pointing object H in the
captured image II, from the detection result. The detecting unit 204 supplies the
detection result to the recognizing unit 206, communication unit 210 and pointing image
generating unit 212, and supplies position information to the communication unit 210
and pointing image generating unit 212. The detecting unit 204 detects the user's face
F and pointing object H using a common image detection technique based on feature
amount information stored in, for example, the storage unit 218. The pointing object H
refers to a part of the body or a tool such as the user's finger of the hand, palm of the
hand, the back of the hand or pointing stick which is generally used to point to an
arbitrary position.
[0030]
The recognizing unit 206 recognizes the user from the detection result of the
face F, and selectively recognizes the shape of the pointing object H from the detection
result of the pointing object H. The recognizing unit 206 supplies the recognition
result to the communication unit 210 and pointing image generating unit 212. The
recognizing unit 206 recognizes the user and shape of the pointing object H using a
common image recognition technique, based on recognition information stored in, for
example, the storage unit 218. The shape of the pointing object H refers to the shape of
the hand or orientation of the hand of the user, and is used to determine a display mode
of a pointing image 13 although described below. In addition, the recognizing unit 206
may be removed if the user and pointing object H are not recognized.
[0031]
11
The mirror image generating unit 208 generates a mirror image 12 from the
captured image II to supply to the pointing image generating unit 212. The mirror
image generating unit 208 also generates a mirror image 12 of a captured image II of
another user. The mirror image 12 means an image which is a mirror image in which
the left and right of the captured image II are reversed.
[0032]
The communication unit 210 functions as a transmitting unit which transmits the
captured image II of the user, detection result, recognition result and position
information, to other user terminals through the communication line 800. Further, the
communication unit 210 functions as a receiving unit which receives the captured image
II of another user, detection result, recognition result and position information, from
another user terminal through the communication line 800. The communication unit
210 encodes the captured image II supplied from the captured image input unit 202 to
transmit, and decodes the received encoded data to the captured image II to supply to
the mirror image generating unit 208.
[0033]
The pointing image generating unit 212 generates the pointing image 13 based
on the detection result, recognition result, position information and mirror image 12 to
supply to the superimposed image generating unit 214. The pointing image generating
unit 212 also generates the pointing image 13 of the captured image II of another user.
The pointing image generating unit 212 selectively generates the pointing image 13 using
user identification information stored in the storage unit 218.
[0034]
The pointing image 13 includes information which enables a pointing position
which the user points to by means of the pointing object H to be specified, and
information which enables the user to be identified. The information which enables the
pointing image to be specified refers to, for example, an image of the pointing object H
included in the captured image II, and an image showing the pointing position specified
from the detection result of the pointing object H. The information which enables the
12
user to be identified refers to, for example, an image of the user's face F included in the
captured image II, and identification information assigned to the user (for example,
user's name and character). Further, the details of the pointing image 13 will be
described below.
[0035]
The superimposed image generating unit 214 generates a superimposed image 14
based on the content IC, pointing image 13 and position information to supply to the
display control unit 216. The superimposed image generating unit 214 may generate
the superimposed image 14 including the pointing images 13 of the user and another user,
and may generate the superimposed image 14 including only the pointing image 13 of
another user depending on cases. The superimposed image 14 refers to an image in
which the pointing image 13 is superimposed on the content image IC displayed on the
display apparatus 102, based on position information. In addition, the details of the
superimposed image 14 will be described below.
[0036]
The display control unit 216 displays the content image IC on the display
apparatus 102. Further, the display control unit 216 makes the display apparatus 102
display the superimposed image 14 in which the pointing image 13 is superimposed on
the content image IC. The storage unit 218 stores feature amount information used to
detect the user's face F and pointing object H. The storage unit 218 stores recognition
information used to recognize the user and shape of the pointing object H. The storage
unit 218 stores, for example, images used to generate the pointing image 13 (for example,
images showing pointing positions and characters representing users) and text (for
example, users' names).
[0037]
In addition, with the configuration illustrated in Fig. 1, images are detected and
recognized using the captured image II. However, instead of the captured image II,
images may be detected and recognized using the mirror image 12. In this case,
information used to detect and recognize images is prepared to match the mirror image
13
12.
[0038]
Further, with the configuration illustrated in Fig. 1, the detection result,
recognition result and position information are transmitted to another user terminal
together with the captured image II. However, by transmitting only the captured image
II (or mirror image 12) to another user terminal, the user terminal which has received the
captured image II may detect and recognize images and generate position information.
Further, by transmitting only the captured image II (or mirror image 12) to the
communication server 300, the communication server 300 which has received the
captured image II (or mirror image 12) may detect and recognize images and generate
position information, and transmit a processing result to other user terminals.
[0039]
Fig. 3 is a block diagram illustrating a hardware configuration of the
information processing apparatus 200. As illustrated in Fig. 3, the information
processing apparatus 200 includes a MPU 230, a ROM 232, a RAM 234, a storage
medium 236, an input/output interface 238, an operation input device 240, a display
device 242, a communication interface 244 and a bus 246. The bus 246 connects the
MPU 230, ROM 232, RAM 234, storage medium 236, input/output interface 238, and
communication interface 244.
[0040]
The MPU 230 reads a program stored in, for example, the ROM 232, RAM 234
and storage medium 236, expands the program on the RAM 234 and executes the
program to control the operation of the information processing apparatus 200. The
MPU 230 functions particularly as the detecting unit 204, recognizing unit 206, mirror
image generating unit 208, pointing image generating unit 212, superimposed image
generating unit 214 and display control unit 216. The RAM 234 functions as the
storage unit 218.
[0041]
The input/output interface 238 inputs and outputs, for example, data to and from
14
an AV system. The input/output interface 238 functions particularly as the captured
image input unit 202 and display control unit 216. The operation input device 240
includes, for example, a keyboard, mouse and touch panel, and supplies operation
information input through the device, to the MPU 230 through the input/output interface
238. The display device 242 displays, for example, image information and text
information indicating the operation state of the system. The communication interface
244 transmits and receives, for example, data between the communication server 300
and other information processing apparatuses 500 and 700 through the communication
line 800. The communication interface 244 functions as particularly as the
communication unit 210.
[0042]
Fig. 4 and Fig. 5 are flowcharts illustrating a procedure of a basic operation of a
television conference system. Fig. 4 and Fig. 5 illustrate the procedure of pointing to
an arbitrary position on the content image IC in a state where a common content image
IC is displayed on a plurality of user terminals. In addition, the operation of pointing
to an arbitrary position on the content image IC is also referred to as "pointing
operation". Hereinafter, although a case will be described where a pointing operation is
performed between the user terminal A of the user A and the user terminal B of the user
B, a case will be described in the same manner where the pointing operation is
performed among three or more user terminals.
[0043]
First, the user terminal A decides whether or not the user A selected a pointing
mode (step SI00). The pointing mode may be selected by operating the user terminal A
or selected by inputting a predetermined gesture. When the pointing mode is selected,
the user terminal A decides that network connection is established with another user
terminal, in this case, the user terminal B (step SI02). Further, if connection to the
network is not established, processing in step SI08 is executed, and, if connection to the
network is established, processing in step SI04 is executed.
[0044]
15
which the user A points to by means of the pointing object H to be specified, and
information which enables the user A to be identified. Fig. 6© illustrates an example of
the pointing image 13 generated from the mirror image 12 illustrated in Fig. 6(B). The
pointing image 13 illustrated in Fig. 6(C) is generated by specifying a surrounding area
of the user's face F from the mirror image 12 based on position information of the face F.
The pointing image 13 is generated by making an image other than the surrounding area
transparent based on the mirror image 12, and clipping the image of the surrounding area.
[0048]
The surrounding area is set as an area in a range of a certain distance from the
position of the face F in the up, down, left and right directions, based on, for example,
position information of the face F. For example, the surrounding area may be set as an
area having a range with the width corresponding to two faces in the left and right
directions based on the position of the face F and the height corresponding to the two
faces F in the up and down directions. The surrounding area is set by a user terminal or
user to include the image of the pointing object H together with the image of the face F.
[0049]
The superimposed image generating unit 214 generates the superimposed image
14 based on the content image IC, pointing image 13 and position information (step
S206). The display control unit 216 displays the superimposed image 14 on the display
apparatus 102 (step S208). Fig. 6(D) illustrates an example of the superimposed image
14 in which the pointing image 13 illustrated in Fig. 6(B) is superimposed on the content
image IC. With the superimposed image 14 illustrated in Fig. 6(D), by matching the
range of the mirror image 12 with the range of the content image IC, the pointing image
13 is superimposed on the content image IC based on position information of the face F.
In addition, with the superimposed image 14, the pointing image 13 may be
superimposed on the content image IC after increasing display transmittance of the
pointing image 13.
[0050]
Fig. 7 are views illustrating the relationship between the user's position in front
17
of the display apparatus 102 and a superimposing position of the pointing image 13. As
described above, the superimposed image 14 is generated by superimposing the pointing
image 13 generated from the mirror image 12 on the content image IC, based on position
information of the face F. Hence, as illustrated in Fig. 7(A), when the user moves to
the right side in front of the display apparatus 102, the pointing image 13 is
superimposed on the right side of the content image IC. By contrast with this, as
illustrated in Fig. 7(B), when the user moves to the left side in front of the display
apparatus 102, the pointing image 13 is superimposed on the left side of the content
image IC.
[0051]
Further, the superimposed image 14 is basically generated by matching the range
of the mirror image 12 (or captured image II), that is, the range of the angle of view of
the image capturing apparatus 104 with the range of the content image IC.
Consequently, when, as illustrated in Fig. 7(A) and Fig. 7(B), the image capturing
apparatus 104 is arranged in the center of the front of the display apparatus 102 and the
range of the angle of view of the image capturing apparatus 104 matches with the range
of the content image IC, the user is positioned in front of the pointing image 13
superimposed on the content image IC.
[0052]
By contrast with this, if the range of the angle of view of the image capturing
apparatus 104 is twice the range of the content image IC, when the user moves to the
right side over the distance x, the pointing image 13 moves to the right side over the
distance x/2. Further, if the range of the angle of view of the image capturing
apparatus 104 is half the range of the content image IC, when the user moves to the right
side over the distance x, the pointing image 13 moves to the right side over the distance
2x.
[0053]
Fig. 8 to Fig. 11 are views illustrating examples of generating the superimposed
image 14. In addition, in Fig. 8 to Fig. 11, the user A points to a predetermined area
18
included in the content image IC (area in which an image of a UFO is shown) by means
of the user A's fmger which is the pointing object H.
[0054]
Similar to the superimposed image 14 illustrated in Fig. 6(D), Fig. 7(A) and Fig.
7(B), with the superimposed image 14 illustrated in Fig. 8, the pointing image Ia3
showing only a surrounding area of the user A's face F is superimposed on the content
image IC. Meanwhile, the superimposed image 14 also includes a background image
included in the surrounding area.
[0055]
With the superimposed image 14 illustrated in Fig. 9, the pointing image Ia3 in
which the background image is removed from the mirror image Ia2 is superimposed on
the content image IC. The pointing image 13 is generated by specifying an area of the
upper body including the user A's face F and pointing object H included in the mirror
image Ia2 based on the detection result of the user A's face F, making an image other
than the specified area transparent and clipping the image of the specified area.
[0056]
With the superimposed image 14 illustrated in Fig. 10, the pointing image Ia3
formed with an image of areas of the user A's face F specified from the mirror image Ia2
and pointing object H is superimposed on the content image IC. The pointing image
Ia3 is generated by making the image other than the specified area transparent and
clipping the image of the specified area. The pointing image Ia3 is generated by
detecting the user A's face F and pointing object H from the captured image lal and
generating position information of the user A's face F and pointing object H.
[0057]
With the superimposed image 14 illustrated in Fig. 11, an image showing the
pointing position and the pointing image Ia3 including a text indicating the user A's
name are superimposed on the content image IC. In this case, the image included in the
mirror image Ia2 is not used for the pointing image Ia3. The pointing image Ia3 is
generated by detecting the user A's face F and pointing object H from the captured image
19
lal and generating position information of the user A's face F and pointing object H.
Further, the user A is recognized from the detection result of the user A's face F, and the
text "user A" showing the recognized user A's name is read from the storage unit 218.
Similarly, the shape of the pointing object H is recognized from the detection result of
the pointing object H, and an image matching the shape of the recognized pointing
object H is read from the storage unit 218.
[0058]
In addition, the pointing image Ia3 may include a character (for example,
likeness) assigned to the user A instead of the text showing the user A's name, and may
include an image of the area of the user A's face F. Further, the pointing image Ia3 may
include an image of the area of the pointing object H instead of the image showing the
pointing position.
[0059]
Next, an operation when it is decided in step SI02 that connection to the
network is established will be described. The user terminals A and B display a common
content image IC. The communication unit 210 of the user terminal A transmits the
captured image lal of the user A, detection result, recognition result and position
information, to the user terminal B through the communication line 800 (step SI04).
Further, the communication unit 210 of the user terminal A receives the captured image
Ibl of the user B, detection result, recognition result and position information, from the
user terminal B through the communication line 800 (step SI06). By this means, the
user terminal A acquires the captured image Ibl of the user B (step S200). In addition,
the same processing is also performed in the user terminal B.
[0060]
Further, similar to the captured image lal of the user A, the mirror image
generating unit 208 generates the mirror image Ib2 from the captured image Ibl of the
user B (step S202). Similar to the pointing image Ia3 of the user A, the pointing image
generating unit 212 generates the pointing image Ib3 of the user B based on the
detection result, selective recognition result, position information and mirror image Ib2
20
(step S204). The superimposed image generating unit 214 generates the superimposed
image 14 based on the content image IC, pointing image Ia3 and position information of
the user A and pointing image Ib3 and position information of the user B (step S206).
The display control unit 216 displays the superimposed image 14 on the display
apparatus 102 (step S208). In addition, the same processing is also performed in the
user terminal B.
[0061]
Fig. 12 are views illustrating examples of generating the superimposed image 14
including the pointing image Ia3 and Ib3 of the users A and B. Fig. 12(A) illustrates an
image of the upper body including the user A's finger H of the hand and face F, on the
viewer's left side, and illustrates the mirror image Ia2 of the captured image lal of the
user A including a background image. Fig. 12(B) illustrates an image of the upper body
including the user B's finger H of the hand and face F, on the viewer's right side, and
illustrates the mirror image Ib2 of the captured image Ibl of the user B including a
background image.
[0062]
Fig. 12(C) illustrates an example of the superimposed image 14 in which the
pointing images Ia3 and Ib3 of the users A and B are superimposed on the content image
IC. The mirror image Ia2 of the user A includes the image of the user A on viewer's left
side, and the mirror image Ib2 of the user B includes the image of user B on the viewer's
right side. Hence, with the superimposed image 14, the pointing image Ia3 of the user
A is superimposed on the content image IC, on viewer's left side, and the pointing image
Ib3 of the user B is superimposed on the content image IC on the viewer's right side.
The pointing images Ia3 and Ib3 of the users A and B include information which enables
each user to be identified, and can readily enable the pointing position of each user to be
specified.
[0063]
In addition, the superimposed image 14 illustrated in Fig. 12(C) assumes a case
where the width of the content image IC and the widths of the mirror images Ia2 and Ib2
21
of the user A and user B are the same. Further, although the superimposed image 14
illustrated in Fig. 12(C) is generated similar to the generation example illustrated in Fig.
8, the superimposed image 14 may be generated similar to the generation examples
illustrated in Fig. 9 to Fig. 11.
[0064]
Fig. 13 are views illustrating modified examples of generating the superimposed
image 14 including the pointing images Ia3 and Ib3 of the users A and B. With the
modified examples, a case is assumed where the pointing images Ia3 and Ib3 of the users
A and B are superimposed on the content image IC overlapping each other. Fig. 13(A)
illustrates the mirror image Ia2 of the user A including the image of the upper body
including the finger H of the hand and face F in the center. Fig. 13(B) illustrates the
mirror image Ib2 of the user B including the image of the upper body including the
finger H of the hand and face F in the center.
[0065]
Fig. 13(C) illustrates an example of the superimposed image 14 in which the
pointing images Ia3 and Ib3 of the users A and B are superimposed on the content image
IC. The mirror image Ia2 of the user A includes the image of the user A in the center,
and the mirror image Ib2 of the user B includes the image of the user B in the center.
Therefore, when the superimposed image 14 is generated similar to the generation
example illustrated in Fig. 12(C), the pointing images Ia3 and Ib3 of the users A and B
are overlaid each other, thereby decreasing the visibility. Hence, as illustrated in Fig.
13(C), by increasing display transmittance of the pointing images Ia3 and Ib3 of the user
A and/or user B, a decrease in the visibility of the pointing images Ia3 and Ib3 which are
superimposed overlapping each other is suppressed.
[0066]
A case has been described above where the range of the angle of view of the
image capturing apparatus 104 is matched with the range of the content image IC to
generate the superimposed image 14. In this case, when the range of the angle of view
is much wider than the range of the content image IC or the user's position is much far
22
from the image capturing apparatus 104, the user is forced to make relatively large
movement for the pointing operation. Therefore, it is proposed to set the pointable
range which the user can point to without moving, based on the position of the user's
face F and match the pointable range with the range of the content image IC instead of
the range of the angle of view to generate the superimposed image 14.
[0067]
Fig. 14 is a view illustrating a setting example of the pointable range. The
pointable range illustrated in Fig. 14 is set as a range which can be pointed by the fmger
of the left hand or right hand in a state where the left arm is extended to the left, upper
left or lower left and the right arm is extended to the right, upper right or lower right,
based on the position of the user A's face F. For example, with the example illustrated
in Fig. 14, the pointable range is set as the range of the width L around the position of
the user A's face F. The pointable range may be set by making the user A actually point
to the range which the user A is about to point to, and detecting an area Ha of the
pointing object H which points to the range. Further, the pointable range may be set as
a range with the width corresponding to four faces F in the left and right directions
based on the size of an area Fa of the user A's face F and the width corresponding to the
two faces F in the up and down directions.
[0068]
Fig. 15 are views illustrating examples of the pointing operation based on the
pointable range. Fig. 15(A) illustrates a mirror image 12 of the captured image II
including the user A and user C. The mirror image 12 includes an image of the user A
capturing the upper body including the fmger H of the hand and face F, on the viewer's
left side, and includes an image of the user C capturing the upper body including the
finger H of the hand and face F, on viewer's right side. The user C is positioned farther
apart from the image capturing apparatus 104 than the user A.
[0069]
On the mirror image 12, rectangular frames representing detection areas Fa, Ha,
Fc and He of the faces F of the users A and C and the pointing objects H, and pointable
23
ranges Pa and Pc are shown. In addition, the rectangular frames of the detection areas
Fa, Ha, Fc and He and pointable ranges Pa and Pc are shown for ease of description to
illustrate processing of the user terminal instead of an image forming part of the mirror
image 12. The user A can perform a pointing operation by moving the pointing object
H in the pointable range Pa of the user A and the range of the mirror image 12.
Similarly, the user C can perform a pointing operation by moving the pointing object H
in the pointable range Pc of the user C and the range of the mirror image 12.
[0070]
Meanwhile, the pointable range Pa of the user A and pointable range Pc of the
user C match with the range of the content image IC, respectively. Consequently, the
users A and C can point to an arbitrary position in the content image IC by moving the
pointing objects H to arbitrary positions in the pointable ranges Pa and Pc of the users A
and C without moving the pointing object H in the range of the angle of view of the
image capturing apparatus 104.
[0071]
Fig. 15(B) illustrates an example of the superimposed image 14 including the
pointing images Ia3 and Ic3 of the users A and C. With the mirror image 12 illustrated
in Fig. 15(A), the user A points to the position closer to the right of the center of the
pointable range Pa of the user A, and the user C points to the position closer to the left
of the center of the pointable range Pc of the user C. Hence, with the superimposed
image 14, the pointing image Ia3 of the user A is superimposed closer to the right of the
center of the content image IC, and the pointing image Ic3 of the user C is superimposed
closer to the left of the center of the content image IC. Meanwhile, the pointing images
Ia3 and Ic3 of the users A and C show the positions which the users A and C point to in
the pointable ranges Pa and Pc of the users A and C.
[0072]
Similarly, when the user A points to the upper right corner of the pointable range
Pa of the user A, the pointing image Ia3 of the user A is superimposed on the upper right
corner of the content image IC, and, when the user C points to the lower left corner of
24
the pointable range Pc of the user C, the pointing image Ic3 of the user C is
superimposed on the lower left corner of the content image IC. In addition, although
the superimposed image 14 illustrated in Fig. 15(B) is generated similar to the generation
example illustrated in Fig. 11, the superimposed image 14 may be generated similar to
the generation example illustrated in Fig. 10.
[0073]
Fig. 16 are views illustrating other examples of the pointing operation based on the
pointable range. Similar to Fig. 15(A), Fig. 16(A) illustrates a mirror image 12
capturing the users A and C. On the mirror image 12, rectangular frames representing
detection areas Fa and Ha of the face F and pointing object H of the user A, and the
pointable range Pa are shown. With the example illustrated in Fig. 16, by detecting the
users A and C's faces F from the captured image II and comparing the sizes of the areas
Fa and Fc of the detected faces F, only the user A having a large area of the face F, that
is, the user A closest to the image capturing apparatus 104 is allowed to perform the
pointing operation.
[0074]
Fig. 16(B) illustrates an example of the superimposed image 14 including the
pointing image Ia3 of the user A. With the mirror image 12 illustrated in Fig. 16(A),
the user A points to the position closer to the right of the center of the pointable range Pa
of the user A. Hence, with the superimposed image 14, the pointing image Ia3 of the
user A is superimposed closer to the right of the center of the content image IC.
[0075]
In addition, instead of the user closest to the image capturing apparatus 104,
only the user farthest from the image capturing apparatus 104 or the user positioned at
an adequate distance from the image capturing apparatus 104 may be allowed to perform
the pointing operation. Further, when a plurality effaces F are detected, only a
predetermined number of users positioned at adequate distances from the image
capturing apparatus 104 may be allowed to perform the pointing operation.
[0076]
25
A case has been described above where the superimposed image 14 by detecting
the position of the pointing object H. By the way, the user wants to superimpose the
pointing image 13 of a different mode on the content image IC depending on cases.
Hence, it is proposed to generate the superimposed image 14 including the pointing
image 13 of a different mode by detecting the position of the pointing object H and
recognizing the shape of the pointing object H. By this means, by changing the mode
of the pointing image 13, it is possible to, for example, transmit additional information to
other users and, thereby, improve convenience upon the pointing operation.
[0077]
Fig. 17 and Fig. 18 are views illustrating examples of the superimposed image
14 including the pointing image 13 of a different mode. With the example illustrated in
Fig. 17, the user forms a ring by bending the thumb and forefinger and moves the ring
from the left to the right in front of the display apparatus 102 drawing a wave. Then,
the user terminal recognizes the shape of the pointing object H, and generates the
superimposed image 14 including the pointing image 13 showing the character
corresponding to the user according to a mode matching the recognized shape.
Meanwhile, the pointing image 13 shows the character according to a mode which
enables clear recognition of the character. With the superimposed image 14, as
indicated by a dotted line in Fig. 17, the pointing image 13 moves on the content image
IC in association with movement of the pointing object H.
[0078]
By contrast with this, with the examples illustrated in Fig. 18, as illustrated in
Fig. 18(A), the user first moves the forefinger from the left to the right in front of the
display apparatus 102 drawing a wave in a state where the forefinger is stretched. Then,
the user terminal recognizes the shape of the pointing object H, and generates the
superimposed image 14 including the pointing image 13 representing the character
according to a mode different from Fig. 17. Meanwhile, the pointing image 13 shows
the character according to a mode which does not enable clear recognition of the
character.
26
[0079]
Further, as illustrated in Fig. 18(B), the user moves the pointing object H and
then makes a ring by bending the thumb and forefinger. Then, the user terminal
recognizes the change of the shape of the pointing object H, and generates the
superimposed image 14 including the pointing image 13 representing the character
according to the same mode as in Fig. 17. In addition, although cases have been
described in Fig. 17 and Fig. 18 where two different shapes are recognized, three or
more different shapes may be recognized.
[0080]
As described above, with the television conference system according to an
embodiment of the disclosure, each user terminal detects the user's face F from the
captured image II and generates information indicating the position of the face F,
generates the pointing image 13 including information which enables the pointing
position of the user to be specified and identification information of the user, and
generates and displays the superimposed image 14 in which the pointing image 13 is
superimposed on the content image IC based on position information. Further, at least
the captured image II is transmitted to other user terminals to display the same
superimposed image 14 on the other user terminals. By this means, between users
utilizing a plurality of user terminals, it is possible to readily point to an arbitrary
position on content information which is commonly displayed, for other users without a
complicated manipulation or operation.
[0081]
It should be understood by those skilled in the art that various modifications,
combinations, sub-combinations and alterations may occur depending on design
requirements and other factors insofar as they are within the scope of the appended
claims or the equivalents thereof
[0082]
For example, a case has been described with the above embodiment where the
user terminal includes the display system 100 and information processing apparatus 200.
27
However, as illustrated in Fig. 19, the user terminal may include an information
processing apparatus 250 in which a configuration of the display system 100 is mounted.
In this case, the information processing apparatus 250 has an image capturing unit 252
(image capturing apparatus) and a display unit 254 (display apparatus) in addition to the
configuration of the information processing apparatus 200 illustrated in Fig. 1. In
addition, there may be display system mount-type information processing apparatuses
and separate-type information processing apparatuses in the television conference
system in a mixed manner.
[Description of Reference Numerals]
[0083]
100, 400, 600 Display system
102 Image capturing apparatus
104 Display apparatus
200, 250, 500, 550, 700, 750 Information processing apparatus
300 Communication server
800 Communication line
202 Captured image input unit
204 Detecting unit
206 Recognizing unit
208 Mirror image generating unit
210 Communication unit
212 Pointing image generating unit
214 Superimposed image generating unit
216 Display control unit
218 Storage unit
252 Image capturing unit
254 Display unit
11 Captured image
12 Mirror image
28
13 Pointing image
14 Superimposed image
IC Content image

[Title of Document] CLAIMS
[Claim 1]
An information processing apparatus comprising:
a captured image input unit which receives from an image capturing apparatus, an input
of a captured image capturing a user positioned in front of a display apparatus;
a detecting unit which detects a face of the user from the captured image, and generates
position information indicating a position of the face;
a pointing image generating unit which generates a pointing image including information
which enables a position which the user points to by means of a pointing object to be
specified, and information which enables the user to be identified;
a superimposed image generating unit which generates a superimposed image in which
the pointing image is superimposed on a content image displayed on the display
apparatus based on the position information;
a display control unit which makes the display apparatus display the superimposed
image; and
a transmitting unit which transmits at least the captured image to another information
processing apparatus to display the superimposed image by means of the another
information processing apparatus connected through a communication line.
[Claim 2]
The information processing apparatus according to claim 1,
wherein the detecting unit detects the pointing object and further generates position
information indicating a position of the pointing object, and
the pointing image generating unit generates a pointing image including information
indicating the position of the pointing object specified based on the position information,
and the information which enables the user to be identified.
[Claim 3]
The information processing apparatus according to claim 2,
wherein the superimposed image generating unit matches a predetermined range which
the user can point to in the captured image, with a range of the content image, and
30
generates a superimposed image in which the pointing image is superimposed on the
content image based on the position information.
[Claim 4]
The information processing apparatus according to claim 1,
wherein the display control unit matches a range of the captured image with a range of
the content image, and generates a superimposed image in which the pointing image is
superimposed on the content image based on the position information.
[Claim 5]
The information processing apparatus according to claim 4,
wherein the pointing image is an image specified from a mirror image of the captured
image based on the position information to include the pointing object and the face of
the user.
[Claim 6]
The information processing apparatus according to claim 5,
wherein the pointing image is an image in which a background image is removed from
the specified image.
[Claim 7]
The information processing apparatus according to claim 3, further comprising:
a recognizing unit which recognizes the user from the detected face of the user,
wherein the pointing image is an image including information indicating a position of
the pointing object specified based on the position information, and information
indicating the recognized user.
[Claim 8]
The information processing apparatus according to claim 3, further comprising:
a recognizing unit which recognizes a shape of the pointing object,
wherein the pointing image is an image which becomes different when a first shape is
recognized and when a second shape is recognized.
[Claim 9]
The information processing apparatus according to claim 3,
31
wherein, when detecting pointing objects and faces of a plurality of users, the detecting
unit detects the pointing objects and the faces of the plurality of users.
[Claim 10]
The information processing apparatus according to claim 9,
wherein, when detecting the pointing objects and the faces of the plurality of users in the
captured image, the detecting unit detects the pointing object and the face of a user
having a maximum area of the detected face.
[Claim 11]
The information processing apparatus according to claim 1, further comprising:
a receiving unit which receives a captured image capturing another user, from the
another information processing apparatus,
wherein the pointing image generating unit generates a pointing image including
information which enables a position which the another user points to by means of a
pointing object to be specified, and information which enables the another user to be
identified, and
the superimposed image generating unit generates a superimposed image in which at
least the pointing image of the another user is superimposed on the content image
displayed on the display apparatus, based on position information indicating a position
of a face of another user.
[Claim 12]
The information processing apparatus according to claim 11,
wherein the superimposed image generating unit generates a superimposed image in
which the pointing image of the user and the pointing image of the another user are
superimposed on the content image displayed on the display apparatus, based on the
position information.
[Claim 13]
The information processing apparatus according to claim 12,
wherein, when the pointing image of the user and the pointing image of the another user
are displayed overlapping on the content image, the superimposed image generating unit
32
increases display transmittance of at least one pointing image and generates a
superimposed image in which the two pointing images are displayed distinguishably.
[Claim 14]
The information processing apparatus according to claim 1, further comprising:
the display apparatus and the image capturing apparatus.
[Claim 15]
An information processing system comprising a first information processing
apparatus and a second information processing apparatus connected to the first
information processing apparatus through a communication line,
wherein the first information processing apparatus includes:
a captured image input unit which receives an input of a captured image capturing a user
positioned in front of a display apparatus,
a detecting unit which detects a face of the user from the captured image, and generates
position information indicating a position of the face,
a pointing image generating unit which generates a pointing image including information
which enables a position which the user points to by means of a pointing object to be
specified, and information which enables the user to be identified,
a superimposed image generating unit which generates a superimposed image in which
the pointing image is superimposed on a content image displayed on the display
apparatus based on the position information,
a display control unit which makes the display apparatus display the superimposed
image, and
a transmitting unit which transmits at least the captured image to the second information
processing apparatus, and
the second information processing apparatus includes:
a receiving unit which receives the captured image capturing the user,
a pointing image generating unit which generates a pointing image including information
which enables a position which the user points to by means of the pointing object to be
specified, and information which enables the user to be identified,
33
a superimposed image generating unit which generates a superimposed image in which
the pointing image is superimposed on the content image based on position information
indicating the position of the face of the user, and
a display control unit which makes a display apparatus connected to the second
information processing apparatus display the superimposed image.
[Claim 16]
An information processing method performed by an information processing
apparatus comprising:
receiving an input of a captured image capturing a user positioned in front of a display
apparatus,
detecting a face of the user from the captured image, and generating position
information indicating a position of the face,
generating a pointing image including information which enables a position which the
user points to by means of a pointing object to be specified, and information which
enables the user to be identified,
generating a superimposed image in which the pointing image is superimposed on a
content image displayed on the display apparatus based on the position information,
making the display apparatus display the superimposed image, and
transmitting at least the captured image to another information processing apparatus to
display the superimposed image by means of the another information processing
apparatus connected with the information processing apparatus through a communication
line.
34
18
WO 2012/046432 PCT/JP2011/005567
position information.
[Claim 10] The apparatus of claim 9, wherein the determined image is' a cursor
image.
[Claim 11] The apparatus of claim 7, wherein the pointing information indicates a
shape of the pointing object.
[Claim 12] The apparatus of claim 11, wherein the pointing image is in accordance
with the shape of the pointing object.
[Claim 13] The apparatus of claim 11, wherein the pointing image indicates a user
corresponding to the pointing object based on the shape of the pointing object.