FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENTS RULES, 2003
COMPLETE SPECIFICATION
(See section 10, rule 13)
“INFORMATION PROCESSING DEVICE,INFORMATION PROCESSING METHOD,INFORMATION PROVISION DEVICE, AND INFORMATION PROVISION SYSTEM”
SONY CORPORATION, of 1-7-1, Konan, Minato-ku, Tokyo
108-0075, Japan
The following specification particularly describes the invention and the manner in
which it is to be performed.
2
Description
Title of Invention
INFORMATION PROCESSING DEVICE, INFORMATION PROCESSING METHOD, INFORMATION PROVISION DEVICE, AND INFORMATION PROVISION SYSTEM
Technical Field
[0001]
10 The technology disclosed in the present specification relates to an
information processing device and an information processing method that extract
feature point information from content for automatic content recognition, an
information provision device that provides information of the content based on the
feature point information extracted from the content, and an information provision
15 system that provides the information of the content based on the feature point
information extracted from the content.
Background Art
[0002]
20 In recent years, technologies for recognizing and understanding images and
sounds have been developed, and a variety of applications of consumer appliances
have used such various technologies for recognizing and understanding images and
sounds. As an example, ACR (Automatic Content Recognition) constructed in a
client server system using the Internet is exemplified, which is called an ACR service,
25 and various applications have been proposed.
[0003]
As one application of the ACR service, content right restriction such as
detecting illegal copies can be exemplified. When content is distributed through the
Internet, for example, the content is identified so that whether the content is
30 legitimate is checked using such a recognition and understanding technology.
Recently, according to the proliferation of file sharing services, a need to more
3
accurately identify content using images and sounds has increased.
[0004]
In addition, as another application of the ACR service, a service of
providing users who view content with added values can be exemplified. For
example, broadcasted or distributed content is identified and information 5 relating to
the content is provided through the Internet so as to be synchronized with viewing of
the content. The CDDB music identification service provided by Gracenote of the
US (Gracenote, Inc., Berkeley, California) identifies a compact disc (CD) and
provides access to information relating to the identified CD (the album title, artist
10 name, track list, relevant content on the Internet (the album cover, artist, fan site),
and the like).
[0005]
In the ACR service, a watermark (electronic watermark) or a fingerprint
(feature point information) is extracted from content and the content is identified in,
15 for example, the final stage in which the content is decoded and displayed, and the
service does not rely on a delivery chain of the content.
[0006]
When content that includes video information and audio information of a
broadcasting program or the like is identified, a method of identifying the content
20 using only one of the video information and the audio information and a method of
identifying the content using both pieces of the information are considered.
[0007]
For example, as a method of identifying content using only audio
information, a method has been proposed in which one or more segments of a
25 waveform that has been digitally sampled are used to form an amplitude signature of
the waveform by counting the number of times of occurrence in the segment of the
waveform in each of a plurality of amplitude bands or slots, fuzzy comparison with
amplitude signatures in a database is executed, and when one or more potential
matching cases are found, more precise comparison is executed, and thereby
30 matching of waveforms is found in a recorded database indicating the waveforms
(for example, refer to Patent Literature 1).
4
[0008]
Whether only one of video information and audio information is used or
both pieces of the information are used when content should be identified is defined
as a part of service specifications or application specifications during system
designing according to an intended application of the ACR service, 5 an index of a
content recognition rate, system design and restriction on operation costs, and the
like.
[0009]
At present, realizing the ACR service using only audio information is
10 considered to be most advantageous to lessening loads on an amount of information
and a number of processes to be dealt with in light of system design and operation
costs. This is because there are many cases in which such audio information may
have a smaller data amount of feature point information for identifying content than
video information.
15 [0010]
However, as there are a larger number of pieces of content to be dealt with,
a data amount of feature point information to be prepared on a server side of the
ACR service increases even though a data amount of feature point information of
each piece of content is small. An increasing physical capacity of a database in the
20 ACR service that deals with audio information is a challenging task in terms of
system design and operations.
[0011]
With regard to the task in terms of system design, if the number of pieces of
content to be dealt with increases, a capacity of a database in which feature point
25 information for identifying the content is stored increases, and a process for the
identification and a time taken in the identification increase as well. In addition,
with regard to the task in terms of a system operation, if the number of pieces of
content to be dealt with increases, a capacity of a database in which feature point
information for identifying the content is stored increases, and investment in
30 facilities for preparing physical databases, maintenance expenses and the like
increase. The present inventors consider it necessary to take measures for such
5
tasks even when only audio information is used.
Citation List
Patent Literature
5 [0012]
Patent Literature 1: JP 2004-511838T
Summary of Invention
Technical Problem
10 [0013]
An object of the technology disclosed in the present specification is to
provide an excellent information processing device and information processing
method that enable appropriate extraction of feature point information from content
that includes audio information so that burdens of system design and operation
15 imposed on a server side of the ACR service are reduced.
[0014]
Another object of technology disclosed in the present specification is to
provide an excellent information provision device that can appropriately provide
information of content based on feature point information extracted from audio
20 information of the content so that burdens of system design and operation are
lessened.
[0015]
In addition, still another object of the technology disclosed in the present
specification is to provide an excellent information provision system that can
25 appropriately provide information of content based on feature point information
extracted from audio information of the content so that burdens of system design and
operation imposed on a server side of the ACR service are reduced.
Solution to Problem
30 [0016]
According to the first aspect of the present application in order to achieve
6
the above-mentioned object, there is provided an information processing device
configured to realize an audio ACR function only with main sound data, the device
including an input unit to which content configured to include a main sound and one
or more sub sounds is input, a sound output unit configured to output a sound by
decoding and reproducing either of the main sound or one 5 or more sub sounds
included in the input content, and a feature point information generation unit
configured to generate feature point information from the main sound included in the
input content.
[0017]
10 According to a second aspect of the present application, the feature point
information generation unit of the information processing device described in the
first aspect is configured to execute a process of generating the feature point
information when an automatic content recognition operation is performed, and to
recognize the main and sub sounds thereby being able to generate feature point
15 information of the main sound at all times even when the sub sounds are listened to.
[0018]
According to a third aspect of the present application, when the sound
output unit for viewing and listening outputs a sound by decoding and reproducing
the sub sounds, the feature point information generation unit of the information
20 processing device described in the first aspect is configured to decode and reproduce
the main sound when an automatic content recognition operation is performed,
thereby generating the feature point information with regard to a decoding result of
the main sound. In other words, the information processing device is characterized
to start decoding and reproduction of the main sound taking the automatic content
25 recognition operation as an opportunity, and therefore it is not necessary to
simultaneously reproduce the main and sub sounds at all times.
[0019]
According to a fourth aspect of the present application, the information
processing device described in the third aspect is configured to generate the feature
30 point information from the decoded and reproduced main sound at the time of
listening to the sub sounds, and not to be output the main sound.
7
[0020]
According to a fifth aspect of the present application, the content input to
the input unit is multiplexed with an image. In addition, the information processing
device described in the first aspect is configured to further include a demultiplexer
unit configured to demultiplex the input content, the demultiplexer 5 unit performs a
process targeting all sounds included in the content, the sound output unit only
decodes and reproduces a sound signal to be listened to among the demultiplexed
main sound and one or more sub sounds and then outputs a sound, and, when an
automatic content recognition operation is performed when the sub sounds are
10 listened to, the feature point information generation unit decodes and reproduces the
demultiplexed main sound, and then generates the feature point information with
regard to a decoding result of the main sound. In other words, the information
processing device is characterized to only decode either sound signal to be listened to
among main and sub signals, and when the sub sounds are listened to, decoding and
15 the reproduction of the main sound start taking the automatic content recognition
operation as an opportunity, and it is not necessary to simultaneously reproduce the
main and sub sounds at all times.
[0021]
According to a sixth aspect of the present application, the information
20 processing device according to the first aspect further includes a content
identification unit configured to identify the input content based on the feature point
information generated from the main sound by the feature point information
generation unit.
[0022]
25 According to a seventh aspect of the present application, the information
processing device according to the sixth aspect further includes a communication
interface unit configured to be connected to a network, and the content identification
unit is configured to check a reference feature point information database in which
content identification information is associated with the feature point information of
30 the main sound included in the content via the communication interface unit, thereby
acquiring the content identification information.
8
[0023]
According to an eighth aspect of the present application, the information
processing device according to the seventh aspect further includes a relevant
information acquisition unit configured to acquire relevant information of the content
identified by the content identification unit by checking a 5 relevant information
database in which relevant information of each piece of content is stored via the
communication interface unit.
[0024]
In addition, according to a ninth aspect of the present application, there is
10 provided an information processing method including the steps of inputting content
configured to include a main sound and one or more sub sounds, outputting a sound
by decoding and reproducing either of the main sound and one or more sub sounds
included in the input content, and generating feature point information from the main
sound included in the input content.
15 [0025]
In addition, according to a tenth aspect of the present application, there is
provided an information provision device configured to include a reference feature
point information database in which feature point information generated from a main
sound of content configured to include the main sound and one or more sub sounds is
20 associated with content identification information and to return the content
identification information in response to an inquiry configured to include the feature
point information, and the feature point information only of the main sound is
compiled as a database, and a capacity of the database is minimized by only using
the main sound when the feature point information of the content is checked.
25 [0026]
In addition, according to an eleventh aspect of the present application, there
is provided an information provision system configured to include a content
distribution device configured to distribute content configured to include a main
sound and one or more sub sounds, a client device configured to be used in viewing
30 and listening to the content distributed from the content distribution device with any
of audio channels and to generate feature point information from the main sound, and
9
an information provision server configured to include a reference feature point
information database in which the feature point information generated from the main
sound of the content is associated with content identification information and to
provide the content identification information corresponding to an inquiry of the
feature point information from 5 the client device.
[0027]
However, the “system” mentioned here means a logical set of a plurality of
devices (or functional modules that realize specific functions) and whether each
device or functional module is disposed in a single housing does not particularly
10 matter.
Advantageous Effects of Invention
[0028]
According to the technology disclosed in the present specification, it is
15 possible to provide an excellent information processing device and information
processing method that can reduce burdens in terms of system design and operation
imposed on a server of an ACR service by extracting feature point information only
from a main sound out of audio information of content.
[0029]
20 In addition, according to the technology disclosed in the present
specification, it is possible to provide an excellent information provision device that
can lessen burdens of system design and operation by providing information of
content based only on feature point information extracted from a main sound out of
audio information of the content.
25 [0030]
Furthermore, according to the technology disclosed in the present
specification, it is possible to provide an excellent information provision system that
can reduce burdens of system design and operation imposed on a server of an ACR
service by providing information of content based only on feature point information
30 extracted from a main sound out of audio information of the content.
[0031]
10
Other aims, characteristics, and advantages of the technique disclosed in the
present specification will be made clear by a more detailed description based on
embodiments and attached drawings which will be described later.
Brief Description 5 of Drawings
[0032]
[FIG. 1] FIG. 1 is a diagram illustrating a configuration of an information provision
system 100 that provides added value information relating to content in
synchronization with viewing.
10 [FIG. 2] FIG. 2 is a diagram illustrating a configuration example of a client device
140.
[FIG. 3] FIG. 3 is a diagram schematically illustrating an internal configuration of a
demultiplexer unit 217 and a sound decoding unit 219 corresponding to content that
includes a plurality of audio channels.
15 [FIG. 4] FIG. 4 is a flowchart showing a processing procedure for performing an
operation of an ACR function under control of a control microcomputer unit 214.
Description of Embodiment
[0033]
20 Hereinafter, an embodiment of the technology disclosed in the present
specification will be described in detail with reference to the drawings.
[0034]
FIG. 1 shows, as an example of an ACR service, a configuration of an
information provision system 100 that provides a user who views content with added
25 value information relating to the content in synchronization with viewing. The
information provision system 100 shown in the drawing is configured with a content
distribution device 110 that distributes broadcasting content, a client device 140 such
as a TV receiver set with which a user views the distributed content, and an
information provision server 150 that provides added information relating to the
30 content.
[0035]
11
The content distribution device 110 is operated by a broadcasting station, a
content provider, or the like and distributes content in the form of a cable, a satellite,
an IPTV, or the like. The content includes video information and audio information,
but there are cases of content which has one or more sub sounds in addition to a main
sound as the audio information. In addition, along with current 5 digitalization of
broadcasting, and the like, metadata that includes relevant application distribution
control information and the like is distributed at the same time as content. Such
metadata includes content identifiers and the like. When original metadata arrives
at the client device 140, the ACR service can be realized by extracting a content
10 identifier from the metadata.
[0036]
Content distributed (or broadcasted) from the content distribution device
110 that is managed by a broadcasting station, a content provider, or the like is not
limited to being directly received by the client device 140 such as a TV receiver set
15 or the like, and there are also cases in which content is first received by a broadcast
relay station 120 such as a cable company of each region and then redistributed. It
can also be assumed that there are a plurality of broadcasting relay stations engaged
therein. The broadcast relay station 120 may distribute content with a time interval
from the original broadcasting time slot, or change composition of programs. For
20 this reason, when the broadcast relay station 120 redistributes broadcasting content,
the original metadata distributed from the content distribution device 110 may be
changed, or replaced by exclusive metadata of the broadcast relay station 120.
According to the change or replacement of metadata described above, there are cases
in which some or all information included in the original metadata such as content
25 identifiers is lost. After the metadata is lost, an attribute of the content is unclear.
[0037]
As described above, the broadcasting content distributed from the content
distribution device 110 or the broadcast relay station 120 arrives at each household
together with the metadata. However, the content is not limited to being directly
30 received by the client device 140 such as a TV receiver set, and as illustrated in the
drawing, there are cases of content being first received by a dedicated set-top box
12
(STB) 130 for cable or satellite broadcasting, or an IPTV. The set-top box 130
generally separates the received broadcast content into a content main body, which is
composed of video information and audio information, and metadata, and transmits
the content main body to the client device 140 such as a TV receiver set via an RF
(Radio Frequency) 5 cable, or the like.
[0038]
In short, the information provision system 100 illustrated in FIG. 1 is a use
case in which the metadata from a broadcasting station is blocked by the broadcast
relay station 120 such as a cable company or a set-top box 130. In such a case, the
10 client device 140 is not able to identify content being viewed from a content
identifier included in the metadata. Therefore, in the information provision system
100 according to the present embodiment, in order to realize the ACR service, feature
point information extracted from video information or audio information constituting
content is used to identify the content, instead of a content identifier.
15 [0039]
The content distribution device 110 operated by a broadcasting station, a
content provider, or the like transmits metadata attached to distributed content to the
information provision server 150 via, for example, the Internet. In addition, the
content distribution device 110 extracts feature point information for each piece of
20 the distributed content, and transmits the information to the information provision
server 150 via the Internet, or the like.
[0040]
The information provision server 150 has a metadata database 151 in which
metadata is stored, and a reference feature point information database 152 in which
25 feature point information of each piece of content is stored. The metadata database
151 manages metadata of each piece of content, advertisements relating to the
content, applications linked to programs, and the like in association with content
identifiers. In addition, the reference feature point information database 152
manages feature point information of each piece of content in association with
30 content identifiers.
[0041]
13
On the other hand, if feature point information is extracted from content that
a user has viewed on the client device 140 side such as a TV receiver set, the client
device makes an inquiry to the information provision server 150 via the Internet or
the like.
5 [0042]
In response to the inquiry from the client device 140 on the feature point
information, the information provision server 150 checks the reference feature point
information database 152 to return a corresponding content identifier to the client
device 140 as a checking result. In this manner, the client device 140 can ascertain
10 an attribute of the content that has lost metadata.
[0043]
When the client device 140 obtains the content identifier as the checking
result, the client device further makes an inquiry to the information provision server
150 based on the content identifier. Then, in response to the inquiry of the content
15 identifier, the information provision server 150 checks the metadata database 151 to
provide a user with metadata of the content, an advertisement relating to the content,
an application linked to the program, and the like. As a provision method, the data
may be distributed to a mobile device such as a smartphone associated with the client
device 140, in addition to being directly transmitted to the client device 140.
20 [0044]
As described above, when content is identified based on feature point
information extracted from the content, instead of a content identifier, to realize the
ACR service, a method of identifying the content using only one of video
information and audio information and a method of identifying the content using
25 both pieces of the information are considered. In the present embodiment, the ACR
service is set to be realized using only audio information in view of system design
and operation costs. This is because audio information is considered to be most
advantageous in lessening a load of an information amount and a process amount to
be dealt with than video information since a data amount of feature point information
30 of audio information for identifying content may be small.
[0045]
14
However, as the number of pieces of content to be dealt with increases, a
data amount of feature point information to be prepared in a server of the ACR
service increases. An increasing physical capacity of a database of the ACR service
that deals with audio information is a challenging task in terms of system design and
5 operations.
[0046]
With regard to the task in terms of system design, if the number of pieces of
content to be dealt with increases, a capacity of the reference feature point
information database 152 in which feature point information for identifying the
10 content is stored increases, and a process for the identification and a time taken in the
identification increase as well. In addition, with regard to the task in terms of a
system operation, if the number of pieces of content to be dealt with increases, a
capacity of the reference feature point information database 152 increases, and
investment in facilities for preparing physical databases, maintenance expenses and
15 the like increase.
[0047]
Here, a case in which feature point information of content that has a
plurality of pieces of audio information such as a main sound and a sub sound is
extracted will be considered. If the ACR service is operated during viewing of the
20 client device 140 listening to a sub sound rather than a main sound, feature point
information is extracted from the sub sound, and the client device makes an inquiry
to the information provision server 150. In this case, in order to identify the content,
it is necessary for the information provision server 150 to prepare the feature point
information not only of the main sound but also of the sub sound in the reference
25 feature point information database 152, and thus it is difficult to solve the abovementioned
task of reducing the capacity of the database. In addition, if only the
feature point information of the main sound is prepared in the reference feature point
information database 152, the ACR service may not function when the sub sound is
listened to in the client device 140.
30 [0048]
Thus, in the present embodiment, the ACR service is designed to function
15
normally during the listening of the sub sound even if the information provision
server 150 has not prepared the feature point information of the sub sound by
extracting only the feature point information of the main sound and making an
inquiry to the information provision server 150 even when the sub sound is listened
to in the client device 140. In addition, even when content 5 that has a main sound
and a plurality of pieces of audio information is distributed from the content
distribution device 110, the client device 140 makes an inquiry on feature point
information extracted from the main sound, and thus it is not necessary for the
information provision server 150 to prepare feature point information of sub sounds,
10 and the capacity of the database may not increase.
[0049]
FIG. 2 shows a configuration example of the client device 140.
[0050]
An input modulation (RF) signal input from a parabolic antenna, a cable line,
15 an STB (none of which is shown in the drawing), or the like is supplied from an
input terminal 201 as an input to a tuner unit 211 in a front-end unit 210, and selected.
A demodulation unit 212 performs, for example, QPSK (Quadrature Phase Shift
Keying) demodulation. An output from the demodulation unit 212 is supplied to an
error detection and correction unit 213. The error detection and correction unit 213
20 performs a correction process at the same time as an error is detected as much as
possible.
[0051]
Keys necessary for decrypting codes are stored in a control microcomputer
unit 214, a ROM (Read Only Memory) unit 215, a RAM (Random Access Memory)
25 unit 216, and the like together with a decryption program. When a digital signal
that has been transmitted is coded, a key for decrypting the code and a decryption
process are necessary. Thus, the control microcomputer unit 214 reads a key to
decrypt the code from the ROM unit 215, the RAM unit 216, and the like, and then
supplies the key to a demultiplexer unit (DEMUX) 217.
30 [0052]
Using the key, the demultiplexer unit 217 decrypts the coded digital signal.
16
Note that, together with the key necessary for the decryption, the decryption program,
charge information, and the like are also stored in the memories.
[0053]
The demultiplexer unit 217 receives the output signal from the error
detection and correction unit 213, and then stores the output 5 signal in a proper
memory region of the RAM unit 216 for a data buffer memory (or in a memory
region inside the demultiplexer unit 217). Then, data is appropriately read from the
RAM unit 216 then separated into a video elementary stream and an audio
elementary stream, and then the video elementary stream is supplied to an image
10 decoding unit 218 and the audio elementary stream is supplied to a sound decoding
unit 219. At this moment, information necessary for re-forming a system clock is
extracted from a multiplexed encoded data stream by a clock generation unit 220,
and then used in processes of the image decoding unit 218 and the sound decoding
unit 219.
15 [0054]
When the content has a plurality of audio channels, the audio elementary
stream extracted by the demultiplexer unit 217 is extracted according to the amount.
For example, when there are main and sub audio channels, two elementary streams
are extracted by the demultiplexer unit 217.
20 [0055]
It should be noted that the process of the demultiplexer unit 217 is
prescribed in detail in, for example, the written standard of ISO (International
Organization for Standardization)/IEC (International Electrotechnical Commission)
13818-1 (Geneva, 1995).
25 [0056]
The image decoding unit 218 appropriately stores the input encoded data in
the RAM unit 216, and executes a decoding process of an image signal that has been
compressed-encoded in the form of MPEG (Moving Picture Experts Group). The
process of the image decoding unit 218 is prescribed in detail in, for example, the
30 written standard of ISO/IEC 13818-2 (Geneva, 1995). The decoded image signal is
supplied to a display image composition unit 221. In the display image
17
composition unit 221, the signal is converted into, for example, a luminance signal
(Y), a chroma signal (C), and a composite signal (CV) in an NTSC (National
Television System Committee) system. Then, the signals are converted into analog
signals by a D/A converter unit 222, and then output. A video signal is output from
an output terminal 223 in the form of, for example, 5 an S video signal.
[0057]
The sound decoding unit 219 causes the RAM unit 216 to appropriately
store the encoded sound signal supplied from the demultiplexer unit 217, and
executes a decoding process of the sound signal that has been compressed and
10 encoded in the form of MPEG. The process of the sound decoding unit 219 is
prescribed in detail in, for example, the written standard of ISO/IEC 13818-3
(Geneva, 1995). The decoded digital sound signal is supplied to a D/A converter
unit 224 to be converted into an analog signal. At this moment, the sound signal of
the right and left channels is properly processed, and then output from an output
15 terminal 25 as a sound output of the right and left channels.
[0058]
When the content has a plurality of audio channels, the demultiplexer unit
217 extracts a plurality of audio elementary streams (as described above), and the
sound decoding unit 219 is provided with decoding sections for each of the audio
20 channels.
[0059]
The control microcomputer unit 214 executes various processes according
to the program stored in the ROM unit 215. For example, the control
microcomputer unit controls processes performed in, for example, the tuner unit 211,
25 the demodulation unit 212, the error detection and correction unit 213, and the like
via a system bus 203. In addition, the control microcomputer unit controls
processing blocks of the demultiplexer unit 217, the image decoding unit 218, the
sound decoding unit 219, and the display image composition unit 221 via the system
bus 203, and controls reading and writing processes and the like of each memory
30 address performed on the ROM unit 215 and the RAM unit 216. On the other hand,
the control microcomputer unit 214 can directly receive an input of a predetermined
18
command from a user interface unit 227 that receives input information from a
remote commander (not shown in the drawing), or the like. The user interface unit
227 inputs instructions for tuning and adjusting volume, selecting an audio channel,
operating the ACR function, and the like.
5 [0060]
An IP (Internet Protocol) interface (IP I/F) unit 230 causes the client device
140 to be connected to a broadband network such as the Internet through a wide-area
network such as Ethernet (a registered trademark), Wi-Fi, or the like. The IP
interface unit 230 is connected to the information provision server 150 at the time of,
10 for example, activation of the ACR service, to be used in transmitting feature point
information extracted from content or acquiring metadata relating to the content.
[0061]
FIG. 3 schematically illustrates an internal configuration of the
demultiplexer unit 217 and the sound decoding unit 219 corresponding to content
15 that includes a plurality of audio channels. As described above, when content input
to the client device 140 has a plurality of audio channels, the demultiplexer unit 217
extracts audio elementary streams according to the amount. In addition, the sound
decoding unit 219 is provided with the decoding sections for each of the audio
channels. Furthermore, when the content that has the plurality of audio channels is
20 viewed and listened to, switching of the audio channels which are targets to view and
listen to is input to the control microcomputer unit 214 from the user interface unit
227.
[0062]
In the present embodiment, for the purpose of low power consumption of
25 the client device 140 and speed-up of an ACR operation, a decoding process is
performed targeting both of main and sub audio channels up until the demultiplexer
unit 217, but the decoding process is performed by the sound decoding unit 219 only
on any one audio channel that is a target to be listened to, using an internal switching
(SW) function. It is assumed that, for example, identification information for
30 identifying a main or sub sound is given to the data of each audio channel, and
thereby determination of whether data is of a main sound or a sub sound can be made
19
during decoding.
[0063]
In this state, when an instruction of an operation of an ACR function is input
from the user interface unit 227, the sound decoding unit 219 executes a process of
acquiring feature point information of the main sound on data of 5 a decoding result of
a desired audio channel that has been stored in the RAM unit 216 by the
demultiplexer unit 217 under control of the control microcomputer unit 214.
[0064]
Then, the client device 140 makes an inquiry of the acquired feature point
10 information of the main sound to the information provision server 150 via the
Internet or the like from the IP interface unit 230.
[0065]
The information provision server 150 manages at least the feature point
information of the main sound of the content distributed from the content distribution
15 device 110 in the reference feature point information database 152 in association
with content identifiers. When the content is identified by checking the reference
feature point information database 152 in response to the inquiry of the feature point
information of the main sound from the client device 140, the information provision
server 150 returns a corresponding content identifier as a checking result to the client
20 device 140.
[0066]
When the client device 140 obtains the content identifier as the checking
result, the client device further makes an inquiry to the information provision server
150 based on the content identifier. Then, in response to the inquiry of the content
25 identifier, the information provision server 150 checks the metadata database 151 to
provide a user with metadata of the content, an advertisement relating to the content,
an application linked to a program, and the like.
[0067]
It should be noted that, using the content identifier by checking the
30 reference feature point information database 152, the information provision server
150 may acquire the metadata by further checking the metadata database 151 and
20
then return the metadata to the client device 140 as the checked result, rather than
returning the content identifier to the client device 140.
[0068]
The client device 140 stores, first in the RAM unit 216, the metadata, the
advertisement relating to the content, the application linked to 5 the program, and the
like received from the information provision server 150 via the IP interface unit 230.
[0069]
The control microcomputer unit 214 processes the metadata, the
advertisement relating to the content, the application linked to the program, and the
10 like which have been stored in the RAM unit 216 according to application
specifications.
[0070]
As a method of providing the metadata, and the like from the information
provision server 150, the metadata may be distributed to a mobile device associated
15 with the client device 140 such as a smartphone, in addition to being transmitted
directly to the client device 140.
[0071]
FIG. 4 shows a processing procedure for performing the operation of the
ACR function under control of the control microcomputer unit 214 in the form of a
20 flowchart.
[0072]
When a start of the ACR operation is instructed by the user interface unit
227, or the like (Yes in Step S401) while viewing and listening of content are
performed in the client device 140, the control microcomputer unit 214 checks
25 whether a main sound is being listened to (Step S402).
[0073]
Here, when the main sound is not being listened to, in other words, a sub
sound is being listened to (No in Step S402), the sound decoding unit 219 starts
decoding of a main sound channel that has been stored in the RAM unit 216 by the
30 demultiplexer unit 217 (Step S407). In addition, when the main sound is being
listened to (Yes in Step S402), the sound decoding unit 219 performs decoding of the
21
main sound channel that is a listening target.
[0074]
Then, the control microcomputer unit 214 starts a process of acquiring
feature point information on decoding result data of the main sound channel (Step
5 S403).
[0075]
When creation of the feature point information of the main sound is
completed (Yes in Step S404), the content being viewed and listened to is identified
based on the feature point information under control of the control microcomputer
10 unit 214 (Step S405). To be specific, an inquiry on the acquired feature point
information of the main sound is made to the information provision server 150 via
the Internet, or the like from the IP interface unit 230.
[0076]
The information provision server 150 manages at least the feature point
15 information of the main sound of the content distributed from the content distribution
device 110 in the reference feature point information database 152 in association
with content identifiers. In response to the inquiry on the feature point information
of the main sound from the client device 140, the information provision server 150
checks the reference feature point information database 152. When the client
20 device receives a checked result from the information provision server 150 and
thereby can identify the content (Yes in Step S406), the present process routine ends.
[0077]
When the client device can extract the feature point information only from
an audio channel being listened to, there are cases in which an audio ACR function
25 does not work with regard to the content that includes a plurality of audio channels
even though the client device is in cooperation with the information provision server
that has prepared only the feature point information of the main sound in the
database while the sub sound is listened to on the client device side. On the other
hand, in the present embodiment, the client device 140 can reliably generate the
30 feature point information of the main sound regardless of what audio channel is
being listened to, and thus the audio ACR function can be realized in cooperation
22
with the information provision server that has only prepared the feature point
information of the main sound in the database.
[0078]
In addition, since it is better for the information provision server 150 to
operate a database only for the feature point information of the 5 main sound with
regard to such content that includes a plurality of audio channels, the capacity of the
database can be suppressed to the minimum level, and accordingly, the optimum
information provision system 100 in terms of operation efficiency, operation costs,
maintenance costs, and the like can be realized.
10 [0079]
Additionally, the present technology may also be configured as below.
(1) An information processing device including:
an input unit to which content including a main sound and one or more sub
sounds is input;
15 a sound output unit configured to output a sound by decoding and
reproducing either the main sound or the one or more sub sounds included in the
input content; and
a feature point information generation unit configured to generate feature
point information from the main sound included in the input content.
20 (2) The information processing device according to (1), wherein the feature point
information generation unit executes a process of generating the feature point
information when an automatic content recognition operation is performed.
(3) The information processing device according to (1), wherein, when the sound
output unit for viewing and listening outputs a sound by decoding and reproducing
25 the sub sounds, the feature point information generation unit decodes and reproduces
the main sound when an automatic content recognition operation is performed,
thereby generating the feature point information with regard to a decoding result of
the main sound.
(4) The information processing device according to (3), wherein the decoded and
30 reproduced main sound is not output as a sound.
(5) The information processing device according to (1),
23
wherein the content input to the input unit is multiplexed with an image,
wherein the information processing device further including:
a demultiplexer unit configured to demultiplex the input content,
wherein the demultiplexer unit performs a process targeting all sounds
included 5 in the content,
wherein the sound output unit outputs a sound by decoding and reproducing
only a sound signal to be viewed and listened to among the demultiplexed main
sound and one or more sub sounds, and
wherein, when an automatic content recognition operation is performed
10 when the sub sounds are viewed and listened to, the feature point information
generation unit decodes and reproduces the demultiplexed main sound, and generates
the feature point information with regard to a decoding result of the main sound.
(6) The information processing device according to (1), further including:
a content identification unit configured to identify the input content based
15 on the feature point information generated from the main sound by the feature point
information generation unit.
(7) The information processing device according to (6), further including:
a communication interface unit configured to be connected to a network,
wherein the content identification unit checks a reference feature point
20 information database in which content identification information is associated with
the feature point information of the main sound included in the content via the
communication interface unit, thereby acquiring the content identification
information.
(8) The information processing device according to (7), further including:
25 a relevant information acquisition unit configured to acquire relevant
information of the content identified by the content identification unit by checking a
relevant information database in which relevant information of each piece of content
is stored via the communication interface unit.
(9) An information processing method including the steps of:
30 inputting content including a main sound and one or more sub sounds;
outputting a sound by decoding and reproducing either the main sound or
24
the one or more sub sounds included in the input content; and
generating feature point information from the main sound included in the
input content.
(10) An information provision device including:
a reference feature point information database in which 5 the feature point
information generated from a main sound of content including the main sound and
one or more sub sounds is associated with content identification information,
wherein the content identification information is returned in response to an
inquiry including feature point information.
10 (11) An information provision system including:
a content distribution device configured to distribute content including a
main sound and one or more sub sounds;
a client device configured to be used in viewing and listening to the content
distributed from the content distribution device with any of audio channels, and to
15 generate feature point information from the main sound; and
an information provision server configured to include a reference feature
point information database in which the feature point information generated from the
main sound of the content is associated with content identification information, and
to provide the content identification information corresponding to an inquiry of the
20 feature point information from the client device.
Industrial Applicability
[0080]
It should be understood by those skilled in the art that various modifications,
25 combinations, sub-combinations and alterations may occur depending on design
requirements and other factors insofar as they are within the scope of the appended
claims or the equivalents thereof.
[0081]
The technology disclosed in the present specification exhibits a main
30 characteristic in a process of generating feature point information from content that
includes a plurality of audio channels and identifying the content based on the feature
25
point information, and the succeeding processes to the identification of the content
are not particularly limited. Although description in the present specification
focused on an embodiment of the information provision system that distributes
metadata of content, an advertisement relating to the content, an application linked to
a program, and the like to a client such as a TV receiver set or a 5 mobile device such
as a smartphone, the technology disclosed in the present specification can be used in
various services which use the ACR function such as content right control.
[0082]
In short, the present invention has been disclosed in a form of illustration
10 and should not be interpreted limitedly. To determine the gist of the technology
disclosed in the present embodiment, patent claims should be taken into account.
Reference Signs List
[0083]
15 100 information provision system
110 content distribution device
120 relay station
130 set-top box
140 client device
20 150 information provision server
151 metadata database
152 reference feature point information database
201 input terminal
203 system bus
25 210 front-end unit
211 tuner unit
212 demodulation unit
213 error detection and correction unit
214 control microcomputer unit
30 215 ROM unit
216 RAM unit
26
217 demultiplexer unit
218 image decoding unit
219 sound decoding unit
220 clock generation unit
221 display image 5 composition unit
222 D/A converter unit
223 output terminal
224 D/A converter unit
225 output terminal
10 227 user interface unit
230 IP interface unit
27
CLAIMS
Claim 1
An information processing device comprising:
an input unit to which content including a main sound and one or more sub
5 sounds is input;
a sound output unit configured to output a sound by decoding and
reproducing either the main sound or the one or more sub sounds included in the
input content; and
a feature point information generation unit configured to generate feature
10 point information from the main sound included in the input content.
Claim 2
The information processing device according to claim 1, wherein the feature
point information generation unit executes a process of generating the feature point
15 information when an automatic content recognition operation is performed.
Claim 3
The information processing device according to claim 1, wherein, when the
sound output unit for viewing and listening outputs a sound by decoding and
20 reproducing the sub sounds, the feature point information generation unit decodes
and reproduces the main sound when an automatic content recognition operation is
performed, thereby generating the feature point information with regard to a
decoding result of the main sound.
25 Claim 4
The information processing device according to claim 3, wherein the
decoded and reproduced main sound is not output as a sound.
Claim 5
30 The information processing device according to claim 1,
wherein the content input to the input unit is multiplexed with an image,
28
wherein the information processing device further comprising:
a demultiplexer unit configured to demultiplex the input content,
wherein the demultiplexer unit performs a process targeting all sounds
included in the content,
wherein the sound output unit outputs a sound by decoding 5 and reproducing
only a sound signal to be viewed and listened to among the demultiplexed main
sound and one or more sub sounds, and
wherein, when an automatic content recognition operation is performed
when the sub sounds are viewed and listened to, the feature point information
10 generation unit decodes and reproduces the demultiplexed main sound, and generates
the feature point information with regard to a decoding result of the main sound.
Claim 6
The information processing device according to claim 1, further comprising:
15 a content identification unit configured to identify the input content based
on the feature point information generated from the main sound by the feature point
information generation unit.
Claim 7
20 The information processing device according to claim 6, further comprising:
a communication interface unit configured to be connected to a network,
wherein the content identification unit checks a reference feature point
information database in which content identification information is associated with
the feature point information of the main sound included in the content via the
25 communication interface unit, thereby acquiring the content identification
information.
Claim 8
The information processing device according to claim 7, further comprising:
30 a relevant information acquisition unit configured to acquire relevant
information of the content identified by the content identification unit by checking a
29
relevant information database in which relevant information of each piece of content
is stored via the communication interface unit.
Claim 9
An information processing method comprising 5 the steps of:
inputting content including a main sound and one or more sub sounds;
outputting a sound by decoding and reproducing either the main sound or
the one or more sub sounds included in the input content; and
generating feature point information from the main sound included in the
10 input content.
Claim 10
An information provision device comprising:
a reference feature point information database in which the feature point
15 information generated from a main sound of content including the main sound and
one or more sub sounds is associated with content identification information,
wherein the content identification information is returned in response to an
inquiry including feature point information.
20 Claim 11
An information provision system comprising:
a content distribution device configured to distribute content including a
main sound and one or more sub sounds;
a client device configured to be used in viewing and listening to the content
25 distributed from the content distribution device with any of audio channels, and to
generate feature point information from the main sound; and
an information provision server configured to include a reference feature
point information database in which the feature point information generated from the
main sound of the content is associated with content identification information, and
30 to provide the content identification information corresponding to an inquiry of the
feature point information from the client device.
30