FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENTS RULES, 2003
COMPLETE SPECIFICATION
(See section 10, rule 13)
“INFORMATION PROCESSING APPARATUS, INFORMATION
PROCESSING METHOD, AND INFORMATION PROCESSING
PROGRAM”
SONY CORPORATION, of 1-7-1, Konan, Minato-ku,
Tokyo 108-0075, Japan
The following specification particularly describes the invention and the manner in which it is to
be performed.

2
Description
INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING
METHOD, AND INFORMATION PROCESSING PROGRAM
5 Field
[0001] The present disclosure relates to an information
processing apparatus, an information processing method, and
an information processing program. Specifically, it
relates to a generation process of a trained model that is
10 generated through machine learning.
Background
[0002] Information processing using machine learning has
been utilized in various technical fields. For example,
15 the feature of content (image, music, or the like) is
learnt by using a neural network representing the mechanism
of a cranial nervous system so that, for example, new
content is automatically generated.
[0003] For example, there is a disclosure of the
20 technique in which the learning of the feature of the
existing song makes it possible to automatically compose an
appropriate song suitable for the lyrics without parameters
input by the user other than the lyrics.
25 Citation List
Patent Literature
[0004] Patent Literature 1: Japanese Laid-open Patent
Publication No. 2011-175006
30 Summary
Technical Problem
[0005] According to a conventional art, the linguistic
feature value calculated from the lyrics data indicating
3
the lyrics of each song and the attribute data indicating
the attribute of the song are learnt and therefore, when
new lyrics data is provided, the song suitable for the new
lyrics data may be automatically generated.
[0006] According to the conventional 5 art, however, a
melody or a code are simply generated in accordance with
the lyrics and, in terms of the structure (structure) of
the entire song, a natural song like the one composed by a
human is not always generated. That is, according to the
10 conventional art, it is difficult to learn the feature of
the overall structure of the content or to generate new
content while maintaining the overall structure.
[0007] Therefore, the present disclosure provides an
information processing apparatus, an information processing
15 method, and an information processing program with which it
is possible to learn the feature of the overall structure
of content.
Solution to Problem
20 [0008] To solve the problem described above, an
information processing apparatus includes: an acquiring
unit that uses a trained encoder to acquire a feature value
of each set of partial data forming first content; a
calculating unit that calculates a relative feature value
25 that is a relative feature value between sets of partial
data from acquired feature values of the sets of partial
data to calculate a relative feature value sequence
indicating a feature of a structure of the first content;
and a generating unit that generates second content based
30 on the relative feature value sequence of the first content
and a feature value of optional data.
Brief Description of Drawings
4
[0009] FIG. 1 is a diagram illustrating an example of
information processing according to an embodiment of the
present disclosure.
FIG. 2 is a diagram illustrating an example of the
configuration of an information processing 5 apparatus
according to an embodiment of the present disclosure.
FIG. 3 is a diagram illustrating an example of a song
data storage unit according to an embodiment of the present
disclosure.
10 FIG. 4 is a flowchart (1) illustrating the steps of
information processing according to an embodiment of the
present disclosure.
FIG. 5 is a flowchart (2) illustrating the steps of
information processing according to an embodiment of the
15 present disclosure.
FIG. 6 is a hardware configuration diagram
illustrating an example of a computer that performs a
function of the information processing apparatus.
20 Description of Embodiments
[0010] Embodiments of the present disclosure are
described below in detail with reference to the drawings.
Furthermore, in each of the following embodiments, the same
components are denoted by the same reference numeral, and
25 duplicated descriptions are omitted.
[0011] The present disclosure is described in accordance
with the order of sections described below.
1. Embodiment
1-1. Outline of information processing according to an
30 embodiment of the present disclosure
1-2. Configuration of an information processing
apparatus according to an embodiment
1-3. Steps of information processing according to an
5
embodiment
2. Other embodiments
3. Hardware configuration
[0012] (1. Embodiment)
[1-1. Outline of information processing 5 according to
an embodiment of the present disclosure]
FIG. 1 is a diagram illustrating an example of
information processing according to an embodiment of the
present disclosure. The information processing according
10 to the embodiment of the present disclosure is performed by
an information processing apparatus 100 illustrated in FIG.
1.
[0013] The information processing apparatus 100 is an
apparatus that performs the information processing
15 according to the present disclosure and is, for example, an
information processing terminal or a server device.
[0014] According to the embodiment, the information
processing apparatus 100 includes a trained model for
extracting the feature of content. According to the
20 embodiment, the content includes digital data in a
predetermined format, e.g., music (song), an image, or a
moving image. In the example illustrated in FIG. 1, the
information processing apparatus 100 uses a song as an
example of the content for processing.
25 [0015] The trained model according to the embodiment
includes: an encoder (encoder) that extracts a feature
value from the data forming the content; and a decoder
(decoder) that reconstructs the content from the extracted
feature value. For example, the information processing
30 apparatus 100 trains the encoder through unsupervised
learning, such as VAE (Variational Auto Encoder) or GAN
(Generative Adversarial Networks). Specifically, the
information processing apparatus 100 inputs the content to
6
the encoder, reconstructs the content from the extracted
feature value, compares the original content with the
reconstructed content, and adjusts the parameters of the
encoder and the decoder. The information processing
apparatus 100 repeats this process 5 to optimize the
parameters of the encoder and the decoder so as to generate
the trained model. This allows the information processing
apparatus 100 to generate the trained model that may obtain
an appropriate feature value from the data forming the
10 content. Furthermore, the feature value is represented by
using, for example, a vector having a lower dimensional
number as compared with the data on the input content.
[0016] Furthermore, the trained model is not limited to
the above example and may be a model of any type as long as
15 a feature value may be extracted and the content may be
reconstructed from the extracted feature value.
[0017] In the example of FIG. 1, the information
processing apparatus 100 inputs a song (in other words, the
digital data representing the sound forming the song) to
20 the encoder to extract, as the feature value of the song,
the element representing the feature of the song, such as
melody line, constituent sound, rhythm (the temporal
structure of music, e.g., the number of notes and rests
included, or the order of sounds to be reproduced), tempo,
25 or beat. Moreover, according to the embodiment, the data
on the song is represented by using the data (e.g., vector
format) indicating the pitch (the data indicating the level
of the sound to be produced), the sound length (the data
indicating how long the produced note is to be retained),
30 and the timing of a rest.
[0018] Here, for the automatic generation of a new song
using the feature value of the existing song as described
above, one of the challenges may be to generate a song
7
having a natural structure like a human-made existing song.
However, extracting the feature value of an optional song
without change leads to the difficulty of generating a song
having a natural structure as a whole although it is
possible to reproduce the feature 5 of the song.
Specifically, even though the similar characteristic melody
line or sound structure (also referred to as motif), which
is a part of the song, is reproduced in an automatically
generated song, it is difficult to reproduce the structure
10 of the song itself, such as how the feature of the motif
changes in the entire song. That is, it is conventionally
difficult to automatically generate a song that has a
natural structure like a human-made existing song and that
has a certain length like an existing song.
15 [0019] Thus, through the information processing
described below, the information processing apparatus 100
according to the present disclosure makes it possible to
calculate the feature value indicating the structure of the
entire song and to automatically generate a natural song by
20 using the feature value. Specifically, the information
processing apparatus 100 uses the trained encoder described
above to acquire the feature value of each set of partial
data (e.g., each bar) forming the song. Further, the
information processing apparatus 100 calculates a relative
25 feature value, which is a relative feature value between
the sets of partial data, from the feature value of each
set of partial data so as to calculate the relative feature
value sequence indicating the feature of the structure of
the song. As described above, the information processing
30 apparatus 100 regards the song having a certain length as
the sequence of partial data arranged, calculates the
relative feature value of them to obtain the feature value
that transitions in the entire song, and then expresses the
8
structure of the entire song. In other words, the
information processing apparatus 100 models the structure
of the entire song by using the relative feature value
sequence. Then, the information processing apparatus 100
uses the relative feature value sequence 5 indicating the
feature of the structure of the entire song to generate a
new song. Thus, the information processing apparatus 100
may automatically generate a new song that maintains the
feature of the structure of the existing song and has a
10 natural structure. Furthermore, the sequence of feature
values is the sequence of sequentially arranged feature
values of the respective sets of partial data forming the
song.
[0020] With reference to FIG. 1, the outline of the
15 information processing according to the present disclosure
is sequentially described below. Furthermore, in the
following description, the target content from which a
feature value is calculated is referred to as "first
content", and the new content generated based on the
20 relative feature value sequence of the first content is
sometimes referred to as "second content". Further, the
trained encoder illustrated in FIG. 1, and the like, is
simply referred to as an "encoder". Moreover, the trained
decoder is simply referred to as a "decoder".
25 [0021] As illustrated in FIG. 1, the information
processing apparatus 100 first acquires a song 30 as the
first content and divides the song 30 into partial data
(Step S1). For example, the information processing
apparatus 100 divides the song 30 on a per bar basis.
30 Although the example in FIG. 1 illustrates the song 30
having six bars, the song 30 may have more bars.
[0022] The song 30 includes the symbol string (digital
data) indicating, for example, a pitch, a sound length, or
9
a rest. For example, the pitch represents the frequency
indicating the level of a sound by using predetermined
levels (e.g., 128 levels). Further, the sound length
represents the length at which the reproduced sound is
retained. Furthermore, the rest represents 5 the timing at
which the sound reproduction is stopped. Moreover, the
data representing the song 30 may include information such
as the symbol indicating the beat or the tempo of the song
30 or the break of a bar, a code at a certain timing, or a
10 constituent sound forming a code.
[0023] For example, the information processing apparatus
100 converts the above-described symbol string so as to be
manipulated in a model. For example, the information
processing apparatus 100 uses the vector representation
15 (e.g., an embedding (embedding) vector having a pitch, a
sound length, or the like, assigned to each dimension and
representing sound information) of the above-described
symbol string for processing. The embedding vector is, for
example, a d-dimensional (d is any integer) vector in which
20 "1" is input to the dimension that corresponds to the
corresponding sound (e.g., information such as "C4"
indicating the level of the sound) and "0" is input to the
other dimensions. Furthermore, the data representing the
sound may be expressed in the MIDI (Musical Instrument
25 Digital Interface) (registered trademark) format, may be
digital data having a known format and reproducible by a
general-purpose sequencer, or may be represented as
waveform data such as the WAV format.
[0024] The information processing apparatus 100 may use
30 various known techniques to divide the song 30 into partial
data. For example, the information processing apparatus
100 detects the pre-set break of a bar to divide the song
30 into partial data (bars). Alternatively, the
10
information processing apparatus 100 may divide the song 30
into partial data based on the relationship between a beat
and a note. For example, in the case of the song 30 with
4/4 time signature, the information processing apparatus
100 detects, as a break, the time when 5 the four sounds
having the length corresponding to a quarter note are
reproduced so as to divide the song 30 into bars.
[0025] Furthermore, the information processing apparatus
100 may use a break other than a bar for partial data. For
10 example, the information processing apparatus 100 may
detect the break (e.g., the position where the rest having
a length more than a predetermined threshold appears) of a
melody line of the song 30 to divide it into partial data.
In this case, the partial data does not necessarily need to
15 match with a bar.
[0026] The information processing apparatus 100 divides
the song 30 into bars at Step S1 and extracts each set of
partial data (Step S2). In the example of FIG. 1, each set
of partial data is illustrated as "xn (n is any natural
20 number)". For example, "x1" indicates the data included in
a first bar of the song 30.
[0027] Subsequently, the information processing
apparatus 100 sequentially inputs the set of extracted
partial data to an encoder 50 (Step S3). Accordingly, the
25 information processing apparatus 100 acquires the feature
value of each set of partial data (Step S4).
[0028] In the example of FIG. 1, the feature value of
each set of partial data is denoted by "zn". For example,
"z1" indicates the feature value of the first bar of the
30 song 30.
[0029] Here, the information processing apparatus 100
calculates a relative feature value that is a relative
feature value between the sets of partial data from the
11
acquired feature value of each set of partial data (Step
S5). For example, as the relative feature value between
the sets of partial data, the information processing
apparatus 100 subtracts the feature value of certain common
partial data from the feature value of each 5 set of partial
data to calculate a relative feature value.
[0030] In the example of FIG. 1, the information
processing apparatus 100 subtracts "z1", which is the
feature value of the first bar, from the feature value of
10 each set of partial data to calculate the relative feature
value. Furthermore, this calculation is an example, and
the information processing apparatus 100 may perform
addition, multiplication, or division on a feature value to
calculate a relative feature value. Furthermore, the
15 information processing apparatus 100 may subtract the
partial data of a different bar from each set of partial
data instead of "z1".
[0031] The information processing apparatus 100
calculates a relative feature value sequence, which is the
20 sequence of feature values indicating the structure of the
song 30, based on the calculated relative feature values
(Step S6). The relative feature value sequence is, for
example, the sequence of sequentially arranged relative
feature values such that they correspond to the structure
25 of the song 30. Furthermore, in the example of FIG. 1, the
relative feature value corresponding to "zn+1-z1" is
indicated by "rn". That is, in the example of FIG. 1, the
relative feature value sequence corresponding to the song
30 is represented as "(r1, r2, r3, r4, r5)".
30 [0032] As above, during the process up to Step S6, the
information processing apparatus 100 calculates the
relative feature value sequence indicating the structure of
the song 30. Then, the information processing apparatus
12
100 acquires optional information that is the motif of the
new song (the second content) to be generated.
[0033] For example, the information processing apparatus
100 acquires the data included in a first bar of a
predetermined song 35 as the information 5 that is the motif
of the new song to be generated (Step S7). Furthermore, as
the information that is the motif of the new song to be
generated, the information processing apparatus 100 does
not always need to acquire the data included in one bar but
10 may acquire, for example, the data on the entire optional
song.
[0034] In the example of FIG. 1, the acquired optional
data (i.e., the data included in the first bar of the song
35) is denoted by "xa".
15 [0035] Subsequently, the information processing
apparatus 100 inputs "xa" to the encoder 50 (Step S8).
Thus, the information processing apparatus 100 acquires the
feature value corresponding to "xa". In the example of
FIG. 1, the feature value corresponding to "xa" is denoted
20 by "za".
[0036] Then, the information processing apparatus 100
generates the feature value sequence corresponding to the
newly generated second content based on the acquired
feature value "za" and the relative feature values
25 indicating the structure of the song 30 (Step S9). For
example, the information processing apparatus 100 generates
the feature value sequence in which "za" is at the
beginning and "za" is attached to each of the relative
feature values of the song 30. Specifically, the
30 information processing apparatus 100 generates the feature
value sequence having the information such as "(za, za+r1,
za+r2, za+r3, za+r4, za+r5)". That is, the information
processing apparatus 100 generates the feature value
13
sequence in which "za" is further attached to the feature
of the structure of the song 30.
[0037] The information processing apparatus 100 inputs
the feature value sequence generated at Step S9 to the
decoder 60 (Step S10). The decoder 60 is 5 a decoder that is
trained to reconstruct content based on the feature value
extracted by the encoder 50. In the example of FIG. 1, the
decoder 60 reconstructs a song (accurately, the digital
data for reproducing a sound) based on the feature value
10 extracted by the encoder 50.
[0038] The information processing apparatus 100 acquires
the sound data on each bar corresponding to the feature
value sequence from the output of the decoder 60. The
information processing apparatus 100 arranges the acquired
15 data in the order of the sequence to generate a song 40
(Step S11). The song 40 is a song that uses the first bar
of the song 35 as a motif and also maintains the feature of
the structure of the song 30.
[0039] Thus, the information processing apparatus 100
20 according to the present disclosure uses the encoder 50 to
acquire the feature value of each set of partial data
forming the first content (the song 30 in the example of
FIG. 1). Then, the information processing apparatus 100
calculates the relative feature value, which is the
25 relative feature value between the sets of partial data,
from the acquired feature value of each set of partial data
to calculate the relative feature value sequence indicating
the feature of the structure of the first content. That
is, the information processing apparatus 100 acquires the
30 sequence in which the feature values of the partial data
forming the first content are arranged in order instead of
calculating the feature value of the first content itself.
Thus, the information processing apparatus 100 may extract
14
the structure of the first content (the flow, the high
point, or the like, of the entire song corresponding to the
time direction if the first content is a song) as a
feature.
[0040] Furthermore, the information processing 5 apparatus
100 according to the present disclosure generates the
second content (the song 40 in the example of FIG. 1) based
on the relative feature value sequence of the first content
and the feature value ("za" that is the feature value of
10 the first bar of the song 35 in the example of FIG. 1) of
optional data. That is, the information processing
apparatus 100 generates a new feature value sequence based
on the feature value sequence indicating the structure of
the first content and the feature value of the new data and
15 reconstructs the content based on the new feature value
sequence. Thus, the information processing apparatus 100
may generate, as the second content, a new song that
maintains the structure of the first content and adopts a
new motif or constituent sound. Furthermore, in the
20 example illustrated in FIG. 1, the information processing
apparatus 100 subtracts the feature value "z1" from the
feature value of each set of partial data to calculate a
relative feature value. However, this example is not a
limitation, and the information processing apparatus 100
25 may calculate a relative feature value by using a feature
value extractor that extracts a feature value like
addition, subtraction, multiplication, division, or
correlation. Furthermore, the information processing
apparatus 100 may extract a graph structure based on the
30 similarity or the causality of the feature value of each
set of partial data and calculate a relative feature value
sequence by using a machine learning technique, or the
like, for graph data.
15
[0041] [1-2. Configuration of the information processing
apparatus according to an embodiment]
Next, the configuration of the information processing
apparatus 100, which is an example of an information
processing apparatus that performs information 5 processing
according to an embodiment, is described. FIG. 2 is a
diagram illustrating an example of the configuration of the
information processing apparatus 100 according to an
embodiment of the present disclosure.
10 [0042] As illustrated in FIG. 2, the information
processing apparatus 100 includes a communication unit 110,
a storage unit 120, and a control unit 130. Furthermore,
the information processing apparatus 100 may include an
input unit (for example, a keyboard or a mouse) that
15 receives various operations from an administrator, or the
like, who manages the information processing apparatus 100
and a display unit (for example, a liquid crystal display)
that displays various kinds of information.
[0043] The communication unit 110 is implemented by
20 using, for example, an NIC (Network Interface Card). The
communication unit 110 is connected to a network N (such as
the Internet) via wired or wireless communications to
transmit/receive information to/from other devices, or the
like, via the network N.
25 [0044] The storage unit 120 is implemented by using, for
example, a semiconductor memory device such as a RAM
(Random Access Memory) or a flash memory (flash memory), or
a storage device such as a hard disk or an optical disk.
The storage unit 120 includes a model storage unit 121 and
30 a song data storage unit 122.
[0045] The model storage unit 121 stores a trained model
that has been trained in advance. Specifically, the model
storage unit 121 includes the encoder 50 that extracts a
16
feature value of the content and the decoder 60 that
reconstructs the content. Furthermore, the model storage
unit 121 may store training data such as the content used
for training.
[0046] The song data storage unit 122 5 stores data about
the content (song) input to the model. FIG. 3 illustrates
an example of the song data storage unit 122 according to
an embodiment. FIG. 3 is a diagram illustrating an example
of the song data storage unit 122 according to an
10 embodiment of the present disclosure. In the example
illustrated in FIG. 3, the song data storage unit 122 has
items such as "song ID", "partial data ID", "pitch
information", "sound-length rest information", "code
information", and "rhythm information".
15 [0047] The "song ID" is the identification information
for identifying a song. The "partial data ID" is the
identification information for identifying partial data.
The partial data corresponds to, for example, one or more
bars forming a song.
20 [0048] The "pitch information" indicates the information
on the pitch (scale) of the sound included in partial data.
The "sound-length rest information" indicates the length
(the reproduction time period or the number of beats to be
reproduced) of a sound included in partial data, and the
25 length and the timing of a rest. The "code information"
indicates the type of code included in partial data, the
constituent sound of a code, the switching of a code within
a bar, and the like. The "rhythm information" indicates a
beat or a tempo in a bar, the positions of a strong beat
30 and a weak beat, and the like.
[0049] Although FIG. 3 has the conceptual description in
which the item such as the pitch information is "C01", each
item actually stores the specific data indicating the sound
17
as described above. Although FIG. 3 illustrates the
example in which "pitch information", "sound-length rest
information", and the like, are stored as different items
for the purpose of description, these sets of information
may be collectively stored in one item, or 5 the like, as the
information indicating a note included in the bar. That
is, the data format representing the song is not limited to
that illustrated in FIG. 3 and may be any format that may
be manipulated in the model.
10 [0050] For example, the example illustrated in FIG. 3
indicates that the song identified with the song ID "A01"
includes the partial data identified with the partial data
IDs "B01" and "B02". Furthermore, it is indicated that the
partial data with the partial data ID "B01" includes the
15 sound data represented with the pitch information "C01",
the sound-length rest information "D01", the code
information "E01", and the rhythm information "F01".
[0051] With reference back to FIG. 2, the description is
continued. The control unit 130 is implemented when, for
20 example, a CPU (Central Processing Unit) or an MPU (Micro
Processing Unit) uses a RAM (Random Access Memory), or the
like, as a work area and executes a program (e.g., an
information processing program according to the present
disclosure) stored in the information processing apparatus
25 100. Moreover, the control unit 130 is a controller
(controller) and may be implemented by using an integrated
circuit such as an ASIC (Application Specific Integrated
Circuit) or an FPGA (Field Programmable Gate Array).
[0052] As illustrated in FIG. 2, the control unit 130
30 includes a training unit 131, an acquiring unit 132, a
calculating unit 133, and a generating unit 134 to
implement or perform the functions and the operation for
the information processing described below. Furthermore,
18
the internal configuration of the control unit 130 is not
limited to the configuration illustrated in FIG. 2 and may
be a different configuration as long as the configuration
enables the information processing described later.
[0053] The training unit 131 uses content 5 as training
data to perform a predetermined training process so as to
generate a trained model.
[0054] For example, to learn the sound data on a song,
or the like, the training unit 131 inputs the sound data to
10 the encoder 50 to extract the feature value of the sound
data. Subsequently, the training unit 131 inputs the
feature value of the sound data to the decoder 60 to
reconstruct the sound data that has been input to the
encoder 50. Then, the training unit 131 adjusts the
15 parameters of the encoder 50 and the decoder 60 so as to
reduce the difference between the successive sets of sound
data. The training unit 131 repeats this process to
generate a trained model with the encoder 50 and the
decoder 60 that have been optimized. As described above,
20 the training unit 131 may generate a model by using various
known techniques such as VAE or GAN.
[0055] The acquiring unit 132 acquires various types of
information. For example, the acquiring unit 132 acquires
the first content to be input to the model trained by the
25 training unit 131.
[0056] Further, the acquiring unit 132 divides the
acquired first content to acquire the partial data forming
the first content. For example, when the first content is
a song, the acquiring unit 132 detects the break of a bar
30 of the song by using the above-described technique and sets
the detected bar as partial data.
[0057] Alternatively, the acquiring unit 132 may detect
a rest that is more than the length of a predetermined
19
threshold in the first content and divide the song into
partial data based on the detected rest. In this case, the
length of the predetermined threshold may be a time length,
the percentage of the rest occupied in the bar, or the
5 like.
[0058] Furthermore, the acquiring unit 132 uses the
encoder 50 trained by the training unit 131 to acquire the
feature value of each set of partial data forming the first
content.
10 [0059] Specifically, the acquiring unit 132 inputs the
symbol string indicating the pitch, the sound length, and
the rest to the encoder 50 as the data indicating the sound
included in the partial data to acquire the feature value
corresponding to the partial data. The feature value is
15 represented as, for example, a vector having a lower
dimensional number as compared with the vector representing
the original partial data.
[0060] The calculating unit 133 calculates a relative
feature value, which is a relative feature value between
20 the sets of partial data, from the feature value of each
set of partial data acquired by the acquiring unit 132 to
calculate a relative feature value sequence indicating the
feature of the structure of the first content.
[0061] For example, the calculating unit 133 executes
25 addition, subtraction, multiplication, or division on the
feature value of certain partial data of the first content
and the feature value of each set of partial data other
than the certain partial data to calculate a relative
feature value so as to calculate a relative feature value
30 sequence. The certain partial data of the first content
indicates specific partial data in the partial data forming
the first content and, in the example of FIG. 1,
corresponds to the feature value "z1" of the first bar of
20
the song 30. Thus, the calculating unit 133 performs an
operation such as addition, subtraction, multiplication, or
division on a plurality of sets of partial data with
specific partial data to calculate a relative feature value
having 5 a relationship.
[0062] Furthermore, FIG. 1 illustrates the example where
the feature value "z1" of the first bar of the song 30 is
subtracted from the feature value of each set of partial
data of the song 30; however, the operation is not limited
10 to this example. For example, the calculating unit 133 may
calculate a relative feature value between the feature
value of certain partial data on the first content and the
feature value of each set of partial data that is other
than the certain partial data and has a causal relationship
15 with the certain partial data so as to calculate a relative
feature value sequence.
[0063] Here, the partial data having a causal
relationship with certain partial data refers to the
partial data having some kind of correspondence
20 relationship with the certain partial data. For example,
if the content is a song, there may be a bar corresponding
to a certain bar (for example referred to as a call-andresponse
relationship) in the song. In this case, as
described above, the calculating unit 133 may
25 preferentially perform the addition (or subtraction)
operation on the feature values of the bars having some
kind of causal relationship so as to emphasize or deemphasize
the structure of the song. Furthermore, the
causal relationship between the sets of partial data of the
30 content may be obtained by using a technique in which a
song is analyzed by using a known machine learning
technique such as causal inference and the relationship
between a certain bar and a factor bar is quantified.
21
[0064] Furthermore, the calculating unit 133 may
calculate a relative feature value by using various known
techniques, such as generating, as a graph, the partial
data in the content based on a predetermined relationship
and adding or subtracting the sets of partial 5 data having a
close relationship in the graph.
[0065] As described above, the calculating unit 133
calculates the relative feature value that is the relative
feature value between the sets of partial data by using
10 various techniques and sequentially arranges the calculated
relative feature values to calculate the relative feature
value sequence indicating the feature of the structure of
the first content. The calculating unit 133 sends the
calculated relative feature value to the generating unit
15 134.
[0066] The generating unit 134 generates the second
content based on the relative feature value sequence of the
first content and the feature value of optional data.
[0067] For example, the generating unit 134 calculates a
20 new feature value sequence based on the relative feature
value sequence of the first content calculated by the
calculating unit 133 and the feature value of optional
data. Then, the generating unit 134 inputs each feature
value included in the new feature value sequence to the
25 decoder 60 and reconstructs the partial data corresponding
to each feature value. Further, the generating unit 134
combines the reconstructed partial data in order of the
sequence to generate the second content that is new
content.
30 [0068] When the relative feature value sequence of the
first content, which is a song, has been acquired, the
generating unit 134 may generate an optional new song as
the second content. In this case, the acquiring unit 132
22
inputs the symbol string indicating the pitch, the sound
length, and the rest to the encoder 50 as the data
indicating the sound included in the partial data and the
optional data to acquire the feature value corresponding to
the partial data and the optional data. Then, 5 based on the
feature value acquired by the acquiring unit 132, the
calculating unit 133 calculates the relative feature value
sequence corresponding to each set of partial data. The
generating unit 134 generates a new feature value sequence
10 based on the relative feature value sequence calculated by
the calculating unit 133 and the feature value
corresponding to optional data (for example, the sound data
that is the motif of a new content) and generates the
second content, which is a song, from the generated new
15 feature value sequence.
[0069] [1-3. Steps of the information processing
according to an embodiment]
Next, the steps of the information processing
according to an embodiment are described with reference to
20 FIG. 4 and FIG. 5. First, the flow of a training process
according to an embodiment of the present disclosure is
described with reference to FIG. 4. FIG. 4 is a flowchart
(1) illustrating the steps of the information processing
according to an embodiment of the present disclosure.
25 [0070] As illustrated in FIG. 4, the information
processing apparatus 100 determines whether the training
data (content) has been acquired (Step S101). When no
training data has been acquired (Step S101: No), the
information processing apparatus 100 stands by until
30 training data is acquired.
[0071] Conversely, when training data has been acquired
(Step S101; Yes), the information processing apparatus 100
generates a model using the training data (Step S102).
23
Then, the information processing apparatus 100 stores the
trained model (the encoder and the decoder) in the storage
unit 120 (Step S103).
[0072] Next, the flow of a generation process according
to an embodiment of the present disclosure 5 is described
with reference to FIG. 5. FIG. 5 is a flowchart (2)
illustrating the steps of the information processing
according to an embodiment of the present disclosure.
[0073] As illustrated in FIG. 5, the information
10 processing apparatus 100 determines whether the first
content has been acquired (Step S201). When the first
content has not been acquired (Step S201; No), the
information processing apparatus 100 stands by until the
first content is acquired.
15 [0074] Conversely, when the first content has been
acquired (Step S201; Yes), the information processing
apparatus 100 divides the first content into partial data
(Step S202). For example, when the first content is a
song, the information processing apparatus 100 divides the
20 song into partial data on each bar.
[0075] Subsequently, the information processing
apparatus 100 inputs each set of partial data to the
encoder 50 to calculate the feature value of each set of
partial data (Step S203). Further, the information
25 processing apparatus 100 performs a predetermined operation
on the feature values of the sets of partial data to
calculate a relative feature value (Step S204).
[0076] Based on the relative feature value, the
information processing apparatus 100 calculates the
30 relative feature value sequence indicating the feature of
the structure of the first content (Step S205).
[0077] Then, the information processing apparatus 100
determines whether optional data (e.g., one bar of a
24
certain song), which is the motif, or the like, of the
second content to be generated, has been acquired (Step
S206). When no optional data has been acquired (Step S206;
No), the information processing apparatus 100 stands by
until optional 5 data is acquired.
[0078] Conversely, when optional data has been acquired
(Step S206; Yes), the information processing apparatus 100
inputs the optional data to the encoder 50 to calculate the
feature value of the optional data (Step S207).
10 [0079] Subsequently, the information processing
apparatus 100 calculates a new feature value sequence,
which is the source of the second content, based on the
relative feature value sequence of the first content and
the feature value of the optional data (Step S208).
15 [0080] Then, the information processing apparatus 100
inputs the new feature value sequence to the decoder 60 to
generate the second content from the new feature value
sequence (Step S209).
[0081] (2. Other embodiments)
20 The processes according to each of the above-described
embodiments may be implemented in various different
embodiments other than the embodiments described above.
[0082] According to the above-described embodiment,
examples of the content include song (music). However, the
25 information processing apparatus 100 may perform the
information processing according to the present disclosure
using text data, a moving image, or the like, as the
content.
[0083] For example, the information processing apparatus
30 100 acquires the feature value of each set of partial data
forming the first content that is text data. Then, the
information processing apparatus 100 generates the second
content, which is text data, on the basis of the relative
25
feature value sequence of the first content obtained based
on the acquired feature value of each set of partial data
and the feature value of optional data, which is text data.
In this case, it is assumed that the information processing
apparatus 100 generates the trained model 5 for outputting
the feature value of text data.
[0084] In this case, examples of the text data include a
poem or a thirty-one-syllable poem. For example, the
information processing apparatus 100 acquires the poem
10 including a plurality of sentences (for example, a text
divided by a line feed symbol). Then, the information
processing apparatus 100 detects a line feed symbol
included in the poem to divide the poem into partial data
(a sentence of each row). Subsequently, the information
15 processing apparatus 100 calculates the relative feature
value of the partial data to obtain the relative feature
value sequence corresponding to the poem.
[0085] When new optional data (for example, the desired
phrase, sentence, or the like, to be included in the poem
20 by the user) is acquired, the information processing
apparatus 100 calculates the feature value of the optional
data and calculates a new feature value sequence based on
the calculated feature value and the relative feature value
sequence corresponding to the poem. Then, the information
25 processing apparatus 100 generates a new poem as the second
content from the new feature value sequence.
[0086] This allows the information processing apparatus
100 to generate a new poem that has a structure (e.g., the
number of characters at the break (rhythm), a phoneme, or
30 the expression of a phrase appearing in the next sentence)
similar to that of the existing poem.
[0087] Furthermore, the information processing apparatus
100 may generate new moving image content based on moving
26
image content instead of text data. In this case, it is
assumed that the information processing apparatus 100
generates the trained model that outputs the feature value
of the image forming the moving image content.
[0088] For example, the information processing 5 apparatus
100 acquires the feature value of each set of partial data
forming the first content that is moving image content. In
this case, the partial data is, for example, the image
corresponding to each frame forming the moving image
10 content. Furthermore, the partial data may be, for
example, the image data obtained by collecting and
averaging several frames as well as a single still image.
Then, the information processing apparatus 100 generates
the second content, which is moving image content, based on
15 the relative feature value sequence of the first content
obtained based on the acquired feature value of each set of
partial data and the feature value of optional data, which
is moving image or image content.
[0089] This allows the information processing apparatus
20 100 to generate new moving image content having a structure
similar to that of the existing moving image content. For
example, on the basis of the moving image content that
captures a series of motions of a blooming flower, the
moving image content that captures the moving image of a
25 walking person, or the like, the information processing
apparatus 100 generates new moving image content of a
different object performing a similar action. Thus, the
information processing apparatus 100 may automatically
generate various types of new content that have a similar
30 content structure in whole as well as songs through the
information processing according to the present disclosure.
[0090] Further, the information processing apparatus 100
may apply the information processing according to the
27
present disclosure so as to perform the process to detect,
for example, a substance that causes a similar taste based
on the taste information on a human. For example, in terms
of the human’s taste, it is supposed that, if a human
perceives elements, sequentially a first 5 element, a second
element, and then a third element, he has a tendency to
feel a specific taste (e.g., sweet taste). In this case,
the information processing apparatus 100 may calculate a
relative feature value of the substance having the
10 structure that causes a human to perceive elements,
sequentially the first element, the second element, and
then the third element, so as to generate a new substance
having a similar structure.
[0091] Further, in the example described according to
15 the above embodiment, the song 30 is divided into bars to
calculate a feature value sequence; however, for example,
the information processing apparatus 100 may regard a
plurality of songs as the first content and regard each of
the songs as one set of partial data to calculate the
20 feature value sequence indicating the songs. In this case,
the information processing apparatus 100 may express the
feature such as the structure of the sequence of the songs.
Specifically, as the order of the sequence of the songs,
the information processing apparatus 100 expresses the
25 feature of the structure such that a positive song (e.g.,
an up-tempo song) comes first and a relatively negative
song comes next. As described in the above embodiment, the
information processing apparatus 100 may shift the feature
of the structure to different content. Thus, for example,
30 when other songs are rearranged, the information processing
apparatus 100 may generate a list having a structure
similar to that of the first content (i.e., a playlist in
which the order of songs is automatically rearranged).
28
[0092] Furthermore, among the processes described in
each of the above embodiments, all or some of the processes
that are automatically performed as described above may be
performed manually, or all or some of the processes that
are manually performed as described above 5 may be performed
automatically by using a well-known method. Furthermore,
the process steps, the specific names, and the information
including various types of data and parameters as described
in the above description and drawings may be optionally
10 changed except as otherwise noted. For example, various
types of information illustrated in each of the figures are
not limited to the information illustrated.
[0093] Furthermore, the components of each device
illustrated are functionally conceptual and do not
15 necessarily need to be physically configured as illustrated
in the drawings. Specifically, specific forms of
separation and combination of devices are not limited to
those depicted in the drawings, and a configuration may be
such that all or some of them are functionally or
20 physically separated or combined in any unit depending on
various types of loads or usage.
[0094] Furthermore, the above-described embodiments and
modifications may be combined as appropriate as long as the
consistency is ensured in the processing details.
25 [0095] Moreover, as the advantage described in this
description is merely an example, there is no limitation,
and other advantages may be produced.
[0096] (3. Hardware configuration)
An information apparatus such as the information
30 processing apparatus 100 according to the above-described
embodiments is implemented by using a computer 1000 having
the configuration illustrated in for example FIG. 6. The
information processing apparatus 100 according to the
29
embodiment is described below as an example. FIG. 6 is a
hardware configuration diagram illustrating an example of
the computer 1000 that performs a function of the
information processing apparatus 100. The computer 1000
includes a CPU 1100, a RAM 1200, a ROM 5 (Read Only Memory)
1300, an HDD (Hard Disk Drive) 1400, a communication
interface 1500, and an input/output interface 1600. The
units of the computer 1000 are connected via a bus 1050.
[0097] The CPU 1100 operates based on a program stored
10 in the ROM 1300 or the HDD 1400 to control each unit. For
example, the CPU 1100 loads a program stored in the ROM
1300 or the HDD 1400 into the RAM 1200 to execute the
processes corresponding to various programs.
[0098] The ROM 1300 stores, for example, a boot program
15 such as BIOS (Basic Input Output System) executed by the
CPU 1100 at the time of the start-up of the computer 1000
and a program that is dependent on the hardware of the
computer 1000.
[0099] The HDD 1400 is a computer-readable recording
20 medium that records a program to be executed by the CPU
1100, data to be used by the program, and the like, in a
non-transitory manner. Specifically, the HDD 1400 is a
recording medium that records an information processing
program according to the present disclosure, which is an
25 example of program data 1450.
[0100] The communication interface 1500 is an interface
with which the computer 1000 is connected to an external
network 1550 (e.g., the Internet). For example, the CPU
1100 receives data from a different device or transmits the
30 data generated by the CPU 1100 to a different device via
the communication interface 1500.
[0101] The input/output interface 1600 is an interface
for connecting the input/output device 1650 and the
30
computer 1000. For example, the CPU 1100 receives data
from an input device such as a keyboard or a mouse via the
input/output interface 1600. Furthermore, the CPU 1100
transmits data to an output device such as a display, a
speaker, or a printer via the input/output 5 interface 1600.
Further, the input/output interface 1600 may function as a
medium interface for reading a program, or the like,
recorded in a predetermined recording medium (media).
Examples of the medium include an optical recording medium
10 such as a DVD (Digital Versatile Disc) and a PD (Phase
change rewritable Disk), a magneto-optical recording medium
such as an MO (Magneto-Optical disk), a tape medium, a
magnetic recording medium, or a semiconductor memory.
[0102] For example, when the computer 1000 functions as
15 the information processing apparatus 100 according to the
embodiment, the CPU 1100 of the computer 1000 executes the
information processing program loaded into the RAM 1200 to
perform the functions of the control unit 130, and the
like. Furthermore, the HDD 1400 stores the information
20 processing program according to the present disclosure and
data in the storage unit 120. Moreover, although the CPU
1100 reads and executes the program data 1450 from the HDD
1400, it may acquire the programs from a different device
via the external network 1550 according to another example.
25 [0103] Further, this technology may also include the
configuration described below.
(1)
An information processing apparatus comprising:
an acquiring unit that uses a trained encoder to
30 acquire a feature value of each set of partial data forming
first content;
a calculating unit that calculates a relative feature
value that is a relative feature value between sets of
31
partial data from acquired feature values of the sets of
partial data to calculate a relative feature value sequence
indicating a feature of a structure of the first content;
and
a generating unit that generates second 5 content based
on the relative feature value sequence of the first content
and a feature value of optional data.
(2)
The information processing apparatus according to (1),
10 wherein the calculating unit executes addition,
subtraction, multiplication, or division on a feature value
of certain partial data of the first content and a feature
value of each set of partial data other than the certain
partial data to calculate the relative feature value so as
15 to calculate the relative feature value sequence.
(3)
The information processing apparatus according to (1)
or (2), wherein the calculating unit calculates a relative
feature value between a feature value of certain partial
20 data of the first content and a feature value of each set
of partial data that is other than the certain partial data
and that has a causal relationship with the certain partial
data so as to calculate the relative feature value
sequence.
25 (4)
The information processing apparatus according to any
one of (1) to (3), wherein the generating unit inputs a
feature value sequence calculated from a relative feature
value sequence of the first content and a feature value of
30 optional data into a trained decoder to generate the second
content.
(5)
The information processing apparatus according to any
32
one of (1) to (4), wherein the generating unit generates an
optional song as the second content based on a relative
feature value sequence of the first content that is a song.
(6)
The information processing apparatus 5 according to (5),
wherein
the acquiring unit inputs, as data indicating a sound
included in the partial data and in the optional data, a
symbol string indicating a pitch, a sound length, and a
10 rest to the trained encoder to acquire feature values
corresponding to the partial data and the optional data,
and
the generating unit inputs the relative feature value
sequence obtained based on acquired feature values of the
15 optional data and the partial data to the trained decoder
to generate the second content.
(7)
The information processing apparatus according to any
one of (1) to (4), wherein
20 the acquiring unit acquires a feature value of each
set of partial data forming the first content that is text
data, and
the generating unit generates the second content that
is text data based on a relative feature value sequence of
25 the first content obtained based on the acquired feature
value of each set of partial data and the feature value of
the optional data that is text data.
(8)
The information processing apparatus according to any
30 one of (1) to (4), wherein
the acquiring unit acquires a feature value of each
set of partial data forming the first content that is
moving image content, and
33
the generating unit generates the second content that
is moving image content based on a relative feature value
sequence of the first content obtained based on the
acquired feature value of each set of partial data and the
feature value of the optional data that 5 is moving image or
image content.
(9)
The information processing apparatus according to any
one of (1) to (6), wherein the acquiring unit detects a
10 break of a bar of the first content that is a song and
acquires a feature value of each detected bar.
(10)
The information processing apparatus according to any
one of (1) to (6), wherein the acquiring unit detects a
15 rest that is more than a length of a predetermined
threshold in the first content that is a song, divides the
song into partial data based on the detected rest, and
acquires a feature value of each set of divided partial
data.
20 (11)
An information processing method causing a computer
to:
use a trained encoder to acquire a feature value of
each set of partial data forming first content;
25 calculate a relative feature value that is a relative
feature value between sets of partial data from acquired
feature values of the sets of partial data to calculate a
relative feature value sequence indicating a feature of a
structure of the first content; and
30 generate second content based on the relative feature
value sequence of the first content and a feature value of
optional data.
(12)
34
An information processing program causing a computer
to function as:
an acquiring unit that uses a trained encoder to
acquire a feature value of each set of partial data forming
5 first content;
a calculating unit that calculates a relative feature
value that is a relative feature value between sets of
partial data from acquired feature values of the sets of
partial data to calculate a relative feature value sequence
10 indicating a feature of a structure of the first content;
and
a generating unit that generates second content based
on the relative feature value sequence of the first content
and a feature value of optional data.
15
Reference Signs List
[0104] 100 INFORMATION PROCESSING APPARATUS
110 COMMUNICATION UNIT
120 STORAGE UNIT
20 121 MODEL STORAGE UNIT
122 SONG DATA STORAGE UNIT
130 CONTROL UNIT
131 TRAINING UNIT
132 ACQUIRING UNIT
25 133 CALCULATING UNIT
134 GENERATING UNIT
35
WE CLAIMS
1. An information processing apparatus comprising:
an acquiring unit that uses a trained encoder to
acquire a feature value of each set of partial data forming
5 first content;
a calculating unit that calculates a relative feature
value that is a relative feature value between sets of
partial data from acquired feature values of the sets of
partial data to calculate a relative feature value sequence
10 indicating a feature of a structure of the first content;
and
a generating unit that generates second content based
on the relative feature value sequence of the first content
and a feature value of optional data.
15
2. The information processing apparatus according to
claim 1, wherein the calculating unit executes addition,
subtraction, multiplication, or division on a feature value
of certain partial data of the first content and a feature
20 value of each set of partial data other than the certain
partial data to calculate the relative feature value so as
to calculate the relative feature value sequence.
3. The information processing apparatus according to
25 claim 1, wherein the calculating unit calculates a relative
feature value between a feature value of certain partial
data of the first content and a feature value of each set
of partial data that is other than the certain partial data
and that has a causal relationship with the certain partial
30 data so as to calculate the relative feature value
sequence.
4. The information processing apparatus according to
36
claim 1, wherein the generating unit inputs a feature value
sequence calculated from a relative feature value sequence
of the first content and a feature value of optional data
into a trained decoder to generate the second content.
5
5. The information processing apparatus according to
claim 1, wherein the generating unit generates an optional
song as the second content based on a relative feature
value sequence of the first content that is a song.
10
6. The information processing apparatus according to
claim 5, wherein
the acquiring unit inputs, as data indicating a sound
included in the partial data and in the optional data, a
15 symbol string indicating a pitch, a sound length, and a
rest to the trained encoder to acquire feature values
corresponding to the partial data and the optional data,
and
the generating unit inputs the relative feature value
20 sequence obtained based on acquired feature values of the
optional data and the partial data to the trained decoder
to generate the second content.
7. The information processing apparatus according to
25 claim 1, wherein
the acquiring unit acquires a feature value of each
set of partial data forming the first content that is text
data, and
the generating unit generates the second content that
30 is text data based on a relative feature value sequence of
the first content obtained based on the acquired feature
value of each set of partial data and the feature value of
the optional data that is text data.
37
8. The information processing apparatus according to
claim 1, wherein
the acquiring unit acquires a feature value of each
set of partial data forming the first 5 content that is
moving image content, and
the generating unit generates the second content that
is moving image content based on a relative feature value
sequence of the first content obtained based on the
10 acquired feature value of each set of partial data and the
feature value of the optional data that is moving image or
image content.
9. The information processing apparatus according to
15 claim 1, wherein the acquiring unit detects a break of a
bar of the first content that is a song and acquires a
feature value of each detected bar.
10. The information processing apparatus according to
20 claim 1, wherein the acquiring unit detects a rest that is
more than a length of a predetermined threshold in the
first content that is a song, divides the song into partial
data based on the detected rest, and acquires a feature
value of each set of divided partial data.
25
11. An information processing method causing a computer
to:
use a trained encoder to acquire a feature value of
each set of partial data forming first content;
30 calculate a relative feature value that is a relative
feature value between sets of partial data from acquired
feature values of the sets of partial data to calculate a
relative feature value sequence indicating a feature of a
38
structure of the first content; and
generate second content based on the relative feature
value sequence of the first content and a feature value of
optional data.
5
12. An information processing program causing a computer
to function as:
an acquiring unit that uses a trained encoder to
acquire a feature value of each set of partial data forming
10 first content;
a calculating unit that calculates a relative feature
value that is a relative feature value between sets of
partial data from acquired feature values of the sets of
partial data to calculate a relative feature value sequence
15 indicating a feature of a structure of the first content;
and
a generating unit that generates second content based
on the relative feature value sequence of the first content
and a feature value of optional data.