extracted from wipo:
formulas and tables are not copied:
FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENTS RULES, 2003
COMPLETE SPECIFICATION
(See section 10, rule 13)
“INFORMATION PROCESSING DEVICE”
SONY CORPORATION, of 1-7-1, Konan, Minato-ku, Tokyo
108-0075, Japan
The following specification particularly describes the invention and the manner in which it is to
be performed.
2
[DESCRIPTION]
INFORMATION PROCESSING APPARATUS
[Technical Field]
[0001]
The present disclosure relates to an information
processing apparatus.
[Background Art]
[0002]
In recent years, with the development of
miniaturization, simplification, and the like of various
types of motion sensors and biological information
sensors, it has become easier to acquire various types of
sensing data with these sensors, and the sensing data
have been used, for example, for assistance of user's
learning of a performance (playing of a musical
instrument, a sport, and the like). For example, as a
technology for evaluating a performance (playing of a
musical instrument) and assisting in the performance, a
technology disclosed in PTL 1 below can be cited. In such
learning assistance, data that can be used in the
learning assistance is obtained by acquiring various
pieces of sensing data from a plurality of sensors and
analyzing the plurality of acquired sensing data.
[Citation List]
3
[Patent Literature]
[0003]
[PTL 1]
Japanese Patent Laid-open No. 2009-47861
[Summary]
[Technical Problem]
[0004]
Meanwhile, in the case of intending to acquire a
plurality of pieces sensing data using a plurality of
different types of sensors, the sensing data sometimes
differs in form. For example, sensing data from a
biological information sensor is normally in an analog
form. In such a case, the analog form of the sensing data
from the biological information sensor is converted into
a digital form to make the sensing data in the form
similar to that of sensing data from other sensors, and
the sensing data from the biological information sensor
in the digital form is then analyzed together with the
sensing data from the other sensors. Furthermore, to
acquire data useful for the assistance of user's learning
of the performance, it is required to accurately
synchronize a plurality of pieces of sensing data even in
a mixture of forms as described above.
[0005]
4
Therefore, the present disclosure proposes a novel
and improved information processing apparatus capable of
accurately synchronizing and handling sensing data in a
mixture of forms and available for assisting in learning
of a performance.
[Solution to Problem]
[0006]
The present disclosure provides an information
processing apparatus including a conversion section that
converts a plurality of pieces of sensing data in
different forms obtained from a plurality of sensors each
sensing a state related to a performance by a motion of a
user, an information processing section that processes
the sensing data converted by the conversion section, and
an information output section that outputs feedback
information to the user on the basis of a processing
result of the information processing section. The
conversion section includes an analog-digital signal
conversion section that converts the sensing data in an
analog form from the sensors into sensing data in a
digital form and outputs the sensing data in the digital
form to the information processing section, and a
digital-analog signal conversion section that converts
the sensing data in the digital form from the sensors
5
into sensing data in the analog form and outputs the
sensing data in the analog form to the analog-digital
signal conversion section.
[Advantageous Effect of the Invention]
[0007]
As described so far, according to the present
disclosure, it is possible to provide an information
processing apparatus capable of accurately synchronizing
and handling sensing data in a mixture of forms and
available for assisting in learning of a performance.
[0008]
It is noted that effects are not always limited to
the above effect, the present disclosure may exhibit any
of the effects described in the present specification or
other effects that can be conceived from the present
specification in addition to or as an alternative to the
above effect.
[Brief Description of Drawings]
[0009]
[FIG. 1]
FIG. 1 is an explanatory diagram of an example of a
configuration of an information processing system 1
according to an embodiment of the present disclosure.
[FIG. 2]
6
FIG. 2 is a block diagram depicting a configuration
of a sensor apparatus 10 according to the embodiment of
the present disclosure.
[FIG. 3]
FIG. 3 is a block diagram depicting a configuration
of a server 30 according to the embodiment of the present
disclosure.
[FIG. 4]
FIG. 4 is a block diagram depicting configurations
of a conversion apparatus 20 and a computing apparatus 26
according to the embodiment of the present disclosure.
[FIG. 5]
FIG. 5 is a block diagram depicting a configuration
of a feedback apparatus 70 according to the embodiment of
the present disclosure.
[FIG. 6]
FIG. 6 is a flowchart illustrating an example of an
information processing method according to the embodiment
of the present disclosure.
[FIG. 7]
FIG. 7 is a flowchart of creating training data
according to the embodiment of the present disclosure.
[FIG. 8]
FIG. 8 is an explanatory diagram (1) of an example
7
of extraction of feature variables.
[FIG. 9]
FIG. 9 is an explanatory diagram (2) of the example
of extraction of feature variables.
[FIG. 10]
FIG. 10 is an explanatory diagram (1) of an example
of constructing a database.
[FIG. 11]
FIG. 11 is an explanatory diagram (2) of the
example of constructing a database.
[FIG. 12]
FIG. 12 is an explanatory diagram (1) of an example
of creating training data.
[FIG. 13]
FIG. 13 is an explanatory diagram (2) of the
example of creating training data.
[FIG. 14]
FIG. 14 is a flowchart of creating comparison data
according to the embodiment of the present disclosure.
[FIG. 15]
FIG. 15 is a flowchart of comparison according to
the embodiment of the present disclosure.
[FIG. 16]
FIG. 16 is an explanatory diagram (1) of an example
8
of comparison.
[FIG. 17]
FIG. 17 is an explanatory diagram (2) of the
example of comparison.
[FIG. 18]
FIG. 18 is a flowchart of a learning assistance
mode according to the embodiment of the present
disclosure.
[FIG. 19]
FIG. 19 is an explanatory diagram of an example of
a hardware configuration of an information processing
apparatus 900 according to the embodiment of the present
disclosure.
[Description of Embodiments]
[0010]
A preferable embodiment of the present disclosure
will be described hereinafter in detail with reference to
the accompanying drawings. It is noted that, in the
present specification and drawings, repetitive
description will be omitted by denoting constituent
elements having substantially identical functional
configurations by the same reference sign.
[0011]
Furthermore, in the present specification and
9
drawings, a plurality of constituent elements having
substantially the same or similar functional
configurations is sometimes distinguished from one
another by putting different numbers after the same
reference sign. However, in the case of no need to
particularly distinguish the plurality of constituent
elements having substantially the same or similar
functional configurations, the constituent elements are
denoted only by the same reference sign. Moreover,
similar constituent elements in different embodiments are
sometimes distinguished from one another by putting
different alphabets after the same reference sign.
However, in the case of no need to particularly
distinguish the similar constituent elements, the
constituent elements are denoted only by the same
reference sign.
[0012]
It is noted that the description is given in the
following order.
1. Circumstances until inventors of present disclosure
reached to creation of embodiment according to present
disclosure
2. Embodiment of present disclosure
2.1. Outline of information processing system 1
10
according to embodiment of present disclosure
2.2. Configuration of sensor apparatus 10 according
to embodiment of present disclosure
2.3. Configuration of server 30 according to
embodiment of present disclosure
2.4. Configurations of conversion apparatus 20 and
computing apparatus 26 according to embodiment of present
disclosure
2.5. Configuration of feedback apparatus 70
according to embodiment of present disclosure
2.6. Information processing method according to
embodiment of present disclosure
2.6.1 Details of creation of training data
2.6.2 Details of creation of comparison data
2.6.3 Details of comparison
2.7. Summary
3. Learning assistance mode
4. Examples of application of embodiment of present
disclosure
5. Hardware configuration
6. Supplemental remarks
[0013]
<<1. Circumstances until inventors of present disclosure
reached to creation of embodiment according to present
11
disclosure>>
Before starting description of an embodiment
according to the present disclosure, circumstances until
the inventors of the present disclosure reached to
creation of the embodiment according to the present
disclosure will first be described.
[0014]
As previously described, in recent years, with the
development of miniaturization, simplification, and the
like of various types of motion sensors and biological
information sensors, it has become easier to acquire
various types of sensing data with these sensors, and the
sensing data have been used, for example, for assistance
of user's learning of a performance (playing of a musical
instrument, a sport, and the like). In such learning
assistance, data that can be used in the learning
assistance is obtained by acquiring various pieces of
sensing data from a plurality of sensors simultaneously
in real time and analyzing the plurality of acquired
sensing data.
[0015]
Furthermore, in the case of performing such
analysis, it is required to match timings of acquiring
the sensing data and sensing time widths of the plurality
12
of sensors with one another, that is, to synchronize the
plurality of pieces of sensing data with one another. It
is assumed that "synchronize/synchronization" means to
match the timings of acquiring sensing data and the
sensing time widths of the plurality of sensing data with
one another in the following description.
[0016]
Meanwhile, the sensing data from the biological
information sensors are normally in an analog form such
as waveform data varying with the passage of time. On the
other hand, many sensors or electronic musical
instruments and the like used by a user normally output
sensing data or data in a digital form. To cope with the
difference, in the learning assistance as described above,
for purposes of simultaneously analyzing these plurality
of pieces of sensing data, the sensing data in the analog
form is converted into sensing data in the digital form
by an analog-digital signal converter or the like before
analysis. It is, therefore, required to accurately
synchronize even the plurality of pieces of sensing data
in a mixture of forms as described above so as to acquire
data useful for the assistance of user's learning of the
performance.
[0017]
13
However, the plurality of pieces of sensing data is
subjected to conversion as described above due to the
difference in form; thus, it is difficult to accurately
synchronize these plurality of pieces of sensing data.
[0018]
In light of such circumstances, therefore, the
inventors of the present disclosure have reached to
creation of the embodiment of the present disclosure
capable of accurately synchronizing and handling sensing
data in a mixture of forms and available for the
assistance of learning of a performance. An information
processing system and an information processing method
according to the embodiment of the present disclosure
described above will be described hereinafter in sequence
and in detail.
[0019]
It is noted that the embodiment of the present
disclosure will be described hereinafter regarding the
case of applying the embodiment to guidance (learning of
a skill) of a piano playing technique. However,
application of the embodiment of the present disclosure
is not limited to the guidance of the piano playing
technique, and the embodiment of the present disclosure
is applicable to playing techniques of other musical
14
instruments, learning of skills of sports and traditional
arts and crafts, rehabilitation of an impaired motility
function, and the like. Furthermore, in the embodiment of
the present disclosure, the musical instrument is not
limited to the piano but may be every type of electronic
musical instrument or every type of acoustic musical
instrument.
[0020]
Furthermore, in the following description, examples
of a user include a learner receiving guidance of the
piano playing technique and a performer who gives or is
to give a performance such as a performer (for example,
professional pianist) regarding whom various types of
sensing data are collected in constructing a database
according to the present embodiment, and also include an
instructor or the like using the information processing
system according to the present embodiment.
[0021]
Moreover, in the following description, it is
assumed that evaluation values (evaluation information)
for a piano performance (performance) mean a performance
speed (a pace, a rhythm) of the performer in the
performance, accuracy of the performance (accuracy of the
rhythm or of a volume of each tone), a volume of a sound
15
generated by the performance (a peak of a sound waveform),
a sound vibration (an integral value of the sound
waveform), a timbre (a spectrum), a volume difference and
a temporal difference (so-called "balance" in a chord (a
group of musical notes)) among musical tones in the chord,
a difference (range) between a maximum value and a
minimum value of each sound parameter, a granularity (a
resolution) of each sound parameter, energy efficiency in
the performance, and the like. Furthermore, in the
following description, parameters in a trade-off relation
(for example, the performance speed and the accuracy) are
present among the plurality of parameters described
above; thus, in consideration of such a case, the
evaluation values of the piano performance may be
proportions of the parameters (numerical values). In
general, an instructor gives guidance while placing
greater importance on preferentially learning the
performance accuracy than on the performance speed for
the following reasons. In a case in which the learner
repeats performances with many false motions and low
accuracy while keeping a high performance speed, the body,
cranial nerves, and the like of the learner are caused to
learn the false motions. On the other hand, in the case
of guidance with excessive importance placed on learning
16
the accuracy, the learner strains himself/herself and
falls into a habit of hardening the muscles, causing a
reduction in energy efficiency in the performance. In the
present embodiment, therefore, the proportions of the
parameters may be used as the evaluation values in
consideration of the balance of the parameters in the
trade-off relation as described above. Moreover, in the
case of applying the embodiment of the present disclosure
not only to the piano performance but also to other user
motions, the evaluation values of the performance may be
a motion pattern of a motion element made by the user, a
motion velocity, motion accuracy, a quantity of motion
(motion power, an impulse, an amount of work, and the
like), energy efficiency in the performance, a state of a
result generated by the performance, and the like.
[0022]
Furthermore, in the following description,
synchronizing a plurality of pieces of sensing data means
to match timings of acquisition and processing of the
plurality of pieces of sensing data with one another and
to match time widths of the plurality of pieces of
sensing data with one another.
[0023]
Additionally, in the following description,
17
performing one test piece (for example, a music, a phrase,
a scale, an arpeggio, a chord, or the like) by one
performer or one learner is referred to as "one trial."
It is noted that, in the embodiment of the present
disclosure, it is possible to acquire a plurality of
pieces of sensing data in one trial.
[0024]
<<2. Embodiment of present disclosure>>
<2.1. Outline of information processing system 1
according to embodiment of present disclosure>
A schematic configuration of an information
processing system (information processing apparatus) 1
according to the embodiment of the present disclosure
will first be described with reference to FIG. 1. FIG. 1
is an explanatory diagram of an example of a
configuration of the information processing system 1
according to the present embodiment. In the information
processing system 1 according to the present embodiment,
a server (information processing section) 30 analyzes a
plurality of pieces of sensing data obtained from a
plurality of sensor apparatuses (sensors) 10 that senses
a state related to a performance given by moving of a
user (learner or performer). Furthermore, the information
processing system 1 feeds back information in various
18
forms to the user via a feedback apparatus (information
output section) 70 on the basis of a result of the
analysis described above.
[0025]
As depicted in FIG. 1, the information processing
system 1 according to the present embodiment includes the
sensor apparatus 10, the server 30, and the feedback
apparatus 70, and these apparatuses are communicably
connected to one another via a network 90. For example,
the sensor apparatus 10, the server 30, and the feedback
apparatus 70 may be connected to the network 90 via base
stations or the like not depicted (for example, base
stations of cellular telephones or access points of a
wireless LAN (Local Area Network)). In other words, any
scheme, regardless of wired or wireless, can be applied
to a communication scheme used in the network 90. An
outline of each apparatus included in the information
processing system 1 according to the present embodiment
will be described hereinafter.
[0026]
(Sensor apparatus 10)
The sensor apparatus 10 can sense a state related
to a performance according to a motion of the performer
or the learner. More specifically, the sensor apparatus
19
10 can be every type of biological information sensor
that can be attached to a part of a body of the learner
or the performer, an imaging apparatus that images the
learner or the performer, a pressure sensor or a
photoreflector sensor provided in the piano played by the
learner or the performer, a sound pick-up apparatus (for
example, a microphone) that collects the sound of the
piano, or the like. Furthermore, the sensor apparatus 10
may be an electronic musical instrument such as an
electronic piano per se capable of outputting signals.
Moreover, the number of the sensor apparatuses 10 and a
type thereof included in the information processing
system 1 according to the present embodiment are not
limited to a specific number and a specific type. It is
noted that details of the sensor apparatus 10 will be
described later.
[0027]
(Server 30)
The server 30 is configured with, for example, a
computer. More specifically, the server 30 collects the
sensing data related to the performance of the performer
or the learner from the sensor apparatus 10, analyzes and
processes the collected sensing data, and outputs
information for feedback (feedback information) to the
20
learner or the like on the basis of results of analysis
and processing. Furthermore, the server 30 may be, for
example, a computer owned by the user, or a computer
owned by a service provider that provides services by the
present embodiment and that is present in a location
different from a location of the user. It is noted that
details of the server 30 will be described later.
[0028]
(Feedback apparatus 70)
The feedback apparatus 70 is an apparatus for
presenting (outputting) the feedback information from the
server 30 to the learner or the like, and notifies the
learner of the feedback information (outputs the feedback
information to the learner) as visible, haptic, auditory,
or audible data during or after the performance of the
learner. For example, the feedback apparatus 70 can be a
device including a display device (display) and an audio
output device (speaker) such as a tablet, a smart phone,
a laptop PC (Personal Computer), or a notebook PC.
Furthermore, the feedback apparatus 70 may be a wearable
device that can be attached to a part of the body of the
learner. More specifically, as the wearable device, every
type of wearable device such as an HMD (Head Mounted
Display) type, an ear device (headphone) type, an anklet
21
type, a wristband type, a choker ring type, an eyewear
type, a glove type, a pad type, a badge type, and a
garment type. It is noted that details of the feedback
apparatus 70 will be described later.
[0029]
While FIG. 1 depicts that the information
processing system 1 according to the present embodiment
includes one sensor apparatus 10 and one feedback
apparatus 70, the configuration of the information
processing system 1 according to the present embodiment
is not limited to that depicted in FIG. 1. In the present
embodiment, the information processing system 1 can
include, for example, a plurality of sensor apparatuses
10 and a plurality of feedback apparatuses 70.
Furthermore, the information processing system 1
according to the embodiment may include, for example,
another communication apparatus such as a relay apparatus
used at the time of transmitting the sensing data from
the sensor apparatus 10 to the server 30.
[0030]
<2.2. Configuration of sensor apparatus 10 according to
embodiment of present disclosure>
A configuration of the sensor apparatus 10
according to the embodiment of the present disclosure
22
will next be described with reference to FIG. 2. FIG. 2
is a block diagram depicting the configuration of the
sensor apparatus 10 according to the present embodiment.
As depicted in FIG. 2, the sensor apparatus 10 according
to the present embodiment mainly has a sensor section 100,
a main control section 140, and a communication section
160. Details of the functional sections of the sensor
apparatus 10 will be described hereinafter.
[0031]
(Sensor section 100)
The sensor section 100 can acquire, when attached
to the body of the performer or the learner, the sensing
data indicating a state of each motion element made by
each part of the body of the learner or the performer
during the performance. For example, the sensor section
100 is realized by one or a plurality of sensor devices
including an acceleration sensor, an angular velocity
sensor, a gyrosensor, a geomagnetic sensor, a position
sensor, a vibration sensor, a pressure sensor, and a
bending sensor. The sensor devices described above each
detect changes in an acceleration, an angular velocity,
or the like applied by the motion elements, and generate
a plurality of pieces of sensing data indicating the
detected changes. In a case in which the sensor section
23
100 is a sensor (first sensor) that acquires sensing data
in an analog form, the sensing data is converted into
sensing data in a digital form by a conversion apparatus
(conversion section) 20 to be described later, and the
sensing data in the digital form is output to the server
30. However, in the present embodiment, the sensor
section 100 described above is not limited to a sensor
that acquires sensing data in the analog form but may be
a sensor (second sensor) that acquires sensing data in
the digital form.
[0032]
Moreover, the sensor section 100 may be, for
example, a key touch detection sensor that detects
vertical movements of keys of the piano (a subject)
moving by a motion (performance) of the learner or the
performer. It is possible to detect the vertical movement
of each key by installing the key touch detection sensor,
for example, below each key. Specifically, the sensor
section 100 can be, for example, a pressure sensor that
detects a pressure applied to each key of the piano by
the motion element of the learner or the performer, or a
photoreflector sensor including a light
receiving/emitting sensor that detects the vertical
movement of each key by reflection of light. It is noted
24
that, in a case in which the sensor section 100 as
described above is the sensor (first sensor) that
acquires the sensing data in the analog form, the sensing
data is converted into the sensing data in the digital
form by the conversion apparatus 20, and the sensing data
in the digital form is output to the server 30. Moreover,
in the present embodiment, the subject for which
detection is performed is not limited to the keys of the
piano but may be another musical instrument (acoustic
musical instrument or electronic musical instrument) per
se or a part of the other musical instrument.
[0033]
Furthermore, in the present embodiment, the sensor
section 100 may be an imaging apparatus that images the
learner or the performer, and in this case, it is
possible to quantitatively detect positions and motions
of joints of the performer or the like by causing a high
speed imaging camera (imaging apparatus) to capture
motions of the performer or the like. Moreover, in the
present embodiment, the imaging apparatus may detect a
motion of an eyeball (eyeball movement) or a size of a
pupil (pupil diameter) of the learner or the performer.
It is noted that, in the case in which the sensor section
100 as described above is the sensor (first sensor) that
25
acquires the sensing data in the analog form, the sensing
data is converted into the sensing data in the digital
form by the conversion apparatus 20, and the sensing data
in the digital form is output to the server 30, similarly
to the description given so far.
[0034]
Moreover, in the present embodiment, the sensor
section 100 may be a nuclear magnetic resonance sensor
that detects an oral cavity state or an intratracheal
state, a motion of a lip or a tongue, and the like of the
learner or the performer using nuclear magnetic resonance.
Specifically, the sensor section 100 can detect the state,
the motion, and the like described above by causing the
learner or the like to execute a performance within an
MRI (Magnetic Resonance Imaging) apparatus. Particularly
in the case of applying the embodiment of the present
disclosure to a playing technique for every type of brass
instrument (a flute, an oboe, a clarinet, a trumpet, or
the like), the MRI is useful since it is possible to
detect the motion of the lip or the tongue that is
difficult to detect by other methods. It is noted that,
in the case in which the sensor section 100 as described
above is the sensor (first sensor) that acquires the
sensing data in the analog form, the sensing data is
26
converted into the sensing data in the digital form by
the conversion apparatus 20, and the sensing data in the
digital form is output to the server 30, similarly to the
description given so far.
[0035]
Furthermore, in the present embodiment, the sensor
section 100 may be a biological information sensor such
as a myoelectric sensor, a heartbeat sensor, a pulse
sensor, a blood flow sensor, a respiration sensor, a
brain wave sensor, a skin temperature sensor, a skin
electrical conductivity (skin resistance) sensor, or a
perspiration sensor. It is noted herein that the
myoelectric sensor is a sensor that senses a feeble
electrical field generated from muscle fibers configuring
muscles. More specifically, the myoelectric sensor can
quantitatively detect muscle active masses of the muscles
by measuring myoelectric potentials by electrical signals
that are generated in muscle fibers when the muscles of
an arm and the like of the performer or the learner
contract and that are propagated through a surface of the
body by a plurality of electrodes attached to the arm and
the like. Furthermore, the heartbeat sensor is a sensor
that detects a heartbeat that is a beat of a heart, and
the pulse sensor is a sensor that detects a pulse that is
27
an arterial pulsation appearing on the surface of the
body or the like when pressure changes occur in artery
linings by feeding a blood to an entire body through
arteries. The blood flow sensor is a sensor that emits an
infrared radiation to the body and that detects a blood
flow rate by reflection of the infrared radiation. The
respiration sensor can be a respiratory flowmeter that
detects a change in a respiratory volume. The brain wave
sensor is a sensor that detects a brain wave by attaching
a plurality of electrodes to a scalp, removing noise from
a fluctuation in a measured potential difference between
the electrodes, and extracting a periodic wave. The skin
temperature sensor is a sensor that detects a body
temperature of the performer or the learner, and the skin
electrical conductivity sensor is a sensor that detects a
skin electrical resistance of the performer or the
learner. Moreover, the perspiration sensor is a sensor
that detects perspiration of the performer or the learner.
It is noted that, in the case in which the sensor section
100 as described above is the sensor (first sensor) that
acquires the sensing data in the analog form, the sensing
data is converted into the sensing data in the digital
form by the conversion apparatus 20, and the sensing data
in the digital form is output to the server 30, similarly
28
to the description given so far.
[0036]
Furthermore, the sensor section 100 may be a sound
pick-up apparatus that collects a sound from the piano
played by the performer or the learner. The sensor
section 100 may be, for example, a microphone provided in
the vicinity of the piano. It is noted that, in the case
in which the sensor section 100 as described above is the
sensor (first sensor) that acquires the sensing data in
the analog form, the sensing data is converted into the
sensing data in the digital form by the conversion
apparatus 20, and the sensing data in the digital form is
output to the server 30, similarly to the description
given so far.
[0037]
Moreover, the sensing data from the sensor section
100 may be an output (sound data) from an electronic
musical instrument played by the learner or the performer,
that is, used in the performance. In other words, in the
present embodiment, the sensor section 100 may be an
electronic musical instrument. In this case, the sensing
data from the sensor section 100 is data in the digital
form compliant with, for example, a MIDI (Musical
Instrument Digital Interface) standard.
29
[0038]
Furthermore, the sensor section 100 may include a
position information sensor such as a GPS (Global
Positioning System) receiver or the like that acquires
position information regarding the learner or the
performer. In addition, the sensor section 100 may
include other various types of sensors such as an
atmospheric pressure sensor, a temperature sensor, and a
humidity sensor for acquiring environmental information
indicating a state of an environment where the learner or
the performer gives the performance.
[0039]
Moreover, in the present embodiment, the sensor
section 100 may be every type of sensor already provided
in a musical instrument (acoustic musical instrument or
an electronic musical instrument) already shipped by each
musical instrument manufacturer.
[0040]
(Main control section 140)
The main control section 140 is provided in the
sensor apparatus 10 and can exercise control over the
blocks in the sensor apparatus 10. The main control
section 140 is realized by hardware which is, for example,
a CPU (Central Processing Unit), a ROM (Read Only Memory),
30
and a RAM (Random Access Memory). Furthermore, the main
control section 140 can function as a data acquisition
section 142, a processing section 144, and an output
control section 152. Details of these functional sections
of the main control section 140 according to the present
embodiment will be described hereinafter.
[0041]
-Data acquisition section 142-
The data acquisition section 142 exercises control
over the sensor section 100, acquires the sensing data
output from the sensor section 100, and outputs the
acquired sensing data to the processing section 144 to be
described later.
[0042]
-Processing section 144-
The processing section 144 converts the sensing
data output from the data acquisition section 142
described above into sensing data in a predetermined form
in which the sensing data can be transmitted via the
network 90, and outputs the sensing data in the
predetermined form to the output control section 152 to
be described later.
[0043]
-Output control section 152-
31
The output control section 152 exercises control
over the communication section 160 to be described later
in such a manner as to transmit the sensing data in the
predetermined form output from the processing section 144
described above to the server 30.
[0044]
(Communication section 160)
The communication section 160 is provided in the
sensor apparatus 10, and can transmit and receive
information to and from an external apparatus such as the
server 30. In other words, the communication section 160
can be said as a communication interface having functions
to transmit and receive data. It is noted that the
communication section 160 is realized by communication
devices such as a communication antenna, a transmittingreceiving
circuit, and a port.
[0045]
It is noted that the sensor apparatus 10 may be a
wearable device of every type including the HMD type, the
ear device type, the anklet type, the wristband type, the
choker ring type, the eyewear type, the pad type, the
badge type, a belt type, and the garment type.
Specifically, these wearable devices can be provided, as
motion capture devices, on a finger, an arm, a leg, the
32
body, a head, and a toe of the learner or the performer
in order to acquire various kinds of sensing data.
Furthermore, the sensor apparatus 10 may be an apparatus,
such as the imaging apparatus or the sound pick-up
apparatus, installed around the learner or the performer,
or may be the musical instrument per se used by the
learner or the performer; thus, the sensor apparatus 10
is not limited to a specific type. Moreover, in the
present embodiment, the sensor apparatus 10 is not
limited to the configuration depicted in FIG. 2.
[0046]
<2.3. Configuration of server 30 according to embodiment
of present disclosure>
A configuration of the server 30 according to the
embodiment of the present disclosure will next be
described with reference to FIG. 3. FIG. 3 is a block
diagram depicting the configuration of the server 30
according to the embodiment of the present disclosure. As
previously described, the server 30 is configured with,
for example, a computer. As depicted in FIG. 3, the
server 30 mainly has an input section 300, an output
section 310, a main control section 340, a communication
section 360, and a storage section 370. Details of the
functional sections of the server 30 will be described
33
hereinafter.
[0047]
(Input section 300)
The input section 300 receives inputs of data and
commands transmitted to the server 30. More specifically,
the input section 300 is realized by a touch panel, a
keyboard, and the like, and can receive inputs of
attribute information regarding the learner or the
performer, subjective evaluations for the performance to
be described later, and the like.
[0048]
(Output section 310)
The output section 310 is configured with, for
example, a display, a speaker, a video output terminal,
an audio output terminal, and the like, and outputs
various types of information by images or audio.
[0049]
(Main control section 340)
The main control section 340 is provided in the
server 30 and can exercise control over the blocks in the
server 30. The main control section 340 is realized by
hardware which is, for example, a CPU, a ROM, and a RAM.
Furthermore, the main control section 340 can function as
a data acquisition section 342, an analysis section
34
(information processing section) 346, and an output
control section (information output section) 352. Details
of these functional sections of the main control section
340 according to the present embodiment will be described
hereinafter.
[0050]
-Data acquisition section 342-
The data acquisition section 342 acquires the
sensing data transmitted from the sensor apparatus 10
described above, and outputs the acquired sensing data to
the processing section 344 to be described later.
[0051]
-Processing section 344-
The processing section 344 analyzes and processes
the sensing data output from the data acquisition section
342 described above. The output control section 352 then
generates feedback information to be fed back to the user
on the basis of results of analysis and processing.
Furthermore, the processing section 344 can construct a
database (DB) 372 (refer to FIG. 10) including various
types of information for generating the feedback
information. More specifically, to realize these
functions described above, the processing section 344
functions as an analysis section 346, a learning section
35
348, and a comparison section 350 as depicted in FIG. 3.
Details of these functional sections of the processing
section 344 according to the present embodiment will be
described hereinafter.
[0052]
The analysis section 346 performs analysis and the
like on a plurality of pieces of sensing data that can be
acquired in one trial, and extracts feature variables
characterizing a state of the performance of the learner
or the performer. The feature variables can be extracted
as a maximum value, a minimum value, a mean value, an
integral value, a period, an amount of change of each
sensing data by statistically processing each sensing
data by the analysis section 346. More specifically, the
feature variables can be timings of peaks of muscle
activities and maximum joint angles, and the like. The
feature variables extracted in this way are linked to the
plurality of pieces of sensing data acquired in the trial
and stored in the storage section 370 to be described
later as the DB 372. Furthermore, the analysis section
346 may output the extracted feature variables to the
learning section 348, the comparison section 350, and the
like to be described later.
[0053]
36
Moreover, the analysis section 346 may perform
analysis and the like on the plurality of pieces of
sensing data (more specifically, for example, audio data
related to the performance) that can be acquired in one
trial, and extract evaluation values for the performance
of the learner or the performer. The evaluation values
are, for example, evaluation values for a piano
performance, and can be the performance speed of the
performer in the performance, the accuracy of the
performance, the volume of the sound generated by the
performance, the sound vibration, the timbre, the volume
difference and the temporal difference among the musical
tones in the chord, the difference between the maximum
value and the minimum value of each sound parameter, the
granularity of each sound parameter, and the like. The
evaluation values extracted in this way are linked to the
plurality of pieces of sensing data acquired in the trial
and stored in the storage section 370 to be described
later as the DB 372. Furthermore, the analysis section
346 may output the extracted evaluation values to the
learning section 348, the comparison section 350, and the
like to be described later.
[0054]
The learning section 348 acquires information
37
associated with relation among the sensing data, the
feature variables, and the evaluation values by
performing multivariate analysis thereon. More
specifically, the learning section 348 is a supervised
learning instrument such as a support vector regression
and a deep neural network, and can perform machine
learning on the relation between the sensing data and the
evaluation values by performing, for example, the
multivariate analysis such as multiple regression
analysis thereon. Information regarding the relation
(relation information) obtained by performing the machine
learning by the learning section 348 can be stored in the
storage section 370 to be described later as the DB 372,
and can be used in selection of training data performed
by the comparison section 350 to be described later.
[0055]
The comparison section 350 selects one or a
plurality of pieces of sensing data that serves as a
model of the learner from the DB 372, and compares the
selected sensing data (training data) with sensing data
(comparison data) of the same item out of a plurality of
pieces of newly acquired sensing data regarding the
learner. More specifically, the comparison section 350
may calculate, for example, a difference (gap) between
38
the training data and the comparison data or calculate a
degree of matching of these pieces of data. Furthermore,
the comparison section 350 may perform comparison by
superimposing the training data on the comparison data.
[0056]
Specifically, the comparison section 350 selects,
as the training data, for example, representative sensing
data that serves as the model of the learner aiming at an
accurate performance from the DB 372, and compares the
selected training data with the sensing data regarding
the learner of the same item. At this time, in the case
of presence of a plurality of pieces of representative
sensing data serving as models, the comparison section
350 calculates a difference between each sensing data and
the sensing data regarding the learner of the same item
as an item of the representative sensing data. The
comparison section 350 then selects the sensing data
greatest in difference as the sensing data related to an
element with high necessity of correction such that the
performance of the learner becomes more accurate. In the
present embodiment, such selection makes it possible to
pay more attention on a technical element which the
learner is slowest in learning; thus, it is possible to
generate the feedback information useful to efficiently
39
acquire the accurate performance.
[0057]
Alternatively, the comparison section 350 may
select, as the training data, the sensing data estimated
to have a higher relation with the accuracy (evaluation
value) of the performance on the basis of the information
regarding the relation obtained by the learning section
348 described above. In the present embodiment, such
selection makes it possible to pay more attention on a
technical element that enables efficient acquisition of
the accurate performance; thus, it is possible to
generate the feedback information useful to efficiently
acquire the accurate performance.
[0058]
In another alternative, in the present embodiment,
the comparison section 350 may select, as the training
data, the sensing data related to a performer having the
same or similar attribute information as or to attribute
information (a gender, an age, a physical size, a muscle
force, tenderness, legerity, and the like) regarding the
learner on the basis of the attribute information
regarding the learner. In the present embodiment, such
selection makes it possible to perform feedback
tailormade to attributes and the like of the learner;
40
thus, the learner can efficiently perform learning.
Moreover, in the case of comparison between a past state
and a current state of the performance of the learner,
the comparison section 350 may select past sensing data
regarding the learner.
[0059]
Moreover, in the present embodiment, the comparison
section 350 is not limited to comparing the sensing data
but may compare the feature variables extracted by the
analysis section 346 described above. Even in such a case,
the comparison section 350 can perform the selection
described above in a case in which a plurality of
representative feature variables serving as models is
present.
[0060]
-Output control section 352-
The output control section 352 generates the
feedback information on the basis of a result of
comparison by the comparison section 350, and exercises
control over the communication section 360 to be
described later in such a manner as to transmit the
feedback information to the feedback apparatus 70. The
transmitted feedback information is output to the user
via the feedback apparatus 70. Furthermore, the output
41
control section 352 may perform processing for
emphasizing or reducing the difference (gap) between the
training data and the comparison data by expanding or
contracting the difference either spatially or temporally,
and generate the feedback information. Specifically, in
the present embodiment, the output control section 352
feeds back the feedback information generated by reducing
the difference to the learner at a technical level
greater in difference from the training data to avoid
decrease of motivation of learning. On the other hand, in
the present embodiment, the output control section 352
feeds back the feedback information generated by
emphasizing the small difference to guide the learner to
a higher technical level in such a manner that the
learner can easily recognize the difference, to the
learner at the technical level smaller in difference from
that of the training data. In other words, in the present
embodiment, performing the processing as described above
makes it possible for the learner to easily recognize the
difference from the training data while avoiding the
decrease of the motivation of the learner; thus, it is
possible to realize assistance of efficient learning of
the performance.
[0061]
42
Moreover, the output control section 352 may select
a sensation modality (such as a visual sensation, an
auditory sensation, or a tactile sensation) suited for
the feedback information on the basis of user's
situations and the like, and transmit the feedback
information to the feedback apparatus 70 according to the
selected sensation modality. In the present embodiment,
the feedback information can be provided to the user by
the sensation modality according to the user's situations
and the like; thus, it is possible to realize assistance
of efficient learning of the performance.
[0062]
(Communication section 360)
The communication section 360 is provided in the
server 30, and can transmit and receive information to
and from an external apparatus such as the sensor
apparatus 10 or the feedback apparatus 70. It is noted
that the communication section 360 is realized by
communication devices such as a communication antenna, a
transmitting-receiving circuit, and a port.
[0063]
(Storage section 370)
The storage section 370 is provided in the server
30 and stores therein programs, information, and the like
43
for the main control section 340 described above to
execute various types of processing. Furthermore, the
storage section 370 stores therein the DB 372 including
the plurality of pieces of sensing data linked to various
types of attribute information and the like (a metafile).
It is noted that the storage section 370 is realized by a
magnetic recording medium such as a hard disk (HD), a
nonvolatile memory such as a flash memory, or the like.
[0064]
It is noted that the metafile (attribute
information) can contain attribute information associated
with the learner or the performer (the name, the gender,
the age, a body height, a body weight, the physical size,
the muscle force, a palm size, the tenderness, the
legerity, years of experience of piano playing, a skill
level, a national origin, an instructor name, and the
like), attribute information associated with the
performance (date and time of performance, a title of a
music, a category of the music, a name of a composer, an
epoch of composition, a category of the composer, a tempo,
a volume, information whether a single tone or complexed
tones, contents of teaching given to the performer or the
learner, the musical instrument for the performance, a
location of the performance, and the like), feature
44
variables characterizing the performance, evaluation
values (evaluation information) for the performance,
attribute information associated with sensing (a sensor
type, a sampling frequency, the number of channels, and
the like), sensitivity evaluation information regarding
the performance (information such as an image, a timbre,
and a vibration the performer is to express), and the
like. The metafile can also contain attribute information
regarding each trial (a competition name, a competition
level, an attendance, the number of fans, evaluation
information regarding attendance, the number of views,
and the like). In other words, the metafile functions as
a label of the plurality of pieces of sensing data
(sensing data group) acquired in one trial, and can
indicate, for example, that the sensing data group is
information sensed when a certain performer performs a
certain piece while aiming to express the piece in a
certain manner.
[0065]
Furthermore, in the present embodiment, the server
30 is not limited to the configuration depicted in FIG. 3
but may include other functional blocks and the like.
[0066]
<2.4. Configurations of conversion apparatus 20 and
45
computing apparatus 26 according to embodiment of present
disclosure>
Meanwhile, as previously described, the sensing
data obtained by the sensor apparatus 10 included in the
information processing system 1 according to the present
embodiment is a mixture of the sensing data in the analog
form and the sensing data in the digital form. To acquire
data useful for assistance of learning of the performance
of the learner, it is required to accurately synchronize
a plurality of pieces of sensing data even in a mixture
of forms described above.
[0067]
In the present embodiment, therefore, it is
preferable to provide the conversion apparatus
(conversion section) 20 and a computing apparatus 26
between the sensor apparatus 10 and the server 30 as
depicted in FIG. 4. Therefore, the conversion apparatus
20 and the computing apparatus 26 according to the
present embodiment will be described hereinafter with
reference to FIG. 4. FIG. 4 is a block diagram depicting
configurations of the conversion apparatus 20 and the
computing apparatus 26 according to the present
embodiment. More specifically, the information processing
system 1 according to the present embodiment includes the
46
conversion apparatus 20 and the computing apparatus 26 as
depicted in FIG. 4 between the sensor apparatus 10 and
the server 30. Details of the conversion apparatus 20 and
the computing apparatus 26 will be described hereinafter.
[0068]
(Conversion apparatus 20)
As depicted in FIG. 4, the conversion apparatus 20
is an apparatus that converts a plurality of pieces of
sensing data in different forms obtained from a plurality
of sensor apparatuses 10, and mainly has a digital-analog
signal conversion section 200 and an analog-digital
signal conversion section 210. Details of the digitalanalog
signal conversion section 200 and the analogdigital
signal conversion section 210 of the conversion
apparatus 20 will be described hereinafter.
[0069]
-Digital-analog signal conversion section 200-
The digital-analog signal conversion section 200
converts sensing data received from a sensor apparatus
(second sensor) 10b that outputs sensing data in the
digital form among the plurality of sensor apparatuses 10
via the computing apparatus 26, to be described later,
into sensing data in the analog form. Furthermore, the
digital-analog signal conversion section 200 outputs the
47
sensing data in the analog form obtained by conversion to
the analog-digital signal conversion section 210 to be
described later.
[0070]
More specifically, the digital-analog signal
conversion section 200 has a plurality of digital-analog
converters (D/A converters) 202 including a microcomputer
board and the like. For example, one sensing data
acquired by one sensor apparatus 10b is extracted as
sensing data in one of a plurality of channels by the
computing apparatus 26 to be described later. The
extracted sensing data per channel is input to each of
the D/A converters 202. Specifically, in a case in which
the sensor apparatus 10b is an electronic piano that
outputs a MIDI signal that is sensing data in the digital
form compliant with the MIDI standard, each packet in the
MIDI signal contains information regarding an event such
as key touch/key release (a status), information
regarding a pitch of a generated sound (a note), and
information regarding a loudness of the sound (a
velocity). In the present embodiment, therefore, the
computing apparatus 26 extracts the MIDI signal as pieces
of sensing data regarding the status, the note, and the
velocity as described above per channel, and inputs the
48
extracted sensing data to the D/A converters 202.
Moreover, each extracted sensing data is converted into
sensing data in the analog form by each D/A converter 202,
and the sensing data in the analog form is input to the
analog-digital signal conversion section 210 to be
described later. In the embodiment, converting the
sensing data in the digital form into the sensing data in
the analog form in this way makes it possible to
simultaneously handle the sensing data in the analog form
with the sensing data that is in the analog form from the
first start and that is output from a sensor apparatus
10a.
[0071]
Moreover, a part of the sensing data the form of
which is converted into the analog form by the digitalanalog
signal conversion section 200 contains, for
example, information regarding a timing of occurrence of
an event such as a moment of touching specific keys at a
strength equal to or higher than a certain level (event).
Therefore, sensing data containing such an event may be
converted into sensing data in the analog form by the
digital-analog signal conversion section 200, and input
to the other sensor apparatus 10 (for example, the sensor
apparatus (first sensor) 10a that acquires the sensing
49
data in the analog form) as a trigger signal. The trigger
signal can be used for the sensor apparatus 10a to start
(drive) acquiring the sensing data.
[0072]
-Analog-digital signal conversion section 210-
The analog-digital signal conversion section 210
converts sensing data from the sensor apparatus (first
sensor) 10a that outputs sensing data in the analog form
among the plurality of sensor apparatuses 10 into sensing
data in the digital form. Furthermore, the analog-digital
signal conversion section 210 converts the sensing data
in the analog form into which the digital-analog signal
conversion section 200 described above converts the
sensing data in the digital form, into sensing data in
the digital form. The analog-digital signal conversion
section 210 then outputs the sensing data obtained by
conversion to the server 30 described above. More
specifically, the analog-digital signal conversion
section 210 has a plurality of analog-digital converters
(A/D converters) 212 including a microcomputer board and
the like, and each A/D converter 212 can convert each
sensing data in the analog form into the sensing data in
the digital form and output the sensing data in the
digital form to the server 30.
50
[0073]
Furthermore, the analog-digital signal conversion
section 210 has a clock (first clock mechanism) 214
mounted on the microcomputer board to synchronize a
plurality of pieces of sensing data. The clock 214
simultaneously outputs a trigger signal to each of the
A/D converters 212 or each sensor apparatus 10a, and
causes the A/D converter 212 or the sensor apparatus 10a
to start acquiring the sensing data by the trigger signal,
thereby making it possible to match timings of acquiring
the sensing data. In other words, in the present
embodiment, the clock 214 enables the pieces of sensing
data from the plurality of sensing apparatuses 10a to be
accurately synchronized with one another. It is noted
that the clock 214 may output clock time information to
the server 30.
[0074]
(Computing apparatus 26)
The computing apparatus 26 acquires the sensing
data from the sensor apparatus 10b that acquires the
sensing data in the digital form, and extracts the
acquired sensing data as sensing data per channel
(containing, for example, the status, the note, and the
velocity). Furthermore, the computing apparatus 26
51
outputs the sensing data extracted for each of a
plurality of channels to the conversion apparatus 20
described above. More specifically, the computing
apparatus 26 mainly has a receiving section 260 and a
computing section 270 as depicted in FIG. 4. Details of
the receiving section 260 and the computing section 270
of the computing apparatus 26 will be described
hereinafter.
[0075]
-Receiving section 260-
The receiving section 260 receives the sensing data
from the sensor apparatus 10b that acquires the sensing
data in the digital form, and outputs the sensing data in
the digital form to the computing section 270 to be
described later.
[0076]
-Computing section 270-
The computing section 270 extracts the sensing data
acquired from the receiving section 260 described above
as the sensing data per channel. For example, as
described above, the computing section 270 extracts the
status, the note, and the velocity contained in each
packet in the MIDI signal that is the sensing data in the
digital form as the sensing data per channel. Furthermore,
52
the computing section 270 outputs the extracted sensing
data per channel to the conversion apparatus 20 described
above.
[0077]
Furthermore, the computing section 270 may have a
clock (second clock mechanism) 272 mounted on a
microcomputer board to synchronize the plurality of
pieces of sensing data. The clock 272 can link clock time
information to the sensing data. In the present
embodiment, since the clock 272 links the sensing data in
the plurality of channels from the sensor apparatus 10b
to the same clock time information, it is possible to
match clock times of the pieces of sensing data with one
another, that is, it is possible to synchronize the
pieces of sensing data. It is noted that the clock 272
may output the clock time information to the server 30.
[0078]
Furthermore, the server 30 can synchronize the
plurality of pieces of sensing data from the conversion
apparatus 20 on the basis of the clock time information
from the clocks 214 and 272. More specifically, the
sensing data from the sensor apparatus 10b that acquires
the sensing data in the digital form is converted into
the sensing data in the analog form in advance, and then
53
the sensing data in the analog form is converted into the
sensing data in the digital form. On the other hand, the
sensing data from the sensor apparatus 10a that acquires
the sensing data in the analog form is converted into the
sensing data in the digital form without being converted
into the sensing data in the analog form in advance.
Therefore, a minor temporal difference is generated
between the sensing data from the sensor apparatus 10b
and the sensing data from the sensor apparatus 10a at a
point in time of arrival of the sensing data at the
server 30. In the present embodiment, therefore, to
accurately synchronize the sensing data from the sensor
apparatus 10b with the sensing data from the sensor
apparatus 10a, the server 30 performs temporal shift
processing on these pieces of sensing data on the basis
of a temporal difference between the clocks 214 and 272
obtained from the clock time information. According to
the present embodiment, therefore, it is possible to
match time bases for the plurality of pieces of sensing
data in a mixture of forms and accurately synchronize and
handle these pieces of sensing data.
[0079]
It is noted that the clock 272 is not necessarily
provided in the computing apparatus 26 and may be
54
provided in the sensor apparatus 10b or in a portion of
the digital-analog signal conversion section 200 closer
to the computing apparatus 26, that is, closer to the
sensor apparatus 10b.
[0080]
It is noted that, in the present embodiment, the
configurations of the conversion apparatus 20 and the
computing apparatus 26 are not limited to those depicted
in FIG. 4. For example, both or one of the conversion
apparatus 20 or the computing apparatus 26 may be
provided in the server 30 described above. Alternatively,
the conversion apparatus 20 may be provided in the sensor
apparatus 10a, or the computing apparatus 26 may be
provided in the sensor apparatus 10b; thus, the
configurations of the conversion apparatus 20 and the
computing apparatus 26 are not limited to specific ones.
Nevertheless, providing any one of the conversion
apparatus 20 or the computing apparatus 26 in the sensor
apparatus 10a or 10b means that a weight of the sensor
apparatus 10 attached to the body of the performer or the
learner or attached to the subject moving according to
the motion of the performer or the learner becomes
heavier. In addition, it is estimated that the weight has
an influence on the performance of the performer or the
55
learner. Moreover, in such a case, it is estimated that
the sensing data from the sensor apparatuses 10a and 10b
tend to contain noise. It is, therefore, preferable that
the conversion apparatus 20 and the computing apparatus
26 are provided in locations other than the sensor
apparatuses 10a and 10b to avoid the influence on the
performance and the noise.
[0081]
As described so far, by having the conversion
apparatus 20 described above, the information processing
system 1 according to the present embodiment can
accurately synchronize the pieces of sensing data even in
the case of a mixture of the sensing data in the analog
form and the sensing data in the digital form. As a
result, the information processing system 1 can
simultaneously analyze the plurality of pieces of sensing
data accurately synchronized with one another and
containing various information; thus, it is possible to
acquire data useful for assistance of learning of the
performance of the learner.
[0082]
<2.5. Configuration of feedback apparatus 70 according to
embodiment of present disclosure>
A configuration of the feedback apparatus 70
56
according to the embodiment of the present disclosure
will next be described with reference to FIG. 5. FIG. 5
is a block diagram depicting the configuration of the
feedback apparatus 70 according to the present embodiment.
As previously described, the feedback apparatus 70 can be
a device such as the tablet, the laptop PC, the notebook
PC, or the wearable device. Furthermore, as depicted in
FIG. 5, the feedback apparatus 70 mainly has a haptic
mechanism 710, a display section 714, an audio output
section 716, a main control section 740, a communication
section 760, and a storage section 770. Details of the
functional sections of the feedback apparatus 70 will be
described hereinafter.
[0083]
(Haptic mechanism 710)
The haptic mechanism 710 is an apparatus that
propagates a sense of force (tactile sensation), for
example, applies a force to a part (for example, a joint)
of the body of the learner on the basis of the feedback
information transmitted from the server 30. For example,
the haptic mechanism can be a glove type wearable device
(wearable apparatus) attached to a hand of the learner.
Furthermore, in the present embodiment, the haptic
mechanism 710 is not limited to the glove type wearable
57
device and may be, for example, a vibration apparatus
that applies a vibration to a part of the body of the
learner or a stimulation apparatus that uses electrical
muscle stimulation to give a stimulus to muscles. In
other words, in the present embodiment, it is sufficient
if the haptic mechanism 710 is capable of sensorily
feeding back the feedback information to the learner
(performing bio-feedback) by giving a tactile stimulus to
a part of the body of the learner.
[0084]
(Display section 714)
The display section 714 is a device for displaying
the feedback information to the user, and outputs the
feedback information toward the user by, for example,
images or light. The display section 714 is realized by,
for example, a display, a light-emitting element (not
depicted), and the like. Furthermore, the display section
714 may be realized by a video output terminal and the
like.
[0085]
Moreover, the display section 714 may be a
projection apparatus that can display an object based on
feedback information 610 by superimposing the object onto
a real space as augmented reality (AR). Such a projection
58
apparatus can be, for example, a smart glass type
wearable device attached in front of learner's eyes. A
transmissive display is provided in the smart glass type
wearable device, and the transmissive display holds a
virtual image optical system including a transparent
light guide section and the like in front of the
learner's eyes using, for example, a half-mirror and a
transparent light guide plate, and displays the object
inside of the virtual image optical system. Furthermore,
the projection apparatus may be an HMD attached to the
learner's head.
[0086]
(Audio output section 716)
The audio output section 716 is a device that
outputs the feedback information to the user as audio,
and may be, for example, a headphone speaker attached to
a learner's ear or a speaker (not depicted) provided in
the vicinity of the learner. Furthermore, the audio
output section 716 may be realized by an audio output
terminal and the like.
[0087]
In this way, in the present embodiment, means
corresponding to a suited sensation modality is selected
from among the haptic mechanism 710, the display section
59
714, and the audio output section 716 according to the
feedback information and the like and the user's
situations and the like, and the feedback information can
be fed back to the learner or the like. Moreover, in the
present embodiment, the haptic mechanism 710, the display
section 714, and the audio output section 716 may
simultaneously perform feedback by a plurality of
sensation modalities, and a method of the feedback is not
limited to a specific method.
[0088]
(Main control section 740)
The main control section 740 is provided in the
feedback apparatus 70 and can exercise control over the
blocks in the feedback apparatus 70. The main control
section 740 is realized by hardware which is, for example,
a CPU, a ROM, and a RAM.
[0089]
(Communication section 760)
The communication section 760 can transmit and
receive information to and from an external apparatus
such as the server 30. It is noted that the communication
section 760 is realized by communication devices such as
a communication antenna, a transmitting-receiving circuit,
and a port.
60
[0090]
(Storage section 770)
[CLAIMS]
[Claim 1]
An information processing apparatus comprising:
a conversion section that converts a plurality of
pieces of sensing data in different forms obtained from a
plurality of sensors each sensing a state related to a
performance by a motion of a user;
an information processing section that processes
the sensing data converted by the conversion section; and
an information output section that outputs feedback
information to the user on a basis of a processing result
of the information processing section, wherein
the conversion section includes
an analog-digital signal conversion section
that converts the sensing data in an analog form from the
sensors into sensing data in a digital form and outputs
the sensing data in the digital form to the information
processing section, and
a digital-analog signal conversion section
that converts the sensing data in the digital form from
the sensors into sensing data in the analog form and
outputs the sensing data in the analog form to the
analog-digital signal conversion section.
[Claim 2]
112
The information processing apparatus according to
claim 1, wherein
the analog-digital signal conversion section has a
first clock mechanism for synchronizing pieces of sensing
data in the analog form from a first sensor that outputs
the pieces of sensing data.
[Claim 3]
The information processing apparatus according to
claim 2, further comprising:
a second clock mechanism that is provided closer to
a second sensor that outputs pieces of sensing data in
the digital form for synchronizing the pieces of sensing
data from the second sensor.
[Claim 4]
The information processing apparatus according to
claim 3, wherein
the information processing section synchronizes the
pieces of sensing data from the first sensor with the
pieces of sensing data from the second sensor
on a basis of a temporal difference between the
first clock mechanism and the second clock mechanism.
[Claim 5]
The information processing apparatus according to
claim 3, wherein
113
the digital-analog signal conversion section
converts the sensing data from the second sensor into
sensing data in the analog form, and outputs the sensing
data in the analog form to the first sensor as a trigger
signal for driving the first sensor.
[Claim 6]
The information processing apparatus according to
claim 3, wherein
the first sensor includes at least one of
an acceleration sensor, a gyrosensor, an angular
velocity sensor, a vibration sensor, a pressure sensor, a
biological information sensor, a bending sensor, or a
position sensor attached to a body of the user,
a pressure sensor or a photoreflector sensor that
is mounted on an object moving by a motion of the user
and that senses a movement of the object,
an imaging apparatus that images the user, or
a sound pick-up apparatus that senses a sound
generated by the performance.
[Claim 7]
The information processing apparatus according to
claim 6, wherein
the biological information sensor senses at least
one of a heartbeat, a pulse, a brain wave, a respiration,
114
a perspiration, a myoelectric potential, a skin
temperature, a skin electrical resistance, an eyeball
movement, a pupil diameter, or a nuclear magnetic
resonance of the user.
[Claim 8]
The information processing apparatus according to
claim 6, wherein
the object includes an acoustic musical instrument
or an electronic musical instrument.
[Claim 9]
The information processing apparatus according to
claim 3, wherein
the second sensor includes at least one of
an acceleration sensor, a gyrosensor, an angular
velocity sensor, a vibration sensor, a pressure sensor, a
bending sensor, or a position sensor attached to a body
of the user, or
an electronic musical instrument used by the user.
[Claim 10]
The information processing apparatus according to
claim 1, further comprising:
a storage section that stores pieces of the sensing
data output from the digital-analog signal conversion
section while linking the pieces of the sensing data to
115
pieces of attribute information regarding the pieces of
the sensing data.
[Claim 11]
The information processing apparatus according to
claim 10, wherein
the attribute information contains at least one of
attribute information associated with the user, attribute
information associated with the performance, feature
variables characterizing the performance, evaluation
information for the performance, attribute information
associated with the sensing, or sensitivity evaluation
information for the performance.
[Claim 12]
The information processing apparatus according to
claim 11, wherein
the information processing section has an analysis
section that analyzes the sensing data output from the
digital-analog signal conversion section and extracts
feature variables characterizing a state of the
performance.
[Claim 13]
The information processing apparatus according to
claim 11, wherein
the sensitivity evaluation information for the
116
performance is input by the user every time the
performance is given.
[Claim 14]
The information processing apparatus according to
claim 11, wherein
the sensitivity evaluation information for the
performance is acquired on a basis of the state related
to the performance by referring to a relation between the
state related to the performance and a sensitivity
evaluation obtained by machine learning in advance.
[Claim 15]
The information processing apparatus according to
claim 11, wherein
the evaluation information for the performance
includes proportions of numerical values related to
different types of the evaluation information for the
performance, the different types of the evaluation
information being in a trade-off relation.
[Claim 16]
The information processing apparatus according to
claim 10, wherein
the information processing section has a comparison
section that selects, as training data, one or a
plurality of pieces of the sensing data from among the
117
plurality of pieces of the sensing data stored in the
storage section and compares the selected training data
with comparison data that is the sensing data which is
related to the performance of the user and which is newly
sensed.
[Claim 17]
The information processing apparatus according to
claim 16, wherein
the comparison section compares the training data
with the comparison data by calculating a difference
between the training data and the comparison data or by
superimposing the training data on the comparison data.
[Claim 18]
The information processing apparatus according to
claim 17, wherein
the information output section outputs, as the
feedback information, the difference on which emphasis or
reducing processing has been performed.
[Claim 19]
The information processing apparatus according to
claim 18, wherein
the information output section outputs the feedback
information via at least one of a display apparatus, a
wearable apparatus attached to a body of the user, or an
118
audio output apparatus.
[Claim 20]
The information processing apparatus according to
claim 19, wherein
the information processing section changes at least
one of the sensing data used as the training data, a
method of comparison by the comparison section, the
processing performed on the difference, or the apparatus
that outputs the feedback information
on a basis of a change in a state of the
performance of the user.