FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENT RULES, 2003
COMPLETE SPECIFICATION
(See Section 10 and Rule 13)
Title of invention:
SYSTEMS AND METHODS FOR MIND RELAXATION USING
GRAMMAR CONSTRAINED MUSIC GENERATION THROUGH
FREEFORM INTERACTION
Applicant
Tata Consultancy Services Limited
A company Incorporated in India under the Companies Act, 1956
Having address:
Nirmal Building, 9th floor,
Nariman point, Mumbai 400021,
Maharashtra, India
Preamble to the description:
The following specification particularly describes the invention and the
manner in which it is to be performed.
2
CROSS-REFERENCE TO RELATED APPLICATIONS AND PRIORITY
[001] The present application claims priority from Indian provisional
application no. 202021046190, filed on October 22, 2020. The entire contents of
the aforementioned application are incorporated herein by reference.
5
TECHNICAL FIELD
[002] The disclosure herein generally relates to the field of improving
mental health, and, more particularly, to systems and methods for mind relaxation
using grammar constrained music generation through freeform interaction.
10
BACKGROUND
[003] In present scenario, mental health has become a concern of
paramount importance. A person’s mental well-being is indicated by presence of
positive emotions such as happiness, contentment, self-confidence and absence of
15 negative conditions like stress, anxiety, and depression. Aspects such as life
fulfilment, life satisfaction and resilience to deal with and recover from setbacks
also affect a person’s mental well-being. Anxiety is one of the important factors
which negatively effects a person’s well-being and mental health. Bad mood, sleep
disturbances, upset stomach, headache and disturbed relationships with family and
20 friends are examples of common mental health-related problems.
[004] Music has been used as a therapy in treating various mental health
problems like depression, dementia, autism, schizophrenia, anxiety, insomnia as
well as substance abuse. Similarly, art as a therapy facilitates an individual to cope
with emotional conflicts and enhance their self-awareness. Integration of music and
25 art therapies combines positive effects of both the therapies providing an overall
effective therapy for mind relaxation. Few conventional systems which combine
music and art therapies require special hardware setups leading to increased cost
and complexity.
30 SUMMARY
3
[005] Embodiments of the present disclosure present technological
improvements as solutions to one or more of the above-mentioned technical
problems recognized by the inventors in conventional systems. For example, in one
embodiment, a processor implemented method is provided. The method
5 comprising receiving, a gesture input from a user on a user interface enabling the
user to create a freeform interaction in real time; generating, via one or more
hardware processors, a sound based on the freeform interaction on the user
interface; mapping, via the one or more hardware processors, (i) a first coordinate
of each point of the freeform interaction to a corresponding unconstrained pitch
10 value to provide a plurality of mapped unconstrained pitch values and (ii) an
instantaneous rate of change of the first coordinate and a second coordinate of each
point of the freeform interaction to a corresponding energy value of the generated
sound; constraining, via the one or more hardware processors, the plurality of
mapped unconstrained pitch values of the generated sound based on a pre-specified
15 grammar to obtain a plurality of constrained pitch values corresponding to the
freeform interaction. In an embodiment, the step of constraining the plurality of
mapped unconstrained pitch values further comprising: identifying a plurality of
silence segments in the plurality of mapped unconstrained pitch values, wherein
each silence segment is indicative of a segment of one or more pitch values from
20 the plurality of mapped unconstrained pitch values having corresponding energy
values below a pre-specified threshold; identifying a plurality of stable segments in
the plurality of mapped unconstrained pitch values, wherein each stable segment is
indicative of a segment of the plurality of mapped unconstrained pitch values with
a predefined minimum duration having one or more corresponding pitch values
25 within a predefined range from a mean pitch value of the segment; identifying, a
plurality of local extrema from the plurality of mapped unconstrained pitch values
between each pair of successive stable segments from the plurality of stable
segments; quantizing, based on a nature of the plurality of mapped unconstrained
pitch values, a first stable segment from the plurality of stable segments to a nearest
30 value in a set of anchors pre-defined in the pre-specified grammar; quantizing a first
4
local extremum from the plurality of local extrema to a nearest value in a set of prespecified extrema associated with the first quantized stable segment; iteratively
performing, based on the nature of plurality of mapped unconstrained pitch values,
quantization of one or more succeeding local extrema, wherein the quantization of
5 the one or more succeeding local extrema alternates between quantization to (i) a
nearest anchor and (ii) a nearest pre-defined extremum associated with the nearest
anchor pre-defined in the pre-specified grammar; and iteratively performing
quantization for the plurality of stable segments and the plurality of local extrema
between the plurality of stable segments till the music corresponding to the freeform
10 interaction is generated. The method further comprising synthesizing, via the one
or more hardware processors, the plurality of constrained pitch values to generate a
music corresponding to the freeform interaction using an interpolation technique
and a musical synthesis technology.
[006] In another aspect, a system for mind relaxation using grammar
15 constrained music generation through freeform interaction is provided. The system
comprising a memory storing instructions; one or more communication interfaces;
and one or more hardware processors coupled to the memory via the one or more
communication interfaces, wherein the one or more hardware processors are
configured by the instructions to receive, a gesture input from a user on a user
20 interface enabling the user to create a freeform interaction in real time; generate,
via one or more hardware processors, a sound based on the freeform interaction on
the user interface; map, via the one or more hardware processors, (i) a first
coordinate of each point of the freeform interaction to a corresponding
unconstrained pitch value to provide a plurality of mapped unconstrained pitch
25 values and (ii) an instantaneous rate of change of the first coordinate and a second
coordinate of each point of the freeform interaction to a corresponding energy value
of the generated sound; constrain, via the one or more hardware processors, the
plurality of mapped unconstrained pitch values of the generated sound based on a
pre-specified grammar to obtain a plurality of constrained pitch values
30 corresponding to the freeform interaction. In an embodiment, the step of
5
constraining the plurality of mapped unconstrained pitch values further comprising:
identifying a plurality of silence segments in the plurality of mapped unconstrained
pitch values, wherein each silence segment is indicative of a segment of one or more
pitch values from the plurality of mapped unconstrained pitch values having
5 corresponding energy values below a pre-specified threshold; identifying a plurality
of stable segments in the plurality of mapped unconstrained pitch values, wherein
each stable segment is indicative of a segment of the plurality of mapped
unconstrained pitch values with a predefined minimum duration having one or more
corresponding pitch values within a predefined range from a mean pitch value of
10 the segment; identifying, a plurality of local extrema from the plurality of mapped
unconstrained pitch values between each pair of successive stable segments from
the plurality of stable segments; quantizing, based on a nature of the plurality of
mapped unconstrained pitch values, a first stable segment from the plurality of
stable segments to a nearest value in a set of anchors pre-defined in the pre-specified
15 grammar; quantizing a first local extremum from the plurality of local extrema to a
nearest value in a set of pre-specified extrema associated with the first quantized
stable segment; iteratively performing, based on the nature of plurality of mapped
unconstrained pitch values, quantization of one or more succeeding local extrema,
wherein the quantization of the one or more succeeding local extrema alternates
20 between quantization to (i) a nearest anchor and (ii) a nearest pre-defined extremum
associated with the nearest anchor pre-defined in the pre-specified grammar; and
iteratively performing quantization for the plurality of stable segments and the
plurality of local extrema between the plurality of stable segments till the music
corresponding to the freeform interaction is generated. The system is further
25 configured to synthesize, via the one or more hardware processors, the plurality of
constrained pitch values to generate a music corresponding to the freeform
interaction using an interpolation technique and a musical synthesis technology.
[007] In yet another aspect, a non-transitory computer readable medium is
provided. The non-transitory computer readable medium comprising receiving, a
30 gesture input from a user on a user interface enabling the user to create a freeform
6
interaction in real time; generating, via one or more hardware processors, a sound
based on the freeform interaction on the user interface; mapping, via the one or
more hardware processors, (i) a first coordinate of each point of the freeform
interaction to a corresponding unconstrained pitch value to provide a plurality of
5 mapped unconstrained pitch values and (ii) an instantaneous rate of change of the
first coordinate and a second coordinate of each point of the freeform interaction to
a corresponding energy value of the generated sound; constraining, via the one or
more hardware processors, the plurality of mapped unconstrained pitch values of
the generated sound based on a pre-specified grammar to obtain a plurality of
10 constrained pitch values corresponding to the freeform interaction. In an
embodiment, the step of constraining the plurality of mapped unconstrained pitch
values further comprising: identifying a plurality of silence segments in the plurality
of mapped unconstrained pitch values, wherein each silence segment is indicative
of a segment of one or more pitch values from the plurality of mapped
15 unconstrained pitch values having corresponding energy values below a prespecified threshold; identifying a plurality of stable segments in the plurality of
mapped unconstrained pitch values, wherein each stable segment is indicative of a
segment of the plurality of mapped unconstrained pitch values with a predefined
minimum duration having one or more corresponding pitch values within a
20 predefined range from a mean pitch value of the segment; identifying, a plurality of
local extrema from the plurality of mapped unconstrained pitch values between
each pair of successive stable segments from the plurality of stable segments;
quantizing, based on a nature of the plurality of mapped unconstrained pitch values,
a first stable segment from the plurality of stable segments to a nearest value in a
25 set of anchors pre-defined in the pre-specified grammar; quantizing a first local
extremum from the plurality of local extrema to a nearest value in a set of prespecified extrema associated with the first quantized stable segment; iteratively
performing, based on the nature of plurality of mapped unconstrained pitch values,
quantization of one or more succeeding local extrema, wherein the quantization of
30 the one or more succeeding local extrema alternates between quantization to (i) a
7
nearest anchor and (ii) a nearest pre-defined extremum associated with the nearest
anchor pre-defined in the pre-specified grammar; and iteratively performing
quantization for the plurality of stable segments and the plurality of local extrema
between the plurality of stable segments till the music corresponding to the freeform
5 interaction is generated. The method further comprising synthesizing, via the one
or more hardware processors, the plurality of constrained pitch values to generate a
music corresponding to the freeform interaction using an interpolation technique
and a musical synthesis technology.
[008] In an embodiment, the freeform interaction includes one or more of
10 doodling, drawing, sketching and scribbling.
[009] In an embodiment, the pre-specified grammar is derived from
histograms of the plurality of stable segments and the plurality of local extrema.
[010] In an embodiment, the nature of the plurality of mapped
unconstrained pitch values is indicative of an upward and a downward movement
15 of the plurality of mapped unconstrained pitch values.
[011] In an embodiment, the step of constraining is performed for a
plurality of filtered pitch values obtained by filtering the plurality of mapped
unconstrained pitch values between one or more successive silence segments of the
plurality of silence segments using a low pass filter.
20 [012] In an embodiment, the generated music comprises continuous pitch
variations indicative of continuous nature of the freeform interaction.
[013] It is to be understood that both the foregoing general description and
the following detailed description are exemplary and explanatory only and are not
restrictive of the invention, as claimed.
25
BRIEF DESCRIPTION OF THE DRAWINGS
[014] The accompanying drawings, which are incorporated in and
constitute a part of this disclosure, illustrate exemplary embodiments and, together
with the description, serve to explain the disclosed principles:
8
[015] FIG. 1 illustrates an exemplary block diagram of a system for mind
relaxation using grammar constrained music generation through freeform
interaction, according to some embodiments of the present disclosure.
[016] FIG. 2 is a flowchart illustrating method for mind relaxation using
5 grammar constrained music generation through freeform interaction, according to
some embodiments of the present disclosure.
[017] FIG. 3 illustrates coordinate system of a user interface of the system
of FIG. 1, according to some embodiments of present disclosure.
[018] FIGS. 4A and 4B illustrate a first example of freeform interaction
10 with corresponding unconstrained pitch and energy curves, according to some
embodiments of the present disclosure.
[019] FIGS. 5A and 5B illustrate a second example of freeform interaction
with corresponding unconstrained pitch and energy curves, according to some
embodiments of the present disclosure.
15 [020] FIG. 6 shows a graph illustrating examples of a plurality of stable
segments and a plurality of local extrema identified in the plurality of mapped
unconstrained pitch values, according to some embodiments of the present
disclosure.
[021] FIG. 7 is a graph illustrating an example of instances of continuous
20 pitch variations conforming to a well-known grammar, in accordance with some
embodiments of the present disclosure.
[022] FIGS. 8A and 8B illustrate an example of freeform interaction with
corresponding constrained pitch and energy curves generated using the method 200,
according to some embodiments of the present disclosure.
25
DETAILED DESCRIPTION OF EMBODIMENTS
[023] Exemplary embodiments are described with reference to the
accompanying drawings. In the figures, the left-most digit(s) of a reference number
identifies the figure in which the reference number first appears. Wherever
30 convenient, the same reference numbers are used throughout the drawings to refer
9
to the same or like parts. While examples and features of disclosed principles are
described herein, modifications, adaptations, and other implementations are
possible without departing from the scope of the disclosed embodiments. It is
intended that the following detailed description be considered as exemplary only,
5 with the true scope being indicated by the following claims.
[024] In general, music as well as art both have been long recognized for
their therapeutic benefits in maintaining and enhancing mental health. Music not
only helps in enhancing general wellbeing it helps combat several mental health
issues. Similarly, art therapy enables individuals to cope with their emotional
10 conflicts and allows self-expression. Doodling in art therapy has especially been
recognized for its various benefits ranging from catharsis to improvement in
memory.
[025] The present disclosure is directed to a system that amalgamates
music and art such as doodling which further helps in mind relaxation of a person.
15 The typical interpretation of results obtained from conventional music generation
systems has been modified to solve a problem of integrating art and music where
freeform interactions to generate musical sounds are not allowed. Further,
conventional systems integrating music and art in the form of doodling have
explored physical, digital as well as hybrid combinations. This is primarily because
20 doodling as well as music both can be generated physically as well as digitally and
users have specific preferences. The present disclosure provides a system for mind
relaxation using grammar constrained music generation through freeform
interaction. In an embodiment, current state of the user determines an initial doodle.
The music generated thus affects upcoming emotional states of the user. For
25 instance, the doodles generated by a person in a relaxed state will be different from
the ones generated in an anxious or hurried state. From a neurological perspective,
if a person is in a negative valence state, the initial doodle generates a piece of
music. This music should trigger Mu rhythms in the brain which in turn enables the
person to have a soothing mental state. This thus creates a closed neurofeedback
10
loop/pathway which is driven by doodling followed by music which in turn affects
the doodling.
[026] In the context of the present disclosure, the expressions ‘sound’, and
‘music’ have been used differently. A sound may not necessarily be musical, but a
5 music will always have a sound. It is observed that some constraints are clearly
needed for generated audio to sound musical. In an embodiment, the generated
audio refers to acoustic, mechanical, or electrical frequencies corresponding to
normally audible sound waves which are of frequencies approximately from 15 to
20,000 hertz. The method of the present disclosure adapts some constraints of
10 musical grammar from some of well-known music genre such as Hindustani and
Carnatic music in detail. Although further description of the present disclosure is
directed to music generation by converting freeform interactions such as doodle,
drawing, scribble, sketching to music, it may be noted that the described application
is non- limiting and systems and methods of the present disclosure may be extended
15 to convert various dance forms to the music by localizing bodily movements during
dance in a 3D plane and time using a Kinect sensor and converting the localized
bodily movements to generate the music. In an embodiment, the system of the
present disclosure stops generating the music when the drawing is stopped from the
user. However, the system of the present disclosure can be improved to keep
20 generating music through extrapolating an already generated music by learning an
inherent pattern structure of the freeform interaction and assigning a suitable
emotion state to the learnt inherent pattern structures.
[027] Referring now to the drawings, and more particularly to FIG. 1
through FIG. 8B, where similar reference characters denote corresponding features
25 consistently throughout the figures, there are shown preferred embodiments and
these embodiments are described in the context of the following exemplary system
and/or method.
[028] FIG. 1 illustrates an exemplary block diagram of a system
implementing method for mind relaxation using grammar constrained music
30 generation through freeform interaction, according to some embodiments of the
11
present disclosure. In an embodiment, the system 100 includes processor (s) 104,
communication interface device(s), alternatively referred as input/output (I/O)
interface(s) 106 or user interface 106, and one or more data storage devices or
memory 102 operatively coupled to the processor (s) 104. The processor (s)
5 alternatively referred as one or more processors 104 may be one or more software
processing modules and/or hardware processors. In an embodiment, the hardware
processors can be implemented as one or more microprocessors, microcomputers,
microcontrollers, digital signal processors, central processing units, state machines,
logic circuitries, component modelers, and/or any devices that manipulate
10 signals/inputs based on operational instructions. Among other capabilities, the
processor(s) is configured to fetch and execute computer-readable instructions
stored in the memory. In an embodiment, the system 100 can be implemented in a
variety of computing systems, such as laptop computers, notebooks, hand-held
devices, workstations, mainframe computers, servers, a network cloud and the like.
15 [029] The I/O interface(s) 106 can include a variety of software and
hardware interfaces, for example, a web interface, a graphical user interface, and
the like and can facilitate multiple communications within a wide variety of
networks N/W and protocol types, including wired networks, for example, LAN,
cable, etc., and wireless networks, such as WLAN, cellular, or satellite. In an
20 embodiment, the I/O interface device(s) can include one or more ports for
connecting a number of devices to one another or to another server. The I/O
interface 106, through the ports can be configured to receive inputs stored external
to the system 100. In an embodiment, the I/O interface device(s) 106 enables a user
to create freeform interactions and transfers pixel coordinates of the freeform
25 interactions to the one or more hardware processors 104. The I/O interface device(s)
106 also provides a music generated by the hardware processors 104 as an output
to the user.
[030] The memory 102 may include any computer-readable medium
known in the art including, for example, volatile memory, such as static random
30 access memory (SRAM) and dynamic random access memory (DRAM), and/or
12
non-volatile memory, such as read only memory (ROM), erasable programmable
ROM, flash memories, hard disks, optical disks, and magnetic tapes. In an
embodiment, the memory 102 includes a data repository 110 for storing data
processed, received, and generated as output(s) by the system 100.
5 [031] The data repository 110, amongst other things, includes a system
database. In an embodiment, the data repository 110 may be external (not shown)
to the system 100 and accessed through the I/O interfaces 106. The memory 102
may further comprise information pertaining to input(s)/output(s) of each step
performed by the processor 104 of the system 100 and methods of the present
10 disclosure. In an embodiment, the system database stores information pertaining to
inputs fed to the system 100 and/or outputs generated by the system (e.g., at each
stage), specific to the methodology described herein. More specifically, the system
database stores information being processed at each step of the proposed
methodology.
15 [032] In an embodiment, the one or more hardware processors 104 can be
configured to perform a method for mind relaxation using grammar constrained
music generation through freeform interaction, which can be carried out by using
methodology, described in conjunction with FIG. 2, and use case examples.
[033] FIG. 2 is a flowchart illustrating method for mind relaxation using
20 grammar constrained music generation through freeform interaction, according to
some embodiments of the present disclosure. In an embodiment, the system 100
comprises one or more data storage devices or the memory 102 operatively coupled
to the one or more processors 104 and is configured to store instructions for
execution of steps of the method 200 by the one or more processors 104. The steps
25 of the method 200 of the present disclosure will now be explained with reference
to the components or blocks of the system 100 as depicted in FIG. 1 and the steps
of flow diagram as depicted in FIG. 2. Although process steps, method steps,
techniques or the like may be described in a sequential order, such processes,
methods and techniques may be configured to work in alternate orders. In other
30 words, any sequence or order of steps that may be described does not necessarily
13
indicate a requirement that the steps to be performed in that order. The steps of
processes described herein may be performed in any order practical. Further, some
steps may be performed simultaneously.
[034] Referring to the steps of the method 200 depicted in FIG. 2, in an
5 embodiment of the present disclosure, at step 202, the one or more hardware
processors 104 are configured to receive, a gesture input from a user on a user
interface 106 enabling the user to create a freeform interaction in real time. In the
context of present disclosure, the expressions ‘user interface’ and ‘screen’ may be
used interchangeably. In an embodiment, freeform interaction may be referred as
10 doodling, drawing, sketching and scribbling. In another embodiment, the freeform
interaction refers to a simple drawing comprising arcs, lines and the like drawn by
the user in a random manner. In an embodiment, the freeform interaction is an art
which is performed by the user for mind relaxation further helping to manage
his/her stress level. In an embodiment, the gesture input to the user interface 106
15 can be provided through various modalities including but not limited to a limb
gesture tracking through Kinect type sensors, tongue tip tracking, eye ball
movement tracking through IR eye trackers or electroencephalograms, drawing
doodles mentally using brain computer interfaces (BCI) through
Electroencephalogram or functional Magnetic Resonance Imaging (fMRI)s. Here,
20 use of BCIs can aid people with physical abnormalities to use the system of the
present disclosure. Further, use of the eyeball movement tracking can aid patients
with Amyotrophic Lateral Sclerosis (ALS) to use the system of the present
disclosure as eyeballs movement is only voluntary control left with such patients.
Further, in an embodiment of the present disclosure, at step 204, the one or more
25 hardware processors 104 are configured to generate a sound based on the freeform
interaction on the user interface 106. In an embodiment, the sound is generated as
a result of a random movement of the gesture input (e.g., movement of a cursor) on
the user interface 106. In an embodiment of the present disclosure, the generated
sound is not musical.
14
[035] In an embodiment of the present disclosure, at step 206 of FIG. 2,
the one or more hardware processors 104 are configured to map, (i) a first
coordinate of each point of the freeform interaction to a corresponding
unconstrained pitch value to provide a plurality of mapped unconstrained pitch
5 values and (ii) an instantaneous rate of change of the first coordinate and a second
coordinate of each point of the freeform interaction to a corresponding energy value
of the generated sound. In an embodiment, the first coordinate corresponds to a
vertical movement and the second coordinate corresponds to a horizontal
movement of the gesture input on the user interface. Assuming a scenario in which
10 the user doodle on the user interface 106 or the screen without any constraints. FIG.
3 illustrates coordinate system of a user interface 106, according to some
embodiments of present disclosure. As the user doodles, coordinates of the location
of gesture inputs (Hereafter referred as cursor coordinates throughout the
description) on the user interface change with time 𝑡. As depicted in FIG. 3, ℎ(𝑡)
15 denotes height of the doodle from the top edge of the user interface 106 and 𝑤(𝑡)
denotes width of the doodle from the left edge of the user interface 106. In an
embodiment, height of the user interface 106 is 𝐻 and width of the user interface
106 is 𝑊, as shown in FIG. 3. Thus, value of ℎ(𝑡) lies between 0 and H, and value
of 𝑤(𝑡) lies between 0 and 𝑊. Further, the first coordinate of the freeform
20 interaction at a particular time instant 𝑡 denoted by 𝑥(𝑡) and the second coordinate
of the freeform interaction at a particular time instant t denoted by 𝑦(𝑡) are
calculated as shown in equation (1) and (2):
x(t) =
w(t)−W/2
W/2
…………. (1)
y(t) =
h(t)−H/2
H/2
................................... (2)
25 Here, center of the user interface 106 is considered as origin of the cursor
coordinates. The first coordinate 𝑥(𝑡) of the freeform interaction at a particular time
instant 𝑡 and
the instantaneous rate of change of the first coordinate and the second coordinate
𝑦(𝑡) of the freeform interaction at a particular time instant 𝑡 are mapped to the
15
corresponding unconstrained pitch value and corresponding energy value of the
generated sound respectively using equations (3) and (4) provided below:
𝑝(𝑡) = 𝑦(𝑡) ∗ 𝑃 − 𝑃0……………..(3)
𝑒(𝑡) = (
𝑑𝑥(𝑡)
𝑑𝑡 )
2
+ (
𝑑𝑦(𝑡)
𝑑𝑡 )
2
………..(4)
5 Here, 𝑒(𝑡) refers to corresponding energy and 𝑝(𝑡) refers to unconstrained pitch
value of generated sound at a particular time instant 𝑡. Further, in an example of the
present disclosure, a range of the unconstrained pitch 𝑃 = 17 semitones and an
offset in the unconstrained pitch 𝑃0 = 5 semitones are set for an user interface 106
of size 17 inches. FIGS. 4A and 4B illustrate a first example of freeform interaction
10 with corresponding unconstrained pitch and energy curves, according to some
embodiments of the present disclosure. In FIG. 4A, the first example of freeform
interaction consisting of approximately horizontal lines is shown and in FIG. 4B,
the corresponding curves for unconstrained pitch in semitones and energy
(pixels/sec)2
are shown. As a result of the mapping performed in step 206, it is
15 observed in FIG. 4A and 4B that moving a cursor horizontally to form a doodle
generates a music with constant pitch value and moving the cursor vertically
generates a music with different pitch values. Further, faster the cursor moves,
louder is the music generated and vice versa. However, not moving the cursor does
not produce any sound, thus speed of movement of the cursor is mapped to
20 amplitude of the generated sound.
[036] Further, in an embodiment of the present disclosure, at step 208 of
FIG. 2, the one or more hardware processors 104 are configured to constrain, a
plurality of mapped unconstrained pitch values of the generated sound based on a
pre-specified grammar to obtain a plurality of constrained pitch values
25 corresponding to the freeform interaction. FIGS. 5A and 5B illustrate a second
example of freeform interaction with corresponding unconstrained pitch and energy
curves, according to some embodiments of the present disclosure. The second
example of freeform interaction as shown in FIG. 5A represents a random doodle
starting from top-left and proceeding to, in order, bottom-right, bottom-left, and
30 top-right of the user interface 106. However, it was observed that mapped
16
unconstrained pitch values and energy value of the generated sound for the first and
second example of freeform interaction as above when synthesized as a pitch curve
and an energy curve do not provide a pleasant music. Unpleasantness was perceived
by listeners that were not familiar with music, and even more so by listeners familiar
5 with music. In addition, effect of the generated sound on mental state of the user is
not predictable. Thus, the plurality of mapped unconstrained pitch values of the
generated sound is required to be constrained to provide a pleasant music so as to
relax mind of a person and reduce his/her stress level.
[037] In an embodiment, the step of constraining the plurality of mapped
10 unconstrained pitch values further comprising identifying a plurality of silence
segments in the plurality of mapped unconstrained pitch values, wherein each
silence segment is indicative of a segment of one or more pitch values from the
plurality of mapped unconstrained pitch values having corresponding energy values
below a pre-specified threshold. In an embodiment, an example value of the pre15 specified threshold value is 10 pixels per second. This corresponds to an average
speed below which the system of the present disclosure stops generating music.
Further, a plurality of stable segments in the plurality of mapped unconstrained
pitch values is identified. Here, each stable segment is indicative of a segment of
the plurality of mapped unconstrained pitch values with a predefined minimum
20 duration having one or more corresponding pitch values with in a predefined range
from a mean pitch value of the segment. In the context of present disclosure,
expressions ‘stable segment’ and ‘anchor’ can be used interchangeably. In an
embodiment, the predefined minimum duration of each of the stable segment is 80
milliseconds and the one or more corresponding pitch values of each of the stable
25 segment are 0.3 semitones from the mean pitch value. Further, a plurality of local
extrema from the plurality of mapped unconstrained pitch values is identified
between each pair of successive stable segments from the plurality of stable
segments. In an embodiment, the plurality of local extrema includes a plurality of
local minima and a plurality of local maxima. FIG. 6 shows a graph illustrating
30 examples of a plurality of stable segments and a plurality of local extrema identified
17
in the plurality of mapped unconstrained pitch values, according to some
embodiments of the present disclosure. As shown in FIG. 6, rectangles show the
plurality of stable segments and circles show the plurality of local extrema.
However, not all stable segments and local extrema are marked in FIG. 6.
5 [038] Upon identifying the plurality of local extrema, a first stable
segment from the plurality of stable segments is quantized to a nearest value in a
set of anchors pre-defined in the pre-specified grammar based on a nature of the
plurality of mapped unconstrained pitch values. For example, in a penta-tonic scale
with musical key as music note C4, the set of anchors comprises music notes C4,
10 D4, E4, G4 and A5, which may be suitably extended to other octaves. In an
embodiment, the nature of the plurality of mapped unconstrained pitch values is
indicative of an upward and a downward movement of the plurality of mapped
unconstrained pitch values. In an embodiment, the upward and the downward
movement of the plurality of mapped unconstrained pitch values may be
15 alternatively referred as upward transient and downward transient respectively. In
an embodiment, the plurality of mapped unconstrained pitch values may form a
pitch curve. Thus, for the first stable segment, if the pitch curve after it rises, then
the first stable segment is quantized to the nearest value in a set of upward transients
denoted by 𝐴𝑢. Similarly, for the first stable segment, if the pitch curve after the
20 first stable segment falls, then the first stable segment is quantized to the nearest
value in a set of downward transients denoted by 𝐴𝑑. In other words, 𝐴𝑢 and
𝐴𝑑 refer to the set of anchors pre-defined in the pre-specified grammar that
comprises the plurality of mapped unconstrained pitch values having upward
movement and downward movement respectively. Further, in an embodiment of
25 the present disclosure, a first local extremum from the plurality of local extrema is
quantized to a nearest value in a set of pre-specified extrema associated with the
first quantized stable segment. For example, in one instance of a doodle, a first value
from the plurality of mapped unconstrained pitch values having upward movement
with a mean pitch of 9.11 semitones is quantized to 9 semitones, and a first local
30 extremum is a local maximum at 9.79 semitones which is quantized to 10
18
semitones. Similarly, in one instance of a doodle, a subsequent value from the
plurality of mapped unconstrained pitch values having downward movement has a
mean pitch of 7.98 semitones which is quantized to 7 semitones and a subsequent
local extremum is a local minimum at 3.45 semitones which is quantized to 2
5 semitones. Further, quantization of one or more succeeding local extrema is
iteratively performed based on the nature of plurality of mapped unconstrained
pitch values. In an embodiment, the quantization of the one or more succeeding
local extrema alternates between quantization to (i) a nearest anchor and (ii) a
nearest pre-defined extremum associated with the nearest anchor pre-defined in the
10 pre-specified grammar. Furthermore, quantization for the plurality of stable
segments and the plurality of local extrema between the plurality of stable segments
is iteratively performed till the music corresponding to the freeform interaction is
generated.
[039] In an example implementation of present disclosure, the step 208 of
15 constraining the plurality of mapped unconstrained pitch values is performed
according to following algorithm:
Identify stable segments in the mapped unconstrained pitch values pitch.
for Pitch curve between every pair of consecutive stable segments do:
Identify the locations of local maxima and minima, {𝑡𝑛}, 0 ≤ 𝑛 <
20 𝑁, where 𝑁 depends on the pitch curve.
end for
state ← TRA
for 𝑡 = {1, 2, . . . . . , 𝑁 – 1} do
if state == ANC then
25 if 𝑡𝑛 > 𝑡𝑛+1 then
𝑝𝑞
(𝑡𝑛
) ← Nearest anchor in 𝐴𝑑 for which downward
transients exist
else
𝑝𝑞
(𝑡𝑛
)← Nearest anchor in 𝐴𝑢
30 end if
19
state ← TRA
else
if 𝑡𝑛−1 > 𝑡𝑛 then
𝑇 ← The set of downward transients of 𝑝𝑞
(𝑡𝑛−1
)
5 else
𝑇 ← The set of upward transients of 𝑝𝑞
(𝑡𝑛−1
)
end if
𝑝𝑞
(𝑡𝑛
) ← Nearest transient in 𝑇
state ← ANC
10 end if
end for
In the above algorithm, for the pitch curve between successive stable segments,
concept of a state for each local maximum or minimum is used. This state can be
ANC (i.e., an anchor) or a TRA (i.e., transient). Initially, all the identified stable
15 segments are in the state ANC. It is observed from the algorithm that successive
local maxima and minima alternate between state TRA and ANC. After this state is
fixed, each local maximum or minimum is quantized to a nearest value in the set of
anchors predefined in the pre-specified grammar anchor, or to a nearest transientnote relative to a preceding anchor in the set of anchors predefined in the pre20 specified grammar.
[040] In the context of the present disclosure, the expression ‘grammar’ is
defined as position of one or more notes, mapping between the one or more notes
in an octave, oscillation about a mean pitch of the one or more notes, glide between
the one or more notes, pitch variations including upward and downward pitch
25 movements (Alternatively referred as transients) in a musical sound. In an
embodiment, the prespecified grammar may include an automatically derived
grammar and a hand-crafted grammar, wherein the automatically derived grammar
may have been learnt from well-known music genres. In the Western music, melody
is specified in terms of musical notes. Indian music is understood in terms of svaras.
30 A svara in Indian music may be rendered in several ways and simplest method is to
20
render the svara with a fixed pitch. In this case, the mapping between svara-names
and notes in Western music is straightforward. In another important method, the
svara starts at an anchor of fixed pitch and finishes with a transient. FIG. 7 is a
graph illustrating an example of instances of continuous pitch variations
5 conforming to a well-known grammar, in accordance with some embodiments of
the present disclosure. Two alternatives for the svara Ni are shown in FIG. 7 where
it starts at the anchor at musical note A, and the transient is to musical note C as
shown in (b) or musical note A# as shown in (c). Conventionally, for each identified
anchor, the transients possible from it are identified and a grammar is thus derived.
10 In practice, a few false alarms and misses are observed. Thus, a hand-crafted
grammar that is free from false alarms is considered to more useful. An example of
handcrafted grammar that approximates the Carnatic raga mohanam (equivalent
Hindustani raga: bhup) is provided in Table 1. The musical notes in table 1 are
according to Western music notation assuming a musical key (musical note S in
15 Carnatic music) corresponds to musical note C. The name of Carnatic music svaras
corresponding to the Western musical notes are provided in brackets in table 1.
Anchor Transients
Upward Downward
C (S)
D (R2)
E (G3)
G (P)
A (D2)
D# (G2)
E (G3)
F (M1), G (P)
A# (N2)
A# (N2), C (S)
A (D2)
D (R2)
E (G3)
Table 1
Some anchors have upward transients and some others have downward transients.
Some anchors may appear in both upward transients and downward transients. As
20 is common in Indian music, the skeletal description of the raga (‘Anchors’ column
shown in the table 1) may not specify the gamakas fully (e.g., G2, M1, N2 appear
only in the transients-column of table 1). This property helps in making audio for a
doodle sound more continuous. However, for the purposes of doodling, the exact
21
adherence to the raga is not required. In an embodiment, the pre-specified grammar
is derived from histograms of the plurality of stable segments and the plurality of
local extrema.
[041] In an embodiment, the step of constraining is performed for a
5 plurality of filtered pitch values 𝑝𝑓(𝑡) obtained by filtering the plurality of mapped
unconstrained pitch values between one or more successive silence segments of the
plurality of silence segments using a low pass filter. In an embodiment, the plurality
of mapped unconstrained pitch values between the one or more successive silence
segments may not necessarily be silent. In an example of present disclosure, a low10 pass Butterworth filter of order 8 with a cutoff frequency of 4Hz is used.
[042] In an embodiment of the present disclosure, it is observed that a
transient in Carnatic music rarely oscillate more than twice between the same two
notes. Also, anchors scale far more than do the transients across speed. Thus, in an
embodiment, post processing steps are performed to limit consecutive oscillations
15 between same pairs of notes to at most two instances and replace the quantized pitch
values at all the local maxima and minima preceding these oscillations by that of
the quantized anchor and duration of transients is limited to an average value of 150
ms and the anchors adjacent to the transients are extended correspondingly.
[043] Referring again to FIG. 2, in an embodiment of the present
20 disclosure, at step 210, the one or more hardware processors 104 are configured to
synthesize, the plurality of constrained pitch values to generate a music
corresponding to the freeform interaction using an interpolation technique and a
musical synthesis technology. In an embodiment, the generated music comprises
continuous pitch variations indicative of continuous nature of the freeform
interaction. In an embodiment, using the quantized values {𝑝𝑓
(𝑡𝑛 25 )}, the plurality of
constrained pitch values denoted by 𝑟(𝑡) for synthesis are generated by setting
𝑟(𝑡𝑛
) = 𝑝𝑞
(𝑡𝑛
) for 1 ≤ 𝑛 ≤ 𝑁 and then interpolating between successive
quantized values using equation (5) provided below for 𝑡𝑛−1 ≤ 𝑡 ≤ 𝑡𝑛.
𝑟(𝑡) = 𝑝𝑞
(𝑡𝑛−1
) + (𝑝𝑞
(𝑡𝑛
) − 𝑝𝑞
(𝑡𝑛−1
)) cos (𝜋
𝑡−𝑡𝑛−1
𝑡𝑛−𝑡𝑛−1
) ……..(5)
22
[044] The equation (5) is repeated for 2 ≤ 𝑛 ≤ 𝑁. FIGS. 8A and 8B
illustrate an example of freeform interaction with corresponding constrained pitch
and energy curves generated using the method 200, according to some
embodiments of the present disclosure. In equation (5), cosine-curves are used as a
5 smooth interpolation function. However, the algorithm is not sensitive to exact
interpolation function, because differences between possible interpolation functions
are not easy to perceive. The music corresponding to the freeform interaction is
then synthesized from 𝑟(𝑡) the musical synthesis technology. In an embodiment,
the musical synthesis is performed using pitch-bend commands through a Musical
10 Instrument Digital Interface (MIDI), with a pitch-bend range set as a difference
between maximum and minimum values of r(t). In an embodiment, the pitch-bend
commands are issued every 10 ms. The generated music is usually relaxing and
helps reducing stress level of a person. In an embodiment, subjective preferences
may also be accommodated by adapting the pre-specified grammar constraining the
15 music generation.
[045] The written description describes the subject matter herein to enable
any person skilled in the art to make and use the embodiments. The scope of the
subject matter embodiments is defined by the claims and may include other
modifications that occur to those skilled in the art. Such other modifications are
20 intended to be within the scope of the claims if they have similar elements that do
not differ from the literal language of the claims or if they include equivalent
elements with insubstantial differences from the literal language of the claims.
[046] The embodiments of present disclosure herein disclose systems and
methods for mind relaxation using grammar constrained music generation through
25 freeform interaction. The embodiments of present disclosure address unresolved
problem of integrating music and art to aid mental health by eliminating the need
of special and expensive hardware setups. The embodiment thus provides a system
that amalgamates music and art by mapping of the freeform interactions created
based on the gesture inputs received from the user to generate pleasant music using
30 some musical grammar constraints.
23
[047] It is to be understood that the scope of the protection is extended to
such a program and in addition to a computer-readable means having a message
therein; such computer-readable storage means contain program-code means for
implementation of one or more steps of the method, when the program runs on a
5 server or mobile device or any suitable programmable device. The hardware device
can be any kind of device which can be programmed including e.g. any kind of
computer like a server or a personal computer, or the like, or any combination
thereof. The device may also include means which could be e.g. hardware means
like e.g. an application-specific integrated circuit (ASIC), a field-programmable
10 gate array (FPGA), or a combination of hardware and software means, e.g. an ASIC
and an FPGA, or at least one microprocessor and at least one memory with software
processing components located therein. Thus, the means can include both hardware
means and software means. The method embodiments described herein could be
implemented in hardware and software. The device may also include software
15 means. Alternatively, the embodiments may be implemented on different hardware
devices, e.g. using a plurality of CPUs.
[048] The embodiments herein can comprise hardware and software
elements. The embodiments that are implemented in software include but are not
limited to, firmware, resident software, microcode, etc. The functions performed by
20 various components described herein may be implemented in other components or
combinations of other components. For the purposes of this description, a
computer-usable or computer readable medium can be any apparatus that can
comprise, store, communicate, propagate, or transport the program for use by or in
connection with the instruction execution system, apparatus, or device.
25 [049] The illustrated steps are set out to explain the exemplary
embodiments shown, and it should be anticipated that ongoing technological
development will change the manner in which particular functions are performed.
These examples are presented herein for purposes of illustration, and not limitation.
Further, the boundaries of the functional building blocks have been arbitrarily
30 defined herein for the convenience of the description. Alternative boundaries can
24
be defined so long as the specified functions and relationships thereof are
appropriately performed. Alternatives (including equivalents, extensions,
variations, deviations, etc., of those described herein) will be apparent to persons
skilled in the relevant art(s) based on the teachings contained herein. Such
5 alternatives fall within the scope of the disclosed embodiments. Also, the words
“comprising,” “having,” “containing,” and “including,” and other similar forms are
intended to be equivalent in meaning and be open ended in that an item or items
following any one of these words is not meant to be an exhaustive listing of such
item or items, or meant to be limited to only the listed item or items. It must also be
10 noted that as used herein and in the appended claims, the singular forms “a,” “an,”
and “the” include plural references unless the context clearly dictates otherwise.
[050] Furthermore, one or more computer-readable storage media may be
utilized in implementing embodiments consistent with the present disclosure. A
computer-readable storage medium refers to any type of physical memory on which
15 information or data readable by a processor may be stored. Thus, a computerreadable storage medium may store instructions for execution by one or more
processors, including instructions for causing the processor(s) to perform steps or
stages consistent with the embodiments described herein. The term “computerreadable medium” should be understood to include tangible items and exclude
20 carrier waves and transient signals, i.e., be non-transitory. Examples include
random access memory (RAM), read-only memory (ROM), volatile memory,
nonvolatile memory, hard drives, CD ROMs, DVDs, flash drives, disks, and any
other known physical storage media.
[051] It is intended that the disclosure and examples be considered as
25 exemplary only, with a true scope of disclosed embodiments being indicated by the
following claims.
25
We Claim:
1. A processor implemented method, comprising:
receiving (202), a gesture input from a user on a user interface
enabling the user to create a freeform interaction in real time;
5 generating (204), via one or more hardware processors, a sound
based on the freeform interaction on the user interface;
mapping (206), via the one or more hardware processors, (i) a first
coordinate of each point of the freeform interaction to a corresponding
unconstrained pitch value to provide a plurality of mapped unconstrained
10 pitch values and (ii) an instantaneous rate of change of the first coordinate
and a second coordinate of each point of the freeform interaction to a
corresponding energy value of the generated sound;
constraining (208), via the one or more hardware processors, the
plurality of mapped unconstrained pitch values of the generated sound based
15 on a pre-specified grammar to obtain a plurality of constrained pitch values
corresponding to the freeform interaction; and
synthesizing (210), via the one or more hardware processors, the
plurality of constrained pitch values to generate a music corresponding to
the freeform interaction using an interpolation technique and a musical
20 synthesis technology.
2. The processor implemented method of claim 1, wherein the step of
constraining the plurality of mapped unconstrained pitch values further
comprising:
25 identifying a plurality of silence segments in the plurality of mapped
unconstrained pitch values, wherein each silence segment is indicative of a
segment of one or more pitch values from the plurality of mapped
unconstrained pitch values having corresponding energy values below a prespecified threshold;
26
identifying a plurality of stable segments in the plurality of mapped
unconstrained pitch values, wherein each stable segment is indicative of a
segment of the plurality of mapped unconstrained pitch values with a
predefined minimum duration having one or more corresponding pitch
5 values within a predefined range from a mean pitch value of the segment;
identifying, a plurality of local extrema from the plurality of mapped
unconstrained pitch values between each pair of successive stable segments
from the plurality of stable segments;
quantizing, based on a nature of the plurality of mapped
10 unconstrained pitch values, a first stable segment from the plurality of stable
segments to a nearest value in a set of anchors pre-defined in the prespecified grammar;
quantizing a first local extremum from the plurality of local extrema
to a nearest value in a set of pre-specified extrema associated with the first
15 quantized stable segment;
iteratively performing, based on the nature of plurality of mapped
unconstrained pitch values, quantization of one or more succeeding local
extrema, wherein the quantization of the one or more succeeding local
extrema alternates between quantization to (i) a nearest anchor and (ii) a
20 nearest pre-defined extremum associated with the nearest anchor predefined in the pre-specified grammar; and
iteratively performing quantization for the plurality of stable
segments and the plurality of local extrema between the plurality of stable
segments till the music corresponding to the freeform interaction is
25 generated.
3. The processor implemented method of claim 1, wherein the freeform
interaction includes one or more of doodling, drawing, sketching and
scribbling.
30
27
4. The processor implemented method of claim 1, wherein the pre-specified
grammar is derived from histograms of the plurality of stable segments and
the plurality of local extrema.
5. The processor implemented method of claim 2, wherein nature of the
5 plurality of mapped unconstrained pitch values is indicative of an upward
and a downward movement of the plurality of mapped unconstrained pitch
values.
6. The processor implemented method of claim 2, wherein the step of
10 constraining is performed for a plurality of filtered pitch values obtained by
filtering the plurality of mapped unconstrained pitch values between one or
more successive silence segments of the plurality of silence segments using
a low pass filter.
15 7. The processor implemented method of claim 1, wherein the generated music
comprises continuous pitch variations indicative of continuous nature of the
freeform interaction.
8. A system (100), comprising:
20 a memory (102) storing instructions;
one or more communication interfaces (106); and
one or more hardware processors (104) coupled to the memory (102) via
the one or more communication interfaces (106), wherein the one or more
hardware processors (104) are configured by the instructions to:
25 receive, a gesture input from a user on a user interface enabling the
user to create a freeform interaction in real time;
generate a sound based on the freeform interaction on the user
interface;
map, (i) a first coordinate of each point of the freeform interaction
30 to a corresponding unconstrained pitch value to provide a plurality of
28
mapped unconstrained pitch values and (ii) an instantaneous rate of change
of the first coordinate and a second coordinate of each point of the freeform
interaction to a corresponding energy value of the generated sound;
constrain, a plurality of mapped unconstrained pitch values of the
5 generated sound based on a pre-specified grammar to obtain a plurality of
constrained pitch values corresponding to the freeform interaction; and
synthesize, the plurality of constrained pitch values to generate a
music corresponding to the freeform interaction using an interpolation
technique and a musical synthesis technology.
10
9. The system of claim 8, wherein the step of constraining the plurality of
mapped unconstrained pitch values further comprising:
identifying a plurality of silence segments in the plurality of mapped
unconstrained pitch values, wherein each silence segment is indicative of a
15 segment of one or more pitch values from the plurality of mapped
unconstrained pitch values having corresponding energy values below a prespecified threshold;
identifying a plurality of stable segments in the plurality of mapped
unconstrained pitch values, wherein each stable segment is indicative of a
20 segment of the plurality of mapped unconstrained pitch values with a
predefined minimum duration having one or more corresponding pitch
values with in a predefined range from a mean pitch value of the segment;
identifying, a plurality of local extrema from the plurality of
mapped unconstrained pitch values between each pair of successive stable
25 segments from the plurality of stable segments;
quantizing, based on a nature of the plurality of mapped
unconstrained pitch values, a first stable segment from the plurality of stable
segments to a nearest value in a set of anchors pre-defined in the prespecified grammar;
29
quantizing a first local extremum from the plurality of local extrema
to a nearest value in a set of pre-specified extrema associated with the first
quantized stable segment;
iteratively performing, based on the nature of plurality of mapped
5 unconstrained pitch values, quantization of one or more succeeding local
extrema, wherein the quantization of the one or more succeeding local
extrema alternates between quantization to (i) a nearest anchor and (ii) a
nearest pre-defined extremum associated with the nearest anchor predefined in the pre-specified grammar; and
10 iteratively performing quantization for the plurality of stable
segments and the plurality of local extrema between the plurality of stable
segments till the music corresponding to the freeform interaction is
generated.
15 10. The system of claim 8, wherein the freeform interaction includes one or
more of doodling, drawing, sketching and scribbling.
11. The system of claim 8, wherein the pre-specified grammar is derived from
histograms of the plurality of stable segments and the plurality of local
20 extrema.
12. The system of claim 9, wherein the nature of the plurality of mapped
unconstrained pitch values is indicative of an upward and a downward
movement of the plurality of mapped unconstrained pitch values.
25
13. The system of claim 9, wherein the step of constraining is performed for a
plurality of filtered pitch values obtained by filtering the plurality of mapped
unconstrained pitch values between one or more successive silence
segments of the plurality of silence segments using a low pass filter.
30
30
14. The system of claim 8, wherein the generated music comprises continuous
pitch variations indicative of continuous nature of the freeform interaction.