FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENT RULES, 2003
COMPLETE SPECIFICATION
(See Section 10 and Rule 13)
Title of invention:
MODELING A NEURONAL CONTROLLER EXHIBITING HUMAN POSTURAL
SWAY
Applicant
Tata Consultancy Services Limited A company Incorporated in India under the Companies Act, 1956
Having address:
Nirmal Building, 9th floor,
Nariman point, Mumbai 400021,
Maharashtra, India
Preamble to the description
The following specification particularly describes the invention and the manner in
which it is to be performed.

TECHNICAL FIELD
[001] The disclosure herein generally relates to postural sway analyses, and,
more particularly, to systems and computer implemented methods for modeling a neuronal controller exhibiting human postural sway.
BACKGROUND
[002] Analyzing human postural sway is critical to detect markers for
progression of balance impairments. Typically, balance impairments are caused by altered functions of the central nervous system or sensory or motor functions which may be due to age or pathology. Parkinson’s disease and peripheral neuropathy may be deciphered by analyzing the postural sway. Conventionally, a neuronal controller located inside the central nervous system governing the maintenance of the upright posture of the human body is designed from a control system perspective using one or more proportional–integral–derivative (PID) controllers, wherein human postural sway is modeled either along a sagittal plan or along a frontal plane separately resulting in limited insights on intricacies of a governing neuronal controller. Some attempts have also been made to model the neuronal controller using a reinforcement learning (RL) paradigm based on complex actor-critic on-policy algorithms.
SUMMARY
[003] Embodiments of the present disclosure present technological
improvements as solutions to one or more of the above-mentioned technical problems recognized by the inventors in conventional systems.
[004] In an aspect, there is provided a processor implemented method for
modeling a neuronal controller exhibiting human postural sway comprising the steps of: modeling, by one or more hardware processors, the neuronal controller in the form of a Reinforcement Learning (RL) agent based on an inverted pendulum with 1 Degree Of Freedom (1DOF) representing a first mechanical model of a human body

in the form of a dynamical system, wherein the RL agent is trained using a Q-learning technique to learn an optimal state-action value (Q-value) function for a tuneable Markov Decision Process (MDP) model, wherein the modeled neuronal controller is configured to reproduce Center Of Pressure (COP) characteristics in the human body either along a sagittal plane or along a frontal plane separately; deriving, by the one or more hardware processors, dynamical equations of a Spherical Inverted Pendulum (SIP) with respect to a global coordinate system, wherein the SIP represents a second mechanical model of the human body that exhibits postural sway along both the frontal plane and the sagittal plane together, wherein the dynamical equations are derived by using Lagrange’s equations with two independent state variables (θx and θy) being angular deviation of the SIP about a pivot joint and along x and y axes respectively of the global coordinate system, wherein the pivot joint characterizes an ankle joint of the human body; and modeling, by the one or more hardware processors, the human postural sway both along the sagittal plane and along the frontal plane together using the modeled neuronal controller and the derived dynamical equations of the SIP by tuning (i) a reward function comprised in the modeled neuronal controller and (ii) a set of parameters to balance the SIP such that the postural sway characteristics generated by the modeled neuronal controller match the postural sway characteristics of one or more control subjects, wherein the set of parameters include parameters of the MDP model and parameters associated with physiology of the human body.
[005] In another aspect, there is provided a system for modeling a neuronal
controller exhibiting human postural sway, the system comprising: one or more data storage devices operatively coupled to one or more hardware processors and configured to store instructions configured for execution by the one or more hardware processors to: model the neuronal controller in the form of a Reinforcement Learning (RL) agent based on an inverted pendulum with 1 Degree Of Freedom (1DOF) representing a first mechanical model of a human body in the form of a dynamical

system, wherein the RL agent is trained using a Q-learning technique to learn an optimal state-action value (Q-value) function for a tuneable Markov Decision Process (MDP) model, wherein the modeled neuronal controller is configured to reproduce Center Of Pressure (COP) characteristics in the human body either along a sagittal plane or along a frontal plane separately; derive dynamical equations of a Spherical Inverted Pendulum (SIP) with respect to a global coordinate system, wherein the SIP represents a second mechanical model of the human body that exhibits postural sway along both the frontal plane and the sagittal plane together, wherein the dynamical equations are derived by using Lagrange’s equations with two independent state variables (θx and θy) being angular deviation of the SIP about a pivot joint and along x and y axes respectively of the global coordinate system, wherein the pivot joint characterizes an ankle joint of the human body; and model the human postural sway both along the sagittal plane and along the frontal plane together using the modeled neuronal controller and the derived dynamical equations of the SIP by tuning (i) a reward function comprised in the modeled neuronal controller and (ii) a set of parameters to balance the SIP such that the postural sway characteristics generated by the modeled neuronal controller match the postural sway characteristics of one or more control subjects, wherein the set of parameters include parameters of the MDP model and parameters associated with physiology of the human body.
[006] In yet another aspect, there is provided a computer program product
comprising a non-transitory computer readable medium having a computer readable program embodied therein, wherein the computer readable program, when executed on a computing device, causes the computing device to: model the neuronal controller in the form of a Reinforcement Learning (RL) agent based on an inverted pendulum with 1 Degree Of Freedom (1DOF) representing a first mechanical model of a human body in the form of a dynamical system, wherein the RL agent is trained using a Q-learning technique to learn an optimal state-action value (Q-value) function for a tuneable Markov Decision Process (MDP) model, wherein the modeled

neuronal controller is configured to reproduce Center Of Pressure (COP) characteristics in the human body either along a sagittal plane or along a frontal plane separately; derive dynamical equations of a Spherical Inverted Pendulum (SIP) with respect to a global coordinate system, wherein the SIP represents a second mechanical model of the human body that exhibits postural sway along both the frontal plane and the sagittal plane together, wherein the dynamical equations are derived by using Lagrange’s equations with two independent state variables (θx and θy) being angular deviation of the SIP about a pivot joint and along x and y axes respectively of the global coordinate system, wherein the pivot joint characterizes an ankle joint of the human body; and model the human postural sway both along the sagittal plane and along the frontal plane together using the modeled neuronal controller and the derived dynamical equations of the SIP by tuning (i) a reward function comprised in the modeled neuronal controller and (ii) a set of parameters to balance the SIP such that the postural sway characteristics generated by the modeled neuronal controller match the postural sway characteristics of one or more control subjects, wherein the set of parameters include parameters of the MDP model and parameters associated with physiology of the human body.
[007] In accordance with an embodiment of the present disclosure, the Q-
learning technique is configured to learn to generate a torque representing an action for each state of the inverted pendulum by: receiving proprioceptive inputs in the form of angle and angular velocity of the pivot joint of the inverted pendulum; and generating the torque at the pivot joint of the inverted pendulum based on the received proprioceptive inputs to maintain an upright posture of the inverted pendulum.
[008] In accordance with an embodiment of the present disclosure, the one
or more hardware processors are configured to perform the step of modeling a neuronal controller by reproducing the Center Of Pressure (COP) characteristics in the human body either along the sagittal or along the frontal plane separately in

accordance with a dynamic equation based on overall mass of the human body
being concentrated at the point of Center Of Mass (COM) located at the 2nd lumbar
vertebrae of the human body, an approximate height L of the 2nd lumbar vertebrae,
moment of inertia of the inverted pendulum I , angle of the inverted pendulum θwith respect to the direction of gravity, gravitational acceleration g , a stiffness constant and a damping constant B , wherein the stiffness constant and the damping constant denote pivot joint properties of the inverted pendulum.
[009] In accordance with an embodiment of the present disclosure, the
parameters of the MDP model include: nθ representing resolution of states in θ
domain representing resolution of states in θ domain (from
representing resolution of states in τ domain
representing the maximum torque exertable on the pivot joint or the boundary of the τ domain; representing the boundary of the θ domain;
representing a finite boundary of the θ domain; representing property of a curve
from which torque values in the τ domain is sampled; representing property of a
curve from which boundaries of states of the θ domain is sampled; and
representing property of a curve from which boundaries of the states of the θ domain
is sampled; and the parameters associated with physiology of the human body
include: White Gaussian Noise ( WGN ) added to the generated torque to represent a
real world noisy biological system; filtering factor (λ ) added to portray signaling characteristics of a neuromuscular junction; and Scaling Factor (SF ) introduced to scale down the magnitude of the generated torque for each state of the inverted pendulum after the training of the RL agent is completed, wherein completion is of the training is represented by a balanced SIP.
[010] It is to be understood that both the foregoing general description and
the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS
[011] The accompanying drawings, which are incorporated in and constitute
a part of this disclosure, illustrate exemplary embodiments and, together with the description, serve to explain the disclosed principles.
[012] FIG.1 illustrates an exemplary block diagram of a system for modeling
a neuronal controller exhibiting human postural sway, in accordance with an embodiment of the present disclosure.
[013] FIG.2 illustrates a high level architecture enabling a method for
modeling a neuronal controller exhibiting human postural sway, in accordance with an embodiment of the present disclosure.
[014] FIG.3 illustrates an exemplary flow diagram of a computer
implemented method for modeling a neuronal controller exhibiting human postural sway, in accordance with an embodiment of the present disclosure.
[015] FIG.4 illustrates a 1 Degree Of Freedom (1DOF) inverted pendulum
representing a free body diagram of a simplified human body in the sagittal plane, in accordance with an embodiment of the present disclosure.
[016] FIG.5 illustrates a Spherical Inverted Pendulum (SIP) having 2 DOF
with two state variables, and in accordance with an embodiment of the present
disclosure.
[017] FIG.6A and FIG.6B illustrate Center Of Pressure (COP)
characteristics along x axis and a corresponding frequency spectrum respectively; and FIG.6C and FIG.6D illustrate the COP characteristics along y axis and a corresponding frequency spectrum respectively, for a set of parameters, without introducing parameters associated with physiology of the human body, in accordance with an embodiment of the present disclosure.
[018] FIG.7A and FIG.7B illustrate variance of the angle of the SIP and the
angular velocity of the SIP with respect to the x axis and the y axis respectively
FIG.7C illustrates a plot of applied torques at a pivot

joint by the neuronal controller and FIG.7D illustrates COP movement along the x axis and the y axis for the set of parameters referred in FIG.6A through FIG.6D.
[019] FIG.8A and 8B illustrate the COP characteristics along the x axis and
a corresponding frequency spectrum respectively; and FIG.8C and FIG.8D illustrate the COP characteristics along the y axis and a corresponding frequency spectrum respectively, after introducing parameters associated with physiology of the human body, in accordance with an embodiment of the present disclosure.
[020] FIG.9A and FIG.9B illustrate the variance of the angle of the SIP and
the angular velocity of the SIP with respect to the x axis and the y axis respectively w.r.t time; FIG.9C illustrates a plot of applied torques at a
pivot joint by the neuronal controller and FIG.9D illustrates COP movement along the x axis and the y axis for the parameters used with reference to FIG.8A through FIG.8D.
DETAILED DESCRIPTION OF EMBODIMENTS
[021] Exemplary embodiments are described with reference to the
accompanying drawings. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. Wherever convenient, the same reference numbers are used throughout the drawings to refer to the same or like parts. While examples and features of disclosed principles are described herein, modifications, adaptations, and other implementations are possible without departing from the spirit and scope of the disclosed embodiments. It is intended that the following detailed description be considered as exemplary only, with the true scope and spirit being indicated by the following claims.
[022] State of the art techniques for analyzing human postural sway, model
the human postural sway either along a sagittal plane or along a frontal plane separately. Modeling separately along the anatomical planes only give lower dimensional projection of higher dimensional dynamics which in turn compromises

the overall picture portraying the human postural sway. Also, the attempts were directed to designing the neuronal controller located inside the central nervous system governing the maintenance of the upright posture of the human body from a control system perspective using PID controllers. Using PID controller only gives rise to limited set of parameters with very less neurophysiological significance. Some attempts to model the neuronal controller using Reinforcement Learning (RL) paradigm were actor-critic on-policy algorithms that address optimization of two redundant parallel entities viz., the ‘policy’ and the ‘actor’ which was relatively complex and computationally inefficient.
[023] The methods and systems of the present disclosure firstly facilitate
designing the neuronal controller using a simplified RL algorithm capable of producing diverse sway characteristics either in the sagittal plane or in the front plane separately using a simple inverted pendulum representing a mechanical model of a human body. Dynamical equations of a Spherical Inverted Pendulum (SIP) are derived with respect to a global coordinate system. The modeled neuronal controller is then adapted to exhibit diverse sway characteristics in the sagittal plane and in the front plane together. The present disclosure also provides parameters which enable the sway characteristics exhibited by the modeled neuronal controller to be close to normal and pathological conditions seen in the real world. Systems and methods of the present disclosure find application in analyzing the postural balance of a subject so that subject specific rehabilitation protocols may be recommended.
[024] Referring now to the drawings, and more particularly to FIG.1 through
FIG.9D, where similar reference characters denote corresponding features consistently throughout the figures, there are shown preferred embodiments and these embodiments are described in the context of the following exemplary system and/or method.
[025] FIG.1 illustrates an exemplary block diagram of a system 100 for
modeling a neuronal controller exhibiting human postural sway, in accordance with

an embodiment of the present disclosure. In an embodiment, the system 100 includes one or more hardware processors 104, communication interface device(s) or input/output (I/O) interface(s) 106, and one or more data storage devices or memory 102 operatively coupled to the one or more processors 104. The one or more processors 104 can be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, graphics controllers, logic circuitries, and/or any devices that manipulate signals based on operational instructions. Among other capabilities, the processor(s) are configured to fetch and execute computer-readable instructions stored in the memory. In the context of the present disclosure, the expressions ‘processors’ and ‘hardware processors’ may be used interchangeably. In an embodiment, the system 100 can be implemented in a variety of computing systems, such as laptop computers, notebooks, hand-held devices, workstations, mainframe computers, servers, a network cloud and the like.
[026] The I/O interface(s) 106 can include a variety of software and
hardware interfaces, for example, a web interface, a graphical user interface, and the like and can facilitate multiple communications within a wide variety of networks N/W and protocol types, including wired networks, for example, LAN, cable, etc., and wireless networks, such as WLAN, cellular, or satellite. In an embodiment, the I/O interface(s) can include one or more ports for connecting a number of devices to one another or to another server.
[027] The memory 102 may include any computer-readable medium known
in the art including, for example, volatile memory, such as static random access memory (SRAM) and dynamic random access memory (DRAM), and/or non-volatile memory, such as read only memory (ROM), erasable programmable ROM, flash memories, hard disks, optical disks, and magnetic tapes. In an embodiment, one or more modules (not shown) of the system 100 can be stored in the memory 102.
[028] FIG.2 illustrates a high level architecture enabling a method for

modeling a neuronal controller exhibiting human postural sway, in accordance with an embodiment of the present disclosure. FIG.3 illustrates an exemplary flow diagram of a computer implemented method 300 for modeling a neuronal controller exhibiting human postural sway, in accordance with an embodiment of the present disclosure. In an embodiment, the system 100 includes one or more data storage devices or memory 102 operatively coupled to the one or more processors 104 and is configured to store instructions configured for execution of steps of the method 300 by the one or more processors 104. The steps of the method 300 will now be explained in detail with reference to the components of the system 100 of FIG.1 and the architecture of FIG.2. Although process steps, method steps, techniques or the like may be described in a sequential order, such processes, methods and techniques may be configured to work in alternate orders. In other words, any sequence or order of steps that may be described does not necessarily indicate a requirement that the steps be performed in that order. The steps of processes described herein may be performed in any order practical. Further, some steps may be performed simultaneously.
[029] In accordance with an embodiment of the present disclosure, the one
or more processors 104 are configured to model, at step 302, the neuronal controller in the form of a Reinforcement Learning (RL) agent based on an inverted pendulum with 1 Degree of Freedom (1DOF) representing a first mechanical model of a human body in the form of a dynamical system. In an embodiment, the RL agent is trained using a Q-learning technique to learn an optimal state-action value (Q-value) function for a tuneable Markov Decision Process (MDP) model, wherein the modeled neuronal controller is configured to reproduce Center Of Pressure (COP) characteristics in the human body either along the sagittal plane or along the frontal plane separately.
[030] The modeling of the MDP for the RL agent, in accordance with an
embodiment of the present disclosure is performed as described herein after. Markov

decision process contains the elements wherein S
represents a set of states, A represents a set of actions, represents a
transition probability,γ represents a discount factor and represents an
expected reward for a current state s, immediate action and next state . The model
based algorithm has no stochasticity associated with state transition. Accordingly,
given a particular state ( in terms of θ and θ) and action (in terms of torque at a
pivot joint), system dynamics reveal an immediate next state. A reward policy or a
reward function (used interchangeably in the context of the present disclosure) is also
modeled deterministically. The set of states S , the set of actions A and the reward
policy are modeled as described hereinafter.
[031] Modeling the set of actions A : The actions are sampled using a curve
represented as:

where nA represents number of actions, p represents an exponent that defines the
characteristics of a curve represented in equation 1. and τmax represents a maximum
torque applied to the pivot joint. Let set,
and n denotes an integer number. To obtain,nA
number of actions, the curve represented by equation (1) is sampled for all integer values in the range for all elements of the set

[032] Modeling the set of States ( S ): The system state comprises of two state
variables θand θ. The scope of the state variable θspans from and is
represented as . Since the present disclosure deals with discrete MDP, the θ and
θ space is discretized by defining boundaries. Boundaries of ‘sub-states’ in θ space

are sampled from a curve represented by equation (2) below.

where represents number of sub-states in the θ space inside and wherein
is chosen to be an odd integer. If a set
number of boundaries required to define
number of states, may be samples from equation (2) for all elements of the set The set containing these boundaries may be represented as given below.

[033] On similar lines, scope of the state variable θ spans from
Sub-state -1 spans from spans fromwhere
represents number of sub-states in the θ domain. From there are
states, boundaries of which are defined by taking samples of all
elements of a where

where represents number of sub-states in the θ space inside .
[034] Also, the set containing these boundaries may be represented as given below.



[035] If Sθ is a set of sub-states corresponding to the state variable θ, then,

The elements of the set S is a combination of elements of the sets Sθ and
respectively.
[037] Modeling the Reward In accordance with the present
disclosure, the reward policy or the reward function may be modeled in multiple approaches as discussed hereinafter.
[038] Reward policy-1: The RL agent gets a reward of +1 when s (state of
the system at t+1) falls inside (which spans between two nearest θ boundaries on
either side of i.e. the reward states comes under the set:



[039] Reward policy-2: The RL agent gets a reward of +1 when s falls inside
the sub-state and simultaneously, in which case the reward state is only one
state out of all the elements of the set The significance of
this reward policy is that it encodes the objective of making the inverted pendulum reach the vertical state and stay in the same state.
[040] Reward policy-3: A Gaussian reward policy over the state space is
represented as follows:

where and are standard deviations of the Gaussian reward policy over the θ
and θ dimensions respectively. The lower values of and reduce the scope of
getting the reward. Also, this is a continuous reward policy as opposed to the Reward policy-1 and the Reward policy-2.
[041] Reward policy-4: A negative reward policy is also adapted frequently
where the RL agent gets a negative reward of -10 or a punishment of 10 when the inverted pendulum falls out of
[042] Transitional probability (( S, S ,)) and discount factor ( γ ): Considering
the system dynamics are known, the need for finding the transitional probability is averted. The discount factor ( γ) which is a parameter of temporal difference learning, is chosen to be 0.1.
[043] In accordance with an embodiment, the parameters of the MDP model
may include:
nθrepresenting resolution of states in θ domain
nθrepresenting resolution of states in θ domain
nArepresenting resolution of states in τ domain

representing the maximum torque exertable on the pivot joint or the boundary of the τ domain;
representing the boundary of the θ domain; representing a finite boundary of the θ domain; representing property of a curve from which torque values in the τ domain is sampled;
representing property of a curve from which boundaries of states of the θ domain is sampled; and
pθ representing property of a curve from which boundaries of the states of the θ domain is sampled.
Increasing parameter values gives more resolution to the system states (S )
and actions (A), at the cost of increased training time for the RL agent.
[044] In an embodiment, the Q-learning technique, which is an off-policy algorithm for temporal difference learning, is configured to learn to generate a torque representing an action for each state of the inverted pendulum by receiving proprioceptive inputs in the form of angle and angular velocity of the pivot joint of the inverted pendulum; and generating the torque at the pivot joint of the inverted pendulum based on the received proprioceptive inputs to maintain an upright posture of the inverted pendulum. In an embodiment of the present disclosure, the optimal state-action value (Q-value) function is in the form of a look-up table.
[045] In accordance with the present disclosure, the algorithm for Q-learning technique may be represented as given below: Initialize Q ( s, a ) arbitrarily Repeat (for each episode): Initialize s Repeat (for each step of the episode):
Choose a from s using a policy derived from Q (e.g., ε-greedy )


Until is terminal, where represents a learning rate, value of which typically lies between 0 and 1.
is not updated or there is no learning, whereas a = 1 means
Q(s,a ) is updated depending on an expected total reward achieved in a recent episode and is independent of a previous Q(s,a ) value for a particular state and action.. Also, ε represents an exploration factor associated with the ε-greedy policy that aims to define balance between exploration and exploitation of immediate reinforcement learning. A value of ε near 0 ensures the RL agent prefers exploitation over exploration and vice versa.
[046] The s and a are the current state and immediate action taken at time t
and ś and á are the state and the action taken at time t+1 respectively. It is proven that
once the Q values are learnt for a significant number of episodes under the e-greedy
policy, the Q- learning algorithm converges to close approximation of optimal state-
action value function (* (s,a )) with probability 1. Once the RL agent has learnt the
Q values for significant number of episodes, the RL agent is ready for exploitation
only (means ε= 0), i.e. at a given state it will choose the action with maximum Q
value by arg max Q( s,a ).
a
[047] In accordance with an embodiment of the present disclosure, the Center Of Pressure (COP) characteristics in the human body either along the sagittal or along the frontal plane are reproduced separately in accordance with a dynamical equation based on overall mass of the human body being concentrated at the point of Center Of Mass (COM) located at the 2nd lumbar vertebrae of the human body, an approximate height of the 2nd lumbar vertebrae, moment of inertia of the inverted pendulum, angle of the inverted pendulum with respect to the direction of gravity, gravitational acceleration, a stiffness constant and a damping constant, wherein the

stiffness constant and the damping constant denote pivot joint properties of the inverted pendulum. FIG.4 illustrates a 1 Degree Of Freedom (1DOF) inverted pendulum representing a free body diagram of a simplified human body in the sagittal plane, in accordance with an embodiment of the present disclosure. In an embodiment, the dynamical equation may be represented as:

and
where m represents the overall mass of the human body being concentrated at the
point of Center Of Mass (COM) located at the 2nd lumbar vertebrae of the human
body;
represents the approximate height of the 2nd lumbar vertebrae; θ represents angle of the inverted pendulum;
g represents gravitational acceleration; I represents moment of inertia of the inverted pendulum;
K represents the stiffness constant; and
B represents the damping constant.
[048] In accordance with an embodiment of the present disclosure, the one
or more processors 104 are configured to derive, at step 304, dynamical equations of a Spherical Inverted Pendulum (SIP) with respect to the global coordinate system. In accordance with the present disclosure, the SIP represents a second mechanical model of the human body that is capable of exhibiting postural sway along both the frontal plane and the sagittal plane together. In an embodiment, the dynamical equations are derived by using Lagrange’s equations with two independent state variables (θx and θy) that represent angular deviation of the SIP about a pivot joint and along x and y axes respectively of the global coordinate system, wherein the pivot joint characterizes an ankle joint of the human body. FIG.5 illustrates an SIP having 2 DOF with the two state variables, θx and θy, in accordance with an embodiment of

the present disclosure. As the reference plan passes through the origin, there are two
sets of equations for the SIP, a set representing above and below the reference planes
respectively. In FIG.5, ( lx, ly, lz ) is a Cartesian co-ordinate system representation of
the position of the pendulum mass. m and l represent pendulum mass and length of
the shaft of the SIP respectively.
[049] In accordance with the present disclosure, the dynamical equations are
derived as described hereinafter. The Lagrange’s equations are as given below.



[050] Further, in accordance with the present disclosure, kinetic energy,
potential energy and Lagrangian may be derived as below.
Using the values of Cosθ, Sinθ, Cosϕ and Sin ϕ in equation (10),

Plugging equation (13) in equation (9) gives the velocity components along the x, y, and z axes as follows:

The kinetic energy and the potential energy derived from equations (7), (8), (13) and (14) may be represented as below.


[051] In accordance with an embodiment of the present disclosure, using the kinetic energy and potential energy in equation (6), provides the Lagrangian (L) and plugging the Lagrangian in equation (5), the dynamical equations of the SIP are represented as: for θ= 0 to 90°, θ being the polar or zenith angle of the spherical coordinate system,



where θx and θy represent the angular acceleration with reference to x and y axes
respectively.
[052] The dynamical equations of the SIP were simulated for the following
initial condition:

wherein given a θx , the θy is calculated by making vz = 0 in the equation vz =
and wherein vz represents velocity along the z axes,
θx and θy being angles of the SIP with respect to the x axis and the y axis respectively and θx and θy being angular velocity of the SIP respectively. Again the dynamical equations of the SIP were also simulated with changed initial angular velocity of 1.2 rad/sec, the initial condition being:
to find out θx given θy , θx is calculated by
making vz = 0 in the following equation The

SIP equations were also checked for above the ground scenario by simulating the dynamical equations for the following initial condition:

[053] In accordance with an embodiment of the present disclosure, the one or more processors 104 are configured to model, at step 306, the human postural sway both along the sagittal plane and along the frontal plane together using the modeled neuronal controller and the derived dynamical equations of the SIP by tuning (i) a reward function comprised in the modeled neuronal controller and (ii) a set of parameters to balance the SIP such that the postural sway characteristics generated by the modeled neuronal controller match the postural sway characteristics of one or more control subjects.
[054] To validate model generated sway characteristics of the present
disclosure, experimental data pertaining to the human postural sway, in the form of
the COP characteristics of four control subjects in the age group 25 to 40 years
having height in the range 5′3″ to 5′10″, and body weight in the range 50 to 70kg,
when standing bipedally for approximately 30 secs were collected. It was noted from
the frequency spectrum of the COP-x signal where |X (k )| is Fourier coefficient
corresponds to the kth frequency spectrum, that the frequency spectrum is almost
uniformly distributed from 0 to 1 Hz except few peaks at very low frequency component, comparable to DC component. The stated experimental data is also referred later in the description with reference to the description for FIG. 6A through FIG.6D, FIG.7A through FIG.7D, FIG.8A through FIG.8D and FIG.9A through FIG.9D. To match the human postural sway seen in the control subjects, in accordance with the present disclosure, three additional parameters are introduced in the model. In accordance with the present disclosure, the set of parameters include parameters of the MDP model and the three additional parameters associated with physiology of the human body.
[055] In accordance with an embodiment, the parameters associated with

physiology of the human body may include White Gaussian Noise ( WGN ) added to
the generated torque to represent a real world noisy biological system; filtering factor (λ ) added to portray signaling characteristics of a neuromuscular junction; and Scaling Factor (SF ) introduced to scale down the magnitude of the generated torque for each state of the inverted pendulum after the training of the RL agent is completed, wherein completion of the training is represented by a balanced SIP.
[056] In accordance with an embodiment of the present disclosure, a
relationship between the generated torque by the modeled neuronal controller and the torque applied to the dynamical system based on the parameters associated with physiology of the human body is represented as:

τ( t ) represents the torque applied to the dynamical system,
represents the torque after introducing the Scaling Factor, represents the generated torque by the modeled neuronal controller, WGN represents the White Gaussian Noise of magnitude 1, and c represents noise amplitude. EXPERIMENTAL RESULTS
[057] The model sway characteristics were modeled without incorporating
the filtering factor, the scaling factor and the noise. The closest human postural sway characteristics generated by the model of the present disclosure are illustrated in FIG.6A through FIG.6D and FIG.7A through FIG.7D. Particularly, FIG.6A and FIG.6B illustrate the COP characteristics along the x axis and a corresponding frequency spectrum while FIG.6C and FIG.6D illustrate the COP characteristics along the y axis and a corresponding frequency spectrum for the following set of parameters:


that the frequency spectrum of COP-x is closely matched with the experimental data (detailed above) but the frequency spectrum is twice as spread as the experimental data for COP-y.
[058] FIG.7A and FIG.7B illustrate variance of the angle of the SIP and the
angular velocity of the SIP with respect to the x axis and the y axis respectively
w.r.t time. FIG.7C illustrates a plot of applied torques at the pivot
joint by the neuronal controller of the present disclosure. It may be noted that the maximum torque is approximately 166.67 Nm. FIG.7D illustrates COP movement along the x axis and the y axis with parameter values stated with reference to FIG.6A through FIG.6D.
[059] The additional parameters viz., the filtering factor the noise
amplitude and the scaling factor were then introduced which
resulted in the human postural sway characteristics as seen in the experimental data. After tuning the parameters of the MDP model, the additional parameters and choosing the Reward policy-3 with appropriate standard deviation, the model of the present disclosure produced an expected sway as depicted in FIG.8A through FIG.8D and FIG.9A through FIG.9D. Particularly, FIG.8A and 8B illustrate the COP characteristics along the x axis and a corresponding frequency spectrum while FIG.8C and FIG.8D illustrate the COP characteristics along the y axis and a corresponding frequency spectrum. It may be noted that the frequency spectrum is spread between 0 to 1 Hz as observed in the experimental data. FIG.9A and FIG.9B illustrate variance of the angle of the SIP and the angular velocity of the SIP with respect to the x axis and the y axis respectively w.r.t time. FIG.9C
illustrates a plot of applied torques at the pivot joint by the neuronal controller

of the present disclosure. FIG.9D illustrates COP movement along the x axis and the y axis with parameter values used with reference to FIG.8A through FIG.8D. The illustrated sway characteristics were observed using the following set of parameters:

[060] In accordance with the present disclosure, by obtaining the COP
characteristics of a test subject, using say, a pressure sensing board, a frequency spectrum representative of the postural sway of the test subject may be derived. Accordingly, using the neuronal controller model described above a best set of parameters that correspond to the postural sway of the test subject can be derived. Deviation of the derived set of parameters from those of the one or more control subjects provides insights into balance impairments, if any, associated with the test subject. Thus the mathematical concepts disclosed in the description of the method of the present disclosure are integrated into a practical application of modeling a neuronal controller that exhibits human postural sway so that an analyses of the postural balance of the test subject can be performed and subject specific rehabilitation protocols may be prescribed.
[061] The written description describes the subject matter herein to enable
any person skilled in the art to make and use the embodiments. The scope of the subject matter embodiments is defined by the claims and may include other modifications that occur to those skilled in the art. Such other modifications are intended to be within the scope of the claims if they have similar elements that do not differ from the literal language of the claims or if they include equivalent elements with insubstantial differences from the literal language of the claims.

[062] It is to be understood that the scope of the protection is extended to
such a program and in addition to a computer-readable means having a message therein; such computer-readable storage means contain program-code means for implementation of one or more steps of the method, when the program runs on a server or mobile device or any suitable programmable device. The hardware device can be any kind of device which can be programmed including e.g. any kind of computer like a server or a personal computer, or the like, or any combination thereof. The device may also include means which could be e.g. hardware means like e.g. an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination of hardware and software means, e.g. an ASIC and an FPGA, or at least one microprocessor and at least one memory with software modules located therein. Thus, the means can include both hardware means and software means. The method embodiments described herein could be implemented in hardware and software. The device may also include software means. Alternatively, the embodiments may be implemented on different hardware devices, e.g. using a plurality of CPUs.
[063] The embodiments herein can comprise hardware and software
elements. The embodiments that are implemented in software include but are not limited to, firmware, resident software, microcode, etc. The functions performed by various modules described herein may be implemented in other modules or combinations of other modules. For the purposes of this description, a computer-usable or computer readable medium can be any apparatus that can comprise, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
[064] The illustrated steps are set out to explain the exemplary embodiments
shown, and it should be anticipated that ongoing technological development will change the manner in which particular functions are performed. These examples are presented herein for purposes of illustration, and not limitation. Further, the

boundaries of the functional building blocks have been arbitrarily defined herein for the convenience of the description. Alternative boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed. Alternatives (including equivalents, extensions, variations, deviations, etc., of those described herein) will be apparent to persons skilled in the relevant art(s) based on the teachings contained herein. Such alternatives fall within the scope and spirit of the disclosed embodiments. Also, the words “comprising,” “having,” “containing,” and “including,” and other similar forms are intended to be equivalent in meaning and be open ended in that an item or items following any one of these words is not meant to be an exhaustive listing of such item or items, or meant to be limited to only the listed item or items. It must also be noted that as used herein and in the appended claims, the singular forms “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise.
[065] Furthermore, one or more computer-readable storage media may be
utilized in implementing embodiments consistent with the present disclosure. A computer-readable storage medium refers to any type of physical memory on which information or data readable by a processor may be stored. Thus, a computer-readable storage medium may store instructions for execution by one or more processors, including instructions for causing the processor(s) to perform steps or stages consistent with the embodiments described herein. The term “computer-readable medium” should be understood to include tangible items and exclude carrier waves and transient signals, i.e., be non-transitory. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, nonvolatile memory, hard drives, CD ROMs, DVDs, flash drives, disks, and any other known physical storage media.
[066] It is intended that the disclosure and examples be considered as
exemplary only, with a true scope and spirit of disclosed embodiments being indicated by the following claims.

WE CLAIM:
1. A processor implemented method for modeling a neuronal controller exhibiting
human postural sway (300), the method comprising the steps of:
modeling, by one or more hardware processors, the neuronal controller in the form of a Reinforcement Learning (RL) agent based on an inverted pendulum with 1 Degree Of Freedom (1DOF) representing a first mechanical model of a human body in the form of a dynamical system, wherein the RL agent is trained using a Q-learning technique to learn an optimal state-action value (Q-value) function for a tuneable Markov Decision Process (MDP) model, wherein the modeled neuronal controller is configured to reproduce Center Of Pressure (COP) characteristics in the human body either along a sagittal plane or along a frontal plane separately (302);
deriving, by the one or more hardware processors, dynamical equations of a Spherical Inverted Pendulum (SIP) with respect to a global coordinate system, wherein the SIP represents a second mechanical model of the human body that exhibits postural sway along both the frontal plane and the sagittal plane together, wherein the dynamical equations are derived by using Lagrange’s equations with two independent state variables (θx and θy) being angular deviation of the SIP about a pivot joint and along x and y axes respectively of the global coordinate system, wherein the pivot joint characterizes an ankle joint of the human body (304); and
modeling, by the one or more hardware processors, the human postural sway both along the sagittal plane and along the frontal plane together using the modeled neuronal controller and the derived dynamical equations of the SIP by tuning (i) a reward function comprised in the modeled neuronal controller and (ii) a set of parameters to balance the SIP such that the postural sway characteristics generated by the modeled neuronal controller match the postural

sway characteristics of one or more control subjects, wherein the set of parameters include parameters of the MDP model and parameters associated with physiology of the human body (306).
2. The processor implemented method of claim 1, wherein the Q-learning
technique is configured to learn to generate a torque representing an action for
each state of the inverted pendulum by:
receiving proprioceptive inputs in the form of angle and angular velocity of the pivot joint of the inverted pendulum; and
generating the torque at the pivot joint of the inverted pendulum based on the received proprioceptive inputs to maintain an upright posture of the inverted pendulum.
3. The processor implemented method of claim 1, wherein the step of modeling a neuronal controller comprises reproducing the Center Of Pressure (COP) characteristics in the human body either along the sagittal or along the frontal plane separately in accordance with a dynamical equation based on overall mass of the human body being concentrated at the point of Center Of Mass (COM) located at the 2nd lumbar vertebrae of the human body, an approximate height of the 2nd lumbar vertebrae, moment of inertia of the inverted pendulum, angle of the inverted pendulum with respect to the direction of gravity, gravitational acceleration, a stiffness constant and a damping constant, wherein the stiffness constant and the damping constant denote pivot joint properties of the inverted pendulum.
4. The processor implemented method of claim 3, wherein the step of modeling a neuronal controller comprises reproducing the Center Of Pressure (COP)

characteristics in the human body either along the sagittal or along the frontal plane separately in accordance with the dynamical equation represented as:

m represents overall mass of the human body being concentrated at the point of Center Of Mass (COM) located at the 2nd lumbar vertebrae of the human body;
L represents the approximate height of the 2nd lumbar vertebrae; θ represents angle of the inverted pendulum with respect to the direction of gravity;
g represents gravitational acceleration;
I represents moment of inertia of the inverted pendulum; K represents the stiffness constant; and B represents the damping constant.
5. The processor implemented method of claim 2, wherein the parameters of the
MDP model include:
nθ representing resolution of states in θ domain
nθrepresenting resolution of states in θ domain
nArepresenting resolution of states in τ domain
τmaxrepresenting the maximum torque exertable on the pivot joint or the boundary of the τ domain;
θmax representing the boundary of the θ domain;
θmax representing a finite boundary of the θ domain;
pa representing property of a curve from which torque values in the τ domain is sampled;
pθrepresenting property of a curve from which boundaries of states of the θ domain is sampled; and
pθrepresenting property of a curve from which boundaries of the states of the θ domain is sampled.

6. The processor implemented method of claim 2, wherein the parameters
associated with physiology of the human body include:
White Gaussian Noise (WGN ) added to the generated torque to represent
a real world noisy biological system;
filtering factor (λ ) added to portray signaling characteristics of a neuromuscular junction; and
Scaling Factor (SF ) introduced to scale down the magnitude of the generated torque for each state of the inverted pendulum after the training of the RL agent is completed, wherein completion of the training is represented by a balanced SIP.
7. The processor implemented method of claim 6, wherein a relationship between
the generated torque by the modeled neuronal controller and the torque applied
to the dynamical system based on the parameters associated with physiology of
the human body is represented as:

τ(t) represents the torque applied to the dynamical system,
represents the torque after introducing the Scaling Factor, represents the generated torque by the modeled neuronal controller, represents the White Gaussian Noise of magnitude 1, and represents noise amplitude.
8. The processor implemented method of claim 7, wherein the dynamical
equations of the SIP are represented as:

9. A system (100) for modeling a neuronal controller exhibiting human postural
sway, the system comprising: one or more data storage devices (102) operatively coupled to one or more hardware processors (104) and configured to store instructions configured for execution by the one or more hardware processors to:
model the neuronal controller in the form of a Reinforcement Learning (RL) agent based on an inverted pendulum with 1 Degree Of Freedom (1DOF) representing a first mechanical model of a human body in the form of a dynamical system, wherein the RL agent is trained using a Q-learning technique to learn an optimal state-action value (Q-value) function for a tuneable Markov Decision Process (MDP) model, wherein the modeled neuronal controller is configured to reproduce Center Of Pressure (COP) characteristics in the human body either along a sagittal plane or along a frontal plane separately;
derive dynamical equations of a Spherical Inverted Pendulum (SIP) with respect to a global coordinate system, wherein the SIP represents a second mechanical model of the human body that exhibits postural sway along both the frontal plane and the sagittal plane together, wherein the dynamical equations are derived by using Lagrange’s equations with two independent state variables (θx and θy) being angular deviation of the SIP about a pivot joint and along x and y axes respectively of the global coordinate system, wherein the pivot joint characterizes an ankle joint of the human body; and

model the human postural sway both along the sagittal plane and along the frontal plane together using the modeled neuronal controller and the derived dynamical equations of the SIP by tuning (i) a reward function comprised in the modeled neuronal controller and (ii) a set of parameters to balance the SIP such that the postural sway characteristics generated by the modeled neuronal controller match the postural sway characteristics of one or more control subjects, wherein the set of parameters include parameters of the MDP model and parameters associated with physiology of the human body.
10. The system of claim 9, wherein the Q-learning technique is configured to learn
to generate a torque representing an action for each state of the inverted
pendulum by:
receiving proprioceptive inputs in the form of angle and angular velocity of the pivot joint of the inverted pendulum; and
generating the torque at the pivot joint of the inverted pendulum based on the received proprioceptive inputs to maintain an upright posture of the inverted pendulum.
11. The system of claim 9, wherein the one or more hardware processors are
configured to perform the step of modeling a neuronal controller by
reproducing the Center Of Pressure (COP) characteristics in the human body
either along the sagittal or along the frontal plane separately in accordance with
a dynamic equation based on overall mass m of the human body being
concentrated at the point of Center Of Mass (COM) located at the 2nd lumbar
vertebrae of the human body, an approximate height L of the 2nd lumbar
vertebrae, moment of inertia of the inverted pendulum I , angle of the inverted
pendulum θ with respect to the direction of gravity, gravitational acceleration
g, a stiffness constant K and a damping constant B , wherein the stiffness

constant and the damping constant denote pivot joint properties of the inverted pendulum.
12. The system of claim 11, wherein the wherein the parameters of the MDP model include:
n θ representing resolution of states in θ domain nθrepresenting resolution of states in θ domain nArepresenting resolution of states in τ domain
τmaxrepresenting the maximum torque exertable on the pivot joint or the boundary of the τ domain;
representing the boundary of the θdomain; representing a finite boundary of the θ domain; Pa representing property of a curve from which torque values in the domain is sampled;
pθrepresenting property of a curve from which boundaries of states of the θ domain is sampled; and
pθ representing property of a curve from which boundaries of the states of the θ domain is sampled; and the parameters associated with physiology of the human body include:
White Gaussian Noise (WGN ) added to the generated torque to
represent a real world noisy biological system;
filtering factor (λ ) added to portray signaling characteristics of a neuromuscular junction; and
Scaling Factor (SF ) introduced to scale down the magnitude of the generated torque for each state of the inverted pendulum after the training of the RL agent is completed, wherein completion is of the training is represented by a balanced SIP.

13. The system of claim 12, wherein a relationship between the generated torque by
the modeled neuronal controller and the torque applied to the dynamical system
based on the parameters associated with physiology of the human body is
represented as:

τ(t ) represents the torque applied to the dynamical system, τSF( t) represents the torque after introducing the Scaling Factor, τNC represents the generated torque by the modeled neuronal controller, WGN represents the White Gaussian Noise of magnitude 1, and C represents noise amplitude.
14. The system of claim 13, wherein the dynamical equations of the SIP are
represented as:
for θ = 0 to 90°, θ being the polar or zenith angle of the spherical coordinate system,