Description:F O R M 2
THE PATENTS ACT, 1970
(39 of 1970)
AND
THE PATENT RULES, 2003
COMPLETE SPECIFICATION
(See section 10; rule 13)
KNOWLEDGE SYSTEM AND METHOD FOR REAL-TIME ASSESSMENT
OF LEARNING COMPREHENSION OF A USER
APPLICANT
Educational Initiatives Private Limited
having an address at
The First Building, Corporate House A2, 1st Floor, Nyay Marg, Vastrapur,
Ahmedabad, Gujarat, India 380015
The following specification particularly describes the invention and the manner
in which it is to be performed.
1
KNOWLEDGE SYSTEM AND METHOD FOR REAL-TIME ASSESSMENT
OF LEARNING COMPREHENSION OF A USER
BACKGROUND
5 [0001] Embodiments of the present specification relate generally to automated
evaluation of learning of a user, and more particularly to knowledge systems and
methods for real-time assessment of learning comprehension of a user.
[0002] Rapid advances in educational technology have led to an exponential
increase in the demand for educational tools that are designed to aid students in
10 learning. Additionally, there is an increased need for educational tools that also
aid the educators in evaluating the learning comprehension of the students.
[0003] In a classroom setting, a teacher typically evaluates a student’s work
product to pinpoint any errors and provide appropriate feedback to help the
student learn. However, tasking the teacher to provide such individual feedback
15 to every student/learner for every problem attempted by the student is an effort
intensive task.
[0004] Currently, there exist intelligent tutoring systems (ITSs) that aid in
providing customized instruction to students to enable learning. However, most
currently existing ITSs typically evaluate only the final answer of a problem,
20 while expecting a student to work out a detailed procedure listing each step in
his/her notebook. As will be appreciated, while solving problems, students do
commit errors and an erroneous intermediate step may eventually lead to an
incorrect final answer. The currently available ITSs fail to enable the student to
identify occurrence of an erroneous intermediate step, thereby depriving the
25 student the chance to correct the error prior to arriving at the final answer.
Additionally, the presently available ITSs fail to provide real-time assessment of
learning comprehension of the user.
2
BRIEF DESCRIPTION
[0005] In accordance with aspects of the present specification, a knowledge
system for real-time automated assessment of learning comprehension of a user is
presented. The system includes a learning platform configured to present a
5 problem statement corresponding to a learning objective to the user of the system,
enable the user to enter one or more solution steps to the problem statement,
evaluate, in real-time, each of the one or more solution steps entered by the user
to assess comprehension of the learning objective through adequacy of each of the
one or more solution steps, and instantaneously generate a customized error
10 message based on the evaluation of each of the one or more solution steps,
wherein the customized error message is configured to enhance learning and
guide the user towards a correct solution of the problem statement presented to
the user. The system also includes a data processing unit in operative associative
with the learning platform, wherein the data processing unit is configured to
15 maintain an error repository of one or more error patterns, one or more error
types, a frequency of occurrence of the one or more error types, one or more error
messages corresponding to the one or more error types, or combinations thereof.
Further, the system includes an interface unit in operative association with the
learning platform, the data processing unit, or both and configured to at least
20 provide the customized error message to the user, wherein the learning platform is
configured to assess the learning comprehension of the user based at least on one
or more efficiency metrics, one or more probability values, or both.
[0006] In accordance with another aspect of the present specification, a
method for real-time automated assessment of learning comprehension of a user is
25 presented. The method includes presenting a problem statement corresponding to
a learning objective to the user of the system. Further, the method includes
enabling the user to enter one or more solution steps to the problem statement.
The method also includes evaluating, in real-time, each of the one or more
solution steps entered by the user to assess comprehension of the learning
30 objective through adequacy of each of the one or more solution steps. In addition,
3
the method includes instantaneously generating a customized error message based
on the evaluation of each of the one or more solution steps, wherein the
customized error message is configured to enhance learning of the user.
Moreover, the method includes communicating the customized error message to
5 the user to guide the user towards a correct solution of the problem statement
presented to the user, where assessing the learning comprehension of the user is
based at least on one or more efficiency metrics, one or more probability values,
or both.
[0007] In accordance with yet another aspect of the present specification, a
10 system for real-time assessment of learning comprehension of a user is presented.
The system includes a communications network. Moreover, the system includes
one or more computing devices corresponding to one or more users, wherein each
of the one or more computing devices is operatively coupled to the
communications network, wherein each of the one or more computing devices
15 comprises a knowledge system for real-time automated assessment of learning
comprehension of a user, and wherein the knowledge system comprises a learning
platform configured to present a problem statement corresponding to a learning
objective to the user of the system, enable the user to enter one or more solution
steps to the problem statement, evaluate, in real-time, each of the one or more
20 solution steps entered by the user to assess comprehension of the learning
objective through adequacy of each of the one or more solution steps, and
instantaneously generate a customized error message based on the evaluation of
each of the one or more solution steps, wherein the customized error message is
configured to enhance learning and guide the user towards a correct solution of
25 the problem statement presented to the user. Furthermore, the system also
includes a data processing unit in operative associative with the learning platform,
wherein the data processing unit is configured to maintain an error repository of
one or more error patterns, one or more error types, a frequency of occurrence of
the one or more error types, one or more error messages corresponding to the one
30 or more error types, or combinations thereof. Additionally, the system includes
4
an interface unit in operative association with the learning platform, the data
processing unit, or both and configured to at least provide the customized error
message to the user, where the system is configured to assess learning
comprehension of the user based on one or more efficiency metrics, one or more
5 probability values, or both.
DRAWINGS
[0008] These and other features and aspects of embodiments of the present
specification will become better understood when the following detailed
10 description in read with reference to the accompanying drawings in which like
characters represent like parts throughout the drawings, wherein:
[0009] FIG. 1 is a schematic representation of an overall learning environment
including a knowledge system for real-time automated assessment of learning
comprehension of a user, in accordance with aspects of the present specification;
15 [0010] FIG. 2 is a schematic representation of one embodiment of an
exemplary knowledge system for real-time automated assessment of learning
comprehension of a user for use in the learning environment of FIG. 1, in
accordance with aspects of the present specification;
[0011] FIG. 3 is a flow chart illustrating a method for real-time automated
20 assessment of learning comprehension of a user, in accordance with aspects of the
present specification;
[0012] FIG. 4 is a flow chart illustrating a method of working of a learning
platform of FIG. 2, in accordance with aspects of the present specification;
[0013] FIG. 5 is a flowchart illustrating a method of working of a digital
25 processing unit of FIG. 2, in accordance with aspects of the present specification;
5
[0014] FIGs. 6(a)-6(b) represent a flowchart illustrating a method of
generating an error repository for use in the knowledge system of FIG. 2, in
accordance with aspects of the present specification;
[0015] FIG. 7 is a flowchart illustrating a method of maintaining an error
5 repository of FIGs. 6(a)-6(b), in accordance with aspects of the present
specification;
[0016] FIG. 8 is a diagrammatical illustration of one embodiment of the
working of the knowledge system of FIG. 2 , in accordance with aspects of the
present specification;
10 [0017] FIG. 9 is a diagrammatical illustration of another embodiment of the
working of the knowledge system of FIG. 2 , in accordance with aspects of the
present specification;
[0018] FIGs. 10(a), 10(b), and 10(c)) are diagrammatical illustrations of
generation of customized error messages by the knowledge system of FIG. 2 , in
15 accordance with aspects of the present specification; and
[0019] FIG. 11 is a schematic representation of a digital processing system
implementing the knowledge system of FIG. 2, in accordance with aspects of the
present specification.
DETAILED DESCRIPTION
20 [0020] The following description presents exemplary computerized knowledge
systems and methods for real-time automated assessment of learning
comprehension of a user. Particularly, embodiments described hereinafter present
exemplary systems and methods that facilitate enhanced quality of learning for a
user such as a student in real-time. Use of the present systems and methods
25 presents significant advantages in reliably providing significant enhancement in
the quality of learning for students by providing real-time assessment of learning
6
comprehension of the students, thereby overcoming the drawbacks of currently
available methods of learning.
[0021] For ease of understanding, the exemplary embodiments of the present
systems and methods are described in the context of mathematical problems.
5 However, use of the exemplary embodiments illustrated hereinafter in other
systems and applications such as solving problems in physics, chemistry, and the
like is also contemplated. An exemplary environment that is suitable for
practising various implementations of the present systems and methods is
discussed in the following sections with reference to FIG. 1.
10 [0022] As used herein, the term “user” refers to a person using a learning
environment of FIG. 1. For example, the user may be a student that uses the
learning environment for learning. Also, as used herein, the term “computing
device” refers to a device such as, but not limited to, a mobile phone, a tablet, a
smart television (TV), a laptop, a personal computer, and the like. As used
15 herein, the term “customized error message” is an error message that is suitably
indicative of an error committed by a user and aids in guiding the user to correct
the error and hence enhances the learning comprehension of the user. It may be
noted that the terms “customized error message(s)” and “customized feedback”
may be used interchangeably.
20 [0023] Referring now to the drawings, FIG. 1 illustrates an exemplary system
or learning environment 100 for real-time automated assessment of learning
comprehension of a user. In particular, the learning environment 100 is
configured to enhance quality of learning experience for the user. In one
embodiment, the learning environment 100 is configured to monitor learning
25 comprehension of the user and provide an assessment of the learning
comprehension of the user in real-time. To that end, the learning environment
100 is configured to present a learning concept/objective to the user. It may be
noted that the following description is presented with reference to a learning
7
environment for teaching mathematical concepts. However, the present systems
and methods may also find application in other learning environments.
[0024] By way of example, the learning environment 100 may be configured
to present a mathematical problem statement to the user. In response, the learning
5 environment 100 is configured to receive an input provided by the user. The
input may include one or more solution steps to the mathematical problem
statement. It may be noted that the one or more solution steps may include one or
more intermediate solution steps and a final solution step. Additionally, the
learning environment 100 is configured to assess each solution step entered by the
10 user to evaluate the learning comprehension of the user of the learning
concept/objective associated with the problem statement. Moreover, in response
to the real-time assessment of each solution step, the learning environment 100 is
configured to generate and provide customized feedback, in real-time, regarding
the assessment of that solution step, thereby guiding the user towards a correct
15 final solution step. The learning environment 100 is configured to communicate
the customized feedback to the user in real-time on a display, for example. In one
embodiment, the customized feedback may be stored in a repository.
[0025] As depicted in FIG. 1, the learning environment 100 includes a
plurality of computing devices 102, 104, 106. In a presently contemplated
20 configuration, each of the computing devices 102, 104, 106 may include an
exemplary knowledge system 108. The knowledge system 108 is configured to
facilitate the real-time automated assessment of learning comprehension of a user.
The working of the knowledge system 108 will be described in greater detail with
reference to FIGs. 2-5, 6(a)-6(b), and 7-11.
25 [0026] Further, the computing devices 102, 104, 106 are communicably
coupled to a data center 112 that provides resources such as, but not limited to,
data storage, computing power, databases, networking, analytics, and the like, in
one example. In some embodiments, the data center 112 may include the cloud,
fog, data lake, and the like. Moreover, in certain embodiments, customized
8
feedback may also be stored in the data center 112. However, it may be noted
that the customized feedback/error messages may also be retrieved from other
storage means such as, but not limited to, physical storage devices such as local or
remote hard disks, CDs, DVDs, Blu-ray disks, and the like. The customized
5 feedback may be transmitted from the data center 112 to the knowledge system
108 via the Internet 110 to be communicated to the user. It may be noted that in
the following description reference is made to use of the cloud 114 as a data
center. However, use of other means of storage and computing is also envisaged.
The data center 112 may also include a data storage 116.
10 [0027] FIG. 2 illustrates one embodiment of a part of the computing device
102, 104, 106 including the knowledge system 108 of FIG. 1, in accordance with
aspects of the present specification. It may also be noted that although the
embodiment of FIG. 2 depicts the knowledge system 108 as a part of the
computing device 102, in some embodiments, the knowledge system 108 may be
15 a stand-alone application that is external to the computing device 102.
[0028] As previously noted, the knowledge system 108 is configured to
facilitate real-time automated assessment of learning comprehension of a user 202
such as a student. As will be appreciated, traditional currently available
intelligent tutoring systems (ITSs) typically evaluate only a final answer of a
20 problem, while expecting students to work out a detailed procedure listing of each
step in his/her notebook. However, if the students commit errors while arriving at
the final answer, any erroneous intermediate step may result in an incorrect final
answer. This process disadvantageously prevents the student 202 from
recognizing the erroneous step. Moreover, in the absence of real-time customized
25 feedback regarding the error(s) committed, the student is deprived of an
opportunity to correct himself/herself, thereby preventing improved learning.
Additionally, it is desirable to assess each solution step entered by the student and
provide customized feedback regarding that solution step in real-time. The
customized feedback is representative/indicative of the error committed by the
30 student 202 and is configured to aid in providing appropriate information
9
regarding the erroneous step to enhance learning while also guiding the student
202 towards a correct/valid solution step. Moreover, it is also desirable to
evaluate the learning comprehension of the concept presented to the student 202
based on the responses provided by the student 202.
5 [0029] In accordance with exemplary aspects of the present specification, the
knowledge system 108 is configured to perform this task of evaluating the student
202 as he/she works out each step of a solution to a problem statement and
generate and provide, in real-time, customized feedback to the student 202 to
address any errors committed by the student 202. In particular, the knowledge
10 system 108 is configured to generate the customized feedback based on the
solution step of the student 202. Additionally, the knowledge system 108 is
configured to evaluate the learning comprehension of the student 202 based on
solution steps entered by the student 202 and the customized feedback provided to
the student 202.
15 [0030] In a presently contemplated configuration, the knowledge system 108 is
depicted as including a learning platform 204 and a data processing unit (DPU)
206. The learning platform 204 is configured to present a problem statement to
the student 202. The problem statement may correspond to a learning objective
such as, but not limited to, solving linear equations in one (1) variable,
20 simplifying an algebraic expression, factorizing and expanding algebraic
expressions applying identities, simplifying expressions involving exponents
applying laws of exponents, evaluating algebraic expressions, evaluating
trigonometric expressions, adding, subtracting, multiplying and dividing rational
numbers, and the like. In one example, the learning platform 204 may present the
25 problem statement to the student 202 via an interface unit 212. As will be
appreciated, a solution to the presented problem statement may include one or
more solution steps, where the solution steps may include one or more
intermediate solution steps and a final solution step. The learning platform 204 is
configured to allow the student 202 to enter one or more solution steps to the
30 problem statement via the interface unit 212. Moreover, the learning platform
10
204 is also configured to evaluate, in real-time, each of the one or more solution
steps entered by the student 202 to assess comprehension of the learning objective
through adequacy or sufficiency of each of the one or more solution steps entered
by the student 202. In certain embodiments, the learning platform 204 may
5 include a step-by-step evaluating unit 208 that is configured to facilitate
evaluation of each solution step entered by the student 202. However, in some
embodiments, the learning platform 208 may not include the step-by-step
evaluating unit 208.
[0031] Furthermore, the learning platform 204 is configured to instantaneously
10 generate customized feedback based on the evaluation of each of the one or more
solution steps. The customized feedback is designed to provide the student 202
with real-time feedback that is tailored based on the solution step entered by the
student 202. In particular, the feedback may be tailored/customized based on an
error committed by the student 202 at a particular solution step. Providing this
15 tailored feedback aids in enhancing the learning comprehension of the student
202.
[0032] With continuing reference to FIG. 2, the DPU 206 is operatively
coupled to the learning platform 204. Further, the DPU 206 is configured to
generate and maintain an error repository 210. The error repository 210 may
20 include one or more error patterns, one or more error types, a frequency of
occurrence of the one or more error patterns, one or more error messages
corresponding to the one or more error patterns, one or more efficiency metrics,
probability values, or combinations thereof. Some non-limiting examples of error
patterns include erroneous simplification of algebraic expressions such as the
25 expression ax(by + cz) may be erroneously simplified as abxy + cz. Another
example of an error pattern includes erroneous manipulation of a linear equation
such as the expression ax + b = c may be erroneously manipulated as ax = c + b .
Yet another example of an error pattern includes incorrectly simplifying
expressions involving exponents such as ap × bp may be incorrectly simplified as
(ab)
(p× p) instead of (ab)
p 30 . Another example of an error pattern includes an
11
erroneous expansion or simplification of an algebraic expression such as (ax +
by)
2 may be erroneously expanded or simplified as a
2
x
2 + abxy + b2
y
2
. Yet
another example of an error pattern may include an incorrect addition of rational
numbers such as ??
??
+ ??
?? may be incorrectly added as ??+??
??+??
.
5 [0033] Also, some non-limiting examples of error types that may be
committed by the student 202 include cognitive errors or misconceptions, careless
computational errors or slips, absence of knowledge and/or procedure, errors due
to lack of comprehension of working with the learning environment 100, and the
like. By way of example, the error types may include typographical errors,
10 incorrect inputs like use of a different variable, and the like. It may be noted that
the error repository 210 is configured to store a pool of customized error
messages, where the pool of error messages is associated with the same error type
E.
[0034] Moreover, the knowledge system 108 may also include the interface
15 unit 212 that is operatively coupled to the learning platform 204, the DPU 206, or
both. The interface unit 212 may be configured to at least provide the customized
feedback to the student 202 to enable learning and guide the student 202 towards
a correct solution of the problem statement presented to the student 202. In one
example, the interface unit 212 may be employed to present the problem
20 statement to the student 202 and allow the student 202 to enter one or more
solution steps. Further, in one embodiment, the interface unit 212 may include a
display 214 and a user interface 216. The interface unit 212 may be configured to
present the problem statement and/or the customized feedback to the student 202
via the display 214. Also, the student 202 may enter the solution steps via the
25 user interface 216. The display 214 and the user interface 216 may overlap in
some embodiments such as a touch screen. Further, in some embodiments, the
display 214 and the user interface 216 may include a common area.
[0035] The user interface 216 may include a human interface device (not
shown) that is configured to aid the user in providing inputs or manipulating
12
content visualized on the display 214. In certain embodiments, the human
interface device may include a trackball, a joystick, a stylus, a mouse, or a touch
screen. It may be noted that the user interface 216 may be configured to aid the
student 202 in navigating through the inputs provided to the learning platform 204
5 and/or outputs generated by the learning platform 204. Although the embodiment
presented in FIG. 2 depicts the knowledge system 108 as including the interface
unit 212, it may be noted that the interface unit 212 including the display 214 and
user interface 216 may be external to the knowledge system 108. Moreover, in
certain embodiments, the computing device 102, 104, 106 may also include a data
10 repository 218. However, in other embodiments, the knowledge system 108 may
include the data repository 218.
[0036] In one embodiment, based on the evaluation of each solution step, the
learning platform 206, the DPU 206, or both may retrieve the customized
feedback from the data center 112. As previously noted, the data center 112
15 provides resources such as, but not limited to, data storage, computing power,
databases, networking, analytics, and the like, in one example. In some
embodiments, the data center 112 may include the cloud, fog, data lake, and the
like. However, the customized feedback may also be retrieved from other storage
means such as, but not limited to, physical storage devices such as local or remote
20 hard disks, CDs, DVDs, Blu-ray disks, and the like. The requested customized
feedback may be transmitted from the data center 112 to the knowledge system
108 via the Internet 110. As previously noted, in some embodiments, the
customized feedback may be generated and stored locally in the error repository
210.
25 [0037] Implementing the knowledge system 108 having the learning platform
204 and the DPU 206 as described hereinabove aids in enhancing the learning
comprehension of the student 202 by providing an enriched learning environment
for the student 202, while obviating the shortcomings of the presently available
ITSs. Additionally, use of the knowledge system 108 provides real-time
30 assessment of the learning comprehension of the student 202, thereby optimizing
13
system level learning workflow. Moreover, each solution step entered by the
student 202 is automatically evaluated in real-time and specific customized
feedback corresponding to the error committed is provided to the student 202,
which allows the student 202 to correct any erroneous intermediate solution steps
5 and proceed towards a correct final solution step. The knowledge system 108 is
configured to not only provide customized feedback on the correctness of a
solution step but also help pinpoint the exact error and a communicate a
customized error message to lead the student 202 to the correct step.
[0038] Additionally, the knowledge system 108 is also configured to assess, in
10 real-time, the learning comprehension of the student 202 of a learning objective
via use of one or more parameters. These parameters are related to efficacy of
each customized error message in leading the student 202 to a subsequent
valid/correct solution step and/or a correct solution. Also, these parameters may
generally be referred to as efficiency metrics and probability values. The
15 assessment of the learning comprehension via use of the efficiency metrics and/or
the probability values will be described in greater detail with reference to FIGs.
6(a)-6(b) and 7.
[0039] In accordance with exemplary aspects of the present specification,
systems and methods that facilitate significant improvement in quality of learning
20 experience for a user such as a student, while facilitating real-time assessment of
the learning comprehension of the student are presented.
[0040] Embodiments of the exemplary methods of FIGs. 3-5, 6(a)-6(b), and 7
may be described in a general context of computer executable instructions on
computing systems or a processor. Generally, computer executable instructions
25 may include routines, programs, objects, components, data structures, procedures,
modules, functions, and the like that perform particular functions or implement
particular abstract data types.
[0041] Moreover, the embodiments of the exemplary methods may be
practised in a distributed computing environment where optimization functions
14
are performed by remote processing devices that are linked through a wired
and/or wireless communication network. In the distributed computing
environment, the computer executable instructions may be located in both local
and remote computer storage media, including memory storage devices.
5 [0042] In addition, in FIGs. 3-5, 6(a)-6(b), and 7, the exemplary methods are
illustrated as a collection of blocks in a logical flow chart, which represents
operations that may be implemented in hardware, software, firmware, or
combinations thereof. It may be noted that the various operations are depicted in
the blocks to illustrate the functions that are performed. In the context of
10 software, the blocks represent computer instructions that, when executed by one
or more processing subsystems, perform the recited operations.
[0043] Moreover, the order in which the exemplary methods are described is
not intended to be construed as a limitation, and any number of the described
blocks may be combined in any order to implement the exemplary methods
15 disclosed herein, or equivalent alternative methods. Further, certain blocks may
be deleted from the exemplary methods or augmented by additional blocks with
added functionality without departing from the spirit and scope of the subject
matter described herein.
[0044] Turning now to FIG. 3, a flow chart 300 of an exemplary method for
20 real-time assessment of learning comprehension of a user, in accordance with
aspects of the present specification, is presented. In particular, the method 300
entails processing each solution step entered by a student and providing, in realtime, customized feedback to the student based on the evaluation of the entered
solution step, thereby facilitating real-time assessment of the learning
25 comprehension of the student. The method 300 provides significant improvement
in quality of experience for end users, while circumventing the shortcomings of
the currently available techniques. Also, the method 300 of FIG. 3 is described
with reference to the components of FIGs. 1-2. Moreover, in certain
embodiments, the method 300 may be performed by the knowledge system 108.
15
[0045] The method 300 starts at step 302 when the user/student 202 initiates a
learning session by logging into the learning environment 100 using the
computing device 102, 104, 106 that includes the exemplary knowledge system
108. Further, the student 202 may identify a learning objective/concept for the
5 learning session. By way of example, the student 202 may identify algebraic
equations as the learning objective. More particularly, the student 202 may
identify linear equations in one variable as the learning objective. Also, in certain
embodiments, the student 202 may select the learning objective via use of the
interface unit 212 having the display 214 and the user interface 216.
10 [0046] In response, the knowledge system 108 may present a problem
statement related to the learning objective selected by the student 202, as
indicated by step 304. In one example, the problem statement may be presented
to the student 202 via use of the interface unit 212 and the display 214. In
particular, according to aspects of the present application, along with the problem
15 statement presented to the student 202, a corresponding interface with appropriate
input fields and a customized toolbar may also be presented to the student 202.
The input fields allow the student 202 to input necessary information
corresponding to the solution steps, while the customized toolbar aids the student
202 in entering the solution steps in the input fields. By way of example, for a
20 solution step for the problem statement associated with linear equations, the
interface unit 212 may provide two input fields on either side of an “=” sign on
the display 214. Similarly, if a solution step entails expansion of an algebraic
equation, the interface unit 212 may provide a single input field to the student 202
on the display 214. As previously noted, the solution to the problem statement
25 may include one or more solution steps, where the solution steps may include one
or more intermediate solution steps, a final step, or combinations thereof.
[0047] Responsive to the problem statement presented to the student 202, the
student 202 may enter one of the one or more solution steps via use of the user
interface 216 and/or the display 214, for example. Accordingly, at step 306, the
30 knowledge system 108 and the learning platform 204 in particular may receive a
16
solution step entered by the student 202. Subsequently, at step 308, the
knowledge system 108 may evaluate the solution step entered by the student 202.
In one embodiment, the learning platform 202 having the step-by-step evaluating
unit 208 may be configured to evaluate the entered solution step. In particular,
5 the learning platform 204 may be configured to determine if the currently entered
solution step is valid/correct or invalid/incorrect. For example, if the solution step
is determined to be valid, the learning platform 204 may be configured to
generate, in real-time, an indicator of a valid solution step and communicate the
indicator to the student 202 in real-time. In a similar fashion, if the solution step
10 is determined to be invalid, the learning platform 204 may be configured to
generate an indicator of an invalid solution step and communicate the indicator to
the student 202 in real-time. The evaluation of the solution step will be described
in greater detail with reference to FIGs. 4-5.
[0048] Furthermore, as indicated by step 310, the learning platform 204 may
15 be configured to instantaneously provide customized feedback to the student 202
based on the evaluation of the solution step entered by the student 202. In certain
embodiments, the learning platform 204 is configured to provide the customized
feedback to the student 202 when the currently entered solution step is determined
to be invalid. To facilitate providing the customized feedback, the learning
20 platform 204 may be configured to obtain user information. The user information
may include information related to the student 202 such as a student ID, the
student’s grade, indicators of student’s math ability like accuracy of solving math
problems in the ITS, indicators of the student’s solutions with the same error
committed in solving similar problems, probability of the student committing
25 errors related to the problems being solved, learning rate of the student signifying
the number of incorrect solutions to similar problems before the student solves the
problem correctly, and the like. The learning platform 204 is configured to
communicate the user information and currently entered invalid solution step to
the DPU 206. It may be noted that if the currently entered invalid solution step is
30 not the first solution step entered by the student 202, then a previous valid
17
solution step entered by the student 202 may also be communicated to the DPU
206.
[0049] Once the user information, the current invalid solution step, and the
previous valid solution step are received by the DPU 206, the DPU 206 is
5 configured to identify a type of the problem statement and mathematical forms of
the previous valid solution step and the current invalid solution step. Moreover,
the DPU 206 is configured to identify an error pattern based at least on
mathematical forms of the previous valid solution step and the current invalid
solution step. In addition, the DPU 206 is configured to retrieve a customized
10 error message based on mathematical forms of the previous valid solution step
and the current invalid solution step. In certain embodiments, the DPU 206 is
configured to dynamically identify an error message in the error repository 210
based on the mathematical form of the previous valid solution step and the
mathematical form of the current solution step, the error pattern, or combinations
15 thereof. The identified customized error message may then be communicated to
the learning platform 204, which in turn is configured to communicate the
retrieved customized error to the student 202. As previously noted, the DPU 206
may be configured to generate and maintain the error repository 210 of one or
more error patterns, one or more error types, a frequency of the one or more error
20 types, error messages, or combinations thereof. The process of generating and
maintaining the error repository 210 will be described in greater detail with
reference to FIGs. 6(a)-6(b).
[0050] Subsequently, a check is carried out to verify if the current solution
step entered is indicated as his/her final solution step by the student 202, as
25 depicted by step 312. At step 312, if it is verified that the current solution step is
not the final solution step, then control is returned to step 306 and steps 306-310
are repeated. However, at step 312, if it determined that the current solution step
is the final solution step, control is passed to step 314, where the final solution
step entered by the student 202 is evaluated as described with reference to step
30 308. Furthermore, based on the evaluation of the final solution step, customized
18
feedback is generated and provided to the student 202 to enable learning and to
guide the student 202 towards a correct solution of the presented problem
statement, as indicated by step 316.
[0051] It may be noted that at steps 310 and/or 316 if the DPU 206 is unable to
5 identity a suitable customized error message/feedback in the error repository 210,
a default error message may be communicated to the student 202. In certain
embodiments, the default error message may be retrieved from the cloud 114.
Alternatively, the default error message may be retrieved from the local data
repository 218, the error repository 210, or a combination thereof.
10 [0052] As noted hereinabove, the knowledge system 108 is configured to
generate customized feedback based on the evaluation of each solution step
entered by the student 202. Additionally, use of the exemplary error repository
210 aids is providing insightful comments that are tuned to address the errors
committed by the student 202 to enhance the learning comprehension of the
15 student 202.
[0053] Moreover, at step 318, the learning comprehension of the student 202
may be assessed in real-time. The evaluation of each solution step entered by the
student 202 allows the knowledge system 108 to objectively assess, in real-time,
the learning comprehension of the student 202 and accordingly present
20 appropriate customized feedback to the student 202. Furthermore, the assessment
of the learning comprehension of the student 202 may also be communicated to
the teachers/tutors and/or the parents of the student 202. In accordance with
aspects of the present specification, the learning comprehension of a learning
objective of the student 202 may be assessed in real-time via use of one or more
25 parameters that are related to efficacy of each customized error message in
leading the student 202 to a subsequent valid/correct solution step and/or a correct
solution. These parameters may generally be referred to as efficiency metrics and
probability values. The assessment of the learning comprehension via use of the
19
efficiency metrics and/or the probability values will be described in greater detail
with reference to FIGs. 6(a)-6(b) and 7.
[0054] FIG. 4 is a flow chart 400 of one example of the working of the
learning platform 204 (see FIG. 2), in accordance with aspects of the present
5 specification. In particular, FIG. 4 presents one example of the exemplary
method for real-time assessment of learning comprehension of a user of FIG. 3.
The method 400 of FIG. 4 is described with reference to the components of FIGs.
1-3. Also, the method 400 may be performed by the knowledge system 108 and
the learning platform 204 in conjunction with the DPU 206, in particular.
10 [0055] In accordance with aspects of the present specification, the knowledge
system 108 having the learning platform 204 and the DPU 206 is configured to
assess, in real-time, learning comprehension of the student 202. Accordingly, the
knowledge system 108 may be configured to evaluate each solution step entered
by the student 202. Additionally, in certain embodiments, the knowledge system
15 108 and the learning platform 204 in particular is configured to provide, in realtime, customized feedback to the student 202 based on the evaluation of the
entered solution step to enhance the learning of the student 202. Further, the
knowledge system 108 and the DPU 206 in particular is also configured to build
and maintain an error repository 210 based on responses provided by users of the
20 knowledge system 108. Moreover, the knowledge system 108 is also configured
to generate one or more metrics that may in turn be employed to assess the
learning comprehension of the student 202.
[0056] As depicted in FIG. 4, in response to a learning session initiated by the
student 202, an appropriate problem statement may be presented to the student
25 202 by the knowledge system 108, as indicated by step 402. In one embodiment,
the learning platform 204 may be configured to present the appropriate problem
statement to the student 202 via the display 214. The student 202 may enter a
solution step in response to the presented problem statement. The student 202
may enter the solution step via use of the user interface 216 and/or the display
20
214, in one embodiment. Accordingly, at step 404, the learning platform 204 may
receive the solution step entered by the student 202.
[0057] In accordance with aspects of the present specification, as depicted by
step 406, the learning platform 204 is configured to evaluate each solution step
5 entered by the student 202 to assess learning comprehension of the student 202 of
a learning objective associated with the presented problem statement.
Furthermore, the knowledge system 108 is configured to provide, in real-time,
customized feedback to the student 202 based on the evaluation of the entered
solution step to enhance the learning of the student 202.
10 [0058] To that end, the learning platform 204 is configured to verify if the
solution step entered by the student 202 is valid or invalid, as indicated by step
408. In one example, the learning platform 204 is configured to verify
mathematical equivalence of the currently entered solution step with the presented
problem statement. It may be noted that for a problem associated with solving a
linear equation of the type ax + b = c, ax = c – b, x = ??-??
?? , x + ??
?? = ??
?? 15 are all
mathematically equivalent steps. For example, for solving linear equation 3x + 5
= 14, 3x = 14 - 5, x + 5
3
=
14
3 , and x =
14-5
3 may all be considered as mathematically
equivalent steps. Based on the evaluation of the mathematical equivalence at step
408, it may be determined if the entered solution step is valid or invalid.
20 [0059] Accordingly, at step 408, if it is verified that the entered solution step is
invalid, an “invalid solution step” indicator is generated and presented to the
student 202, as depicted by step 410. In one embodiment, the invalid solution
step indicator may be a symbol “X.” Furthermore, the learning platform 204 is
configured to communicate, in real-time, a customized error message to the
25 student 202, where the customized error message is an error message that is
suitably representative of the error committed by the student 202. Moreover, the
customized error message is configured to guide the student 202 to correct the
erroneous step, thereby enhancing the learning of the student 202. Accordingly,
at step 412, the learning platform 204 is configured to communicate at least the
21
current invalid step and a previous valid solution step, if any, and the problem
statement to the DPU 206. Other information such as the student information
may also be communicated at step 412. In response to the communication of step
412, the DPU 206 is configured to generate and/or retrieve a customized error
5 message based on the evaluation of the invalid solution step entered by the
student 202. The retrieval of the customized error message via use of the DPU
206 will be described in greater detail with reference to FIGs. 5 and 6(a)-6(b).
[0060] Furthermore, as depicted by step 414, the customized error message is
retrieved by the DPU 206 and communicated to the learning platform 204. It may
10 be noted that for a particular mathematical problem statement presented to the
student 202, if the student 202 enters an incorrect/invalid step, the knowledge
system 108 and the DPU 206 in particular is configured to retrieve the customized
error message by identifying an error message from a pool of error messages in
the error repository 210, where the pool of error messages is associated with the
15 same error type E. The DPU 206 may be configured to select a customized error
message randomly or based on weights associated with each of the error messages
in the pool of error messages. In one example, an error message with a higher
weight has a greater chance of being selected.
[0061] However, in another embodiment, the DPU 206 may be configured to
20 identify the customized error message from the pool of error messages based on
values of associated efficiency metrics and/or probability values (PE). In some
embodiments, for each error message, the knowledge system 108 is configured to
compute probability values PE for a student 202 based on his/her math ability
when he/she commits an error type E. Accordingly, in this example, the DPU 206
25 retrieves and displays the error message for which the student 202 has the highest
probability of entering the final step correctly. The computation of the efficiency
metrics and probability values will be described in greater detail with reference to
FIGs. 6(a)-6(b).
22
[0062] In addition, at step 416, the customized error message is presented to
the student 202 to guide the student 202 to correct the error committed. In certain
embodiments, the customized error message may be visually presented to the
student 202 via the display 214. However, in certain other embodiments, the
5 customized error message may be communicated to the student 202 via audio
means. Other means of communicating the customized error message are also
envisaged. Control is returned to step 404.
[0063] With returning reference to step 408, if it is determined that the
currently entered solution step is valid, an indicator of a “valid” solution step”
10 may be communicated to the student 202, as depicted by step 418. In one nonlimiting example, the valid solution step indicator may be a symbol “?.”
[0064] Also, at step 420, a check is carried out to verify if the currently entered
step is a final solution step. At step 420, if it is determined that the currently
entered solution step is not a final solution step, control is returned to step 404
15 and steps 404-418 may be repeated. However, at step 420, if it is determined that
the currently entered solution step is a final solution step, then the final solution
step entered by the student 202 is evaluated, as indicated by step 422. As
previously noted with reference to step 406, the validity of a final solution step is
verified by determining the mathematical equivalence of the final solution step
20 with the mathematical problem statement.
[0065] Subsequent to the evaluation of the final solution step, a check is
carried out at step 424 to verify if the entered final solution step is valid or
invalid. If it is determined that the final solution step is invalid, an “invalid final
solution step” indicator is presented to the student 202, as depicted by step 426.
25 In one embodiment, the invalid solution step indicator may be a symbol “X.”
Moreover, at step 428, the learning platform 204 is configured to communicate
the one or more solution steps entered by the student 202 along with their
corresponding indicators and student information to the DPU 206 for storage and
further processing. In one example, the DPU 206 is configured to store the
23
solution steps entered by the student 202 along with their indicators in the error
repository 210.
[0066] Referring again to step 424, if at step 424 it is verified that the final
solution step is valid, the learning platform 204 is configured to instantaneously
5 provide an indicator of a “valid final solution step” to the student 202, as depicted
by step 430. In one example, the valid final solution step indicator may be a
symbol ?.” Subsequently, as depicted by step 432, the learning platform 204 is
configured to communicate the one or more solution steps entered by the student
202 along with their corresponding indicators and student information to the DPU
10 206 for storage and further processing. In one example, the DPU 206 is
configured to store the solution steps entered by the student 202 along with their
indicators in the error repository 210.
[0067] It may be noted that the DPU 206 may use the information regarding
the solutions steps and their corresponding indicators and/or the student
15 information to generate one or more customized error messages and update the
error repository 210 to store additional customized error messages. Furthermore,
the DPU 206 may also be configured to use this information to generate one or
more metrics corresponding to each of the one or more customized error
messages to evaluate efficacy of the customized feedback in the learning
20 comprehension of the student 202. The computation of the efficiency metrics and
the probability value will be described in greater detail with reference to FIGs.
6(a)-6(b).
[0068] Referring now to FIG. 5, a flow chart 500 of one example of the
working of the DPU 206 (see FIG. 2), in accordance with aspects of the present
25 specification, is presented. As previously noted, the DPU 206 is configured to
generate and maintain the error repository 210. Accordingly, a method for realtime generation of a customized error message of steps 310, 316 (see FIG. 3) and
step 414 (see FIG. 4), in accordance with aspects of the present specification, is
presented. The method 500 of FIG. 5 is described with reference to the
24
components of FIGs. 1-4. Also, the method 500 may be performed by the
knowledge system 108 and the DPU 206, in particular. The DPU 206 may work
in conjunction with the learning platform 204 to generate the customized error
messages.
5 [0069] The method 500 starts at step 504 in response to input 502 received by
the DPU 206 from the learning platform 204. In one example, the input 502
received may include a trigger indicative of an invalid solution step entered by the
student 202. Additionally, the input 502 may also include the current invalid
solution step, a previous valid solution step, the problem statement presented to
10 the student 202, and user information. It may be noted that if the first solution
step entered by the student 202 is an invalid solution step, then a previous valid
solution step is not communicated to the DPU 206. Instead, in such a case, the
actual mathematical problem statement is passed as the previous valid solution
step to the DPU 206. By way of example, if the mathematical problem statement
15 posed is “Solve: 3x + 5 = 14,” and the first step is erroneously entered as 3x = 19,
then 3x + 5 = 14 is passed as the previous valid solution step to the DPU 206.
[0070] In response to the trigger received, at step 506, the DPU 206 is
configured to identify a type of the problem statement. By way of a non-limiting
example, the types of the problem statement may include “solve,” “simplify,”
20 “factorize,” and “evaluate.” Some examples of the “solve” problem statements
include linear equations and addition/subtraction of rational numbers. Also, some
examples of the “simplify” problem statements include simplifying algebraic
expressions and simplifying expressions involving exponents. In a similar
fashion, some examples of the “factorize” problem statements include factorizing
25 algebraic expressions. Further, some examples of the “evaluate” problem
statements include evaluating algebraic expressions and trigonometric
expressions.
[0071] Furthermore, at step 508, the DPU 206 is configured to identify a
mathematical form of the previous valid solution step and a mathematical form of
25
the current invalid solution step. By way of a non-limiting example, for a
problem 3x + 5 = 14, if a solution step is 3x = 14 – 5, the mathematical form of
the solution step is Ax = C – B and that for the previous valid step is Ax + B = C,
where x is a variable and B and C are constants. Similarly, for a solution step 3v
5 – 2×3u – 2×5v, the mathematical form is Ax – BCy – BDx, where x and y are
variables, and A, B, C, and D are constants.
[0072] In addition, step 510 entails verifying a match of the mathematical form
of the previous valid solution step and the mathematical form of the current
invalid solution step. In one embodiment, verifying a match of the mathematical
10 forms of the previous valid solution step and the current invalid solution step may
include identifying variables and corresponding values of the variables and any
constants corresponding to the previous valid solution step and the current invalid
solution step.
[0073] Subsequently, at step 512, a match of one or more predefined
15 conditions may be verified. Some non-limiting examples of the predefined
conditions are presented in Table 1.
Table 1
Type
Previous Correct
Step
Current Invalid
Step
Predefined
Condition
Simplify K(AX – BY) CX – DY C=K×A & D=B
Linear
equation
AX + B = C X = D D = (C + B)/A
[0074] At step 512, if it is determined that there is no match of the predefined
conditions, then a generic message indicative of the invalid solution step may be
20 generated, as depicted by step 514. Also, the generic message may be
communicated to the learning platform 204. The learning platform 204 in turn
may be configured to communicate this generic message to the student 202 via
26
the interface unit 212, for example. However, at step 512, if a match of the
predefined conditions is verified, then the DPU 206 is configured to identify an
error pattern based on a combination of the previous valid solution step, the
current invalid solution step, and the predefined conditions, as indicated by step
5 516. In certain embodiments, the DPU 206 may be configured to identify the
error pattern based on the verified match of the mathematical forms of the
previous valid solution step and the current solution step in accordance with the
one or more predefined conditions.
[0075] Further, at step 518, the DPU 206 may be configured to
10 identify/retrieve a customized error message based on the verified match of the
mathematical forms of the previous valid solution step and the current solution
step, the error pattern, or a combination thereof. In one embodiment, to retrieve
the customized error message, the DPU 206 is configured to dynamically identify
an error message in the error repository 210 based on the verified match of the
15 mathematical forms of the previous valid solution step and the current solution
step, the error pattern, or a combination thereof.
[0076] In accordance with aspects of the present specification, for a particular
mathematical problem presented to the student 202 by the knowledge system 108,
if the student 202 enters an incorrect/invalid solution step, the DPU 206 is
20 configured to identify an appropriate customized error message from a pool of
error messages associated with the same error type E, where the pool of
customized error messages are stored in the error repository 210. In certain
embodiments, the DPU 206 is configured to select a customized error message
randomly or based on weights associated with each error message in the pool of
25 customized error messages stored in the error repository 210. In certain
embodiments, an error message with a higher weight has a greater chance of
being selected.
[0077] In other embodiments, the DPU 206 is configured to select a
customized error message based on values of associated efficiency metrics, the
27
probability values PE, or a combination thereof. Accordingly, in this example, for
an error type E, the DPU 206 retrieves and communicates the customized error
message for which the student 202 has the highest probability of entering
subsequent solution steps correctly.
5 [0078] Moreover, at step 520, the DPU 206 is configured to communicate the
retrieved customized error message to the learning platform 204. Subsequently,
the customized error message may be presented/communicated to the student 202.
As previously noted, the customized error message is pointedly indicative of the
error committed by the student 202. Hence, presenting the customized error
10 message aids in enhancing the learning comprehension of the student 202.
[0079] Turning now to FIGs. 6(a)-6(b), a flow chart 600 of one example of
generation and maintenance of a customized error database such as the error
repository 210 (see FIG. 2), in accordance with aspects of the present
specification, is presented. More particularly, a method for generating and
15 maintaining the error repository 210 having customized error messages is
presented. It may be noted that the method 600 of FIGs. 6(a)-6(b) is described
with reference to the components of FIGs. 1-5. Also, the method 600 may be
performed by the knowledge system 108 and the DPU 206, in particular.
[0080] As noted hereinabove, the DPU 206 is configured to generate and
20 maintain an error repository of one or more customized error messages. It may be
noted that each of the one or more customized error messages may correspond to
one or more identified error types. To that end, the DPU 206 is configured to
receive an input corresponding to an analysis of user response data corresponding
to items to be assessed 602, an analysis of user interactions and user generated
25 solutions to problem statements 604, one or more hypotheses corresponding to
errors committed by users 606, or combinations thereof. Moreover, an input
corresponding to frequently occurring erroneous steps committed by one or more
users 608 may also be obtained. In addition, an input corresponding to identified
erroneous steps and/or identified ineffective error messages 610 may also be
28
obtained. In some embodiments, one or more of these inputs 602-610 may be
provided by subject matter experts. It may be noted that other inputs may also be
used to aid in the generation of the error repository 210.
[0081] Subsequently, at step 612, a type of a mathematical problem may be
5 identified. In one embodiment, the type of mathematical problem may be
identified based on the inputs 602-610. Additionally, associated concepts may
also be identified. By way of example, for a mathematical problem such as 3x +
14 = 5, the associated concepts may include solving a linear equation in one
variable and operations on integers/rational numbers. Similarly, for a
10 mathematical problem 3v – 2(3u + 5v), the associated concepts include applying
distributivity of multiplication over addition for algebraic expressions and
adding/subtracting like terms.
[0082] Moreover, at step 614, mathematical forms of a previous valid solution
step and a current invalid solution step associated with the mathematical problem
15 are identified. Further, as depicted by step 616, an error pattern may be identified
based on the mathematical forms of the previous valid solution step and the
current valid solution step. Some non-limiting examples of identification of error
patterns are presented in Table 2.
Table 2
Previous
correct
step
Current
step
Predefined
Condition
Error pattern Error type
Simplify K(AX –
BY)
CX -
DY
C=K×A &
D=B
Multiplying
the common
factor only
with the first
term and not
all the terms
while applying
distributivity
Cognitive or
misconception
Linear
equation
AX + B
= C
NX = D N = A and
D = C+B
Doing the
same
operation
Cognitive or
misconception
29
instead of
‘inverse’
operation in
transposing
constants to
isolate the
variable term
[0083] It may be noted that each error pattern may have a corresponding set of
error types. Accordingly, at step 618, each error type corresponding to the
identified error pattern may be classified. In one non-limiting example, the error
types may include cognitive errors or misconceptions, careless computational
5 errors or slips, absence of knowledge and/or procedure, and errors due to lack of
comprehension of working with the learning environment 100. Accordingly, each
of the one or more error types corresponding to the identified error pattern may be
classified into one or more of the categories noted hereinabove.
[0084] Subsequently, at step 620, a check is then carried out to verify if the
10 error type includes a careless error committed by the student 202. At step 620, if
it is determined that the error pattern is a careless error, then that error type is
ignored and discarded from further analysis, as indicated by step 622. Further, in
this situation, a generic message may be populated in the error repository 210, as
depicted by step 624.
15 [0085] However, at step 620, if it is determined that the error type is not a
careless error, then one or more relationships between variables in the previous
valid solution step and the current invalid solution step may be determined, as
indicated by step 626. Subsequently, at step 628, one or more customized error
messages may be generated. More particularly, the one or more customized error
20 messages corresponding to the error type may be generated based at least on the
determined relationship between variables in the previous valid solution step and
the current invalid solution step.
[0086] As previously noted, the error pattern identified at step 616 may
include a corresponding set of one or more error types. Accordingly, at step 628,
30
one or more different types of error messages may also be generated
corresponding to the error types associated with the identified error pattern. In
certain embodiments, a pool of one or more customized hints/prompts/error
messages for each error pattern may be generated based on the mathematical
5 structures of the previous valid solution step and the current invalid solution step.
[0087] Moreover, each error message may also be correspondingly labeled. In
particular, for any error type E, each error message in the pool of one or more
customized hints/prompts/error messages may be labeled with one or more
categories. Some non-limiting examples of the different types of label categories
10 of error messages include, “error pointing,” “explanation of error illustrating
similar math problem of same type,” “hint to rule/property/formulae to apply,”
and “explanation of error and illustrating correct form with a simple related math
problem.”
[0088] One example label category may point out an erroneous part in a
15 solution step. For example, for a mathematical problem “Simplify 2(x + 3),” if a
solution step entered by the student 202 includes “2x + 3,” then the label category
may indicate, “error in the term ‘3’.” In another example label category, the label
category may share an alternative mathematical problem or property to verify if a
solution step is erroneous. By way of example, for a mathematical problem
20 “Simplify 2(x + 3),” if a solution step entered by the student 202 includes “2x +
3,” the error message may indicate, “Note, 2(1 + 3) = 2×1 + 2×3 and NOT 2×1 +
3.”
[0089] Yet another example label category may share the formulae, rules, or
properties to apply to correct an erroneous step entered by the student 202. For
25 example, for a mathematical problem “Simplify 2(x + 3),” if a solution step
entered by the student 202 includes “2x + 3,” then the label category may
indicate, “Note, the distributive property to be applied here is k(a + b) = ka + kb.”
Another example label category may share an illustration of a correctly solved
similar mathematical problem highlighting the solution step corresponding to the
31
erroneous solution step and the following correct solution step(s) in the illustrated
example. By way of example, for a mathematical problem “Simplify 2(x + 3),” if
a solution step entered by the student 202 includes “2x + 3,” the label category
may indicate, “Note, 2(a + 3) = 2a + 2×3 and NOT 2a + 3. Both the terms inside
5 the parentheses ‘a’ and ‘3’ should be multiplied with 2 and added.”
[0090] In accordance with further aspects of the present application, once the
error messages are generated and labeled, these error messages may be stored. By
way of example, the error repository 210 may be populated with these error
messages.
10 [0091] Also, as indicated by step 630, each error message may be converted
into a corresponding machine-readable form. In accordance with aspects of the
present specification, the error message may be converted into a machinereadable form based on one or more of the mathematical forms of the previous
valid solution step and the current invalid solution step and the relationship(s)
15 between variables in the previous valid solution step and the current invalid
solution step using identified constants and variables. Converting the error
message into a machine-readable form aids in ensuring that the error message
displayed will use the same variables and constants present in the actual solution
steps. By way of example, to display an error message “2x + 4 = 6 implies 2x = 6
20 – 4 and NOT 2x = 6 + 4,” where 4 is a constant identified from the current invalid
solution step entered by the student 202.
[0092] Further, at step 632, information regarding the mathematical forms of
the previous valid solution step, the current invalid solution step, the
relationship(s) between variables in the previous valid solution step and the
25 current invalid solution step, the error message and its type, labels, associated
mathematical problems, and associated mathematical concepts may also be stored
in the error repository 210.
[0093] As previously noted, in accordance with aspects of the present
specification, the learning comprehension of a learning objective of the student
32
202 may be assessed in real-time based on one or more efficiency metrics and/or
probability values. To that end, the knowledge system 108 and the DPU 206 in
particular is configured to generate and store values of one or more parameters
that are related to efficacy of each customized error message in leading the
5 student 202 to a subsequent valid/correct solution step and/or a correct solution,
as depicted by step 634. These parameters are generally referred to as efficiency
metrics. The efficiency metrics may be used to assess the learning
comprehension of the student 202. Further, in certain embodiments, the
efficiency metrics may be determined based on historical/past users’ data in the
10 error repository 210, if available. However, in other embodiments, the efficiency
metrics may also be determined based on subject matter expertise. In addition,
the values of these efficiency metrics may be updated based on solution steps
entered by one or more students 202. The values of efficiency metrics
corresponding to each customized error message are employed to evaluate the
15 effectiveness of the customized error message in enabling student learning and
comprehension. In certain embodiments, values of these efficiency metrics may
be determined by the knowledge system 108 by processing the actual student
solution steps. Also, the efficiency metrics may be computed and updated each
time the knowledge system 108 encounters a student solution step or periodically
20 at a fixed time and/or at a frequency that is daily, weekly, monthly, or
combinations thereof.
[0094] One example of utilizing the values of efficiency metrics corresponding
to each customized error message to evaluate its effectiveness in enabling student
learning/comprehension includes establishing a proportion of all solutions with
25 the error message Emessage that contain the correct final answer.
[0095] For example, for an error message of the form “Note, 2(1 + 3) = 2×1 +
2×3 and NOT 2×1 + 3,” if 20 solutions out of a total of 100 solutions where such
an error message was communicated to any student for the erroneous step of the
form 2x + 3 for the math problem simplify: 2(x + 3) led to a correct final solution
30 step as 2x + 6, then a value of 0.2 is assigned to a corresponding efficiency metric.
33
In one embodiment, the DPU 206 may be configured to compute this metric for a
particular student 202 as well as globally for the occurrence of error message
Emessage in solutions of any student.
[0096] Another example of an efficiency metric entails determining a
5 proportion of all solutions with the error message Emessage communicated to any
student 202 on encountering an associated erroneous solution step (error type) in
which the subsequent solution step entered by the student 202 is correct. By way
of example, out of 100 solutions if 40 solutions from the same student 202 or a
different student has an instance of Emessage communicated and in 10 cases the
10 following solution step in these 40 solutions was correct, then the proportion is
10/40 among all the instances with a solution containing an erroneous step.
[0097] Furthermore, yet another example of an efficiency metric calls for
determining a proportion of instances of solutions with the error message Emessage
for error type E where subsequently presented similar mathematical problems do
15 not have an erroneous solution step associated with the error type E. For
example, out of 100 instances of solutions with an error message, “Note, 2(1 + 3)
= 2×1 + 2×3 and NOT 2×1 + 3” generated due to an error type associated with the
erroneous step 2x + 3 for mathematical problem 2(x + 3), 30 solutions to a
subsequently presented mathematical problem such as 5(v + 10) did not contain
20 the erroneous step 5v + 10, then the proportion is 30/100.
[0098] Additionally, the knowledge system 108 is also configured to compute
a probability value of the student 202 committing an identified error pattern.
Accordingly, as depicted by step 636, a probability value (PE) of user committing
an identified error pattern may be generated. The probability values (PE) may be
25 generated based on historical/past users’ data if available and/or based on subject
matter expertise. In one embodiment, the probability value (PE) may be
determined by the knowledge system 108 by processing the solution steps entered
by the student 202. By way of example, probability values PE of a student 202
having a particular mathematical ability committing an error type E and having
34
encountered the associated error message Emessage, providing a subsequent
valid/correct solution step, a correct final step, and solving another similar
mathematical problem solved correctly may be computed. Additionally, the PE
values may be computed and updated each time the knowledge system 108
5 encounters a student solution step or periodically at a fixed time and/or at a
frequency that is daily, weekly, monthly, or combinations thereof.
[0099] Subsequent to the processing of steps 602-636, an error repository 638
may be generated. The error repository may be representative of the error
repository 210 of FIG. 2.
10 [0100] In accordance with aspects of the present specification, for a particular
mathematical problem presented to the student 202 by the knowledge system 108,
if the student 202 enters an incorrect/invalid solution step, the knowledge system
108 is configured to display a customized error message by identifying an
appropriate customized error message from the pool of error messages associated
15 with the same error type E, where the pool of customized error messages are
stored in the error repository 210. In certain embodiments, the knowledge system
108 is configured to randomly select a customized error message or select a
customized error message based on weights associated with each error message in
the pool of customized error messages stored in the error repository 210. It may
20 be noted that an error message with a higher weight has a greater chance of being
selected.
[0101] In other embodiments, the knowledge system 108 is configured to
select a customized error message based on values of the associated efficiency
metrics, the probability values PE, or a combination thereof. The knowledge
25 system 108 is configured to compute probability values PE for a student 202 based
on his/her math ability for each error message when he/she commits an error type
E and accordingly retrieves and displays the error message for which he/she has
the highest probability of entering the final step correctly.
35
[0102] It may be noted that the DPU 206 may use the information regarding
the solutions steps and their corresponding indicators to generate additional
customized error messages and update the error repository 210 to store the
additional customized error messages. Moreover, the DPU 206 is configured to
5 generate additional efficiency metrics corresponding to each of the one or more
customized error messages to evaluate efficacy of the customized feedback in the
learning comprehension of the student 202. Further, the knowledge system 108 is
also configured to compute additional probability values of the student 202
committing an identified error pattern. The efficiency metrics and the probability
10 values may be updated in the error repository 210 and may be used by the
knowledge system 108 to assess the learning comprehension of the student 202.
[0103] Referring now to FIG. 7, a flow chart 700 of one example of
maintaining a customized error database such as the error repository 210, in
accordance with aspects of the present specification, is presented. More
15 particularly, a method for maintaining the error repository 210 having customized
error messages corresponding to different error types is presented. It may be
noted that the method 700 of FIG. 7 is described with reference to the components
of FIGs. 1-5 and 6(a)-6(b). Also, the method 700 may be performed by the
knowledge system 108 and the DPU 206, in particular.
20 [0104] The method starts at step 704 where one or more solution steps 702 are
received and processed to identify one or more error patterns. As previously
noted, some non-limiting examples of error patterns include erroneous
simplification of algebraic expressions, erroneous manipulation of a linear
equation, incorrect simplification of expressions involving exponents, erroneous
25 expansion or simplification of an algebraic expression, incorrect addition of
rational numbers, and the like.
[0105] Subsequently, at step 706, one or more error types corresponding to
each of the one or more error patterns may be identified based on the one or more
error patterns. Furthermore, as indicated by step 708, the one or more error types
36
may be classified based on the one or more error patterns. As previously noted,
the error types include cognitive errors or misconceptions, careless computational
errors or slips, absence of knowledge and/or procedure, errors due to lack of
comprehension of working with the learning environment 100. Accordingly, each
5 of the one or more error patterns may be classified into one or more of the
categories noted hereinabove (see step 618 of FIGs. 6(a)-6(b)).
[0106] The classification of the one or more error types may be described with
use of following mathematical problem noted in equation (1).
5 - 2(4x - 5) = -6 (1)
10 [0107] One example of an invalid solution step entered by the student 202 may
include the equation presented in equation (2).
-2(4x - 5) = -1 (2)
[0108] The DPU 206 may be configured to classify an error as a cognitive
error or misconception: In one example, the DPU 206 may be configured to
15 classify the current invalid solution step depicted in equation (2) as a “cognitive
error associated with algebraic manipulation to generate equivalent equations in
solving a linear equation” if a connected previous solution step entered by the
student 202 is as depicted in equation (3).
-2(4x - 5) = -6 + 5 (3)
20 [0109] In another example, the DPU 206 may be configured to classify the
current invalid solution step of equation (2) as a “cognitive error associated with
addition/subtraction of integers” if a connected previous solution step entered by
the student is as indicated in equation (4). As will be appreciated, the solution
step of equation (4) is a valid solution step.
25 -2(4x - 5) = -6 – 5 (4)
37
[0110] With continuing reference to the mathematical problem presented in
equation (1), in yet another example, the DPU 206 may be configured to classify
a current invalid solution step as a “slip or careless computational error” in the
following example.
5 [0111] The current invalid solution step is as depicted in equation (5) and a
connected previous solution step is as indicated in equation (6).
2(4x - 5) = -10 (5)
- 2(4x - 5) = -6 – 5 (6)
[0112] The error type may be classified as a slip or a careless computational
10 error in the above example since a solution of “-10” cannot be obtained via use of
any of the four basic arithmetic operations between “-6” and “5” on the right hand
side of equation (6). Additionally, if such an error pattern is not repeated in the
student’s solutions to subsequently presented similar mathematical problems, the
error is classified as a slip or a careless computational error.
15 [0113] Furthermore, in another example, the DPU 206 may be configured to
classify an error type as an absence/lack of knowledge/procedure. Referring
again to the mathematical problem of equation (1), a current solution step such as
a first solution step to the mathematical problem may be as depicted in equation
(7).
20 4x – 5 = -6 (7)
[0114] If such an error pattern is not repeated for similar mathematical
problems, the error type corresponding to the current invalid solution step of
equation (7) may be considered as one stemming from absence/lack of
knowledge/procedure.
25 [0115] Yet another category of errors may be classified as errors due to lack of
comprehension of working with the system such as the knowledge system 108,
38
the computing device 102, and/or the learning environment 100 of FIG. 1. Some
non-limiting examples of this category of errors include typographical errors,
incorrect inputs like use of a different variable, and the like.
[0116] In accordance with aspects of the present specification, this type of
5 classification or categorization of error types of step 708 may be performed based
on repetition of an error type in similar mathematical problems. Additionally, a
connection or relation between one or more erroneous steps corresponding to the
error type in one or more mathematical problems may be determined.
Subsequently, the error type may be classified or categorized based on the
10 determined correlation. Further, in one example, the classification of the error
types may be performed by a subject matter expert, artificial intelligence (AI)
using machine learning algorithms, or a combination thereof. In certain
embodiments, subsequent to the classification or categorization of error types, the
error types may be correspondingly labeled. Moreover, at step 710, a frequency
15 of occurrence of the one or more error patterns may be determined. For example,
if the presented mathematical problem is 3x + 14 = 5 and the incorrect step is 3x =
5 + 14, the DPU 206 is configured to convert the previous correct step and the
incorrect step into respective equivalent mathematical forms as ax + b = c, and ax
= c + b. Subsequently, the DPU 206 is configured to search for a match for these
20 in the error repository 210 having a database of error patterns. By way of
example, the error repository 210 may include an error pattern stored
corresponding to the following mathematical forms of a previous correct step, an
incorrect current step and conditions as follows as depicted in Table 3.
Table 3
Problem
Type
Previous
correct step
Current
incorrect
step
Condition Error
pattern
Linear
equation
AX + B = C NX = D N = A and D =
C + B
Doing the
same
operation
instead of
‘inverse’
39
operation
in
transposing
constants
to isolate
the
variable
term (E)
[0117] Hence, in this example, the DPU 206 finds a matching error pattern E
in the error repository 210 that corresponds to the previous correct step and the
current incorrect step having respective forms AX + B = C and NX = D where N
= A and D = C + B that are associated with error pattern E in the error repository
5 210. Since a matching error pattern E has been identified in the error repository
210, the frequency of the error pattern E is incremented by 1. Similarly, for
another presented mathematical problem 3x + 20 = 10 – 2x, if the previous correct
step is 5x + 20 = 10 and the incorrect step is 5x = 10 + 20, here again the DPU
206 will identify a match for the same error pattern E based on the mathematical
10 forms of previous correct step and current incorrect step and increment the
frequency of the error pattern E by 1.
[0118] Subsequently, as indicated by step 712, the one or more error types, the
classification of the one or more error types, the labels, the frequency of
occurrence of the one or more error types, or combinations thereof may be stored
15 in the error repository 210.
[0119] In addition, other information corresponding to each category of an
error type may be stored in the error repository 210. In one example, for each
error category type, a statement describing the error may be stored in the error
repository 210. By way of example, the statement related to a misconception type
20 error may include “error in transposing a constant to isolate a variable for an error
type E associated with an erroneous step as ax = c + b for the mathematical
problem as ax + b = c.” This statement may be entered by a team of subject
matter experts, in certain embodiments.
40
[0120] Additionally, for each category of the error type stored in the error
repository 210, associated mathematical problems such as types of mathematical
problems that elicit a particular error may also be identified and stored in the error
repository 210 as sources of a corresponding error type. By way of example, for a
5 cognitive error type E: ax = c + b, if the given mathematical equation is ax + b =
c, the associated mathematical problems for error type E are of the form ax + b =
c and not a(x + b) = c. It may be noted that the mathematical problems associated
with each error type E may be entered by a team of subject matter experts, in one
embodiment.
10 [0121] Moreover, for each category of the error type E associated
mathematical structures of the one or more mathematical problems and
mathematical concepts associated with each error type may be identified and
stored in the error repository 210. Furthermore, a correlation between one or
more erroneous solution steps corresponding to each category of the error type in
15 the one or more problem statements may be determined. In one example, for each
category of the error type E described hereinabove, the associated mathematical
structures of the mathematical problems may include ax + b = c, the erroneous
step may be ax = c + b, and the previous correct step may be ax = c - b. In certain
embodiments, the information related to the mathematical structures, the
20 associated mathematical concepts, the correlation between the current
erroneous/invalid solution steps, and the previous correct valid solution steps may
be entered by a team of subject matter experts.
[0122] Furthermore, for each category of the error type, mathematical skills
and/or concepts associated with that error type may also be stored in the error
25 repository 210. By way of example, for the error type E described hereinabove
while solving a single-step linear equation in one variable, the associated
mathematical skills and/or concepts may include simplifying algebraic
expressions applying distributivity for the error type as ‘ax + b’ for the associated
mathematical problem of the form ‘Simplify, a(x + b).’ It may be noted that the
30 mathematical skills and/or concepts associated with that error type E may be
41
entered by a team of subject matter experts, in one embodiment. Similarly, in the
following sequence of solution steps entered by the student 202 (equations 8-10),
the error type is identified as a cognitive error in step 2 (equation 9) of the
sequence of solution steps. In this example of a cognitive error type, the
5 associated pre-requisite mathematical skills and/or concepts may include
“addition/subtraction of integers.”
5 – 2(4x – 5) = -6 (8)
-2(4x – 5) = -6 – 5 (9)
-2(4x - 5) = -1 (10)
10 [0123] It may be noted that the mathematical skills and/or concepts associated
with each error type E may be entered by a team of subject matter experts, in one
embodiment.
[0124] FIG. 8 is a diagrammatic illustration 800 of one embodiment of the
working of the knowledge system 108 (see FIG. 1). It may be noted that FIG. 8 is
15 described with reference to the components of FIGs. 1-5, 6(a)-6(b) and 7.
[0125] Reference numeral 802 is generally representative of the display 214 of
the interface unit 212 of the knowledge system 108. As previously noted, once
the student 202 initiates a learning session, the knowledge system 108 and the
learning platform 204 in particular present a mathematical problem statement 804
20 to the student 202 on the display 214. Further, one or more intermediate solution
steps entered by the student 202 is generally represented by reference numeral
806. Based on the evaluation of each of the solution steps entered by the student
202, the learning platform 204 is configured to provide a valid or invalid solution
step indicator to the student 202. Reference numeral 808 is an example indicator
25 of valid solution steps entered by the student 202, while an example indicator of
an invalid solution step is represented by reference numeral 810.
42
[0126] As previously noted, the learning platform 204 and the DPU 206 are
configured to generate a customized error message/feedback based on the
evaluation of the erroneous solution step entered by the student 202. In the
present example, reference numeral 812 is representative of the customized
5 feedback provided in real-time to the student 202 based on the evaluation of the
erroneous solution step. Further, the student 202 is guided to correct the
erroneous solution step based on the customized feedback, thereby enhancing
learning of the student 202 of the learning objective. Moreover, reference
numeral 814 represents a corrected solution step entered by the student 202 based
10 on the customized feedback presented to the student 202. Also, space provided to
enable the student 202 to enter subsequent solution steps is represented by
reference numeral 816. Further, reference numeral 818 generally represents
shortcut keys provided to enable the student to enter mathematical operators.
[0127] Referring now to FIG. 9, a diagrammatic illustration 900 of another
15 embodiment of the working of the knowledge system 108 is presented. It may be
noted that the example presented in FIG. 8 is described with reference to the
components of FIGs. 1-5, 6(a)-6(b) and 7-8.
[0128] The display 214 of the interface unit 212 of the knowledge system 108
may be represented by reference numeral 902. Moreover, reference numeral 904
20 represents a mathematical problem statement presented to the student 202 on the
display 214. Also, one or more intermediate solution steps entered by the student
202 is generally represented by reference numeral 906. Furthermore, reference
numeral 908 is an example indicator of valid solution steps entered by the student
202, while an example indicator of an invalid solution step is represented by
25 reference numeral 910.
[0129] In the example presented in FIG. 9, reference numeral 912 is
representative of the customized feedback provided in real-time to the student
202. Also, the student 202 is guided to correct the erroneous solution step based
on the customized feedback, thereby enhancing learning of the student of the
43
learning objective. Reference numeral 914 represents a corrected solution step
entered by the student 202 based on the customized feedback presented to the
student 202. Further, reference numeral 916 generally represents a final solution
step entered by the student 202. Also, reference numeral 918 is used to indicate
5 that the final solution step is saved by the knowledge system 108.
[0130] Turning now to FIGs. 10(a)-10(c), diagrammatic illustrations of
generation of different customized error messages/feedback corresponding to
different types of errors in solution steps committed by a student 202 in solving
problem statements provided by the knowledge system 108 are presented.
10 Reference numerals 1002, 1004, and 1006 are generally representative of nonlimiting examples of customized feedback generated by the knowledge system
108 and presented to the student 202.
[0131] FIG. 11 is a schematic representation 1100 of one embodiment 1102 of
a digital processing system implementing the knowledge system 108 (see FIG. 1),
15 in accordance with aspects of the present specification. Also, FIG. 11 is
described with reference to the components of FIGs. 1-5, 6(a)-6(b), 7-8, 9(a)-9(c),
and 10.
[0132] It may be noted that while the knowledge system 108 is shown as being
a part of the computing device 102, in certain embodiments, the knowledge
20 system 108 may also be integrated into other end user systems. Moreover, the
example of the digital processing system 1102 presented in FIG. 11 is for
illustrative purposes. Other designs are also anticipated.
[0133] The digital processing system 1102 may contain one or more
processors such as a central processing unit (CPU) 1104, a random access
25 memory (RAM) 1106, a secondary memory 1108, a graphics controller 1110, a
display unit 1112, a network interface 1114, and an input interface 1116. It may
be noted that the components of the digital processing system 1102 except the
display unit 1112 may communicate with each other over a communication path
44
1118. In certain embodiments, the communication path 1118 may include several
buses, as is well known in the relevant arts.
[0134] The CPU 1104 may execute instructions stored in the RAM 1106 to
provide several features of the present specification. Moreover, the CPU 1104
5 may include multiple processing units, with each processing unit potentially being
designed for a specific task. Alternatively, the CPU 1104 may include only a
single general-purpose processing unit.
[0135] Furthermore, the RAM 1106 may receive instructions from the
secondary memory 1108 using the communication path 1118. Also, in the
10 embodiment of FIG. 11, the RAM 1106 is shown as including software
instructions constituting a shared operating environment 1120 and/or other user
programs 1122 (such as other applications, DBMS, and the like). In addition to
the shared operating environment 1120, the RAM 1106 may also include other
software programs such as device drivers, virtual machines, and the like, which
15 provide a (common) run time environment for execution of other/user programs.
Moreover, in certain embodiments, the RAM 1106 may also include an error
repository 1124 such as the error repository 210 (see FIG. 2).
[0136] With continuing reference to FIG. 11, the graphics controller 1110 is
configured to generate display signals (e.g., in RGB format) for display on the
20 display unit 1112 based on data/instructions received from the CPU 1104. The
display unit 1112 may include a display screen to display images defined by the
display signals. Furthermore, the input interface 1116 may correspond to a
keyboard and a pointing device (e.g., a touchpad, a mouse, and the like) and may
be used to provide inputs. In addition, the network interface 1114 may be
25 configured to provide connectivity to a network (e.g., using Internet Protocol),
and may be used to communicate with other systems connected to a network, for
example.
[0137] Moreover, the secondary memory 1108 may include a hard drive 1126,
a flash memory 1128, and a removable storage drive 1130. The secondary
45
memory 1108 may store data generated by the learning environment 100 (see
FIG. 1) and software instructions (for example, for implementing the various
features of the present specification), which enable the digital processing system
1102 to provide several features in accordance with the present specification. The
5 code/instructions stored in the secondary memory 1108 may either be copied to
the RAM 1106 prior to execution by the CPU 1104 for higher execution speeds or
may be directly executed by the CPU 1104.
[0138] Some or all of the data and/or instructions may be provided on a
removable storage unit 1132, and the data and/or instructions may be read and
10 provided by the removable storage drive 1130 to the CPU 1104. Further, the
removable storage unit 1132 may be implemented using medium and storage
format compatible with the removable storage drive 1130 such that the removable
storage drive 1130 can read the data and/or instructions. Thus, the removable
storage unit 1132 includes a computer readable (storage) medium having stored
15 therein computer software and/or data. However, the computer (or machine, in
general) readable medium can also be in other forms (e.g., non-removable,
random access, and the like.).
[0139] It may be noted that as used herein, the term “computer program
product” is used to generally refer to the removable storage unit 1132 or a hard
20 disk installed in the hard drive 1126. These computer program products are
means for providing software to the digital processing system 1102. The CPU
1104 may retrieve the software instructions and execute the instructions to
provide various features of the present specification.
[0140] Also, the term “storage media/medium” as used herein refers to any
25 non-transitory media that store data and/or instructions that cause a machine to
operate in a specific fashion. Such storage media may include non-volatile media
and/or volatile media. Non-volatile media include, for example, optical disks,
magnetic disks, or solid-state drives, such as the secondary memory 1108.
Volatile media include dynamic memory, such as the RAM 1106. Common
46
forms of storage media include, for example, a floppy disk, a flexible disk, hard
disk, solid-state drive, magnetic tape, or any other magnetic data storage medium,
a CD-ROM, any other optical data storage medium, any physical medium with
patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM,
5 any other memory chip or cartridge.
[0141] Storage media is distinct from but may be used in conjunction with
transmission media. Transmission media participates in transferring information
between storage media. For example, the transmission media may include coaxial
cables, copper wire, and fiber optics, including the wires that include the
10 communication path 1118. Moreover, the transmission media can also take the
form of acoustic or light waves, such as those generated during radio-wave and
infra-red data communications.
[0142] Reference throughout this specification to “one embodiment,” “an
embodiment,” or similar language means that a particular feature, structure, or
15 characteristic described in connection with the embodiment is included in at least
one embodiment of the present specification. Thus, appearances of the phrases
“in one embodiment,” “in an embodiment” and similar language throughout this
specification may, but do not necessarily, all refer to the same embodiment.
[0143] Furthermore, the described features, structures, or characteristics of the
20 specification may be combined in any suitable manner in one or more
embodiments. In the description presented hereinabove, numerous specific
details are provided such as examples of programming, software modules, user
selections, network transactions, database queries, database structures, hardware
modules, hardware circuits, hardware chips, and the like, to provide a thorough
25 understanding of embodiments of the specification.
[0144] The aforementioned components may be dedicated hardware elements
such as circuit boards with digital signal processors or may be software running
on a general-purpose computer or processor such as a commercial, off-the-shelf
47
personal computer (PC). The various components may be combined or separated
according to various embodiments of the invention.
[0145] Furthermore, the foregoing examples, demonstrations, and process
steps such as those that may be performed by the system may be implemented by
5 suitable code on a processor-based system, such as a general-purpose or specialpurpose computer. It should also be noted that different implementations of the
present specification may perform some or all of the steps described herein in
different orders or substantially concurrently, that is, in parallel. Furthermore, the
functions may be implemented in a variety of programming languages, including
10 but not limited to C++, Python, and Java. Such code may be stored or adapted for
storage on one or more tangible, machine readable media, such as on data
repository chips, local or remote hard disks, optical disks (that is, CDs or DVDs),
memory or other media, which may be accessed by a processor-based system to
execute the stored code. Note that the tangible media may include paper or
15 another suitable medium upon which the instructions are printed. For instance,
the instructions may be electronically captured via optical scanning of the paper
or other medium, then compiled, interpreted, or otherwise processed in a suitable
manner if necessary, and then stored in the data repository or memory.
[0146] Embodiments of the systems and methods described hereinabove
20 advantageously present a robust framework for enhancing the real-time
assessment of learning comprehension of a user such as a student by providing an
enriched environment for learning to the student, while obviating the
shortcomings of the currently available techniques. Additionally, the knowledge
system may be utilized to enable greater learning among students using the
25 system on a computer, a tablet, or a mobile device. Moreover, use of the
knowledge system facilitates evaluation of each solution step entered by the
student in real-time and provides customized feedback related to the errors
instantaneously. Furthermore, the generation of customized feedback based on
the error type aids in guiding the student towards a correct solution, thereby
30 enhancing learning comprehension and achieving superior learning. The
48
knowledge system is configured to not only provide customized feedback on the
correctness of a solution step but also help pinpoint the exact error and a
communicate a customized error message to lead the student to the correct step.
Additionally, the knowledge system also facilitates the real-time assessment of
5 the learning comprehension of the student based on efficiency metrics and/or
probability values. It may also be noted that the knowledge system could be a
stand-alone application or a part of an intelligent tutoring system.
[0147] Furthermore, the error depository that includes errors and associated
error messages may also be used by a teacher to improve his/her instructions,
10 devise worksheets for practice, and/or build assessments. Moreover, the error
repository may also be used by a researcher to conduct experiments and publish
results related to user learning and errors committed by the users. In addition, the
error repository may also be used by content developers to create better learning
content in intelligent tutoring systems. The error repository may also be used to
15 author assessment items and develop assessments that enhance the diagnosis of
misconceptions. By way of example, the diagnosis may entail framing distractors
(options) in multiple choice items/questions (MCQs) that capture most prominent
errors/misconceptions.
[0148] Although specific features of embodiments of the present specification
20 may be shown in and/or described with respect to some drawings and not in
others, this is for convenience only. It is to be understood that the described
features, structures, and/or characteristics may be combined and/or used
interchangeably in any suitable manner in the various embodiments.
[0149] While only certain features of the present specification have been
25 illustrated and described herein, many modifications and changes will occur to
those skilled in the art. It is, therefore, to be understood that the present
specification is intended to cover all such modifications and changes as fall within
the true spirit of the invention.
49
, Claims:Claims
We Claim:
1. A knowledge system (108) for real-time automated assessment of
learning comprehension of a user (202), the system (108) comprising:
5 a learning platform (204) configured to:
present a problem statement (804, 904) corresponding to a learning
objective to the user (202) of the system (108);
enable the user (202) to enter one or more solution steps (806, 814,
906, 914) to the problem statement (804, 904);
10 evaluate, in real-time, each of the one or more solution steps 806,
814, 906, 914) entered by the user (202) to assess learning comprehension
of the learning objective through adequacy of each of the one or more
solution steps (806, 814, 906, 914);
instantaneously generate a customized error message (812, 912,
15 1002, 1004, 1006) based on the evaluation of each of the one or more
solution steps (806, 814, 906, 914), wherein the customized error message
(812, 912, 1002, 1004, 1006) is configured to enhance learning and guide
the user (202) towards a correct solution of the problem statement (804,
904) presented to the user (202);
20 a data processing unit (206) in operative associative with the learning
platform (204), wherein the data processing unit (206) is configured to maintain
an error repository (210) of one or more error patterns, one or more error types, a
frequency of occurrence of the one or more error types, one or more error
messages corresponding to the one or more error types, or combinations thereof;
25 and
an interface unit (212) in operative association with the learning platform
(204), the data processing unit (206), or both and configured to at least provide
the customized error message to the user (202),
50
wherein the learning platform (204) is configured to assess the learning
comprehension of the user (202) based at least on one or more efficiency metrics,
one or more probability values, or both.
2. The system (108) of claim 1, wherein the one or more solution
5 steps (806, 814, 906, 914) comprise one or more intermediate solution steps, a
final solution step, or a combination thereof.
3. The system (108) of claim 1, wherein to evaluate, in real-time,
each of the one or more solution steps to assess the comprehension of the learning
objective, the learning platform (204) is configured to:
10 verify mathematical equivalence of an entered solution step (806, 814,
906, 914) with the problem statement (804, 904); and
determine if the entered solution step is valid or invalid.
4. The system (108) of claim 3, wherein, if the solution step (806,
814, 906, 914) is determined to be valid, the learning platform (204) is configured
15 to generate, in real-time, an indicator of a valid solution step (808, 908), and
wherein, if the solution step is determined to be invalid, the learning platform
(204) is configured to generate an indicator of an invalid solution step (810, 910).
5. The system (108) of claim 4, wherein the learning platform (204)
is configured to provide, in real-time, the customized error message (812, 912,
20 1002, 1004, 1006) to the user (202) based on the invalid solution step.
6. The system (108) of claim 5, wherein to provide, in real-time, the
customized error message (812, 912, 1002, 1004, 1006) to the user (202) the
learning platform (204) is configured to:
obtain user information, wherein the user information comprises at least a
25 user identification number;
51
communicate a previous valid solution step, a current invalid solution
step, the problem statement (804, 904), the user information, or combinations
thereof to the data processing unit (206); and
receive the customized error message (812, 912, 1002, 1004, 1006) from
5 the data processing unit (206).
7. The system (108) of claim 6, wherein the data processing unit
(206) is configured to:
receive a trigger indicative of the current invalid solution step;
receive the previous valid solution step, the current invalid solution step,
10 the problem statement, the user information, or combinations thereof;
identify a type of the problem statement (804, 904);
identify a mathematical form of the previous valid solution step of the
problem statement (804, 904);
identify a mathematical form of the current invalid solution step of the
15 problem statement (804, 904); and
verify a match of the mathematical form of the previous valid solution
step and the mathematical form of the current invalid solution step.
8. The system (108) of claim 7, wherein to verify the match of the
mathematical form of the previous valid solution step and the mathematical form
20 of the current invalid solution step, the data processing unit (206) is configured to:
identify variables and corresponding values in the mathematical form of
the previous valid solution step;
identify variables and corresponding values in the mathematical form of
the current invalid solution step; and
25 verify a match of one or more predefined conditions.
52
9. The system (108) of claim 8, wherein the data processing unit
(206) is further configured to:
identify an error pattern based on the verified match of the mathematical
form of the previous valid solution step and the mathematical form of the current
5 solution step of the one or more predefined conditions;
retrieve a customized error message (812, 912, 1002, 1004, 1006) based
on the verified match of the mathematical form of the previous valid solution step
and the mathematical form of the current solution step, the error pattern, or a
combination thereof; and
10 communicate the customized error message (812, 912, 1002, 1004, 1006)
to the learning platform (204).
10. The system (108) of claim 9, wherein to retrieve the customized
error message (812, 912, 1002, 1004, 1006), the data processing unit (206) is
configured to dynamically identify an error message in the error repository (210)
15 based on the verified match of the mathematical form of the previous valid
solution step and the mathematical form of the current solution step, the error
pattern, or a combination thereof.
11. The system (108) of claim 1, wherein the data processing unit
(206) is configured to generate one or more error messages, wherein the one or
20 more error messages correspond to the one or more error types, and wherein to
generate the one or more error messages, the data processing unit (206) is
configured to:
receive an input corresponding to an analysis of user response data on
assessment items (602), an analysis of user interactions and user generated
25 solutions to problem statements (604), one or more hypotheses corresponding to
errors committed by users (606), or combinations thereof;
obtain an input corresponding to frequently occurring erroneous steps
committed by one or more users (608); and
53
obtain an input corresponding to identified erroneous steps, identified
ineffective error messages, or a combination thereof (610).
12. The system (108) of claim 11, wherein the data processing unit
(206) is further configured to:
5 identify a type of a mathematical problem;
identify a mathematical form of a previous valid solution step and a
mathematical form of a current invalid solution step;
identify an error pattern based on the mathematical form of the previous
valid solution step and the mathematical form of the current invalid solution step;
10 classify one or more error types corresponding to the identified error pattern; and
label the one or more error types.
13. The system (108) of claim 12, wherein the data processing unit
(206) is further configured to:
determine a relationship between variables in the previous valid solution
15 step and the current invalid solution step;
generate one or more customized error messages (812, 912, 1002, 1004,
1006) corresponding to the one or more error types based at least on the
determined relationship between variables in the previous valid solution step and
the current invalid solution step; and
20 store the error types, the mathematical forms of the previous valid solution
step and the current invalid solution step, the relationship between variables in the
previous valid solution step and the current invalid solution step, the one or more
customized error messages, the labels of the one or more error types, or
combinations thereof in the error repository (210).
25 14. The system (108) of claim 13, wherein to maintain the error
repository (210) of one or more error patterns, one or more error types, a
54
frequency of the one or more error types, or combinations thereof the data
processing unit (206) is configured to:
process the one or more solution steps (806, 814, 906, 914, 916) to
identify one or more error patterns;
5 identify one or more error types based on the one or more error patterns;
classify the one or more error types based on the one or more error
patterns;
determine a frequency of occurrence of the one or more error types; and
store the one or more error types, the classification, the frequency of
10 occurrence, or combinations thereof corresponding to each error type in the error
repository (210).
15. The system (108) of claim 14, wherein to classify the one or more
error types, the data processing unit (206) is configured to:
identify repetition of an error type in one or more problem statements
15 (804, 904);
determine a correlation between one or more erroneous steps
corresponding to the error type in the one or more problem statements (804, 904);
and
categorize the error type based on the correlation.
20 16. The system (108) of claim 15, wherein the data processing unit
(206) is further configured to:
provide a description of each category of an error type,
identify one or more problem statements associated with a source of each
category of the error type;
25 identify mathematical structures of the one or more problem statements
and mathematical concepts associated with each category of the error type;
55
determine a correlation between one or more erroneous steps
corresponding to each category of the error type in the one or more problem
statements; and
store the description, the mathematical structures, the problem statements
5 (804, 904), the correlation corresponding to each category of the error type, or
combinations thereof in the error repository (210).
17. The system (108) of claim 16, wherein the data processing unit
(204) is further configured to:
generate one or more customized error messages (812, 912, 1002, 1004,
10 1006) corresponding to an error type based on the mathematical structures,
wherein the one or more customized error messages (812, 912, 1002, 1004, 1006)
corresponding to the error type are labeled with one or more of the error
categories;
generate one or more metrics corresponding to each of the one or more
15 customized error messages to evaluate efficacy of the customized error message
(812, 912, 1002, 1004, 1006) in the learning comprehension of the user (202); and
compute a probability value of the user (202) committing an identified
error pattern.
18. A method (300) for real-time automated assessment of learning
20 comprehension of a user (202), the method (300) comprising:
presenting (304) a problem statement (804, 904) corresponding to a
learning objective to the user (202) of a system (100, 108);
enabling (306) the user (202) to enter one or more solution steps (806,
814, 906, 914, 916) to the problem statement (804, 904);
25 evaluating (308), in real-time, each of the one or more solution steps (806,
814, 906, 914, 916) entered by the user (202) to assess learning comprehension of
56
the learning objective through adequacy of each of the one or more solution steps
(806, 814, 906, 914, 916);
instantaneously (310) generating a customized error message (812, 912,
1002, 1004, 1006) based on the evaluation of each of the one or more solution
5 steps (806, 814, 906, 914, 916), wherein the customized error message (812, 912,
1002, 1004, 1006) is configured to enhance learning of the user (202); and
communicating (316) the customized error message (812, 912, 1002,
1004, 1006) to the user (202) to guide the user (202) towards a correct solution of
the problem statement presented to the user (202),
10 wherein assessing (318) the learning comprehension of the user (202) is
based at least on one or more efficiency metrics, one or more probability values,
or both.
19. The method (300) of claim 18, further comprising maintaining
(700) an error repository (210) of one or more error patterns, one or more error
15 types, a frequency of occurrence of the one or more error types, one or more error
messages corresponding to the one or more error types, or combinations thereof.
20. The method (300) of claim 19, wherein evaluating (308), in realtime, each of the one or more solution steps (806, 814, 906, 914, 916) comprises:
verifying (406) mathematical equivalence of an entered solution step with
20 the problem statement (804, 904);
determining (408, 418) if the entered solution step (806, 814, 906, 914,
916) is valid or invalid;
if the solution step (806, 814, 906, 914, 916) is determined to be valid,
generating (416, 428), in real-time, an indicator of a valid solution step (808,
25 908); and
if the solution step (806, 814, 906, 914, 916) is determined to be invalid,
generating (410) an indicator of an invalid solution step (810, 910).
57
21. The method (300) of claim 20, further comprising providing (310,
316), in real-time, a customized error message (812, 912, 1002, 1004, 1006) to the
user (202) based on the invalid solution step, wherein providing, in real-time, the
customized error message (812, 912, 1002, 1004, 1006) to the user (202)
5 comprises:
obtaining user information, wherein the user information comprises at
least a user identification number;
communicating (412) a previous valid solution step, a current invalid
solution step, the problem statement (804, 904), the user information, or
10 combinations thereof to a data processing unit (206); and
receiving (414) the customized error message (812, 912, 1002, 1004,
1006) from the data processing unit (206).
22. The method (300) of claim 21, further comprising:
receiving (504) a trigger indicative of the current invalid solution step;
15 receiving (502) the previous valid solution step, the current invalid
solution step, the problem statement, the user information, or combinations
thereof;
identifying (506) a type of the problem statement (804, 904);
identifying (508) a mathematical form of the previous valid solution step
20 of the problem statement (804, 904);
identifying (508) a mathematical form of the current invalid solution step
of the problem statement (804, 904); and
verifying (510) a match of the mathematical form of the previous valid
solution step and the mathematical form of the current invalid solution step,
25 wherein verifying (510) the match of the mathematical form of the previous valid
solution step and the mathematical form of the current invalid solution step
comprises identifying variables and corresponding values in the mathematical
58
form of the previous valid solution step, identifying variables and corresponding
values in the mathematical form of the current invalid solution step, and verifying
a match of one or more predefined conditions.
23. The method (300) of claim 22, further comprising:
5 identifying (516) an error pattern based on the verified match of the
mathematical form of the previous valid solution step and the mathematical form
of the current solution step of the one or more predefined conditions;
retrieving (518) a customized error message (812, 912, 1002, 1004, 1006)
based on the verified match of the mathematical form of the previous valid
10 solution step and the mathematical form of the current solution step, the error
pattern, or a combination thereof, wherein retrieving (518) the customized error
message (812, 912, 1002, 1004, 1006) comprises dynamically identifying an error
message in the error repository (210) based on the verified match of the
mathematical form of the previous valid solution step and the mathematical form
15 of the current solution step, the error pattern, or a combination thereof; and
communicating (520) the customized error message (812, 912, 1002,
1004, 1006) to the learning platform (204).
24. The method (300) of claim 18, further comprising generating (600)
one or more error messages, and wherein the one or more error messages
20 correspond to one or more error types, wherein generating the one or more error
messages comprises:
receiving (602, 604, 606) an input corresponding to an analysis of user
response data on assessment items, an analysis of user interactions and user
generated solutions to problem statements, one or more hypotheses corresponding
25 to errors committed by users, or combinations thereof;
obtaining (608) an input corresponding to frequently occurring erroneous
steps committed by one or more users; and
59
obtaining (610) an input corresponding to identified erroneous steps,
identified ineffective error messages, or a combination thereof.
25. The method (300) of claim 24, further comprising:
identifying (612) a type of a mathematical problem (804, 904);
5 identifying (614) a mathematical form of a previous valid solution step
and a mathematical form of a current invalid solution step;
identifying (616) an error pattern based on the mathematical forms of the
previous valid solution step and the current invalid solution step;
determining (626) a relationship between variables in the previous valid
10 solution step and the current invalid solution step;
generating (628) one or more customized error messages (812, 912, 1002,
1004, 1006) corresponding to the error type based at least on the determined
relationship between variables in the previous valid solution step and the current
invalid solution step;
15 labeling (628) the one or more error types; and
storing (632) the error type, the mathematical forms of the previous valid
solution step and the current invalid solution step, the relationship between
variables in the previous valid solution step and the current invalid solution step,
the one or more customized error messages (812, 912, 1002, 1004, 1006), the
20 labels of the one or more error types, or combinations thereof.
26. The method (300) of claim 25, wherein maintaining (600, 700) the
error repository (210) of one or more error patterns, one or more error types, a
frequency of the one or more error types, or combinations thereof comprises:
processing (704) the one or more solution steps (806, 814, 906, 914, 916)
25 to identify one or more error patterns;
identifying (706) one or more error types based on the one or more error
patterns;
60
classifying (708) the one or more error types based on the one or more
error patterns;
determining (710) a frequency of occurrence of the one or more error
types; and
5 storing (712) the one or more error types, the classification, the frequency
of occurrence, or combinations thereof corresponding to each error type in the
error repository (210).
27. The method (300) of claim 26, wherein classifying (708) the one
or more error types comprises:
10 identifying repetition of an error type in one or more problem statements
(804, 904);
determining a relation between one or more erroneous steps corresponding
to the error type in the one or more problem statements (804, 904); and
categorizing the error type based on the correlation.
15 28. The method (300) of claim 27, further comprising:
providing a description of each category of an error type;
identifying one or more problem statements (804, 904) associated with a
source of each category of the error type;
identifying mathematical structures of the one or more problem statements
20 (804, 904) and mathematical concepts associated with each category of the error
type;
determining (626) a correlation between one or more erroneous steps
corresponding to each category of the error type in the one or more problem
statements;
25 generating one or more customized error messages (812, 912, 1002, 1004,
1006) corresponding to an error type based on the mathematical structures,
61
wherein the one or more customized error messages (812, 912, 1002, 1004, 1006)
corresponding to the error type are labeled with one or more of the error
categories;
generating one or more metrics corresponding to each of the one or more
5 customized error messages (812, 912, 1002, 1004, 1006) to evaluate efficacy of
the customized error message in the learning comprehension of the user (202);
computing (636) a probability value of the user committing an identified
error pattern; and
storing (632) the description, the mathematical structures, the
10 mathematical statements, the correlation corresponding to each category of the
error type, the one or more customized error messages (812, 912, 1002, 1004,
1006), the one or more metrics, the probability value, or combinations thereof.
29. A system (100) for real-time assessment of learning
comprehension of a user (202), the system (100) comprising:
15 a communications network (110);
one or more computing devices (102,104, 106) corresponding to one or
more users (202), wherein each of the one or more computing devices (102,104,
106) is operatively coupled to the communications network (110), wherein each
of the one or more computing devices (102,104, 106) comprises a knowledge
20 system (108) for real-time automated assessment of learning comprehension of
the user (202), and wherein the knowledge system (108) comprises:
a learning platform (204) configured to:
present a problem statement (804, 904) corresponding to a
learning objective to the user (202) of the system (108);
25 enable the user (202) to enter one or more solution steps
(806, 814, 906, 914, 916) to the problem statement;
62
evaluate, in real-time, each of the one or more solution
steps (806, 814, 906, 914, 916) entered by the user (202) to assess
learning comprehension of the learning objective through
adequacy of each of the one or more solution steps (806, 814, 906,
5 914, 916);
instantaneously generate a customized error message (812,
912, 1002, 1004, 1006) based on the evaluation of each of the one
or more solution steps (806, 814, 906, 914, 916), wherein the
customized error message (812, 912, 1002, 1004, 1006), is
10 configured to enhance learning and guide the user (202) towards a
correct solution of the problem statement (804, 904) presented to
the user (202);
a data processing unit (206) in operative associative with the
learning platform (204), wherein the data processing unit (206) is
15 configured to maintain an error repository (210) of one or more error
patterns, one or more error types, a frequency of occurrence of the one or
more error types, one or more error messages corresponding to the one or
more error types, or combinations thereof; and
an interface unit (212) in operative association with the learning
20 platform (204), the data processing unit (206), or both and configured to at
least provide the customized error message (806, 814, 906, 914, 916) to
the user (202),
wherein the learning platform (204) is configured to assess the
learning comprehension of the user (202) based at least on one or more
25 efficiency metrics, one or more probability values, or both.
Dated this 30th of January, 2023
Ankush Mahajan
Agent for the Applicant (IN/PA-1523)
30 OF CoreIP Legal Services Pvt. Ltd.
63