Specification
FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENT RULES, 2003
COMPLETE SPECIFICATION (See Section 10 and Rule 13)
Title of invention:
HUMAN INTERVENTION LEVEL DETERMINATION BASED ON
EVIDENCE AND DECISION OF BUSINESS PROCESSES FOR
REGULATORY COMPLIANCES
Applicant
Tata Consultancy Services Limited A company Incorporated in India under the Companies Act, 1956
Having address:
Nirmal Building, 9th floor,
Nariman point, Mumbai 400021,
Maharashtra, India
Preamble to the description
The following specification particularly describes the invention and the manner in which it is to be performed.
TECHNICAL FIELD [001] The disclosure herein generally relates to quality assurance and regulatory compliances, and, more particularly, to human intervention level determination based on evidence and decision of business processes for regulatory compliances.
BACKGROUND [002] Artificial Intelligence (AI) enables machines to mimic the human mind by learning through historic data and helps users in problem solving and decision-making capabilities. Currently, there are lot of processes in various domains (e.g., life sciences and healthcare (LS & HC)) which are extremely human oriented. These processes have a lot of stereotyped actions which can be potentially automated using artificial intelligence. The nature of decisions in such domain processes are extremely deterministic and require human brain intervention. The lack of evidence of accuracy and failures in most AI automation is causing disapproval by regulatory bodies. Main causes for non-acceptance of artificial intelligence in domain processes by regulators are: (a) for quality management system purposes, every decision needs to have everything documented and everything justified - many AI algorithms rely on very complex, and difficult to explain logic(s) behind associated result(s)/outcome(s). The inability to clarify the inner workings of an algorithm impact the likelihood that the regulators may approve. Regulatory Authorities are specific about not replacing the judgement aspect with Artificial Intelligence, (b) most processes in such domains require qualified personnel, mainly trained people are engaged in making judgments or decisions – Person’s qualifications justify the decision taken by him/her which is not possible in case the machine takes the decision, (c) lack of evidence of accuracy of machine processed cases as compared with human processed cases, (d) lack of justification for a particular decision and the evidence in all circumstances, and (e) no analysis of failures available. In addition to obstacles for regulatory approval, AI algorithms may also face difficulties in achieving the trust and approval of end
users. Therefore, completely relying on AI automation for any decisions and capabilities continues to remain as a challenge.
SUMMARY
[003] Embodiments of the present disclosure present technological improvements as solutions to one or more of the above-mentioned technical problems recognized by the inventors in conventional systems. For example, in one aspect, there is provided a processor implemented method for determining intervention level using evidence-based justification. The method comprises receiving, via one or more hardware processors, a business process, an associated business input, and an associated business output, wherein the business process comprises content having at least one of a text description, an audio, a video, an image, or a combinations thereof; determining (i) a representation of an evidence, (ii) an inference of the evidence, (iii) strength analysis of the evidence, and (iv) a significance of the evidence based on an execution of the business process; identifying an evidence level amongst a plurality of evidence levels based on a computed score for each of (i) the representation of the evidence, (ii) the inference of the evidence, (iii) strength analysis of the evidence, and (iv) the significance of the evidence; identifying a decision level amongst a plurality of decision levels based on at least one of (i) the associated business input, being converted or extracted to a specific form, (ii) the business process, being processed by using at least one of (a) one or more rules, (b) one or more formulae, and (c) a decision tree, (iii) the business process, being processed required a logic to select at least one output from a plurality of outputs, and (iv) the business process, being processed using at least one of (a) one or more options, (b) one or more decisions, and (c) one or more determinations being made; generating a justification matrix based on a mapping between the identified evidence level and the identified decision level; and determining a (human) intervention level using the generated justification matrix and one or more decision metrics.
[004] In an embodiment, the one or more decision metrics comprise at least one of (i) repeatability of a decision associated with the business process, the
associated business input, and the associated business output, (ii) criticality of the decision, and (iii) an applicability of the decision.
[005] In an embodiment, the determined intervention level is based on the justification matrix and an attribute of the one or more decision metrics.
[006] In an embodiment, the method further comprises classifying the determined intervention level as one of an extensive review, a moderate review, or an optional review based on the justification matrix and the attribute of the one or more decision metrics.
[007] In an embodiment, the representation of the evidence is based on at least one of a correctness, a completeness, a consumption level, and quality of the associated business input.
[008] In an embodiment, the inference of the evidence is based on at least one of a generalization, and utilization of the associated business input.
[009] In an embodiment, the strength analysis of the evidence is based on at least one of a Type I error, Type II error, and F1 score of the associated business output.
In an embodiment, the significance of the evidence is based on at least one statistical test of the associated business output.
[010] In an embodiment, when (i) the representation of the evidence, (ii) the inference of the evidence, (iii) the strength analysis of the evidence, and (iv) the significance of the evidence are not complementing each other, the method further comprises modifying a corresponding score of each of the (i) the representation of the evidence, (ii) the inference of the evidence, (iii) the strength analysis of the evidence, and (iv) the significance of the evidence.
[011] In another aspect, there is provided a processor implemented system for determining intervention level using evidence-based justification. The system comprises: a memory storing instructions; one or more communication interfaces; and one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to: receive, a business process, an associated business input, and an associated business output, wherein the business process comprises
content having at least one of a text description, an audio, a video, an image, or a combinations thereof; determine (i) a representation of an evidence, (ii) an inference of the evidence, (iii) strength analysis of the evidence, and (iv) a significance of the evidence based on an execution of the business process; identify an evidence level amongst a plurality of evidence levels based on a computed score for each of (i) the representation of the evidence, (ii) the inference of the evidence, (iii) strength analysis of the evidence, and (iv) the significance of the evidence; identify a decision level amongst a plurality of decision levels based on at least one of (i) the associated business input, being converted or extracted to a specific form, (ii) the business process, being processed by using at least one of (a) one or more rules, (b) one or more formulae, and (c) a decision tree, (iii) the business process, being processed required a logic to select at least one output from a plurality of outputs, and (iv) the business process, being processed using at least one of (a) one or more options, (b) one or more decisions, and (c) one or more determinations being made; generate a justification matrix based on a mapping between the identified evidence level and the identified decision level; and determine a (human) intervention level using the generated justification matrix and one or more decision metrics.
[012] In an embodiment, the one or more decision metrics comprises at least one of (i) repeatability of a decision associated with the business process, the associated business input, and the associated business output, (ii) criticality of the decision, and (iii) an applicability of the decision.
[013] In an embodiment, the determined intervention level is based on the justification matrix and an attribute of the one or more decision metrics.
[014] In an embodiment, the one or more hardware processors are further configured by the instructions to classify the determined intervention level as one of an extensive review, a moderate review, or an optional review based on the justification matrix and the attribute of the one or more decision metrics.
[015] In an embodiment, the representation of the evidence is based on at least one of a correctness, a completeness, a consumption level, and quality of the associated business input.
[016] In an embodiment, the inference of the evidence is based on at least one of a generalization, and utilization of the associated business input.
[017] In an embodiment, the strength analysis of the evidence is based on at least one of a Type I error, Type II error, and F1 score of the associated business output.
[018] In an embodiment, the significance of the evidence is based on at least one statistical test of the associated business output.
[019] In an embodiment, when (i) the representation of the evidence, (ii) the inference of the evidence, (iii) the strength analysis of the evidence, and (iv) the significance of the evidence are not complementing each other, the one or more hardware processors are further configured by the instructions to modify a corresponding score of each of the (i) the representation of the evidence, (ii) the inference of the evidence, (iii) the strength analysis of the evidence, and (iv) the significance of the evidence.
[020] In yet another aspect, there are provided one or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause a method for determining intervention level using evidence-based justification. The method comprises receiving, via one or more hardware processors, a business process, an associated business input, and an associated business output, wherein the business process comprises content having at least one of a text description, an audio, a video, an image, or a combinations thereof; determining (i) a representation of an evidence, (ii) an inference of the evidence, (iii) strength analysis of the evidence, and (iv) a significance of the evidence based on an execution of the business process; identifying an evidence level amongst a plurality of evidence levels based on a computed score for each of (i) the representation of the evidence, (ii) the inference of the evidence, (iii) strength analysis of the evidence, and (iv) the significance of the evidence; identifying a decision level amongst a plurality of decision levels based on at least one of (i) the associated business input, being converted or extracted to a specific form, (ii) the business process, being processed by using at least one of (a) one or more rules, (b) one or more formulae, and (c) a
decision tree, (iii) the business process, being processed required a logic to select at least one output from a plurality of outputs, and (iv) the business process, being processed using at least one of (a) one or more options, (b) one or more decisions, and (c) one or more determinations being made; generating a justification matrix based on a mapping between the identified evidence level and the identified decision level; and determining a (human) intervention level using the generated justification matrix and one or more decision metrics.
[021] In an embodiment, the one or more decision metrics comprise at least one of (i) repeatability of a decision associated with the business process, the associated business input, and the associated business output, (ii) criticality of the decision, and (iii) an applicability of the decision.
[022] In an embodiment, the determined intervention level is based on the justification matrix and an attribute of the one or more decision metrics.
[023] In an embodiment, the method further comprises classifying the determined intervention level as one of an extensive review, a moderate review, or an optional review based on the justification matrix and the attribute of the one or more decision metrics.
[024] In an embodiment, the representation of the evidence is based on at least one of a correctness, a completeness, a consumption level, and quality of the associated business input.
[025] In an embodiment, the inference of the evidence is based on at least one of a generalization, and utilization of the associated business input.
[026] In an embodiment, the strength analysis of the evidence is based on at least one of a Type I error, Type II error, and F1 score of the associated business output.
[027] In an embodiment, the significance of the evidence is based on at least one statistical test of the associated business output.
[028] In an embodiment, when (i) the representation of the evidence, (ii) the inference of the evidence, (iii) the strength analysis of the evidence, and (iv) the significance of the evidence are not complementing each other, the method further comprises modifying a corresponding score of each of the (i) the representation of
the evidence, (ii) the inference of the evidence, (iii) the strength analysis of the evidence, and (iv) the significance of the evidence.
[029] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
BRIEF DESCRIPTION OF THE DRAWINGS
[030] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and, together with the description, serve to explain the disclosed principles:
[031] FIG. 1 depicts an exemplary system for determining human intervention level determination based on evidence and decision of business processes for regulatory compliances, in accordance with an embodiment of the present disclosure.
[032] FIG. 2 depicts an exemplary high level block diagram of the system for determining human intervention level determination based on evidence and decision of business processes for regulatory compliances, in accordance with an embodiment of the present disclosure, in accordance with an embodiment of the present disclosure.
[033] FIG. 3 depicts an exemplary flow chart illustrating a method for determining human intervention level determination based on evidence and decision of business processes for regulatory compliances, using the systems of FIG. 1-2, in accordance with an embodiment of the present disclosure.
[034] FIG. 4 depicts a flow chart illustrating a method of identifying an evidence level amongst a plurality of evidence levels, in accordance with an embodiment of the present disclosure.
[035] FIG. 5 depicts a flow chart illustrating a method of identifying a decision level amongst a plurality of decision levels, in accordance with an embodiment of the present disclosure.
DETAILED DESCRIPTION OF EMBODIMENTS
[036] Exemplary embodiments are described with reference to the accompanying drawings. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. Wherever convenient, the same reference numbers are used throughout the drawings to refer to the same or like parts. While examples and features of disclosed principles are described herein, modifications, adaptations, and other implementations are possible without departing from the scope of the disclosed embodiments.
[037] Referring now to the drawings, and more particularly to FIGS. 1 through 5, where similar reference characters denote corresponding features consistently throughout the figures, there are shown preferred embodiments and these embodiments are described in the context of the following exemplary system and/or method.
[038] FIG. 1 depicts an exemplary system 100 for determining human intervention level determination based on evidence and decision of business processes for regulatory compliances, in accordance with an embodiment of the present disclosure. In an embodiment, the system 100 includes one or more hardware processors 104, communication interface device(s) or input/output (I/O) interface(s) 106 (also referred as interface(s)), and one or more data storage devices or memory 102 operatively coupled to the one or more hardware processors 104. The one or more processors 104 may be one or more software processing components and/or hardware processors. In an embodiment, the hardware processors can be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and/or any devices that manipulate signals based on operational instructions. Among other capabilities, the processor(s) is/are configured to fetch and execute computer-readable instructions stored in the memory. In an embodiment, the system 100 can be implemented in a variety of computing systems, such as laptop computers, notebooks, hand-held devices (e.g., smartphones, tablet phones, mobile communication devices, and the like), workstations, mainframe computers, servers, a network cloud, and the like.
[039] The I/O interface device(s) 106 can include a variety of software and hardware interfaces, for example, a web interface, a graphical user interface, and the like and can facilitate multiple communications within a wide variety of networks N/W and protocol types, including wired networks, for example, LAN, cable, etc., and wireless networks, such as WLAN, cellular, or satellite. In an embodiment, the I/O interface device(s) can include one or more ports for connecting a number of devices to one another or to another server.
[040] The memory 102 may include any computer-readable medium known in the art including, for example, volatile memory, such as static random-access memory (SRAM) and dynamic-random access memory (DRAM), and/or non-volatile memory, such as read only memory (ROM), erasable programmable ROM, flash memories, hard disks, optical disks, and magnetic tapes. In an embodiment, a database 108 is comprised in the memory 102, wherein the database 108 comprises information on business processes, associated business input and associated business output. The database 108 further comprises at least one of a text description, an audio, a video, an image, or a combination thereof corresponding to the business processes, associated business input and associated business output, and the like. The memory 102 further comprises (or may further comprise) information pertaining to input(s)/output(s) of each step performed by the systems and methods of the present disclosure. In other words, input(s) fed at each step and output(s) generated at each step are comprised in the memory 102 and can be utilized in further processing and analysis.
[041] FIG. 2, with reference to FIG. 1, depicts an exemplary high level block diagram of the system 100 for determining human intervention level determination based on evidence and decision of business processes for regulatory compliances, in accordance with an embodiment of the present disclosure, in accordance with an embodiment of the present disclosure.
[042] FIG. 3, with reference to FIGS. 1-2, depicts an exemplary flow chart illustrating a method for determining human intervention level determination based on evidence and decision of business processes for regulatory compliances, using the systems 100 of FIGS. 1-2, in accordance with an embodiment of the present
disclosure. In an embodiment, the system(s) 100 comprises one or more data storage devices or the memory 102 operatively coupled to the one or more hardware processors 104 and is configured to store instructions for execution of steps of the method by the one or more processors 104. The steps of the method of the present disclosure will now be explained with reference to components of the system 100 of FIG. 1, the block diagram of the system 100 depicted in FIG. 2, and the flow diagram as depicted in FIG. 3.
[043] In an embodiment, at step 202 of the present disclosure, the one or more hardware processors 104 receive a business process, an associated business input, and an associated business output, wherein the business process comprises content having at least one of a text description, an audio, a video, an image, or combinations thereof. Consider, a classification machine learning (ML) algorithm is developed for complaint coding/Product experience coding in medical device complaint management. The algorithm predicts the complaint code based upon the complaint description and Product Family. The objective is to assign/map predicted code to new complaint, and whether human intervention is required or not? Below illustrates an exemplary, business process input (also referred as associated business input and interchangeably used herein), business process, and business process output (also referred as associated business output and interchangeably used herein):
Business Process Input - Complaint Description and Product Family Name: Complaint Description - “The manufacturer received information alleging a "service required" alarm condition occurred. there was no harm or injury reported. During the evaluation of the device at the manufacturer's service center, a "service required" code was found in the ventilator's downloaded error log. The device's oxygen blending module board and interface board were replaced to address the issue.”
Product Family Name - TRILOGY O2
Business Process - Complaint coding/Product experience coding in medical device complaint management (ML classification algorithm) Business Process Output - Predicted complaint codes (with probability score)
Predicted complaint code 1- CIRCUIT FAILURE
Predicted complaint code probability 1 - 0.9058
Predicted complaint code 2 - FAILURE TO RECALIBRATE
Predicted complaint code probability 2 - 0.0376
Predicted complaint code 3 - BATTERY PROBLEM
Predicted complaint code probability 3 - 0.0116
[044] Referring to steps of FIG. 3, in an embodiment, at step 204 of the present disclosure, the one or more hardware processors 104 determine (i) a representation of an evidence, (ii) an inference of the evidence, (iii) strength analysis of the evidence, and (iv) a significance of the evidence based on an execution of the business process.
In an embodiment, the representation of the evidence is based on at least one of a correctness, a completeness, a consumption level, and quality of the associated business input. More specifically, to determine/evaluate representation of an evidence, it is determined whether the data is correctly represented in a way that can be consumed by a statistical software or by a machine immediately, or is it something that is very complex, and it requires lot of processing before the data becomes useful. Therefore, model for correctness includes - Text Data considered for training is imbalanced wherein only say x% (e.g., 15%) of complaint codes comprises y% (e.g., 80%) of entire data. Thus, score is low for this scenario. More specifically, for correctness, it is important to know how much skewness is in the data, and how much kurtosis is in the data. These give number of outliers and deficient values. If skewness and kurtosis is high for the data, then correctness of data is doubtful/challenging.
[045] Similarly, model for Completeness includes performing a comparison between missing data versus total data. If very few missing data, then score can be high. Similarly, model for Consumable includes determination of whether data is 100% structured data i.e., properly tabulated, then the data is considered as highly consumable. If data is 100% unstructured i.e., completely text/image, then the data is least consumable. In the above example described in step 202, the input is text description and 100%, 100% unstructured, and hence data
is least consumable (or the consumption level is least/low). Likewise, model for Quality includes determining whether data quality is dependent on how the data is linked with the outcome. Therefore, inference and data are tightly coupled. In the example, inferencing requirement and representation of data, both are highly matching, hence the data quality is high. The score for each of the above 4 models is determined (via one or more inputs - say by the user). Adding all (e.g., low+high+least+high) gets an average score for Representation of an evidence.
[046] Similarly, in an embodiment, the inference of the evidence is based on at least one of a generalization, and utilization of the associated business input. If representation of evidence is not good, then inference is also not going to be strong. Inference may be based on statistical method or machine learning method as known in the art. In the present disclosure, the system 100 challenges these methods. The method(s) can be challenged based upon first, how the inference can be generalized, and second, how the inference can be utilized. Model of generalization included determining if evidence is used in the source agnostic way, and if yes, then it is general. If it is source specific, then it is not general. If evidence is correctly used for more than one source(s)/utilities, it is highly generalized. For instance, in the above example, it is specific to the source, hence generalization is low. Similarly, model for utilization includes determining how strongly the evidence can be used in a specific use case gives indication of its utilization. In the above example, the evidence has high utilization as same decision can be used for similar cases. Combining both low and high individual score gives an average score for the inference of the evidence.
[047] In an embodiment, the strength analysis of the evidence is based on at least one of a Type I error, Type II error, and F1 score of the associated business output. Type I error, and Type II error are referred as statistical measures α and β respectively. Relationship of α and β gives strength analysis of evidence. α and β can put together to identify strength of evidence. The following is calculated: α * (1 - β) or β (1 - α), to score the strength of evidence. α should be < ‘x’ wherein x=0.5 and β should be < ‘y’ where y=0.2. Lower the values of α and β, higher is the inference.
[048] F1 score also can give the strength score, as it is majorly about value of data. When precision and recall is high, inference is high. In the complaint code prediction example as mentioned above, the strength calculation can be computed as in the given Table 1 below:
Table 1
Data %(A) F1 score(B) A*B
0.1 0.9 0.09
0.65 0.75 0.4875
0.25 0.6 0.15
Summation(A*B)/Summation(A) 0.7275
[049] The system and method of the present disclosure may consider any one parameter among Type I error, Type II error and Confusion matrix F1 score, etc. In the example, the confusion matrix is considered wherein for 10% of classes, precision and recall is high (>90%). For 65% of classes between (90% - 75%). And 25% of classes between (60% - 75%). Hence, average score for the strength analysis of the evidence is low.
[050] In an embodiment, the significance of the evidence is based on at least one statistical test of the associated business output. In the complaint code example, for ML classification, significance of the coefficients was evaluated which are significant i.e., p-value <0.05. Below description is provided for better understanding of the significance of evidence:
[051] Significance of Evidence is the strength by probability. This is the machine probabilistic determination. The system 100 can include some statistical test such as z-test or any other test which gives the significance (p-value) as mentioned above. The above p-value can be arrived by using below any of the (or both of the) mentioned methods:
1. Create k separate train and test datasets to calculate independent scores for each dataset and then correctly apply the statistical test.
2. K-fold cross-validation can be used with statistical test to get the significance. [052] Once (i) the representation of the evidence, (ii) the inference of the
evidence, (iii) strength analysis of the evidence, and (iv) significance of the evidence are determined based on the execution of the business process, at step 206 of the present disclosure, the one or more hardware processors 104 identify an evidence level amongst a plurality of evidence levels based on a computed score for each of (i) the representation of the evidence, (ii) the inference of the evidence, (iii) strength analysis of the evidence, and (iv) the significance of the evidence (refer above examples for scores computed). Various scores are computed for (i) the representation of the evidence, (ii) the inference of the evidence, (iii) strength analysis of the evidence, and (iv) significance of the evidence and are depicted in below Table 2:
Table 2
Total score available Representation System
generated
score
5 Correctness 2
5 Completeness 5
5 Consumable 1
5 Quality 5
20 Summation 13
Inference
5 Generalization 1
5 Utilization 5
10 Summation 6
Strength
1 Type I error, Type II error, and F1 score of the associated business output 0.7275
Significance
5 Statistical test of the associated business output 5
1000 Representation*Inference*Strength*Significance 283.725
Score (S) (Actual score/Total score available) *100 28.3725%
Now, Score S = 28.3725%, i.e., 50%
Documents
Application Documents
| # |
Name |
Date |
| 1 |
202121046588-STATEMENT OF UNDERTAKING (FORM 3) [12-10-2021(online)].pdf |
2021-10-12 |
| 2 |
202121046588-REQUEST FOR EXAMINATION (FORM-18) [12-10-2021(online)].pdf |
2021-10-12 |
| 3 |
202121046588-PROOF OF RIGHT [12-10-2021(online)].pdf |
2021-10-12 |
| 4 |
202121046588-FORM 18 [12-10-2021(online)].pdf |
2021-10-12 |
| 5 |
202121046588-FORM 1 [12-10-2021(online)].pdf |
2021-10-12 |
| 6 |
202121046588-FIGURE OF ABSTRACT [12-10-2021(online)].jpg |
2021-10-12 |
| 7 |
202121046588-DRAWINGS [12-10-2021(online)].pdf |
2021-10-12 |
| 8 |
202121046588-DECLARATION OF INVENTORSHIP (FORM 5) [12-10-2021(online)].pdf |
2021-10-12 |
| 9 |
202121046588-COMPLETE SPECIFICATION [12-10-2021(online)].pdf |
2021-10-12 |
| 10 |
Abstract1.jpg |
2021-12-22 |
| 11 |
202121046588-FORM-26 [20-04-2022(online)].pdf |
2022-04-20 |
| 12 |
202121046588-FER.pdf |
2024-02-05 |
| 13 |
202121046588-OTHERS [02-07-2024(online)].pdf |
2024-07-02 |
| 14 |
202121046588-FER_SER_REPLY [02-07-2024(online)].pdf |
2024-07-02 |
| 15 |
202121046588-DRAWING [02-07-2024(online)].pdf |
2024-07-02 |
| 16 |
202121046588-COMPLETE SPECIFICATION [02-07-2024(online)].pdf |
2024-07-02 |
| 17 |
202121046588-CLAIMS [02-07-2024(online)].pdf |
2024-07-02 |
| 18 |
202121046588-ABSTRACT [02-07-2024(online)].pdf |
2024-07-02 |
Search Strategy
| 1 |
202121046588E_22-01-2024.pdf |