[0001] The present disclosure relates to healthcare informatics and diagnostics, and in particular to a method and device for analysing of intelligent automated testing tools in the field of healthcare based on artificial intelligence/machine learning and visualization of results for performance evaluation.

Background:
[0002] Worldwide, several intelligent automated testing tools based on artificial intelligence/machine learning algorithm are being developed and deployed to augment and improve the healthcare workflow by performing predictions, diagnosis and prognostication of disease conditions, diagnosis of disease, prediction of disease and treatment of disease.

[0003] Various methods to evaluate and train the machine learning modules for clinical set up have been disclosed in prior art. One such method described in U.S. Pat. No. US10565477B2, discloses a method to automatically generate an image quality metric for a medical image and process a first medical image using a deployed learning network model. Another U.S. Pat. No. US10628943B2 provides methods and apparatus for improved deep learning for image acquisition.

[0004] The reference U.S. Pat No. US20180144244A1 provides technique for processing of input data (such as input medical imaging data) being performed at a client computing device with the execution of an algorithm of a deep neural network and generates a set of updated training parameters to update the algorithm of the deep neural network, based on user interaction activities (such as user acceptance and user modification in a graphical user interface) that occur with the results of the executed algorithm. Another patent no. EP3101599A2 provides machine learning systems and computerized methods to compare candidate machine learning algorithms.

[0005] Further, none of the references suggest any method or device to enable a clinician to test any intelligent automated tool for diagnostics and determine its utility from a clinical point of view. Moreover, no single framework exists for the validation and testing of these intelligent automated testing tools in the clinical workflow. Most metrics currently in use revolve around measuring the performance of such tools from a statistical perspective which does not take into account clinical scenarios and clinical outcomes. The existing devices for measuring this performance are not intuitive for the clinicians to use them.

[0006] Thus arises, a strong need for a device and a method overcoming the above discussed challenges present in the art and which allows a clinician to test any intelligent automated tool based on artificial intelligence/ machine learning and determine its utility from a clinical point of view, enables clustering of outputs of different intelligent automated testing tools to improve their accuracy and performance, enables visualization of results of intelligent automated testing tools using intuitive graphs for faster performance analysis and provides simplified view and access to various statistical tools.
Objective of invention:
[0007] The main objective of the present disclosure is to provide a method for analysing a plurality of intelligent automated testing tools in a clinical workflow implemented on a computing device. Another objective of the present disclosure is to provide a computing device to analyse a plurality of intelligent automated testing tools in a clinical workflow.

Summary:
[0008] Some or all of the above mentioned problems related to informatics and diagnostics in a healthcare system are proposed to be addressed by certain embodiments of the present disclosure. The problem of linking the intelligent automated testing tools and intelligent solutions to existing hospital information system can be provided by certain embodiments of the invention.
[0009] According to an aspect of the invention, there is disclosed a method for analysing a plurality of intelligent automated testing tools in a clinical workflow implemented on a computing device, the method comprising: executing evaluation techniques of intelligent automated testing tools; adjusting the capability of the intelligent automated testing tools with the application of dynamic thresholding; displaying the capability of the intelligent automated testing tools by using different visualization tools; and linking the intelligent automated testing tools to existing hospital information systems using communication techniques.
[0010] According to another aspect of the invention, there is disclosed a computing device computing device to analyse a plurality of intelligent automated testing tools in a clinical workflow comprising: a processor for executing evaluation techniques of the intelligent automated testing tools; a machine learning module configured to adjust the capability of the intelligent automated testing tools; an output device coupled with the processor to display the capability of intelligent automated tools; and communication system to link the intelligent automated testing tools to existing hospital information systems.
[0011] According to an embodiment of the invention, the evaluation techniques of intelligent automated testing tools are executed on any operating system of the computing device.
[0012] According to another embodiment of the invention, the communication techniques include Dicom communication, HL7 (health level 7) communication and REST APIs.

[0013] According to yet another embodiment of the invention, the displaying the capability of the intelligent automated testing tools comprises: loading the intelligent automated testing tools onto the computing device; plotting a chart with Y axis indicating the probability estimate of each case; displaying the distribution of the cases in the performance space of the intelligent automated testing tools; and determining the capability of the intelligent automated testing tools by an accuracy metric for different situations with different thresholds.

[0014] According to yet another embodiment of the invention, the accuracy metric of the intelligent automated testing tools comprises: loading the intelligent automated testing tools onto the computing device; plotting a chart with Y axis indicating the distance of localization between the location determined by the intelligent automated testing tools and the real location as determined by the existing standards; measuring the accuracy of detection of the intelligent automated testing tools for different situations with different thresholds; and determining cases for review.

[0015] According to yet another embodiment of the invention, the intelligent automated testing tools provide probability estimates of a particular finding.

[0016] According to yet another embodiment of the invention, the dynamic thresholding is applied on the probability estimate to find the presence of a particular finding using an intelligent automated testing tool comprises: a probability estimate higher than the dynamic thresholding implies the presence of a particular finding; a probability estimates lower than the dynamic thresholding implies the absence of a particular finding; and dynamic thresholding is based on the clinical situation.
[0017] According to yet another embodiment of the invention, the analysis of the intelligent automated testing tools enables clustering of various intelligent solutions to improve their accuracy and performance.
[0018] According to still another embodiment of the invention, the computing device is selected from any one of a hand held device a computer or a virtual computer on cloud.
[0019] Other embodiments, systems, methods, apparatus aspects, and features of the invention will become apparent to those skilled in the art from the following detailed description, the accompanying drawings, and the appended claims.

Brief Description of drawings:
[0020] The invention will be further illustrated in detailed description with reference to the appended drawings, which are not necessarily drawn to scale.

FIG. 1 depicts a schematic representation of the proposed Analytics Platform in the clinical environment, to which an embodiment of this invention is applicable;

FIG. 2 shows a block diagram indicating the generation of diagnostic test report using the proposed analytics platform of the present disclosure;

FIG. 3 shows a system integration architecture for deployment of various intelligent automated testing tools for diagnosis;

FIG. 4(a) and (b) show visualization of testing techniques of the proposed analytics platform in a viewer according to an embodiment of this invention;

Figure 5 (a) and (b): Visualization 1: Determining Far North and Far South Failures;

Figure 5(c): Visualization 2: Measuring accuracy of detection; and

Figure 5(d): Visualization 3: Determining relevant cases for review.

Detailed description:
[0021] The invention is now described more fully hereinafter with reference to the accompanying drawings. It should be noted that each of the exemplary embodiments presented and discussed herein should not insinuate limitations of the present subject matter.

[0022] With the digitization of healthcare processes, arises a need to analyse a large volume of patient imaging data generated at every medical facility. A number of artificial intelligence and computer assisted evaluation techniques and applications are used to evaluate the medical data and diagnose a disease. These techniques assist the healthcare professionals to diagnose and treat a patient well by addressing gaps in several sub domains of medicine. The subdomains that are most benefitted include Radiology, Pathology, Dermatology and Ophthalmology among others as these involve analysing medical images. Traditionally a special skill set is developed by Doctors after several years of training to be able to analyse the patient’s imaging data for accurately diagnosing a disease.

[0023] Among the most addressed field is the field of radiology and imaging, which generally involves the acquisition of ‘pictures’ using devices such as MRI scanners, CT scanners, X-Ray Scanners, Ultrasound Scanners, PET scanners and other imaging modalities. Today, most radiology scans are read by highly skilled doctors called ‘radiologists’ who look at these scans and make a diagnosis about what disease might be present in a patient.

[0024] A large number of research groups and companies around the world are creating tools for intelligent automating some of the tasks that radiologists perform. The end-customer for most of these developers is either a hospital or an imaging centre, or a doctor. However, the ecosystem is faced with a paradoxical state, where these intelligent automated testing tools built to address the shortage of workforce are to be tested by the same workforce in demand, and further acutely worsen the demand-supply gap.

[0025] The present disclosure provides a computing device with a proposed Analytics Platform which is an integrative platform comprising of machine learning module. The proposed analytics platform allows the users to test and deploy any kind of artificial intelligence based intelligent automated testing tools in a time and resource efficient manner.

[0026] Figure 1 depicts a schematic representation of the proposed Analytics Platform in the clinical environment, to which an embodiment of this invention is applicable. The proposed analytics platform is hosted on a computing device (3) in a clinical environment. The images (1) from the scanner (2) are pushed to the proposed analytics platform in the computing device (3). The proposed analytics platform comprises of intelligent automated tool (4) and testing and visualization tool (5) and after processing the results (6) get pushed to the output device (7) for display and to radiologist work station (8).

[0027] The proposed analytics platform has three main features:
1) Testing of intelligent automated testing tools (4): The computing device (3) hosting the proposed analytics platform, presents detailed evaluation techniques and methodologies for various types of intelligent automated testing tools (4). These evaluation techniques are used by doctors, technologists, engineers, scientists etc., to test the capability of the intelligent automated tool (4) to determine its implementability into the real clinical workflow.

2) Deployment of intelligent automated tool (4): The proposed analytics platform allows to implement these intelligent automated testing tools (4) into the clinical workflow using the concept of Dynamic Thresholds. The dynamic thresholding enables the user to tweak the results of an intelligent automated tool (4) to better suit their clinical environment. After establishing the dynamic thresholds, the proposed analytics platform connects to the users existing technology infrastructure (Hospital Information Systems (HIS), Picture Archival and Communication (PACS) and Radiology Information System (RIS)) to give the user access to intelligent automated tool (3) from external developers.

3) Dedicated Viewer (7) for intelligent automated tool (4): The proposed analytics platform also features a dedicated Dicom medical image viewer or output device (7) which enables the user to view the results (6) of intelligent automated testing tools (4) without actually storing them on their own IT infrastructure. The viewer (7) takes output results (6) from intelligent automated tool (4) in various formats (JSON, NII, DicomSeg, GSPS, etc) and converts them to a standard editable ROI and/or text field which the user can edit. The Viewer (7) enables the user to change the results/findings (6) of the intelligent automated tool (4), and store the changed findings (6) in the backend by the proposed analytics platform. Also the proposed analytics platform allows the documentation of this change in findings (6) which is critical to advance the development of intelligent automated tool (4). The proposed analytics platform enables quick and efficient documentation of training of the intelligent automated tool (4).

[0028] Figure 2 shows a block diagram indicating the generation of diagnostic test report using proposed analytics platform. The images (1) generated by any scanning device (2) are pushed to the intelligent automated tool (4) having a machine learning module (12) and to truth determination/analysis (9). The intelligent automated tool (4) provides the result (6) and the truth determination/analysis (9) provides a prediction output result (13). Both the results (6) and the prediction output result (13) can be viewed on the viewer (7). Then the results (6) and prediction output result (13) are compare and after the visualization of results (10), a detailed report (11) is generated.

[0029] The proposed analytics platform has the following components: back end – integration, front-end integration and validation of intelligent automated tool (4). The backend integration comprises integration of intelligent automated tool (4) and integration of DICOM. The integration of the intelligent automated tool (4) is done through Docker containers which is a containerization environment and allows the execution of intelligent automated tool (4) on any operating system of the computing device (3). The DICOM integration involves using various communication techniques to link up with various healthcare softwares like, – Hospital Information Systems (HIS), Picture Archival and Communication (PACS) and Radiology Information System (RIS). These techniques include DICOM communication, HL7 (Health Level 7) communication and REST APIs.

[0030] The integration architecture for deployment of various intelligent automated testing tools (4) for diagnostics is shown in figure 3. The front-end integration by proposed analytics platform comprises a dedicated viewer (7) for intelligent automated testing tools (4). The dedicated viewer (7) has the ability to show results produced by any intelligent automated testing tools (4). The intelligent automated testing tools (4) are based on three kinds of computing techniques: classifiers, segmentation and super-resolution. DICOM represents Digital Imaging and Communications in Medicine and is the standard for the communication and management of medical imaging information and related data. DICOM is most commonly used for storing and transmitting medical images (1) enabling the integration of medical imaging devices (2) such as scanners, servers, workstations, printers, network hardware, and picture archiving and communication systems (PACS) from multiple manufacturers. The standard includes a file format definition and a network communications protocol that uses TCP/IP to communicate between systems.

[0031] Each of these computing techniques generate different type of output results (6). In fact, each developer of such intelligent automated testing tools (4) provides output results (6) in different ways (JSON, DicomSeg, GSPS, etc). proposed analytics platform’s dedicated viewer (7) takes various types of output results (6) and converts it all into the same type of output results (6) to be seen by the user. Note that the output results (6) generated by proposed analytics platform’s dedicated viewer (7) can be changed by the user and both the original and changed outputs results (6) are preserved by proposed analytics platform enabling the future improvement of such intelligent automated testing tools (4). Figure 4(a) and (b) show visualization of testing techniques of the proposed analytics platform in a viewer according to an embodiment of this invention.

[0032] Validation of intelligent automated testing tools (4) comprises of visualization techniques which enables users to validate and test any kind of intelligent automated testing tools (4) using some proprietary metrics, techniques and visualizations. The description of these visualization tools are as follows: Visualization 1 for determining Far North and Far South Failures as shown in figures 5(a) and (b).

[0033] These figures 5(a) and (b) demonstrate a comparison (14) of inference results (6) of an intelligent automated tool (4) against the original truth (aka Ground truth) (13) for the inferred class of pathology. The Y- axis represents the prediction output results (10) of the machine learning module (12) of intelligent automated tool (4). Each of the individual dots represents a discrete subject or study. These dots will be represented in four different colours, one for each of the true positive, true negative, false positive and false negative classes. The false positive inference results (10) near the top end of the graph on y-axis is a Far-North failure and the false negative inference result near the lower end of the graph is a Far South failure.

[0034] The visualization techniques measure the accuracy of detection of the results (6) provided by the intelligent automated testing tools (4). This visualization tool (5) shows comparison of inference results (6) and the original truth (13) in more than one dimension as shown in figures 5(c). The dimensions represented are size and location of disease entity, lung nodule in this particular example. The distance of the dot on the y axis represents the distance of the prediction from the ground truth (9) whereas the difference in the size of the multiple coloured dots represents the difference in size between the predictions and the ground truth (9).

[0035] After measuring the accuracy of detection of the results (6) provided by the intelligent automated testing tools (4), relevant cases for review are determined. This visualization tool (5) enables comparison of predictions of multiple focal points for classes among themselves without ground truth (9) in its true sense as shown in figure 5(d). It utilises a function to segregate the foci using a threshold function to enable faster review of clinically relevant foci. These visualization results (10) are evolving and present a framework for quick and efficient measurement of performance of intelligent automated testing tools (4).

[0036] Dynamic Thresholding: Most intelligent automated testing tools (4) give output results (6) in the form of probability estimates for the presence/absence of a particular finding. A threshold needs to be applied onto these probabilities based on which a decision is made with respect to a finding being present or not in a particular image. A probability that is higher than the threshold implies that the finding is present and a probability lower than the threshold implies that the finding is not present. The concept of dynamic thresholds says that the threshold at which a finding is called out as present or absent should change based on the clinical situation – for example the threshold for a life-threatening acute finding (such as pneumothorax) should be high in a situation where a patient walks in for a screening chest X-ray vs the situation where the patient is in the intensive care unit. proposed analytics platform allows the determination of such dynamic thresholds for each clinical scenario using the visualizations above.

[0037] The embodiments of the present invention may be configured as a system, method, or computer program product as described above. The embodiments of the present invention may comprise of various means including completely of hardware, completely of software, or any combination thereof. Further, the embodiments of the present invention may be realized by a computer program product on a computer readable storage medium. The computer readable storage medium may comprise of CD-ROMs, hard disks, magnetic storage devices or optical storage devices.

[0038] Although the illustrative embodiments of the present disclosure have been described herein with reference to the accompanying drawings, it is to be understood that the disclosure is not limited to those precise embodiments, and that various other changes and modifications may be affected therein by one skilled in the art. That is, those skilled in the art will envision other modifications within the scope and spirit of the claims appended hereto.

WE CLAIM:

1.A method for analysing a plurality of intelligent automated testing tools (4) in a clinical workflow implemented on a computing device (3), the method comprising:

executing evaluation techniques of intelligent automated testing tools (4);

adjusting the capability of the intelligent automated testing tools (4) with the application of dynamic thresholding;

displaying the capability of the intelligent automated testing tools (4) by using different visualization tools; and

linking the intelligent automated testing tools (4) to existing hospital information systems using communication techniques.

2. The method as claimed in claim 1, wherein the evaluation techniques of intelligent automated testing tools (4) are executed on any operating system of the computing device (3).

3. The method as claimed in claim 1, wherein the communication techniques include Dicom communication, HL7 (health level 7) communication and REST APIs.

4. The method as claimed in claim 1, wherein the displaying the capability of the intelligent automated testing tools (4) comprises:

loading the intelligent automated testing tools (4) onto the computing device (3);

plotting a chart with Y axis indicating the probability estimate of each case;

displaying the distribution of the cases in the performance space of the intelligent automated testing tools (4); and

determining the capability of the intelligent automated testing tools (4) by an accuracy metric for different situations with different thresholds.

5. The method as claimed in claim 4, wherein the accuracy metric of the intelligent automated testing tools (4) comprises:

loading the intelligent automated testing tools (4) onto the computing device (3);
plotting a chart with Y axis indicating the distance of localization between the location determined by the intelligent automated testing tools (4) and the real location as determined by the existing standards; and

measuring the accuracy of detection of the intelligent automated testing tools (4) for different situations with different thresholds.

6. The method as claimed in claim 1, wherein the intelligent automated testing tools (4) provide probability estimates of a particular finding.

7. The method as claimed in claim 1, wherein the dynamic thresholding is applied on the probability estimate to find the presence of a particular finding using an intelligent automated testing tool (4) comprises:

a probability estimate higher than the dynamic thresholding implies the presence of a particular finding;

a probability estimates lower than the dynamic thresholding implies the absence of a particular finding; and

dynamic thresholding is based on the clinical situation.

8. The method as claimed in claim 1, wherein the analysis of the intelligent automated testing tools (4) enables clustering of various intelligent solutions to improve their accuracy and performance.

9. A computing device to analyse a plurality of intelligent automated testing tools (4) in a clinical workflow comprising:

a processor for executing evaluation techniques of the intelligent automated testing tools (4);

a machine learning module configured to adjust the capability of the intelligent automated testing tools (4);

an output device coupled with the processor to display the capability of intelligent automated tools (4); and

communication system to link the intelligent automated testing tools (4) to existing hospital information systems.

10. The computing device as claimed in claim 9 is selected from any one of a hand held device a computer or a virtual computer on cloud.