Generally, the invention relates to surgical instruments. More
specifically, the invention relates to method and system for detecting missing surgical
instruments.
Background
[002] Recent changes in healthcare, such as better resource management
and a reduction in instrument inventories necessitates collaboration between
healthcare professionals and manufacturers to establish policies and methods, as well
as equipment, to meet current demands. A successful surgical operation needs a
collaborative effort from a group of caring specialists with various roles. Current
industry practice of surgery includes very densely packed case and trays with
instruments that are hard to find if misplaced. Moreover, in order to perform a
successful surgery,many instruments are used. Hence, if any surgical instrument goes
missing while performing surgery, a patient’s life could be at risk. Therefore, surgical
instruments processing is essential for enhancing a safe surgery provided to patients
in hospitals.
[003] Traditionally, keeping track of all the necessary instruments required for
performing the surgery has been a tedious task, as it requires a substantial amount of
time and human concentration. Moreover, in many instances, intraoperative surgery
may be delayed due to absences of surgical instruments. This may lead to increase in
surgery time and cost, and also increases the risk of patient’s life. The burden on
hospital staff to ensure timely availability of surgical instruments can be reduced if
problem of missing surgical instruments required for performing surgery can be
successfully solved. This in turn may reduce conflict and stress among hospital staff,
save money for providing better facilities to the patients, and provide better level of
care to the patients.
[004] Therefore, there is a need for implementing an efficient and reliable
system and method for detecting missing surgical instruments.
Docket No: IIP-HCL-P0093
-3-
SUMMARY OF INVENTION
[005] In one embodiment, a method for detecting missing surgical instruments
is disclosed. The method may include capturing an image of a kit configured to store
each of a plurality of surgical instruments in an associated slot from a plurality of slots
within the kit. It should be noted that, the plurality of surgical instruments are uniquely
mapped to the kit. The method may include processing the image using an Artificial
Intelligence (AI) model. It should be noted that, the AI model is trained using a plurality
of images of each of the plurality of surgical instruments, the plurality of slots within
the kit, and the kit. The method may include determining presence of each of the
plurality of surgical instruments within the kit based on a result of the processing using
the AI model. The method may include identifying at least one issue associated with
one or more surgical instruments from the plurality of surgical instruments in response
to the determining. The method may include highlighting, via a User Interface (UI) the
at least one issue associated with one or more surgical instruments. It should be noted
that, the at least one issue is highlighted by generating a graphical element
corresponding to the one or more surgical instruments. The method may include
prompting a user to perform a corrective action corresponding to the one or more
surgical instruments to resolve the at least one issue.
[006] In another embodiment, an electronic device for detecting missing
surgical instruments. The electronic device comprising a camera, an Input/Output
display, a processor communicatively coupled to the camera and the I/O display unit,
and a memory communicatively coupled to the processor. The memory stores
processor executable instructions, which, on execution, causes the processor to
prompt the camera to capture an image of a kit. It should be noted that, the kit is
configured to store each of a plurality of surgical instruments in an associated slot from
a plurality of slots within the kit. Additionally, the plurality of surgical instruments are
uniquely mapped to the kit. The processor-executable instructions, on execution, may
further cause the processor to process the image using an Artificial Intelligence (AI)
model. It should be noted that, the AI model is trained using a plurality of images of
each of the plurality of surgical instruments, the plurality of slots within the kit, and the
kit. The processor-executable instructions, on execution, may further cause the
processor to determine presence of each of the plurality of surgical instruments within
Docket No: IIP-HCL-P0093
-4-
the kit based on a result of the processing using the AI model. The processorexecutable instructions, on execution, may further cause the processor to identify at
least one issue associated with one or more surgical instruments from the plurality of
surgical instruments in response to the presence determined. The processorexecutable instructions, on execution, may further cause the processor to highlight,
via a User Interface (UI) of the I/O display, the at least one issue associated with one
or more surgical instruments. It should be noted that, the at least one issue is
highlighted by generating a graphical element corresponding to the one or more
surgical instruments. The processor-executable instructions, on execution, may further
cause the processor to prompt a user to perform a corrective action corresponding to
the one or more surgical instruments to resolve the at least one issue.
[007] In yet another embodiment, a system for detecting missing surgical
instruments. The system includes a processor and a memory communicatively
coupled to the processor. The memory may store processor-executable instructions,
which, on execution, may causes the processor to capture an image of a kit configured
to store each of a plurality of surgical instruments in an associated slot from a plurality
of slots within the kit. It should be noted that, the plurality of surgical instruments are
uniquely mapped to the kit. The processor-executable instructions, on execution, may
further cause the processor to process the image using an Artificial Intelligence (AI)
model. It should be noted that, the AI model is trained using a plurality of images of
each of the plurality of surgical instruments, the plurality of slots within the kit, and the
kit. The processor-executable instructions, on execution, may further cause the
processor to determine presence of each of the plurality of surgical instruments within
the kit based on a result of the processing using the AI model. The processorexecutable instructions, on execution, may further cause the processor to identify at
least one issue associated with one or more surgical instruments from the plurality of
surgical instruments in response to the determining. The processor-executable
instructions, on execution, may further cause the processor to highlight, via a User
Interface (UI), the at least one issue associated with one or more surgical instruments.
It should be noted that, the at least one issue is highlighted by generating a graphical
element corresponding to the one or more surgical instruments. The processorexecutable instructions, on execution, may further cause the processor to prompt a
user to perform a corrective action corresponding to the one or more surgical
instruments to resolve the at least one issue.
Docket No: IIP-HCL-P0093
-5-
[008] It is to be understood that both the foregoing general description and the
following detailed description are exemplary and explanatory only and are not
restrictive of the invention, as claimed.
BRIEF DESCRIPTION OF THE DRAWINGS
[009] The present application can be best understood by reference to the
following description taken in conjunction with the accompanying drawing figures, in
which like parts may be referred to by like numerals.
[010] FIG. 1 illustrates a functional block diagram of a system for detecting
missing surgical instruments, in accordance with an embodiment.
[011] FIG. 2 illustrates a flowchart of a method for detecting missing surgical
instruments, in accordance with an embodiment.
[012] FIG. 3 illustrates a flowchart of a method for training an Artificial
Intelligence (AI) model to detect at least one issue associated with one or more
surgical instruments from a plurality of surgical instruments, in accordance with an
embodiment.
[013] FIG. 4 represents a flow diagram of a process for pre-processing an input
data required to train an AI model, in accordance with some exemplary embodiments.
[014] FIG. 5 illustrates a flowchart of a method for performing a corrective
action by a user in response to identification of absence of one or more surgical
instruments, in accordance with an embodiment.
[015] FIG. 6 represents a flow diagram of a process for training an AI model
for detecting absence of one or more surgical instruments, in accordance with some
exemplary embodiments.
[016] FIG. 7 illustrates detection of absence of one or more surgical
instruments and subsequent corrective action, in accordance with some exemplary
embodiments.
[017] FIG. 8 illustrates a flowchart of a method of performing a corrective
action by a user in response to identification of misplacement of one or more surgical
instruments, in accordance with an embodiment.
[018] FIG. 9 illustrates a flow diagram of a process for training an AI model for
detecting misplacement of one or more surgical instruments, in accordance with some
exemplary embodiments.
Docket No: IIP-HCL-P0093
-6-
[019] FIG. 10 illustrates detection of misplacement of one or more surgical
instruments and subsequent corrective action, in accordance with some exemplary
embodiments.
[020] FIGs. 11A - 11B illustrate assembling of a surgical kit with a plurality of
surgical instruments, in accordance with some exemplary embodiments.
[021] FIG. 12 illustrates various steps executed to assemble a surgical kit, in
accordance with some exemplary embodiments.
[022] FIG. 13 illustrates pre-processing of multimedia information captured for
a surgical kit, in accordance with some exemplary embodiments.
[023] FIG. 14 represents a table depicting performance evaluation of
localization to determine absence of one or more surgical instruments, in accordance
with some exemplary embodiments.
[024] FIG. 15 represents results of localization for determining absence of one
or more surgical instruments, in accordance with some exemplary embodiments.
DETAILED DESCRIPTION OF THE DRAWINGS
[025] The following description is presented to enable a person of ordinary skill
in the art to make and use the invention and is provided in the context of particular
applications and their requirements. Various modifications to the embodiments will be
readily apparent to those skilled in the art, and the generic principles defined herein
may be applied to other embodiments and applications without departing from the spirit
and scope of the invention. Moreover, in the following description, numerous details
are set forth for the purpose of explanation. However, one of ordinary skill in the art
will realize that the invention might be practiced without the use of these specific
details. In other instances, well-known structures and devices are shown in block
diagram form in order not to obscure the description of the invention with unnecessary
detail. Thus, the invention is not intended to be limited to the embodiments shown, but
is to be accorded the widest scope consistent with the principles and features
disclosed herein.
[026] While the invention is described in terms of particular examples and
illustrative figures, those of ordinary skill in the art will recognize that the invention is
not limited to the examples or figures described. Those skilled in the art will recognize
that the operations of the various embodiments may be implemented using hardware,
Docket No: IIP-HCL-P0093
-7-
software, firmware, or combinations thereof, as appropriate. For example, some
processes can be carried out using processors or other digital circuitry under the
control of software, firmware, or hard-wired logic. (The term “logic” herein refers to
fixed hardware, programmable logic and/or an appropriate combination thereof, as
would be recognized by one skilled in the art to carry out the recited functions.)
Software and firmware can be stored on computer-readable storage media. Some
other processes can be implemented using analog circuitry, as is well known to one of
ordinary skill in the art. Additionally, memory or other storage, as well as
communication components, may be employed in embodiments of the invention.
[027] A functional block diagram of a system 100 for detecting missing
instruments, is illustrated in FIG. 1, in accordance with an embodiment. The system
100 may be configured to detect missing surgical instruments, misplaced surgical
instruments, wrong placement of surgical instruments from multiple combinations of
kits without requirement of human intervention. The kits may be surgical kits
specifically configured to receive and store surgical instruments. In particular, the
system 100 may include an electronic device 102. The electronic device 102 may be
configured to identify at least one issue associated with one or more surgical
instruments. The at least one issue, for example, may include, but is not limited to
missing surgical instruments and misplaced or wrongly placed surgical instruments.
[028] In order to detect the at least one issue associated with one or more
surgical instruments, the electronic device 102 may be communicatively coupled to an
Artificial Intelligence (AI) model 104. In some embodiments, the electronic device 102
may include the AI model 104 as an edge implementation. The AI model 104 may be
trained based on a plurality of images of each of the plurality of surgical instruments,
a plurality of slots within the kit, and the kit itself. The process of training the AI model
104 has been explained in greater detail in conjunction with FIG. 3 and FIG. 4.
Examples of the electronic device 102 may include, but are not limited to, a server, a
desktop, a laptop, a notebook, a tablet, a smartphone, a mobile phone, an application
server, or the like.
[029] The electronic device 102 may include a camera 106, an Input/Output
unit 108, a memory 112, and a processor 114. The I/O unit 108 may further include a
user interface 110. A user or an administrator may interact with the electronic device
102 and vice versa through the I/O unit 108. Initially, the electronic device 102 may
scan a unique kit identifier code associated with a kit. The unique kit identifier code
Docket No: IIP-HCL-P0093
-8-
may correspond to a Quick Response (QR) code or a bar code. The unique kit identifier
code may be scanned to identify the kit, determine a utilization category of the kit, and
a type of each of the plurality of surgical instruments that go into the kit. Moreover, the
kit may be configured to store each of a plurality of surgical instruments in an
associated slot from a plurality of slots within the kit. Additionally, each of the plurality
of surgical instruments may be uniquely mapped to the kit.
[030] In an embodiment, the scanning of the unique kit identifier code may be
done via the camera 106 inbuilt in the electronic device 102 or a plurality of external
devices 124 communicatively coupled to the electronic device 102. Examples of the
external devices 124 may include, but is not limited to fixed-lens rangefinder camera,
digital single-lens reflex (DSLR) camera, robotic-arm based camera, wearable
camera, smart glasses, and HoloLens. In addition, the electronic device 102 may work
in conjunction with the Augmented Reality (AR) and mixed reality (MR) devices in
order to determine the at least one issue in real-time.
[031] Once the kit is identified based on the unique kit identifier code
associated with the kit, the electronic device 102 may be configured to retrieve
information associated with the plurality of surgical instruments.In an embodiment, the
information associated with the plurality of surgical instruments may include at least
one of number of surgical instruments in the kit, type of surgical instruments, material
used to build surgical instruments, shape of surgical instruments, size of surgical
instruments, and nature of surgical process associated with surgical instruments.
[032] In one embodiment, once the information associated with each of the
plurality of surgical instruments is retrieved, the electronic device 102 may prompt a
camera to capture a multimedia information associated with the kit. The multimedia
information associated with the kit may include, but is not limited to text, an image, a
video, a live view of the kit, or a combination thereof. The camera may correspond to
the inbuilt camera 106 of the electronic device 102 or a camera of one of the plurality
of external device 124.
[033] In another embodiment, the electronic device 102 may receive the
multimedia information associated with the kit from one of the plurality of external
device 124. Upon receiving, the multimedia information, the electronic device 102 may
be configured to process each of the multimedia information associated with the kit
using the AI model 104. Based on a result of the processing of the multimedia
Docket No: IIP-HCL-P0093
-9-
information, the electronic device 102 may be configured to determine presence of
each of the plurality of surgical instruments within the kit.
[034] Based on the presence determined for each of the plurality of surgical
instruments, the electronic device 102 may be configured to identify the at least one
issue associated with one or more surgical instruments from the plurality of surgical
instruments. The at least one issue associated with the one or more surgical
instruments may include, but is not limited to absence of the one or more surgical
instruments, misplacement of the one or more surgical instruments, incorrect
placement of the one or more surgical instruments, and incorrect orientation of the one
or more surgical instruments. It should be noted that, for ease of explanation, the
identification of the at least one issue is explained in conjunction of one kit.
[035] Upon identifying the at least one issue associated with the one or more
surgical instruments, the electronic device 102 may be configured to highlight the at
least one issue associated with the one or more surgical instruments. The highlighting
of the at least one issue may be done via the user interface 110. Moreover, the at least
one issue may be highlighted by generating a graphical element corresponding to the
one or more surgical instruments. Further, the electronic device 102 may prompt a
user to perform a corrective action corresponding to the one or more surgical
instruments to resolve the at least one issue. In an embodiment, the corrective actions
may correspond to placing of the one or more surgical instruments that is currently
absent from the kit, repositioning of the one or more surgical instruments misplaced
within the kit, changing the one or more surgical instruments incorrectly placed in the
kit, and correcting the incorrect orientation of the one or more surgical instruments
within the kit.
[036] The I/O unit 108 may be used to display results (i.e., the result of
processing of the multimedia information, and the at least one issue associated with
the one or more surgical instruments, the at least one issue highlighted using the
graphical element, etc.) based on actions performed by the electronic device 102, to
the user (i.e., a person working in a warehouse of surgical instruments, or a person
working in a sterile processing department (SPD) of a hospital). Moreover, the I/O unit
108 may be used to display the utilization category of the kit, the type of each of the
plurality of surgical instruments that go into the kit, and the information associated with
the plurality of surgical instruments. In addition, the I/O unit 108 may be used to display
an option provided to the user to select a job from the list of jobs to be performed
Docket No: IIP-HCL-P0093
-10-
corresponding to the kit, the corrective action performed in response to identification
of the at least one issue identified associated with the one or more surgical
instruments.
[037] The user interface 110 may also be used by the user to provide inputs
to the electronic device 102. Thus, for example, in some embodiment, the electronic
device 102 may ingest an input that includes a user selection of the corrective action
in response to the at least one issue identified. Further, in some embodiments, the
electronic device 102 may render intermediate results (e.g., the information associated
with each of the plurality of surgical instruments, and the at least one issue highlighted)
or final results (e.g., the corrective action) to the user via the user interface 110.
[038] The memory 112 may store instructions that, when executed by the
processor 114, may cause the processor 114 to detect missing surgical instruments
from the kit. The processor 114 may perform detection of missing surgical instruments
based on the multimedia information captured, in accordance with some
embodiments. As will be described in greater detail in conjunction with FIG. 2 to FIG.
11, in order to perform detection of missing surgical instruments, the processor 114 in
conjunction with the memory 112 may perform various functions including processing
of the multimedia information, detection of presence of each of the plurality of surgical
instruments within the kit, identification of the at least one issue associated with the
one or more surgical instruments from the plurality of surgical instruments, and
highlighting of the at least one issue associated with one or more surgical instruments,
etc.
[039] The memory 112 also store various data (e.g., the multimedia
information of the kit, the information of each of the plurality of surgical instrument, the
result of the processing of the multimedia information, etc.) that may be captured,
processed, and/or required by the electronic device 102. The memory 112 may be a
non-volatile memory (e.g., flash memory, Read Only Memory (ROM), Programmable
ROM (PROM), Erasable PROM (EPROM), Electrically EPROM (EEPROM) memory,
etc.) or a volatile memory (e.g., Dynamic Random Access Memory (DRAM), Static
Random-Access memory (SRAM), etc.).
[040] The electronic device 102 may be connected to a database 116. The
database 116 may be used to store the information associated with each of the
plurality of surgical instruments, and information of the kit based on the unique kit
identifier code associated with the kit. In addition, the database 116 may store results
Docket No: IIP-HCL-P0093
-11-
generated based on the processing of the multimedia information associated with the
kit. Additionally, the database 116 may be periodically updated based on the corrective
action performed by the user in response to the at least one issue identified for the
one or more surgical instruments.
[041] Further, the electronic device 102 may interact with a server 118 or the
external devices 124 over a network 122 for sending and receiving various data. The
network 122, for example, may be any wired or wireless communication network and
the examples may include, but may be not limited to, the Internet, Wireless Local Area
Network (WLAN), Wi-Fi, Long Term Evolution (LTE), Worldwide Interoperability for
Microwave Access (WiMAX), and General Packet Radio Service (GPRS).
[042] In some embodiment, the electronic device 102 may fetch information
associated with each of the plurality of surgical instruments from the server 118.
Moreover, the server 118 may be a cloud server. The server 118 may enable the
processor 114 of the electronic device 102 to perform various functionalities including
processing of the multimedia information, detection of presence of each of the plurality
of surgical instruments within the kit, identification of the at least one issue associated
with the one or more surgical instruments from the plurality of surgical instruments,
and highlighting of the at least one issue associated with the one or more surgical
instruments, etc. In addition, the server 118 may provide information associated with
the plurality of surgical instruments to the plurality of users. The server 118 may further
include a database 120. The database 120 may store information associated with the
plurality of surgical instruments. By way of an example, the database 120 may store
the information associated with the plurality of surgical instruments in order to
distinguish a new surgical instrument from existing once. The database 120, may be
periodically updated based on information associated with the new surgical
instrument. Alternatively, the electronic device 102 may receive the user input from
one of the external devices 124.
[043] Referring now to FIG. 2, a flowchart of a method for detecting missing
surgical instruments is illustrated, in accordance with an embodiment. At step 202, a
unique kit identifier code associated with the kit is scanned. The unique kit identifier
code may be scanned to identify the kit, determine a utilization category of the kit, and
a type of each of the plurality of surgical instruments that go into the kit. In an
embodiment, the unique kit identifier code may correspond to one of a QR code or a
bar code. At step 204, information associated with each of the plurality of surgical
Docket No: IIP-HCL-P0093
-12-
instruments may be retrieved in response to identification of the kit. In other words,
once the kit is identified, the information associated with each of the plurality of surgical
instruments present in the kit may be retrieved. In an embodiment, the information
associated with the plurality of surgical instruments includes at least one of number of
surgical instruments in the kit, type of surgical instruments, material used to build
surgical instruments, shape of surgical instruments, size of surgical instruments, and
nature of surgical process associated with surgical instruments.
[044] Once the information associated with the plurality of surgical instruments
that goes into the kit is retrieved, at step 206, a multimedia information associated with
the kit may be captured. In an embodiment, the kit may be configured to store each of
the plurality of surgical instruments in an associated slot from a plurality of slots within
the kit. Each of the plurality of surgical instruments may be uniquely mapped to the kit.
The multimedia information associated with the kit may correspond to one of a text, an
image, a video, a live view of the kit, or a combination thereof. Once the multimedia
information associated with the kit is captured, at step 208, the retrieved multimedia
information may be processed using an AI model. In reference to FIG. 1, the AI model
may correspond to the AI model 104. Moreover, the AI model used for processing the
retrieved multimedia information may be trained using a plurality of images of each of
the plurality of surgical instruments, the plurality of slots within the kit, and the kit. In
reference to FIG. 1, the processing of the retrieved multimedia information may
happen in the cloud server (i.e., the server 118). Further, the result of the processing
may be rendered to the user via the I/O unit 108 of the electronic device 102.
[045] Further, at step 210, presence of each of the plurality of surgical
instruments within the kit may be determined. In an embodiment, the presence of each
of the plurality of surgical instruments may be determined based on the processing of
the multimedia information of the kit and the information retrieved for each of the
plurality of surgical instruments based on scanning of the unique kit identifier code
associated with the kit. In some embodiments, in addition to determining the presence
of each of the plurality of surgical instruments, at step 212, ageing of each of the
plurality of surgical instruments may be detected. The ageing of each of the plurality
of surgical instruments may be detected based on a plurality of ageing factors. In an
embodiment, the plurality of ageing factors may include, but are not limited to at least
one of discoloration or fading of each of the plurality of surgical instruments, surface
Docket No: IIP-HCL-P0093
-13-
wear, scratches, kinks, bends, and dull edge for sharp instruments from the plurality
of surgical instruments.
[046] At step 214, at least one issue associated with one or more surgical
instruments from the plurality of surgical instruments may be identified. In an
embodiment, the at least one issue associated with the one or more surgical
instruments may include, but is not limited to absence of the one or more surgical
instruments, misplacement of the one or more surgical instruments, incorrect
placement of the one or more surgical instruments, and incorrect orientation of the one
or more surgical instruments.
[047] Upon identifying the at least one issue associated with the one or more
surgical instruments, at step 216, the at least one issue associated with the one or
more surgical instruments may be highlighted. In an embodiment, in order to highlight
the at least one issue, a graphical element may be generated corresponding to the
one or more surgical instruments. The graphical element generated for highlighting
the at least one issue may be a bounding box generated around the at least one issue.
[048] In some embodiments, the at least one issue may be highlighted by
overlaying the captured multimedia information of the kit with the retrieved information
in response to identification of the kit. By way of an example, an image captured of the
kit configured to store the plurality of surgical instrument in the associated slot may be
overlayed with an image of the kit identified based on scanning of the unique kit
identifier code. In the image, the kit may include each of the plurality of surgical
instruments in the designated slots. A process of identification of the at least one issue
associated with the one or more surgical instruments has been explained in greater
detail in conjunction with FIG. 5 - FIG. 8.
[049] Once the at least one issue associated with the kit is highlighted, at step
218, a user may be prompted to perform a corrective action. The corrective action is
to resolve the at least one issue associated with the one or more surgical instrument.
In an embodiment, the corrective action performed to resolve the at least one issue
may correspond to placing of a surgical instrument that is currently absent from the kit
in the associated slot within the kit, repositing a surgical instrument misplaced within
the kit, changing a surgical instrument that is incorrectly placed within the kit, and
correcting the incorrect orientation of a surgical instrument in the kit.
[050] Referring now to FIG. 3, a flowchart of a method for training an AI model
to detect at least one issue associated with one or more surgical instruments from a
Docket No: IIP-HCL-P0093
-14-
plurality of surgical instruments is illustrated, in accordance with an embodiment. With
reference to FIG. 1, the AI model may correspond to the AI model 104. In an
embodiment, the training of the AI model is done using a deep learning algorithm.
Examples of deep learning algorithms may include, but is not limited to Convolutional
Neural Network (CNN), Recurrent Neural Networks (RNNs), Long Short-Term Memory
Networks (LSTMs), Stacked Auto-Encoders, Deep Boltzmann Machine (DBM), Deep
Belief Networks (DBN).
[051] At step 302, an image of a kit may be captured. The kit may include the
plurality of surgical instruments. Each of the plurality of surgical instruments may be
placed in an associated slot in the kit. Once the image of the kit is captured, at step
304, an input data may be received. The input data may include a set of training
images. In an embodiment, the set of training images may include training images of
the kit, training images of each of the plurality of surgical instruments, and training
images of the empty kit. Additionally, each of the set of training images may be
captured in a plurality of orientations and lighting conditions. The orientation and the
lighting condition may include rotated camera condition, different viewing angles,
different perspectives of angle, day light, room light, camera flashlight, and dimly-lit
environment. In an embodiment, the different perspectives of angle may correspond
to a normal angle and a wide angle of camera. In reference to FIG.1, the camera may
be the camera 106 and or a camera within one of the plurality of external devices 124.
[052] Upon receiving the input data, at step 306, the input data may be preprocessed. In an embodiment, the pre-processing of the input data is done to assess
and enhance quality of each of the set of training images of the input data. Moreover,
the pre-processing of the input data may be based on a Structural Similarity Index
(SSI) metric and Visual Information Fidelity (VIF) metric associated with each of the
set of training images. In an embodiment, in order to perform the pre-processing of the
input data a Continuous Image Quality Assessment and Enhancement (CIQAE)
technique may be used. The CIQAE technique assesses visual quality of the input
data and converts the input data in a suitable form using enhancements.
[053] In order to convert the input data into the suitable form, quality of each
of the set of training images may be enhanced using image pre-processing techniques.
Examples of image pre-processing techniques may include, but is not limited to
averaging, histogram equalization, and image sharpening. In an embodiment,
histogram equalization may use cumulative distribution function to improve contrast of
Docket No: IIP-HCL-P0093
-15-
each of the set of training images in order to handle illumination issue. Further, image
sharpening may be done to generate high frequency images (obtained through high
pass filters) for each of the set of training images. The image sharpening preprocessing technique may enhance high frequency components of each of the set of
training images which may get blurred or smoothed out and are not easily extracted.
An advantage of performing the pre-processing techniques includes understanding of
overall shape and structure of an object (e.g., each of the plurality of surgical
instruments, the associated slot, etc.) that may disappear due to poor quality of the set
of training images.
[054] Moreover, the conversion of the input data in the suitable form provides
more generic detection of patterns in the input data of interest irrespective of capturing
conditions of each of the set of training images of the input data. In an embodiment,
capturing conditions associated with each of the set of training images may correspond
to the plurality of orientations and lighting conditions. In the CIQAE technique, the SSI
metric and the VIF metric may be used to assess quality of originally captured input
data and augmented input data. In other words, quality of each of the set of training
images captured initially may be compared with each of the set of training images
produced after augmentation. Augmentation is used to artificially expand size of each
of the set of training images by creating modified versions of each of the set of training
images of the input data.
[055] The SSI metric may capture structural similarity between each of the set
of training images. This may help to make the set of training images more inclusive for
object of interest to efficiently train the AI model. Additionally, the VIF metric may be
used to verify visual quality of each of the set of training images that are augmented,
without manual inspection. Further, a threshold may be pre-defined for each of the SSI
metric and the VIF metric in order to decide a level of allowed degradation of each of
the set of training images. In one embodiment, when each of the SSI metric and the
VIF metric may be determined to be less than the pre-defined threshold, then one or
more of the set of training images may be processed using image enhancement.
Further, even after performing image enhancement for the one or more of the set of
training images, if the visual quality the one or more training images is not improved,
then the one or more training images may be discarded. This is done to avoid feeding
bad quality data to train the AI model. The process of the pre-processing of the input
data has been explained in greater detail in conjunction to FIG. 5.
Docket No: IIP-HCL-P0093
-16-
[056] Once the input data is pre-processed, at step 308, each of the plurality
of surgical instruments may be mapped with the associated slot in the empty kit.
Moreover, the mapping of each of the plurality of surgical instruments may be done
based on the set of training images. Further, at least one error may be generated
based on the mapping of each of the plurality of surgical instruments with the
associated slot in the empty kit.
[057] At step 310, the at least one error generated may be corrected. Errors
thus generated may be iteratively corrected. Based on iterative corrections performed
for the at least one error, the AI model may get trained corresponding to each of the
plurality of surgical instruments with the associated slot in the empty kit. In an
embodiment, the training of the AI model for each of the plurality of surgical
instruments with the associated slot in the empty kit includes detection of each of the
plurality of surgical instruments with the associated slot in the empty kit in different
array/configurations of the empty kit. In other words, the AI model may be iteratively
trained to detect the associated slot in the empty kit assigned to each of the plurality
of surgical instruments even in different array/configurations of the empty kit.
[058] By way of an example, initially, the AI model may receive the input data.
The input data may include the set of training images. Further, the set of training
images may include training images of the kit, training images of each of the plurality
of surgical instruments, and training images of the empty kit. In an embodiment, the
training images of the kit may include images of a set of kits. Each of the set of kits
may be associated with the unique kit identifier code. Further, each of the set of kit
may include the plurality of surgical instruments placed in the associated slot in the kit.
Moreover, each of the plurality of surgical instruments and the associated slot may be
provided with the unique identifier. Upon receiving the input data, the AI model may
iteratively train itself by mapping training images of each of the plurality of surgical
instruments with the associated slot present in the training images of the empty kit.
[059] The mapping of each of the plurality of surgical instruments with the
associated slot present in the training images of the empty kit may be done using the
training images of each of the set of kits. Further, upon receiving a new image of the
kit, the AI model may map the new image with the training images of each of the set
of kits. Based on mapping, the AI model may be configured to identify the at least one
issue associated with the kit. Further, the AI model may prompt the user to perform
the corrective action in order to rectify the at least one issue identified.
Docket No: IIP-HCL-P0093
-17-
[060] Referring now to FIG. 4, a flow diagram of a process for pre-processing
an input data required to train an AI model is represented, in accordance with an
exemplary embodiment. At step 402, the AI model may receive the input data. The
input data may include training images of the kit, training images of each of the plurality
of surgical instruments, and training images of the empty kit. With reference to FIG. 1,
the AI model may correspond to the AI model 104. Upon receiving the input data, the
AI model may perform Image Quality Assessment (IQA) for the input data. With
reference to FIG. 3, the IQA may be performed using the CIQAE technique. Based on
the IQA performed, at step 404, the AI model may obtain the IQA metrices for the input
data. In an embodiment, the IQA metrics may correspond to the SSI metric and the
VIF metric generated for the input data.
[061] Once the IQA metrices are obtained, at step 406, data augmentation for
the input data may be done. In an embodiment, the data augmentation may be done
to increase volume of the input data required for training the AI model. The AI model
may be trained using the deep learning algorithms. During data augmentation, any
issue in the input data may be addressed using the data augmentation techniques.
Examples of the data augmentation techniques may include, but are not limited to,
averaging, rotation, gaussian blurring, noise addition, cropping, flipping, translation,
color spaces. In an embodiment, the issue associated with the input data may
correspond to blurriness, noisiness, dullness, brightness, low contrast, intra-class
variation, scale variation, view-point variation, occlusion, illumination, and similar other
issues. However, quality of the input data may get impacted due to extreme
augmentation conditions. Hence, identification of objects of interest from the input data
may become difficult. Due to this, desired object from the input data may be missed.
Moreover, an aspect ratio which is one of the useful features in object identification
may get disturbed leading to false predictions by the AI model. Therefore, to avoid
such false prediction, the CIQAE technique is used to perform the data augmentation.
[062] Once the data augmentation for the input data is done, at step 408, the
IQA metrices for the augmented input data may be obtained. Based on the IQA
metrices received, at step 410, a check may be performed to determine quality of the
input data. Once the check is performed, at step 412, the input data (i.e., training data)
may be fed to the AI model, in order to train the AI model. Upon receiving the input
data by the AI model, at step 414, a classification model may be configured to classify
the received input data. In an embodiment, data classification for the input data may
Docket No: IIP-HCL-P0093
-18-
be performed to organize the input data. In order to organize the input data, a unique
identifier code may be assigned to each of the training images of the kit, training
images of each of the plurality of surgical instruments, and the associated slot present
in each of the training images of the empty kit.
[063] Further, at step 416, localization of the input data may be performed by
a localization model. In an embodiment, the localization of the input data may be done
to enable storing of the input data within boundaries in order to process the input data
locally, thereby protecting the input data and its misuse. At step 418, a hyper
parameter tuning may be performed for the classified input data and the localized input
data. The hyper parameter tuning may be performed in order to choose a set of optimal
hyperparameters from the classified input data and the localized input data. Once the
set of optimal hyperparameters are chosen, at step 420, optimization and training of
the Al model may be performed based on the set of optimal hyper parameters.
[064] Referring now to FIG. 5, a flowchart of a method of a corrective action
performed by a user in response to identification of absence of one or more surgical
instruments from a kit is illustrated, in accordance with an embodiment. Once absence
of one or more surgical instruments from a kit has been identified, the user, at step
502, may be prompted to perform the corrective action corresponding to the absence
of the one or more surgical instruments. In other words, the user may be prompted to
perform the corrective action upon detecting the one or more surgical instruments
missing from the kit. By way of an example, in a hospital the absence of one or more
surgical instruments in the kit may be identified before starting a surgery or even after
the surgery has been completed to make sure that all surgical instruments are placed
back into the kit. By way of another example, absence of one or more surgical
instruments may be identified in the warehouse during completion of the kit in order to
assemble the kit and dispatch it to the hospital to perform a particular surgery.
Similarly, the absence of one or more surgical instruments may be identified in the
warehouse, when the kit is received back from the hospital. A process performed to
setup the kit with the plurality of surgical instruments required to perform the particular
surgery has been explained in greater detail in conjunction with FIG. 9 – FIG. 11.
[065] Upon identifying that the one or more surgical instruments are absent in
the kit, at step 504, a first graphical element may be generated to highlight absence of
the one or more surgical instruments that are absent from the kit. The first graphical
element, for example, may be a bounding box that may be rendered around slots
Docket No: IIP-HCL-P0093
-19-
associated with the one or more surgical instruments that are absent in the kit.
Contemporaneous to generation of the first graphical element, at step 506, a second
graphical element may be generated. The second graphical element may provide
directions to the user to place at least one of the one or more surgical instruments in
the associated slot within the kit. By way of an example, in order to provide directions
to the user, the second graphical element may be generated corresponding to the at
least one of the one or more surgical instruments from a pile of surgical instruments
placed on a table. The second graphical element, for example, may be a combination
of a bounding box placed over the absent surgical instrument placed on the table and
an arrow indicating placing back the absent surgical instrument into the associated
slot in the kit.
[066] Further, at step 508, a database may be updated upon detecting
successful completion of the corrective action performed by the user. By way of an
example, the database may be updated once the user places the at least one of the
one or more surgical instrument absent from the kit in the associated slot within the
kit. In an embodiment, the database may be allocated on a cloud server or a local
repository. A process to detecting absence of the one or more surgical instruments is
explained in greater detail in conjunction to FIG. 7.
[067] Referring now to FIG. 6, a flow diagram of a process for training an AI
model to detect absence of the one or more surgical instruments is illustrated, in
accordance with an exemplary embodiment. With reference to FIG. 1, the AI model
may correspond to the AI model 104. At step 602, an input data may be received by
the AI model. In an embodiment, the input data may include a set of training images.
Further, the set of training images may include training images of the kit, training
images of each of the plurality of surgical instruments, and training images of the
empty kit. Once the input data is received, at step 604, the classification model may
be configured to detect absence of the one or more surgical instruments.
[068] Based on processing performed by the classification model to detect the
absence, at step 606, if absence of the one or more surgical instruments is detected
to be true, at step 608, a localization model may be configured to determine the one
or more surgical instruments that are absent in the kit. In addition to the determination,
the localization model may be configured to predict the unique identifier associated
with the one or more surgical instrument that is absent from the kit.
Docket No: IIP-HCL-P0093
-20-
[069] Referring now to FIG. 7, detection of absence of one or more surgical
instruments and subsequent corrective action is illustrated, in accordance with an
exemplary embodiment. As explained before, the detection may be via a mobile device
or an AR device. Initially the multimedia information associated with the kit may be
captured. For ease of explanation, the multimedia information captured may
correspond to an image 702 of the kit as depicted in FIG. 7. The kit may include the
plurality of surgical instruments placed in the associated slots within the kit. Further,
the captured image 702 of the kit may be processed using the AI model. Based on
processing of the captured image 702 of the kit, presence of each of the plurality of
surgical instruments in the kit may be determined. Further, based on determination of
the presence of each of the plurality of surgical instruments, the at least one issue
associated with the one or more surgical instruments may be identified.
[070] In the present embodiment, the at least one issue associated with the kit
may correspond to absence of a surgical instrument from the plurality of surgical
instruments. Upon identification of the absence of the surgical instrument, the surgical
instrument in the captured image 702 of the kit may be highlighted using a bounding
box 704 (i.e., the first graphical element) that may be rendered over a slot that is
configured to receive the surgical instrument.
[071] Thereafter, a bounding box 706 (i.e., the second graphical element) may
be rendered over the surgical instrument placed over a surgical table 708 within a pile
of surgical instruments. This may provide directions to the user to place the surgical
instrument currently resting on the surgical table 708 in the associated slot highlighted
by the bounding box 704 within the kit. By using the bounding boxes 704 and 706, the
user may be prompted to perform the corrective action, that is, to place the surgical
instrument in the associated slot in the kit as depicted in 710.
[072] Referring now to FIG. 8, a flowchart for a method of performing a
corrective action by a user in response to identification of misplacement of one or more
surgical instruments is illustrated, in accordance with an embodiment. Once
misplacement of one or more surgical instruments from a kit has been identified, at
step 802, the user may be prompted to perform the corrective action in response to
identification of misplacement of the one or more surgical instruments. In other words,
the user may be prompted to perform the corrective action upon detecting that the one
or more surgical instruments misplaced within the kit.
Docket No: IIP-HCL-P0093
-21-
[073] The misplacement may imply that the one or more surgical instruments
are incorrectly positioned within the kit. Incorrectly positioned surgical instrument may
imply that either the surgical instrument is placed in the wrong slot or is not oriented
properly in its slot. In an embodiment, the misplacement of the one or more surgical
instruments may be identified at the hospital or the warehouse. By way of an example,
in the hospital the misplacement of the one or more surgical instruments may be
identified before starting a surgery or after completion of the surgery. In addition, the
misplacement of the one or more surgical instruments may be identified in the
warehouse during completion of the kit in order to assemble the kit to perform a
particular surgery or when the kit has been received from the hospital.
[074] Upon identifying the one or more surgical instruments being incorrectly
positioned within the kit, at step 804, a third graphical element may be generated. The
third graphical element may be used to highlight misplacement of the one or more
surgical instruments based on the identification. Once the third graphical element is
generated, at step 806, a fourth graphical element may be generated to provide
directions to the user to reposition at least one of the one or more surgical instruments
that are incorrectly positioned within the kit. By way of an example, in order to provide
directions to the user, the fourth graphical element may be a bounding box that may
be rendered over the at least one of the one or more surgical instruments incorrectly
positioned within the kit.
[075] Further, at step 808, a database may be updated upon detecting
successful completion of the corrective action performed by the user. By way of an
example, the database may be updated once the user correctly places (i.e.,
repositions) the at least one of the one or more surgical instrument that were
incorrectly positioned within the kit. In an embodiment, the database may in a cloud
server or a local repository. A process to detecting the misplacement of the one or
more surgical instruments is explained in greater detail in conjunction to FIG. 10.
[076] Referring now to FIG. 9, a flow diagram of a process for training an AI
model for detecting misplacement of one or more surgical instruments is illustrated, in
accordance with an exemplary embodiment. With reference to FIG. 1, the AI model
may correspond to the AI model 104. At step 902, an input data may be received by
the AI model. In an embodiment, the input data may include a set of training images.
Further, the set of training images may include training images of the kit, training
images of each of the plurality of surgical instruments, and training images of the
Docket No: IIP-HCL-P0093
-22-
empty kit. Once the input data is received, at step 904, the classification model may
be configured to detect misplacement of the one or more surgical instruments.
[077] Based on processing performed by the classification model to detect the
misplacement, at step 906, if the misplacement of the one or more surgical instruments
is detected to be true, at step 908, a localization model may be configured to determine
the one or more surgical instruments being incorrectly positioned within the kit. In
addition to the determination, the localization model may be configured to predict the
unique identifier associated with the one or more surgical instrument that is being
incorrectly positioned within the kit.
[078] Referring now to FIG. 10, detection of misplacement of one or more
surgical instruments and subsequent corrective action is illustrated, in accordance with
an exemplary embodiment. As explained before, the detection may be via a mobile
device or an AR device. Initially themultimedia information associated with the kit may
be captured. For ease of explanation, the multimedia information captured may
correspond to an image 1002 of the kit as depicted in the present FIG. 10. The kit may
include the plurality of surgical instruments placed in the associated slots within the
kit. Further, the captured image 1002 of the kit may be processed using the AI model.
Based on processing of the captured image 1002 of the kit, presence of each of the
plurality of surgical instruments in the kit may be determined. Further, based on
determination of the presence of each of the plurality of surgical instruments, the at
least one issue associated with the one or more surgical instruments may be identified.
[079] In the present embodiment, the at least one issue associated with the kit
may correspond to misplacement of a surgical instrument from the plurality of surgical
instruments within the kit. Upon identification of the misplacement of the surgical
instrument, the surgical instrument in the captured image 1002 of the kit, may be
highlighted using a bounding box 1004 (i.e., the third graphical element) that may be
rendered over a slot that is configured to correctly place the surgical instrument. The
bounding box 1004 rendered over the slot may depict wrong placement of the surgical
instrument in the kit.
[080] Thereafter, a bounding box 1006 (i.e., the fourth graphical element) may
be rendered over the slot to depict correct placement of the surgical instrument in the
kit. This may provide directions to the user to reposition the surgical instrument in the
associated slot within the kit based on the highlighted bounding box 1006. By using
the bounding boxes 1004 and 1006, the user may be prompted to perform the
Docket No: IIP-HCL-P0093
-23-
corrective action, that is, to correctly place the surgical instrument in the associated
slot in the kit as depicted in 1008.
[081] Referring now to FIGs. 11A – 11B, assembling of a surgical kit with a
plurality of surgical instruments is illustrated, in accordance with some exemplary
embodiments. In FIG. 11A, an AR headset 1102 is used to assemble the surgical kit.
The AR headset 1102 may correspond to an AR device, for example, a HOLOLENSTM
or GOOGLETM glasses. In some embodiment, instead of the AR headset 1102, a
Mixed Reality (MR) headset may be used. At 1104, a worker 1106 (also referred as
the user) may place an empty kit 1108 on a table in order to assemble the empty kit
1108 with the plurality of surgical instruments that may also be placed on the same
table as a pile. At 1110, the worker 1106 wears the AR headset 1102 and views the
kit 1108via the AR headset 1102. Thereafter, the worker 1106 may start placing each
of the plurality of surgical instruments one by one in the empty kit 1108. In an
embodiment, the worker 1106 may be a technician working in the warehouse or a
technician working in the SPD department of the hospital.
[082] Seeing through the AR headset 1102 may enable the worker 1106 to
see various graphical elements superimposed over the view of the empty kit 1108
and/or the pile of the plurality of surgical instruments. For example, as depicted at
1112,when the worker 1106 looks at the kit 1108 via the AR headset 1102, an empty
slot may be highlighted by rendering a bounding box 1114 over the empty slot in order
to indicate that one of the one or more of the plurality of surgical instruments may be
placed in that empty slot. Thereafter, at 1116, the worker 1106 may look at the pile of
the plurality of surgical instruments placed on the table.
[083] Further, using the AR headset 1102, at 1118, the worker 1106 may
identify a surgical instrument from the pile of the plurality of surgical instruments that
is to be placed in the empty slot identified by the bounding box 1114. To this end, the
AR headset 1102 may highlight the surgical instruments by rendering a bounding box
1120 around the surgical instrument.Thus, AR headset 1102 may prompt and provide
directions to the worker 1106 to place the surgical instrument currently placed on the
table, into the associated slot highlighted by the bounding box 1114. This may be
repeated for each empty slot in the it 1108, thereby assembling the kit 1108 with each
of the plurality of surgical instruments placed inside the kit 1108 in the designated slots
as depicted in 1122. In a similar manner, FIG. 11B depicts assembling of the kit 1108
Docket No: IIP-HCL-P0093
-24-
with the plurality of surgical instrument using a mobile device 1124. In this exemplary
embodiment, the AR device 1102 is replaced by the mobile device 1124.
[084] Referring now to FIG. 12, various steps executed to assemble a surgical
kit are illustrated, in accordance with some exemplary embodiments. At step 1202, a
technician 1204 (i.e., the user) may use an electronic device 1206 to launch an
application on the electronic device 1206. In reference to FIG. 1, the electronic device
1206 may correspond to the electronic device 102. In an embodiment, the technician
1204 may correspond to the person working in the warehouse or the person working
in the SPD of the hospital. Once the application is launched on the electronic device
1206, at step 1208, the technician 1204 may login in a Job portal configured for the
plurality of surgical instruments stored in the surgical kit. In order to login in the job
portal, the technician 1204 may use his user credentials. The user credentials may
include, but are not limited to, the username and the password.
[085] Once the technician 1204 has logged in to the job portal, at step 1210, a
list of jobs corresponding to the plurality of surgical instruments stored in the surgical
kit that are required to be performed by the technician 1204 may be displayed. In an
embodiment, the list of jobs may include, but is not limited to, setup of the kit,
determination of presence of each of the plurality of surgical instruments in the kit,
identification of the at least one issue associated with the one or more surgical
instruments, highlighting the at least one issue associated with the one or more
surgical instruments, and providing directions to the technician 1204 to perform the
corrective action associated with the at least one issue.
[086] At step 1212, the technician 1204 may select a job from the list of jobs
displayed. By way of an example, the technician 1204 may select identification of the
at least one issue associated with the one or more surgical instruments. In present
embodiment, the at least one issue may correspond to the incorrect placement of at
least one of the one or more surgical instruments within the kit. In other words, the
incorrect placement may refer to wrong instrument being placed in the slot designated
for another surgical instrument. Based on selection of the job, at step 1214, the
technician 1204 may scan the unique identifier (also referred as the unique device
identifier code) associated with each of the plurality of surgical instruments as depicted
via 1218. The unique identifier associated with each of the plurality of surgical
instruments may be scanned to capture the multimedia information associated with
the each of the plurality of surgical instruments. The multimedia information may
Docket No: IIP-HCL-P0093
-25-
correspond to an image captured for the each of the plurality of surgical instruments.
In an embodiment, the technician 1204 may scan the unique identifier using a camera
1216 (also referred as commodity camera). With reference to FIG. 1, the camera 1216
may correspond to one of the camera 106 and the plurality of external devices 124.
[087] Further, at step 1220, the technician 1204 may place a surgical kit 1222
under the camera 1216 in order to scan the surgical kit 1222. The surgical kit 1222
may be configured to store each of the plurality of surgical instruments in the
designated slot within the surgical kit 1222. In an embodiment, the surgical kit 1222
may also be referred as a case or a tray. Moreover, the surgical kit 1222 may be
scanned to capture the multimedia information associated with the surgical kit 1222.
In present embodiment, the multimedia information may correspond to an image of
the surgical kit 1222. Once the multimedia media information associated with each of
the plurality of surgical instruments and the surgical instrument kit is captured, at step
1224, the captured multimedia information may be sent to a cloud server database
1226 for further analysis. In an embodiment, further analysis of the multimedia
information captured may be performed using an image processing technique.
[088] Based on the analysis performed, at step 1228, the application installed
on the electronic device 1206 may be configured to detect absence of a surgical
instrument from the surgical kit 1222. In an embodiment, absence of the surgical
instrument may be detected by highlighting an empty slot or the missing surgical
instrument. In order to highlight the empty slot, a bounding box 1230 may be rendered
on the empty slot. Further, the application may prompt the technician 1204 to perform
the corrective action in order to correct the at least one issue, i.e., placement of the
missing surgical instrument in the surgical kit 1222. Once the corrective action is
performed by the technician 1204, at step 1232, the application may perform a check
to determine completeness of the surgical kit 1222.
[089] In one embodiment, if the technician 1204 fails to perform the corrective
action, the surgical kit 1222 may be determined to be incomplete as depicted via step
1234. Upon determining the surgical kit 1222 to be incomplete, the surgical kit 1222
may be marked as not ready to be used. However, if the technician 1204 performs the
corrective action, the surgical kit 1222 may be determined to be complete as depicted
via step 1236. Upon determining the surgical kit 1222 to be complete, the surgical kit
1222 may be marked as ready to be used.
Docket No: IIP-HCL-P0093
-26-
[090] Referring now to FIG. 13, pre-processing of multimedia information
captured for a surgical kit is illustrated, in accordance with some exemplary
embodiments. At 1302, the multimedia information captured for the kit is represented.
In an embodiment, the surgical kit may be configured to store each of the plurality of
surgical instrument in the associated slot from a plurality of slots in the surgical kit.
Moreover, each of the plurality of surgical instruments may be uniquely mapped to the
surgical kit. The multimedia information captured for the surgical kit may include, but
is not limited to, a text, a video, and an image. In the present FIG. 13, the multimedia
information captured for the surgical kit may correspond to an image 1304 captured
for the surgical kit.
[091] Further, the captured image 1304 may be pre-processed using the
CIQAE technique. In order to pre-process the captured image 1304, initially, value for
the SSI metric and the VIF metric associated with the captured image 1304 may be
calculated. By way of an example, the value for the SSI metric calculated for the
captured image 1304 may be ‘0.037’. In addition, the value of the VIF metric calculated
for the captured image 1304 may be ‘0.70’. Once the values for the SSI metric and the
VIF metric associated with the captured image 1304 have been calculated, the
captured image 1304 may be pre-processed using the CIQAE technique. The preprocessing performed for the captured image 1304 using the CIQAE technique may
include enhancement of values of the SSI metric and the VIF metric. In other words,
the pre-processing of the captured image 1304 may be performed to assess and
enhance quality of the captured image 1304.
[092] Once the captured image 1304 is pre-processed, a pre-processed image
1306 may be depicted as represented, via step 1308. By way of an example, in order
to assess and enhance quality of the captured image 1304, enhancement of values of
the SSI metric and the VIF metric may be performed. The enhanced value of the SSI
metric for the pre-processed image 1306 may then become ‘0.88’ and the enhanced
value of the VIF metric for the pre-processed image 1306 may become ‘0.99’.
[093] Referring now to FIG. 14, a table 1400 depicting performance evaluation
of localization to determine absence of one or more surgical instruments is illustrated,
in accordance with some exemplary embodiments. In order to perform performance
evaluation of localization to determine absence of the one or more surgical
instruments, percentage mean Average Precision (mAP) may be used. In order to
calculate percentage mAP for the one or more surgical instruments absent from the
Docket No: IIP-HCL-P0093
-27-
surgical kit, a true positive value and a false negative value may be calculated for each
of the one or more surgical instruments.
[094] For calculating the true positive values and the false negative values, an
intersection of union (IoU) value may be considered. In an embodiment, the IoU value
may correspond to a threshold value. The threshold value may be pre-defined in order
to determine probability of absence of the one or more surgical instruments. The
probability of absence of the one or more surgical instruments may be determined
based on ground truth associated with the one or more surgical instruments absent
from the surgical kit and predictions made for the absence of the one or more surgical
instruments by an AI model. With reference to FIG. 1, the AI model may correspond
to the AI model 104. By way of an example, the pre-defined threshold value (i.e., value
defined for the IoU) may correspond to ‘0.3’. In one embodiment, when the IoU
calculated for at least one of the one or more surgical instruments absent from the
surgical kit considering the ground truth and the predictions made is greater than ‘0.3’,
than the IoU may be considered to be the true positive value. In other words, when the
IoU calculated for the at least one of the one or more surgical instruments absent from
the surgical kit is greater than ‘0.3,’ then the at least one of the one or more surgical
instruments may be determined to be absent from the surgical kit.
[095] In another embodiment, when the IoU calculated for at least one of the
one or more surgical instruments absent from the surgical kit considering the ground
truth and the predictions made is less than ‘0.3,’ then the IoU is considered to be the
false negative value. In other words, when the IoU calculated for the at least one of
the one or more surgical instruments absent from the kit is less than ‘0.3,’ then the at
least one of the one or more surgical instrument may be determined to be present in
the kit. The precision for absence of the one or more surgical instruments may be
calculated using an equation (1) one represented below:
Precision = True positive / (True positive + False positive) … (1)
[096] Further, recall curve for the absence of the one or more of the plurality
of surgical instruments may be calculated using an equation (2) represented below:
Recall = True positive / (True positive + False negative) … (2)
Docket No: IIP-HCL-P0093
-28-
[097] In the equation (1) and the equation (2) above, true positive, false
positive, and false negative may be computed based on intersection over union (IoU).
The IoU may be ratio of area of intersection and area of union of predicted bounding
box and ground truth bounding box as represented via an equation (3) below:
IoU = Area of intersection/Area of union … (3)
[098] Once the precision and the recall curve are calculated, an average
precision for all classes of the one or more surgical instrument that are absent from
the surgical kit may be calculated by subtracting the recall curve from the precision
calculated using an equation (4) as depicted below:
Mean Average Precision (mAP) = Average precision for all classes (here
instruments) … (4)
[099] In the equation (4) above, the average precision may be computed as
area under precision-recall curve. Moreover, the precision-recall curve may be
obtained by calculating values for precision and recall at various IoU thresholds. In an
embodiment, classes associated with each of the one or more surgical instruments
may correspond to one of a plurality of categories to which the each of the one or more
surgical instruments belongs to. Examples of the plurality of categories associated with
the plurality of surgical instruments may include, but is not limited to, cutting and
dissecting instruments, grasping or holding instruments, hemostatic instruments,
retractors, and tissue unifying instruments. Once the average precision is calculated,
mean of the average precision may be calculated for all classes of the one or more
surgical instruments absent from the surgical kit. In order to calculate the mean
average precision, addition of the precision calculated for all classes associated with
the one or more surgical instruments may be divided by total number of the one or
more surgical instruments.
[0100] As represented in the table 1400, a column 1402may represent number
of classes in which each of the one or more surgical instruments that are absent from
the surgical kit may be categorized. Further, a column 1404 may represent number of
samples. The number of samples may correspond to the number of the one or more
surgical instruments that belongs to a particular class from the number of classes. By
Docket No: IIP-HCL-P0093
-29-
way of an example, 42 surgical instruments that may be absent may belong to class
one. A column 1406 may represent the true positive value calculated for the one or
more surgical instruments absent from the surgical kit that belongs to each of the
number of classes. A column 1408 may represent the false negative value calculated
for the one or more surgical instruments absent from the surgical kit that belongs to
each of the number of classes. Based on the true positive value and the false negative
value calculated for the one or more surgical instruments, the percentage mAP may
be calculated for the one or more surgical instruments absent from the surgical kit.
The percentage mAP calculated for the one or more surgical instruments may
correspond to percentage average precision as depicted via a column 1410. Further,
the last row of the table 1400 may represent overall values calculated for each column
(i.e., column 1404, column 1406, column 1408, and column 1410) except for the
column 1402 representing number of classes.
[0101] Referring now to FIG. 15, results of localization to determine absence of
the one or more surgical instruments is illustrated, in accordance with some exemplary
embodiments. Initially, an image of the surgical kit may be captured. The surgical kit
may be configured to store the plurality of surgical instruments in the associated slot
within the surgical kit. In the FIG. 15, the image of kit may correspond to an image
1502. Once the image 1502 is captured, the pre-processing of the image 1502 may
be performed using an AI model. In reference to FIG. 1, the AI model may correspond
to the AI model 104. Based on the pre-processing performed, presence of each of the
plurality of surgical instruments in the surgical kit may be detected. Further, the at least
one issue associated with the one or more surgical instruments from the plurality of
surgical instruments placed in the associated slot within the surgical kit may be
identified. In present embodiment, the at least one issue associated with the one or
more surgical instruments may correspond to the absence of the one or more surgical
instruments. In order to predict the absence of the one or more surgical instruments,
initially, the pre-processing of the image 1502 of the surgical kit may be performed as
explained in the FIG. 13. The pre-processing of the image 1502 may be performed to
assess and enhance quality of the image 1502. In an embodiment, the pre-processing
of the image 1502 may be performed based on the SSI metric and the VIF metric.
[0102] Once the image 1502 is pre-processed and the absence of the one or
more surgical instruments is predicted, the performance evaluation of localization of
the one or more surgical instruments may be performed. Based on the performance
Docket No: IIP-HCL-P0093
-30-
evaluation of localization of the one or more surgical instruments, a resulting image
highlighting the absence of the one or more surgical instruments may be depicted as
represented via an image 1504.
[0103] Various embodiments provide method and system for detecting missing
surgical instruments. The disclosed method and system may capture an image of a kit
configured to store each of a plurality of surgical instruments in an associated slot from
a plurality of slots within the kit. The plurality of surgical instruments are uniquely
mapped to the kit. Moreover, the disclosed method and system may process the image
using an Artificial Intelligence (AI) model. The AI model is trained using a plurality of
images of each of the plurality of surgical instruments, the plurality of slots within the
kit, and the kit. Further, the disclosed method and system may determine presence of
each of the plurality of surgical instruments within the kit based on a result of the
processing using the AI model. In addition, the disclosed method and system may
identify at least one issue associated with one or more surgical instruments from the
plurality of surgical instruments in response to the presence determined. Further, the
disclosed method and system may highlight, via a User Interface (UI), the at least one
issue associated with one or more surgical instruments. The at least one issue is
highlighted by generating a graphical element corresponding to the one or more
surgical instruments. Thereafter, the disclosed method and system may prompt a user
to perform a corrective action corresponding to the one or more surgical instruments
to resolve the at least one issue.
[0104] The system and method also disclose an electronic device that may
provide some advantages like, the disclosed electronic device mitigates risk pertaining
non-availability of right surgical kits for orthopedic surgeries. Further, the disclosed
electronic device may be used by a Hospital SPD technician or a warehouse
technician for assembling and verifying a kit carrying a plurality of surgical instruments.
Moreover, the disclosed electronic device may reduce human efforts, thereby reducing
fatigue and minimizing errors. Further, the disclosed electronic device may accelerate
the kit completion and verification process. In addition, the disclosed electronic device
may improve detection accuracy of each of the plurality of surgical instruments, and
processing speed as compared to existing systems such as machine vision systems,
Radio Frequency Identification (RFID) tags with smart boards, color code tagging,
barcodes, and QR codes.
Docket No: IIP-HCL-P0093
-31-
[0105] Further, the disclosed electronic device may work with any commodity
camera. This means that, the disclosed electronic device may not require any
specialized developed equipment’s and tagging devices for detecting missing surgical
instrument. Moreover, the disclosed electronic device minimizes any effects of any
external factors that may affect picture quality such as lighting, shake while handling,
camera resolution, etc. with aid of picture correcting techniques. In addition, the
disclosed electronic device may itself learns over a period, based on each of the
plurality of surgical instruments. The disclosed electronic device does not require
continuous re-training even when configuration of the kit is varying. This is because,
the deep learning algorithm is trained with individual surgical instruments as compared
to specific array of instruments laid out in the kit. Moreover, any condition including
wear and tear, scratches, and discoloration of each of the plurality of surgical
instruments due to its multiple re-use does not affect overall accuracy of the deep
learning algorithm. Additionally, an application installed in the disclosed electronic
device runs from cloud database, hence the application may be light weight and could
be installed in commodity devices.
[0106] It will be appreciated that, for clarity purposes, the above description has
described embodiments of the invention with reference to different functional units and
processors. However, it will be apparent that any suitable distribution of functionality
between different functional units, processors or domains may be used without
detracting from the invention. For example, functionality illustrated to be performed by
separate processors or controllers may be performed by the same processor or
controller. Hence, references to specific functional units are only to be seen as
references to suitable means for providing the described functionality, rather than
indicative of a strict logical or physical structure or organization.
[0107] Although the present invention has been described in connection with
some embodiments, it is not intended to be limited to the specific form set forth herein.
Rather, the scope of the present invention is limited only by the claims. Additionally,
although a feature may appear to be described in connection with particular
embodiments, one skilled in the art would recognize that various features of the
described embodiments may be combined in accordance with the invention.
[0108] Furthermore, although individually listed, a plurality of means, elements
or process steps may be implemented by, for example, a single unit or processor.
Additionally, although individual features may be included in different claims, these
Docket No: IIP-HCL-P0093
-32-
may possibly be advantageously combined, and the inclusion in different claims does
not imply that a combination of features is not feasible and/or advantageous. Also, the
inclusion of a feature in one category of claims does not imply a limitation to this
category, but rather the feature may be equally applicable to other claim categories,
as appropriate.

CLAIMS
WHAT IS CLAIMED IS:
1. A method for detecting missing surgical instruments, the method comprising:
capturing (206) an image of a kit configured to store each of a plurality of
surgical instruments in an associated slot from a plurality of slots within the kit, wherein
the plurality of surgical instruments are uniquely mapped to the kit;
processing (208) the image using an Artificial Intelligence (AI) model (104),
wherein the AI model (104) is trained using a plurality of images of each of the plurality
of surgical instruments, the plurality of slots within the kit, and the kit;
determining (210) presence of each of the plurality of surgical instruments
within the kit based on a result of the processing using the AI model (104);
identifying (214) at least one issue associated with one or more surgical
instruments from the plurality of surgical instruments in response to the determining;
highlighting (216), via a User Interface (UI) (110), the at least one issue
associated with one or more surgical instruments, wherein the at least one issue is
highlighted by generating a graphical element corresponding to the one or more
surgical instruments; and
prompting (218) a user to perform a corrective action corresponding to the one
or more surgical instruments to resolve the at least one issue.
2. The method of claim 1, further comprising:
scanning (202) a unique kit identifier code associated with the kit, wherein the
unique kit identifier code is scanned to identify the kit, determine a utilization category
of the kit, and a type of each of the plurality of surgical instruments that go into the kit;
and
retrieving (204) information associated with the plurality of surgical instruments
in response to identifying the kit, wherein the information associated with the plurality
of surgical instruments comprises at least one of number of surgical instruments in the
kit, type of surgical instruments, material used to build surgical instruments, shape of
Docket No: IIP-HCL-P0093
-34-
surgical instruments, size of surgical instruments, and nature of surgical process
associated with surgical instruments.
3. The method of claim 1, wherein training the AI model (104) comprises:
capturing (302), by the AI model (104), an image of a kit comprising the plurality
of surgical instruments, wherein each of the plurality of surgical instruments are placed
in a designated slot in the kit;
receiving (304), by the AI model (104), an input data, wherein the input data
includes a set of training images, and wherein the set of training images comprises
training images of the kit, training images of each of the plurality of surgical
instruments, and training images of an empty kit, and wherein the set of training
images is captured in a plurality of orientations and lighting conditions, and wherein
the orientation and the lighting condition includes rotated camera condition, different
viewing angles, different perspectives of angle, day light, room light, camera flashlight,
and dimly lit environment;
mapping (308), by the AI model (104), each of the plurality of surgical
instruments with the designated slot in the empty kit, wherein mapping of each of the
plurality of surgical instruments is done based on the set of training images; and
iteratively correcting (310) at least one error generated in response to mapping
of each of the plurality of surgical instruments with the designated slot in the empty kit
by the AI model (104), wherein iterative corrections train the AI model (104).
4. The method of claim 3, further comprises pre-processing (306) the input data to
assess and enhance quality of each of the set of training images of the input data,
wherein pre-processing of the input data is based on a Structural similarity index (SSI)
metric and Visual Information Fidelity (VIF) metric associated with each of the set of
training images.
5. The method of claim 1, wherein determining (210) presence of each of the plurality
of surgical instruments within the kit further comprises predicting (212), by the AI
model (104), ageing of each of the plurality of surgical instruments based on a plurality
of ageing factors, wherein the plurality of ageing factors comprises at least one of
discoloration or fading of each of the plurality of surgical instruments, surface wear,
Docket No: IIP-HCL-P0093
-35-
scratches, kinks, bends, and dull edge for sharp instruments from the plurality of
surgical instruments.
6. The method of claim 1, wherein the at least one issue associated with one or more
surgical instruments corresponds to absence of the one or more surgical instruments,
and wherein prompting (502) the user to perform the corrective action corresponding
to the absence of the one or more surgical instruments comprises:
identifying, by the AI model (104), the at least one issue as the one or more
surgical instruments being absent in the kit;
generating (504), by the AI model (104), a first graphical element highlighting
absence of the one or more surgical instruments based on the identification, wherein
the first graphical element is overlayed on the captured image to highlight the absence;
generating (506), by the AI model (104), a second graphical element providing
directions to the user to place at least one of the one or more surgical instruments in
a designated slot within the kit, wherein the one or more surgical instruments are
placed near the kit, wherein the second graphical element is overlayed on a current
view displayed to the user via an electronic device (102); and
updating (508), by the AI model (104), a database (116) upon detecting
successful completion of the corrective action performed by the user.
7. The method of claim 1, wherein the at least one issue associated with one or more
surgical instruments corresponds to misplacement of the one or more surgical
instruments, and wherein prompting (802) the user to perform the corrective action
corresponding to the misplacement of the one or more surgical instruments comprises:
identifying, by the AI model (104), the at least one issue as the one or more
surgical instruments being incorrectly positioned within the kit;
generating (804), by the AI model (104), a third graphical element highlighting
the one or more surgical instruments based on the identification, wherein the third
graphical element is overlayed on the captured image to highlight misplacement;
generating (806), by the AI model (104), a fourth graphical element providing
directions to the user to reposition at least one of the one or more surgical instruments,
Docket No: IIP-HCL-P0093
-36-
wherein the fourth graphical element is overlayed on a current view displayed to the
user via an electronic device (102); and
updating (808), by the AI model (104), a database (116) upon detecting
successful completion of the corrective action performed by the user.
8. An electronic device (102) for detecting missing surgical instruments, the electronic
device (102) comprising:
a camera (106); wherein the camera (106) corresponds to a commodity
camera, and wherein the commodity camera includes one of a mobile phone camera,
a camera in a mobile phone attachment, a fixed-lens rangefinder camera, a digital
single-lens reflex (DSLR) camera, an industrial machine vision camera, a robotic-arm
based camera, and a wearable camera;
an Input/Output (I/O) display unit (108);
a processor (114) communicatively coupled to the camera (106) and the I/O
display unit (108); and
a memory (112) communicatively coupled to the processor (114), wherein the
memory (112) stores processor executable instructions, which, on execution, causes
the processor (114) to:
prompt the camera to capture (206) an image of a kit, wherein the kit is
configured to store each of a plurality of surgical instruments in an associated
slot from a plurality of slots within the kit, and wherein the plurality of surgical
instruments are uniquely mapped to the kit;
process (208) the image using an Artificial Intelligence (AI) model (104),
wherein the AI model (104) is trained using a plurality of images of each of the
plurality of surgical instruments, the plurality of slots within the kit, and the kit;
determine (210) presence of each of the plurality of surgical instruments
within the kit based on a result of the processing using the AI model (104);
identify (214) at least one issue associated with one or more surgical
instruments from the plurality of surgical instruments in response to the
presence determined;
Docket No: IIP-HCL-P0093
-37-
highlight (216), via a User Interface (UI) (110) of the I/O display (108),
the at least one issue associated with one or more surgical instruments,
wherein the at least one issue is highlighted by generating a graphical element
corresponding to the one or more surgical instruments; and
prompt (218) a user to perform a corrective action corresponding to the
one or more surgical instruments to resolve the at least one issue.
9. A system (100) for detecting missing surgical instruments, the system (100)
comprising:
a processor (114); and
a memory (112) communicatively coupled to the processor (114), wherein the
memory (112) stores processor executable instructions, which, on execution, causes
the processor (114) to:
capture (206) an image of a kit configured to store each of a plurality of
surgical instruments in an associated slot from a plurality of slots within the kit,
wherein the plurality of surgical instruments are uniquely mapped to the kit;
process (208) the image using an Artificial Intelligence (AI) model (104),
wherein the AI model (104) is trained using a plurality of images of each of the
plurality of surgical instruments, the plurality of slots within the kit, and the kit;
determine (210) presence of each of the plurality of surgical instruments
within the kit based on a result of the processing using the AI model (104);
identify (214) at least one issue associated with one or more surgical
instruments from the plurality of surgical instruments in response to the
determining;
highlight (216), via a User Interface (UI) (110), the at least one issue
associated with one or more surgical instruments, wherein the at least one
issue is highlighted by generating a graphical element corresponding to the one
or more surgical instruments; and
prompt (218) a user to perform a corrective action corresponding to the
one or more surgical instruments to resolve the at least one issue.
Docket No: IIP-HCL-P0093
-38-
10. The system (100) of claim 9, wherein the processor executable instructions cause
the processor (114) to:
scan (202) a unique kit identifier code associated with the kit, wherein the
unique kit identifier code is scanned to identify the kit, determine a utilization category
of the kit, and a type of each of the plurality of surgical instruments that go into the kit;
and
retrieve (204) information associated with the plurality of surgical instruments
in response to identifying the kit, wherein the information associated with the plurality
of surgical instruments comprises at least one of number of surgical instruments in the
kit, type of surgical instruments, material used to build surgical instruments, shape of
surgical instruments, size of surgical instruments, and nature of surgical process
associated with surgical instruments.
11. The system (100) of claim 9, wherein, to train the AI model (104), the processor
executable instructions cause the processor (114) to:
capture (302) an image of a kit comprising the plurality of surgical instruments,
wherein each of the plurality of surgical instruments are placed in a designated slot in
the kit;
receive (304) an input data, wherein the input data includes a set of training
images, and wherein the set of training images comprises training images of the kit,
training images of each of the plurality of surgical instruments, and training images of
an empty kit, and wherein the set of training image is captured in a plurality of
orientations and lighting conditions, and wherein the orientation and the lighting
condition includes rotated camera condition, different viewing angles, different
perspectives of angle, day light, room light, camera flashlight, and dimly lit
environment;
map (308) each of the plurality of surgical instruments with the designated slot
in the empty kit, wherein mapping of each of the plurality of surgical instruments is
done based on the set of training images; and
Docket No: IIP-HCL-P0093
-39-
iteratively correct (310) at least one error generated in response to mapping of
each of the plurality of surgical instruments with the designated slot in the empty kit by
the AI model (104), wherein iterative corrections train the AI model (104).
12. The system (100) of claim 11, wherein the processor executable instructions cause
the processor (114) to pre-process (306) the input data to assess and enhance quality
of each of the set of training images of the input data, and wherein pre-processing of
the input data is based on a Structural similarity index (SSI) metric and Visual
Information Fidelity (VIF) metrics associated with each of the set of training images.
13. The system (100) of claim 9, wherein the processor executable instructions cause
the processor (114) to determine (210) presence of each of the plurality of surgical
instruments within the kit by predicting (212) ageing of each of the plurality of surgical
instruments based on a plurality of ageing factors, wherein the plurality of ageing
factors comprises at least one of discoloration or fading of each of the plurality of
surgical instruments, surface wear, scratches, kinks, bends, and dull edge for sharp
instruments from the plurality of surgical instruments.
14. The system (100) of claim 9, wherein the at least one issue associated with one
or more surgical instruments corresponds to absence of the one or more surgical
instrument, and wherein, to prompt (502) the user to perform the corrective action
corresponding to the absence of the one or more surgical instruments, the processor
executable instructions cause the processor (114) to:
identify the at least one issue as the one or more surgical instruments being
absent in the kit;
generate (504) a first graphical element highlighting absence of the one or more
surgical instruments based on the identification, wherein the first graphical element is
overlayed on the captured image to highlight the absence;
generate (506) a second graphical element providing directions to the user to
place at least one of the one or more surgical instruments in a designated slot within
the kit, wherein the one or more surgical instruments are placed near the kit, wherein
the second graphical element is overlayed on a current view displayed to the user via
an electronic device; and
Docket No: IIP-HCL-P0093
-40-
update (508) a database (116) upon detecting successful completion of the
corrective action performed by the user.
15. The system (100) of claim 9, wherein the at least one issue associated with one
or more surgical instruments corresponds to misplacement of the one or more surgical
instruments, and wherein, to prompt (802) the user to perform the corrective action
corresponding to the misplacement of the one or more surgical instruments, the
processor executable instructions cause the processor (114) to:
identify the at least one issue as the one or more surgical instruments being
incorrectly positioned within the kit;
generate (804) a third graphical element highlighting the one or more surgical
instruments based on the identification, wherein the third graphical element is
overlayed on the captured image to highlight the misplacement;
generate (806) a fourth graphical element providing directions to the user to
reposition at least one of the one or more surgical instruments, wherein the fourth
graphical element is overlayed on a current view displayed to the user via an electronic
device; and
update (808) a database (116) upon detecting successful completion of the
corrective action performed by the user.