Claims:1. A computer-implemented method for object parts detection at a computing device, the method comprising:
receiving a video of an object captured using a Augmented Reality (AR) device via one or more hardware processors, the object associated with an object identifier and comprising a plurality of object parts;
assigning, via the one or more hardware processors, priority scores to a set of object parts from the plurality of object parts appearing in a Field of View (FOV) of the AR device in the video based on a deep learning model;
prioritizing the inspection of the set of object parts based on the priority scores of the set of object parts, via the one or more hardware processors; and
retrieving, via the one or more hardware processors, a set of inspection check-lists associated with the inspection of the set of object parts from a data repository associated with the computing device for enabling the inspection of the set of object parts.

2. The method as claimed in claim 1, further comprising identifying the set of object parts using a fast region based convolution network (R-CNN) model.

3. The method as claimed in clam 1, further comprising retrieving device information, inspection history of the device, a question list and a corresponding answer list corresponding to each of the set of inspection lists based on the object identifier from the data repository.

4. The method as claimed in claim 3, wherein the questions list is arranged in a hierarchical order.

5. The method as claimed in claim 1, wherein the video is received from the AR device via a User Data Program (UDP) communication link.

6. The method as claimed in claim 1, further comprising receiving and storing one or more visual and audio evidences of the inspection.

7. The method as claimed in claim 1, wherein assigning the priority scores based on the deep learning model comprises:
generating a plurality of object part proposals for a plurality of segments captured from a plurality of frames of the video, the plurality of object part proposals being dominant in the plurality of frames of the video; and
extracting one or more features of each of the plurality of object part proposals using a deep convolutional neural network.

8. A system for object detection at a computing device, the system comprising:
one or more memories; and
one or more hardware processors, the one or more memories coupled to the one or more hardware processors, wherein the one or more hardware processors are capable of executing programmed instructions stored in the one or more memories to:
receive a video of an object captured using a Augmented Reality (AR) device, the object associated with an object identifier and comprising a plurality of object parts;
assign priority scores to a set of object parts from the plurality of object parts appearing in a Field of View (FOV) of the AR device in the video based on a deep learning model;
prioritize the inspection of the set of object parts based on the priority scores of the set of object parts; and
retrieve a set of inspection check-lists associated with the inspection of the set of object parts from a data repository associated with the computing device for enabling the inspection of the set of object parts.

9. The system as claimed in claim 8, wherein the one or more hardware processors are further configured by the instructions to identify the set of object parts using a fast R-CNN model.

10. The system as claimed in clam 8, wherein the one or more hardware processors are further configured by the instructions to retrieve device information, inspection history of the device, a question list and a corresponding answer list corresponding to each of the set of inspection lists based on the object identifier from the data repository.
11. The system as claimed in claim 10, wherein the questions list is arranged in a hierarchical order.

12. The system as claimed in claim 8, wherein the one or more hardware processors are further configured by the instructions to receive the video from the AR device via a User Data Program (UDP) communication link.

13. The system as claimed in claim 8, wherein the one or more hardware processors are further configured by the instructions to receive and store one or more visual and audio evidences of the inspection.

14. The system as claimed in claim 8, wherein to assign the priority scores based on the deep learning model, wherein the one or more hardware processors are further configured by the instructions to:
generate a plurality of object part proposals for a plurality of segments captured from a plurality of frames of the video, the plurality of object part proposals being dominant in the plurality of frames of the video; and
extract one or more features of each of the plurality of object part proposals using a deep convolutional neural network.
, Description:FORM 2

THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENTS RULES, 2003

COMPLETE SPECIFICATION
(See section 10 and rule 13)

Title of invention:
SYSTEM AND METHOD FOR OBJECT PARTS INSPECTION IN
INDUSTRIAL APPLICATIONS

Applicant:
Tata Consultancy Services Limited
A company Incorporated in India under the Companies Act, 1956
Having address:
Nirmal Building, 9th floor,
Nariman point, Mumbai 400021,
Maharashtra, India

The following specification particularly describes the invention and the manner in which it is to be performed.
TECHNICAL FIELD
[001] The embodiments herein generally relate to object inspection, and, more particularly, to system and method for object parts inspection using Augmented Reality (AR) based re-configurable inspection framework.

BACKGROUND
[002] Inspection refers to an organized examination of particular equipment/process. Inspection typically involves a set of checks to be performed in accordance to a guideline provided by the product, or equipment in an industry. For example, for a vehicle, typically an annual inspection is performed to conform to laws regarding emissions, safety, or both. In the case of industrial assembly and maintenance operations, a number of inspection checks are performed before the product is delivered. For an automobile plant having a flexible manufacturing line, different models with multiple variants are produced in the same assembly line. Hence, the number of unique set of checks that need to be performed in a line is proportional to the number of variants and the number of models.
[003] The objective of said checks involved in inspection is to verify that no defects are present and all compliance tests have been performed. All these checks have to be carried out in a limited amount of time. Apart from vehicles, there are a number of areas where inspection is carried out for safety and quality. Examples of such areas includes but are not limited to oil refineries, petrochemical complexes, chemical plants, cement plants, cross-country pipelines, in shipping industries, for wind mill towers, fabrication shops, and so on.

SUMMARY
[004] The following presents a simplified summary of some embodiments of the disclosure in order to provide a basic understanding of the embodiments. This summary is not an extensive overview of the embodiments. It is not intended to identify key/critical elements of the embodiments or to delineate the scope of the embodiments. Its sole purpose is to present some embodiments in a simplified form as a prelude to the more detailed description that is presented below.
[005] In view of the foregoing, an embodiment herein provides methods and systems for object part detection at a computing device. In one aspect, a computer implemented method for object part detection at a computing device is provided, where the method includes receiving a video of an object captured using an Augmented Reality (AR) device via one or more hardware processors, the object associated with an object identifier and comprising a plurality of object parts. Further the method includes assigning, via the one or more hardware processors, priority scores to a set of object parts from the plurality of object parts appearing in a Field of View (FOV) of the AR device in the video based on a deep learning model. Furthermore, the method includes prioritizing the inspection of the set of object parts based on the priority scores of the set of object parts, via the one or more hardware processors. Also, the method include retrieving, via the one or more hardware processors, a set of inspection check-lists associated with the inspection of the set of object parts from a data repository associated with the computing device for enabling the inspection of the set of object parts.
[006] In another aspect, a system for object part detection at a computing device is provided. The system includes one or more memories; and one or more hardware processors, the one or more memories coupled to the at least one processor and the at least one media sensor wherein the at least one processor is capable of executing programmed instructions stored in the one or more memories to: receive a video of an object captured using an Augmented Reality (AR) device, the object associated with an object identifier and comprising a plurality of object parts. Further, the one or more hardware processors are further configured by the instructions to assign priority scores to a set of object parts from the plurality of object parts appearing in a Field of View (FOV) of the AR device in the video based on a deep learning model. Furthermore, the one or more hardware processors are further configured by the instructions to prioritize the inspection of the set of object parts based on the priority scores of the set of object parts. Also, the one or more hardware processors are further configured by the instructions to retrieve a set of inspection check-lists associated with the inspection of the set of object parts from a data repository associated with the computing device for enabling the inspection of the set of object parts.
[007] In yet another aspect, a non-transitory computer-readable medium having embodied thereon a computer program for executing a method for object part detection at a computing device is provided. The method includes receiving a video of an object captured using an Augmented Reality (AR) device, the object associated with an object identifier and comprising a plurality of object parts. Further the method includes assigning priority scores to a set of object parts from the plurality of object parts appearing in a Field of View (FOV) of the AR device in the video based on a deep learning model. Furthermore, the method includes prioritizing the inspection of the set of object parts based on the priority scores of the set of object parts. Also, the method include retrieving a set of inspection check-lists associated with the inspection of the set of object parts from a data repository associated with the computing device for enabling the inspection of the set of object parts.

BRIEF DESCRIPTION OF THE FIGURES
[008] The detailed description is described with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The same numbers are used throughout the drawings to reference like features and modules.
[009] FIG. 1 illustrates a network implementation of system for object parts detection in industrial inspection, in accordance with an embodiment of the present subject matter;
[0010] FIG. 2 illustrates a block diagram of a system for object parts detection in industrial inspection, in accordance with an embodiment;
[0011] FIG. 3 illustrates a flow diagram of a method for object parts detection in industrial inspection, in accordance with an example embodiment;
[0012] FIG. 4 illustrates an example framework for object parts detection in case of a printer, in accordance with an example embodiment;
[0013] FIG. 5 illustrates an example of object parts detection in industrial inspections, in accordance with an example embodiment;
[0014] FIGS. 6A, 6B and 6C illustrate detection of object parts by object parts detection system, in accordance with an example embodiment; and
[0015] FIGS. 7A-7E illustrates a user interface (UI) for object parts detection system, in accordance with an example embodiment.
[0016] It should be appreciated by those skilled in the art that any block diagrams herein represent conceptual views of illustrative systems and devices embodying the principles of the present subject matter. Similarly, it will be appreciated that any flow charts, flow diagrams, and the like represent various processes which may be substantially represented in computer readable medium and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.
DETAILED DESCRIPTION
[0017] Systems and methods for object parts detection in industrial inspection are disclosed in present subject. In general, the inspection is carried out by employing complex checks for safety and quality in various fields such as oil refineries, petrochemical complexes, chemical plants, cement plants, cross-country pipelines, in shipping industries, for wind mill towers and fabrication shops, and so on. Such a wide variety of inspection scenarios may impose a number of requirements. For example, from an inspector’s perspective the inspection procedure can be made convenient by provisioning of a precise check-list for object parts under inspection. In most of the scenarios, the object inspection is a cumbersome procedure involving inspection of multiple object parts. During inspection, object part specification should be available when as and when required. Moreover, priority of the tests/checks to be performed for object parts should be known. This is due to the fact that during inspection, a large number of check-lists pertaining to a specific model of the machine and variant of the product would have to be addressed. In such scenarios some checks may need to be prioritised. In the present disclosure, unlike conventional systems where prioritisation is hard-coded, the disclosed system enables object-detection driven checklist priorities. In addition, accurate recording of observed inconsistencies should be possible. Various requirements from the perspective of a company may include: inspection tasks should be completely quickly and accurately and it should be possible to change priorities of checks due to external factors. Also, all inspection records should be available easily post inspection.
[0018] Various embodiments disclosed herein provide methods and systems for object parts detection in industrial inspection, where the checklists are prioritized. The priorities of the check-lists are dynamic and driven by object detection paradigm. The object parts are detected and tracked by using robust fast region based convolution network (R-CNN) model, thereby obviating the need for markers.
[0019] In addition, the disclosed methods and systems provides for accurate recording of status of inspection through evidence capturing of images, audio, notes, and videos. In addition, the device type is detected automatically and its corresponding device specific layout is rendered. This is done by fetching hardware build model type and parsing information.
[0020] The embodiments herein and the various features and advantageous details thereof are explained more fully with reference to the non-limiting embodiments that are illustrated in the accompanying drawings and detailed in the following description. The examples used herein are intended merely to facilitate an understanding of ways in which the embodiments herein may be practiced and to further enable those of skill in the art to practice the embodiments herein. Accordingly, the examples should not be construed as limiting the scope of the embodiments herein.
[0021] The methods and systems are not limited to the specific embodiments described herein. In addition, the method and system can be practiced independently and separately from other modules and methods described herein. Each device element/module and method can be used in combination with other elements/modules and other methods.
[0022] The manner, in which the system and method for object parts detection in industrial inspection shall be implemented, has been explained in details with respect to the FIGS. 1 through 7C. While aspects of described methods and systems for summarizing multi-sensor data can be implemented in any number of different systems, utility environments, and/or configurations, the embodiments are described in the context of the following exemplary system(s).
[0023] Referring now to FIG. 1, a network implementation 100 of system 102 for object parts detection in industrial inspection is illustrated, in accordance with an embodiment of the present subject matter. In one embodiment, the system 102 provides a marker less method for detection of object parts in industrial inspection. Herein, it will be noted that the system utilizes markers such as Quick Response(QR) codes as the only marker to trigger pre-inspection step. The remaining steps of inspection in proposed system are marker-less. Object part detection without markers using deep learning principles and supervised learning to train the machine to learn the object appearance obviates the need of deploying markers for industrial inspection.
[0024] Although the present subject matter is explained considering that the system 102 is implemented for object parts detection in industrial inspection, it may be understood that the system 102 may is not restricted to any particular machine or environment. The system 102 can be utilized for a variety of domains where industrial assembly and maintenance operations are involved. The system 102 may be implemented in a variety of computing systems, such as a laptop computer, a desktop computer, a notebook, a workstation, a mainframe computer, a server, a network server, and the like.
[0025] Herein, the system 102 may receive the media information, for example, videos captured from multiple devices and/or machines 104-1, 104-2…104-N, collectively referred to as devices 104 hereinafter. Examples of the devices 104 may include, but are not limited to, industrial machines, a portable computer, a personal digital assistant, a handheld device, a workstation, VR camera embodying devices, storage devices equipped to capture and store the videos, and so on. In an embodiment, the devices 104 may include devices capable of capturing the videos, for example, augmented reality (AR) based videos of industrial objects and/or machines. The devices 104 are communicatively coupled to the system 102 through a network 106, and may be capable of transmitting the captured videos to the system 102.
[0026] In one implementation, the network 106 may be a wireless network, a wired network or a combination thereof. The network 106 can be implemented as one of the different types of networks, such as intranet, local area network (LAN), wide area network (WAN), the internet, and the like. The network 106 may either be a dedicated network or a shared network. The shared network represents an association of the different types of networks that use a variety of protocols, for example, Hypertext Transfer Protocol (HTTP), Transmission Control Protocol/Internet Protocol (TCP/IP), Wireless Application Protocol (WAP), and the like, to communicate with one another. Further the network 106 may include a variety of network devices, including routers, bridges, servers, computing devices, storage devices, and the like.
[0027] The devices 104 may send the videos to the system 102 via the network 106. The system 102 is caused to analyze the videos captured by the AR devices such as the devices 104. Herein, the AR devices are the devices that may embody AR technologies. AR technologies enhance user’s perception and help the user to see, hear, and feel the environments in enriched ways. With the use of AR to incorporate instruction and assistance manuals associated with the objects interactively and directly within the task domain, and directly referencing the equipment at which the user is looking, has the potential to eliminate the current need for personnel to continually switch their focus of attention between the inspection task and its separate documentation in Paper-based or Tablet-based instructions. During repair and maintenance tasks, the use of AR increases productivity, reduces personal risk when working in difficult environment and also helps in performing complex tasks faster with less errors.
[0028] AR system implementation consists of three steps, namely recognition, tracking, and augmentation. In recognition step, object is recognized in a 3D space and virtual content is superimposed on it. Tracking involves real-time localization of object and finally media such as audio, video, 3D, 2D and text are superimposed for better user perception. Tracking is an important step as it should continuously trigger and modify relevant media according to the user motion and FoV (field-of-view). Tracking in AR primarily classified into Marker-based tracking and Sensor Fusion Based Tracking. Marker-based tracking uses physical-world symbols as a reference for virtual objects to be overlaid on them. The main downfall of the existing marker based implementations for triggering AR content is that markers have to be always present in the camera FoV and cannot be obscured by other objects during the augmentation. Another tracking method used is Sensor Fusion Based Tracking where a combination of sensor data from GPS, compass, accelerometer to calculate actual position and orientation are used. To overcome these problems in overlaying AR content, the disclosed system utilizes deep learning based object detection and tracking.
[0029] In an embodiment, the system 102 may be embodied in a computing device 110. Examples of the computing device 110 may include, but are not limited to, a desktop personal computer (PC), a notebook, a laptop, a portable computer, a smart phone, a tablet, and the like. The system 102 may also be associated with a data repository 112 to store a plurality of object identifiers corresponding to a plurality of objects for which the inspection can be performed, various check-lists associated with the plurality of object parts, inspection history of object parts and so on. In an embodiment, the data repository may be stored configured outside and communicably coupled to the computing device 110 embodying the system 102. Alternatively, the data repository 112 may be configured within the system 102. An example implementation of the system 102 for object parts detection in industrial inspection is described further with reference to FIG. 2.
[0030] FIG. 2 illustrates a block diagram of a system 200 for object parts detection in industrial inspection, in accordance with an embodiment of the present disclosure. In an example embodiment, the system 200 may be embodied in, or is in direct communication with a computing device, for example the computing device 110 (FIG. 1). The system 200 includes or is otherwise in communication with one or more hardware processors such as a processor 202, one or more memories such as a memory 204, and a network interface unit such as a network interface unit 206. In an embodiment, the processor 202, memory 204, and the network interface unit 206 may be coupled by a system bus such as a system bus 208 or a similar mechanism.
[0031] The processor 202 may include circuitry implementing, among others, audio and logic functions associated with the communication. For example, the processor 202 may include, but are not limited to, one or more digital signal processors (DSPs), one or more microprocessor, one or more special-purpose computer chips, one or more field-programmable gate arrays (FPGAs), one or more application-specific integrated circuits (ASICs), one or more computer(s), various analog to digital converters, digital to analog converters, and/or other support circuits. The processor 202 thus may also include the functionality to encode messages and/or data or information. The processor 202 may include, among other things, a clock, an arithmetic logic unit (ALU) and logic gates configured to support operation of the processor 202. Further, the processor 202 may include functionality to execute one or more software programs, which may be stored in the memory 204 or otherwise accessible to the processor 202.
[0032] The one or more memories such as a memory 204, may store any number of pieces of information, and data, used by the system to implement the functions of the system. The memory 204 may include for example, volatile memory and/or non-volatile memory. Examples of volatile memory may include, but are not limited to volatile random access memory (RAM). The non-volatile memory may additionally or alternatively comprise an electrically erasable programmable read only memory (EEPROM), flash memory, hard drive, or the like. Some examples of the volatile memory includes, but are not limited to, random access memory, dynamic random access memory, static random access memory, and the like. Some example of the non-volatile memory includes, but are not limited to, hard disks, magnetic tapes, optical disks, programmable read only memory, erasable programmable read only memory, electrically erasable programmable read only memory, flash memory, and the like. The memory 204 may be configured to store information, data, applications, instructions or the like for enabling the system 200 to carry out various functions in accordance with various example embodiments. Additionally or alternatively, the memory 204 may be configured to store instructions which when executed by the processor 202 causes the system to behave in a manner as described in various embodiments.
[0033] In an embodiment, the memory 204 includes a data repository having a plurality of object identifiers corresponding to a plurality of objects for which the inspection can be performed. It will be noted that the data repository may be embodied in a computing device (for example, the computing device 110 of FIG. 1) embodying the system 200. Alternatively, the data repository may be configured outside and communicatively coupled to the computing device.
[0034] The network interface unit 206 is configured to facilitate communication between the first and the second computing devices. The network interface unit 206 may be in form of a wireless connection or a wired connection. Examples of wireless network interface element 206 may include, but are not limited to, IEEE 802.11 (Wifi), BLUETOOTH®, or a wide-area wireless connection. Example of wired network interface element 206 includes, but is not limited to Ethernet.
[0035] The system 200 may be caused to detect a marker attached to the device/object/machine to be inspected and extract a device identifier encrypted within the marker. In an example embodiment, the marker may include a QR code. In an example embodiment, a Zxing’s library of QR codes may be utilized for detecting product to be inspected by recognizing QR code attached to it. The QR code facilities in retrieving a device identifier, for instance a serial number of the device, as will be explained below. The device identifier helps in obtaining the relevant device information and different check-lists that need to be executed during the inspection of that device. Also, the authenticity of the product is confirmed by matching the serial number with the entries in the data repository. Said step of detecting the marker attached to the device/object/machine can also be configured for inspector authentication to make sure that the inspection is being conducted by a relevant/authorized person.
[0036] Upon detecting the marker, the system 200 sends a query to match the detected device identifier with the plurality of device identifiers stored in the data repository, and in response, extract relevant device information therefrom. The relevant information may include device information, for instance, while inspecting a printer, information such as make, type, year of manufacture and inspection history may be included as the relevant information, and may be retrieved along with corresponding check-lists to be performed during inspection. The system 200 may be caused to facilitate display of safety instructions and manuals associated with the object part on the user device screen. The system 200 may subsequently send request to initiate the inspection process.
[0037] It will be noted that the three initial steps, namely, capturing the device identifier, transmitting the device identifier to the computing device, and retrieving the relevant device information along with corresponding check-lists may be construed as pre-processing steps, and said steps make sure that all the relevant contents for the device are extracted prior to commencement of actual inspection of object parts.
[0038] To commence the inspection, the system 200 is caused to receive, via the network interface unit 206, a captured video of the object. Herein, the video may be captured using an AR device. In an embodiment, the AR device’s camera captures the scene, and the feed is sent to the remote server using User Data Program (UDP) communication link. UDP is a connectionless and datagram oriented protocol that facilitates real-time communication, where a small percentage of packet-loss rate is permissible to the overhead of a Transmission Control Protocol (TCP) connection. Fast data transmission can be achieved using this connection making object localization and virtual object overlay in the FoV possible at interactive rates.
[0039] In an example embodiment, the system may be caused to detect a blur in the captured videos. For instance, in case the AR device’s camera is very close to the object to be inspected, the images/videos rendered by the camera may be blurred. In an embodiment, a non-referential blur detection based on cumulative probability of blur detection may be utilized to quantify the level of
blur in an image. Upon detection of image’s blur to be beyond a predefined threshold, the system may send a message to the user device to capture the video from a distance which is beyond a threshold distance, then resume inspection.
[0040] The system 200 is caused to detect a plurality of object parts that may appear in a field of view (FOV) of the AR device as captured in the video. As is understood, there may be multiple object parts that may appear in the FOV of the AR device. The system 200, instead of initiating inspection for the multiple objects directly, may prioritize some of the object parts over remaining object parts for conducting the inspection.
[0041] In an embodiment, the system 200 assigns priority scores to a set of object parts from the plurality of object parts appearing in the FOV of the AR device in the video based on a deep learning model. In an embodiment, the deep learning model may include an R-CNN detector. For tracking of parts in subsequent frames, the system utilizes tracking by a detection approach, wherein an R-CNN based part detector is run on continuous frames which identifies position of different object parts appearing at all scales. It will be noted that in the present scenario, occlusion may not be a challenge as the system is expected to display information for only those objects which are visible in the view.
[0042] In an embodiment, a confidence score of the RCNN based object detector may be determined by generating a plurality of object part proposals for a plurality of segments captured from a plurality of frames of the video. Herein, the plurality of object part proposals includes those object parts that may be dominantly occurring in the plurality of frames of the video. In other words, the object part proposals are the regions or segments of various sizes in the captured frames/images that may have at least one dominant object part in the segment. In an embodiment, label of the segment may be defined based on the dominant object part. The system 200 may be caused to extract one or more features of each of the plurality of object part proposals using a deep convolutional neural network. In an embodiment, the deep convolutional neural network is trained to extract fixed-length feature representation of each of the candidate object proposals. In an embodiment, a classification model may be utilized for recognizing the object proposals and assigning confidence scores to the detected object class/type. The confidence score may be represented as ScoreConf.
[0043] The system 200 prioritizes the inspection of the set of object parts based on the priority scores of the set of object parts. In an embodiment, the system 200 may be caused to obtain priority scores of the object parts based on a location of the said object parts in the FOV and the confidence score of faster R-CNN detector. An example of computing the priority scores is described below.
[0044] In an embodiment, the priority scores are determined based on a combined probability estimation of:
1) Confidence score of detected object class/type (represented as “ScoreConf”), which is given by Faster R-CNN (as explained above), and
2) The distance of object parts from center of user's FOV. A normalized distance of the object parts from image center can be computed in Euclidean space.
[0045] The priority score can be computed as:
Priority_Score = 1/2 [ ScoreConf + || Centroid of object distance - Image View center||]
As is evident from the equation above, more the confidence score and closer the object from image center, the higher is the priority score of the object for inspection.
[0046] Based on the priority scores, the system 200 is caused to retrieve a set of inspection check-lists associated with the inspection of the set of object parts from the data repository associated with the computing device for enabling the inspection of the set of object parts. The set of inspection check-lists may include a question list and a corresponding answer list corresponding to each of the set of inspection check-lists. Additionally or alternatively, the system 200 may be caused to retrieve an inspection history of the object from the data repository based on the object identifier. In an embodiment, the questions list is arranged in a hierarchical order. In an embodiment, a mode of inspection where tasks are to be completed quickly and accurately, the questions pertaining to that object part are displayed sequentially till the list goes empty. This helps in overriding the usual full mode inspection route for quick and essential checks that are required when performing a complex inspection task under. An example of presenting is checklist for the object part to be inspection is described further with reference to FIGS. 7A-7C.
[0047] In an embodiment, the system 200 enables user interaction with the system 200 during the inspection. For instance, a speech recognition module recognizes the user commands and helps in choosing options similar to Interactive Voice Recognition (IVR), for authoring the comments, and recording the visual and audio evidences of defects during the inspection. The speech recognition module may continuously listen for user voice input through the device microphone and immediately returns the text output. This text output is then compared with the predefined keywords to execute the relevant task.
[0048] Upon completion of all the checks, inspection data pertaining to the object is transferred to a remote server, for instance using HTTP protocol which ensures the security of the data. As HTTP utilizes reliable TCP connection, user data is not dropped and the accurate transmission of the inspector data is guaranteed. An example scenario of object parts detection in industrial inspection using the system 200 is described further with reference to FIGS. 4-7C.
[0049] FIG. 3 illustrates a flow diagram of a method 300 for object parts detection in industrial inspection, in accordance with an example embodiment. The method 300 may be described in the general context of computer executable instructions. Generally, computer executable instructions can include routines, programs, objects, components, data structures, procedures, modules, functions, etc., that perform particular functions or implement particular abstract data types. The method 300 may also be practiced in a distributed computing environment where functions are performed by remote processing devices that are linked through a communication network. The order in which the method 300 is described is not intended to be construed as a limitation, and any number of the described method blocks can be combined in any order to implement the method 300, or an alternative method. Furthermore, the method 300 can be implemented in any suitable hardware, software, firmware, or combination thereof. In an embodiment, the method 300 depicted in the flow chart may be executed by a system, for example, the system 200 of FIG. 2. In an example embodiment, the system 200 may be embodied in a computing device, for example, the computing device 104 (FIG. 1).
[0050] At 302, the method 300 includes receiving a video of an object captured using an AR device. The object may be associated with an object identifier and may include a plurality of object parts. For instance, if the object is a printer, then the printer may include a plurality of parts such as a touch screen panel, knobs, power socket, and so on. In an embodiment, the video is received at the system from the AR device via a UDP communication link.
[0051] At 304, the method 300 includes assigning priority scores to a set of object parts from the plurality of object parts appearing in a Field of View (FOV) of the AR device in the video based on a deep learning model. In an embodiment, the priority scores are determined based on a combined probability estimation of:
1) Confidence score of detected object class/type (represented as “ScoreConf”), which is given by Faster R-CNN, and
2) The distance of object parts from center of user's FOV. A normalized distance of the object parts from image center can be computed in Euclidean space.
[0052] The priority score can be computed as:
Priority_Score = 1/2 [ ScoreConf + || Centroid of object distance - Image View center||]
[0053] At 306, the method 300 includes prioritizing the inspection of the set of object parts based on the priority scores of the set of object parts. Based on the priority scores, a set of inspection check-lists associated with the inspection of the set of object parts may be retrieved from the data repository associated with the computing device for enabling the inspection of the set of object parts. The set of inspection check-lists may include a question list and a corresponding answer list corresponding to each of the set of inspection check-lists at 308. Additionally or alternatively, an inspection history of the object may be retrieved based on the object identifier from the data repository. In an embodiment, the questions list is arranged in a hierarchical order.
[0054] In an embodiment, the method provides for various modes of user interaction with the system during the inspection. For instance, a speech recognition module recognizes the user commands and helps in selecting options for authoring the comments, and recording the visual and audio evidences of defects during the inspection. Upon completion of all the checks, inspection data pertaining to the object is transferred to a remote server, for instance using HTTP protocol which ensures the security of the data. An example scenario illustrating application of the disclosed method for object parts detection in case of a printer is described in further detail with reference to FIGS. 4-7C.
[0055] FIG. 4 illustrates an example framework for object parts detection in case of a printer, in accordance with an example embodiment. Initially, at Step 1, the system detects the marker, for example, the QR code attached to the printer device to be inspected and extracts a Serial Number encrypted within the code.
[0056] At Step 2, the system queries to match the extracted serial number with the data stored in the a data repository in a back-end server, and based on the matching, extracts the relevant device information such as make, type, year of manufacture and inspection history of the printer along with corresponding check-lists to be performed during inspection. At Step 3, safety instructions and object part manuals are displayed on the user device screen. Upon completion of the steps 1-3, the system may request to start the inspection process. Said three initial steps make sure that all the relevant contents for the device are extracted and the user is aware of the complete inspection process.
[0057] The inspection is performed by a combination of graphic and augmented interfaces, object detection on a remote server, a speech recognition module, and user gestures at steps 4, 5 and 6. At step 4, the AR device’s camera captures a scene of the printer via a video, and the video feed is sent to the remote server using UDP communication link. At the server, different objects in the stream are tracked which guides the display of part/object specific information including the check-lists. An example of tracking the object parts in the video stream is described further with reference to FIG. 6A-6C. A speech recognition module recognizes the user commands and helps in selecting the options for authoring the comments, and recording the visual and audio evidences during the inspection. Once the user completes all checks provided in the check-list, the inspection data pertaining to the device is transferred to the remote server using HTTP protocol which ensures the security of the data. FIG. 5 illustrates an example of data handling in the
[0058] FIG. 5 illustrates an example of object parts detection in industrial inspections, in accordance with an example embodiment. For instance in case of the printer application, as described with reference to FIG. 4, the data handling includes: (i) sending live video stream to the server for object detection and tracking, for example at 502 (ii) fetching the relevant check-lists and object information, for example at 504, and (iii) storing the evidence of defects in the form of audio or images on the server. Object specific information files and check-lists are stored on the server and loaded into an application post QR code scanning. Herein, the disclosed object parts detection system may include hardware and software that may be collectively configured to host an IT application for performing functions for detection of object parts of machines and various industrial equipment.
[0059] Referring to the reconfigurable back-end 500 with the help of printer use-case, and as illustrated in FIG. 5, the files for inspection data includes printer’s metadata file 508, questions file 510, and answers file 512. The printer’s metadata file 502 includes details/meta data of objects to be inspected is contained in this file, where a particular object is represented by unique QR/ID number. First column is QR Number/ID number of the object. Second column gives object’s information such as Make, Type and Year for Printer Inspection use case. Last column contains checklist numbers to be followed during the inspection for that object.
[0060] As each check-list is associated with its own questions file, each question in the question file 504 has a unique identifier, and all the nodes information in the hierarchical structure as shown in the figure. In the answer file 506, the first column is QID. Second and third column contains Response and Evidence. Where response can be only YES/No, and evidence is used for multimedia record containing inspection information such as snapshot, video and audio of the scene. All the three types of files are stored on the remote server and all the relevant check-lists are loaded into application once the QR code is scanned.
[0061] The questions are grouped in a hierarchical structure, where each level of hierarchy represents the abstraction of the groups. The abstraction levels can be configured for different applications where the leaf nodes are the questions and the abstraction layer above the leaf nodes define the first level grouping of questions. Numbers of abstraction levels are decided by the steps in inspection process. The system may retrieve a relevant check-list (Ck) (for example, at 508) to follow for that particular part inspection. It refers to the Question file of Ck check-list. In an embodiment, the system may enable a user to manually traverse the abstractions in both directions (for example, at 514) and address a particular question. The exploration on questions is achieved at a user interface at 516. Against each question, responses may be recorded in the answer file 506.
[0062] FIGS. 6A, 6B and 6C illustrate detection of object parts by object parts detection system, in accordance with an example embodiment. The present example is explained using a printer as the object for the ease of description, however, it will be noted that the disclose system is configured for object parts detection for any machine or industrial application, without limiting to the disclosed examples.
[0063] Herein, FIGS. 6A, 6B and 6C illustrate sample detection of printer parts through a bounding box overlay. For example, in FIGS. 6A and 6B illustrates object parts such as knobs 610, 620, respectively, on the printer 600, while FIG. 6C illustrate printer back side 630.
[0064] FIGS. 7A-7E illustrates a user interface (UI) 700 for object parts detection system, in accordance with an example embodiment. The UI may initially present a window 702 showing the object to the inspected. In the present example, the window 702 illustrates a printer in FIG. 7A. For initiating the scanning, the subsequent window, for example, a window 704 is presented, as illustrated in FIG. 7B. The window 704 facilitates in initiating marker based detection of object. Window 706 in FIG. 7C is presented on the UI 700 to show the marker, for example, the QR code. Based on the QR code, the device information of the printer can be retrieved. For example as illustrated in FIG. 7D, the device identifier/serial number, make, type, are year of manufacture are retrieved and shown in the window 708. In addition, the window 708 provides an option to initiate inspection using button 710. FIG. 7E illustrates check-list retrieved for corresponding object part to be inspected, and the options to record results of inspection.
[0065] Various embodiments provide method and system for object parts detection in industrial applications. The system represents an AR based re-configurable inspection framework that can be utilized in cross-domain applications and with multiple devices. The disclosed system facilitates in check-list prioritization by detecting the parts viewed in inspector’s field with the aid of the state-of-the-art recurrent neural networks that works close to human performance and at real-time. Additionally the system provides a reconfigurable back-end for different application domains.
[0066] The written description describes the subject matter herein to enable any person skilled in the art to make and use the embodiments. The scope of the subject matter embodiments is defined by the claims and may include other modifications that occur to those skilled in the art. Such other modifications are intended to be within the scope of the claims if they have similar elements that do not differ from the literal language of the claims or if they include equivalent elements with insubstantial differences from the literal language of the claims.
[0067] It is, however to be understood that the scope of the protection is extended to such a program and in addition to a computer-readable means having a message therein; such computer-readable storage means contain program-code means for implementation of one or more steps of the method, when the program runs on a server or mobile device or any suitable programmable device. The hardware device can be any kind of device which can be programmed including e.g. any kind of computer like a server or a personal computer, or the like, or any combination thereof. The device may also include means which could be e.g. hardware means like e.g. an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination of hardware and software means, e.g. an ASIC and an FPGA, or at least one microprocessor and at least one memory with software modules located therein. Thus, the means can include both hardware means and software means. The method embodiments described herein could be implemented in hardware and software. The device may also include software means. Alternatively, the embodiments may be implemented on different hardware devices, e.g. using a plurality of CPUs.
[0068] The embodiments herein can comprise hardware and software elements. The embodiments that are implemented in software include but are not limited to, firmware, resident software, microcode, etc. The functions performed by various modules described herein may be implemented in other modules or combinations of other modules. For the purposes of this description, a computer-usable or computer readable medium can be any apparatus that can comprise, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
[0069] The foregoing description of the specific implementations and embodiments will so fully reveal the general nature of the implementations and embodiments herein that others can, by applying current knowledge, readily modify and/or adapt for various applications such specific embodiments without departing from the generic concept, and, therefore, such adaptations and modifications should and are intended to be comprehended within the meaning and range of equivalents of the disclosed embodiments. It is to be understood that the phraseology or terminology employed herein is for the purpose of description and not of limitation. Therefore, while the embodiments herein have been described in terms of preferred embodiments, those skilled in the art will recognize that the embodiments herein can be practiced with modification within the spirit and scope of the embodiments as described herein.
[0070] The preceding description has been presented with reference to various embodiments. Persons having ordinary skill in the art and technology to which this application pertains will appreciate that alterations and changes in the described structures and methods of operation can be practiced without meaningfully departing from the principle, spirit and scope.