DESC:F O RM 2
THE PATENTSACT, 1970(39of 1970)
&
THE PATENTS RULES, 2003COMPLETESPECIFICATION
[Seesection10 andrule13]

1. TITLE OF THE INVENTION: SMART INSPECTION SYSTEM

2. APPLICANT (A) NAME: RADOME TECHNOLOGIES AND SERVICES PVT. LTD

(B) ADDRESS: No. 207, 1st FLOOR, “JAYANIVAS”,
PADUKA MANDIRA ROAD, NEAR
BRIGADE 7 GARDENS, BHCS,
BENGALURU KARNATAKA-560061

3. NATIONALITY (C) INDIA

THE FOLLOWING SPECIFICATION PARTICULARLY DESCRIBES THE NATURE OF THIS INVENTION AND THE MANNER IN WHICH IT IS TO BE PERFORMED
PRIORITY CLAIM
[001] The instant patent application is related to and claims priority from the co-pending India provisional patent application entitled, “SMART INSPECTION SYSTEM”, Patent Application no.:202241062785, Filed on: 3 November 2022, which is incorporated in its entirety herewith.
BACKGROUND OF THE DISCLOSED EMBODIMENT
[002] Technical field
[003] The disclosed embodiment is in the technical field of a process of performing quality inspection of radial tyre. More particularly, it relates to an inspection system that uses a combination of cameras and laser sensors which results in increased throughput and ensures maximum quality.
[004] Related art
[005] Manufacturing and supply chain processes are becoming increasingly sophisticated through the adoption of advanced, high-speed automation systems.
[006] Given the high throughput of objects through these systems, frequent changeover of parts, as well as increased manufacturing and logistics demands, quality assurance activities can be difficult to implement.
[007] It is important to make the processes of procurement, setup and monitoring as easy as possible in order to drive up adoption of automated sensor and camera-based quality inspection systems.
[008] Techniques such as minimizing hardware configurations, moving solutions from hardware to software domains, and providing insights and explain ability around algorithm performance are examples of ways in which the process of implementing automated sensor and camera-based quality assurance systems can be made simpler.
[009] The known quality control methods typically comprise two steps: the step of checking the appearance of the work piece by looking for visual defects like cracks, blow, peel-off, chip-off etc.., inspecting the surface quality for both the external surface of the work piece; and the step of checking the dimensions of the work piece, either by manually measuring the dimensions, such as with a calliper, depth gauge, etc., or by using a template or the like.
[010] The known quality control steps are usually carried out on the first work pieces produced by a new manufacturing run. Indeed, the initial parts may be subjected to a complete dimensional check, which may include cutting the part open to inspect any interior dimensions. Once the inspection has been satisfactorily completed, it is performed only on randomly selected manufactured work pieces.
[011] To date, such quality control checks have been carried out entirely manually. The manual operations, which may include spot marking to emphasize the presence of the defect and location of the defect on the tyre for further analysis and decision making. These defects are sometimes reworkable, reparable if not will be scrapped. The process of decision making is always time consuming and laborious.
[012] Such time-consuming operations cause production delays, thereby increasing the manufacturing costs of the work piece. Typically, such quality control checks on a rough, as cast work piece may take several weeks.
[013] Human factors in quality inspection of manufactured items are susceptible to errors and minute defects have high chances of being overlooked due to factory lighting conditions and fatigue factors. The manual inspection process is usually unreliable, expensive, time consuming, have a low detection rate and are difficult to scale.
[014] Therefore, there is a critical need to make an inspection system that uses artificial intelligence to solve these challenges of inefficiency. An inspection system that can use combination of cameras and sensors resulting in increased throughput while ensuring maximum quality.
SUMMARYOFTHEDISCLOSED EMBODIMENT
[015] According to an exemplary aspect of the disclosed embodiment, an inspection system is provided comprising a mechanical system to pick a tyre and to perform a set of rotations of the tyre to a set of positions and an electronic system comprising a set of cameras and a set of sensors, wherein the set of cameras capture a set of images of the tyre at respective positions in the set of positions and the set of sensors capture a set of profiles of the tyre at respective positions in the set of positions. The inspection system also contains a software system to perform analysis of the set of images and the set of profiles to determine whether there are defects or anomalies in the tyre.
[016] According to another aspect of the disclosed embodiment, wherein at a first position in the set of positions, the electronic system captures an image of a barcode printed on the tyre andthe software system retrieves information on the tyre from an MIS (management information system) connected to the software system.
[017] According to further aspect of the disclosed embodiment, the software system matches, using a template matching technique, the set of profiles to one or more reference templates, each reference template representing a defect-free tyre surface. If a match is found, the software system indicates to a user that the tyre had no anomalies and is defect free.
[018] According to yet another aspect of the disclosed embodiment, each profile of the set of profiles comprises a 3D (3 dimensional) point cloud data, wherein the template matching technique is an Isolation Forest technique.
[019] According to one more aspect of the disclosed embodiment, if no match is found, the software system detects defects using a classification technique that compares the set of images to previously maintained images of defects in a defect library. If a defect is detected, the software system indicates to the user that the tyre has the defect and one or more images contained in the subset of images based on which the defect is detected.
[020] According to another aspect of the disclosed embodiment, the classification technique is a combination of YOLO (You Only Look Once) v8 and Slicing Aided Hyper Inference (SAHI) techniques.
[021] According to further aspect of the disclosed embodiment, if no match is found, the software system performs anomaly detection by applying an anomaly detection technique on the set of images of the tyre. If an anomaly is determined to be present, the software system indicates to the user that the tyre has the anomaly and one or more images contained in the subset of images based on which the anomaly is determined.
[022] According to yet another aspect of the disclosed embodiment, the anomaly detection technique is an Isolation Forest technique.
[023] According to an aspect of the disclosed embodiment, a method of performing inspection of a tyre comprises capturing a set of images of the tyre at respective positions in a set of positions of the tyre and a set of profiles of the tyre at respective positions in the set of positions and matching, using a template matching technique, the set of profiles to one or more reference templates, each reference template representing a defect-free tyre surface. If amatch is found, the method comprises indicating to a user that the tyre had no anomalies and is defect free.
[024] According to another aspect of the disclosed embodiment, if no match is found, the method comprisesdetecting defects using a classification technique that compares the set of images to previously maintained images of defects in a defect library and performing anomaly detection by applying an anomaly detection technique on the set of images of the tyre. If a defect is detected or an anomaly is determined to be present, the method comprises indicating to the user that the tyre has the defect or the anomaly and one or more images contained in the subset of images based on which the defect or the anomaly is determined.
[025] Several aspects of the invention are described below with reference to examples for illustration. However, one skilled in the relevant art will recognize that the invention can be practiced without one or more of the specific details or with other methods, components, materials and so forth. In other instances, well-known structures, materials, or operations are not shown in detail to avoid obscuring the features of the invention. Furthermore, the features/aspects described can be practiced in various combinations, though only some of the combinationsaredescribedhereinforconciseness.
BRIEFDESCRIPTIONOFTHEDRAWINGS
[026] Exampleembodimentsofthedisclosed embodiment willbedescribedwith reference tothe accompanyingdrawingsbrieflydescribedbelow
[027] FIG.1 illustratesthe Input–Output Flow Correlation, according to the aspects of the disclosed embodiment.
[028] FIG.s2A-2Cillustrate a mechanical system assembly, according to the aspects of the disclosed embodiment.
[029] FIG. 3 illustrates example results of a computer vision-based defect detection, according to the aspects of the disclosed embodiment.
[030] FIG. 4 illustrates a process flow of the smart inspection system, according to the aspects of the disclosed embodiment.
[031] FIG. 5 is a block diagram illustrating the details of a digital processing system in which various aspects of the disclosed embodiment are operative by execution of appropriate execution modules.
[032] Inthedrawings,likereferencenumbersgenerallyindicateidentical,functionallysimilar,and/orstructurallysimilarelements.Thedrawinginwhichanelementfirstappearsis indicatedbytheleftmostdigit(s)inthecorrespondingreferencenumber.
DETAILEDDESCRIPTIONOFTHEDISCLOSED EMBODIMENT
[033] It is to be understood that the present disclosure is not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the drawings. The present disclosure is capable of other embodiments and of being practiced or of being carried out in various ways. Also, it is to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting.
[034] Reference throughout this specification to “one embodiment”, “an embodiment”, or similar language means that a particular feature, structure, or characteristic described in disclosed embodiment. Thus, appearances of the phrases “in one embodiment”, “in an embodiment” and similar language throughout this specification may, but do not necessarily, all refer to the sameembodiment.
[035] Theuseof“including”,“comprising”or“having”andvariationsthereofhereinis meantto encompass the items listed thereafter and equivalents thereof as well asadditional items. The terms “a” and “an” herein do not denote a limitation of quantity, butrather denote the presence of at least one of the referenced items. Further, the use of terms“first”,“second”,and“third”,andthelike,hereindonotdenoteanyorder,quantity,or importance, butratherareusedtodistinguishoneelementfromanother.
[036] As used herein, the singular forms “a”, “an”, and “the” include both singularand plural referents unless the context clearly dictates otherwise. By way of example, “adosage” referstooneormorethanonedosage.Theterms“comprising”,“comprises”and“comprisedof”asusedhereinare synonymouswith“including”,“includes”or“containing”,“contains”,andareinclusiveoropen-endedanddonotexcludeadditional,non-recitedmembers, elements or method steps.
[037] All documents cited in the present specification are hereby incorporated by reference in their totality. In particular, the teachings of all documents herein specifically referred to are incorporated by reference.
[038] Example embodiments of the present disclosure are described with reference to the accompanying figures.
[039] DEFINITIONS
[040] The term ‘Inspection’ means an instance of viewing, examining, or critically analyzing something with the purpose of improving it or highlighting something.
[041] The term ‘Mechanical system’ meansa system that manages the power of forces and movements to accomplish a task.
[042] The term ‘Electronic system’ means a system working depending on electronic variables like power or voltage or current and formed of a number of blocks having different input–output relationships connected together.
[043] The term ‘Software system’ means a system of intercommunicating components based on software forming part of a computer system.
[044] The term ‘YOLOv8’ means the newest state-of-the-art YOLO (You Only Look Once) model that can be used for object detection, image classification, and instance segmentation tasks.
[045] 1. SMART INSPECTION SYSTEM
[046] The primary objective of the present embodiments is to use a combination of cameras and sensors resulting in increased throughput while ensuring maximum quality.
[047] Smart inspection system is a combination of a mechanical system, an electrical/electronic system, and a software system. The mechanical system consists of tyre holding system, tilting & rotating assembly, base and column assembly, main frame assembly, electrical and control panel assembly, slide assembly, camera holding assembly and tyre clamping assembly. According to an aspect, the mechanical system is designed to pick a tyre and to perform a set of rotations of the tyre to a set of positions.
[048] The electronic system consists of 3D triangulation sensor, line scan cameras, illumination system and accessories. According to an aspect, the electronic system comprises a set of cameras and a set of sensors, the set of cameras capturing a set of images of the tyre at respective positions in the set of positions and the set of sensors capturing a set of profiles of the tyre at respective positions in the set of positions. In one embodiment, each profile of the set of profiles comprises a 3D (3 dimensional) point cloud data described in detail below.
[049] The software system includes backend, database, and front end with tier 2 architecture. The backend includes algorithm and script to interact with the sensor and camera systems. The solution is developed with python-based Django front end framework for user interface. This includes a MongoDB database. The application is a web-based application which provides user-access at various locations to get the insights of the system. According to an aspect, the software system is designed to perform analysis of the set of images and the set of profiles to determine whether there are defects or anomalies in the tyre.
[050] FIG. 1 illustrates the Input–Output flow correlation, according to the aspects of the disclosed embodiment. The smart inspection system is shown containing 4 parts -inputs102, mechanical automation108, software and AI (Artificial Intelligence) system 116 and output 132. Each of the partsof the Figure is described in detail below.
[051] Inputs 102 is categorized into two types. The first is categorized as tyre data 104 which includes tyre geometrical input parameters like section width, aspect ratio, section height, and inner diameter. The second is categorized as barcode data 106 which includes manufacturing data of a radial tyre that is received through barcode reading, well known in the arts.According to an aspect, the electronic system (noted above) captures an image of a barcode printed on the tyre and software and AI system 116 retrieves information on the tyre from an MIS (management information system) connected to the software system.
[052] Mechanical automation108 includes tyre holding assembly 110, actuation system for vision system processing 114and the control system 112 with PLCs and drives. Mechanical automation 108is primarily used to handle the various sizes of the tyres for inspection. The correct setting of the machine is automatically done using input 102received in the first part.
[053] Software and AI System116performs a quality check of a tyre loaded in tyre holding assembly 110 based on cameras and sensors. Whenever the tyre is on boarded onto the system,a complete cycle of datacapturing using vision systems with cameras & 3D triangulation (laser)sensors 130 is performed. Once the capturing is completed, all the captured data is processed, which comprises of sensor operation programing, machine interaction/communication for sequencing programing, computer vision algorithm, template matching algorithm, database122, backend programing and decision making 124, a web application for dashboard,weight file from trained deep learning model, mathematical model 118, high resolution images of the tyre outer and inner surfaces 126, inference of tyre quality and defect 120, 3D point cloud coordination data of the tyre outer surface contour 128.PLC layer 112 integrates the interrelation between the software decision making and the hardware actuation.
[054] Output 132 is the output of the smart inspection system. The data captured is processed in the software and AI system 116and the result of tyre defect detecting and classification measurement and segregation 134 of defect is communicated back to the mechanical system which decides to perform the acceptance or reject action. A front-end user interface and analytics dashboard 136 is also provided.
[055] Thus, a smart inspect systems for radial tyre is provided according to aspects of the disclosed embodiment. The system uses sensor fusion to combine the 3D point cloud data generated by the laser sensors and the high-definition images captured by a line scan camera.The data captured is processed by AI modules including built in mathematical models which uses computer vision technology to detect, recognize and classify the defects into the trained classes. The module uses historic data and pertained models to accurately identify the defects that are present on the manufactured product.The models extract unique features and attributes of the defects to distinguish clearly between defects and establishes a confidence level of identification of the defects.
[056] As such, smart inspection system offers several advantages such as increased throughput and production quantity, reduces the inspection time with repeatability and reliability, reduces human intervention and human factors contributing to the errors in inspection. The description is continued with the manner in which a mechanical system in the smart inspection system may be implemented.
[057] 2. EXAMPLE MECHANICAL SYSTEM
[058] The mechanical system is designed in such a way that it accommodates various sizes of tyres ranging from 12 inches to 18 inches and its categories.The system is designed to accommodate tyre and sensors like laser sensors, and line scan cameras. The mechanical system has 7 drives to pick up and hold a tyre, rotate the tyre at a specific RPM, positioning the laser and the camera based on the tyre size.Based on the decision given by the computer vision algorithms regarding acceptance and rejection of the tyre, the system can place the tyre in the respective location.The mechanical system has a positional accuracy of 20 micrometers though the tyre is a rubber product and could be flexible with force. The machine is designed to achieve a reliability of 2400 cycles or inspections/day.
[059] FIG.s 2A-2C illustrate a mechanical system assembly, according to the aspects of the disclosed embodiment. Referring to FIG. 2A, the mechanical system assembly is designed to accommodate different sizes of the tyres and inspect for defects in less than 30 seconds.The capability of accommodating any size is built based on automation techniques. As opposed to any other inspection machines, the size of the tyre feeding to the mechanical system assembly of the present disclosure is not the same in a continuous process.
[060] The tyres can be fed randomly with the size ranging from 12 inch to 18 inch. Such a flexibility is not there in any automated inspection systems available in the market. According to an aspect, flaps are designed to receive the tyres from the conveyor, storage column. The flaps center the tyres automatically and holds it with the optimal pressure for rotation without wobbling. Once the tyre is in position, an automatic controller rotates the tyre and a barcode scanner recognizes the tyre size and sends it to a PLC (112 noted above).
[061] Based on the tyre dimension, the PLC calculates the optimal distance from the tyres, the vision system needs to be placed for inspection based on one or more empirical formulas and communicates the spatial coordinates information with the positioning actuators powered by pneumatic and electric controllers.
[062] Referring to FIG. 2B, the Figure illustrates the system dimensions when the dimensions are in the X direction 206 showing the tyre center 204,according to the aspects of the disclosed embodiment. Referring to FIG. 2C, the Figure illustrates the system dimensions when the dimensions are in the Y direction 206 showing the roller center 208,according to the aspects of the disclosed embodiment.
[063] The optimal distancepositions the vision system in an optimal location for the images to be captured with high definition and the captured images are sent to the computer vision algorithms. Such automation removes the dependency on a human intervention to set the machine to accommodate a different tyre size incoming in the sequence.The closest systems to the disclosed embodiment are the inspection systems developed by Pirelli R&D for their specific tyre inspection within their plants.
[064] The description is continued with the manner in which an electrical/ electronic system in the smart inspection system may be implemented.
[065] 3. EXAMPLE ELECTRONIC SYSTEM
[066] The electronic system in smart inspection system consists of a combination of 3D triangulation sensors, 2k line scan cameras, illumination systems, GigE enabled communications and power accessories.The triangulation sensors have a uniqueness of digitally recreating the point cloud of the 3D contour of the tyres’ internal and external surfaces, as described in sections below. The 2k line scan cameras capture the details of the tyres with high definition to capture the information of the dimension as small as 6 microns. The illumination system is selected and arranged in such a way that the lighting condition are at an appropriate distance and angle for the cameras to capture the image of a black tyre without blind spots and matching the characteristics of the defects.
[067] The description is continued with the manner in which a software and AI system in the smart inspection system may be implemented.
[068] 4. EXAMPLE SOFTWARE SYSTEM
[069] The software system consists of a template matching algorithm as well as computer vision and analytics algorithms for intelligence layer which uses YOLO V8 with Slicing Aided Hyper Inference (SAHI) for image detection and recognition.
[070] Template Matching algorithm
[071] Template matching with laser 3D point cloud data is a powerful technique in computer vision and 3D modelling. Matching involves the comparison of a predefined template (a set of 3D points or shapes) with the acquired 3D point cloud data from laser sensors. By aligning the template with the point cloud data, the system can detect and recognize objects or features in the environment. The laser 3D point cloud data offers high precision and accuracy, making such data ideal for template matching tasks. Template matching allows for robust and real-time identification of objects even in complex and cluttered scenes. Template matching with laser data plays a crucial role in enhancing the autonomy and decision-making capabilities of various autonomous systems. Some of the steps that may be implemented as part oftemplate matching in a tyre inspection system is described in detail below.
[072] Pre-processing:Before implementing template matching for tyre inspection, the 3D laser data is pre-processed. Such pre-processing may involve tasks such as rearranging the 3D point cloud data and ensuring that the 3D point cloud data starts from the same point. These enhancements are necessary to improve the accuracy of the matching process.
[073] Template Selection: Select a reference template image that represents a "good" or defect-free tyre surface. The selected template is used for comparison with the actual tyre images to detect discrepancies.
[074] Matching Algorithm: In the matching algorithm, the Isolation Forest technique (well known in the arts) is chosen to compare the template image with the target tyre image. Isolation Forest calculates a similarity score for each possible location of the template within the tyre image, making it a suitable choice for your specific application. The Isolation Forest algorithm is designed to identify anomalies or outliers in the data, and using it for template matching may help improve the accuracy of the tyre inspection process.
[075] The smart/tyre inspection system using template matching on 3D laser data takes as inputs:
[076] 3D Laser Data: These are the 3D point cloud representations of the tyres that need to be inspected;
[077] Reference Template: A 3D template representing a defect-free tyre surface; and
[078] Pre-processing Parameters: Parameters for pre-processing the 3D laser data rearranging the laser input data
[079] As noted above, the software and AI system in the smart inspection system uses YOLO (You Only Look Once) v8 and Slicing Aided Hyper Inference (SAHI) algorithms for computer vision analysis. A brief introduction to the usage of these algorithms in the smart inspection system is provided below.
[080] YOLO v8 Algorithm
[081] YOLOv8 is the latest version of the acclaimed real-time object detection and image segmentation model. YOLOv8 is built on cutting-edge advancements in deep learning and computer vision, offering unparalleled performance in terms of speed and accuracy. The streamlined design of YOLO v8 makes the algorithm suitable for various applications and easily adaptable to different hardware platforms, from edge devices to cloud APIs.
[082] YOLOv8 utilizes a convolutional neural network that can be divided into two main parts: the backbone and the head.
[083] Backbone: A modified version of the CSPDarknet53 architecture forms the backbone of YOLOv8. This architecture consists of 53 convolutional layers and employs cross-stage partial connections to improve information flow between the different layers.
[084] Head: The head of YOLOv8 consists of multiple convolutional layers followed by a series of fully connected layers. These layers are responsible for predicting bounding boxes, objectness scores, and class probabilities for the objects detected in an image.
[085] SAHI Algorithm
[086] A framework called Slicing Aided Hyper Inference (SAHI) is used that provides a generic slicing aided inference and fine-tuning pipeline for small object detection. The proposed technique is generic in the sense that the technique can be applied on top of any available object detector without any fine-tuning. Detection of small objects and inference on large images are still major issues in practical usage.
[087] Sliced Inference: Sliced inference refers to the practice of subdividing a large or high-resolution image into smaller segments (slices), conducting object detection on these slices, and then recompiling the slices to reconstruct the object locations on the original image.
[088] The SAHI algorithm has the benefits of reduced computational burden, preservingdetection quality and enhanced scalability. The smart inspection system incorporates the SAHI algorithm in combination with the YOLOv8 model to develop a custom model. SAHI is utilized to slice the images to detect small objects in a large image of the tyre to be inspected to improve feature extraction and enhance accuracy. YOLOv8, a powerful object detection model, serves as the backbone for the object detection capabilities of the system, ensuring real-time and efficient detection and localization of objects. The combination allowed to leverage the strengths of both SAHI and YOLOv8 to achieve high accuracy in object detection tasks in real-time performance.
[089] Thus, the fusion of YOLOv8 and the SAHI algorithm presents an innovative and highly effective approach to object detection and image segmentation. The adaptability, accuracy, and efficiency of the fusion makes the resultant fusion a versatile tool with the potential to significantly impact a wide range of applications, from automated quality control in manufacturing to intelligent surveillance systems. The success of this integration underscores the ongoing potential of deep learning and computer vision in solving complex real-world problems.
[090] One challenge with tyre inspection is the identification of defects on a black surface. The first step in the disclosed embodiment is, after the image is captured, to annotate the defects for the system to learn their characteristics. The feature engineering used in computer vision models of the system is the distinctive physical, visual features of the defects in a way that they are generalized for the characteristics pertaining to the defects such as cracks, blows, inclusion of foreign material etc.,
[091] The defects are annotated with a method which not only includes the defect but also the necessary surrounding information related to the defect. Such annotation is necessary to train the model not to get confused with similar features that otherwise can be confused as a defect. Also, such annotation techniques take care of the position of the defect accurately. For example, a crack on the sidewall is differentiated by a crack on the tread or bead although the characteristic of the crack remains the same. An example anomaly detection algorithm used to identify defects is described below.
[092] Anomaly Detection Algorithm
[093] Anomaly detection is used to identify data points or objects that deviate significantly from the norm or exhibit unusual behavior. The detection is used for finding rare events or outliers within a dataset. Anomaly detection techniques involve statistical methods, machine learning algorithms, or domain-specific rules to identify outliers or anomalies. The system implements isolation forest for detecting the anomalies. The Isolation Forest is a powerful and efficient anomaly detection technique that excels in identifying anomalies within large and complex datasets.
[094] The 3D triangulation sensor data from the unit under test is pre-processed to match the starting and ending position of the tyre scanned. The pre-processed data is then passed into the template matching algorithm for anomaly detection. Then, the anomaly part of the data (if any) is extracted and fused using camera image data for final classification of the defects.
[095] According to an aspect, the software system matches, using a template matching technique (Isolation Forest technique noted above), the set of profiles (containing 3D point cloud data) to one or more reference templates, each reference template representing a defect-free tyre surface. If a match is found, the software system indicates to a user that the tyre had no anomalies and is defect free. If no match is found, the software system detects defects using a classification technique (a combination of YOLO v8 and SAHI techniques noted above) that compares the set of images to previously maintained images of defects in a defect library. If a defect is detected, the software system indicates to the user that the tyre has the defect and one or more images contained in the subset of images based on which the defect is detected. if no match is found, the software system also performs anomaly detection by applying an anomaly detection technique (Isolation Forest technique noted above) on the set of images of the tyre. If an anomaly is determined to be present, the software system indicates to the user that the tyre has the anomaly and one or more images contained in the subset of images based on which the anomaly is determined.
[096] The outputs of the smart/tyre inspection system using 3D laser data consists of:
[097] Defect Detection: Regions of the 3D point cloud where defects are detected, often identified in 3D space; and
[098] Defect Information: Details about the defects (Sidewall blow, Crack, Flash) found, including their 3D coordinates, size, and shape characteristics. Including visualizations of the detected defects in the 3D point cloud data stored in viewable format in our system.
[099] The software system of the smart/tyre inspection system includes a Python-Django based web application This includes a MongoDB database. The web application provides user access at various locations to get the insights of the system.
[0100] FIG. 3 illustrates example results of a computer vision-based defect detection, according to the aspects of the disclosed embodiment. The results of the Figure may be displayed to users on a display unit associated with client systems used by the users. The client system may represent a system such as a personal computer, workstation, mobile phone (e.g., iPhone available from Apple Corporation), tablet, portable device (also referred to as “smart” devices”) that operate with a generic operating system such as Android operating system available from Google Corporation, etc.,
[0101] Image 310depicts a tread crack being detected by the smart inspection system with a confidence of 100% on the tyre. Image 320depicts and IL crackbeing detected by the smart inspection system with a confidence of 74% on the tyre. Image 330 depicts cord visible being detected by thesmart inspectionsystem with a confidence of 99% on the tyre. Image 340 depicts cord visible being detected by the smart inspection system with a confidence of 100% on the tyre.
[0102] It may be appreciated that the integration of laser 3D data and template matching techniques in tyre inspection represents a groundbreaking leap in quality control and safety assurance for the automotive industry. Such an innovative approach capitalizes on the high precision and accuracy offered by laser 3D point cloud data, enabling robust and real-time object recognition, even in challenging environments.
[0103] The tyre inspection system outlined in the present disclosure combines meticulous preprocessing, template selection, and the application of the Isolation Forest method to achieve unparalleled accuracy in defect detection. By harnessing the power of 3D laser data, the present disclosure paves the way for a comprehensive assessment of tyre quality while manufacturing in the tyre industry. The manner in which smart inspection system operates to provide various aspects of the disclosed embodiment is described in detail below.
[0104] 5. GENERAL FLOW
[0105] FIG. 4 illustrates a process flow of the smart inspection system, according to the aspects of the disclosed embodiment. The flowchart is described with respect to the blocks of FIG. 1 merely for illustration. However, various features can be implemented in other systems and/or other environments also without departing from the scope of various aspects of the present invention, as will be apparent to one skilled in the relevant arts by reading the disclosure provided herein.
[0106] In addition, some of the steps may be performed in a different sequence than that depicted below, as suited in the specific environment, as will be apparent to one skilled in the relevant arts. Many of such implementations are contemplated to be covered by several aspects of the present invention.
[0107] In step 402, a tyre is picked from the bay.When the tyre inspection cycle is complete, the next step is to pick the next tyre from the bay where the tyres will be in pipeline for inspection.
[0108] In step 404, the tyre is positioned to the center of the machine.Once the tyre is sensed in the bay, the tyre is automatically picked by titling and holding in the tyre holding part.
[0109] In step 406, a first rotation is performed to read the barcode. In step 408, the system gets all necessary configuration data from the connected MIS on the shop floor to load the appropriate programme for the selected tyre within the inspection system.
[0110] In step 410, laser sensors and cameras are positioned to capture sidewall (1) and inner liner (1) area. The sensors and camera reach a first position and capture the required data within a scheduled time and transfer the data for analysis in step 414A.
[0111] In step 412, laser sensors and cameras are positioned to capture tread and inner liner (2) area. The sensors and camera reach a second position and capture the required data within a scheduled time and transfer the data for analysis in step 414B.
[0112] In step 416, laser sensors and cameras are positioned to capture sidewall (2) and inner liner (3) area. The sensors and camera reach a third position and capture the required data within a scheduled time and transfer the data for analysis in step 414C.
[0113] In step 418, image & 3D point cloud processing layer processes the 3D point cloud data obtained from the tyre under test using template matching algorithm in step420and the result of which is taken for further analysis. In step 430, if the template is a 100% match for the obtained 3D point cloud data,the system indicates “accept”in step 432to indicate that there are no anomaly and the tyre is defect free.
[0114] In step 426, if the result of template matching is less than 100%,the system takes up the area of anomaly and passes that section to defect detection & classificationalgorithm in step 422 or to anomaly detection algorithm in step 424. In step 428, if a new defect not found in a library of defects is identified, the system adds the new defect to the library. In step 434, the system indication “reject” of the tyre under test, and may also provide the details of the defects detected (such as the images shown in FIG. 3).
[0115] According to an aspect, the process flow noted above represents a method of performing inspection of a tyre comprising capturing a set of images of the tyre at respective positions in a set of positions of the tyre and a set of profiles of the tyre at respective positions in the set of positions and matching, using a template matching technique, the set of profiles to one or more reference templates, each reference template representing a defect-free tyre surface. If a match is found, the method comprises indicating to a user that the tyre had no anomalies and is defect free.
[0116] If no match is found, the method comprises detecting defects using a classification technique that compares the set of images to previously maintained images of defects in a defect library and performing anomaly detection by applying an anomaly detection technique on the set of images of the tyre. If a defect is detected or an anomaly is determined to be present, the method comprises indicating to the user that the tyre has the defect or the anomaly and one or more images contained in the subset of images based on which the defect or the anomaly is determined.
[0117] The uniqueness in such an approach is meeting high repeatability, scalability (as in the same model detects various sizes of defects in different sizes of tyres at different locations) and accuracy by patching the images with annotation and training using YOLO V8 with SAHIalgorithms for each defect.
[0118] After training the models, a unique approach which uses a slicing hyper inference is adopted for the best inference accuracy. Such an approach slices the test images into the defined number of slices and performs the inference on each slice which increases the accuracy of prediction. Such a method allows the detection of defects without having the need for the images to be having same characteristics of trained images.
[0119] The inference performs with a high accuracy even with the images flipped or warped which makes the system very robust. Incorporation of such an approach in the inspection system is a novelty of present disclosure and makes the accuracy more than other followed methods.This is claimed as the first successful attempt in identifying, classifying, and segregating the tyre defects using AI.
[0120] The integration of GigE camera for image capturing using Python and setting up the camera suitable to the appropriate size, form and factor is also one of the key elements. Similarly, integration of laser point cloud data and development of mathematical equations to correlate with the defect identified and measure the attributes associated with the defect is also a unique capability of the present disclosure. The analytical algorithms are built to identify the trend of the defect occurrences and its attributes.
[0121] The 3D triangulation sensors capture the contour information of the tyre outer surface in the form of a 3D point cloud. In the disclosed embodiment, inventors have integrated the image captured by the 2k line scan camera with the 3D point cloud data of the triangulation sensors which is another novelty in the present disclosure. The location of the defect identified from the camera is correlated to the location of the defect in the triangulation sensor data to exactly pinpoint the coordinates of the defects. The 3D coordinates of the defects are then isolated from the whole tyre and the dimension of the defect is measured upto an accuracy of 240 micrometers. The length, the width and the depth of the defect is measured using the triangulation sensor which will describe the defect comprehensively.
[0122] Such a technique is claimed as the first successful attempt in fusion of camera and 3D triangulation sensor for identifying the exact coordination of the defects and measurement by the present disclosure.All the above is integrated in a Python environment and the goal of the smart inspection system is to identify up to 60 defects within 30 seconds.The Django framework allows the application to be accessed on cloud by multiple users who can analyze the information from any machine from any plant at any given point in time.
[0123] Thus, aspects of the present disclosure provide for a smart inspection system that performs quality inspection of radial tyres using a combination of cameras & laser sensors and computer vision techniques.Currently used techniques follow basic image processing techniques for detection and recognition of the defects. However, present disclosure includes a combination of laser and line scan camera, data fusion and computer vision techniques to extract the features of the defect which makes the learning robust and allows the models to identify the defects even if they differ in form and factor. The algorithms built also includes measurement of attributes such as length, width, depth, and area covered by the defects which is not currently available in any existing systems.
[0124] Smart inspection system offers several advantages such as increased throughput and production quantity. The system reduces the inspection time with repeatability and reliability, reduces human intervention and human factors contributing to the errors in inspection.
[0125] The instant disclosure has multiple uses, applications, and benefits. Manual Inspection takes 50 seconds and smart inspection system can inspect at 30 seconds per tyre. The system can run 2400 cycles per day which is on an average 40% more than the current capacity followed manually. The system ensures saving of defective tyres getting into the market which can lead to accidents and fatalities. The system has increased throughput which indirectly gives more revenue and profits, allows retention of customers which helps in better branding, and is scalable across various production units for different types of tyres. The system provides reduction in cost of quality.
[0126] The disclosed embodiment is a part of a challenge provided by one of the leading tyre manufacturers in India to develop and deploy commercially. The disclosed embodiment is ready to be bought and use commercially.
[0127] It should be further appreciated that the above noted features can be implemented in various embodiments as a desired combination of one or more of hardware, execution modules and firmware. The description is continued with respect to one embodiment in which various features are operative when execution modules are executed.
[0128] 6. HARDWARE
[0129] FIG. 5 is a block diagram illustrating the details of digital processing system 500 in which various aspects of the disclosed embodiment are operative by execution of appropriate execution modules. Digital processing system 500 may correspond to a system implementing the software and AI system(116) in the smart inspection system described above.
[0130] Digital processing system 500 may contain one or more processors (such as a central processing unit (CPU) 501), random access memory (RAM) 502, secondary memory 503, graphics controller 506, display unit 507, network interface 508, and input interface 509. All the components except display unit 507 may communicate with each other over communication path 505 which may contain several buses as is well known in the relevant arts. The components of FIG. 7 are described below in further detail.
[0131] CPU 501 may execute instructions stored in RAM 502 to provide several features of the present invention. CPU 501 may contain multiple processing units, with each processing unit potentially being designed for a specific task. Alternatively, CPU 501 may contain only a single general-purpose processing unit. RAM 502 may receive instructions from secondary memory 503 using communication path 505.
[0132] Graphics controller 506 generates display signals (e.g., in RGB format) to display unit 507 based on data/instructions received from CPU 501. Display unit 507 contains a display screento display the images defined by the display signals (e.g., portions of the images shown in FIG. 3, etc.). Input interface 509 may correspond to a keyboard and a pointing device (e.g., touch-pad, mouse), which enable the various inputs to be provided.
[0133] Network interface 508 provides connectivity to a network (e.g., using Internet Protocol), and may be used to communicate with other connected systems. Network interface 508 may provide such connectivity over a wire (in the case of TCP/IP based communication) or wirelessly (in the case of Wi-Fi, Bluetooth based communication).
[0134] Secondary memory 503 may contain hard drive 503a, flash memory 503b, and removable storage drive 503c. Secondary memory 503 may store the data (e.g., 3Dpoint cloud data, inputs/outputs of the algorithms noted above, etc.) and software instructions (e.g., for implementing the steps of FIG. 4, the blocks of FIG. 1), which enable digital processing system 700 to provide several features in accordance with the present invention.
[0135] Some or all of the data and instructions may be provided on removable storage unit 504, and the data and instructions may be read and provided by removable storage drive 503c to CPU 501. Floppy drive, magnetic tape drive, CD-ROM drive, DVD Drive, Flash memory, and removable memory chip (PCMCIA Card, EPROM) are examples of such removable storage drive 503c.
[0136] Removable storage unit 504 may be implemented using storage format compatible with removable storage drive 503c such that removable storage drive 503c can read the data and instructions. Thus, removable storage unit 504 includes a computer readable storage medium having stored therein computer software (in the form of execution modules) and/or data.
[0137] However, the computer (or machine, in general) readable storage medium can be in other forms (e.g., non-removable, random access, etc.). These "computer program products" are means for providing execution modules to digital processing system 500. CPU 501 may retrieve the software instructions (forming the execution modules) and execute the instructions to provide various features of the disclosed embodiment described above.
[0138] Merely for illustration, only representative number/type of graph, chart, block, and sub-block diagrams were shown. Many environments often contain many more block and sub-block diagrams or systems and sub-systems, both in number and type, depending on the purpose for which the environment is designed.
[0139] While specific embodiments of the invention have been shown and described in detail to illustrate the inventive principles, it will be understood that the invention may be embodied otherwise without departing from such principles.
[0140] It should be understood that the figures and/or screen shots illustrated in the attachments highlighting the functionality and advantages of the disclosed embodiment are presented for example purposes only. The disclosed embodiment is sufficiently flexible and configurable, such that it may be utilized in ways other than that shown in the accompanying figures.
[0141] It should be understood that the examples and embodiments described herein are for illustrative purposes only and that various modifications or changes in light thereof will be suggested to persons skilled in the art and are to be included within the spirit and purview of this application and scope of the appended claims. All publications, patents, and patent applications cited herein are hereby incorporated by reference in their entiretyforallpurposes.
,CLAIMS:I/WE Claim:
1. A inspection system for tyres, comprising:
a mechanical system to pick a tyre and to perform a set of rotations of the tyre to a set of positions;
an electronic system comprising a set of cameras and a set of sensors, wherein the set of cameras capture a set of images of the tyre at respective positions in the set of positions and the set of sensors capture a set of profiles of the tyre at respective positions in the set of positions; and
a software system to perform analysis of the set of images and the set of profiles to determine whether there are defects or anomalies in the tyre.

2. The inspection system of claim 1, wherein at a first position in the set of positions:
the electronic system captures an image of a barcode printed on the tyre; and
the software system retrieves information on the tyre from an MIS (management information system) connected to the software system.

3. The inspection system of claim 1, wherein the software system is operable to:
match, using a template matching technique, the set of profiles to one or more reference templates, each reference template representing a defect-free tyre surface; and
if a match is found, indicate to a user that the tyre had no anomalies and is defect free.

4. The inspection system of claim 3, wherein each profile of the set of profiles comprises a 3D (3 dimensional) point cloud data, wherein the template matching technique is an Isolation Forest technique.

5. The inspection system of claim 3, if no match is found, the software system is operable to:
detect defectsusing a classification technique that compares the set of images to previously maintained images of defects in a defect library; and
if a defect is detected, indicate to the user that the tyre has the defect and one or more images contained in the subset of images based on which the defect is detected.

6. The inspection system of claim 6, wherein the classification technique is a combination of YOLO (You Only Look Once) v8 and Slicing Aided Hyper Inference (SAHI) techniques.

7. The inspection system of claim 3, if no match is found, the software system is operable to:
perform anomaly detection by applying an anomaly detection technique on the set of images of the tyre; and
if an anomaly is determined to be present, indicate to the user that the tyre has the anomaly and one or more images contained in the subset of images based on which the anomaly is determined.

8. The inspection system of claim 7, wherein the anomaly detection technique is an Isolation Forest technique.

9. A method of performing inspection of a tyre, the method comprising:
capturing a set of images of the tyre at respective positions in a set of positions of the tyre and a set of profiles of the tyre at respective positions in the set of positions;
matching, using a template matching technique, the set of profiles to one or more reference templates, each reference template representing a defect-free tyre surface; and
if a match is found, indicating to a user that the tyre had no anomalies and is defect free.

10. The method of claim 9, if no match is found, the method comprising:
detecting defects using a classification technique that compares the set of images to previously maintained images of defects in a defect library;
performing anomaly detection by applying an anomaly detection technique on the set of images of the tyre; and
if a defect is detected or an anomaly is determined to be present, indicating to the user that the tyre has the defect or the anomaly and one or more images contained in the subset of images based on which the defect or the anomaly is determined.

Dated this 3rd day of November, 2023
Signature.......................................................
(LIPIKA SAHOO)
Registration Number: IN/PA-2467
Agent for Applicant
This document is signed with the digital signature of Patent Agent for the Applicant LIPIKA SAHOO (IN/PA-2467)