Sign In to Follow Application
View All Documents & Correspondence

Blood Vessel Extraction In Two Dimensional Thermography

Abstract: BLOOD VESSEL EXTRACTION IN TWO-DIMENSIONAL THERMOGRAPHY A system and method for isolating blood vessels in a thermographic image of a patient"s breast or any other muscular region of the body. The thermographic image is received and a temperature-based analysis is performed to detect vessel pixels. An intensity-based method analysis is performed on the image. A shape-based analysis is also performed to detect pixels of vessel-like structures. Candidate pixels which satisfy one or more of intensity-based or temperature-based or shaped-based criterion are identified. A constraint of local maximallity is thereafter imposed on each candidate pixel that satisfies both criterion to eliminate spurious non-vessel pixels. The satisfied criterion is then marked with a different color such that the vessel structures in the breast tissue can be visually differentiated. The vessel structures are provided to a classifier system which classifies the tissue in the thermal image as malignant and non-malignant otherwise, based on a tortuosity of the vessel structures. FIG. 1

Get Free WhatsApp Updates!
Notices, Deadlines & Correspondence

Patent Information

Application #
Filing Date
18 December 2018
Publication Number
08/2019
Publication Type
INA
Invention Field
BIO-MEDICAL ENGINEERING
Status
Email
Parent Application
Patent Number
Legal Status
Grant Date
2021-04-28
Renewal Date

Applicants

NIRAMAI HEALTH ANALYTIX PVT. LTD.
Flat A7-506, Elita Promenade JP Nagar 7th Phase, Bangalore-560078, Karnataka

Inventors

1. TEJA KAKILETI, Siva
D. No: 1-45, Sundar Nagar Street, Annaipeta, Draksharamam R.C Puram Mandal East, Godavari Dist., Kakinada -533262, Andhra Pradesh

Specification

TECHNICAL FIELD [0001] The present invention is directed to systems and methods for isolating blood vessels in a thermographic image of a person. Background [0002] Breast cancer leads in incidence rates among all cancers in women, contributing to about one-fourth of the cancers detected. Such high incidence rates in the overall population, with nearly 1 in 8 women in the west and 1 in 11 women in India getting breast cancer sometime in their lifetime, requires intervention in terms of detection and treatment. Early detection is key to survival here as the breast cancer can be completely cured if detected in the early stages. Thermography is an emerging alternative non-radiation and non-contact screening method for breast cancer detection, whose sensitivity does not depend on the age of the woman. In the recent decade interest has been rekindled in thermography as a breast cancer screening approach with the improvement in thermal camera resolution and technology. Thermography can also be used for imaging other body parts and it accurately depicts the temperature distribution on the skin. [0003] Malignancy increases the number of blood vessels. The experimental evidence for this dependence of malignant tumor growth and angiogenesis is known. Microcirculation is altered in cancer due to presence of nitric oxide which is released from cancerous tissue. This nitric oxide increases blood circulation creates new vessels and also recruits dormant vessels. By contact temperature measurements, it was observed that the malignant tumor is at a higher temperature than surrounding tissue. It was also observed that the malignant tumor is hotter than the temperature of blood vessels associated with the tumor. The increase in metabolic activity of the cancerous cells generates heat. This heat can be detected by a thermal camera. Malignancy also triggers regional changes to vessel shape and increases the vessel dimensions in terms of width and length compared to the normal vessels. The significant difference between the widths of normal and malignant blood vessels is due to increased blood supply. It is shown that tortuosity (twists and turns of vessel growth) is exhibited in early stages of cancer and that tumor vessels have a profound sort of tortuosity, with many smaller bends upon each larger bend. This increase in vessel caliber, vessel length, and its resultant tortuosity is also an effect of the increase in blood flow to the cancerous cells. It is important to be able to accurately isolate (extract) vessel structures in a thermographic image of breast tissue so that the tortuosity can be analyzed. The present invention is specifically directed to this effort. Brief Summary [0004] The present invention is focused on breast cancer screening, and hence in extracting features from thermal images that will distinguish one or more malignant sub¬sets from one or more subsets of the nonmalignant category. Some of these features will be having an association to the visual observations made by trained thermographers, but will not be exactly the same. The algorithm(s) to extract such features will be different in representation to the visual features and provide the advantage of quantifying the observations from thermal images. Other features are that difficult to interpret visually are additionally present, and have been designed due to their medical relevance to differentiate the malignant cases from some or more of the non-malignant cases. These features are divided into vascular and non-vascular features, and are described herein. The present invention extracts the blood vessels in thermal images even in presence of hotspots and non-uniform heat in images by considering intensity and shape based descriptors. One embodiment of the present method is used for isolating blood vessels in a thermographic image of a patient's breast involves the following. A thermographic image of a breast of a patient is received. A temperature-based analysis is performed on the image to detect vessel pixels. A shape-based analysis is also performed to detect pixels of vessel-like structures. Candidate pixels which satisfy both the temperature-based and shaped-based criterion are identified. A constraint of local maximallity is thereafter imposed on each candidate pixel that satisfies both criterion to eliminate spurious non-vessel pixels. Candidate pixels which satisfy both criterion are then marked with a different color such that the vessel structures in the breast tissue can be visually differentiated. The vessel structures are provided to a classifier system which classifies the tissue in the thermal image as malignant and non-malignant otherwise, based on a tortuosity of the vessel structures. [0005] Features and advantages of the above-described method will become readily apparent from the following detailed description and accompanying drawings. Brief Description of the Drawings [0006] The foregoing and other features and advantages of the subject matter disclosed herein will be made apparent from the following detailed description taken in conjunction with the accompanying drawings, in which: [0007] FIG. 1 shows an example female patient with a thermal camera mounted on a slideable and axially rotatable robotic arm for moving the camera along a semi-circular trajectory from side-to-side in front of the patient; [0008] FIG. 2 shows a thermographic image of a breast of a patient; [0009] FIG. 3 shows the image of FIG. 2 which has been processed in accordance with the teachings hereof wherein the blood vessels have been identified; [0010] FIG. 4 is a flow diagram which illustrates one embodiment of the present method for isolating blood vessels in a thermographic image of a patient's breast; and [0011] FIG. 5 which shows a functional block diagram of one example image processing system for processing thermographic images for breast cancer screening in accordance with the embodiment described with respect to the flow diagram of FIG. 4. Detailed Description [0012] What is disclosed is a system and method for isolating blood vessels in a thermographic image of a patient. NON-LIMITING DEFINITIONS [0013] A "patient' refers to either a male or a female person. Gender pronouns are not to be viewed as limiting the scope of the appended claims strictly to females. Moreover, although the terms "subject', "person" or "patient' are used interchangeably throughout this disclosure, it should be appreciated that the patient undergoing cancer screening may be something other than a human such as, for example, a primate. Therefore, the use of such terms is not to be viewed as limiting the scope of the appended claims to humans. [0014] A "thermal camera" refers to either a still camera or a video camera with a lens that focuses infrared energy from objects in a scene onto an array of specialized sensors which convert infrared energy into electrical signals on a per-pixel basis and outputs a thermal image comprising an array of pixels with color values corresponding to temperatures of the objects in the image across a desired thermal wavelength band. FIG. 1 shows a thermal camera 101 mounted on a slideable and axially rotatable robotic arm 102 capable of moving the camera along a semi-circular trajectory 103 in the front of the patient from side-to-side such that thermographic images can be captured in a right-side view 104, a front view 105, and a left-side view 106, and various oblique angles in between. The thermal camera can be any of: a single-band infrared camera, a multi-band infrared camera in the thermal range, and a hyper spectral infrared camera in the thermal range. The resolution for a thermal camera is effectively the size of the pixel. Smaller pixels mean that more pixels will go into the thermal image giving the resulting image higher resolution and thus better spatial definition. Although thermal cameras offer a relatively large dynamic range of temperature settings, it is preferable that the camera's temperature range be relatively small, centered around the person's body surface temperature so that small temperature variations are amplified in terms of pixel color changes to provide a better measure of temperature variation. Thermal cameras are readily available in various streams of commerce. In one embodiment, the thermal camera is placed in wired or wireless communication with a workstation which enables manual or automatic control of various aspects of the thermal camera such as, for instance, adjusting a focus of the thermal camera lens, changing a resolution of the thermal camera, and changing a zoom level of the thermal camera. [0015] A "thermographic image" or simply "thermal image" comprises a plurality of pixels with each pixel having an associated corresponding temperature value. Pixels in the thermal image with a higher temperature value being displayed in a first color and pixels with a lower temperature value are displayed in a second color. Pixels with temperature values between the lower and higher temperature values are displayed in gradations of color between the first and second colors. Thermal images can be retrieved from a memory or storage device of the thermal imaging device, or obtained from a remote device over a network. Thermal images may be retrieved from a media such as a CDROM or DVD. Thermal images may be downloaded from a web-based system which makes such images available for processing. Thermal images can also be retrieved using an application such as those which are widely available for handheld cellular devices and processed on the user's cellphone or other handheld computing device such as an iPad or tablet. Use of the term "image" is intended to also mean "video". This thermal image can also be stored and retrieved purely as a two-dimensional matrix of real numbered values (also known as radiometric image) which are derived as a function of the measured temperature values that are represented by the color of each pixel in the thermal image. [0016] "Receiving a thermal image" of a patient for cancer screening is intended to be widely construed and includes retrieving, capturing, acquiring, or otherwise obtaining video image frames. The image can be received or retrieved from a remote device over a network, or from a media such as a CDROM or DVD. The image may be downloaded from a web-based system or application which makes video available for processing in accordance with the methods disclosed herein. The image can also be received from an application such as those which are available for handheld cellular devices and processed on the cellphone or other handheld computing device such as an iPad or Tablet-PC. The image can be received directly from a memory or storage device of the imaging device used to capture that image or video. The thermal image of the contra-lateral breast is analyzed to determine whether a hot spot exists in that breast. [0017] A "classifier system" or simply "classifier" comprises at least a processor and a memory with the processor retrieving machine readable program instructions from memory and executing those instructions causing the processor to classify tissue in a thermal image of the breast based on the determined vesselness measure. In another embodiment, the tissue in the thermal image of the breast is classified based on the tortuosity of the vessel structures identified therein. Classifiers can take any of a variety of forms including a Support Vector Machine (SVM), a neural network, a Bayesian network, a Logistic Regression, Naive Bayes, Randomized Forests, Decision Trees and Boosted Decision Trees, K-nearest neighbor, and a Restricted Boltzmann Machine (RBM), as are understood in the machine learning arts, including a hybrid system comprising any combination hereof. For an in-depth discussion, the reader is directed to any of a wide variety of texts on classifiers, including: "Foundations of Machine Learning", MIT Press (2012), ISBN-13: 978-0262018258, and "Design and Analysis of Learning Classifier Systems: A Probabilistic Approach", Springer (2008), ISBN-13: 978-3540798651. The classifier is training using a training set which, in various embodiments, comprises patient medical records and historical data. Based on the training set, the classifier sets a threshold value. Once trained, the classifier then utilizes the threshold for classification. The threshold can be user adjusted or user manipulated as needed to minimize false positives and/or false negatives. As new data sets or additional parameters are added to the training set used to train the classifier, the threshold or decision boundary used by the classifier will likely change accordingly. [0018] It should be appreciated that the steps of "receiving", "analyzing", "communicating", "performing", "determining", "selecting", "providing", "identifying", "removing", and the like, as used herein, include the application of any of a variety of techniques as well as mathematical operations according to any specific context or for any specific purpose. Such steps may be facilitated or otherwise effectuated by a microprocessor executing machine readable program instructions such that the intended functionality is effectively performed. Flow Diagram of One Embodiment [0019] Reference is now being made to the flow diagram of FIG. 4 which illustrates one embodiment of the present method for isolating blood vessels in a thermographic image of a patient's breast. Flow processing begins at step 400 and immediately proceeds to step 402. [0020] At step 402, receive a thermographic image of a breast of a patient. The thermographic image can be a single image of both breasts or an image of either a left or right breast. FIG. 2 shows a thermographic image of a breast of a patient which is received for processing. [0021] At step 404, perform vessel detection on the received image comprising a temperature-based method to detect vessel pixels and a shape-based method to detect pixels of vessel-like structures. [0022] At step 406, identify candidate pixels associated with vessel structures in the breast tissue in the thermal image that satisfy both the temperature-based and shaped-based methods. [0023] At step 408, remove non-vessel pixels (from the pool of candidate pixels) by imposing a strict constraint of local maximallity on each candidate pixel that satisfies both criterion. FIG. 3 shows the image of FIG. 2 which has been processed in accordance with the teachings hereof wherein the blood vessels have been identified. In this embodiment further processing stops. In another embodiment, the remaining candidate pixels that satisfy both criterion are marked with a different color so that these pixels can be visually differentiated in the thermal image of the breast tissue. Thereafter, in this embodiment, the marked vessel structures are provided to a classifier system which classifies tissue in the breast as being malignant, and non-malignant otherwise, based on a tortuosity of the marked vessel structures. [0024] It should be understood that the flow diagrams depicted herein are illustrative. One or more of the operations illustrated in the flow diagrams may be performed in a differing order. Other operations may be added, modified, enhanced, or consolidated. Variations thereof are intended to fall within the scope of the appended claims. All or portions of the flow diagrams may be implemented partially or fully in hardware in conjunction with machine readable/executable program instructions. 5.1 Automatic extraction of non-vascular thermal features [0025] We want to differentiate the temperature increases due to malignancy from the rest of the non-malignancy conditions, as we are interested in cancer screening. For this, we first want to extract the region of increased temperature automatically. [0026] Each feature is designed to differentiate malignancy from one or more of the non-malignancy cases. The temperature increase for malignancy is typically higher than for other non-malignancy cases. Hence, two different settings of this automatic extraction of the high temperature region is used, one tuned for malignancy and another tuned for non-malignancy cases. To distinguish malignancy from hormonal response, we design a feature to detect if there a similar increased temperature region in the contralateral breast in the corresponding region, as hormonal responses is expected to be present in both breasts. Malignancy in one breast causes the temperature rise in that breast to be significantly higher. To differentiate between benign conditions and malignancy, we use a feature to check if the boundary of the increased temperature region is regular or irregular. The boundaries of the malignant tumors are generally irregular, while the benign tumors are more regular. These features may be composed of multiple criteria, which are described in detail as follows. [0027] These features are agnostic to the imaging protocols and the camera resolution. We have used 3 different types of cameras, each of different resolutions and dynamic range. We have used two imaging protocols: one with a video with the person rotating from one side to the other side so that all relevant views are observed, and another where 3 different angles, frontal/oblique/lateral views of each breast are taken. Features can be extracted from the video using the best view where the highest temperature region is most clearly seen (based on its area) in one breast, and the contralateral view of the other breast. [0028] From the images, the best view corresponds to one of the 3 views in which the high temperature region is most prominently present. • Whether a tumor (likely malignant) is present: Tumor is defined to be a patch in the region satisfying the following conditions: (1) The temperature of the region is above a threshold, given by the mean of the mode of the temperature histogram and the highest temperature present. (2) The temperature of the region must be greater than or equal to (overall maximum temperature-2). (3) The temperature of the region is less than or equal to the overall maximum temperature. (4) The size of the region is greater than B pixels depending on camera resolution (say 64 pixels). • Number of hot patches (likely benign) in best view and its contralateral view: Hot patch (likely benign) is defined to be a patch in the region satisfying the following conditions: (1)The temperature of the region is above a threshold, given by the mean of the medians of the temperatures in 3 views (frontal, lateral and oblique). (2) The temperature of the region must be greater than or equal to (overall maximum temperature-2). (3) The size of the region is greater than 64 pixels. • The size of the tumor/hot patch detected with respect to the Region of Interest in the best view. (1) The size of the tumor or the hot patch divided by the size of corresponding Region of Interest. (2) The size of the hot patch is considered in its contralateral view also, as it serves as a measure of symmetry for hormonal cases. • (D) Similarity of hot patch(es) in contralateral breast to distinguish hormonal response from malignancy. (1) The extent of overlap of the tumor/hot patch present on one view and the corresponding contralateral view (if present): (i) Convolve the larger of the two increased temperature regions present in both breasts to determine the area of overlap and the percentage of overlap, (ii) In case of hot patches, lower temperature patches are also considered on the contra lateral side, to get a measure of the symmetry. These lower temperature hot patches are considered to be the hot patches with a lower threshold: (overall maximum temperature - 3). (2) The difference in area of the tumor/hot patch (if present, else it is 0) in the frontal/oblique/lateral views. • (E) The difference in temperature of the region of the tumor/hot patch detected and the surrounding region. (1) This is calculated by taking the mean temperature of the ROI without the tumor/hot patch region detected in the best view, and the mean temperature of the tumor/hot patch region and their difference in considered. (2) The difference of temperature is considered in the contralateral view also in case of hot patch as it accounts to the symmetry measure. • Irregularity of the tumor/hot patch shape with respect to a circle or ellipse. (1) Irregularity measure with respect to a circle is calculated using the formula below. where (xj,yj)are the points on the boundary, and x and y are the mean of xt and yt, respectively, and N(R) is the number of points within the region R. (i) Irregularity measure is taken in the best view of the hot patch detected and also in its contralateral view. (ii) In case of tumor, this measure is considered only in the best view. 2) Deviation of the tumor shape from a best fitted ellipse. This is calculated by considering how much it deviates from the best fitted ellipse, by taking the dot product of the coefficient vector and the point coordinates. An ellipse can be defined as the set of points X = (x,y) such that F(a,X) = f{a, (x,y)) = D.a = 0, where D = (x2, xy.y2, x,y,l) and a = (axx, axy, ayy, ax, ay, %) and Aaxxayy - axy > 0 for an ellipse. This is equivalent to aTCa > 0, where C is a 6X6 matrix with values, C13 = C31 = 2, C2,2 = -1. and a" other Ctj = 0. We can fit the ellipse to N data points, by minimizing the distanceA(a.x) = E/"(a>*i)2 = ^(aTZ)fZ)ja) = aTSa, where5 =1Z(DfDi). So, find a such that A(a,x) is minimum and aTCa = 6 for some positive^. To solve the above constrained problem, introduce a LaGrange multiplier X and a LaGrangian L{a) = A(a,x)-l(a'rCa- 6) and minimizeL(a), ^H^ = o, Sa = XCa,-a = S~1Ca. Solving the above eigen value problem with Eigen da A value 1/1, we get the Eigen vector a, which gives us the equation of the ellipse. The measure by which the tumor boundary deviates from this ellipse is given by the dot product of D and a. 3) Deviation of the tumor shape from a circle with center as centroid of all the points. This is calculated by taking the standard deviation of the distances of the points of the tumor boundary from the centroid of the tumor. The distances are calculated as dj = (XJ -x)2 + (yt -y)2, where x and y are the centroid of xt and yt. The standard deviation of these distances is calculated as.Std deviation a = — 7 — sqrt((Z(di - d) )/n), where dis the mean of the distances dt and n is the number of points in the boundary. 5.2 Automatic extraction of vascular features [0029] Vascular Features play an important role in the classification of malignancy. The importance of these features can also be seen from their significant role in classification with other modalities like MRI, Mammogram while classification/grading. In fact, thermographers consider these features during thermo-biological grading. In thermography, normal blood vessels are detected only when they show significant temperature difference than surrounding tissues. In case of malignancy, heat radiated from the vessels increases due to large amount of blood flow, which can be captured in the radiometric image/ thermo-gram. To design and analyze the features from blood vessels, we first need to extract the blood vessels from the Region of Interest (ROI). Section5.2.1 gives details of the blood vessel extraction in thermography. After extraction, the vascular features are mentioned in Section 5.2.2. 5.2.1 Automatic extraction of blood vessels [0030] Existing 2D vessel detection algorithms, when applied on thermo-graphic images, fail in extracting the vessels properly. Instead they might even pick up diffusions of heat and edges of tumor as vessels. The present algorithm extracts the blood vessels correctly even in presence of hotspots and non-uniform heat in images by considering intensity and shape based descriptors. The present algorithm uses matched filter response of a vertically shifted Gaussian as an enhancement technique followed by a vesselness criterion defined by us to extract the vessel like structures based on their shape from the Eigen values of Hessian Matrix. We use morphological extraction of vessels to detect the pixels based on their intensity relative to the surroundings. The pixels that are detected in both cases are considered as vessel pixels. This kind of approach might pick edges of tumor/hotspot as vessel pixels in some cases of thermo-graphic images due to irregular diffusion of heat in breast region. To avoid that, we used right and left Gaussians for calculating matched filter response separately to extract the valid blood vessel pixels which would be discussed in the algorithm. Matched Filter: [0031] Vessels are darker than the background and can be approximated to an inverted Gaussian curve along x - axisshifted by its absolute mean with vessel direction along y - axis. This kind of matched filter response approach for detecting blood vessels is proposed in references [1] "Detection Of Blood Vessels In Retinal Images Using Two-Dimensional Matched Filters", Chaudhuri, S.; Chatterjee, S.; Katz, N.; Nelson, M.; Goldbaum, M., IEEE Transactions on Medical Imaging, Vol.8, No.3, pp.263-269 (1989), [2] "An Efficient Algorithm For Extraction Of Anatomical Structures In Retinal Images", Thitiporn, C. and Fan, G.L., Proc. Of Intl. Conf. on Image Processing, vol. 1, pp. 1093-1096 (2003), and [3] "Retinal Vessel Extraction By Matched Filter With First-Order Derivative Of Gaussian", Zhang B, et al., Computers In Biology And Medicine, 40:438-445, (2010). This matched filter can be mathematically written as: where A represents the amplitude, ^corresponds to half the width of the vessel, and £"(.) represents expected/mean value. It is sufficient to assume |A2|, and Vs = 1 for tubular structures, and Vs = 0 or Vs = -1, otherwise. 5. The method as claimed in claim 1, wherein the system comprising marking vessel pixels which satisfy both criterion in the image with a different color such that the vessel structures in the breast tissue can be visually differentiated. 6. The method as claimed in claim 5, wherein the system comprising providing the marked vessel structures to a classifier system which proceeds to classify the breast tissue as malignant, and non-malignant otherwise, based on a tortuosity of the vessel structures. 7. The method as claimed in claim 6, wherein the classifier system comprises any of: Support Vector Machine, a neural network, a Bayesian network, a Logistic regression, Naive Bayes, Randomized Forests, Decision Trees and Boosted Decision Trees, K-nearest neighbor, a Restricted Boltzmann Machine, and a hybrid system comprising any combination hereof. 8. A system for isolating blood vessels in a thermographic image of a patient, the system comprising: a storage device; and a processor retrieving machine readable instructions from the storage device which, when executed by the processor, enable the processor to: receive a thermographic image of a patient; perform blood vessel detection on the received image to detect pixels of vessel-like structures; identify pixels associated with vessel structures in the thermal image draw lines on the thermal image to represent identified blood vessel structures; and communicate the candidate pixels that satisfy both criterion to the storage device. 9. A system as claimed in claim 8, wherein the blood vessel structures are detected based on temperature of the pixels, intensity of pixels or a shape-based method to detect a patch of pixels that are vessel-like structures, wherein said processor further enabled to identify candidate pixels associated with vessel structures in the breast tissue in the thermal image that satisfy one or all of the temperature-based, intensity-based and shaped-based methods; and remove non-vessel pixels from the identified candidate pixels by imposing a strict constraint of local maximallity on each candidate pixel that satisfies both criterion. 10. The system as claimed in claim 9, wherein the temperature-based method comprises: dividing the image into non-overlapping blocks of pixels of a predefined size; and binarizing each of said non-overlapping blocks with a threshold that is equal to a mean of pixels in that block to identifying candidate pixels. 11. The system as claimed in claim 9, wherein the shape-based method comprises: Vsth.h) = \ sign (sinc (ii^ki)"f)h < ° *■ 0 otherwise where Vs is the vessel-ness measure at scale s, t is a threshold for deciding vessel and non-vessel areas, Xlt X2 are Eigen vectors of a Hessian matrix, where \XX\ > |A2|, and Vs = 1 for tubular structures, and Vs = 0 or Vs = -1, otherwise. 12. The system as claimed in claim 9, wherein the system comprising marking vessel pixels which satisfy both criterion in the image with a different color such that the vessel structures in the breast tissue can be visually differentiated. 13. The system as claimed in claim 12, wherein the system comprising communicating the marked image to any of: a display device, the storage device, and a remote device over a network. 14. The system as claimed in claim 12, wherein the system comprising providing the marked vessel structures to a classifier system which proceeds to classify the breast tissue as malignant, and non-malignant otherwise, based on a tortuosity of the vessel structures. 15. The system as claimed in claim 14, wherein the classifier system comprises any of: Support Vector Machine, a neural network, a Bayesian network, a Logistic regression, Naive Bayes, Randomized Forests, Decision Trees and Boosted Decision Trees, K-nearest neighbor, a Restricted Boltzmann Machine, and a hybrid system comprising any combination hereof. 16. The system as claimed in claim 14, wherein the system comprising communicating the classification to any of: a display device, the storage device, and a remote device over a network.

Documents

Application Documents

# Name Date
1 201847047949.pdf 2018-12-18
2 201847047949-STATEMENT OF UNDERTAKING (FORM 3) [18-12-2018(online)].pdf 2018-12-18
3 201847047949-PROOF OF RIGHT [18-12-2018(online)].pdf 2018-12-18
4 201847047949-POWER OF AUTHORITY [18-12-2018(online)].pdf 2018-12-18
5 201847047949-FORM FOR SMALL ENTITY(FORM-28) [18-12-2018(online)].pdf 2018-12-18
6 201847047949-FORM 1 [18-12-2018(online)].pdf 2018-12-18
7 201847047949-EVIDENCE FOR REGISTRATION UNDER SSI(FORM-28) [18-12-2018(online)].pdf 2018-12-18
8 201847047949-DRAWINGS [18-12-2018(online)].pdf 2018-12-18
9 201847047949-DECLARATION OF INVENTORSHIP (FORM 5) [18-12-2018(online)].pdf 2018-12-18
10 201847047949-COMPLETE SPECIFICATION [18-12-2018(online)].pdf 2018-12-18
11 abstract_201847047949.jpg 2018-12-21
12 Correspondence by Agent_Form 26 And Assignment_28-12-2018.pdf 2018-12-28
13 201847047949-FORM 3 [04-03-2019(online)].pdf 2019-03-04
14 201847047949-STARTUP [27-11-2019(online)].pdf 2019-11-27
15 201847047949-FORM28 [27-11-2019(online)].pdf 2019-11-27
16 201847047949-FORM 18A [27-11-2019(online)].pdf 2019-11-27
17 201847047949-FER.pdf 2020-01-20
18 201847047949-OTHERS [28-05-2020(online)].pdf 2020-05-28
19 201847047949-FER_SER_REPLY [28-05-2020(online)].pdf 2020-05-28
20 201847047949-CORRESPONDENCE [28-05-2020(online)].pdf 2020-05-28
21 201847047949-CLAIMS [28-05-2020(online)].pdf 2020-05-28
22 201847047949-US(14)-HearingNotice-(HearingDate-05-08-2020).pdf 2020-07-13
23 201847047949-Correspondence to notify the Controller [29-07-2020(online)].pdf 2020-07-29
24 201847047949-Correspondence to notify the Controller [04-08-2020(online)].pdf 2020-08-04
25 201847047949-Annexure [04-08-2020(online)].pdf 2020-08-04
26 201847047949-US(14)-ExtendedHearingNotice-(HearingDate-17-08-2020).pdf 2020-08-05
27 201847047949-Correspondence to notify the Controller [16-08-2020(online)].pdf 2020-08-16
28 201847047949-Annexure [16-08-2020(online)].pdf 2020-08-16
29 201847047949-Correspondence to notify the Controller [31-08-2020(online)].pdf 2020-08-31
30 201847047949-Annexure [31-08-2020(online)].pdf 2020-08-31
31 201847047949-Written submissions and relevant documents [21-09-2020(online)].pdf 2020-09-21
32 201847047949-Correspondence to notify the Controller [14-02-2021(online)].pdf 2021-02-14
33 201847047949-Annexure [14-02-2021(online)].pdf 2021-02-14
34 201847047949-Written submissions and relevant documents [01-03-2021(online)].pdf 2021-03-01
35 201847047949-PatentCertificate28-04-2021.pdf 2021-04-28
36 201847047949-IntimationOfGrant28-04-2021.pdf 2021-04-28
37 201847047949-US(14)-ExtendedHearingNotice-(HearingDate-15-02-2021).pdf 2021-10-17
38 201847047949-US(14)-ExtendedHearingNotice-(HearingDate-07-09-2020).pdf 2021-10-17
39 201847047949-RELEVANT DOCUMENTS [30-09-2022(online)].pdf 2022-09-30
40 201847047949-RELEVANT DOCUMENTS [28-08-2023(online)].pdf 2023-08-28

Search Strategy

1 2020-06-0915-03-59AE_09-06-2020.pdf
2 2020-01-0710-43-54_07-01-2020.pdf

ERegister / Renewals

3rd: 01 Jul 2021

From 29/06/2019 - To 29/06/2020

4th: 01 Jul 2021

From 29/06/2020 - To 29/06/2021

5th: 01 Jul 2021

From 29/06/2021 - To 29/06/2022

6th: 01 Jul 2021

From 29/06/2022 - To 29/06/2023

7th: 01 Jul 2021

From 29/06/2023 - To 29/06/2024

8th: 01 Jul 2021

From 29/06/2024 - To 29/06/2025

9th: 01 Jul 2021

From 29/06/2025 - To 29/06/2026

10th: 01 Jul 2021

From 29/06/2026 - To 29/06/2027

11th: 01 Jul 2021

From 29/06/2027 - To 29/06/2028

12th: 01 Jul 2021

From 29/06/2028 - To 29/06/2029

13th: 01 Jul 2021

From 29/06/2029 - To 29/06/2030

14th: 01 Jul 2021

From 29/06/2030 - To 29/06/2031

15th: 01 Jul 2021

From 29/06/2031 - To 29/06/2032