DESC:TECHNICAL FIELD
[0001] The present disclosure relates, in general, to object tracking, and more specifically, relates to a system and method for artificial intelligence (AI) based object tracking while an electro-optic sensor is in scanning mode.

BACKGROUND
[0002] An example is recited in a patent WO2008107925A1 titled “Panoramic Tracking Camera” describes a Vision system device integrating a moving camera (with motors for pan and/or tilt, and/or zoom) and a panoramic camera of any kind (catadioptric or polidiotric). The two cameras thanks to intelligent image processing (realized on hardware and/or on software) actively cooperate to monitor the complete surrounding environment by exploiting the advantages of both types of cameras. Another example is recited in a patent US9947108B1 titled “Method and System for Automatic Detection and Tracking of Moving Objects in Panoramic” describes Panoramic imaging systems and techniques disclosed. In one aspect, a technique for automatically detecting and tracking a foreground object includes the steps of receiving a first set of raw images in an image sequence captured by a panoramic imaging system; stitching, by the panoramic imaging system, the first set of raw images to generate a panoramic image; detecting, by the panoramic imaging system, a foreground object in the panoramic image; tracking, by the panoramic imaging system, a movement of the foreground object in the image sequence; and generating, by the panoramic imaging system, a panoramic video based on the image sequence with tracking the movement of the foreground object.
[0003] Another example is recited in a patent CN109686109B titled “Parking lot safety monitoring management system and method based on artificial intelligence” describes an artificial intelligent parking lot safety management system and method, which comprises a video image target detection subsystem module, a parking management module, an abnormal behaviour detection module, a database storage module, a display module, a hardware equipment module and a management server of a central control system connected with the modules or the subsystem group. The system can realize intelligent safety monitoring management on vehicles and pedestrians in the parking lot, improve the safety management efficiency of the parking lot and reduce the manual workload.
[0004] Yet another example is recited in a patent KR101995107B1 titled “Method and system for artificial intelligence-based video surveillance using deep learning” describes an artificial intelligence-based video surveillance method and system using deep learning is disclosed. The artificial intelligence-based video surveillance system according to an embodiment of the present invention includes a detector for detecting at least one object present for a surveillance camera image using a predefined sensing deep learning network; A recognition unit for recognizing information about the detected object based on a detection result for the object; And a tracking unit for tracking the detected object based on a predefined tracking deep learning network and a detection result for the detected object, wherein the detecting unit detects a heat map based on a convolution neural network (CNN) the object can be detected using a region proposal extraction including a heat map generation method.
[0005] Therefore, it is desired to overcome the drawbacks, shortcomings, and limitations associated with existing solutions, and develop a system that detects an object of interest accurately and can be further fined tuned to improve its performance.

OBJECTS OF THE PRESENT DISCLOSURE
[0006] An object of the present disclosure relates, in general, to object tracking, and more specifically, relates to a system and method for artificial intelligence (AI) based object tracking while an electro-optic sensor is in scanning mode.
[0007] Another object of the present disclosure is to provide a system that detects objects of interest highly accurately and it can be further fined tuned to improve its performance.
[0008] Another object of the present disclosure is to provide a system that is trained to detect and identify other objects of interest.
[0009] Another object of the present disclosure is to provide a system that calculates new information present in each image from the previous image and keeps appending to generate a panorama which helps the operator to visualize the surroundings to monitor any possible threat.
[0010] Yet another object of the present disclosure is to provide a system that detects and tracks objects of interest, where these detections are shown with different symbols on the created panorama.

SUMMARY
[0011] The present disclosure relates in general, to object tracking, and more specifically, relates to a system and method for artificial intelligence (AI) based object tracking while an electro-optic sensor is in scanning mode. The main objective of the present disclosure is to overcome the drawbacks, limitations, and shortcomings of the existing system and solution, by tracking objects of interest while the electro-optic sensor is in scanning mode using an artificial intelligence technique. The method is mainly for surveillance purposes. In this method, images are continuously captured by an image-capturing unit mounted on a pan-tilt platform which rotates to capture the entire 360 views. The panorama creation unit creates a 360-degree panorama for the operator to be able to see the entire view on a user interactive graphical user interface (GUI) display. The tracking unit uses a predefined artificial intelligence-based deep neural network to detect and track the object of interest in the created panorama. This method also provides a facility for the operator to click on the objects of interest to see the zoomed version of the objects for more detail.
[0012] The system for tracking objects in surveillance operations comprises an image-capturing unit configured to continuously capture a set of images of an area of interest using an electro-optic sensor. These images are obtained while the image-capturing unit is mounted over a pan-tilt platform, which rotates continuously at a preset speed in the azimuth direction. A server coupled to the pan-tilt platform performs various operations including panorama creation, object tracking, and user interactive GUI display. The panorama creation unit processes the continuous set of images captured by the image capturing unit to create real-time panoramas. A tracking unit utilizes a learning engine, such as an artificial intelligence-based deep neural network, to detect objects of interest in the created panoramas for tracking. Detected objects are displayed with distinct symbols on a user interactive GUI display, which also presents the created panoramas, live feed from the image capturing unit, and zoomed versions of detected objects.
[0013] This GUI display facilitates operator interaction for viewing and analyzing the detected objects, allowing for enhanced scrutiny. Additionally, objects of interest with various detections and trajectories are visually differentiated on the GUI display through distinct symbols and colors, with unique object identifiers assigned to different objects. Furthermore, the system includes a dataset used to refine the deep neural network for improved detection of objects of interest. The user interactive GUI display is partitioned into three sections, encompassing outputs from the panorama creation unit, tracking unit, and live feed from the image capturing unit.
[0014] Various objects, features, aspects, and advantages of the inventive subject matter will become more apparent from the following detailed description of preferred embodiments, along with the accompanying drawing figures in which like numerals represent like components.

BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The following drawings form part of the present specification and are included to further illustrate aspects of the present disclosure. The disclosure may be better understood by reference to the drawings in combination with the detailed description of the specific embodiments presented herein.
[0016] FIG. 1A illustrates an exemplary block diagram for AI-based object tracking in a panoramic view, in accordance with an embodiment of the present disclosure.
[0017] FIG. 1B shows the user interactive GUI display unit in accordance with an embodiment of the present disclosure.
[0018] FIG. 2 is a flow chart of a method for AI-based object tracking while the electro-optic sensor is in scanning mode, in accordance with an embodiment of the present disclosure.
[0019] FIG. 3 illustrates an exemplary flow chart of a method for tracking objects in surveillance operations, in accordance with an embodiment of the present disclosure.

DETAILED DESCRIPTION
[0020] The following is a detailed description of embodiments of the disclosure depicted in the accompanying drawings. The embodiments are in such detail as to clearly communicate the disclosure. If the specification states a component or feature “may”, “can”, “could”, or “might” be included or have a characteristic, that particular component or feature is not required to be included or have the characteristic.
[0021] As used in the description herein and throughout the claims that follow, the meaning of “a,” “an,” and “the” includes plural reference unless the context clearly dictates otherwise. Also, as used in the description herein, the meaning of “in” includes “in” and “on” unless the context clearly dictates otherwise.
[0022] The present disclosure relates, in general, to object tracking, and more specifically, relates to a system and method for artificial intelligence (AI) based object tracking while an electro-optic sensor is in scanning mode.
[0023] The present invention provides a method for AI-based object tracking while the electro-optic sensor is for surveillance purposes. The system consists of an image-capturing unit mounted on a pan-tilt platform which can rotate at a specified rpm speed (user configurable). The image-capturing unit continuously captures images from the surrounding area under surveillance. The panorama creation unit then creates panorama continuously by appending new information content present in each image of the scene. The tracking unit uses a fine-tuned predefined artificial intelligence-based deep neural network model to detect the objects of interest in the created panorama and mark different objects by different symbols and colours such as white rectangular boxes for detected persons and black circular shapes for ships. All the outputs are shown on the GUI display unit. The created panorama is shown in two-row strips on the GUI display unit, where the first-row strip shows 0-180-degree panorama and the second-row strip shows 180-360-degree panorama. GUI display shows a live feed from the electro-optic sensor in the bottom left part, detections of objects of interest are marked on the panorama itself and it also provides a facility for the user/operator to click on the detections to see the zoomed version of the detected objects for better detailing.
[0024] The present disclosure relates to a system for tracking objects in surveillance operations including an image-capturing unit configured to capture a set of images of an area of interest using an electro-optic sensor. A pan-tilt platform is configured to rotate continuously at a preset speed in azimuth direction, where the image capturing unit is mounted over the pan-tilt platform. The image-capturing unit captures the set of images continuously to cover a 360-degree view under surveillance. The panorama created by the panorama creation unit is displayed in two strips on the user interactive GUI display, wherein a first strip covers a 0-180-degree view and a second strip covers a 180-360-degree view of the entire scene.
[0025] Further, a server coupled to the pan-tilt platform, the server is configured to perform panorama creation, object tracking, and user-interactive GUI display operations. A panorama creation unit is configured to create real-time panoramas from a continuous set of images captured by the image-capturing unit.
[0026] A tracking unit configured to detect objects of interest using a learning engine in the created panoramas for tracking and displaying the detected objects with distinct symbols. The learning engine is an artificial intelligence-based deep neural network. The tracking unit employs a deep neural network fine-tuned for detecting objects of interest, where the objects of interest are selected from persons, ships and any combination thereof.
[0027] Moreover, a user-interactive GUI display is configured to display the created panoramas, live feed from the image capturing unit, and zoomed versions of detected objects, where the GUI display allows an operator to interactively view and analyze the detected objects. The user-interactive GUI display enables an operator to interact with the detected objects, allowing for viewing of the zoomed versions for enhanced scrutiny. The objects of interest with various detections and trajectories are visually differentiated on the user-interactive GUI display through distinct symbols and colors. The user-interactive GUI display is partitioned into three sections, including outputs from the panorama creation unit, tracking unit, and live feed from the image capturing unit.
[0028] Besides, the system comprises a dataset utilized to refine the deep neural network for the detection of objects of interest. The present disclosure can be described in enabling detail in the following examples, which may represent more than one embodiment of the present disclosure.
[0029] The advantages achieved by the system of the present disclosure can be clear from the embodiments provided herein. The system detects objects of interest highly accurately and it can be further fined tuned to improve its performance. The system is trained to detect and identify other objects of interest. The system calculates new information present in each image from the previous image and keeps appending to generate a panorama which helps the operator visualize the surroundings to monitor any possible threat. Further, the present invention provides a system that detects and tracks objects of interest, where these detections are shown with different symbols on the created panorama. The description of terms and features related to the present disclosure shall be clear from the embodiments that are illustrated and described; however, the invention is not limited to these embodiments only. Numerous modifications, changes, variations, substitutions, and equivalents of the embodiments are possible within the scope of the present disclosure. Additionally, the invention can include other embodiments that are within the scope of the claims but are not described in detail with respect to the following description.
[0030] FIG. 1A illustrates an exemplary block diagram for AI-based object tracking in a panoramic view, in accordance with an embodiment of the present disclosure.
[0031] Referring to FIG. 1A, the system 100 and method for artificial intelligence (AI)-based object tracking is running on a graphics processing unit (GPU)-based server 106. The present disclosure pertaining to artificial intelligence-based object tracking when an electro-optic (EO) sensor is in scanning mode for surveillance applications. The system 100 can include image capturing unit 102, pan-tilt platform 104, GPU-based server 106, the panorama creation unit 108, tracking unit 110 and user interactive GUI display 112. This method is a single GPU-based server 106 solution shown in FIG. 1A i.e., panorama creation block, tracking and user interactive GUI display is performed in the GPU server 106.
[0032] In this method, the electro-optic sensor is the image-capturing unit 102 which is placed over the pan-tilt platform 104. The pan-tilt platform 104 rotates continuously at the specified rpm which is user-configurable in the azimuth direction. The image capturing unit 102 mounted over pan-tilt platform 104 captures images continuously to cover the entire 360-degree view under surveillance. These images are the input to the panorama creation unit 108 which creates the real-time panorama of the surroundings. The panorama is displayed in a user-interactive GUI display 112 in two strips to cover the entire 360 views, the first strip covers 0-180 degrees and the second covers 180-360-degree view of the entire scene for better detailing of objects. The tracking unit 110 then detects the objects of interest using a predefined artificial intelligence-based deep neural network in the created panorama for tracking and denotes the detection with different symbols e.g., white rectangular box for the person, black circular shape for ships. The tracking unit 110 output is also shown in the user interactive GUI display 112.
[0033] The object tracking is performed by a predefined artificial intelligent-based deep neural network fine-tuned for detecting persons and ships. The user interactive GUI 112, which displays created panorama, live feed and zoomed version of the detected object. The GUI display unit 112 provides a facility for the user/operator to click on the detected objects to see their zoomed version for better detailing. The different detections and trajectories of objects of interest are shown with different symbols and colors such as white rectangular boxes for persons, and black circular shapes for ships. Further, different objects of interest are marked with unique object IDs.
[0034] The dataset for detecting objects of interest has been created for fine-tuning the deep neural network. The artificial intelligent based deep neural network is applied to the image created by appending pixels which contain new information in each frame to detect objects of interest
[0035] The electro-optic sensor-based object tracking systems, methods for real-time object tracking, are widely available. Most of the EO-based object trackers are working on the principal of background modelling and suppression of background. The foreground object after background suppression is the target which is selected for tracking. In this method, the persons and ships are selected as targets for tracking.
[0036] The available existing methods use multiple static fixed images capturing sensors for surveillance of an area under surveillance for continuous monitoring or conventional methods for tracking objects of interest. However, the present invention uses only a single image-capturing sensor 102 mounted over the pan-tilt platform 104, which rotates at a specific rpm speed provided by the operator to cover the entire 360-degree area under surveillance. The speed of the pan-tilt platform 104 is configurable. The present disclosure uses a predefined artificial intelligence-based deep neural network module for the detection and tracking of desired objects of interest. This present invention also provides the GUI display 112 for displaying all the outputs and this display is user-interactive which means the operator can click on any of the detections to view the zoomed version of that area for better analysis.
[0037] FIG. 1B shows the user interactive GUI display unit in accordance with an embodiment of the present disclosure. The user interactive GUI display unit 112 where the final output is shown. The GUI is divided into three parts such as panorama creation unit output, tracking unit output and the live feed from the image capturing unit 102. The live feed which is captured by the electro-optic sensor is shown in the bottom left corner of the GUI, the 0-180-degree view of panorama is shown in the first half of the GUI and the remaining 180-360-degree view of panorama is shown in the second half of the GUI. Since GUI is user-interactive so the operator can click on the display to see the zoomed view of the detected object for better detailing which is shown in the bottom right corner of the GUI display.
[0038] FIG. 2 is a flow chart of a method for AI-based object tracking while the electro-optic sensor is in scanning mode, in accordance with an embodiment of the present disclosure.
[0039] The images from the surrounding area under surveillance are captured continuously by the image capturing unit 102 i.e., by an electro-optic sensor mounted over pan-tilt platform 104, which rotated continuously at the specified rpm speed in the azimuth direction. From these images, the panorama creation unit 108 continuously creates panorama in real time which is shown on GUI display unit 112. The tracking unit 110 then performs real-time detection using a predefined artificial intelligence-based deep neural network model to identify all the objects of interest in the created Panorama and denotes different objects with different symbols e.g., rectangular box for humans, black circular shape for ships. The tracking unit 110 output is also displayed on the created panorama in the GUI display unit 112. The operator can click on any of the detections from tracking unit 110 to see the zoomed version of the detected objects for better detailing.
[0040] At block 202, the system can perform capturing of images continuously by an electro-optic sensor mounted on a pan-tilt platform to cover 360 views. At block 204, the images are sent to the panorama creation unit through Ethernet Interface. At block 206, perform 360-degree panorama creation. At block 208, the tracking unit detects objects of interest in the created panorama. At block 210, the operator clicks on objects of interest. Further, at block 212, the zoomed version of detected objects currently clicked by the operator is shown on the GUI display unit for better detailing along with created panorama and live feed and at block 214, the previous zoomed version if any is shown on GUI display unit along with created panorama and live feed.
[0041] For tracking unit 110 to be able to detect and track objects of interest, a database containing all the relevant images is required with objects of interest. It is required to fine-tune the artificial intelligence-based deep neural network to be able to detect the objects of interest. These detections are shown with different symbols on the created panorama.
[0042] Thus, the present invention overcomes the drawbacks, shortcomings, and limitations associated with existing solutions, and provides the advantage of using a predefined artificial intelligence-based deep neural network to detect objects of interest highly accurately and it can be further fined tuned to improve its performance. The predefined deep neural network has been trained to detect persons and ships but it can be further trained to detect and identify other objects of interest. For the deep neural network to be able to detect other objects it has to be trained and fine-tuned with the database created from the images containing those objects. When the image capturing unit 102 rotates and captures images then very few pixels only in an image contain new information from the previous image. The panorama creation unit 108 calculates new information present in each image from the previous image and keeps appending to generate a panorama which helps the operator to visualize the surroundings to monitor any possible threat.
[0043] FIG. 3 illustrates an exemplary flow chart of a method for tracking objects in surveillance operations, in accordance with an embodiment of the present disclosure.
[0044] The method 300 includes block 302, capturing a set of images of an area of interest using an electro-optic sensor by the image capturing unit 102. At block 304, rotating, continuously at a preset speed in azimuth direction, wherein the image capturing unit 102 is mounted over the pan-tilt platform 104 by a pan-tilt platform 104.
[0045] At block 306, performing panorama creation, object tracking, and user interactive GUI display operations, the server coupled to the pan-tilt platform 104 at a server 106. At block 308, creating real-time panoramas from a continuous set of images captured by the image capturing unit 102 at the panorama creation unit 108.
[0046] At block 310, detecting objects of interest using a learning engine in the created panoramas for tracking by the tracking unit 110. At block 312, displaying the created panoramas, live feed from the image capturing unit 102, and zoomed versions of detected objects with distinct symbols, wherein said GUI display 112 allows an operator to interactively view and analyze the detected objects by the user interactive GUI display 112.
[0047] It will be apparent to those skilled in the art that the system 100 of the disclosure may be provided using some or all of the mentioned features and components without departing from the scope of the present disclosure. While various embodiments of the present disclosure have been illustrated and described herein, it will be clear that the disclosure is not limited to these embodiments only. Numerous modifications, changes, variations, substitutions, and equivalents will be apparent to those skilled in the art, without departing from the spirit and scope of the disclosure, as described in the claims.

ADVANTAGES OF THE PRESENT INVENTION
[0048] The present invention provides a system that detects objects of interest highly accurately and it can be further fined tuned to improve its performance.
[0049] The present invention provides a system that is trained to detect and identify other objects of interest.
[0050] The present invention provides a system that calculates new information present in each image from the previous image and keeps appending to generate a panorama which helps the operator to visualize the surroundings to monitor any possible threat.
[0051] The present invention provides a system that detects and tracks objects of interest, where these detections are shown with different symbols on the created panorama.
,CLAIMS:1. A system (100) for tracking objects in surveillance operations, the system comprising:
an image capturing unit (102) configured to capture a set of images of an area of interest using an electro-optic sensor;
a pan-tilt platform (104) configured to rotate continuously at a preset speed in azimuth direction, wherein the image capturing unit (102) is mounted over the pan-tilt platform (104);
a server (106) coupled to the pan-tilt platform (104), the server configured to perform panorama creation, object tracking, and user interactive graphical user interface (GUI) display operations;
a panorama creation unit (108) configured to create real-time panoramas from a continuous set of images captured by the image capturing unit (102);
a tracking unit (110) configured to detect objects of interest using a learning engine in the created panoramas with distinct symbols; and
a user interactive GUI display (112) configured to display the created panoramas, live feed from the image capturing unit (102), and zoomed versions of detected objects, wherein said GUI display (112) allows an operator to interactively view and analyze the detected objects.
2. The system as claimed in claim 1, wherein the image capturing unit (102) captures the set of images continuously to cover a 360-degree view under surveillance.
3. The system as claimed in claim 1, wherein the panorama created by the panorama creation unit (108) is displayed in two strips on the user interactive GUI display (112), wherein a first strip covers a 0-180-degree view and a second strip covers a 180-360-degree view of the entire scene.
4. The system as claimed in claim 1, wherein the learning engine is an artificial intelligence-based deep neural network, wherein the artificial intelligent based deep neural network is applied on the set of images created by appending pixels which contains new information in each frame to detect the objects of interest.
5. The system of claim 1, wherein the tracking unit (110) employs a deep neural network fine-tuned for detecting the objects of interest, wherein the objects of interest are selected from persons, ships and any combination thereof.
6. The system of claim 1, wherein the user interactive GUI display (112) enables an operator to interact with the detected objects, allowing for viewing of the zoomed versions for enhanced scrutiny.
7. The system as claimed in claim 1, wherein the objects of interest with various detections and trajectories are visually differentiated on the user interactive GUI display (112) through distinct symbols and colors, wherein different objects of interest are marked with unique object identifiers (IDs).
8. The system as claimed in claim 1, wherein the system comprises a dataset utilized to refine the deep neural network for the detection of the objects of interest.
9. The system as claimed in claim 1, wherein the user interactive GUI display (112) is partitioned into three sections, including outputs from the panorama creation unit, the tracking unit, and the live feed from the image capturing unit (102).
10. The method (300) for tracking objects in surveillance operations, the method comprising:
capturing (302), by an image capturing unit (102), a set of images of an area of interest using an electro-optic sensor;
rotating (304), by a pan-tilt platform (104), continuously at a preset speed in azimuth direction, wherein the image capturing unit (102) is mounted over the pan-tilt platform (104);
performing (306), at a server (106), panorama creation, object tracking, and user interactive GUI display operations, the server coupled to the pan-tilt platform (104);
creating (308), at a panorama creation unit (108), real-time panoramas from continuous set of images captured by the image capturing unit (102);
detecting (310), by a tracking unit (110), objects of interest using a learning engine in the created panoramas with distinct symbols; and
displaying (312), by a user interactive GUI display (112), the created panoramas, live feed from the image capturing unit (102), and zoomed versions of detected objects, wherein said GUI display (112) allows an operator to interactively view and analyze the detected objects.