Abstract:
The invention relates to a method for detecting and tracking targets in a series of images, at least one track, called an existing track, having being formed from blips detected in images of said series preceding a current image. Said method comprises, for the current image: obtaining (30) at least one blip contained in said current image, and for each blip, a value that represents a characteristic of said blip; sorting (31) each blip according to said representative value into a first category when said representative value is higher than a first predefined threshold or into a second category when said representative value is between the first predefined threshold and a second predefined threshold lower than the first threshold; and, for each existing track, allocating (32) preferably a first category blip to said existing track and allocating a second category blip to said existing track when none of the first category blips is compatible with said existing track.
Get Free WhatsApp Updates!
Notices, Deadlines & Correspondence
c/o SAFRAN ELECTRONICS & DEFENSE, 72-76 rue Henry Farman, 75015 Paris, France
Specification
TARGET DETECTION AND TRACKING PROCESS
The invention relates to a method for detecting and tracking targets, as well as a device and equipment suitable for implementing said method.
Methods using target detection and tracking (“object detection and tracking” in English terminology) are commonly used to carry out environmental monitoring, in particular in the air and maritime fields. These methods consist of successively capturing a series of optronic images of the environment, looking in each of the images for signs of the presence of a potential target, then when clues have been found, confirming or denying the real presence. of a target by determining whether a repeat of coherent detections exists in several successive images.
The following terms are used in target detection and tracking:
• Plot or hot point ("spot" in English terminology): sub-part of an image comprising adjacent pixels having characteristics which are homogeneous between them and different from the pixels of a neighborhood close to the sub-part in the picture. A plot is detected by applying a detection criterion. This detection criterion can be established as a function of an average signal level and / or a noise level in the image. For example, a pixel can be considered as belonging to a plot when its luminance level has a difference with an average luminance level of the pixels of its close vicinity in the image greater than a predefined detection threshold. More generally, the detection of a plot can be based on an observation of signal to noise ratio (“Signal to Noise Ratio (SNR)” values of the pixels of the image. A pixel whose SNR is greater than a predefined detection threshold is considered to belong to a plot. A plot can correspond to a target or to a detection aberration in an image.
• Track: spatio-temporal chain of plots detected in several successive images. A track can have three states: candidate track (being initialized or forked), track confirmed or track abandoned. The states of a track are updated using a track management function.
track. The runway management function allocates the candidate runway state to each new runway. A candidate track is usually initialized from plots that have not been used to update an existing confirmed track. The track management function can confirm a candidate track using more or less sophisticated techniques depending on a desired level of performance or false alarm. For example, a track can be confirmed when a spatio-temporal chain of plots can be constructed from plots found in Q images of a series of P consecutive images (P> Q).
• Algorithm for assigning plots to tracks: An algorithm for assigning plots to tracks allows a plot to be associated with an existing track (confirmed or in the process of being initialized). When a new image (called the “current image”) is obtained by a target detection and tracking process, it is analyzed in order to search for spots therein. In parallel, a prediction of a position of a stud, called a “predicted stud” is carried out from each existing track in order to determine where a stud may be located making it possible to continue said track. This prediction of the position of the predicted stud is generally done using a runway maintenance function. One or two validation windows are then defined around the position of the predicted stud. We then speak of a prediction window and an evolution window. We then look for a suitable plot to continue said track in these validation windows. The assignment can consist of two stages: the pre-correlation and the final assignment. During the pre-correlation, the candidate plots for the association of each track are indicated. It is a question of taking into account all the plots which fall in the validation window of the track. A table is thus formed indicating whether a plot is one of the candidates that can be associated with a track. During the final assignment, a more detailed analysis of the relationships between the pads and the tracks is carried out. There are various methods of final assignment. We can the candidate plots for association of each track are indicated. It is a question of taking into account all the plots which fall in the validation window of the track. A table is thus formed indicating whether a plot is one of the candidates that can be associated with a track. During the final assignment, a more detailed analysis of the relationships between the pads and the tracks is carried out. There are various methods of final assignment. We can the candidate plots for the association of each track are indicated. It is a question of taking into account all the plots which fall in the validation window of the track. A table is thus formed indicating whether a plot is one of the candidates that can be associated with a track. During the final assignment, a more detailed analysis of the relationships between the pads and the tracks is carried out. There are various methods of final assignment. We can There are various methods of final assignment. We can There are various methods of final assignment. We can thus cite for example the method of the global nearest neighbor ("Global Nearest Neighbor Approach (GNN)" in English terminology) or the method of the sub-optimal closest neighbor ("Suboptimal Nearest Neighbor (SNN)" in English terminology ) both described in the article "A study of a target Tracking Algorithm Using Global
Nearest Neighbor Approach, P. Konstantinova, International Conference on computer Systems and technologies (CompSysTech 2003) ”.
• Track maintenance function: a track maintenance function provides an estimate at a current instant and a prediction at a time following the current instant of the kinematic properties of a target tracked from a track. This function is generally implemented by a Kalman filter.
Target detection and tracking methods must find a compromise between effectively detecting plots and avoiding increasing a probability of false alarms (ie creating tracks for sets of pixels not corresponding to targets). It is known that the more the number of pads detected increases, the more the probability of false alarm increases. In order for the number of pads to remain reasonable, the value of the detection threshold is generally increased.
On the other hand, it is known that a target can evolve in a variable environment.
Figs. 1A, 1B and 1C represent an image acquired by an optronic device such as a camera (not shown) in which an aircraft 1 operates in an environment comprising a cloud 2. During its movement in this environment, the aircraft 1 passes successively from a cloud-free area to a cloud-free area, then reverts to a cloud-free area. The aircraft 1 is potentially a target that can be detected and tracked by a target detection and tracking method.
In the image shown in FIG. 1A, the pixels corresponding to the aircraft 1 have characteristics (luminance values, SNR values, etc.) that are very different from the pixels of the vicinity of the aircraft 1. A plot corresponding to the aircraft is then detected without difficulty. 1 in this picture. Indeed, for example, the pixels corresponding to the aircraft 1 have SNR values greater than a predetermined detection threshold, which is not the case with the pixels located in the vicinity of the aircraft 1. This plot can then be assigned to an existing track that would have been created from images preceding the image of Fig. 1A.
In the image shown in FIG. 1B, the pixels corresponding to the aircraft 1 are embedded in pixels corresponding to the cloud 2. It is then difficult to distinguish the pixels corresponding to the aircraft 1 from the pixels corresponding to the cloud 2 because these pixels have similar characteristics. In fact, cloud 2 corresponds to a textured area. A textured area greatly increases the variation in pixel values in the vicinity of a target. The textured area causes an increase in noise and therefore a drop in the SNR values. For example, the pixels corresponding to the aircraft 1 have low SNR values, lower than the predefined detection threshold, like the pixels located in the vicinity of the aircraft 1. The probability of not detecting a plot corresponding to the aircraft 1 is then important.
In the image represented by Lig. 1C, the aircraft 1 returns to a zone in which the pixels which correspond to it have very different characteristics from the pixels of its neighborhood. Again, the pixels corresponding to the aircraft 1 have SNR values greater than the predetermined detection threshold. It then becomes easy again to detect a plot corresponding to the aircraft 1. If the runway has been abandoned in the image corresponding to Lig. 1B, there is a risk of tracking discontinuity, that is to say that a new track is created with a new identifier in the image corresponding to Lig. 1C, while the target (ie aircraft 1) is the same.
This example shows that using a high detection threshold makes it possible to avoid increasing the number of spots detected, but on the other hand, prevents the detection of spots corresponding to real targets, when the pixels corresponding to these targets are located in a environment unfavorable to detection.
It is desirable to overcome these drawbacks of the state of the art. It is in particular desirable to provide a method which makes it possible to follow a target in an environment unfavorable to detection.
It is also desirable to provide a method which is simple to implement and which has a low computational cost.
According to a first aspect of the present invention, the present invention relates to a method for detecting and tracking targets in a series of images supplied by an optronic system. The method comprises for a current image: obtaining at least one plot included in said current image and for each plot, a value representative of a characteristic of said plot; classify each plot according to said representative value in a first category when said representative valueis greater than a first predefined threshold or in a second category when said representative value is between the first predefined threshold and a second predefined threshold lower than the first predefined threshold; for each existing spatio-temporal chain of plots of a first type, called an existing track of the first type, having been formed from plots detected in images of said series preceding the current image, assign a plot of the first category to said existing track of the first type when at least one first category pad is compatible according to a predefined criterion with said existing track of the first type, and when no first category pad is compatible with said existing track of the first type, assign a second category plot when at least one second category plot is compatible according to the predefined criterion with the existing track of the first type, an assignment to an existing track of the first type making it possible to continue said existing track of the first type in the current image; creating a new track of the first type for each plot of the first category that has not been assigned to an existing track; classifying each second category pad that has not been assigned to an existing track of the first type in a third category if a number of pixels making up this pad is greater than a third predefined threshold and in a fourth category otherwise; for each existing spatio-temporal chain of plots of at least one second type, called the existing track of the second type, having been formed from pads detected in images of said series preceding the current image, assigning a third category pad to said existing track of the second type when at least one third category pad is compatible according to the predefined criterion with said existing track of the second type and transforming said existing track of the second type into an existing track of the first type when the number of pads making up said track exceeds a predefined number of pads; creating a new track of the second type for each plot of the third category that has not been assigned to an existing track of the second type. assign a third category pad to said existing track of the second type when at least one third category pad is compatible according to the predefined criterion with said existing track of the second type and transform said existing track of the second type into an existing track of the first type when the number of pads making up said track exceeds a predefined number of pads; creating a new track of the second type for each plot of the third category that has not been assigned to an existing track of the second type. assign a third category pad to said existing track of the second type when at least one third category pad is compatible according to the predefined criterion with said existing track of the second type and transform said existing track of the second type into an existing track of the first type when the number of pads making up said track exceeds a predefined number of pads; create a new track of the second type for each plot of the third category that has not been assigned to an existing track of the second type.
According to one embodiment, the value representative of a characteristic of said pad is an SNR value representative of SNR values of pixels belonging to said pad, the first and second predefined thresholds being predefined values of SNR.
According to one embodiment, a plot is compatible with an existing track of the first or of the second type when it is included in a sub-part of the current image, called a validation window, corresponding to said track, a validation window being defined around a prediction of a position, called the predicted position, in which there must be a stud corresponding to said track in the current image, said position being obtained from a runway maintenance function by taking into account the kinematic properties of a target tracked from said runway.
According to one embodiment, when a plurality of first category pads is compatible with an existing track of the first type, the first category pad closest to the predicted position according to a predefined proximity criterion is assigned to said track and when 'a plurality of second category pads is compatible with an existing track of the second type, the second category pad closest to the predicted position according to the predefined proximity criterion is selected, only first or second category pads can be assigned to a first type runway.
According to one embodiment, when a plurality of third category pads is compatible with an existing track of the second type, the third category pad closest to the position predicted according to the predefined proximity criterion is assigned to said track, only third category pads that can be assigned to a second type track.
According to one embodiment, the tracks of the second type include two subtypes of tracks, a track of the second type being of the first subtype as long as it comprises a number of pads of the third category less than the predefined number of pads, a track of the second type being of the second subtype when it comprises a number of pads greater than the first predefined number of pads, a track of the second type of the second subtype being transformed into a track of the first type when a sum of values SNR representative of pads constituting said track of the second type is greater than a fourth predefined threshold over a predefined number of successive pads, or when a change in the size of the pads of said track of the second type is strictly increasing over a second number of predefined successive plots,or when an evolution of the SNR of the pads of said track of the second type is strictly increasing over a third predefined number of successive pads.
According to one embodiment, following the assignment of the first or second category pads to the existing track of the first type, and before the creation of a new track of the first type for each first category pad that has not been assigned to an existing track, the method further comprises, for each existing track of the second type of the second subtype, assigning a first category pad to said track when at least one first category pad is compatible according to an assignment criterion predefined with said track and, when no first category pad is compatible with said track, assigning a second category pad when at least one second category pad is compatible according to the predefined allocation criterion with said track.
According to a second aspect of the invention, the invention relates to a device comprising means for implementing the method according to the first aspect.
According to a third aspect of the invention, the invention relates to equipment comprising a device according to the second aspect.
According to a fourth aspect of the invention, the invention relates to a computer program, comprising instructions for implementing, by a device, the method according to the first aspect, when said program is executed by a processor of said device.
According to a fifth aspect of the invention, the invention relates to storage means, storing a computer program comprising instructions for implementing, by a device, the method according to the first aspect, when said program is executed by a device. processor of said device.
The characteristics of the invention mentioned above, as well as others, will emerge more clearly on reading the following description of an exemplary embodiment, said description being given in relation to the accompanying drawings, among which:
- Figs. 1A, 1B and 1C schematically illustrate images acquired by an optronic device such as a camera (not shown) in which an aircraft is operating in an environment comprising a cloud;
- Fig. 2A schematically illustrates equipment suitable for implementing the invention;
- Fig. 2B schematically illustrates a hardware architecture of an analysis module able to implement the invention;
- Fig. 3 schematically illustrates a method for detecting and tracking targets according to the invention;
- Fig. 4 schematically illustrates a first detail of the method for detecting and tracking targets according to the invention;
- Fig. 5 schematically illustrates a second detail of the method for detecting and tracking targets according to the invention; and,
- Fig. 6 schematically illustrates a third detail of the method for detecting and tracking targets according to the invention.
The invention is described below in the context of aerial surveillance using an image acquisition device making image acquisitions in a visible domain. The invention could equally well be applied in different surveillance contexts such as maritime surveillance. Moreover, the invention applies to other types of images (ie other spectral domains) such as infra-red images, thermal images and radar images.
Fig. 2A schematically illustrates equipment 10 suitable for implementing the invention.
The equipment 10 comprises an image acquisition module 100 such as a camera operating in the visible spectral domain. The image acquisition module 100 supplies the images that it has acquired to an image processing module 101. The image processing module 101 is able, among other things, to detect spots in each image and to calculate at least one representative value of each plot.
The image processing module 101 supplies the acquired images, the position of each detected plot and for each detected plot, a value representative of said plot to an analysis module 102. The analysis module 102 implements a method of target detection and tracking according to the invention. This process, which we describe below in relation to Figs. 3, 4, 5 and 6 is able to create new tracks, extend tracks and abandon tracks. The analysis module 102 is responsible for confirming the tracks and for this uses a track management function.
For each image acquired, the analysis module 102 transmits information representative of each confirmed track to a display module 103 which displays this information so that it can be used by an operator. For example, the analysis module 102 supplies the display module 103 with information representative of a position of each dot detected in the last image acquired which could be assigned to a confirmed track.
The equipment 10 is considered to form an optronic system.
In Fig. 2A, the image acquisition 100, image processing 101, analysis 102 and display 103 modules are shown as forming part of the same equipment 10. In one embodiment, these modules could also be included in different equipment or devices each comprising a communication module allowing them to communicate with each other. These
equipment or devices then form an optronic system capable of implementing the invention.
Fig. 2B schematically illustrates an example of a hardware architecture of an analysis module capable of implementing the invention.
According to the example of hardware architecture shown in FIG. 2B, the analysis module 102 then comprises, connected by a communication bus 1020: a processor or CPU (“Central Processing Unit” in English) 1021; a random access memory RAM (“Random Access Memory” in English) 1022; a ROM read only memory 1023; a storage unit such as a hard disk or a storage medium reader, such as an SD (“Secure Digital”) 1024 card reader; at least one communication interface 1025 allowing the analysis module 102 to communicate with the image processing module 101 and the display module 103.
Processor 1021 is capable of executing instructions loaded into RAM 1022 from ROM 1023, external memory (not shown), storage media (such as an SD card), or a communication network. When equipment 10 is powered on, processor 1021 is able to read instructions from RAM 1022 and execute them. These instructions form a computer program causing the complete or partial implementation, by the processor 1021, of the methods described below in relation to FIGS. 3, 4, 5 and 6.
The methods described in relation to Figs. 3, 4, 5 and 6 can be implemented in software form by executing a set of instructions by a programmable machine, for example a DSP (“Digital Signal Processor”) or a microcontroller, or be implemented in hardware form by a machine or a dedicated component, for example an FPGA (“Field-Programmable Gâte Array” in English) or an ASIC (“Application-Specific Integrated Circuit” in English).
Fig. 3 schematically illustrates a method for detecting and tracking targets according to the invention.
The method is implemented for each image acquired by the image acquisition module 100. An image being processed by the equipment 10 is called the current image below. In one embodiment, we assume that at least one track, called an existing track, has been formed from pads detected in images preceding the current image.
As we will see below, the method uses a plurality of types of tracks comprising tracks of nominal type, called nominal tracks, and at least a second type of tracks comprising tracks of TBD type, called TBD tracks (Track before detecting. : "Track Before Detect" in Anglo-Saxon terminology).
A track is considered of nominal type when it consists of a pad or a set of pads having an interest in carrying out target tracking according to at least a first predefined condition. A first example of a first predefined condition is such that a track comprising at least one pad having an SNR value greater than a predefined SNR threshold is considered to be a nominal track. A second example of a first predefined condition is such that a track comprising a number of pads greater than a predefined number is considered to be a nominal track. Each nominal track can be a confirmed track, a track being initialized, or a forked track created from a confirmed track.
A track is considered of type TBD when it does not fulfill any first predefined condition. A TBD track must, however, fulfill at least a second predefined condition that we describe below.
Note that only the nominal tracks can be confirmed by the analysis module 102. To do this, the analysis module 102 performs a track management function consisting of confirming a nominal track when said nominal track has been constructed from. of Q plots found in Q images of a sequence of P consecutive images (P> Q), P and Q being predefined positive integers. In one embodiment P = 20 and 0 = 16. A second type lead is never confirmed.
In a step 30, the analysis module 102 obtains at least one plot included in said current image and for each plot, a value representative of a characteristic of said VR plot. In one embodiment, the value representative of a VR plot is for example an average SNR value of the pixels constituting said plot.
In a step 31, the analysis module 102 classifies each plot as a function of said representative value VR in a first category when said value VR is greater than a first predefined threshold T 1 or in a second category when said representative value VR is included. between the first predefined threshold T 1 and a second predefined threshold T 2 less than the first predefined threshold T 1 . The first category includes plots for which the probability of false alarms is low, called standby plots. The second category includes plots for which the probability of false alarms is higher than for standby plots. The plots classified in thesecond category, specific plots. In the embodiment in which the value representative of a VR plot is an average SNR value, the first and second thresholds T 1 and T 2 are SNR values obtained on training image sequences. We then detail the classification of the plots in categories in relation with FIG. 4.
In a step 32, the analysis module 102 implements an algorithm for assigning pads to the existing nominal tracks.
Within the framework of this algorithm, for each existing nominal track, the analysis module 102 assigns a standby pad to said existing nominal track when at least one standby pad is compatible according to a predefined allocation criterion with said existing nominal track. When no standby pad is compatible with said existing nominal track, the analysis module 102 assigns a specific pad when at least one specific pad is compatible according to the predefined allocation criterion with the existing nominal track.
In this way, we favor the assignments of standby plots since these are the plots that present the least risk of false alarms (i.e. the least risk of generating false nominal tracks or of continuing nominal tracks while the corresponding target n 'is no longer observed by the image acquisition module 100). However, the specific pads are taken into account in order to continue with existing nominal tracks for which no compatible standby pad has been found. Thus, nominal tracks are continued with plots of poorer quality in terms of the probability of false alarms. However, using specific plots to pursue an existing nominal track allows the pursuit of nominal tracks that would have been abandoned by mistake, when for example,
Note that when no existing nominal track has been formed on the previous images, the track allocation step 32 is not implemented. The analysis module 102 then goes directly from step 31 to step 33.
In step 33, the analysis module 102 checks whether standby plots have not been assigned to an existing track.
If this is the case, step 33 is followed by a step 34 during which the analysis module 102 creates a candidate track from the standby plots that have not been assigned to an existing track.
Following step 34 (or following step 33 if there are no standby plots left after the implementation of the plot allocation algorithm), the analysis module 102 implements a step 35 during which the analysis module 102 determines whether at least one specific pad has not been assigned to an existing nominal track.
If each specific plot has been assigned, the analysis module 102 waits for a new image in a step 45 and returns to step 30 when a new image is supplied to it by the image processing module. 101.
If at least one specific plot has not been assigned, the analysis module 102 determines for each remaining specific plot, if this specific plot must be definitively rejected, or if this plot must be kept to create or continue a TBD track. To do this, the analysis module 102 executes steps 36 to 39 for each specific unassigned pad. During step 36, the analysis module 102 classifies said specific pad in a third category during a step 37 if a number of pixels making up this specific pad is greater than a threshold T 3 and in a fourth category otherwise during of a step 38. A third category plot is called a TBD plot. The fourth category plots are definitively rejected by the analysis module 102. In one embodiment T 3= 6 pixels.
In a step 39, the analysis module 102 determines whether at least one other specific unassigned pad remains to be processed. If so, the analysis module 102 returns to step 36. If not, the analysis module 102 executes a step 40.
During step 40, the analysis module 102 implements an algorithm for assigning pads to existing TBD tracks. Within the framework of this algorithm, for each existing TBD track, the analysis module 102 assigns a TBD plot to said existing TBD track when at least one TBD plot is compatible according to the predefined allocation criterion with said existing TBD track. Step 40 is detailed in relation to FIG. 6.
An example of a second predefined condition that a TBD track must comply with is therefore that it is made up of TBD pads, that is to say made up of at least one specific pad not assigned to an existing nominal track, having a representative value. VR less than T 1 and greater than T 2 and consisting of a number of pixels greater than the threshold T 3 .
In a step 41, the analysis module 102 determines, for each existing TBD track, whether it can transform said TBD track into a nominal track. In one embodiment, the analysis module 102 transforms the TBD track into a nominal track whenthe number of pads making up said track TBD exceeds a predefined number NB1. In one embodiment NB1 = 40.
In a step 43, the analysis module 102 determines whether TBD pads have not been assigned to an existing track. If all the TBD pads have been assigned, the analysis module 102 executes step 45 already explained. Otherwise, during a step 44, the analysis module 102 creates a new TBD track for each TBD plot that has not been assigned to an existing track. Step 44 is followed by step 45.
In one embodiment, the tracks of the second type include the TBD tracks and so-called transient tracks. It can then be considered that the second type of track comprises two subtypes. Transient tracks are intermediate tracks between TBD tracks and nominal tracks. The analysis module 102 transforms a TBD track into a transient track when the number of pads making up said TBD track exceeds a predefined number NB2
Documents
Orders
Section
Controller
Decision Date
Application Documents
#
Name
Date
1
202017021346-IntimationOfGrant29-10-2024.pdf
2024-10-29
1
202017021346-TRANSLATIOIN OF PRIOIRTY DOCUMENTS ETC. [21-05-2020(online)].pdf
2020-05-21
2
202017021346-PatentCertificate29-10-2024.pdf
2024-10-29
2
202017021346-STATEMENT OF UNDERTAKING (FORM 3) [21-05-2020(online)].pdf