IMAGE RESTORATION PROCESS

The invention relates to a method for restoring images from a sequence of images, and to a device implementing said method.

Some applications using images or image sequences require very high image quality. This is the case for example of surveillance applications where low image quality can lead to incorrect interpretations of an image and, for example, false alarms triggering. Image restoration consists in improving the quality of an image by applying various image processing techniques to it, such as noise reduction or suppression, edge enhancement, contrast enhancement, etc. .

An image of a sequence of images (ie of a video sequence) is a particular case of an image since this image is generally visually close to images that are temporally neighboring in the sequence of images. An image sequence acquisition device, such as a video camera, in fact generally has a sufficiently high image acquisition frequency for temporal correlations to remain between successive images of a sequence of images. Used wisely, these temporal correlations can allow an image to benefit from improvements made to one or more neighboring images thanks to image restoration methods. However, identifying temporal correlations that exist between images is not always an easy task. This is particularly the case when a sequence of images represents at least one moving object on a background which is itself moving, the background then being able to be considered as an object. In fact, in order to be able to benefit efficiently from the temporal correlations existing between two images, it is desirable to take into account the movements of the object and the movements of the background. Known methods of restoring images comprising moving objects include a movement analysis phase. This phase of motion analysis makes it possible to match correlated pixels between two images. An improvement made to a first pixel of a first image can then benefit a second pixel of a second image correlated with the first pixel. The phase of

(or at least one subset of pixels) of an image to be restored is associated, by a motion vector, with a pixel of a reference image with respect to which the motion is determined. These image restoration techniques are considered insufficiently reliable to enable quality image restoration. In addition, an estimate of a dense field of motion has a significant computational cost. Such a computational cost is not compatible with certain on-board or portable systems likely to implement image restoration methods, such as video cameras, digital binoculars, augmented reality glasses, etc. We therefore prefer in general, during the motion analysis phase, replace the techniques for estimating a dense field of motion by motion analysis techniques having a lower computational cost, such as global motion estimation techniques. Global motion estimation techniques are particularly effective when a hypothesis of stationarity of the scenes represented by the sequence of images is verified.

Moreover, following the motion analysis, the same image restoration method is generally applied to each pixel of the image to be restored. However, all the pixels of an image are not of equal interest. It is in fact common for an image to represent an object and a background. Obtaining a detailed visual rendering of the object is useful, for example, when the object needs to be identified precisely, while in the best case the background is only useful for placing the object in context. Methods of restoring images are known in which the movement analysis phase is followed by segmentation of the image. The segmentation of the image makes it possible to divide the image into homogeneous sub-parts according to a predefined criterion. Knowing the sub-parts constituting the image, it is possible to apply more or less efficient (and therefore more or less complex) image processing depending on the interest of each sub-section. When the segmentation follows an estimate of a dense field of motion, this segmentation can be based on the dense field obtained to divide the image into sub-parts which are homogeneous in the sense of the movement. This method of segmentation can however give approximate results, especially when the dense field of motion is noisy. press on the dense field obtained to divide the image into homogeneous sub-parts in the sense of movement. However, this segmentation method can give approximate results, especially when the dense field of motion is noisy. press on the dense field obtained to divide the image into homogeneous sub-parts in the sense of movement. However, this segmentation method can give approximate results, especially when the dense field of motion is noisy.

Other segmentation methods exist, for example segmentation methods based on active contours. However, these methods are generally intended for still images and, when applied to images of a sequence of images, they make little (or no use) of the temporal correlations existing between the images.

It is desirable to overcome these drawbacks of the state of the art.

It is in particular desirable to provide a method and a device for restoring images suitable for images comprising moving objects. It is further desirable that this method and this device be suitable for images where the background of the image is itself moving. Finally, it is desirable that said method has a low computational cost and can be implemented by a system having low computational capacities such as an on-board or portable system.

According to a first aspect of the invention, the invention relates to a method for restoring images of a sequence of images comprising, when it is applied to a first image of the sequence of images:

estimating information representative of an overall movement of a background of the first image relative to a second image;

compensating for the overall movement of the background in the second image by using said information representative of the overall movement of the background in order to obtain a registered version of the second image, called the second registered image; obtaining an outline of an object of the first image by applying a segmentation method, said segmentation method being iterative and comprising, during an iteration, a modification of an outline of the object in the first image obtained during 'a previous iteration of said segmentation method, said previous contour, so as to obtain an outline of the object in the first image, said current contour, such that a cost of the current contour is less than a cost of the previous contour, a final contour of the object being obtained when a predefined condition for stopping said segmentation process is fulfilled, the cost of a contour of the object in the first image being a sum between a first value representative of an energy internal to said contour and a second value representative of a energy external to said contour, the energy external to said contour being a function of at least one energy dependent on an overall movement of the object between the first image and the second registered image and on an energy, called contour energy, corresponding to a sum of gradient modulus values ​​calculated for pixels of a second set of pixels belonging to the current contour of the object; a representative value of the energy dependent on a global movement of thesame spatial positions as the pixels of the first set of pixels in the second image;

using the contour of the object thus obtained to estimate information representative of an overall movement of the object; and,

applying to the first image an image restoration method using the information representative of the overall movement of the background and of the overall movement of the object estimated.

Using an external energy integrating an energy dependent on a global movement of the object makes it possible to take into account the movement of the object in the segmentation of the contour of the object. We thus take advantage of the correlations existing between two images to improve the segmentation.

In one embodiment, to calculate the value representative of the energy internal to the current contour, a local first derivative and a local second derivative of the contour are calculated for pixels of a third set of pixels of the current image belonging to the current contour of the object, said value representative of the internal energy being a function of said calculated derivatives.

In one embodiment, the first, second and third sets of pixels are identical, and each set comprises at least a sub-part of the pixels of the current image belonging to the current contour of the object.

In one embodiment, during a first iteration of said method, an initial contour of the object in the current image is obtained from a final contour obtained during an application of the segmentation method to the image reference or an operator-specified contour in the reference image.

In one embodiment, during each estimation of information representative of an overall movement, information representative of the shape and position of the object is obtained, said information representative of the shape and position of the object. 'object being used to mask pixels not to be taken into account in said estimate.

In one embodiment, following the estimation of said piece of information representative of the overall movement of the object, called the first piece of information, filtering is applied to said first piece of information in order to guarantee regular variations in the movement of the object between two images. successive images of the sequence of images, said filtering comprising the following steps: determining a first matrix making it possible to estimate a movement of the object in a frame of reference centered on a barycenter of

the object in the first image and a second matrix making it possible to estimate a movement of the object in a frame of reference centered on a barycenter of the object in the second registered image; using the first and second matrices to calculate information representative of the movement of the object, called second information, from said first information; using the second information to obtain a third matrix representative of translational components of the movement of the object; using the second information and the third matrix to obtain a fourth matrix representative of components of the movement of the object other than the translation components; obtain a filtered version of the third matrix, called the third filtered matrix, by calculating a weighted sum between the third matrix and a third filtered matrix obtained during the implementation of said method on the second image; obtaining a filtered version of the fourth matrix, called the fourth filtered matrix, by calculating a weighted sum between the fourth matrix and a fourth filtered matrix obtained during the implementation of the method on the second image; and, obtaining information representative of a filtered overall movement of the object using the first and second matrices, the third filtered matrix and the fourth filtered matrix. by calculating a weighted sum between the fourth matrix and a fourth filtered matrix obtained during the implementation of the method on the second image; and, obtaining information representative of a filtered overall movement of the object using the first and second matrices, the third filtered matrix and the fourth filtered matrix. by calculating a weighted sum between the fourth matrix and a fourth filtered matrix obtained during the implementation of the method on the second image; and, obtaining information representative of a filtered overall movement of the object using the first and second matrices, the third filtered matrix and the fourth filtered matrix.

In one embodiment, the second information is calculated as follows:

where V k is the first matrix, V k – 1 is the second matrix,
the first information and the second information.

In one embodiment, the third matrix is ​​calculated as follows:

where is the third matrix and

ApproxT (X) is a translation approximation of the homographic matrix

X.

In one embodiment, the fourth matrix is ​​calculated as follows:

where is the fourth matrix.

In one embodiment, the third filtered matrix is ​​calculated as follows:

where
is the third filtered matrix,
is the third filtered matrix obtained during the implementation of said method on the second image and a is a predefined constant between “0” and “1”.

In one embodiment, the fourth filtered matrix is ​​calculated as follows:

where
is the fourth filtered matrix, I is an identity matrix and β is a predefined constant between "0" and "1".

In one embodiment, the information representative of a filtered global movement of the object is calculated as follows:

According to a second aspect of the invention, the invention relates to a device for restoring images of a sequence of images comprising:

estimation means for estimating information representative of an overall movement of a background of a first image of the sequence of images relative to a second image of the sequence of images;

movement compensation means for compensating for the overall movement of the background in the second image by using said information representative of the overall movement of the background in order to obtain a registered version of the second image, called the second image failed;

obtaining means making it possible to obtain an outline of an object in the first registered image, comprising means for modifying an outline of the object in the first image obtained during a previous iteration, called the previous outline, making it possible to obtain an outline of the object in the first image, called a current outline, such that a cost of the current outline is less than a cost of the previous outline, a final outline of the object being obtained when a predefined condition stop is met, the cost of a contour of the object in the first image being a sum between a first value representative of an energy internal to said contour and a second value representative of an energy external to said contour, the energy external to said contour being a function ofat least one energy dependent on an overall movement of the object between the first image and the second registered image and on an energy, called edge energy, corresponding to a sum of values ​​of gradient moduli calculated for pixels of a second set of pixels belonging to the contour

current of the object; a value representative of the energy dependent on an overall movement of the object between the first image and the second image being calculated as a sum of differences between values ​​representative of pixels of a first set of pixels of the first image belonging to the current contour and values ​​representative of pixels located at the same spatial positions as the pixels of the first set of pixels in the second image;

movement estimation means using the contour of the object in the first image to estimate information representative of an overall movement of the object between the first image and the second image; and,

image restoration means using the information representative of the overall movement of the background and of the overall movement of the object estimated According to a third aspect of the invention, the invention relates to a computer program, comprising instructions to implement, by a device, the method according to the first aspect, when said program is executed by a processor of said device.

According to a fourth aspect of the invention, the invention relates to storage means, storing a computer program comprising instructions for implementing, by a device, the method according to the first aspect, when said program is executed by a device. processor of said device.

The characteristics of the invention mentioned above, as well as others, will emerge more clearly on reading the following description of an exemplary embodiment, said description being given in relation to the accompanying drawings, among which:

- Fig. 1 schematically illustrates an example of a context in which the invention can be implemented;

- Fig. 2 schematically illustrates an example of an image containing an object and a background;

- Fig. 3 schematically illustrates an example of the hardware architecture of a processing module included in an image acquisition system;

- Fig. 4 schematically illustrates a method for restoring images according to the invention;

- Fig. 5 schematically illustrates an image segmentation method according to the invention;

- Fig. 6 schematically illustrates a method for calculating a cost of an outline of an object included in the segmentation method according to the invention;

- Fig. 7 schematically illustrates a method for estimating overall movement; and,

- Fig. 8 schematically illustrates a motion filtering method. The invention is described below in a context where a display system comprises an image acquisition device, a processing module and an image display device. The invention can however be implemented in a context where the image acquisition device, the processing module and the display device are separate and geographically distant. In this case, the image acquisition device, the processing module and the image display device comprise communication means for communicating with one another.

Furthermore, the method according to the invention is based on a segmentation method based on active contours based on contours. We subsequently show that other types of segmentation methods based on active contours can be used such as for example, segmentation methods based on region-based active contours, segmentation methods based on implicit active contours based on sets of levels ("level sets ”in Anglo-Saxon terminology), ...

In addition, the images used in the context of the invention are essentially monochrome images where each pixel of an image has only one component. The invention can however be applied to multi-component images in which each pixel of an image has a plurality of components.

Fig. 1 schematically illustrates an example of a context in which the invention can be implemented.

In Fig. 1, a scene 1 is observed by a viewing system 5 along an optical field 3. Scene 1 comprises an object 4. The viewing system 5 comprises an image acquisition device 51, a processing module 52 capable of implement an image restoration method and a segmentation method according to the invention and an image display device 53. The image acquisition device 51 comprises an optical assembly and an image sensor such as for example a CCD sensor (“Charge-Coupled Device” in English terminology, charge transfer device) or a CMOS sensor (“Complementarity metal-oxide-semiconductor”, in English terminology, metal-oxide semiconductor complementary). The image sensor provides a sequence ofthe processing module 52, the processing module 52 supplies improved images to the image display device 53. The image display device 53 is for example a screen.

In one embodiment, the images supplied by the image acquisition device 51 are monochrome images.

In one embodiment, the images supplied by the image display device 51 are multi-component images.

Fig. 2 schematically illustrates an example of an image containing an object and a background.

The example described in relation to FIG. 2 represents an image 2 supplied by the viewing system 5. In image 2, we find the object 4 (here a vehicle) moving on a background (here a landscape). One can imagine that the display system 5 is mobile so that the background has movement.

Fig. 3 schematically illustrates an example of the hardware architecture of the processing module 52 included in the display system 5.

According to the example of hardware architecture shown in FIG. 3, the processing module 52 then comprises, connected by a communication bus 520: a processor or CPU (“Central Processing Unit” in English) 521; a random access memory RAM (“Random Access Memory” in English) 522; a read only memory (ROM) 523; a storage unit such as a hard disk or a storage medium reader, such as an SD (“Secure Digital”) card reader 524; at least one communication interface 525 allowing the processing module 52 to communicate with the image acquisition device 51 and / or the image display device 53.

Dans un mode de réalisation dans lequel le dispositif d'acquisition d'images 51, le module de traitement 52 et dispositif de visualisation 53 sont séparés et distants, le dispositif d'acquisition d'images 51 et le dispositif de visualisation 53 comprennent aussi une interface de communication pouvant communiquer avec l'interface de communication 525 par l'intermédiaire d'un réseau tel qu'un réseau sans fils.

Le processeur 521 est capable d'exécuter des instructions chargées dans la RAM

522 from ROM 523, an external memory (not shown), a storage medium (such as an SD card), or a communications network. When the processing module 52 is powered on, the processor 521 is able to read instructions from the RAM 522 and execute them. These instructions form a program

computer causing the implementation, by the processor 521, of all or part of the method described below in relation to FIGS. 4 to 8.

The method described below in relation to FIGS. 4 to 8 can be implemented in software form by executing a set of instructions by a programmable machine, for example a DSP (“Digital Signal Processor”), a microcontroller or a GPU (graphics processor, “Graphics Processing Unit”). "In English terminology), or be implemented in hardware form by a machine or a dedicated component, for example an FPGA (" Field-Programmable Gâte Array "in English) or an ASIC (" Application-Specific Integrated Circuit "in English ).

Fig. 4 schematically illustrates a method for restoring images according to the invention.

The method described in relation to FIG. 4 is an iterative process, implemented by the processing module 52 on each image of a sequence of images supplied by the image acquisition device 51, except the first image of the sequence of images. Hereinafter, we call current image, an image being processed by the processing module 52 and we denote the current image I k , where k represents an index of an image. The index k indicates that the image I k appears in the sequence of images at an instant T 0 + k. τ. T 0 corresponds to the start of the sequence of images (and therefore to the first image of the sequence) and Let I k – 1 , an image, called image

preceding, immediately preceding the current image I k in the sequence of images supplied by the image acquisition device 51.

In a step 41, the processing module 52 estimates information representative of an overall movement of a background of the current image I k (or movement of the background) with respect to the previous image I k – 1 . The previous image I k – 1 is then a reference image for the current image I kfor estimating information representative of bottom movement. This step is implemented by a global motion estimation method. An estimate of global movement assumes that a set of pixels of the same image is animated by the same movement. This movement can be simple, such as translational or rotational movement, or complex represented, for example, by affine transformation or homography. An homography is an eight-parameter projective coordinate transformation. In one embodiment, the processing module considers that the movement of the background between two successive images of the sequence of images is represented by a homography. Let (x, y) be the coordinates of a pixel

belonging to the background in the current image I k and (x ', y') coordinates of the same pixel belonging to the background in the previous image I k – 1 . The estimate of

global movement implemented during step 41 consists in determining the eight parameters of a homography making it possible to transform the coordinates (x ', y') of each pixel in the previous image I k – 1 into coordinates (x, y) of a pixel

in the current image I k . By determining the eight parameters of the homography, one determines a piece of information representative of a global movement of the background between the previous image I k – 1 and the current image I k . We note
the homography representative of the global movement of the background between the previous image I k – 1 and the image I k .

In a step 42, the processing module 52 compensates for the movement of the background in the previous image I k – 1 in order to obtain a registered previous image. To
do this , the processing module 52 applies the homography found during the step

41 to the set of pixels of the previous image I k – 1 .

In a step 43, the processing module 52 obtains a contour C of the object 4 by applying a segmentation method to the current image I k . We describe below in relation to Figs. 5 and 6 a segmentation method according to the invention.

In a step 44, the processing module 52 estimates information representative of an overall movement of the object 4, between the current image and a previous image. As during step 41, the processing module 52 considers that the overall movement of the object 4 is represented by a homography, denoted
In one embodiment, the homography representative of the movement

global of object 4 is obtained by using the current image I k and the previous image I k – 1 and by taking into account the bottom movement
measured during step 41. Let (x, y) be the coordinates of a pixel belonging to object 4 in the current image

I k and (x ', y') coordinates of the same pixel belonging to object 4 in the image

previous I k – 1 . The overall motion estimation implemented during step 44 comprises a determination of the eight parameters of a homography
making it possible to transform the coordinates (x, y) of each pixel in the
current image I k into coordinates (x ' , y ') of the pixel in the previous image I k – 1 . We

can write:

The representative homography is then obtained as follows:

In one embodiment, the homography representative of

global movement of the object 4 is measured between a registered image and a non-registered image, which makes it possible not to involve the homography
For example, the homography is measured between the previous registered image and the

current image I k .

In a step 45, the processing module 52 applies an image restoration method to the current image I k . In one embodiment, the image restoration method applied uses the information representative of the overall movement of the background and the information representative of the overall movement of the object 4 estimated during steps 41 and 44 to match the pixels of the background. 'current image I k and the pixels of the previous image I k – 1 . Let P k be a pixel of the current image I k and P k – 1 a pixel of the previous image I k – 1 matched with the pixel P k using the homographies The pixel Pk (respectively

pixel P k – 1 ) has a non-zero positive integer N c of components
(respectively i
∈ [1; N C ]. In this embodiment, the value
of each component of each pixel P k of the image I k is replaced by a weighted sum calculated as follows:

where W k and W k – 1 are predefined weight values. For example, the weights W k and W k – 1 can be such that W k ≤ W k – 1 .

In one embodiment, the image restoration method applied uses an image window comprising a number N I of images preceding the current image I k . The pixels of each of the images are matched by using the information representative of the movement of the background and the movement of the object 4 obtained for each of the images of the image window.

In this embodiment, the value of each component of each pixel

P k of the image I k is replaced by a weighted sum calculated as follows:

where each Wj is a predefined weighting value. It is noted that the restoration method using a weighting involving two images is a special case of the restoration method based on an image window where N I = 2.

In one embodiment, only the pixels of the current image I k belonging to the object 4 are restored.

In one embodiment, several restoration methods are used depending on the object to which each pixel of the image I k belongs . A first restoration method is applied to the pixels belonging to the background of the current image I k and a second restoration method is applied to the pixels belonging to the object 4 in the current image I k . The first restoration method is for example the restoration method using an image window in which the image window comprises two images. The second restoration method is for example the restoration method using an image window in which the image window comprises five images.

Fig. 7 schematically illustrates a method for estimating overall movement.

The method of FIG. 7 is applied by the processing module 52 during step 41 to determine the overall movement of the background (ie the homography and during
step 44 to determine the overall movement of the object 4 (ie the homography

During step 41, only the movement of the pixels corresponding to the background is sought. The movement of pixels corresponding to object 4 must not be taken into account.

In a step 410, the processing module obtains a position and a shape of the object 4 in the image I k .

In one embodiment, the shape and the position of the object 4 in the image I k are given by an ordered list of pixels, called control points, belonging to a contour C of the object 4. The ordered list of control points can include all the pixels belonging to the contour C of the object 4 or a subset of pixels of the contour C making it possible to obtain a good approximation of the contour C. A browse of the ordered list of control points allows get the contour C.

In one embodiment, during step 410, the processing module 52 makes an assumption of weak movement of the object 4 between two successive images of the sequence of images. As said above, the method described in relation to FIG. 4 isiterative so that when the current image I k is being processed by this method, the previous image I k – 1 has already been processed by this method. The position and shape of object 4 in the image I k – 1 are therefore known. Based on the hypothesis of weak movement, the processing module 52 considers that the shape and the position of the object 4 in the image I k are identical to the shape and the position of the object 4 in the image I k – 1 . The processing module 52 therefore reuses the ordered list of control points of the image I k – 1 to define the contour C in the image I k .

The first image of the sequence I k = 0 is a special case since this image is not preceded by any other image. In one embodiment, the position and the shape of the object 4 in the first image I 0 of the sequence of images are given by an operator. To do this, the operator can crop the object 4 using a pointing device, such as a mouse, on the display device 53, which in this case is a touch screen. The background of the first image is considered to have zero motion. Therefore the movement of the background does not have to be compensated for for the first frame of the sequence. The method described in relation to FIG. 4 is not applied to the first I 0 image of the sequence,is used in this method to determine the movement of the background and the movement of the object 4 between this image I 0 and the image I k = 1 which follows it in the sequence of images.

In a step 411, the position and the shape of the object in the current image I k being known, the processing module masks each pixel of the current image I k belonging to the object 4, ie the processing module masks each pixel belonging to the contour or internal to the contour of the object 4. In one embodiment, the processing module 52 associates each pixel of the current image I k with a first mask value when said pixel is masked and with a second mask value when said pixel is not masked. The first mask value is for example the value “1” and the second mask value is for example the value “0”.

In a step 412, the processing module 52 estimates the overall movement of the background between the current image I k and the previous image I k – 1 (ie the processing module 52 estimates the homography During this estimate, only the pixels of the image

I k that are not masked (ie the pixels associated with the second mask value) are taken into account. In addition, only the pixels of the previous image I k – 1 not masked are taken into account, using the mask obtained during the application of the method described in relation to FIG. 4 to the previous image I k – 1 . In one embodiment, the determination of the eight parameters of the homography
uses the method

of projective fit ("projective fit" in English terminology) or the method of projective flow ("projective flow" in English terminology) described in the article "Video orbit of the projective group: a simple approach to estimation without parameter characteristics ("video Orbits of the Projective Group: a simple approach to featureless estimation of parameters" in English terminology) ", Steve Mann and Rosaling W. Picard, IEEE Tr. On Image Processing, Vol. 6, no. 9, September 1997.

During step 44, only the movement of the pixels corresponding to the object 4 is sought. The movement of the pixels corresponding to the background should not be taken into account.

During step 44, the processing module implements step 410 of obtaining the position and the shape of the object 4. The position and the shape of the object 4 in the current image I k are obtained thanks to the result of step 43

During step 411, the processing module 52 masks each pixel of the current image I k . belonging to the background, ie not belonging to the object 4.

During step 412, the processing module 52 estimates the movement of the object between the current image I k and the previous image I k – 1 (ie the processing module estimates the homography
and then deduces the homography During this
estimation, only the pixels of the current image I k and of the previous image I k – 1 not masked are taken into account. Again, the determination of the eight parameters of the homography
uses the adjustment method projective or the method of projective flows.

In one embodiment, during the implementation of step 410 during step 41, the processing module 52 makes an assumption of continuous movement of the object 4 in the sequence of images. The continuous motion assumption implies that the motion of object 4 between the current frame I k and the previous frame I k – 1 is the same as the movement of object 4 between the previous frame I k– 1 and an image I k – 2 preceding the previous image I k – 1 . The method described in relation to FIG. 4 being iterative, during the processing of the current image I k , the movement of the object 4 between the previous image I k – 1 and theis known. Moreover, the position and the shape of object 4 in the previous image I k – 1 are also known. The position and the shape of the object 4 in the current image I k can therefore be found using a homography representing the movement of

object 4 between the previous frame I k – 1 and the previous frame I k – 2 . Homography

is a combination of a homography representative of the

movement of the background between the previous image I k – 1 and the previous image I k – 2 and a homography
representative of the movement of the object 4 between the previous image I k –1 and the previous image I k – 2 . Homography is applied to

object 4 in the previous image I k – 1 . More precisely the homography is
applied to the control points of the ordered list of control points representing the contour of the object 4 in the previous image I k – 1 in order to obtain the ordered list of control points representing the contour of object 4 in image I k . The assumption of continuous movement is applicable during the implementation of step 410 during step 44.

In one embodiment, to take into account that the assumptions of weak movement and continuous movement only make it possible to obtain an approximation of the shape and the position of the object 4 in the current image I k , a dilation is applied to the contour of the object 4 in the current image I k . The expansion is obtained, for example, by using a mathematical morphology method.

In other embodiments, other known methods of estimating the parameters of a homography can be used.

In other embodiments, the processing module considers that the movement between two successive images of the sequence of images is represented by other models of movement such as a translation, a rotation, an affine transformation or a transformation. bilinear.

In one embodiment, prior to each estimation of overall movement (of the background or of the object 4), each image involved in the estimation of overall movement is interpolated to a half, a quarter or an eighth of a pixel. In this way, the accuracy of the overall motion estimation is improved.

The projective adjustment method (respectively the projective flow method) consists in finding among a set of parameters of a motion model (here the 8 parameters of a homography), the parameters of the motion model minimizing a representative metric of 'an error between an actual movement of an object in an image and a movement of the object represented by the movement model. In the projective adjustment method (respectively the projective flow method), every possible combination of parameters of the motion model is tested. Such an exhaustive method of finding the parameters of the motion model can have a significant computational cost. It is possible to reduce the computational cost ofprojective adjustment method (respectively the projective flow method) using, for example, a gradient descent algorithm rather than an exhaustive search. However, a known problem with gradient descent methods is that, when the metric to be minimized has several local minimums, the gradient descent method can converge to a local minimum which is not a global minimum, ie which is not not the minimum value that the metric can take. One method making it possible to ensure rapid convergence towards the global minimum of the metric consists in initializing the gradient descent method with a value close to the desired global minimum. In one embodiment, the exhaustive search for the parameters of the motion model of the method of

During the implementation of step 41 on the image I k , the gradient descent method is initialized to a value
representative of the movement of the background found for the image I k – 1 . More precisely, the eight parameters of the homography
representative of the bottom movement between the previous image I k – 1 and the previous image I k – 2 are used to initialize the eight parameters of the homography representative of the bottom movement between the current image I k and the image

previous I k – 1 in the gradient descent method.

Likewise, during the implementation of step 44 on the image I k , the gradient descent method is initialized to a value.
More precisely, the eight parameters of the homography are used to initialize

the eight parameters of the homography
representative of the movement of object 4 measured between the current image I k and the previous image I k – 1 in the gradient descent method. The homography
is then deduced from the homography and the homography

In one embodiment, following the estimation of the information representative of the movement of the object 4 between the current image I k and the previous image adjusted I k – 1 (ie following the estimation of the eight parameters of the homography, the information
representative of the estimated movement is filtered in order to guarantee regular variations in the movement of the object between two successive images of the sequence of images. The method described in relation to Fig. 4 is particularly effective when 'an inertial motion hypothesis of the object 4 is verified. When the estimated motion information of the object 4 is too variable, it is preferable to correct theseinformation so that they approximate motion information compatible with the inertial motion hypothesis. Fig. 8 describes a motion filtering method for correcting motion information. The method described in relation to FIG. 4 then no longer uses the estimated movement information but the corrected estimated movement information.

Fig. 8 schematically illustrates a motion filtering method.

The method described in relation to FIG. 8 makes it possible to guarantee regular variations in the movement of the object 4 between two successive images of the sequence of images.

In a step 800, the processing module 52 determines a passage matrix V k (respectively a passage matrix V k – 1 ) making it possible to estimate a movement of the object in a frame of reference centered on a barycenter of the object 4 in the current image I k (respectively in the previous image registered

where
(respectively
are coordinates of the barycenter of object 4 in the current image I k (respectively in the previous image registered

In a step 801, the processing module 52 calculates information
representative of the movement of the object 4 in the frame of reference centered on the barycenter of the object 4.

The coefficients of the matrix noted “. »Are coefficients which are not

not used subsequently due to an approximation implemented in a following step 802.

In step 802, the processing module 52 obtains a matrix H £ representative of components of translation of the movement of the object 4 between the current image I k and the previous image registered (represented by the homography
in the reference frame centered on the barycenter of object 4 as follows:

where T x and T y are parameters of a translation and ApproxT (X) is a translation approximation of the homographic matrix X.

In a step 803, the processing module 52 obtains a matrix
representative of components of the movement of the object 4 between the current image I k and the previous image registered other than the components of translation in the manner

next :

The components of the movement of the object 4 other than the translation components can for example be components of rotation, zoom, etc.

In a step 804, the processing module 52 filters the translation components of the movement of the object 4 between the current image I k and the previous image registered as follows:

where
is a matrix representative of the filtered translation components of the movement of the object 4 between the previous image I k – 1 and the previous readjusted image and α is a predefined constant between “0” and “1”. In one embodiment, α = 0.8.

In a step 805, the processing module 52 filters the components of the movement of the object 4 between the current image I k and the previous registered image
other than the translation components as follows:

is a matrix representative of the filtered components of the movement of the object 4 other than the translation components, I is an identity matrix of size 3 x 3 and β is a predefined constant between “0” and “1”. In one embodiment β = 0.5.

In a step 806, the processing module 52 determines information representative of the filtered overall movement of the object 4 in the manner

next :

In the embodiment in which the estimated movement is filtered in order to guarantee small variations in the movement of the object between two successive images of the sequence of images, the filtered overall movement of the object 4 represented by

the filtered homography
replaces the overall movement of the object 4 represented by the homography
during the restoration of the current image I k .

Fig. 5 schematically illustrates an image segmentation method according to the invention.

The method described in relation to FIG. 5 is implemented during step 43.

In one embodiment, the image segmentation method implemented during step 43 is a segmentation method based on active contours. One principle of segmentation methods based on active contours is to define an initial contour in the vicinity of an object and then to iteratively modify this contour so that it matches the shape of the object as well as possible. At each iteration, the contour of the object obtained during a previous iteration, called the previous contour, is modified so as to obtain an contour of the object, called the current contour, such that a cost of the current contour is less than a cost of the previous contour. In segmentation methods based on active contours, the cost of a contour is a function of an internal energy and an external energy of the contour. We subsequently give examples of methods of calculating values ​​representative of an internal energy and an external energy of a contour. A final contour of the object is obtained when a predefined condition for stopping the active contour based segmentation process is met. A stop condition can for example be a maximum number of iterations or obtaining a difference between two contour costs obtained in two successive iterations less than a predefined threshold. It is noted that the closer the initial contour is to the real contour of the object, the more quickly the segmentation method based on active contours converges to a contour close to the real contour of the object. A judicious choice of a position and a shape of the initial contour therefore makes it possible to improve the performance of the segmentation process based on active contours. In one embodiment, the active contour based segmentation method is contour based.

In a step 431, the processing module 52 obtains an initial contour C of the object 4 in the current image I k .

In one embodiment, during step 431, the processing module makes the assumption of weak movement of the object 4 between the current image I k and the previous image I k – 1 . In this case, as during the implementation of step 411 during step 44, the processing module 52 reuses the ordered list of control points determined during the implementation of the method described in relation to the Fig. 4 on

the previous image I k – 1 to obtain the initial contour C of the object in the current image I k .

In a step 432, the processing module 52 calculates a cost of the current contour C by applying a method which we describe below in relation to FIG. 6. During the first iteration of the segmentation method based on active contours described in relation to FIG. 5, the current contour C is the initial contour C.

In a step 433, the processing module 52 checks whether a condition for stopping the segmentation method based on active contours is fulfilled. In one embodiment, said iterative method stops when a number of iterations of the segmentation method based on active contours reaches a maximum number of iterations.

When the stop condition is fulfilled, the segmentation method based on active contours ends during step 434 and the processing module 52 implements step 44 already explained.

When the stop condition is not fulfilled, the processing module 52 implements a step 435. During the step 435, the processing module 52 implements a procedure for refining the contour C of the object. 4 obtained during the previous iteration of the active contour based segmentation process. During step 435, the processing module 52 modifies the contour C of the object 4 obtained during the previous iteration of the segmentation method based on the active contour, called the previous contour, so as to obtain a contour C of the object, said current contour, such that a cost of the current contour is less than a cost of the previous contour. The modification of contour C uses, for example, a method described in the article "snakes: Active Contour Models" Michael Kass, Andrew Witkin, Demetri Terzopoulos,

Step 435 is followed by step 432.

In one embodiment, during step 431, the processing module makes the assumption of continuous movement of the object 4 between the current image I k and the previous image I k – 1 . In this case, the processing module 52 moves the control points of the ordered list of control points to obtain the initial contour C of the object 4 in the current image l k . These control points, determined during the implementation of the method described in relation with FIG. 4 on the previous image I k – 1 , are displaced by the movement of the object 4 represented by the homography

In one embodiment, the control points of the ordered list of control points are moved from the filtered movement of the object 4 represented by the homography to obtain the initial C contour of object 4 in the image

current I k .

Fig. 6 schematically illustrates a method of calculating a cost of an outline of an object included in the segmentation method according to the invention.

The method described in relation to FIG. 6 is implemented during step 432. In a step 4321, the processing module 52 calculates an internal energy E int of the contour C as follows:

where a and b are predefined constants equal for example to the value "0.01", N is a number of control points in the list of control points representing the curve C, PC i is the z ' -th control point of the list of control points representing the curve C in the current image I k , is a local first derivative of the curve

C in the current image I k calculated at the level of the control point PC i, and is a
local second derivative of the curve C in the current image I k calculated at the level of the control point PC i .

In a step 4322, the processing module 52 calculates an external energy E ext of the contour C as follows:

where W edge and W mvt are predefined constants, for example equal to the value “1”. E edge is an energy, called edge energy, calculated on a gradient modulus image obtained from the current image I k :

where is an image gradient modulus value

corresponding to the position of the control point PC i .

It should be noted that different methods of calculating a gradient modulus image are applicable here. To obtain the image of gradient modulus, we can for
example apply to each pixel of the image I k :

• a linear combination of adjacent pixels of said pixel, each adjacent pixel being weighted by a weight, the sum of said weights being equal to zero, then calculating the amplitude (ie the modulus) of this linear combination;

• a Sobel filter;

• a Canny filter;

• ...

In one embodiment, the image is not calculated, and the values ​​of

gradient modulus used in the calculation of the contour energy E edge are calculated only at the positions of the N control points PC i .

E mvt is an energy dependent on the movement of object 4 between the current image the previous registered image

where I k (PC [ ) is a value of a pixel of the current image I k corresponding to the control point PC i and
is a value of a pixel of the image
located at the same position as the pixel of l 'image I k corresponding to the control point PC i .

In a step 4323, the processing module 52 calculates the cost J of the current contour C as follows:

J = E ext + E int

It can therefore be seen that the movement of the object 4 is taken into account in the segmentation method according to the invention, which makes it possible to obtain better segmentation of the object. Minimizing the cost J makes it possible to maximize E mvt and E edge on the control points of the contour, in order to favor areas with strong spatial and / or temporal gradients.

The principle of the invention remains the same in the case of a use of another type of segmentation methods based on active contours than the segmentation methods based on active contours based on contour. Each segmentation method based on active contours comprises an estimate of an external energy E ext . However, since the active contour-based segmentation methods are suitable for still images, they do not take into account the movements in a sequence of images during the segmentation. The invention makes it possible to take these movements into account by integrating an energy representative of the movement in the estimation of the external energy E ext . This principle applies to external energies E ext computed within the framework of active contour based region-based segmentation methods and active contour based segmentation methods based on sets of levels.

So far we have considered images comprising only one object. The invention is applicable when the images of the sequence of images comprise a plurality of objects. During steps 41 and 42, each object is masked during the estimationand compensating for background movement in an image. Steps 43, 44 and 45 are implemented independently on each object.

Furthermore, until then, we have considered that object 4 was rigid and that therefore the apparent shape of the object was approximately constant. In a real case, depending on the movements of the object and / or the camera, the object can be seen from different angles of view which can cause deformations in the apparent shape of the object. In one embodiment, when a variation in the shape of the object over a plurality of successive images of the sequence of images exceeds a predefined threshold, the processing module 52 considers that the object appearing in the images has exchange. In this case, when the processing module 52 detects a change of object, it considers that a new sequence of images has started and invites the operator to crop the object again. In another embodiment, the processing module 52 applies the segmentation method described in relation to FIG. 5 regardless of the variations in the shape of the object, without requiring the intervention of an operator.

In one embodiment, when the images supplied by the image acquisition device are multi-component images, the processing module 52 applies the restoration method described in relation to FIG. 4 to each component independently. Each component can then be displayed independently or in combination with one or more other components on the display device 53.

In one embodiment, when the images supplied by the image acquisition device are multi-component images, the processing module 52 applies the restoration method described in relation to FIG. 4 to at least one of the components, or to at least one component calculated from the components available in the images. Only the restoration step 45 is applied to each component independently using the information representative of the movement of the bottom and of the movement of the object 4 obtained during the previous steps. Each component can then be displayed independently or in combination with one or more other components on the display device 53. For example, when the multi-component images comprise a luminance component and two chrominance components, the restoration method described in relation to FIG. 4 is applied only to the luminance component, the restoration step 45 being applied to the three components

CLAIMS

1) Method for restoring images of a sequence of images, characterized in that the method comprises, when it is applied to a first image of the sequence of images:

estimating (41) information representative of an overall movement of a background of the first image relative to a second image;

compensating (42) for the overall movement of the background in the second image using said information representative of the overall movement of the background in order to obtain a registered version of the second image, called the second registered image; obtain (43) an outline of an object of the first image by applying a segmentation process,

said segmentation method being iterative and comprising, during an iteration, a modification (435) of a contour of the object in the first image obtained during a previous iteration of said segmentation method, said previous contour, so in obtaining an outline of the object in the first image, called the current outline, such that a cost of the current outline is less than a cost of the previous outline, a final outline of the object being obtained when a predefined condition of stopping said segmentation process is fulfilled (433),

the cost of an outline of the object in the first image being a sum

(4323) between a first value representative of an energy internal to said contour and a second value representative of an energy external to said contour, the energy external to said contour being a function of at least one energy dependent on an overall movement of the object between the first image and the second readjusted image and of an energy, called edge energy, corresponding to a sum of gradient modulus values ​​calculated for pixels of a second set of pixels belonging to the current contour of the object;

a value representative of the energy dependent on an overall movement of the object between the first image and the second image being calculated as a sum of differences between values ​​representative of pixels of a first set of pixels of the first image belonging to the current contour and values ​​representative of pixels located at the same spatial positions as the pixels of the first set of pixels in the second image;

using the contour of the object thus obtained to estimate (44) information representative of an overall movement of the object; and,

applying (45) to the first image an image restoration method making it possible to replace, for at least each pixel of the first image corresponding to the object, each component of said pixel by a component calculated using a component of a pixel of at least the second image matched with said pixel of the first image using information representative of the overall movement of the background and the overall movement of the object estimated.

2) Method according to claim 1, characterized in that to calculate the value representative of the energy internal to the current contour, a first local derivative and a second local derivative of the contour are calculated for pixels of a third set of pixels of the current image belonging to the current contour of the object, said value representative of the internal energy being a function of said calculated derivatives.

3) Method according to claim 2, characterized in that the first, second and third sets of pixels are identical, and each set comprises at least a sub-part of the pixels of the current image belonging to the current contour of the object.

4) Method according to any one of the preceding claims, characterized in that during a first iteration of said method, an initial contour of the object in the current image is obtained (431) from a final contour obtained when applying the segmentation method to the reference image or to an operator-specified contour in the reference image.

5) Method according to any one of the preceding claims, characterized in that during each estimation of information representative of an overall movement, information representative of the shape and position of the object is obtained, said information representative of the shape and position of the object being used to mask pixels not to be taken into account in said estimate.

6) Method according to claim 5, characterized in that following the estimation of said information representative of the overall movement of the object, said first information, a filtering is applied to said first information in order to guarantee regular variations of the movement of the object. the object between two successive images of the sequence of images, said filtering comprising the following steps:

determining (800) a first matrix making it possible to estimate a movement of the object in a frame of reference centered on a barycenter of the object in the first registered image and a second matrix making it possible to estimate a movement of the object in a frame of reference centered on a barycenter of the object in the second registered image;

using the first and second matrices to calculate (801) information representative of the movement of the object in said frame of reference, called second information, from said first information;

using the second information to obtain (802) a third matrix representative of translational components of the movement of the object;

using the second information and the third matrix to obtain (803) a fourth matrix representative of components of the movement of the object other than the translational components;

obtaining (804) a filtered version of the third matrix, called the third current filtered matrix, by calculating a weighted sum between the third matrix and a previous third filtered matrix obtained during the implementation of said method on the second image;

obtaining (805) a filtered version of the fourth matrix, called the fourth current filtered matrix, by calculating a weighted sum between the fourth matrix and a previous fourth filtered matrix obtained during the implementation of the method on the second image; and,

obtaining information representative of a filtered overall movement of the object using the first and second matrices, the third current filtered matrix and the fourth current filtered matrix.

7) Method according to claim 6, characterized in that the second information is calculated as follows:

where V k is the first matrix, V k – 1 is the second matrix, the first
information and the second information.

8) Method according to claim 7, characterized in that the third matrix is ​​calculated as follows:

where is the third matrix and ApproxT (X) is an approximation in

translation of a homographic matrix X.

9) Method according to claim 8, characterized in that the fourth matrix is ​​calculated as follows:

where
is the fourth matrix.

10) Method according to claim 9, characterized in that the third current filtered matrix is ​​calculated as follows:

where
is the third current filtered matrix,
is the third previous filtered matrix obtained during the implementation of said method on the second image and a is a predefined constant between “0” and “1”.

11) Method according to claim 10, characterized in that the fourth current filtered matrix is ​​calculated as follows:

where
is the fourth current filtered matrix, I is an identity matrix and β is a predefined constant between "0" and "1".

12) Method according to claims 10 and 11, characterized in that the information representative of a filtered global movement of the object is calculated as follows:

13) Device for restoring images of a sequence of images, characterized in that the device comprises when it is applied to a first image of the sequence of images:

estimation means for estimating (41) information representative of an overall movement of a background of the first image relative to a second image;

motion compensation means for compensating (42) for the overall movement of the background in the second image using said information representative of the overall movement of the background to obtain a registered version of the second image, said second registered image;

means for obtaining contours for obtaining (43) an outline of an object of the first image by using segmentation means,

said segmentation means comprising:

an overall movement of the object between the first image and the second registered image and of an energy, called the edge energy, corresponding to a sum of gradient modulus values ​​calculated for pixels of a second set of pixels belonging to the current contour of the object; a value representative of the energy dependent on an overall movement of the object between the first image and the second image being calculated as a sum of differences between values ​​representative of pixels of a first set of pixels of the first image belonging to the current contour and values ​​representative of pixels located at the same spatial positions as the pixels of the first set of pixels in the second image; object between the first image and the second registered image and of an energy, called edge energy, corresponding to a sum of gradient modulus values ​​calculated for pixels of a second set of pixels belonging to the current contour of the object ; a value representative of the energy dependent on an overall movement of the object between the first image and the second image being calculated as a sum of differences between values ​​representative of pixels of a first set of pixels of the first image belonging to the current contour and values ​​representative of pixels located at the same spatial positions as the pixels of the first set of pixels in the second image; object between the first image and the second registered image and of an energy, called edge energy, corresponding to a sum of gradient modulus values ​​calculated for pixels of a second set of pixels belonging to the current contour of the object ; a value representative of the energy dependent on an overall movement of the object between the first image and the second image being calculated as a sum of differences between values ​​representative of pixels of a first set of pixels of the first image belonging to the current contour and values ​​representative of pixels located at the same spatial positions as the pixels of the first set of pixels in the second image; corresponding to a sum of values ​​of gradient moduli calculated for pixels of a second set of pixels belonging to the current contour of the object; a value representative of the energy dependent on an overall movement of the object between the first image and the second image being calculated as a sum of differences between values ​​representative of pixels of a first set of pixels of the first image belonging to the current contour and values ​​representative of pixels located at the same spatial positions as the pixels of the first set of pixels in the second image; corresponding to a sum of gradient modulus values ​​calculated for pixels of a second set of pixels belonging to the current contour of the object; a value representative of the energy dependent on an overall movement of the object between the first image and the second image being calculated as a sum of differences between values ​​representative of pixels of a first set of pixels of the first image belonging to the current contour and values ​​representative of pixels located at the same spatial positions as the pixels of the first set of pixels in the second image;

movement estimation means using the contour of the object thus obtained to estimate (44) information representative of an overall movement of the object; and,

means for applying an image restoration method for applying (45) to the first image an image restoration method making it possible to

replace, for at least each pixel of the first image corresponding to the object, each component of said pixel by a component calculated using a component of one pixel of at least the second image matched with said pixel of the first image using information representative of the overall movement of the background and the overall movement of the object estimated.

14) Computer program, characterized in that it comprises instructions for implementing, by a device (52), the method according to any one of claims 1 to 12, when said program is executed by a processor ( 521) of said device (52).

15) Storage means, characterized in that they store a computer program comprising instructions for implementing, by a device (52), the method according to any one of claims 1 to 12, when said program is executed by a processor (521) of said device (52).