Enhancing Motion Pictures With Accurate Motion Information
Abstract:
Methods and systems for digitally enhancing the characteristics of an image sequence including video and motion picture images are provided. Highly accurate motion information represented by trajectories are generated through analyzing available image frames in the image sequence. Some embodiments of the present invention relate to generating multiple layer trajectories from an image sequence. Certain aspects may be applicable to the enhancement of three-dimensional (3D) image sequences including 3D video and 3D motion pictures.
Get Free WhatsApp Updates!
Notices, Deadlines & Correspondence
ENHANCING MOTION PICTURES WITH ACCURATE MOTION INFORMATION
Cross-Reference to Related Applications
[0001 ] This claims priority to U.S. Provisional Application Serial No. 61/804,814, titled "Methods and Systems for Enhancing Motion Pictures with Accurate Motion Information," and filed March 25, 2013, the entirety of which is incorporated herein by reference.
Technical Field
[0002] This disclosure relates generally to the field of digitally enhancing the characteristics of image content and, more particularly, the characteristics of video and motion picture content for improved visual display quality.
Background
[0003] For real-world images, particularly in videos and motion pictures, certain image content may appear dissimilar in different image frames due to many complex factors, such as the presence of noise, lighting change, lens-focus change, object motion, object rotation, occlusion, etc. As a result, an image enhancement method that is based on tracking similar appearances of image content over multiple image frames, such as a conventional motion tracking algorithm, may not be able to follow the motion of an image content accurately and, in some cases, may even fail to recognize the image content in some image frames due to different appearances.
Summary
[0004] In one example, a method for enhancing an image sequence is provided. The image sequence is received with image frames. A decomposition feature is determined for the image frames. Feature connections are computed for image pixels based on the decomposition feature. At least one anchor trajectory is calculated based on at least one selected feature connection among the feature connections. A relationship of the image pixel with the at least one anchor trajectory is determined. A new trajectory is created for an image pixel of the decomposed image frames based on the at least one anchor trajectory and the relationship of the image pixel with the at least one anchor trajectory. At least one trajectory feature is defined for the new trajectory and whether the at least one trajectory feature needs to be modified at a new trajectory intersection is determined. The new trajectory intersection is refined and a final trajectory is produced. An enhanced image pixel is created by modifying a pixel image feature based on the final trajectory. An enhanced image sequence is created based on the enhanced image pixel. The enhanced image sequence is outputted.
[0005] In another example, a method for enhancing an image sequence is provided. The image sequence with image frames is received. The image sequence is decomposed into at least two layers that form decomposed image frames. A decomposition feature is determined for the image frames in a layer. Feature connections are computed for image pixels in the layer based on the decomposition feature. At least one anchor trajectory is calculated for the layer based on at least one selected feature connection. A relationship of an image pixel of the image frames with the at least one anchor trajectory in each of the at least two layers is determined. A new trajectory is created for the image pixel of the decomposed image frames based on the at least one anchor trajectory and the relationship of the image pixel with the at least one anchor trajectory from the at least two layers. At least one trajectory feature is defined for the new trajectory in the layer and whether the at least one trajectory feature needs to be modified at a new trajectory intersection in the layer is determined. The new trajectory intersection is refined and a final trajectory is produced. An enhanced image pixel in at least one layer is created by modifying image pixel features in each of the at least two layers based on the final trajectory. An enhanced image sequence is created based on the enhanced image pixel in decomposed image layers. The enhanced image sequence is outputted.
[0006] In another example, a method is provided that includes generating at least one image processing layer from image frames of an image sequence. Decomposition features are determined that are characteristics of image content in the at least one image processing layer for the image frames. Feature connections are determined based on the decomposition features. The feature connections include links that connect image pixels in the at least one image processing layer among space-time locations. Anchor trajectories are calculated for the at least one image processing layer based on the feature connections. The anchor trajectory represent space-time locations of selected image content elements over a length of the image sequence. Relationships are determined of the image pixels with the
anchor trajectories. New trajectories for the image pixels are generated based on the relationships and the anchor trajectories. A trajectory feature is defined for the new trajectories and a modification level is determined for the trajectory feature at new trajectory intersections in the at least one image processing layer. The trajectory feature representing characteristics selected for targeted enhancement in the image sequence. The new trajectory intersections recording intersections of the new trajectories with the image frames in a form of space-time locations. Final trajectories are generated from the new trajectories. At least one enhanced image layer is generated by modifying pixel image features in the at least one image processing layer based on the final trajectories. An enhanced image sequence is generated using the at least one enhanced image layer.
[0007] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon that are executable by a processor device to perform operations, the operations comprising:
generating at least one image processing layer from image frames of an image sequence;
determining decomposition features that are characteristics of image content in the at least one image processing layer for the image frames;
determining feature connections based on the decomposition features, the feature connections including links that connect image pixels in the at least one image processing layer among space-time locations;
calculating anchor trajectories for the at least one image processing layer based on the feature connections, the anchor trajectories representing space-time locations of selected image content elements over a length of the image sequence; determining relationships of the image pixels with the anchor trajectories;
generating new trajectories for the image pixels based on the relationships and the anchor trajectories;
defining a trajectory feature for the new trajectories and determining a modification level for the trajectory feature at new trajectory intersections in the at least one image processing layer, the trajectory feature representing characteristics selected for targeted enhancement in the image sequence, the new trajectory intersections recording intersections of the new trajectories with the image frames in a form of space-time locations;
generating final trajectories from the new trajectories;
generating at least one enhanced image layer by modifying pixel image features in the at least one image processing layer based on the final trajectories; and
generating an enhanced image sequence using the at least one enhanced image layer
Brief Description of the Drawings
[0001 ] FIG. 1 is a table that includes examples of information recorded by an a trajectory according to one example.
[0002] FIG. 2 is a block diagram of a method of enhancing an image sequence by computing multi-layer trajectories according to one example.
[0003] FIG. 3 is a block diagram of a method of decomposition of an image frame into layers according to one example.
[0004] FIG. 4 is a block diagram of a method of enhancing an image sequence by computing trajectories according to one example.
[0005] FIG. 5 is a block diagram of an image processing device according to one example.
[0006] These illustrative examples are mentioned not to limit or define the disclosure, but to aid understanding thereof. Additional examples, features, and embodiments are discussed in the Detailed Description, and further description is provided there. Advantages offered by one or more of the various examples may be further understood by examining this specification or by practicing one or more examples presented.
Detailed Description
[0007] Certain aspects and features of the present disclosure relate to enhancing a characteristic or characteristics of an image sequence by computing trajectories across multiple image frames of the image sequence and optimizing the performance of the enhancement along the trajectories. The characteristics of an image sequence, including video and motion picture images, can be digitally enhanced by analyzing available image frames in the image sequence and generating accurate motion information. Accurate motion information can be obtained by computing trajectories for each image pixel of each image frame of the image sequence. The structure of the trajectory can allow an image enhancement method to achieve a superior performance by following the trajectories that represent more accurate motion and to perform based on similar appearances of the image content and on dissimilar appearances of the image content.
[0008] An image frame can include multiple image pixels, and each pixel can define an image content element at a space-time location in the image frame. Because motion can exist in an image sequence, the same content element may appear in different space-time locations in other image frames of the image sequence. The movement of the image content element may be represented by a path connection that records the space-time locations of the same image content element across multiple image frames of the image sequence. Trajectories can include the space-time locations of an image content element over the length of the image sequence, and the features of the image content element in the vicinity of each space-time location in a form of multiple feature channels.
[0009] A trajectory of an image content element can include three types of elements: a trajectory intersection, a trajectory feature and a trajectory feature flag. A trajectory intersection can record the intersection of a trajectory with an image frame, and the trajectory intersection can be represented as a space-time location, such as
for image frame n. A trajectory can begin at a pixel location in a frame, which can be an anchor pixel of the trajectory, and the trajectory can intersect with other image frames at space-time locations that are either pixel positions or sub-pixel positions.
[0010] In one example, a trajectory has a structure that contains trajectory intersections, trajectory features, and trajectory feature flags. A trajectory intersection can record the intersection of a trajectory with an image frame in a form of a space-time location. A trajectory feature represents certain characteristics of the image sequence that is selected for a targeted enhancement application. Examples of a targeted enhancement application include super-resolution, de-noising, and de-occlusion. Because different trajectory features may be selected for different targeted enhancement applications, trajectories may be customized for a targeted enhancement application. For a given targeted enhancement application, a trajectory feature may be considered trustworthy in some image frames but unreliable in other image frames. The changing status of a trajectory feature can be tracked by a trajectory feature flag, which marks the trajectory feature as an essential image
component or as an outlying image component in different image frames. An essential image component represents a "good" space-time location where the trajectory feature can be used to compute a targeted enhancement or to predict or refine trajectories. An outlying image component represents a "bad" space-time location where the trajectory feature may be harmful to related computations. Essential image components are used for computations that are based on similarity, while outlying image components are used for dissimilarity. Trajectories may be computed for any image sequence.
[001 1 ] In one example, each image frame of an image sequence can be decomposed into multiple component layers and trajectories may be computed in each layer. Layer decomposition can be based on pixel image features that are identified for enhancement based on the targeted enhancement application. Pixel image features may be extracted from the image sequence, and then organized or grouped into layers. Since the pixel image features are selected for the purpose of enhancement, the pixel image features may appear dissimilar in some image frames in certain layers and may be difficult to track reliably and accurately. Decomposition features can be identified for each layer to facilitate the tracking of pixel image features within the layer. A decomposition feature can be different from a pixel image feature, and the decomposition feature can be prominent within the layer and easier to track. A layer with additional decomposition feature channels can be an augmented layer in which each image pixel is assigned an additional value from each feature channel. Trajectories may be computed in each layer, and the trajectory features may be selected from the feature channels in the augmented layer.
[0012] In one example, anchor trajectories are computed in each layer for selected image pixels. The feature connections for those pixels in a layer can be computed based on decomposition feature channels, and anchor trajectories can be computed from the feature connections using a sub-space method. The anchor trajectories can be used for computing trajectories for all or substantially all pixels in all or multiple layers.
[0013] In one example, trajectories for each pixel in each layer are computed based on the anchor trajectories. Relationships between a pixel and the anchor trajectories in each layer can be computed, and new trajectories for each pixel may be computed based on the iterative refinement of trajectory intersections. Once trajectories are established, for pixels of the image frames, the appearance of the
image content at a pixel can be optimized along the trajectories to generate an enhanced appearance. Certain aspects provide the ability of tracking dissimilar appearances along a trajectory in order to produce a preferred appearance.
[0014] In one example, the appearance of image content may be adjusted to meet different quality criteria from different targeted applications, including motion picture theatrical presentations, television broadcasting, video streaming through internet, video up-conversion, and VFX (visual effects) applications. The appearance of the image content can be produced by an optimization process along the trajectory based on the selected criteria.
[0015] In one example, a trajectory can be generalized for stereoscopic three-dimensional (3D) motion pictures and 3D videos that include a left-eye image sequence and a right-eye image sequence. Correlations between the left-eye image sequence and the right eye sequence can be identified, and trajectories can be computed in each sequence. The trajectories of both image sequences can be unified so that the trajectories represent the same image content. A joint optimization can be applied to optimize the appearance of the image content along trajectories.
[0016] Every pixel of an image frame may have a trajectory across multiple image frames of the image sequence. The trajectory features can be represented by M channels, f f^ ^ ^. For example, given an image sequence that includes N
image frames, and each image frame has a spatial resolution of ^ pixels per row and H rows, a pixel P in image frame n is a spot that has a trajectory T( ) represented by:
½ - f , ί*»Χ,Ί f,Λl #Λs * -Λ^ -.¾Κ; ^ f* V fi¾ - fsf¾ f -M t ( lEiqquuaattiioonn 1 i ))
[0017] The trajectory T( has ^ entries, with each entry corresponding to one of the image frames of the image sequence. Each entry of TC ) corresponds to an image frame, and each entry has M+3 elements:
«*f¾* (1 < t≤ The frame number 1≤ i≤M represents the frame position in the image sequence. Trajectory T(¾ has an anchor position in image frame n at which space-time locations of the trajectory can be represented by integers. The intersections of T{ in other frames with
t≠ m, may be represented with sub-pixel precision. A trajectory may be defined for the entire length of the image sequence or only a portion of the image sequence.
[0018] The trajectory T(F|may have multiple features / ,f ·»* *(£, and each feature represents a specific characteristic of the trajectory at space-time location. The value of a feature in each image frame is recorded with an appropriate flag. The value of a feature flag may change over the length of the trajectory, indicating changes of computing strategy. In one example, each flag takes a value of " or "0". A flag value "1 " of a feature can be the essential image component of the feature, and a value of "0" can be the outlying image component of the feature. The flag values of the features of a trajectory can control the change of computing strategies for the trajectory.
[0019] Essential image components can represent "strong" space-time locations along a trajectory that is trustworthy to be used for computing and predicting other trajectories or for refining the current trajectory. Outlying image components represent "weak" space-time locations along the trajectory that may be harmful to trajectory calculation. Essential image components can be used for computations that are based on similarity, while outlying image components can be used for computations that allow dissimilarity. During computing, essential image components can contribute to a data cost term. The outlying image components may be ignored in some cases or may be manipulated along a trajectory as long as the trajectory is computed based on the essential image components.
[0020] In FIG. 1 there is shown a table that includes examples of information recorded by an a trajectory that has M = 3 features.
[0021 ] The trajectory example shown in FIG. 1 can connect five space-time locations in five image frames, and the space-time locations can be trajectory intersections, denoted by {(xo,yo,to),(xi ,yi,ti), (x2,y2,t2),(x3,y3,t3),(x4,y4,t4)}, where xn,yn represent the spatial coordinates and tn represents the temporal coordinate. The trajectory has three trajectory features. Feature f1 may represent large-scale image details. Feature z may represent fine image details, and feature f3 may represent a lighting change between neighboring image frames. The trajectory in the example also has a feature flag array denoted by {(1 , 1 , 1 , 1 , 1 ), (1 ,0, 1 ,0, 1 ), (0,0,0,0,0)}. The feature flag array represents the characteristics of the image content at each trajectory intersection along the trajectory. For example, all "zero" flags for feature
three indicates that the image pixels along the trajectory are not subject to lighting changes; all "one" flags for feature one indicates that the image pixels along the trajectory share similar large scale details. The mixture of "zero" and "one" flags of feature two indicates that the fine-scale detail of the image pixels at certain space-time locations may be significantly modified.
[0022] The trajectory may be defined using two computation methods. In the first case, all trajectory features are flagged with "one"s, indicating that the trajectory is trustworthy and the information collected along the trajectory includes essential image components that can be used in computing. An example of such methods may include a block-based, motion-compensated algorithm in which pixels in the vicinity of a trajectory are used for image enhancement. In the second case, trajectory features are flagged with "zero"s, indicating that the information collected along the trajectory cannot be used automatically for computations. An example of such methods includes human-assisted VFX operations in which image pixels may be replaced with new content based on human judgment.
[0023] Trajectories may be computed in layers after an image sequence is decomposed into two or more layers. An image layer can be a collection of deconstructed image frames that describe a certain characteristic of the image sequence. FIG. 2 is a block diagram of an example of a process for enhancing an image sequence. An image sequence that that multiple image frames is received by an image processing device in block 102. The image processing device can perform the computations for the enhancement of certain targeted characteristics of the image sequence. Examples of targeted enhancement applications include fine-detail enhancement, noise reduction, image sharpening, and upsizing.
[0024] In block 104, the image frames of the image sequence are decomposed into layers by the image processing device. Each image frame of the image sequence can be first decomposed into at least two layers. The layer decomposition can be performed using a decomposition method that is selected based on the targeted enhancement application. For example, a wavelet transform may be used for the decomposition of an image sequence when the target is to enhance the fine details of the image sequence. Different decomposition methods may be selected for different targeted enhancement applications. An inverse transform can be available for a selected decomposition method so that each image frame of the image sequence can be recovered from the decomposed layers.
[0025] Multiple layers of an image sequence can be generated by organizing or grouping pixel image features that are extracted from decomposition. FIG. 3 depicts an example of a process for performing layer decomposition of image frames in the image sequence into multiple layers. In block 202, pixel image features are identified based on a targeted enhancement application by the image processing device. Given a targeted enhancement application, a certain pixel image features of interest can be identified. A pixel image feature can represent a specific characteristic of image pixels that has a direct impact on the targeted enhancement over the image sequence. Examples of pixel image features include color, detail, contrast, and noise level. In the case that the targeted enhancement is super-resolution, image detail may be identified as a pixel image feature. Multiple pixel image features may be identified to be jointly enhanced for a given targeted enhancement application. For example, image detail and noise level may be identified when the targeted enhancement application is to increase the perceived resolution and to reduce noise.
[0026] In block 204, the image processing device extracts identified pixel image features from the image sequence. In the case that image detail is the identified pixel image feature, a wavelet transform may be applied to each image frame of the image sequence. The wavelet transform can separate fine details of an image frame in the image sequence into multiple scales and various spatial directions. Each image frame may be decomposed into four sub-bands, namely the HH, HL, LH and LL sub-bands. The HH, HL and LH sub-bands contain fine image detail at a first scale, and the LL sub-band can further be decomposed into another set of HH, HL, LH and LL sub-bands at a coarse scale. This process may be repeated to generate fine image detail at increasingly finer scales.
[0027] In block 206, the image processing device characterizes the extracted pixel image features. The extracted pixel image features may be further characterized to be organized into layers. In the case of wavelet sub-bands, the image details may be characterized by scales or by spatial orientations. Characterizing can include determining a mathematical description and organization of a particular feature detail of an image pixel, a layer, or a trajectory, etc. Examples of a feature detail include color, illumination intensity, lighting changes, motion, etc. In block 208, the image processing device organizes the pixel image features into multiple layers. Each layer can include a deconstructed image sequence that contains one or more channels. The formation of layers can be used in computing
high-quality trajectories because multiple layers can jointly provide information for determining "strong" and "weak" space-time locations. The correlations between different layers can also provide useful information.
[0028] As an example, the pixel image features separated through wavelet transform may be grouped into three layers. Layer one can represent large-scale image details, and it may be generated by grouping HH, HL and LH sub-bands at a certain range of large scales. The large-scale image details in Layer one are not classified by spatial orientations. As a result, Layer one is a bandpass-filtered image sequence with only large scale details. Layer two can represent fine-scale image details, and it may be generated by selecting the HL, LH and HH sub-bands at a certain range of finer scales. The resulting Layer two image sequence can be a high-pass filtered image sequence. Layer three can represent brightness distribution, and it may use only the highest scale LL band. As a result, Layer three can be a low-pass filtered image sequence with finer scale details removed. Layer three may also be generated using other types of filters, such as edge-preserving smoothing filters that can remove textures while retaining sharp edges of image content. A layer may contain a number of image sequences, which can be channels.
[0029] Trajectories can be computed independently from the image sequences in each layer. Due to the differences from one layer to another, a trajectory of a given anchor pixel in one layer may not share the same space-time locations as a trajectory of the same anchor pixel in another layer. Such layer discrepancies provide valuable information that can be used for image enhancement. The correlations between the trajectories in different layers can be analyzed and used to refine the accuracy of the trajectories. In one example, the computation of trajectories and layer decompositions may be executed iteratively. Prior to layer decomposition, certain anchor points may be identified, and trajectories of those selected anchor points may be computed over the image sequence. The resulting trajectories may be used to guide layer decomposition and to generate more accurate layer trajectories.
[0030] After an image sequence is decomposed into layers, trajectories can be computed by tracking pixel image features. However, the pixel image features in a layer may appear dissimilar from one image frame to another image frame, and therefore may be difficult to track reliably and accurately. For example, the existence of noise in Layer two may make accurate tracking the fine scale details in the layer more difficult. Additional information may be needed to facilitate tracking of pixel image features in a specific layer, and a collection of such information can be decomposition features.
[0031 ] A decomposition feature may be a characteristic of the image content in a specific layer that is prominent within the layer and thus is easier to track. A decomposition feature is typically different from a pixel image feature. Returning to FIG. 2, the image processing device can determine decomposition features for the image frames in each layer in block 106. The decomposition features can be determined such that the decomposition features may be used to facilitate the computation of trajectories in a layer to account for dissimilarities of pixel image features. For example, if the fine-scale details in Layer two are corrupted by noise and become difficult to track, strong edges in the layer may be identified as decomposition features, which may be more likely to be tracked accurately even under noise corruption. An edge-detection method may be used to extract strong edges in Layer two, and the resulting strong edges may be used to facilitate trajectory computation in the layer.
[0032] The decomposition features may be computed from the same layer where the trajectories are to be computed, but the decomposition features may alternatively be computed from a different layer or from a number of different layers. For example, another type of decomposition features may be object masks. Object masks may be computed from another layer, such as Layer three in the above example, or the object masks may be obtained directly from the image sequence using a color segmentation method. Object masks may also be generated by a human-assisted interactive method, such as rotoscoping. Object masks may be used to facilitate trajectory computation in a specific layer, such as Layer two in the above example. The decomposition features may be added to a layer as additional feature channels. A layer with decomposition feature channels can be an augmented layer in which each image pixel is assigned an additional value from each feature channel. The feature channels of an augmented layer may include both decomposition features and the pixel image features. For example, Layer two may be an augmented layer with a pixel image feature channel containing fine-scale details and two added decomposition-feature channels including a strong edge channel and an object mask channel. Subsequently in the computation of trajectories in the layer, each trajectory may use some feature channels and determine whether those channels can be
significantly modified. The selected feature channels may be as trajectory features. Trajectory features can be a subset of the feature channels in an augmented layer.
[0033] In block 108, the image processing device computes feature connections for image pixels in each layer based on the decomposed image features. A feature connection can be a link that connects an image pixel p from one space-time location (%*¾#¾3 to another space-time location where i p. The image pixel p may have feature connections with a number of neighboring image frames. A feature connection can be computed based on the decomposition feature channels. In one example, preliminary feature connections of pixel p are first computed in a layer over a number of neighboring image frames using an optical flow method. The computation of optical flow can be based on a Bayesian approach that minimizes an energy function that includes data costs and link costs. The image pixels with the lowest energy values and those with the most reduced energy values can be selected as feature connections. In general, each image pixel in a layer may have feature connections with other frames in the same layer. In practice, however, only a limited number of feature connections may be selected to reduce the required computation. For example, pixel p may have four feature connections with the following space-time locations
^¾-$ρ3¾.-2* ¼-g i (½-l*3 -t* ¾■-!.)' 3¾*1* % ·*ΐ)>· ¾#-S* 'n f°ur neighboring image frames.
[0034] In block 1 10, the image processing device calculates anchor trajectories for each layer based on select feature connections. Anchor trajectories can be computed based on selected feature connections in the layer using a sub-space method. A feature connection across M space-time locations in M frames may be represented by a 2M long column vector:
f Ϊ = I w—*m*7t*T (Equation 2)
[0035] If N feature connections are selected, the column vectors are concatenated horizontally into a 2ΜχΝ matrix F
* - fft. -f. - ftj (Equation 3)
Each feature connection is represented as a separate column in the matrix. It assumes that all feature connections across those M frames exist in a rank R subspace, so that matrix F may be factorized into:
F = (Equation 4)
B is a 2MxR matrix with each column representing a base trajectory, and C is a RxN matrix of coefficients. One example of computing the rank-R base matrix B is by applying singular value decomposition and truncation to rank R. Rank R can be related to the complexity of motion in the scene. Anchor trajectories are obtained from base matrix B after factorization since the R columns of matrix B are recognized as the intersections of anchor trajectories. In order to generate sufficient number of anchor trajectories, the value of M is sufficiently large since of? < ¾ . In one example, multiple matrices E¾¾¾,? may be computed by selecting multiple local areas in an image frame to generate a large number of anchor trajectories.
[0036] Each anchor trajectory may have a number of trajectory features, and those features can be selected from the feature channels from an augmented layer. The feature flags for each anchor trajectory can be set based on the characteristics of the layer decomposition as well as based on the targeted enhancement applications. The values of feature flags may be used as the control parameters for a given targeted enhancement application. For example, assuming anchor trajectories are defined as in the table of FIG. 1 for a three-layer decomposition where Layer one represents large-scale image details, Layer two represents fine-scale image details, and Layer three represents brightness distribution, the anchor trajectories in Layer one may have the feature flags for feature 1 set to all "one's," indicating large-scale detail is the essential image component for the layer. At the same time, the feature flags for feature f 2 and feature s may be set to all "zero's" so that both fine-scale detail and lighting changes are treated as outlying image components. Similarly, in Layer two, the feature flags for feature fz may be set to all "one's," while the flags for feature f1 and 3 may be set to all "zero's." If lighting change is prominent in the image sequence, the feature flags for /3 may be set to all "one's" so that the process can be aware of lighting changes in the computation. The trajectory feature flags may be set differently for different layers.
[0037] The number of anchor trajectories can be smaller than the total number of image pixels in a layer because anchor trajectories are generated from a comparatively small number of selected image pixels. The feature connections of those selected pixels can be computed using methods such as optical flow. Anchor trajectories can be used to compute the feature connections for other image pixels
that are not on the path of anchor trajectories. In block 1 12, the image processing device determines the relationships of each image pixel of the decomposed image frames with anchor trajectories in each layer. A relationship may be a feature association or a mathematical association in which there are similarities, differences, or similar differences between image pixels, trajectories, or image pixels and trajectories, etc. Relationships can be described in terms of space and time expressions or other mathematical expressions. Assuming pixel p is not on the path of any anchor trajectory, the feature connections of pixel p may be determined by establishing a relationship with one or more anchor trajectories in the neighborhood. An example of the relationships of pixel p with anchor trajectories are represented by Equation 5 in which its feature connection fn may be expressed as a sparse representation by anchor trajectories:
%, = ® ™ i«¾ subject te f9. = Be (Equation 5)
[0038] In Equation 5, !*¾ is a L s norm that equals the number of non-zero entries in coefficient vector ε. Equation 5 may be computed using linear programming. The non-zero entries in the Lc norm equals to the number of non-zero entries in the coefficient vector. The number of non-zero entries in implies that each image pixel may be related to a reduced number of anchor trajectories.
[0039] Blocks 108, 1 10, 1 12 may be applied to all decomposed layers for each pixel. For pixel p at space-time location ; a set of information
≥ l) is computed in each layer /. As described previously, matrix B: stores the intersections of computed anchor trajectories, coefficient vector