Claims:WHAT IS CLAIMED IS:
1. An apparatus of registering biomedical images of 2D and 3D dimensions to reduce imaging artifacts caused by movement of objects in the input image which comprises of the following steps:
a. Providing two images of input image and reference image which are of multi-modal and consist of cross sectional features of the same medical object, the two images consist of two or three dimensional array of pixels or voxles;
b. Reference image has objects with known landmarks, features and feature positions in the 2D or 3D geometry, the input image is the deformed image or image captured using different method has the similar object as in the reference image but is time shifted, scaled, rotated, sheared and has also has features that are not in the reference image in addition to the features in the reference image;
c. Transforming both images using complex wavelets or in particular dual tree complex wavelets to generate multiple sub bands of both input image and reference image, the DTCWT algorithm decomposes the 2D or 3D image into sub bands at different resolutions, the number of decomposition levels is set to D, which generates only two frames of low pass and high pass bands after D-level decomposition of 3D image. For 2D the D-level decomposition results in low pass and high pass bands with only 32 x 32 pixels.
d. Image registration is carried in two steps: training and evaluation
e. During training phase of neural network structure the network weights and biases are obtained by training the network using multiple data sets of 3D or 2D images that are obtained after DTCWT transformation. The trained network estimates the transformation parameters for a given input that is used to register input image with the reference image.
f. During validation phase input image that were not part of training data set is considered for image registration using the algorithm proposed in the present invention
g. Evaluation of registration algorithm is carried out by computing MSE and PSNR metrics considering reference image and registered image, input image and registered image and comparing them with MSE and PSNR obtained from reference image and input image.
h. In addition to computing MSE and PSNR the transformation vectors estimated from NN is compared with actual transformation vectors and the corresponding error is measured to determine image registration algorithm accuracy.
2. An apparatus for registering a reference image with a input image both are of 4D images, the input image containing multiple 3D images with 3D objects, the reference image containing multiple 3D images with 3D objects, the input image is required to register all 3D images with all 3D images of reference image leading to 4D image registration comprises the following steps:
a. Reference 4D image that comprises of multiple 3D images are first identified from the 4D data set, similarly for 4D input image number of 3D image are identified.
b. For each of the 3D data image registration algorithm with the method in Claim 1 is used to estimate the transformation vectors
c. With estimation of transformation vectors for multiple 3D images, the average of transformation vectors are estimated
d. If there are N 3D data sets, there will be N transformation vectors estimated from the method in Claim 1, which are grouped into two by considering 1 to (N/2)-1 as the first group of transformation vectors and N/2 to N as the second group of transformation vectors.
e. The average of first group of transformation vectors and average of second group of transformation vectors are computed that are used for transforming the first N/2 3D frames and second N/2 3D frames respectively.
f. 4D registration algorithms are evaluated by computing MSE and PSNR metrics as in Claim 1.
3. The apparatus of Claim 1, wherein said image is transformed into wavelet sub bands is performed using 3D DTCWT that decomposes input data at every level into 56 high pass and 8 low pass bands. At every level of decomposition the data size is reduced by half. For example if the input data size is 512 x 512 x 64 after first level of decomposition the data size of each of the 64 sub bands will be 256 x 256 x 32. The number of levels of decomposition is limited to D wherein for the example considered D is set to five and after five levels of decomposition there will be four low pass bands of size 16 x 16 x 2 and 56 high pass bands of size 16 x 16 x 2 at level 5, 56 high pass bands of size 32 x 32 x 4 at level 4, 56 high pass bands of size 64 x 64 x 8 at level 3, 56 high pass bands of size 128 x128 x 16 at level 2, 56 high pass bands of size 256 x 256 x 32 at level 1. In total there will be 8 low pass bands and 280 high pass bands. The apparatus in Claim 1 requires low pass bands for image registration that is obtained after D levels of decomposition.
4. The apparatus in Claim 1, where in the DTCWT sub bands are used for estimating the transformation vectors is carried out using one of the neural network algorithms defined as “2DFFNN_1” that comprises of M (M is number of frames that have been obtained after DTCWT decomposition) three layered (input layer, hidden layer, output layer) Feed Forward Neural Network Structure with input layer having 16 vectors, hidden layer having 32 neurons and tansig activation function, output layer consisting of 16 neurons with tansig activation function. The FFNN structure is arranged into M-arrays that process each of the M frames of DTCWT to generate 16 transformation vectors.
5. The apparatus in Claim 1, where in the DTCWT sub bands are used for estimating the transformation vectors is carried out using another neural network structure defined as “2DFFNN_2” that comprises of three FFNN structure as defined in Fig. 12, that are arranged connected into two stage structure as shown in Fig. 15. Stage 1 consists of two FFNNs forming 2D network structure that process the two frames of low pass DTCWT sub band and generates 16 outputs from each FFNN structure. The second stage processes both vectors generated from stage 1 and generate 16 output vectors that represent transformation vectors for image registration.
6. The apparatus in Claim 1, where in the DTCWT sub bands are used for estimating the transformation vectors is carried out using another neural network structure defined as “2DFFNN_3” that comprises of multiple neurons that process the DTCWT sub bands to generate transformation vectors as shown in Fig. 16. Every neuron in the hidden layer is connected with four pixels from the two frames of DTCWT sub bands to generate one output; each of the hidden layer neuron output generated is processed by the output layer to generate the transformation vector that is required for image registration. With hidden layer processing data from both frames of DTCWT sub bands, the hidden layer is a two dimensional structure that converts 2D input data to 1D output vector, the output layer is a 1D structure.
7. The apparatus in Claim 1, where in the DTCWT sub bands are used for estimating the transformation vectors, is carried out using another neural network structure defined as “2DFFNN_4” that comprises of multiple neurons as shown in Fig. 17 that processes DTCWT frames to generate transformation vectors for image registration. With DTCWT sub bands consisting of two frames after D level decomposition, and each frame consisting of N x N pixels, there are N-1 x N-1 neurons in the hidden layer, with each of the neuron processing four neighboring pixels that are grouped by overlapping 2 x 2 sub images. The N-1 x N-1 neurons in the hidden layer generated (N-1)(N-1) outputs that are processed by 16 neurons in the output layer to generate 16 transformation vectors that are required for image registration.
8. The DTCWT sub bands that are being processed by the apparatus in Claim 4 to Claim 6 are obtained by image reordering of DTCWT sub bands. DTCWT low pass sub bands are sub grouped into non-overlapping 4 x 4 blocks and each of the 4 x 4 blocks is scanned in zig-zag method (shown in Fig. 14) to convert 4 x 4 sub image into 16 x 1 vector. For example if the frame size of DTCWT sub band is 16 x 16, they are grouped into 16 sub images each of 4 x 4 size and are reordered into 16 columns with each column comprising of 16 pixels. As there are two frames the reordered matrix will have two data inputs representing frame 1 and frame 2 each having 16 column vectors, with each column vector having 16 elements.
9. The neural network apparatus as described in Claim 4 and Claim 5 processes the two data inputs as in Claim 8 independently. The reordered data of two frames form the input layer for the neural network structure 2DFFNN_1 and 2DFFNN_2 that are processed by the two dimensional structures.
10. The neural network apparatus as described in Claim 6 processes the reordered data inputs as in Claim 8 by considering data vectors from both the reordered data sets. The cross link connection of input layer improves inter pixel correlation among the hidden layer neurons of 2DFFNN_3 structure in generation of transformation vectors.
11. The neural network apparatus as described in Claim 7 is designed to processes the DTCWT sub bands without reordering scheme. The DTCWT frames obtained after D levels of decomposition are sub grouped into 2 x 2 blocks, each of the pixels in the 2 x 2 blocks of both frames are connected to one hidden layer neuron, there are N-1 x N-1 hidden layer neurons that process the two frames to generate the transformation vectors. The N-1 x N-1 outputs of hidden layer is further processed by 16 neurons in the output layer to generate 16 transformation vectors.
12. The neural network apparatus as described in Claim 7 is designed is designed to have N-1 x N-1 neurons in the hidden layer, the output of these hidden layers are further processed by another hidden layer represented as hidden layer that comprises of 128 neurons to generate 128 intermediate outputs. The outputs of hidden layer 2 are processed by the output layer consisting of 16 neurons to generate the final transformation vectors.
13. The transformation vectors generated from the apparatus as described in Claim 4 to Claim 7 are also used to generate transformation vectors considering high pass sub bands. As DTCWT generates six directional sub bands at each level in addition to low pass sub bands, from each level of high pass bands obtained six bands are selected for each of the six selected high pass bands six 2D FFNN structures can be designed at each level to generate transformation vectors.
14. From the six transformation vectors estimated as in Claim 13 at each level average of these six transformation vectors are computed from each level, further the average of transformation vectors from all levels are computed. The algorithm described in present invention generates two transformation vectors one from low pass sub band processing and the other from high pass sub band processing.
15. Registration of low pass DTCWT sub bands are performed by considering low pass processed transformation vectors as in Claim 13 and high pass bands are registered by considering high pass processed transformation vectors as in Claim 14.
16. Inverse DTCWT is carried out to compute the registered 3D image by D-levels of inverse process. At each level the transformation vectors generated are correspondingly used to perform image registration as in Claim 13 and Claim 14.
17. The apparatus in Claim 2 for processing of 4D images is carried out by considering multiples of 3D images that are independently registered as per the apparatus described in Claim 1 and apparatus as in Claim 13 and Claim 14.
18. An apparatus for 4D image registration considers only half number of 3D images that are used in generation of transformation vectors, with the transformation vectors generated as in Claim 13 and Claim 14 two consecutive 3D images are registered using the transformation vectors and the entire set of 3D images forming 4D data set is registered.
, Description:The present invention overcomes the limitations of the conventional techniques of 3D image registration by providing a methodology that combines two techniques. Specifically, the present invention uses landmark manifolds to identify features in both real and imaginary domain and estimates transformation parameters for image registration.
The advantages and additional features of the present invention will be detailed in the description which follows, and in part, will be apparent during further description, or may be learned by practicing the invention. The objectives and techniques described in this invention will be realized by the method particularly pointed out in the written description and the claims hereof as well in the associated diagrams listed.
In one embodiment of medical image registration according to the present invention, image registration algorithm and methods are based on neural network that is trained to estimate transformation parameters considering DTCWT features.
To achieve these and other advantages and in accordance with the purpose of the invention, as embodied and broadly described, according to the present invention for registering input image with the reference image there are several steps, including identifying significant features from the real and imaginary sub bands of complex wavelets of both input image and reference image. Once the features have been identified, the method includes the process of computing transformation parameters considering the trained neural network structure that has been specifically trained to perform the task. The transformation parameters are then used to transform the input image.
Both the foregoing general description and the following detailed description are exemplary and the explanatory and are intended to provide further explanation of the invention as claimed.