FORM 2
THE PATENTS ACT 1970
(39 of 1970)
AND
The Patents Rules, 2003
COMPLETE SPECIFICATION
(See section 10 and rulel3)
1. TITLE OF THE INVENTION:
"A SYSTEM FOR REAL-TIME IMAGE CORRECTION FOR CURVILINEAR SURFACES"
2. APPLICANT:
(a) NAME: KPIT CUMMINS INFOSYSTEMS LIMITED
(b)NATIONALITY: Indian Company incorporated under the Companies Act, 1956
(c) ADDRESS: 35 & 36 Rajiv Gandhi Infotech Park, Phase 1, MIDC, Hinjewadi, Pune - 411057, Maharashtra, India.
3. PREAMBLE TO THE DESCRIPTION:
The following specification describes the invention and the manner in which it is
to be performed.

Field of Invention:
The invention relates to a real time system for performing geometrical image correction. More specifically, the present invention relates to system of image correction for the images projected on curvilinear surfaces.
Background of Invention:
Displaying geometrically straight image on a curvilinear surface is of great importance to certain applications like a Head-up-display system where vehicle information is projected onto a curvilinear windshield of aircrafts and land vehicles or projection of images on curved surfaces using a projector and similar other applications. Such projected images when viewed by a viewer through the centroid of the eye box region, should look geometrically straight.
Projecting an image on a plane typically preserves all the geometric characteristics of the image provided the projection angle is 90°. However, projecting image on non-planar surface and/or with projection angle other than 90°. adds perceptual deformations in the projected image. To illustrate, a rectilinear grid when projected on curved surface will not 'look" as a rectilinear grid, in order for it to look as a rectilinear grid, we need to apply opposite or inverse operations on input image before projection so that it looks as perfect grid after projection.
Therefore, to correct an image, one applies a set of image transformation operations (geometric transforms) in sequence such that the output image looks straight or normal to the user. A system is developed with a set of such image correcting operations and ported to the embedded hardware. However, the present system shows that applying a sequence of operations consumes noticeable processing time and it may not be suitable for present real time applications with targeted frames per second (FPS). This is due to the fact that it involves heavy mathematical computing which needs to be carried out for each operation every time a projected image needs correction.

US patent No. 20070291051 proposes a method whereby image correction is achieved through digitally manipulating an image by performing horizontal and vertical scan. This increases time complexity and consequently the performance efficiency of the system reduces.
JP2001208999 proposes a method whereby displayed image is corrected with the help of a concave mirror. The difference in radius of curvature is calculated and applied to correct distortion caused by the windshield. The use of different component according to the method results in a substantial processing time. Further, the device disclosed therein is bulky and hence not suitable for daily applications.
The systems as known hitherto used processor intensive codes for rectification of the images. Consequently, the processing time is substantially increased hence, making them inconvenient for real-time applications. With a view of obtaining real time distortion-less image, the current invention provides for a system for image correction for curvilinear screens for projecting real-time images with a substantially reduced processing time,
Summary of Invention:
The current invention provides a system for performing geometrical image correction operations in real time for the images projected on curvilinear surfaces.
In an aspect, the said system for image correction uses a Transformation Matrix for image correction.
In another aspect, the transformation matrix is generated, which is only a pixel to pixel mapping of source to destination, representing the entire sequence of geometric transforms.
Brief Description of Drawings:
Fig, 1 illustrates image correction through sequence of warping operations.

Fig. 2 illustrates a conventional system of image correction and display.
Fig. 3 illustrates an embodiment of the present invention.
Fig. 4 illustrates components of projection means.
Fig. 5 illustrates contents of a transformation block.
Fig. 6 illustrates a diagram for smile-frown unit.
Fig. 7 illustrates a real-time application of transformation matrix
Fig. 8 illustrates a transformation matrix based sequence of operations.
Fig. 9 illustrates a transformation matrix based image correction.
Detailed description of Drawings:
Fig. 1 illustrates a conventional image correction mechanism through sequence of warping operations.
Fig. 2 illustrates a conventional method for image correction.
Fig, 3 illustrates an embodiment of the present invention for real-time image projection comprising a viewing means (1), viewer (2), projecting means (3), curvilinear surface (4), and corrected image (5).
Fig. 4 illustrates components of projection means comprising a processor (301) and transformation block (302).
Fig. 5 illustrates contents of transformation block (302) comprising a rotation unit (3021), parallelogram unit (3022). smile-frown unit (3023). and push-pull unit (3024).
Fig. 6 illustrates a diagram for smile frown unit where the distance d is from the center of the edge along which the operation is carried out, m is number of rows in the input image and n is number of columns in the input image.
Fig. 7 illustrates a real time application for transformation matrix wherein the photographic input is corrected using transformation matrix based operation.

Fig. 8 illustrates a sequential process of image correction using a transformation matrix.
Fig. 9 illustrates a transformation matrix based image correction.
Detailed description of Invention:
The present invention is described in detail herein below with respect to the figures provided.
As illustrated in Fig. 1, a common image correction method involves a sequence of warping operations to be performed on an image comprising a photographic input means, a processor and an output means. Accordingly, the image received from the input means is firstly rotated, then processed through a parallelogram operation and finally processed using smile operation. The image thus processed is then transmitted to the output means for display.
Fig. 2, illustrates an example of the conventional method of image correction/ rectification wherein the input image is passed through a series of geometrical transformation operations one after another to produce a corrected output image.
Since image correction operation involves a set of geometric transformations that transform an input image into a desirably/optimally distorted image, a unique system has been devised. In this system, instead of having complex mathematical operations performed on itself, an input image is transformed into an output image using transformation matrix operations. The transformation matrix is only a pixel to pixel mapping of source to destination representing the entire sequence of geometric transforms.
The current invention describes a system for real-time image correction operations
comprising:
a) Dynamic creation of a transformation matrix for image correction operation.

b) Direct mapping of source image to target image using single transformation matrix representing sequence of image correction operations.
An embodiment of the present invention, as shown in Fig. 3 comprises of a projecting means (3) that projects a corrected image (5) on a viewing means (1). which is placed on a curvilinear surface (4). The projecting means consist of a processor (301) and a transformation block (302). The processor receives an input image (A) and converts the input image (A) into input transformation matrix. The input transformation matrix is then processed by the transformation block (302). The transformation block (302) performs a series of operation to convert the input transformation matrix to an output transformation matrix. The resultant output transformation matrix is then passed to the processor (30 i). The processor (301) then applies the output transformation matrix to input image (A) to give output image (B).
In fig. 4 illustrates components of projection means comprising a processor (301) and transformation block (302) where the input image A is processed through processor (301) and transformation block (302) results in output image B.
In a further embodiment, the transformation block (Fig. 5) comprises of a rotation unit (3021), a parallelogram unit (3022), a smile-frown unit (3023), and a push-pull unit (3024). The descriptions of these units are explained below.
Rotation Unit (3021):
The rotation unit (3021) is the first unit of the transformation block (3 02) and comprises of a rotation operator. It acquires the photographic input transformation matrix from the processor (301) and performs the rotation operation .The rotation operation is now further explained below.
The rotation operator performs a geometric transform which maps the position (xl, y 1) of a picture element in an input image onto a position(x2, y2) in an output image by rotating it through a user-specified angle 6 about an origin O.

The rotation operation is carried out by using following equations:

Where,
: Coordinates of the center of rotation (in the input image)
θ : Angle at which the image has to be rotated. It can be +ve (clockwise) or -ve
(anticlockwise) depending on the direction of rotation.
: Coordinate of the Input Image Pixels.
: Coordinate of the Output Image Pixels.
Parallelogram unit (3022):
The Parallelogram unit (3022) is the second unit of the transformation block (302). It acquires the photographic intermediate input transformation matrix from the rotation unit (3021) and performs the parallelogram operation .The parallelogram operation is now further explained below.
Parallelogram is an affine transformation, which maps parallel lines onto parallel fines. Its effect leaves all fixed points on one axis and other points are shifted parallel to the axis by a distance proportional to their perpendicular distance from the axis. It is a shear operation in one direction.
A shear is a transformation that distorts the shape of an object along either or both of the axes. Like scale and translate, a shear can be done along just one or along both of the

A shear parallel to the y axis of lines is accomplished by the linear mapping.
coordinate axes. A shear along one axis (say, the x-axis) is performed in terms of the point's coordinate in the other axis (the y-axis). To shear in the x direction the equation is:


The vertical shear leaves vertical lines x = a invariant, but maps horizontal lines y - b into lines

Smile-Frown Unit (3023):
The Smile-Frown unit (3023) is the third unit of the transformation block (302). It receives the intermediate photographic input transformation matrix from the parallelogram unit (3022) and performs the Smile -Frown operation. The Smile-Frown operation is now further explained below.
Smile & Frown transformations are a form of affine transformations where, an input image is geometrically altered from its center either towards inside or towards outside. The operation which transforms an image towards inside is called Frown operation and the operation which transforms an image towards outside is called Smile operation.

In frown operation, the edges of the input image shrink inside proportionately in a symmetric manner along the given direction i.e. vertical or horizontal.
As illustrated in Figure 6 where,
d: distance from the center of the edge along which the operation is carried out m: no. of input image rows n: no. of input image column
The general equation of a circle is

Now the coefficients g, f and c can be calculated in terms of image pixel parameters (m, n, and d) as,

Push-pull Unit (3024):
The Push-pull unit (3024) is the final unit of the transformation block (302). It receives the intermediate photographic input transformation matrix from the Smile-Frown unit (3023) and it further performs the Push-pull operation. The Push-pull operation is now further explained below.

Push operation is a general image warping operation applied on an image to fit it onto the desired area on the screen. The image push operation can be viewed as pushing or pulling a corner of image inside or outside the defined image area respectively, so that the image appears straight and proportionately spread onto the display to the viewer.
This is expressed in general by the following equation

Source image corner coordinates are given by

Where,
= x coordinates of source image corner = y coordinates of source image corner

Where,
xl, x2, x3, x4 = x coordinates of output image corner
yl, y2, y3, y4 = y coordinates of output image corner


The linear equations are solved to get the coefficient vectors A and B of polynomial equation, which can be represented in matrix notation as
x = au
y = bv
Where,
a and b are the column vectors of coefficients a and b respectively.
Calculate mapped source image coordinates (u, v) for each of the output image pixel (x, y) by using the same polynomial equations given by above equations.
Fig. 7 illustrates an embodiment of the invention wherein the transformation matrix block rectifies the input image A received from processor to result in output image B.
Fig. 8 illustrates a sequential process of image correction as performed by the transformation block comprising a photographic input transformation matrix subjected to an image rotation unit (3021), resultant transformation matrix further subjected to a parallelogram unit (3022). Further, the transform matrix obtained from parallelogram unit (3022) is processed with a smile -frown unit (3023) followed by a push-pull unit (3024) and finally resulting in an output transformation matrix. While in the present context the transformation block performs the geometric transformations as mentioned here above, it should be noted that the sequence or number of repetitions for performing these operations is determined by the processor/transformation block as identified for a particular image and corresponding viewing means of the system. Thus, the processor/transformation block can adapt the system for any embodiment of its components and does not limit the system to a particular set sequence of operations.
Fig. 9 illustrates a transformation matrix based image correction where the input transformation matrix as obtained from the processor corresponding to an input image is inputted to transformation block, where a sequence of operations is performed on it to

obtain the resultant output transformation matrix. This output transformation matrix is then passed on to the processor to be mapped onto the input image in order to achieve the final corrected output image for display,
In an embodiment, the sequence of image correction operations is applied on the transformation matrix. The output transformation matrix represents the entire sequence of geometric operations. So, in a real time system, only the output transformation matrix (representing a sequence of operations) is used. This output transform matrix transforms an input image to an output image for display or projection in a real time system. Since the transform matrix represents the entire sequence of operation, the displayed or projected image appears geometrically correct on a curvilinear surface like a windshield. In further embodiment, the image correction on a curvilinear surface in real time provides for many applications such as image correction required for geometric distortions introduced due to optical lenses, head-up-display in aeronautical applications, correcting movie projection on a curved screen to appear as flat screen view, generating bird-eye view from multiple cameras for an around view systems and other such applications, where similar image correction is required.
In an embodiment, the system of present invention may be used for correcting geometric distortions introduced due to optical lenses. ]n optical world, use of cheaper wide angle lenses cause a straight line to be bent away from the center of the image. So, a rectangle looks like a barrel or a pin cushion shaped object. Correction of these distortions is performed according to the method of the present invention by shifting each pixel radially. The distortion caused due to lens property is corrected by using the transformation block of the present invention, which houses a set of geometrical transformations which can make a rectangular grid look like a rectangular grid itself without any distortions. The transformation matrix according to the invention represents the entire sequence of operations required to un-warp the distorted image. Symmetric Smile/Frown operations in vertical and horizontal operations are performed to correct the geometric distortions. Based on the nature of the deformation introduced by lens, it may require implying asymmetric operations for correction. According to an embodiment of the invention, all such sequence of operations will be represented by a single

transformation matrix. In such a system, the image captured using the camera is provided as an input and it is transformed using the transformation matrix according to the present invention and the corrected output is sent for display.
In an embodiment, the system of present invention is used in aeronautical operations. The input images/videos need fast and real-time correction and subsequent smooth and straightened projection in a continuous feed. The transformation block can reduce the processor activity by handling all the image transformation operations. The processor (301) can communicate with the aircraft processor to manage continuous input feed while the transformation block corrects for geometrical image distortions.
In an embodiment, the system of present invention can be used in video correction for curved screen movie to provide a Flat-screen appearance. The continuous feed, as received by the processor, will be provided to the transformation block. The transformation block according to the invention represents a series of transformation and warping operations to generate a straightened corrected image providing a smooth flat-screen experience on a curved viewing surface.
In an embodiment, the system of present invention may be used in generating Bird's-eye View for camera-on-vehicle, consisting of a single or a multiple camera system. In case of multiple cameras, for an around view system, the images from cameras will be stitched resulting in single view of the region of interest. A transformation matrix representing the sequence of image correction operations according to the invention, will then generate a geometrically corrected image from the stitched image.
The use of transformation matrix as per current invention provides following advantages
a) Faster Execution as only input pixel to output pixel mapping is required. Therefore, highly applicable for all real time applications. Streamed video can be displayed at the same rate after image warping operations.
b) Eliminate usage of mathematical equations and computations for conventional geometric transforms.
c) Floating point operations are totally eliminated.

d) Eliminate the need for additional high end hardware with floating point and math coprocessor.
e) Easy and convenient configuration with some smallexternal application along with some interface device (H/W).
While the embodiments of the present invention have been described with certain examples, those with ordinary skill in the art will appreciate that various modifications and alternatives to those details could be developed in the light of the overall teachings of the disclosure, without departing from the scope of the invention. For example, the system of the present invention may be used to correct images for various similar applications utilizing curvilinear surfaces and the transformation block may be utilized to perform similar image processing operations depending on the input image and the system requirement. As such, the examples used to illustrate the embodiments of the present invention, in no way limit the applicability of the present invention to them.
In an embodiment, the following is an example of working of the transformation block.
Inputs:
Image Size: 480x240 pixels
An example, one set of warping operations: [Rotate, Smile, Parallelogram, Push]

S.No Operation Parameter
1. Rotate Angle: 13°
2. Smile Direction: Both Shift: 75
2. Parallelogram Direction: Horizontal Shift: 80
3. Push Corner: TL Vertical Shift: -80 Horizontal Shift: -90

For a given input image, the individual geometrical operations like (rotate, smile/ frown etc) to be executed in a sequence on a target platform approximately takes thrice the time as compared to the method of the present invention, to execute a single transformation matrix containing the same set of geometric operations.

We Claim,
1. A system for real-time image/video correction for curvilinear surfaces comprising projection means (3) having a transformation block (302) to convert a given input image to a corrected output image.
2. The system for real-time image correction for curvilinear surfaces as claimed in claim 1 wherein, the said transformation block (302), on receiving an input transformation matrix corresponding to the received input image as processed by the processor (301), produces an output transformation matrix to be applied to the said input image to subsequently produce a corrected output image for projecting on the viewing means (I).
3. The system for real-time image correction for curvilinear surfaces as claimed in claim 1 wherein, the said transformation block (302) comprises of a rotation unit (3021), a parallelogram unit (3022), a smile-frown unit (3023) and a push-pull unit (3024).
4. The system for real-time image correction for curvilinear surfaces as claimed in claim 1 wherein, the use of transformation matrix reduces processor dependency by restricting the transformation operations to the said transformation block (302).
5. The system for real-time image correction for curvilinear surfaces as claimed in claim 1 wherein, geometric distortions introduced due to optical lenses are corrected by the said transformation block.
6. The system for real-time image correction for curvilinear surfaces as claimed in claim 1 wherein, the system has operational applications in vehicles for use on land, water and air such as aircrafts.

7. The system for real-time image correction for curvilinear surfaces as claimed in claim 1 wherein, the system generates a bird-eye view using inputs from multiple cameras.
8. The system for real-time image correction for curvilinear surfaces as claimed in claim 1 wherein, the system provides a flat-screen video projection on a curved viewing surface.