Claims:I/We claim
1. An Imaging System consisting of one visible camera (2) and one SWIR camera (3) which simultaneously capture the visible and SWIR images which is feed to the Image processing unit (1) which converts visible color image to YCbCr from RGB color space; the Luma(Y) component(22) and Chroma(CbCr) components(21) are separated, the edge component (17) from the visible gray is extracted and masked so that it is used in Image fusion(19) which takes to transform SWIR image data (18), visible grayscale image (16) and edge component (17) of the visible gray image and combine them in proportion to form fused luma image(19), the luma component(22) of original visible YCbCr image is replaced with fused luma image(19) to form the modified YCbCr image, which is converted to the visible RGB image(20) using YCbCr to RGB conversion such that the resultant image contains texture, color and SWIR properties.

2. The method of producing a fused image as claimed in claim 1 wherein Visible light spectrum (2); visible to human eyes consisting of blue, green and red spectrum ranging from 380nm to 740nm.

3. The method of producing a fused image as claimed in claim 1 wherein SWIR cameras (3) captures infrared components from the SWIR spectrum ranging from 900nm to 2500nm.

4. The method of producing a fused image as claimed in claim 1 wherein Image fusion takes transformed SWIR image (12) data in 40% to 90%, visible gray scale image (13) in 10% to 60% and edge component (17) of the visible gray image in 0% to 20% and combine them in such a way that the total image fusion proportion of first two components is 100%.

5. The method of producing a fused image as claimed in claim 1 wherein intensity correction is done to bring the average intensities of both images to the same level before the fusion process.

6. The method of producing a fused image as claimed in claim 1 wherein registration correction is performed on SWIR image (12) or visible image (11) or both to correct for misalignment in objects and background before the fusion process.

7. The method of producing a fused image as claimed in claim 1wherein the edge Information (17) can be extracted either from SWIR (12) or visible Image (11) based on the sharpness of the SWIR and visible image.

8. The method of producing a fused image as claimed in claim 1wherein HSV color space can be used by replacing the Luma component (22) with the Value (V) component of Hue-Saturation-value color space.
, Description:Title of the invention: -

Adding Color Information to SWIR Images

Background of the invention:-

Objects in Space have a tendency to absorb and reflect light which they are exposed to. Not all of the light which is reflected by the object is visible to human. But if imaged in multi spectral domain, it can be detected using specialized cameras. The visible light reflected by the object can be captured by visible camera. Where as a portion of Infrared light known as SWIR can be detected by SWIR cameras. As a result, both the cameras provide particular details. The motivation is to combine the benefits of both the SWIR and visible Images. Thus the details from both the spectrums can be reaped from a single image.

The SWIR and visible image fusion task is an important problem in the image processing field. It attempts to extract salient features from source images, and then these features are integrated into a single image by the appropriate fusion process. For decades, these fusion methods achieve extraordinary fusion performance and are widely used in many applications, like video surveillance and military applications. Firstly, the salient features are extracted by image decomposition methods. Then, an appropriate fusion strategy is utilized to obtain the final fused image.

Diagram 1
Light Spectrum

Electromagnetic Spectrum: -
The electromagnetic spectrum (refer diagram 1) or the light spectrum as described in the diagram 1can be bifurcated into various spectrums such as visible light, Infrared, microwave and etc. All such spectrums wavelengths possess special reflectance and refractance behavior. Owing to such behavior by various spectrums enables Imaging systems to use their wavelength characteristics to obtain imaging beyond human visible range.

Visible Light Spectrum: -
Visible light spectrum (refer diagram 1) is visible to human eyes and consists of blue, green and red spectrum. Information of all three spectrums gives us the color information. It ranges from 380nm to 740nm. Visible color cameras (2) are meant for capturing color components from the visible light spectrum.

SWIR Spectrum: -
SWIR is similar to visible light in which photons are reflected or absorbed by an object, providing a strong contrast for imaging. SWIR cameras (3) are meant for capturing infrared components from the SWIR spectrum. SWIR imaging is also similar to visible light where reflected light is used. SWIR spectrum ranges from 900nm to 2500nm. SWIR spectrum can cut through fog and hence can produce good images in foggy condition. SWIR cameras can be also utilized for night-time applications and also in scenarios where the visible light content is limited or in foggy conditions, thus providing extra contrast information.

Gray scale and Color Images: -
Visible images can be broadly classified into two categories. Grayscale (monochrome) images and color (true color) images. Grayscale images, the picture elements vary in shades of gray from pure white to black. On the other hand, the color images are made of small picture elements contains primary colors red, green, and blue.

RGB Color Space: -
Also known as primary colors because all other colors are generated using these primary colors mixed in varying proportions.

YCbCr Color Space: -
YCbCr is a practical approximation to color processing and perceptual uniformity, where the primary colors corresponding roughly to red, green and blue are processed into perceptually meaningful information.

Pixel of an image: -
The picture elements that make up an image are called a pixel. Hence a digital image can be considered as a special arrangement of these tiny dots called pixels. The shades of these pixels can be represented as intensity values ranging from 0 to 255 in case of an 8bit deep image.

Image Registration: -
Image registration (15) is a process of aligning two misaligned images. Images when captured from slightly different angles or positions may result in a different view or misalignment of objects in the image. The procedure is set to correct this misalignment in the images. In Diagram 2 image in left side is fused without proper registration, whereas image in right is fused after registration correction.

Diagram 2 Image Registration examples Source Image

Image Fusion: -
Combining two or more aligned images is known as image fusion. Image fusion provides features present in various images to be grouped and viewed in a single image.

Field of the Invention: -

The present invention relates to the color fusion of visible, contrast information of SWIR and edge information of Visible. The invention relates to acquiring and processing an image multi-spectrally. Visible Color Camera gives the color information, but lags in contrast. SWIR (Short Wave Infrared) camera doesn’t have color information, but has good contrast details and can cut through fog. The Objective of our invention is to combine color information from Visible Camera and contrast from SWIR Camera.

Description of related art:-

JP2006072401A
Titled: - Image compounding device and method
Discloses: -:A gray scale image acquiring means 11 acquires a gray scale image, and an RGB image acquiring means 12 acquires an RGB color image, and a luminance information color information separating means 13 separates the acquired RGB color image into luminance information and color information. An image compounding means 14 calculates composite luminance information by operating an inter-image arithmetic operation to the separated luminance information and the acquired gray scale image, and a luminance information color information compounding means 15 prepares a composite RGB image by compounding the calculated composite luminance information with the color information separated from the RGB color image, and a display means 16 displays the prepared composite RGB image.

Problem: - The cited patent discloses the principle of combining luminance of gray scale image with a luminance of color image without disturbing the color information. The cited patent tried making it general, but they were more focused on thermal. The decomposition method is used to separate RGB into luminance and Chrominance is different. The cited patent is using RGB to HIS and HIS to RGB mode.

Solution: - Our invention focuses on SWIR using RGB to YCbCr mode. It is to be noted RGB to HIS convertor are computationally more expensive and complicated than RGB to YCbCr convertors. The luminance component is not split into high and low frequency component, and our invention is uniform for all frequency components. In the fusion process (19), we would be also using edge information (17) from Visible Camera to improve the sharpness and texture information of output image.

US 8836793 B1
Titled: - True color night vision (TCNV) fusion.
Discloses: -True color images are produced by combining data collected from one or more color cameras with data collected from one or more infrared cameras. The produced images are the result of combining portions of the visible light data with portions of the infrared light data that may have been captured at dark, at daytime or at low light level conditions. These images appear as normal color images, with infrared information highlighted in a non-distracting fashion. The true color images that are produced in this fashion can also facilitate identification and reproduction of various objects that may not be visible or readily identifiable in infrared or false color imagery.

Problem: - The cited patent is more a generic patent for combining infrared and visible. It is basically based on edge detection of infrared images. Even though the invention mentions SWIR, the patent is more specific to thermal (TIR). Edges of infrared images are used as part of fused images.

Solution: - Our invention is more specific to SWIR which talks about merging and retaining contrast of SWIR, and same time adding color to the final merged image. In our invention Edges of visible are used to form a fused image. Our invention uses weighted Image fusion which can be used even for machine vision applications. Our approach has better texture as compared to the cited patent.

US 8711256 B2
Titled: - Image processing apparatus, image processing method, and program to create a composite image from color image data and monochrome image data
Discloses: - There is provided an image processing apparatus including an image acquisition unit that obtains color image data composed of wavelength components in a visible light region only, and monochrome image data which does not contain the wavelength components in the visible light region and which is composed of wavelength components in other than visible light region only, a color information extraction unit that extracts color information from the color image data, a luminance information extraction unit that extracts luminance information from the monochrome image data, and a synthesis unit that synthesizes the extracted color information and the extracted luminance information to generate composite image data.

Problem: - The cited patent is only limited to NIR and is restricted to combining color image (400nm to 700 nm wavelengths) and near infrared images (700 nm to 1000 nm). Also to be noted near infrared images does not help in providing good contrast images; they are more used for night only with help of an IR illuminator.

Solution: - Our invention is covering up to SWIR spectrum. Moreover, our invention is combining color (380 nm to 740 nm) with SWIR (900 nm to 2500 nm). Our invention uses SWIR image which provides a good contrast images which can be used even for machine vision applications also.

As seen, the above cited inventions suffer from critical drawbacks as mentioned. Our invention provides the solution of above problems by making possible to combine color information of Visible Camera and contrast of SWIR Camera. Visible Color Camera gives the color information, but lags in contrast. SWIR (Short Wave Infrared) camera doesn’t have color information, but has good contrast details and can cut through fog. The invention would be a process of combining Visible and SWIR images to get desired results. The image formed contain color information form Visible color image, edge content from visible gray image, gray content of both visible and SWIR image.

Summary of the Invention: -

Our invention is a process to add Color Information to SWIR Images. The problems of the prior art have been solved by our invention by combining the features of both the SWIR and visible Images.

The invention relates to a unique process of combining SWIR and Visible Color image. The most important aspect of our invention is to combine texture context, SWIR data and visible color information. Our invention is a new process of adding color information to the high contrast SWIR images. The process is carried out in YCbCr color space. In the proposed process, the fusion of visible (Visible spectrum is the spectrum, which is visible to human eyes. It consists of Blue spectrum, Green spectrum and Red spectrum) and SWIR image(SWIR is form of infrared light, but very similar to visible as imaging is done using reflected light.) is possible without any compromise on scene details.

Our invention registers SWIR and Visible Color images. Image registration is a process of aligning two misaligned images. Images when captured from slightly different angles or positions may result in a different view or misalignment of objects in the image. The procedure is set to correct this misalignment in the images. After registering images our invention converts visible color image from RGB mode to YCbCr mode. This would segregate the color component from Luma component. Then it would replace the Y channel of a color visible image with a weighted average of SWIR image and Y-channel of color images and edge information of Y-channel of color image and at last convert the modified YCbCr color image to RGB image. Thus the details from both the spectrums can be reaped from a single image.

Our invention is useful for Electro Optics Systems having both SWIR and Visible Camera Core. Applicability of our invention would be for industries like defense, Industrial Vision and Pharmaceutical. Fusion of SWIR Image and Visible color Image forming a composite image which would be a self descriptor of the contrast in the scene or object while preserving its color.

Brief description of the drawing(s): -

Our invention is a process to combine the benefits of both the SWIR and visible Images. Thus the details from both the spectrums can be reaped from a single image.

Figure 1: - In accordance to Figure 1, Imaging System used consists of one visible camera (2) and one SWIR camera (3) connected to the Image processing unit (1) This system would be able to capture the visible and SWIR data simultaneously. The Image data captured would be then fed to the Image processing Unit (1).

Figure 2: - Image processing unit (refer Figure 2 Image processing Flow Diagram) first captures the image data from both the sensors. The visible color image (11) would be converted to a grayscale image (13) for registration and edge content extraction. For effective registration the intensity of visible gray and SWIR image data should be near about same else registration would fail in capturing common features. Intensity correction can even be carried out on SWIR Image.

Registration (15) is performed on the images obtained from a SWIR camera (12) and intensity corrected visible gray. SWIR imaging is transformed as per the visible gray. Visible color image is converted to YCbCr from RGB color space.

The Luma(Y) component (22) and Chroma (CbCr) components (21) are separated (16) and used independently ahead. Simultaneously, the edge component (17) from the visible gray is extracted and masked so that it could be used in Image fusion. Image fusion (19) takes to transform SWIR image data (18), visible grayscale image (16) and edge component (17) of the visible gray image and combine them in certain proportion to form fused luma image(19), the luma component(22) of original visible YCbCr image is replaced with fused luma image(19) to form the modified YCbCr image, which is converted to the visible RGB image(20) using YCbCr to RGB conversion such that the resultant image contains texture, color and SWIR properties.

Detailed Description of the invention:-

The present invention relates to the process to combine the benefits of both the SWIR and visible Images. Our invention is a process of adding color information to the high contrast SWIR images. The process is carried out in YCbCr color space. In the proposed process, the fusion of visible (Visible spectrum is the spectrum, which is visible to human eyes. It consists of Blue spectrum, Green spectrum and Red spectrum) and SWIR image (SWIR is form of infrared light, but very similar to visible as imaging is done using reflected light.) is possible without any compromise on scene details. Image Fusion includes combining texture context, SWIR data and visible color information.

Visible light spectrum is visible to human eyes and consists of blue, green and red spectrum. Information of all three spectrums gives us the color information. It ranges from 380nm to 740nm. Visible color cameras (2) are meant for capturing color components from the visible light spectrum.

SWIR is similar to visible light in which photons are reflected or absorbed by an object, providing a strong contrast for imaging. SWIR cameras (3) are meant for capturing infrared components from the SWIR spectrum. SWIR imaging is also similar to visible light where reflected light is used. SWIR spectrum ranges from 900nm to 2500nm. SWIR cameras may be utilized for nighttime applications and also in scenarios where the visible light content is limited, thus providing extra contrast information.

The Configuration and Operation of an Embodiment: -

In accordance to Figure 1, Imaging System used consists of one visible camera (2) and one SWIR camera (3) connected to the Image processing unit (1) This system would be able to capture the visible and SWIR data simultaneously. The Image data captured would be then fed to the Image processing Unit (1).

Image processing unit (refer Figure 2) first captures the image data from both the sensors. The visible color image (11) would be converted to a grayscale image (13) for registration and edge content extraction. For effective registration the intensity of visible gray and SWIR image data should be near about same else registration would fail in capturing common features. Intensity correction can even be carried out on SWIR Image.

Registration (15) is performed on the images obtained from a SWIR camera (12) and intensity corrected visible gray. Image registration (15) is a process of aligning two misaligned images. Images when captured from slightly different angles or positions may result in a different view or misalignment of objects in the image. The procedure is set to correct this misalignment in the images.

SWIR imaging is transformed as per the visible gray. Visible color image is converted to YCbCr from RGB color space.

When coding images our invention usually doesn’t use the RGB color space. Instead the image is described in another color space, where the pixel values are given using a luminance (or Luma) component (called Y), that tells us how bright the pixel is (i.e. basically a grayscale signal) and two chrominance (or chroma) components (called Cb and Cr) that tells the actual color of the pixel. The chrominance components can often be down sampled to a lower resolution, without a human observer noticing any reduction in image quality. There are many variants of luminance-chrominance color spaces, but they are rather similar to each other.

The Example of converting RGB into YCbCr: -

Suppose that ER, EG and EB are analog values between 0 and 1 that describe how much red, green and blue there is in a pixel (given eight bit quantization we have ER = R/255, EG = G/255 and EB = B/255). A typical conversion (ITU-R Recommendation 624-4 System B, G) to luminance-chrominance is then given by: -

{EY = 0.299 • ER + 0.587 • EG + 0.114 • EB
{ECb = -0.169 • ER - 0.331 • EG + 0.500 • EB
{ECr = 0.500 • ER - 0.419 • EG - 0.081 • EB

Where EY is between 0 and 1 and ECb and ECr are between -0.5 and 0.5. Conversions for 8-bit values are then done by: -

{Y = 219 • EY + 16
{Cb = 224 • ECb + 128
{Cr = 224 • ECr + 128

The Luma(Y) component (22) and Chroma (CbCr) components (21) are separated (16) and used independently ahead. Simultaneously, the edge component (17) from the visible gray is extracted and masked so that it could be used in Image fusion. Image fusion (19) takes to transform SWIR image data (18), visible grayscale image (16) and edge component (17) of the visible gray image and combine them in certain proportion to form fused luma image(19), the luma component(22) of original visible YCbCr image is replaced with fused luma image(19) to form the modified YCbCr image, which is converted to the visible RGB image(20) using YCbCr to RGB conversion such that the resultant image contains texture, color and SWIR properties.

Our invention is useful for Electro Optics Systems having both SWIR and Visible Camera Core and also as an independent Process application. Industrial applications include defense, Industrial Vision and Pharmaceutical. Fusion of SWIR Image and Visible color Image to form a composite image which would be self descriptory of the contrast in the scene or object while preserving its color. Process developed can be used to enhance the end user experience.

Results of our invention on different appliances: -


Diagram 1
Diagram 1 discloses Result on Consumer Applications.

Diagram 2
Diagram2 discloses Result on Industrial Applications include use in defense, anti-counterfeiting, solar cell inspection, identifying and sorting and much more.

Diagram 3
Diagram 3 discloses Result on Consumer Applications.


Diagram 4
Diagram 4 discloses Result on Consumer Applications.

Diagram 5
Diagram 5 discloses Result on Industrial Applications include use in defense, anti-counterfeiting, solar cell inspection, identifying and sorting and much more.

Example: -

Case 1 – Innovation used as part of electro-optic system

• “SWIR Visible combiner” is the main novelty. This can be in the form of either a software running on a PC or firmware running on an electronic board which may be part of overall electro-optic system
• Here the electro-optic system consists of both SWIR camera and Visible camera
• The output of both the cameras will be combined into a single image using “SWIR Visible Combiner”

Case 2– Innovation used it independent.

• Capture SWIR and Visible images independently
• Feed the images to “SWIR-Visible Combiner” software
• This would generate a SWIR Color image (i.e. highly contrast color image)
Claims: -

I/We claim
1. An Imaging System consisting of one visible camera (2) and one SWIR camera (3) which simultaneously capture the visible and SWIR images which is feed to the Image processing unit (1) which converts visible color image to YCbCr from RGB color space; the Luma(Y) component(22) and Chroma(CbCr) components(21) are separated, the edge component (17) from the visible gray is extracted and masked so that it is used in Image fusion(19) which takes to transform SWIR image data (18), visible grayscale image (16) and edge component (17) of the visible gray image and combine them in proportion to form fused luma image(19), the luma component(22) of original visible YCbCr image is replaced with fused luma image(19) to form the modified YCbCr image, which is converted to the visible RGB image(20) using YCbCr to RGB conversion such that the resultant image contains texture, color and SWIR properties.

2. The method of producing a fused image as claimed in claim 1 wherein Visible light spectrum (2); visible to human eyes consisting of blue, green and red spectrum ranging from 380nm to 740nm.

3. The method of producing a fused image as claimed in claim 1 wherein SWIR cameras (3) captures infrared components from the SWIR spectrum ranging from 900nm to 2500nm.

4. The method of producing a fused image as claimed in claim 1 wherein Image fusion takes transformed SWIR image (12) data in 40% to 90%, visible gray scale image (13) in 10% to 60% and edge component (17) of the visible gray image in 0% to 20% and combine them in such a way that the total image fusion proportion of first two components is 100%.

5. The method of producing a fused image as claimed in claim 1 wherein intensity correction is done to bring the average intensities of both images to the same level before the fusion process.

6. The method of producing a fused image as claimed in claim 1 wherein registration correction is performed on SWIR image (12) or visible image (11) or both to correct for misalignment in objects and background before the fusion process.

7. The method of producing a fused image as claimed in claim 1wherein the edge Information (17) can be extracted either from SWIR (12) or visible Image (11) based on the sharpness of the SWIR and visible image.

8. The method of producing a fused image as claimed in claim 1wherein HSV color space can be used by replacing the Luma component (22) with the Value (V) component of Hue-Saturation-value color space.

For, and on behalf of the Applicant
Harpreetsingh Banker
Patent Agent (IN/PA: 2131)
Abstract:-

Our invention Titled “Adding Color Information to SWIR Images” is a process to combine the benefits of the SWIR (12) and Visible images (11). Our invention is a new process of adding color information to the high contrast SWIR images. The process is carried out in YCbCr color space. Our invention will obtain SWIR image (12) and visible color image (11) and then will register both the images without any compromise on scene details. Converting RGB image to YCbCrand then extract Y component (22) of the image and replace it with the weighted average of SWIR (18) and Y channel image and at last regenerate RGB image from YCbCr image formed. Image fusion includes combining texture context, SWIR data and visible color information. Our invention is useful for Electro Optics Systems and also as an independent Process application. Industrial applications include defense, Industrial Vision and Pharmaceutical.

Figure of abstract.