Claims:I/We claim
1. An imaging system consisting of a visible(003) and a thermal(004) image as the inputs which are obtained simultaneously from visible(001) and thermal(002) cameras respectively which are combined in an image combiner(005) to generate the fused output(006); thermal image(004) is converted into False color-coded thermal image(107); the False color coded thermal image is then converted into HSV color space to extract the V component(108); V component which is extracted from False color coded infrared image is mixed with the intensity corrected greyscale visible image (105); the ratio of mixing (109) ensures that the scene details are visible in the resultant image and this fused image is the new value component (VNEW) which is then combined with the original H and S components and converted back to RGB color space (110) and the resultant image is the final fused RGB image.

2. The method of producing a fused image as claimed in claim 1 wherein when the visible image is in color format then it is converted from color (103) to gray scale(104);

3. The method of producing a fused image as claimed in claim 1 wherein the ratio of mixing (109) ranges from 20% to 50% (x%) of the V component which is extracted from false color coded infrared image and the intensity corrected (100-x) % of greyscale visible image (105).

4. The method of producing a fused image as claimed in claim 1 wherein Visible light spectrum ranging from 380nm to 740nm which is visible to human eyes and consists of blue, green and red spectrum.

5. The method of producing a fused image as claimed in claim 1 wherein LWIR (generally referred to as Thermal spectrum) is part of infrared light, which is used for imaging thermal emission which ranges from 8000 nm to 15000 nm.

6. The method of producing a fused image as claimed in claim 1 wherein intensity correction is done to bring the average intensities of both images to the same level.

7. The method of producing a fused image as claimed in claim 1 wherein registration correction is performed on either visible image or thermal image or on both images to correct for misalignment in objects and background before the fusion process.

8. The method of producing a fused image as claimed in claim 1 wherein the method of applying fusion can be also restricted to only objects of interest by selecting the object region based on temperature or equivalently intensity of infrared image thus displaying objects of interest in fused view, and rest of the image will be displayed in visible mode.
, Description:Title of the invention: -

Fusing Visible and Thermal image by preserving scene details.

BACKGROUND OF THE INVENTION: -

Every object emits its heat signature in the form of infrared radiation. But human eyes cannot detect these radiations. It can be detected using a thermal camera. Thermal cameras are designed to capture the information from the thermal spectrum. In a sense, the temperature emitted by an object can be captured using these cameras. These cameras are also used for capturing night conditions. In short, the information obtained from the thermal image cannot be provided by the visible spectrum image. Also, the visible image details of the scene cannot be obtained from the thermal image. The motivation is to combine the benefits of both the thermal and visible images. Thus, the details from both the spectrums can be reaped from a single image at once.

This section briefs the basic terms, terminologies, and descriptions in image processing that are relevant to understanding the innovation.

Visible and Thermal spectrum
The visible spectrum ranges from 380nm to 750nm. Human eyes can only perceive this part of the electromagnetic spectrum. Images that an eye can see are called visible images. Refer to Fig 2.1 for the spectrum details.

Diagram 1

Types of images
Visible images can be broadly classified into two categories. Gray scale (monochrome) images and colour (true colour) images. Gray scale images (Diagram 2.2), the picture elements vary in shades of gray from pure white to black. On the other hand, the colour images (Diagram 2.1) are made of small picture elements contains primary colours red, green, and blue. These are called primary colours because all other colours can be generated using these primary colours mixed in varying proportions.

Diagram 2.1 Diagram 2.2

Picture elements (Pixels)
The picture elements that make up an image are called a pixel. Hence a digital image can be considered as a special arrangement of these tiny dots called pixels. The shades of these pixels can be represented as intensity values ranging from 0 to 255 in case of an 8bit image.

Color Spaces
Colour images can be represented in many colour spaces. The commonly used colour space is the RGB colour space. This is suitable for electronic display purposes. In this format image is represented in its primary color format. Each pixel will have contribution of three channels Red, Green and Blue respectively. The other colour spaces are the HSV (Hue, Saturation, Value), and the YCbCr (Luma, Component blue, Component red) colour space.
In HSV colour space, each pixel of the image is represented by three indicators namely Hue, Saturation and Value.
• Hue is the color information of pixel specified in an angle between 0o to 360o.
• Saturation is the level of color applied. Higher the saturation higher the impact of color and vice-versa.
•Value is the intensity value of the pixel equivalent to grey value. This color space is also referred as HSI (Diagram 3).

Diagram 3
In YCbCr format the color information is segregated from grey information. The Y channel represents the grey information. The Cb and Cr provide contribution of blue channel and red channel respectively. The Y component is equivalent to V component of HSV, which is similar to gray scale image of the RGB colour space. Conversion from one colour space to another is possible with the help of standard software techniques.

Thermal Spectrum
Coming to the infrared spectrum, the thermal information is in the LWIR spectrum (8000nm to 15000nm) which is part of the IR spectrum. These are not visible to human eyes. But can be captured using a thermal camera in the form of a grayscale image. Thermal spectrum captures temperature emitted by object. The pixel value of thermal image would be proportional to temperature of the object at that particular point.

False Color coding
While displaying thermal images, they are usually converted into a color scale. This method of conversion is called false-color conversion. Most commonly Blue is considered to be low temperature, and yellow is considered to be high temperature. So low intensity or low temperature value of thermal is mapped to blue and high intensity or high temperature is mapped to yellow. Intermediate intensity values are generally mapped to purple; red and orange (Diagram 4).

Diagram 4
Image Registration
Image registration or just registration in image processing is a procedure for aligning two images. Images when taken from slightly different angles or positions may result in misalignment of objects in the image. The registration procedure will correct the misalignment. Standard registration process are available based on the level of correction needed. Registration is an important step in the merging two images. Without registration, merged image can create undesirable artifacts on the image. Refer Diagram 5.

Diagram 5

Brief on what a Video is
A sequence of images displayed in succession forms a video. Each image in a video is called a frame. The number of frames that make up a one-second video is called its frame rate or frames per second (FPS). Refer Diagram 6.

Diagram 6
Field of the Invention: -

The present invention relates to the color fusion of visible and thermal images. Our invention is a process to combine information from visible spectrum and thermal spectrum. The process is achieved through combining gray scale image of visible spectrum with color information of false color-coded thermal image. Our invention provides a single image to infer both thermal and visible information. Temperature can be perceived from the image itself and texture and contrast details will be retained to large extent. Information like patterns, edges, contrast and texture from visible spectrum along with temperature information from Thermal spectrum will be present in the final image. Our invention can be applied to medical grade thermal scanners, surveillance cameras, cameras used for heating/cooling troubleshooting in industries etc.

Description of related art: -

Patent: US7620265B1
Titled: -Color invariant image fusion of visible and thermal infrared video.
Disclosure: -A methodology for forming a composite color image fusion from a set of N gray level images takes advantage of the natural decomposition of color spaces into 2-D chromaticity planes and 1-D intensity. This is applied to the color fusion of thermal infrared and reflective domain (e.g., visible) images whereby chromaticity representation of this fusion is invariant to changes in reflective illumination.

Problem:-In this invention cited patent are generating Hue and Saturation from Infrared image, whereas value is generated from a combination of Grey channel of visible image and Infrared image. Saturation value would be proportional to infrared image, which would result in fading of color information for lower temperature objects or background.
Solution: - Our invention uses a fixed value for saturation (can lie between 0.5 to 1), so the color would be visible when infrared intensity is lower. Approach for computing intensity in our invention’s approach is simpler and would be less CPU intensive.

Way of mapping value field is different as shown below.
US7620265B1: Out Value (Intensity) = VIS’ + (1-ß)*VIS’*IR’.
Note: VIS’, IR’ are visible and infrared inputs, ß is parameter specified
Our Invention: Out Value (Intensity) = a*VIS’ + (1-a)*IR’
Note: VIS’, IR’ are visible and infrared inputs, a is parameter specified lying between 0 to 1.

Patent: US20180300906A1
Titled: -Methods and systems for fusion display of thermal infrared and visible image
Disclosure: -A method relating to an image fusion includes acquiring a thermal infrared image and a visible image. The method also includes receiving a fusion parameter corresponding to a color space and generating, based on the fusion parameter, a fused image of the thermal infrared image and the visible image. The method further includes receiving a regulation parameter, the regulation parameter including a color scheme or a partial contrast, and adjusting the fused image according to the regulation parameter.

Problem: - Hue value determined based on Infrared image. Saturation is taken based on combination of Infrared and Visible. Value (Intensity) is taken fully from Visible.
Solution: - Our invention uses fixed value of Saturation. For Value (Intensity) our invention takes linear sum of Infrared and Visible.
Since our invention taking parameterized linear sum (parameter = a) for Value (Intensity) contribution of Infrared or Visible can be tuned. For higher contribution of visible, a can be set to close to 1, similarly for higher contribution of Thermal a can be set to close to 0. Our approach of saturation computation is simpler as it is a constant value.

Patent: US8836793B1
Titled: - True color night vision (TCNV) fusion.
Disclosure: -True color images are produced by combining data collected from one or more color cameras with data collected from one or more infrared cameras. The produced images are the result of combining portions of the visible light data with portions of the infrared light data that may have been captured at dark, at daytime or at low light level conditions. These images appear as normal color images, with infrared information highlighted in a non-distracting fashion. The true color images that are produced in this fashion can also facilitate identification and reproduction of various objects that may not be visible or readily identifiable in infrared or false color imagery.

Problem: - The cited patent suggest to Extract Luma and chroma from color visible image. Perform edge detection on Infrared image. Add extracted information to Luma of visible image. Combined the Luma and Chroma to generate fused image. The final output would not display the temperature information as color information (Hue and Saturation) is picked up from only visible spectrum. Hence, the applicability of Patent would be limited to night Surveillance only.
Solution: - Our invention doesn’t involve edge detection. In our invention Chroma (Hue) value is picked from false color code Infrared image. In this scenario it is picked from visible image and also in our invention temperature information can be viewed in form of color.

Above inventions suffer from critical drawbacks as mentioned our invention provides the solution to above problems by simplifying the process of combining visible and thermal spectrums and retaining best of both. The process is achieved through combining gray scale image of visible spectrum with color information of false color-coded thermal image. The processing is carried out in HSV color space. Our objective is to combine information from both the spectrums and obtain Patterns, edges, contrast and texture from visible spectrum and Temperature information from Thermal spectrum.

Summary of the Invention: -

Our invention is a process of Fusing Visible and Thermal image by preserving scene details. The problems of the prior art have been solved by our invention by simplifying the process of combining visible and thermal spectrums and retaining best of both.

In the proposed process, the fusion of visible and thermal images with negligible compromise on scene details is possible. The ratio of blending the images can be controlled easily according to the viewer's preference. The gray single channeled thermal image is false color-coded to form an RGB three channeled image. The image is then mapped into HSV color space. The V component of the image is modified by mixing it with the gray image obtained from the visible image. The modified HSV image is then converted back to the RGB image which will be the final fused image.
Our objective is to combine information from both the spectrums. Patterns, edges, contrast and texture from visible spectrum along with temperature information from Thermal spectrum would be retained in the final image. The process is achieved through combining gray scale image of visible spectrum with color information of false color-coded thermal image. Colors will indicate the temperature as brighter color (Yellow) indicating warm area and darker Color (Blue) indicating cold region.

In general, our invention would be useful for products having both thermal and visible camera. A single image is only required to get the details about the scene and the temperature information. Our invention is feasible both with visible color and gray images. Our invention is also suitable if the thermal image is available in grayscale or false color form. Our invention can also run as an independent invention add-on to any image processing platform.

The scope of this innovation is spread across multiple domains such as consumer products, medical, industry, defense, etc.
• For example:-Medical grade thermal scanners such as COVID thermal Scanners are one such application. With this innovation it is possible to see both the temperature info and the facial features of the person from a single image.
• A camera incorporated with this feature will be very useful in low light or even night surveillance.
• Other applications such as fire detection, heating-cooling troubleshooting, etc. raise the scope of this innovation.

Brief description of the drawings: -

Figure 1: - The diagrammatic representation of this arrangement is shown in Figure 1. The invention takes two images, a visible (003) and a thermal (004) image as the inputs. These two images are obtained from visible (001) and thermal (002) cameras respectively that are arranged in a single unit. The images are combined in an image combiner (005) to generate the output (006).

Figure 2: - The complete block diagram representation of the process is shown in Figure 2. Assigning colors to this gray thermal image (106) on the base of intensities is called false color coding. A false color-coded thermal image (107) can also be fed as input to the algorithm. Similarly, the visible image can also be color (103) or gray scale (104). The thermal image is then converted into HSV color space to extract the V component (108). This can be done using standard conversion techniques used in image processing. Our invention aims to preserve the scene details by introducing more information into the false color-coded thermal image. This is done by mixing the V component with the intensity corrected greyscale visible image (105). The intensity correction is done to bring the average intensities of both images to the same level. The ratio of mixing (109) ranges from x% of the V component which is extracted from False color coded infrared image and the intensity corrected (100-x) % of greyscale visible image (105). (For getting the best fused results prescribed value of x can be between 20% to 50%, but in practice it can be any value between 0% to 100% which would be user or application configurable.) And this ratio of mixing (109) ensures that the scene details are visible in the resultant image. This blended image is the new value component (VNEW). VNEW is then combined with the original H and S components and converted back to RGB colour space (110). The resultant image is the final fused RGB image.

Detailed description of the invention: -

The present invention is a process to combine the benefits of visible and thermal spectrum. The processing is carried out in HSV color space. In the proposed method, the fusion of visible and thermal images with negligible compromise on scene details is possible. The ratio of blending the images can be controlled easily according to the viewer's preference. The gray single channeled thermal image is false color-coded to form an RGB three channeled image. The image is then mapped into HSV color space. The V component of the image is modified by mixing it with the gray image obtained from the visible image. The modified HSV image is then converted back to the RGB image which will be the final fused image.

The process is achieved through combining gray scale image of visible spectrum with color information of false color-coded thermal image.

The configuration and operation of this process: -

The diagrammatic representation of this arrangement is shown in Figure 1. The invention takes two images, a visible (003) and a thermal (004) image as the inputs. These two images are obtained from visible (001) and thermal (002) cameras respectively that are arranged in a single unit. The images are combined in an image combiner (005) to generate the output (006).

The complete block diagram representation of the process is shown in Figure 2. Assigning colors to this gray thermal image (106) on the base of intensities is called false color coding. A false color-coded thermal image (107) can also be fed as input to the algorithm. Similarly, the visible image can also be color (103) or gray scale (104). The thermal image is then converted into HSV color space to extract the V component (108). This can be done using standard conversion techniques used in image processing. Our invention aims to preserve the scene details by introducing more information into the false color-coded thermal image. This is done by mixing the V component with the intensity corrected greyscale visible image (105). The intensity correction is done to bring the average intensities of both images to the same level. The ratio of mixing (109) ranges from x% of the V component which is extracted from False color coded infrared image and the intensity corrected (100-x) % of greyscale visible image (105). (For getting the best fused results prescribed value of x can be between 20% to 50%, but in practice it can be any value between 0% to 100% which would be user or application configurable.) and this ratio of mixing (109) ensures that the scene details are visible in the resultant image. This blended image is the new value component (VNEW). VNEW is then combined with the original H and S components and converted back to RGB color space (110). The resultant image is the final fused RGB image. In final fused image Brighter color (Yellow) indicating warm area and Darker Color (Blue) indicating cold region.

The objective of our invention is to combine information such as Patterns, edges, contrast and texture from visible spectrum and Temperature information from Thermal spectrum.

In our invention Single image is enough to infer both thermal and visible information. Temperature can be perceived from the image itself and Texture and contrast details will be retained to large extent. Our invention would be even applicable if a visible image is in grayscale. In general this would be useful for products having both thermal and visible camera.

Key points of Innovation
The innovation enables us to do the following: -
• It is possible to apply fusion on the entire image
• It is also possible to apply fusion only to the object of interest by selecting the object region based on temperature (intensity of infrared image). Areas of interest will only be displayed with the fused image pixels; the rest of the image will be displayed in visible.
• A single image is only required to get the details about the scene and the temperature information.
• The innovation is feasible both with visible color and gray images.
• The innovation is feasible if the thermal image is available in grayscale or false color form.
• The innovative process can run independently or serves an add-on to any other image processing platform.

The application of our invention is spread across multiple domains such as consumer products, medical, industry, defense, etc.
Examples of application: -

• Medical grade thermal scanners such as COVID thermal Scanners are one such application. With this innovation it is possible to see both the temperature info and the facial features of the person from a single image.
• A camera incorporated with this feature will be very useful in low light or even night surveillance.
• Other applications such as fire detection, heating-cooling troubleshooting, etc. raise the scope of this innovation.

Results of our invention: -

Diagram 1
Diagram 1 discloses fused image of our invention on entire image. This would be an example image for Medical Thermal Scanners.

Diagram 2
Diagram 2 shows fusion applied only on area of interest.

Diagram 3
Diagram 3 discloses fused image of our invention on entire image. This would be an example usage in industrial applications.

Examples of use cases: -

Use Case 1 – Innovation used as part of electro-optic system

• “Thermal Visible Fusion” model is the main novelty. This can be in form of either a software running on a PC or firmware running on a electronic board which may be part of overall electro-optic system
• Here the electro-optic system consists of both Thermal camera and Visible camera
• The output of both the cameras will be combined into a single image using “Thermal Visible Fusion” model
• Use case 1 can be either of the cases
• Covid Thermal Scanner
• Surveillance Camera
• Machine Vision based Camera
• Cameras for monitoring heating and cooling systems.

Use case II - The innovation as independent invention.

• Capture Thermal and Visible images independently
• Feed the images to “Thermal Vision” software
• This would generate the fused image
• The application can run either on a PC or cloud-based application.

Claims: -

I/We claim
1. An imaging system consisting of a visible(003) and a thermal(004) image as the inputs which are obtained simultaneously from visible(001) and thermal(002) cameras respectively which are combined in an image combiner(005) to generate the fused output(006); thermal image(004) is converted into False color-coded thermal image(107); the False color coded thermal image is then converted into HSV color space to extract the V component(108); V component which is extracted from False color coded infrared image is mixed with the intensity corrected greyscale visible image (105); the ratio of mixing (109) ensures that the scene details are visible in the resultant image and this fused image is the new value component (VNEW) which is then combined with the original H and S components and converted back to RGB color space (110) and the resultant image is the final fused RGB image.

2. The method of producing a fused image as claimed in claim 1 wherein when the visible image is in color format then it is converted from color (103) to gray scale(104);

3. The method of producing a fused image as claimed in claim 1 wherein the ratio of mixing (109) ranges from 20% to 50% (x%) of the V component which is extracted from false color coded infrared image and the intensity corrected (100-x) % of greyscale visible image (105).

4. The method of producing a fused image as claimed in claim 1 wherein Visible light spectrum ranging from 380nm to 740nm which is visible to human eyes and consists of blue, green and red spectrum.

5. The method of producing a fused image as claimed in claim 1 wherein LWIR (generally referred to as Thermal spectrum) is part of infrared light, which is used for imaging thermal emission which ranges from 8000 nm to 15000 nm.

6. The method of producing a fused image as claimed in claim 1 wherein intensity correction is done to bring the average intensities of both images to the same level.

7. The method of producing a fused image as claimed in claim 1 wherein registration correction is performed on either visible image or thermal image or on both images to correct for misalignment in objects and background before the fusion process.

8. The method of producing a fused image as claimed in claim 1 wherein the method of applying fusion can be also restricted to only objects of interest by selecting the object region based on temperature or equivalently intensity of infrared image thus displaying objects of interest in fused view, and rest of the image will be displayed in visible mode.

For, and on behalf of the Applicant
Harpreetsingh Banker
Patent Agent (IN/PA: 2131)

Abstract:-

Our invention Titled “Fusing Visible and Thermal image by preserving scene details” is a process of combining the images in the visible (103) and thermal (106) spectrums. The main aspect of our invention is to combine information such as Patterns, edges, and texture from visible spectrum and Temperature information from Thermal spectrum. The processing is carried out in HSV color space. The ratio of blending the images can be controlled easily according to the viewer's preference. The gray thermal image is false color-coded (107) to form an RGB three channeled image. The image is then mapped into HSV color space. The V-component (108) of the image is modified by mixing it with the gray image obtained from the visible image (104). The modified HSV image is then converted back to the RGB image which will be the final fused image.

Figure of abstract.