The present invention relates to the degraded image restoration. More particularly, the present invention relates to a system and method for fog/haze removal from visible and near-infrared spectrum image or video.
5 BACKGROUND OF THE INVENTION
[002] Smog is a common atmospheric phenomenon in which dust, smoke and other dry particles will reduce the transparency of the atmosphere. Light needs to pass through the dense atmosphere to image distant objects. Due to the influence of light scattering from haze particles, the contrast of images taken by general 10 imaging systems is very low and the visibility is poor. In particular, it has a huge impact on automatic driver assistance systems and road surveillance cameras.
[003] Dehazing has always been a very challenging problem, because the concentration of fog is different when the object is at different depths. Therefore, the image enhancement techniques such as histogram equalization are not 15 effective for image defogging.
[004] Based on the statistics of fog-free images and foggy images, Dr. Kaiming He found that among the three RGB color channels of each image, there is always one channel whose gray value is very low, almost tending to zero. According to this rule, the dark channel prior algorithm was invented, which is simple and easy 20 to use, and is widely employed in the field of image defogging. However, this algorithm is very computationally intensive and will fail when calculating sky fog images. In addition, it cannot be applied tonear-infrared (NIR) band of the electromagnetic spectrum.
[005] Reference may be made to the following:
25 [006] Publication No. CN 110992293 relates to a self-adaptive video defogging method and device. This method utilized an infrared distance measuring module to estimate visibility of the environment and a camera module to capture the video

and a processing chip if the visibility lies in the predefined range. On the other hand, the proposed invention has dual spectrum camera setup with a mode selector that decides which video feed should pass to the processor and display. The dehazing is performed in visible as well as NIR spectrum that makes this 5 invention suitable for a wide range of fog conditions. In addition, it can be accommodated on single FPGA/ASIC chip.
[007] Indian Application No. 202011029506 relates to a real time de-hazing device, comprising; a video capturing module for capturing polluted/ hazed video; plurality of video frames for estimating intensity. This approach used transitional
10 convolutional neural network that consist of feature extraction, feature enhancement and transition layers. The approach is more resource hungry and does not map the physical model of scattering and is not designed for NIR imaging. Moreover, it is not self-adaptive to the flickers in the video frames. On the other hand, the proposed approach is self-adaptive to flickers, consistent with
15 physical model of scattering and suitable for dedicated hardware design such as FPGA/ASIC. Moreover, it is cable to restore both visible and NIR band image frames.
[008] Indian Application No. 201811027075 relates to a method for dehazing images and/or videos in real time based on Color Uniformity Principle (CUP).
20 The method receives by an image acquisition system (100), an image or video having one or more hazy regions. This method presented a prior based image/video dehazing approach. The transmission map is estimated using a prior based color uniformity principle (CUP), later this transmission map is utilized to estimate the atmospheric light. The estimated transmission and airtight were
25 employed to restore image using physical model of scattering. The restored frames were also enhanced using histogram equalization technique. The approach is a prior based and is not self-adaptive to estimated airtight. Moreover, airtight can be estimated only after the transmission map is estimation that restrict the parallelism and increase processing time. On the other hand, the present disclosure
30 is an end-to-end learning approach that estimated and adapt transmission map and

airtight for the best possible image quality. The proposed technique estimates transmission map and airtight simultaneously that reduces processing time and makes it suitable for dedicated parallel architectures such as FPGA/ASIC.
[009] Publication No. CN110544216 relates to a real-time video defogging 5 system based on deep learning. This method utilized an end-to-end multiscale convolutional neural network based image dehazing approach for video frames dehazing. The approach first segmented foreground and background. The background is down sampled using the bicubic interpolation that is followed by dehazing and upsampling. Finally, foreground and background fusion is 10 performed. On the other hand, the proposed method does not require segmentation and down/up sampling. Therefore, less computational resources and processing time are required. Hence, the presented invention is suitable for real-time embedded deployment. Moreover, it is a multispectral dehazing/defogging approach that enables it to restore a wide range of foggy images and videos.
15 [010] Publication No. CN109978799 relates to a deep learning-based method for defogging of maritime drone video images, which can effectively solve the blurring of video images acquired by the drone vision system under foggy conditions. This method is based on a multiscale convolutional neural network that learns an intermediate variable to restore the image using scattering model of
20 dehazing. The approach may not manage video flickering artifacts and hardware deployment may be resource hungry. On the other hand, the proposed method is suitable to handle flickering artifacts in the video frames. The proposed design is also highly suitable for lightweight parallel dedicated hardware (FPGA/ASIC) deployment. In addition, it works for visible as well as NIR spectrum.
25 [011] Publication No. CN 106600560 relates to an image defogging method for an automobile data recorder. With a quad-tree method, an atmospheric light value is obtained by calculation; a rough transmissivity map is obtained by using a trained convolution neural network and the transmissivity map is optimized by using a guidance filtering method; and then inverse solution is carried out by using an

atmospheric scattering model to obtain a restored image. This approach estimated airtight using a quadtree based method and estimated transmission map using convolutional neural network that is refined using guided filtering and then the image frames are restored using scattering model. On the other hand, the proposed 5 approach is an end-to-end solution that learn the feature for the input image and produces restored image directly. The method also adapt flickers in the video frames for better temporal consistency and designed for wide spectrum defogging.
[012] Publication No. US9508129 relates to methods and systems for dehazing images with increased accuracy and reduced error enhancement. This method 10 performs a piecewise smoothing on the estimated coarse transmission map to reduce the artifacts in the dehazed image. The method uses visible band spectrum of electromagnetic (EM) radiations for image dehazing.On the other hand, the present invention utilized multispectral data, namely: visible band and near-infrared band pixels, to remove haze from the captured images and videos.
15 [013] Publication No. US9189830 relates to an image defogging method and system. This method uses minimum intensity in the color image and edge-preserving filtering to estimate the transmission map and the airtight. The method can be considered as a conventional approach, because prior based approach is utilized for image dehazing. On the other hand, the present invention uses data
20 driven approach that automatically learn the best possible way to dehaze the image that shows significantly better performance. It is designed to work for a wide spectrum of imaging.
[014] Publication No. US2013071043 relates to image defogging method and system, constructing a pixel-based dark channel map of a fogging image. The 25 method estimated a dark channel and optimized it. The whole processing isdone using prior based approach. In contrast, the present invention uses the data driven approach that automatically learn the best possible way to dehaze the image in visible as well as NIR spectrum.

[015] Publication No. WO2017175231 relates to a digital image and for restoring an underwater digital image. This method consists of transmission map estimation and smoothing operations in visible spectrum using a haze line based assumption. On the other hand, the present invention utilized a self transmission estimation 5 and adaptation approach to recover the information loss occurred during the propagation of light through the atmosphere. It is designed to work invisible as well as NIR spectrum.
[016] Publication No. US8290294 relates to an image may be dehazed using a three-dimensional reference model. In an example embodiment, a device-
10 implemented method for dehazing includes acts of registering, estimating, and producing. This method rely on the reference model and haze curve in the visible band of electromagnetic spectrum. However, the haze removal capabilities are limited, because the methodmay not work well if the observation fails. On the other hand, the proposed invention presented a convolutional neural network
15 (CNN) based approach that is very effective for haze removal in visible and NIR bands.
[017] Publication No. CN103413305 relates to a single-image rapid defogging method. The single-image rapid defogging method comprises the following steps that de-noising processing is conducted on a foggy day image, so that a foggy day
20 image generated after de-noising processing is obtained. This publication presented a single image defogging and denoising approach. The method utilized a dark channel based approach to estimate airlight and transmission map. The estimated transmission map is adjusted using maximum and minimum values of the estimated transmission map. On the other hand, the present invention utilized
25 CNN based self adaptive transmission map estimation and image dehazing approach that is suitable for visible and NIR image restoration. In addition, it is a hardware based approach.
[018] Publication No. CN107316284 relates to a defogging method and a defogging device for images under a strong light source. The method comprises

the following steps of carrying out image segmentation on an input foggy image, and separating the non-light source area from the strong light source area; dividing atmospheric light components into an atmospheric light constant component and a strong light source influence component, and respectively 5 calculating the atmospheric light constant component and the strong light source influence component. This method presented a light source segmentation based approach, wherein the influence of the strong light source and atmospheric light are targeted separately. On the other hand, the proposed invention presented CNN based self adaptive approach that is very effective for haze removal in visible and 10 NIR image and video defogging/dehazing.
[019] Publication No. CN103065288 relates to a method and a device of image dehazing, wherein the method comprises calculating a dark primary color image I (x, y) according to a primary image and obtaining airglow A corresponding to the dark primary color image; estimating the dark primary color image and
15 accordingly obtain a first transmission diagram of an atmospheric scattering module, and adopting Gaussian filtering to conduct edge thinning on the first transmission diagram and accordingly obtain a second transmission diagram; and conducting recovery operations on the a primary image I (x, y) according to the second transmission diagram and the airglow and accordingly obtain a fog-free
20 image. The invention estimated a transmission map based on prior and refined it using Gaussian filtering approach. In contrast, the present invention is based on CNN and self adaptive technique data to recover the information loss occurred during the propagation of light through the atmosphere. It is designed to work in wide spectrum of imaging.
25 [020] Publication No. US2014140619 relates to a method of removing fog from the images/videos independent of the density or amount of the fog and free of user intervention and a system for carrying out such method of fog removal from images/videos are disclosed. This method presented an image and video dehazing approach. It estimates airlight map of a preprocessed color hazy image. This
30 airlight map is passed through a refinement stage that is followed by an image

restoration block. In contrast, the proposed invention utilized data driven approach to remove haze from the color and NIR images and videos.
[021] Publication No. CNl02750674 relates to a video image defogging method based on self-adapting allowance and belongs to the field of video image 5 processing. This method estimated the transmission map using dark channel prior and refined it using guided filter approach. The method can be applied to color (visible band) images only. On the other hand, the present invention utilized machine learning approach to remove fog from visible and NIR images and videos.
10 [022] Publication No. CN104767912 relates to a video defogging method based on an FPGA. This method uses minimum and maximum value matrices for transmission map and airlight estimation. The method is conventional prior based technique. However, the present invention uses deep learning based self adaptive approach to defog not only visible but also NIR images and videos.
15 [023] Publication No. US2018308225 relates to a video dehazing module identifies a scene including a set of video frames. The video dehazing module identifies the dark channel, brightness, and atmospheric light characteristics in the scene. This method performed video dehazing by determining haze correction parameters that are based on the dark channel prior, image brightness and
20 atmospheric light. However, the method may not work in wide range of haze and fog conditions. In contrast, the present invention employed a deep learning technique to recover the information loss occurred during the propagation of light through the atmosphere. It is designed to work in visible and NIR spectrum both; therefore, it is suitable for wide range of fog/haze conditions.
25 [024] Publication No. CNl04252698 relates to a semi-inverse method-based rapid single image dehazing algorithm, which comprises the following steps: In light of an atmospheric scattering model, working out an atmospheric global illumination value by utilizing an improved semi-inverse algorithm, wherein the robustness of the obtained atmospheric global illumination value is stronger than that of the

maximum gray value in a dark channel. This method presented a single image dehazing approach using a semi-inverse algorithm. The airlight is estimated using a quad-tree approach that is followed by a gradient based atmospheric streamer estimation stage and image restoration stage. In contrast, the proposed invention 5 utilized deep learning based self adapting transmission map to remove haze and for in an end-to-end manner from the color and NIR images and videos.
[025] Publication No. CN107194894 relates to a video defogging method and a system thereof. This method performed down-sampling of the input image to reduce its dimensions. The dark channel prior is utilized to dehaze the reduced
10 size hazy image that is followed by an up-sampling stage. This down-sampling and up-sampling approach is used to reduce the computational complexity and hardware resources required by the algorithm. On the other hand, the proposed invention can be effectively implemented on hardware (FPGA/ASIC) for original size input hazy image. Therefore, it avoids sampling overhead as well as
15 distortions. Moreover, the proposed approach utilized deep learning approach to self adapt the environment parameters that makes it highly suitable for wide range of haze conditions and imaging spectrum (visible and NIR).
[026] Publication No. CN106846260 relates to a video defogging method in a computer. This publication presented a video dehazing method that utilized an
20 entropy based key frames detection and cross-correlation based frame registration. This is followed by airlight estimation, regression based transmission map estimation, and image recovery. The method is designed to work with visible band images only. On the other hand, the present invention discloses visible and MR defogging solution that is suitable for low cost hardware (FPGA/ASIC)
25 implementation. The proposed invention utilized deep learning based self adaptive approach that drastically improves the haze removal capability.
[027] Publication No. CN105635526 relates to a real-time video defogging method and system applied in a video monitoring system. The method presented a video defogging, wherein inter-frame information of the input video sequence is

utilized to identify common and unknown transmission map areas. This information is employed to finally get the transmission map of the current frame. In contrast, the present invention utilized deep learning based self adaptive approach to recover the information loss occurred during the propagation of light 5 through the atmosphere. In addition, temporal coherence in the dehazed video frames is achieved using an adaptive airlight approach. It is suitable for visible and NIR spectrum dehazing and dedicated hardware implementation.
[028] Publication No. CN107071353 relates to an image defogging device. This publication presented a device for image and video defogging. Apart from the
10 processing units, device also provides a human interface for manual selections and adjustments. The apparatus is designed for visible spectrum only. On the other hand, proposed invention discloses a system as well as method that is capable to capture and process for defogging visible as well as NIR spectral data. In addition, it is highly suitable for dedicated hardware implementation such as FPGA and
15 ASIC.
[029] Publication No. CN108596856 relates to an image defogging method. This method presented an image defogging approach based on the dark channel prior. However, the restoration capabilities of the approach are limited, because the approach may not work well if the haze prior assumption violates. In contrast, the 20 proposed invention utilized data driven self learning technique to remove haze from the color and NIR images. Therefore, it is suitable for wide range of fog/haze conditions.
[030] Publication No. CN108717686 relates to a real-time video defogging method based on dark channel prior and mainly solves the problems of relatively 25 high calculation complexity, and flash and dithering, color distortion and halo effect of a restored video in an existing method. This publication presented a low complexity dark channel prior based video defogging approach. The approach incorporate inter-frame information for airlight estimation. The transmission map is estimated using dark channel prior and refined using guided filter. The approach

is designed for visible band only. On the other hand, the present invention employed deep learning based self adaptive approach for end-to-end image and video dehazing in visible as well as NIR spectrum. In addition, temporal coherence in the dehazed video frames is achieved using an adaptive airtight 5 approach.
[031] Publication No. CN108038831 relates to a colorful video defogging method based on an atmospheric scattering model, and belongs to the technical field of colorful video defogging. This method employed maximum and bilateral filtering for scene depth estimation and the image restoration is performed using the 10 scattering model of dehazing. On the other hand, the present invention data driven approach to recover the information loss occurred during the propagation of light through the atmosphere and it is designed to deploy on dedicated hardware such as FPGA and ASIC.
[032] Publication No. CN107730472 relates to a dark channel prior-based image 15 defogging optimization algorithm, and belongs to the technical field of image processing. This method performed down-sampling of the input image to reduce its dimensions. The dark channel prior is utilized to dehaze the reduced size hazy image. This down-sampling approach is used to speed-up the algorithm. On the other hand, the proposed invention can be effectively implemented for original 20 size input hazy image. Therefore, it avoids sampling overhead as well as distortions. Moreover, the proposed approach utilized CNN based technique that makes it highly suitable for wide range of haze conditions.
[033] Publication No. CN108830803 relates to a defogging optimization algorithm for a traffic video image. This publication presented a traffic video 25 dehazing approach, wherein only the vehicle targets are defogged in each frame and other non-target regions of the image are ignored. The transmission map and airtight are estimated for a group of images. The applications of this method are limited to the static camera. On the other hand, the present invention is valid for static as well as dynamic camera cases.lt used deep learning based self adaptive

approach to recover the information loss occurred during the propagation of light through the atmosphere. The temporal coherence in the dehazed video frames is achieved using an adaptive airtight approach. In addition, the proposed invention works in both visible as well as NHL band of imaging.
5 [034] Publication No. CN103714520 relates to a digital video image enhancement achieving system and method based on an FPGA. This publication presented a dark channel prior based video dehazing approach. The approach is confined to the visible spectrum of the EM radiations and dark channel prior. On the other hand, the present invention deep learning approach to recover the information loss 10 occurred during the propagation of light through the atmosphere. In addition, temporal coherence in the dehazed video frames is achieved using an adaptive airtight approach. The approach can work in visible and NIR dehazing that makes it suitable for wide range of fog/haze conditions.
[035] Publication No. CN109087270 relates to an improved convolution matching 15 tracking pipeline video image defogging enhancement method. This publication presented a dictionary based image defogging method for pipe videos. This is a dedicated approach for pipe images and videos, therefore, this may not be applicable for other applications.On the other hand, the proposed method is generic and can be applied to any type of images and videos. In addition, it is 20 suitable for visible and NIR spectrum.
[036] Publication No. CN109493300A relates to a kind of real-time defogging method of Aerial Images based on FPGA convolutional neural networks and unmanned plane. This method presented a FPGA based defogging approach that estimated airtight using dark channel prior and transmission map using 25 convolutional neural network. The estimated transmission map is then refined by using guided filtering approach. The estimated airtight and transmission map are then utilized to restore the image using physical model of scattering. On the other hand, the proposed invention is an end-to-end learning approach that is self adaptive to the airtight and transmission to ensure best possible restored image

quality. Moreover, it can work in visible and NIR bands that makes it suitable for wide range of haze/fog conditions.
[037] Publication No. KR20180050832A relates to an image dehazing method using a CNN and a system thereof are disclosed. This approach is designed for 5 underwater image dehazing applications. It estimates ambient light and transmission map using CNN that is later used to get the restored image using physical model of dehazing. The proposed method is an end-to-end learning technique that optimizes the parameters to ensure the best possible restoration image quality with less computational resources. Moreover, it can work in visible 10 and NIR bands that makes it suitable for wide range of applications.
[038] The article entitled "End-to-end united video dehazing and detection" by Boyi Li, Xiulian Peng, Zhangyang Wang, Jizheng Xu, and Dan Feng, Association for the Advancement of Artificial Intelligence, Feb. 2018 talks about the end-to-end video dehazing network (EVD-Net), to exploit the temporal consistency
15 between consecutive video frames. This technique uses convolutional neural network structure for image dehazing. The method is computationally expensive and is designed for visible band defogging only. Therefore, the haze removal capability is limited. On the other hand, the present invention is not only suitable for dedicated hardware implementation but also designed for video dehazing.
20 Temporal coherence in the dehazed video frames is achieved using an adaptive airtight approach. In addition, the system is a hybrid approach that contains both color as well as NIR image sensing setup.
[039] The article entitled "Improved single image and video dehazing using morphological operation" by ApurvaKumari, SidharthSahdev, andS.K.Sahoo, 25 International Conference on VLSI Systems, Architecture, Technology and Applications, Jan. 2015 talks about the video defogging. In this paper, we present an effective method for defogging the images and videos based on filtering method. This method employed morphological operation and guided filtering to estimate the transmission map. The method can be operate in limited haze

conditions. On the other hand, the present invention uses deep learning based self adapting approach that shows significantly better dehazing performance. It is designed for visible as well as NIR spectrum image and video restoration.
[040] The article entitled "A video dehazing system based on fast airtight 5 estimation" by Yongmin Park,and Tae-Hwan Kim, IEEE Global Conference on Signal and Information Processing (GlobalSIP), Nov. 2017 talks about the video dehazing system based on the dark channel prior (DCP) is proposed. This method presented a dark channel prior based video dehazing. The original dark channel prior method is modified for faster processing speed and implemented on a
10 system-on-chip platform. On the other hand, the present invention is a data driven method to recover the information loss occurred during the propagation of light through the atmosphere. The temporal coherence in the dehazed video frames is achieved using an adaptive airtight approach. In addition, it can be applied for visible and NIR spectrum that makes it suitable for wide range of fog/haze
15 conditions.
[041] The article entitled "Image dehazing techniques: a review of research trends" by Titiksha N. Bhusari, and S. S. Vasekar, International Journal of Advanced Research in Computer and Communication Engineering, Vol. 5, Issue 4, April 2016 talks about several dehazing approaches. The article reviewed and
20 discussed several dehazing approaches. Regarding the multiple images, early methods used multiple images of the same scene with different polarizations or weather conditions in visible spectrum of electromagnetic radiations. These approaches are confined to visible band of light. Weather is a slow changing phenomenon, therefore, hard to get two consecutive image of different weather
25 conditions in a short interval. This limits the real-time applications of these methods. In order to get different polarization images, special hardware is required and its setting is not trivial. Noise is another problem in these approaches. On the other hand, the proposed invention utilized single image defogging approach that make the proposed approach a perfect candidate for real-
30 time operations.

[042] The article entitled "Dark channel prior based video dehazing algorithm with sky preservation and its embedded system realization for ADAS applications" by Chia-Chi Tsai, Cheng-Yen Lin, and Jiun-In Guo, Optics Express, Vol. 27, Issue 9, April 2019 talks about the dark Channel Prior (DCP) is one of 5 the significant dehazing methods based upon the observation of the key features of the haze-free images. This method presented a dark channel prior based single image dehazing. In addition flickering artifacts are reduced by adjusting the guided filter parameters. However, the method may not work well if the haze condition violates the prior assumption. In contrast, the presented invention 10 employed a deep learning approach to learn and adapt the haze conditions from the input data automatically. Moreover, it can be utilized in visible as well as NIR data.
[043] Thus haze is a common cause of degradation and usually present in the outdoor captured images. It hampers the range of the scene/object viewed. The 15 effect is significantly higher on high frequency components of the image in comparison to the low frequency components. Therefore, image captured at the camera plane lost sharpness and contrast. Haze particles degrade visible band images more severely in comparison to the infrared images, due to scattering phenomena.
20 [044] To overcome the limitations of the above cited prior arts, present invention aims to provide fog removal using visible and near-infrared bands of the electromagnetic spectrum that makes it highly suitable for a wide range of fog conditions. A system and method is provided for color and NIR image and video dehazing/defogging via the convolutional neural network.
25 OBJECTS OF THE INVENTION
[045] The principal object of the present invention is to provide system and method for a wide imaging electromagnetic spectrum (from 400nm to 1400nm) image and video dehazing/defogging

[046] Another object of the present invention is to provide a method and system, wherein defogging is done via the convolutional neural network
[047] Yet another object of the present invention is switching between visible and near- infrared (NIK) bands and system for defogging which provides less 5 execution time and produces high perceptual quality restored images
SUMMARY OF THE INVENTION
[048] The present invention provides a system and method for color and NIR image and video defogging and a deep learning and pixel sharing based self adapting approach is disclosed. It successfully restored hazy images with wide
10 range of haze/fog conditions. Unlike the prior art, the proposed invention perform defogging in wide spectrum of imaging viz. visible and NIR bands that drastically increases the utility and capability of the invention. This invention requires less execution time yet produces high perceptual quality haze free images. In addition, it is highly suitable for FPGA/ASIC implementation; consequently, it can be used
15 for in-camera processing.
[049] The present invention employed a self airtight adaptive deep learning based defogging approach that makes the method more suitable for wide range of applications and haze/fog conditions. In addition, the spatial multiscale features are extracted using completely separable ID layer that allows pipeline stages 20 insertion to improve timing performance and reduces the number of learning weights and memory requirement drastically as compared to the previous methods.
BRIEF DESCRIPTION OF DRAWINGS
[050] It is to be noted, however, that the appended drawings illustrate only 25 typical embodiments of this invention and are therefore not to be considered for limiting of its scope, for the invention may admit to other equally effective embodiments.

[051] FIG. 1 depicts a block diagram of the system components for image dehazing according to the principles of the present innovation.
[052] FIG. 2 depicts an embodiment of the video frame defogging according to the present invention.
5 [053] FIG. 3 depicts a hardware architecture implementation of the method according to the principles of the present innovation.
[054] FIG. 4 depicts a block diagram to train the model and transfer to the inference engine according to the principles of the present innovation.
[055] FIG. 5 depicts a high-level block diagram of the system components for 10 video defogging according to the principles of the present innovation.
[056] FIG. 6a and 6b depicts input foggy image and restored image using proposed method in visible and near-infrared bands, respectively.
DETAILED DESCRIPTION OF INVENTIONS
[057] Accordingly, the present invention provides visible and near-infrared (NIR)
15 image and video defogging system and method. FIG. 1 shows a block
representation of the present disclosure for image defogging, wherein band
selector control 101 that is basically for selection of appropriate imaging band,
using 101 a user can select either color or IR mode of the system, it can also be
replaced with automatic mode selector on the basis of fog density or light
20 conditions, central control unit 102 handles all the timing and streaming of the
system, control circuitry 103 that controls the image sensors and select the pixel
stream from image sensor, 103 operates on the control inputs received from the
band selector, image sensor 104 that receives optical energy in visible 400nm to
750nm and NIR band in 750nm to 1400nm wavelengths, image dehazing unit 105
25 removes fog from the input image and produces a clear image of the scene, it also
performs contrast enhancement to manages contrast problems of the images, a

display unitl06to show the clear image frame that can be dashboard display, smart glasses, etc.
[058] The image sensor that is sensitive to the visible band and NIR band of imaging, senses the light falls upon its pixel arrays. This is scanned by the 5 circuitry associated with the sensor and transfers data to the T estimation and A estimation and reconstruction modules as well as the other modules via the high performance system buses. The T estimation module 202 comprises multiscale convolutional neural network layers with ID kernels. A estimator 203 uses input pixels to estimate atmospheric light of the environment and the image 10 reconstruction unit 204.
[059] According to scattering model, the outdoor image can expressed as:
P(x) = Q(x)xt + A(l-t), (1)
[060] where P{x) and Q(x) represents captured hazy image and original haze
free image, respectively, x denotes image pixel coordinates (represented as single 15 1-dimensional coordinate for notational simplicity). 7 and A indicates transmission map of the environment and atmospheric light, respectively. Eq. (1) can be expressed as
Q(x) = lMzA + A (2)
[061] by rearrange it further as
20 Q(x) = -P(x) + A(l--) (3)
[062] this can be rewritten in simplistic form as
Q(x) = TxP(x) + b (4)
1 ( W
[063] wherer = - and b = A\l — =A(l-T).t represented as t = e~pd{x), so
t { tj
r = e/wW, where j3 and d{x) are scattering parameter and scene depth,
25 respectively. Therefore, J"and Z>are nonlinearly related to the scene depth and scattering parameters of the environment. As a result, the clear image pixel is

highly non-linearly dependent on these parameters. In order to restore the foggy image, we have to estimate Tand b.
[064] A convolutional neural network 202 is utilized to estimate T in the presented embodiment. Any other combination of layers can also be used based 5 upon the computational resources available at hand.
[065] FIG. 2 depicts the proposed multiscale dehazing CNN architecture to learn efficient end-to-end features from the input image and produce a corresponding dehazed image as the direct output. It consists of eight layers, wherein Layer 1,2, 4, 5, and 7 implement convolution and ReLU operations and Layer 3 and 6
10 perform concatenation. Layer 1 and 2extract multiscale features using ID scales lxl, 3x1,5x1,7x1 and lxl, 1x3, 1x5, 1x7, respectively. ThislD decomposition approach is not only easy to implement but also reduces the number of parameters and computational cost as compared to the 2D multiscale implementation. Table I shows number of learning parameters using 2D and
15 lDconvolutions for multiscale feature extraction, which illustrates that the ID decomposition (Layer 1 and Layer2) approach reduces the training parameters 59% as compared to a direct2D implementation of the multiscale layer. Apart from this, we have positioned the multiscale feature layer close to the input to allow pixel sharing in the hardware implementation. The RGB image provided as
20 to the network is processed by multiscale layer, Layerl that produces three output maps for each channel followed by second multiscale layer, Layer2. The output of the multiscale layers is concatenated and processed by a lxl convolutional and ReLU layer whose output is passed through another lxl convolutional and ReLU layer followed by concatenation, convolution, and reconstruction layers. The
25 convolution and non-linear operations are performed as
ii=^'/H+^) (5)
[066] where f,, wt, ft_x, and bt represent, respectively, the nonlinear map of the
current 1th layer, filter weights of 1th layer, the nonlinear map of the previous layer, and bias of the 1th layer. The non-linearity operation (ReLU) is defined as

(0(f) = max(Y,O) (6)
[067] We have used L2 loss function for supervised training of the neural network that is expressed as
C H W 1
4=YLY,-7^\HJ{i>J>k)>e)-J.{i>j*)\ (7)
,=i j=\ k=\ L,liW 5 [068] where C, H, W represents number of channels, image height, and image width, respectively. JandJ, indicate estimated and target clean images, respectively and 6 denote the network learning parameters.
[069] The proposed system has fewer training parameters as compared to the existing lightweight CNNs and is suitable for efficient hardware implementation.
10 The network is composed of only ID convolution operations. The first two layers are responsible for extracting multiscale features using column and row wise convolutions cascaded with ReLU. FIG. 3 shows hardware architecture for the 3x3 scale feature extractor. FIG. 3(a) depicts an input image having r; g; b channels (NIR channels) followed by a set of line buffers. The output buses of the
15 line buffer block will carry three pixels each that correspond to the input maps. These nine pixels are processed by the parallel architecture depicted in 3(b) at each clock cycle. Similarly other two maps for 3x1 scale are generated. A clock¬wise data flow is depicted in 3(c). The other scales (i.e. 1x1,5x1, and 7x1) are also generated using same approach. The output of column-wise convolution is
20 followed by the row-wise operation that implements 1x3 (and lxl, 1x5, and 1x7) scales and depicted in 3(e) wherein input data from the three input maps are shifted and processed by a set of processing elements (PE). The architecture of a PE is depicted in FIG. 3(f). It contains a multiplier, adder, and a weight storing element. Finally, the PE output is accessed through a ReLU function implemented
25 using a multiplexer circuit. The sbit represents sign bit of the resulting pixel from the convolution block. If the sign bit is high (negative number) the output is zero otherwise the output matches the input. Its clock-wise behavior is shown in 3(g). The design is used for all the scales as well. All the scales in first hidden layer share the line buffer pixels. This approach saves 50% of the line buffers required

without a pixel sharing architecture. The other hidden layer Layer 4, 5, and 7 are also implemented in a similar fashion. The input feature maps for the layer Layer 4, 5, and 7 are 12, 3,and 6, respectively. In order to effectively map the architecture to hardware, the weights and activations are quantized to 1 Obits signed fixed point numbers. In addition, the arithmetic operations are performed in fixed point. This greatly reduces the hardware cost as compared to floating point operations.
[070] The airtight can be estimated as the highest value of the current row of a patch in the image corresponding to the identified region as:
max (max (pc (y)\) if M, > M ,
10 A* =
Ac otherwise
[071] whereMx = min [pc{x)),M1= min (pc(x-\)),^represents the airlight, c
belongs to {r,g,b}, and y indicates the coordinates of pixel p\ to pi 5 of the current row for a 15x15 patch, x represents coordinates of the center pixel of a window. Apart from this, the system is adaptive to airlight estimator and also 15 works well if A is prefixed to a constant value 1. This reduces the system complexity on one hand and improves the learning capability on the other.
[072] The image dehazing technique produces flickering artifacts if directly
applied to the video dehazing. Thus, a recursive airlight updating scheme is
formulated to retain the temporal coherence as
20 y;=at/H+(l-ar)t/H (9)
[073] where lft is the current video frame airlight, Ut_x is the previous image airlight estimated, u\_x is the airlight value estimated for the previous video
frame, and a is the learning parameter. The video airlight value lft is updated recursively based on the learning rate, airlight in the previous iteration u]^, and 25 previous atmospheric light Ut_x of the image frame. This approach yields smooth results that are free from flickering artifacts and adapts actual changes in the atmospheric light due to variation in the scene or weather conditions. The flicker

reduction unit expressed in (9) is not needed if the system is trained with a constant airlight value i.e. A = 1.
[074] A representation of the proposed method is shown in FIG. 4 that has two distinct phases viz. training and inference. A large-scale hazy image dataset with 5 ground truth is first utilized to train the software implementation of the model. The trained weights then quantized and converted from floating to fixed point data representation and transferred to an FPGA/ASIC platform to realize a real-time hardware implementation.
[075] The accompanying FIG. 5 illustrated a dehazing system according to the
10 present invention, wherein501 is image sensor. The camera sensor unit may contains separate RGB and NIR sensors, a single RGB-NIR sensor, rotating wheel based RGB-NIR images acquisition device, a prism splitter based color and NIR image capturing device, etc. 502 is image acquisition and control circuitry. The camera sensor signals are processed by the processing system 503, wherein RGB
15 video pipeline and NIR video pipeline preprocess the data coming from the sensor(s). The controller unit generates all the controlling and timing signals for the dehazing as well as other blocks. The controller unit can be a dedicated finite state machine, ARM/MIPS/Intel processor, etc. The sensors data is transferred to the defogging IP via the high performance buses, where the fog removal and
20 contrast enhancement operation is performed. The on-chip-memory (OCM)is a small memory used to buffer the data for high data-rate processing. The I/O ports provide interface to the outer world. The external memory 506unit could be a DRAM, RRAM, MRAM, etc. The storage unit 507 may be a flash memory, SD card, magnetic memory, SSD, CD/DVD, floppy disk, etc. The display interface
25 generates display data and synchronization signals. The display device 504 could be LED display, OLED display, LCD display, smart glasses, etc. 505 is a band selector to switch between RGB and NIR modalities. Apart from this, the system may provide generic I/O ports. Input port of the system may contains HDMI interface, UART interface, MIPI-CSI interface, USB interface, Ethernet interface,

etc. Output port of the system may contains HDMI interface, UART interface, USB interface, Ethernet interface, MIPI-DSI interface, VGA, etc.
[076] The accompanying FIG. 6 (a, b) depicts images restored according to the present invention. FIG. 6a shows input hazy images in visible and NIR bands and 5 restored image in visible band only.FIG.6bshows input hazy images in visible and NIR bands and restored image in NIR band only.
[077] The video dehazing method can be summarized using a stepwise representation as below:
[078] Step 1: Band selection according to the present invention.
10 [079] Step 2: Capture the environment image using image sensor(s) according to the band selected.
[080] Step 3: Transmission map and airlight estimation according to the present invention.
[081] Step 3: Image restoration according to the present invention
15 [082] Step 4: Transmission map adaptation according to the present invention
[083] Step 5: Flicker reduction according to the present invention
[084] Step 6: Optimal quality video frame restoration according to the present invention
[085] Numerous modifications and adaptations of the system of the present 20 invention will be apparent to those skilled in the art and thus it is intended by the appended claims to cover all such modifications and adaptations which fall within the true spirit and scope of this invention.

WE CLAIM:
1. A system and method for image and video dehazing/defogging comprising
a lightweight CNN model that produces dehazed image using input images containing some haze;
a dataset container that contains large set of hazy images and their ground truth images;
a processor with vector processing engine to train the model;
a learning weights quantizer and data converter that quantize the input floating point data and covert it to fixed point;
a system to run the lightweight CNN model with fixed point precision.
2. The system and method for image and video dehazing/defogging, as claimed in claim 1, wherein band selector control 101 that is basically forselection of appropriate imaging band, using 101a user can select either visible or infrared (IR) mode of the system, it can also be replaced with automatic mode selector on the basis of fog density or light conditions, central control unit 102 handles all the timing and streaming of the system, control circuitry 103 that controls the image sensor(s) and select the pixel stream from image sensor(s), 103 operates on the control inputs received from the band selector, image sensor(s)104 that receives optical energy in visible 400nm to 750nm and NIR in 750nm to 1400nm wavelengths, image defogging unit 105 removes fog from the input image and produces a clear image of the scene, it also performs contrast enhancement to manages contrast problems of the images, a display unit 106 that can be a dashboard display, smart glasses, etc.
3. The system and method for image and video dehazing/defogging, as claimed in claim 1, wherein multiscale dehazing CNNarchitecture consists of eight layers, wherein Layer 1,2, 4, 5, and 7 implement convolution and ReLU