DESC:FORM 2
THE PATENTS ACT, 1970
[39 of 1970]
&
THE PATENTS RULES, 2003
COMPLETE SPECIFICATION
(Section 10; Rule 13)

METHOD FOR ENHANCING IMAGE QUALITY WHILE ZOOMING INTO PICTURES

SAMSUNG R&D INSTITUTE INDIA – BANGALORE Pvt. Ltd.
# 2870, ORION Building, Bagmane Constellation Business Park,
Outer Ring Road, Doddanakundi Circle,
Marathahalli Post,
Bangalore -560037, Karnataka, India
Indian Company

The following Specification particularly describes the invention
and the method it is being performed

RELATED APPLICATION
The present invention claims benefit of the Indian Provisional Application No. 4746/CHE/2014 titled "METHOD FOR ENHANCING IMAGE QUALITY WHILE ZOOMING INTO PICTURES” by Samsung R&D Institute India – Bangalore Private Limited, filed on 26th September 2014, which is herein incorporated in its entirety by reference for all purposes.

FIELD OF THE INVENTION
The present invention generally relates to image processing systems and methods and more particularly relates to a method for enhancing quality of an image while zooming into pictures.

BACKGROUND OF THE INVENTION
In the current era of technology, variety of handheld devices and embedded devices are available with inbuilt cameras. Especially most of the mobile devices come with inbuilt camera with zooming option, so that the user can capture images at close range without going near to the object. But, image quality of the pictures captured by the current mobile devices’ deteriorates drastically when the user zooms into pictures beyond its original resolution in the gallery.

For an example, for a mobile device, if a sensor resolution is 16MP and display resolution is 2MP, the largest zoom ratio that can be viewed on the 2MP display without losing the quality of the image being displayed is sqrt (16/2) = 2.828x. The 16 MP images will be downsized to 2MP to render it on the display of the mobile device. If the user intends to zoom the images in the applications like gallery, the largest zoom ratio that can be viewed without losing quality is 2.828x. If the user wishes to view the image or picture beyond the zoom ratio, then the quality of the image gets deteriorated and the user can only see low quality, blurred images.
Current technologies provide features such as, but not limited to, super resolution based image enhancement, object recognition, texture detection, Bi-cubic interpolation, and the like to reduce the deterioration of the image quality during zooming. But the above mentioned features are effective only to a certain level of zooming and the image quality still remains low even after applying any of the above mentioned features.

In view of the foregoing, there is a need for a method which reduces the deterioration of the image and maintains the image quality during zooming.

The above mentioned shortcomings, disadvantages and problems are addressed herein and which will be understood by reading and studying the following specification.

SUMMARY OF THE INVENTION
The various embodiments of the present invention disclose a method for enhancing image quality while zooming into pictures. The embodiments of the present invention provide a Region of Interest (ROI) based residue extraction technique, wherein the residue that is extracted during the capture time is stored as a part of the original image metadata. Further enhancement of the image quality for higher zoom ratios while viewing pictures in gallery is achieved by utilizing the residue to generate a super resolved image.

According to an embodiment of the present invention, a method of rendering a captured image comprising, receiving a request to zoom into the captured image, determining at least one of a zoom level and a Region of Interest (ROI) in the captured image for the zoom request, and displaying an enhanced quality captured image by combining a residual data available for the ROI and the captured image at the zoom level.

According to an embodiment of the present invention, the method of rendering a captured image further comprise of combining an extrapolated image data with the residual data to render the captured image with a high resolution at different zoom levels while viewing the image. According to an embodiment of the present invention, the residue data is extracted using a Region of Interest (ROI) based residue extraction technique during capturing of the image.

According to another embodiment of the present invention, the residue data is stored as a metadata of the captured image as a single file or stored separately with or without compression. According to another embodiment of the present invention, the ROI in the captured image is selected automatically based on the visual content that provides cues about at least one of, but not limited to, one or more image segments having high frequency information, scene context, image content, text content, focus information, and the like. According to another embodiment of the present invention displaying an enhanced captured image comprises of reconstruction of the image during zoom on a display using the residue data which provides the highest resolution.

According to another embodiment of the present invention, an apparatus for rendering a captured image comprises an image signal processor (ISP), a communication module unit for receiving a request to zoom into the captured image, a processing engine for determining at least one of a zoom level and a Region of Interest (ROI) in the captured image for the zoom request, and a display unit for displaying an enhanced captured image by combining a residual data available for the ROI and the captured image at the zoom level. According to another embodiment of the present invention, the apparatus further comprises of a storage module adapted for storing the residue data of the captured image.

The foregoing has outlined, in general, the various aspects of the invention and is to serve as an aid to better understand the more complete detailed description which is to follow. In reference to such, there is to be a clear understanding that the present invention is not limited to the method or application of use described and illustrated herein. It is intended that any other advantages and objects of the present invention that become apparent or obvious from the detailed description or illustrations contained herein are within the scope of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The other objects, features and advantages will occur to those skilled in the art from the following description of the preferred embodiment and the accompanying drawings in which:

Figure 1 is a flow chart illustrating a method of rendering captured image on a device display, according to an embodiment of the present invention.

Figure 2 is a schematic diagram illustrating processing the captured image for enhancing image quality while zooming, according to an embodiment of the present invention.

Figure 3 is a schematic diagram illustrating a method for extracting the residue using Visual Content Analysis for selecting the Region of Interest (ROI) and utilizing Super Resolution (SR) for obtaining enhanced quality ROI Image according to an embodiment of the present invention.

Figure 4 is a schematic diagram illustrating Region of Interest (ROI) based residue extraction, according to an embodiment of the present invention.

Figure 5 is a schematic diagram illustrating residue extraction of a selected ROI using bi-cubic downscaling and bi-cubic upscaling technique, according to an embodiment of the present invention.

Figure 6 is a schematic diagram illustrating upscaling of the ROI for reconstruction of the captured image using bi-cubic interpolation, according to an embodiment of the present invention.

Figure 7 is a schematic diagram illustrating storage and re-construction of the ROI from a captured image, according to an embodiment of the present invention.

Figure 8 is a schematic block diagram of a User Equipment (UE) for rendering of captured images, according to an embodiment of the present invention.

Figure 9 is an exemplary illustration of the present invention.

Figures 10a-10d are schematic diagrams of various use cases illustrating automatic ROI selection in the captured images, according to an embodiment of the present invention.

Figure 11 is a schematic diagram illustrating another use case for automatic ROI selection, according to an embodiment of the present invention.

Figure 12 is a schematic diagram illustrating manual selection of ROI within a captured image, according to an example embodiment of the present invention.

Figure 13 is a schematic diagram illustrating hybrid (combination of automatic and manual) selection of ROI within a captured image, according to another example embodiment of the present invention.

Figures 14a-14b are schematic diagrams illustrating flow for another use case of hybrid ROI selection in a captured image, according to an embodiment of the present invention.

Figure 15 is a schematic diagram illustrating automatic ROI selection settings in user equipment (UE), according to an embodiment of the present invention.

Figure 16 is a schematic diagram illustrating alternatives for residual data storage for a captured image, according to an embodiment of the present invention.

Figure 17 is a schematic diagram illustrating a method for enhancing an image quality while zooming by automatic selection of region of interest (ROI) in the image, according to an embodiment of the present invention.

Figure 18 is a schematic diagram illustrating a method for enhancing an image quality while zooming by manual selection of region of interest (ROI) in the image, according to an embodiment of the present invention.

Although specific features of the present invention are shown in some drawings and not in others, this is done for convenience only as each feature may be combined with any or all of the other features in accordance with the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention provides a method for enhancing image quality while zooming into captured images. In the following detailed description of the embodiments of the invention, reference is made to the accompanying drawings that form a part hereof, and in which are shown by way of illustration specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is defined only by the appended claims.

The specification may refer to “an”, “one” or “some” embodiment(s) in several locations. This does not necessarily imply that each such reference is to the same embodiment(s), or that the feature only applies to a single embodiment. Single features of different embodiments may also be combined to provide other embodiments.

As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless expressly stated otherwise. It will be further understood that the terms “includes”, “comprises”, “including” and/or “comprising” when used in this specification, specify the presence of stated features, integers, steps, operations, elements and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. As used herein, the term “and/or” includes any and all combinations and arrangements of one or more of the associated listed items.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

The present invention provides a method for enhancing image quality while zooming into pictures that uses a Region of Interest based residue extraction technique, where the residue data associated with the captured image is stored as a part of the original image metadata and utilizing the residue data to enhance the image quality for higher zoom ratios while viewing pictures in gallery to generate a super resolved image. Various embodiments are described in the present disclosure to describe the working of the method, but not limiting to the scope of the present invention.

The embodiments herein and the various features and advantages details thereof are explained more fully with reference to the non-limiting embodiments that are illustrated in the accompanying drawings and detailed in the following description. Descriptions of well-known components and processing techniques are omitted so as to not unnecessarily obscure the embodiments herein. The examples used herein are intended merely to facilitate an understanding of ways in which the embodiments herein can be practiced and to further enable those of skill in the art to practice the embodiments herein. Accordingly, the examples should not be construed as limiting the scope of the embodiments herein.

According to an embodiment of the present invention, a method of rendering a captured image is described herein, where the user wishes to access a captured image and initiates a request to zoom into the captured image. A user equipment (UE) comprises of a communication module unit that receives a request to zoom into the captured image. In an embodiment of the present invention, the user equipment can be any of, but not limited to, a mobile device, personal digital assistant (PDA), a tablet, a laptop, a phablet, any other embedded device that can receive the request to zoom the image, without departing from the scope of the invention.

In another embodiment of the present invention, the request for zooming of the captured image can be received for the image which is captured and stored in the image gallery of the user equipment (UE). In another embodiment of the present invention, the request for zooming of the captured image can be received for the image which is captured from the camera unit of the UE and is still under review. The person having ordinarily skilled in the art can understand that request for any of the captured image present in the UE can be received for zooming, without departing from the scope of the invention.

Further, the method comprises of determining at least one of a zoom level and a Region of Interest (ROI) in the captured image based on the zoom request. The UE comprises of the processing engine that can receive and determine the zoom level input for the captured image. Further, the processing engine also identifies the region of interest (ROI) within the captured image, which is to be zoomed according to the received request. The ROI can be present anywhere within the captured image and the size of the ROI can also be selected by the user.

In an embodiment of the present invention, the ROI in the captured image can be selected manually by the user of the UE. In another embodiment of the present invention, the ROI in the captured image can be selected automatically based on the visual content that provides cues about at least one of, but not limited to, one or more image segments having a high frequency content, scene context, image content, text content, focus information, and the like. In another embodiment of the present invention the ROI in the captured image can be done using hybrid method, wherein a portion of the captured image can be highlighted automatically and provided and preferred portion from the highlighted portion is selected manually to obtain the region of interest (ROI) for zooming and enhancing, without departing from the scope of the invention.

In an embodiment of the present invention, the user of the UE can set manual ROI selection as the default in the image selection settings. In another embodiment of the present invention, the user of the UE can set automatic ROI selection as the default in the image selection settings. In another embodiment of the present invention, the user of the UE can set hybrid ROI selection as the default in the image selection settings. The person having ordinary skills in the art can understand that the default settings for image selection and ROI selection can be set according to user requirements, without departing from the scope of the invention.

In an embodiment of the present invention, the region of interest (ROI) can be of rectangular shape, wherein the ROI can comprise of image data from left top corner to right bottom corner. In another embodiment of the present invention, the region of interest (ROI) can be of any of the shapes that include, but not limited to, circular shape, triangular, rhombus, trapezoidal, irregular shape, and the like. The person ordinarily skilled in the art can understand that the region of the interest (ROI) within the captured image can be of any shape and size, without departing from the scope of the invention.

In another embodiment of the present invention, only one ROI can be selected within the captured image. In another embodiment of the present invention, any number of ROIs can be selected and/ or obtained from the captured image, and the person ordinarily skilled in the art can understand that the method can allow selection of plurality of ROIs from the captured image, without departing from the scope of the invention.

Further, the method comprises step of displaying an enhanced captured image by combining a residual data available for the ROI and the captured image at the zoom level. The processing engine receives the ROI of the captured image at the zoom level and the residual data of the ROI in the captured image. In an embodiment of the present invention, the residual data of the captured image is the additional data obtained as a difference between the captured Image and the enhanced quality Image obtained using Super Resolution.

In an embodiment of the present invention, the residual data is extracted using a Region of Interest based residue extraction technique during capturing of the image. In another embodiment of the present invention, any of the known image residue extraction techniques can be used for extracting the residual data of the captured image, without departing from the scope of the invention. In another embodiment of the present invention, the residual data is stored as a metadata of the captured image as a single file or stored separately with or without compression, without departing from the scope of the invention.

In an embodiment of the present invention, the UE comprises of a storage module that stores the residual data of the captured image, wherein the storage module can be any of the storage units, that includes, but not limited to, internal storage device such as hard drive, RAM, and the like, external storage devices such as CD, DVD, pen drive, external hard disk, cloud storage, server, database, and the like, without departing from the scope of the invention.

The processing engine identifies the residual data corresponding to the ROI of the captured image at the zoom level and combines the residual data of the captured image with the ROI of the captured image. Upon combining both the residual data of the captured image with the ROI of the captured image, the residual data fills the image data for the particular ROI of the image and thereby enhancing the quality of the ROI of the captured image that user wishes to zoom in for viewing.

In another embodiment of the present invention, the residual data of the captured can be combined with an extrapolated image data of ROI, such that when the residual data is combined with the ROI of the extrapolated image data, the enhanced captured image can render high resolution at different zoom levels while viewing the image.

Figure 1 is a flow chart illustrating a method of rendering captured image on a device display, according to an embodiment of the present invention. At step 102, the communication module unit receives a request initiated by a user to zoom into the captured image. Further at least one of a zoom level and a Region of Interest (ROI) in the captured image is determined based on the zoom request at step 104.

Further, at step 106, the quality of the captured image is enhanced by combining a residual data available for the ROI and the captured image at the zoom level and the enhanced image is displayed to the user. The residual data of the captured image is obtained by processing the image using an image signal processor (ISP) along with the zoomed and ROI selected captured image. The residual data is combined with the ROI of the captured image to enhance the quality of the ROI of the captured image, thereby obtaining the enhanced captured image.

The present invention can be described in detail with respect to various block diagram and use cases for better understanding, various techniques and applications, without departing from the scope of the invention.

Figure 2 is a schematic diagram illustrating processing the captured image for enhancing image quality while zooming, according to an embodiment of the present invention. According to the diagram 200, a captured image 202 can be selected, wherein the captured image is a downscaled image from its original resolution such that the captured image is made suitable to be displayed on a display unit of user equipment (UE). In an embodiment of the present invention, the captured image can be a new captured image from the camera of the UE, or the image selected from the gallery of the UE that contains the residue data, without departing from the scope of the invention.

Within the captured image 202, a portion of the image can be selected called as selected Region of Interest (ROI) 204 that is to be zoomed and viewed. In an embodiment of the present invention, the ROI can be selected automatically. In another embodiment of the present invention, the ROI can be selected manually without departing from the scope of the present invention.

An upscaling operation 206 is performed on the selected ROI 204 of the captured image so as to enhance the selected ROI 204. Any of the known upscaling technique can be used for upscaling 206 of the selected ROI 204, without departing from the scope of the invention.

Further, residue or residual data of the captured image can be received from storage module of the UE, wherein the residue data can be additional data obtained as a difference between the captured image and the enhanced quality image during capture time. As the selected ROI 204 is to be enhanced, out of the entire residue data, only residue of the selected ROI 208 can be selected.

Further, the upscaled selected ROI 206 and the residue of the selected ROI 208 is combined at an adder module 210, wherein, at the adder module 210, the residue of the selected ROI 208 can be used to fill the missing image data in the upscaled selected ROI 206, and thereby obtaining enhanced ROI image 212. The enhanced ROI image 212 is of the best quality image after zooming that can be displayed on the display of the UE, wherein the resolution of the enhanced ROI image 212 matches with the requirements of the display capacity of the display of the UE.

Figure 3 is a schematic diagram 300 illustrating a method for extracting the residue using one such technique of Visual Content Analysis for selecting the Region of Interest (ROI) and utilizing Super Resolution (SR) for obtaining enhanced quality ROI Image according to an embodiment of the present invention. At step 302 multiple frames captured in succession is utilized to improve the resolution. Multiple frames of the same scene can be captured using several methods that are not limited to the following; burst capture at single view point, single capture at several different viewpoints, burst capture at several different viewpoints, usage of single camera at several different viewpoints at different time instants, usage of multiple cameras at different viewpoints at same time instants and so on.

Further at step 304 automatic/manual selection of the ROI in the original image will provide the necessary input ROI to the step 306 which later makes a decision to generate the residue for a particular ROI. An example for such a decision making process is shown in step 306. However alternatives for the same step can be utilized to enable the same functionality. A typical example scenario where the residues are generated for the ROI containing the house and skipped for the ROI containing the Sky is depicted at steps 314 and 312 respectively. Since the sky does not contain any visual content cues, residue extraction is skipped (can be mathematically visualized as zero residue generated because both 316 and 318 utilizes similar bicubic upscaling). Both 316 and 318 can be replaced with alternative upscaling algorithms.

At step 308 a well-known standard Super Resolution technique or any other alternative techniques can be used to enhance the quality of the image ROI selected at step 306.

Further at step 310 the residue is computed as a difference between the Super Resolved image (enhanced quality capture image) and the Bicubic upscaled captured image at several scale factors as described in the subsequent paragraphs.

Figure 4 is a schematic diagram illustrating rendering of the captured image based on Region of Interest (ROI) based residue extraction, according to an embodiment of the present invention. According to the Figure 4, the residue of the captured image 402 is extracted, and is used to enhance the image quality based on the context of selected Region of Interest (ROI). The user selects a super resolution (SR) enhanced region of interest (ROI) 404 of a captured image 402. Here the SR enhanced ROI 404 is the ROI of the captured image with resolution being enhanced by applying super resolution or equivalent techniques during capture flow. Further, region of interest (ROI) of the captured image is selected to obtain captured region of interest (ROI) 406. Further, the SR enhanced ROI 404 is passed on to a downscale module 408, wherein the SR enhanced ROI 404 is downscaled by a factor a. Further, the captured ROI 406 is sent to an upscale module 410, where the captured ROI 406 is upscaled by a preset factor a’. The downscaled output of SR enhanced ROI 404 from the downscale module 408 and the upscaled output of captured ROI 406 from the upscale module 410 is provided to the adder module 412. The adder module 412 combines both the downscaled and the upscaled outputs of given ROI at the adder 412, to provide the residue A for zoom ratio ‘A’.

Further, the SR enhanced ROI 404 is sent to another downscale module 414, wherein the SR enhanced ROI 404 is downscaled by a factor b, and the captured ROI 406 is sent to an upscaler 416, wherein the captured ROI 406 is upscaled by a factor b’. The downscaled output from the downscale module 414 and the upscaled output from the upscale module 416 are passed to the adder 418. Further the downscaled output and the upscaled output are combined at the adder 418 to obtain the residue B for zoom ratio ‘B’.

Similarly, using different downscale ratios for the SR enhanced ROI and corresponding upscale ratios for the captured ROI, a plurality of residues with different residual data for various zoom ratios can be obtained. In an embodiment of the present invention, different types of downscale and upscale modules, with different downscaling and upscaling techniques can be used to obtain the plurality of residues with different residual data for various zoom ratios. In another embodiment of the present invention, different downscaling techniques and similar upscaling techniques for different zoom ratios can be used for obtaining different residues. In another embodiment of the present invention, similar downscaling techniques and different upscaling techniques for different zoom ratios can be used for obtaining different residues. The person having ordinary skills in the art can understand that any of the known upscaling and downscaling techniques can be used for obtaining residue of the ROI, without departing from the scope of the invention.

The main reason behind obtaining plurality of residues is to obtain the residue which when combined with the upscaled ROI of the captured image, can obtain the enhanced captured image with minimal loss. Therefore, a plurality of residues can be obtained for the captured image and the residue data for each zoom ratio is saved for further process.

In another embodiment of the present invention, the upscaling of the image can be performed to enhance the quality of the captured image. According to the present invention, upscaling of the captured image can comprise of three steps, resolution enhancement, residue extraction, and reconstruction of the image. According to the present invention, currently used single/ multi frame super resolution enhancement technique or any other equivalent techniques can be used for resolution enhancement of the captured image, wherein ROI is selected from the captured image for enhancement. In an embodiment of the present invention, bi-cubic interpolation technique can be used for both downscaling and upscaling of the ROI for residue extraction from the ROI of the captured image. Further, bi-cubic interpolation technique can also be used for upscaling of the ROI during reconstruction of the image.

In an embodiment of the present invention, bi-cubic interpolation technique is used for upscaling and downscaling of the region of interest (ROI) in the captured image to obtain different residues corresponding to each zoom ratio, without departing from the scope of the invention.

In another embodiment of the present invention, the bi-cubic interpolation technique can be used for upscaling of the super resolution ROI to obtain enhanced ROI for the captured image.

Figure 5 is a schematic diagram illustrating residue extraction of a selected ROI using a bi-cubic downscaling and bi-cubic upscaling technique, according to an embodiment of the present invention. According to the diagram 500, a super resolution (SR) enhanced ROIi 502 and a captured ROIi 504 is obtained for the captured image (not shown in the Figure 5). The SR enhanced ROIi 502 can be transmitted to a bi-cubic downscaler 506 that downscales the SR enhanced ROIi 502 by a factor a using bi-cubic interpolation. Further, the captured ROIi 504 can be transmitted to a bi-cubic upscaler 508 that upscales the captured ROIi 504 by a factor a’ using bi-cubic interpolation. The output of the bi-cubic downscaler 506 and the bi-cubic upscaler 508 is then transmitted to an adder 510 that combines both the downscaled ROI and the upscaled ROI to generate residue of ROIi namely RA,i 512 for zoom ratio “A”.

Similarly, the SR enhanced ROIi 502 is transmitted to bi-cubic downscaler 514 which downscales the SR enhanced ROIi 502 by a factor b using bi-cubic interpolation. Further, the captured ROIi 504 can be transmitted to a bi-cubic upscaler 516 that upscale the captured ROIi 504 by a factor b’ using bi-cubic interpolation. The output of the bi-cubic downscaler 514 and the bi-cubic upscaler 516 can be transmitted to an adder 518 that combines both the downscaled ROI and the upscaled ROI to generate residue of ROIi namely RB,i 520 for zoom ratio “B”. Similarly, plurality of residues can be generated for the captured image, as described in the embodiments mentioned herein above, without departing from the scope of the invention.

Figure 6 is a schematic diagram illustrating upscaling of the ROI for reconstruction of the captured image using bi-cubic interpolation, according to an embodiment of the present invention. According to the diagram 600, the selected ROIi 602 of the captured image is fed to a bi-cubic upscaler 604 that upscales the selected ROIi 602 by a factor a using a bi-cubic interpolation technique. Further, the residue of the selected ROIi 606 is obtained by using the residue extraction technique described above, but not described herein again to avoid repetition. The residue of the selected ROIi 606 is then provided to an adder 608 along with the upscaled ROIi. Both the upscaled ROIi and the residue of the ROIi 606 is combined at the adder 608 to obtain enhanced ROIi 610.

Figure 7 is a schematic diagram illustrating storage and re-construction of the ROI from a captured image, according to an embodiment of the present invention. Generally, the field of view (FOV) available at 1x resolution is larger than the field of view available in 4x super resolved image. When zooming a 1x resolved image, according to the embodiments disclosed herein, a pre-computed super-resolved ROI can be used to get a much better scaled image at intermediate scale ratios (between 1x and 4x). Also the parts of image which do not lie in the field of view of the super resolved image can be cropped and merged with an upscaled data computed from 1x resolved image. The outer pyramid shown in Figure 7 corresponds to the same FOV as 1x image, while the inner pyramid has the FOV same as 4x image.

Following are the inner pyramid cases, where FOV is equal to or a subset of 4x image:
1. If scale ratio is between 1x and 4x and the FOV of intermediate scale ratio falls within the inner pyramid, the image is computed for display by downsizing 4x super-resolved image as indicated by (Disp=SR_Downscaled).
2. If the scale ratio is less than 1x, the image is computed for display by downsizing 1x native resolution image as indicated by (Disp=NR_downscaled). This is the naïve downscaling of a captured image and can be considered as prior art.
3. If the scale ratio is more than 4x, the image is computed for display by upscaling 4x super resolved image as indicated by (Disp=SR_Upscaled).

Following are the outer pyramid cases, where FOV is equal to or a part of 1x image and not covered by 4x image completely, with scale ratio between 1x and 4x.
1. If scale ratio is between 1x and 4x and the FOV of intermediate scale ratio falls completely outside the FOV of 4x image, the image for display is computed by upscaling the native resolution image as indicated by Disp=NR_Upscaled.
2. If the FOV of the intermediate scale ratio image overlaps the FOV of 1x and FOV of 4x image (regions falling on the intersection points of the two pyramids), the image for display is computed by merging the upscaled native resolution image (for regions of the image falling outside 4x FOV) and downscaled super resolved image (for FOV of ROIs falling within 4x FOV), as indicated by Disp=NR_Upscaled + SR_Downscaled.
3. If the FOV of the 4x scale ratio image overlaps with the FOV of 1x and FOV of 4x image, the image is computed by cropping the super resolved 4x image and merged with the upscaled native resolved image, as indicated by the Disp=SR_Crop+NR_Upscaled.

Using methods as illustrated by Figure 7, one can realize smooth transition between different scale ratios (during zoom-in or zoom-out operations), with much better quality for the regions of images falling within the FOV of 4x image.

Figure 8 is a schematic block diagram of a User Equipment (UE) for rendering of captured images, according to an embodiment of the present invention. The UE 800 is an embedded device comprising, but not limited to, mobile phone, digital camera, PDA, tablet, laptop, phablet, and the like. The UE 800 comprises of an image sensor 802, a controller module 804, a communication module 806, a processing engine 808, an image signal processor (ISP) 810, a storage module 812, a multi-format coder-decoder (CODEC) 814, a touch sensor engine 816 and a display 818.

The image sensor 802 is adapted to detect and capture the image data. In an embodiment of the present invention, the image sensor 802 is an analog image sensor, digital image sensor, sensors based on color separation technique, special sensors and the like. In an embodiment of the present invention, the analog image sensor can be any of, but not limited to, semiconductor charge-coupled devices (CCD), active pixel sensors in complementary metal–oxide–semiconductor (CMOS), N-type metal-oxide-semiconductor (NMOS, Live MOS), and the like. In another embodiment of the present invention, analog and digital image sensors comprises of sensors that can include flat panel detectors, but not limited thereto. In another embodiment of the present invention, image sensors based on color separation techniques include, but not limited to, Bayer filter sensor, Foveon X3 sensor, 3CCD sensor, and the like. In another embodiment of the present invention, special sensors are sensors used in various applications such as, but not limited to, thermography, creation of multi-spectral images, video laryngoscopes, gamma cameras, sensor arrays for x-rays, other highly sensitive arrays for astronomy, and the like.

In an embodiment of the present invention, any type of the known image sensor 802 that is known to the person having ordinary skill in the art can be used in the UE 800 for capturing of the image data without departing from the scope of the invention. Further, the process of image capturing using the image sensor is already known art to the person having ordinary skills in the art and hence not described in detail herein.

Further, the UE 800 comprises of the controller module 804 that can be used for processes such as, but not limited to, enhanced resolution mode in Camera Application, extracting residual information during capture, storing of residual information, reconstruction of enhanced quality image using residue during zoom, and the like.

Further, the UE 800 comprises of the communication module 806 that enables communication across various modules and components both locally and remotely. Further, the communication module 806 receives request for zooming and enhancing of the captured image.

Further, the UE 800 comprises of the processing engine 808, wherein the processing engine 808 can capture the image, extract residue from the captured image, and reconstruct the captured image upon extracting the residue.

Further, the UE 800 comprises of the image signal processor (ISP) 810 that enhances the image data quality as well as to scale and crop the image. The ISP 810 receives the user request and identifies the region of interest (ROI) based on the request, receive the residue from the processing engine 808 and generate an enhanced captured image by combining the ROI and the residue of the image. The person having ordinarily skilled in the art can understand that any type of the known ISP 810 can be used for scaling and enhancing the captured image using the method of the present invention, without departing from the scope of the invention.

Further, the UE 800 comprises of a storage module 812 adapted for storing the captured image data, residual data corresponding to the captured image data, and control parameters corresponding to the captured image data. The storage module 812 that can be used for storing the captured image data, residual data and the control parameters are being described in the embodiments mentioned hereinabove and thus not described herein again to avoid repetition.

Further, the UE 800 comprises of the multi format CODEC 814 that can be used for encoding and decoding of the image data and residue. In an embodiment of the present invention, any of the known types of image encoding and decoding formats that include, but not limited to, JPEG/JFIF, JPEG 2000, Exif, TIFF, RIF, GIF, BMP, PNG, PPM, PGM, PBM, PNM, WEBP, BPG, CGM, Gerber format (RS-274X), SVG, and the like, and the person having ordinary skill in the art can understand that any of the known encoding and decoding format can be used apart from the formats mentioned herein above to encode decode the captured image and the enhanced captured image, without departing from the scope of the invention.

Further, the UE 800 comprises of the touch sensor engine 816 that receives one or more touch inputs from the user, and processes touch events such as image selection, zoom, ROI selection, image saving, deletion, and the like, without departing from the scope of the invention.

Further, the UE 800 comprises of the display unit 818 that can be used for displaying captured image, image selection, zooming of the image, options available for enhancing the image, updating of the image, saving of the image, and other options that are related to the captured image. In an embodiment of the present invention, the display unit 818 can be any of the displays, such as, but not limited to, LCD type display, LED type display, and the like.

Figure 9 is an exemplary illustration of the present invention. Figure 9 comprises of two images 902 and 904 obtained from the existing image enhancement method, and the current image enhancement method respectively. Both the images 902 and 904 are zoomed, the region of interest (ROI) is selected and enhanced from the same captured image. It can be observed that the image obtained from the existing image enhancement method 902 is unclear, noise level is high, and posing difficulty to differentiate between different objects. Whereas on the other hand, the image obtained from the proposed method of image enhancement 904 clearly shows different objects in the image, texts which can be easily read, and the noise level is minimal. Thus, upon considering the images 902 and 904, it can be easily observed that the quality of the image obtained by the current method of image enhancement is of high quality as compared to the conventional methods.

Figures 10a-10d are schematic diagrams illustrating various use cases of automatic ROI selection in the captured images, according to an embodiment of the present invention. The schematic diagram 1000 illustrates automatically selecting an ROI based on visual content within an image, obtaining residue for the selected ROI and enhancing the quality of the selected ROI of the captured image. For example as shown in Figure 10a, the user has selected the image of the statue of Liberty. The processing engine (Not shown in the Figure) of the UE identifies the statue of Liberty in the image automatically, based on one of the cues from visual content analysis. In an embodiment of the present invention, the visual content can provide cues based on several factors such as, but not limited to, color intensity, color depth, change in color value of the pixel with respect to each other, edges, and the like.

The image signal processor (ISP) can automatically identify the portion of the image as Region of Interest (ROI) based on the visual content, obtains residue for the selected ROI, and combines residue of the ROI with the selected ROI to obtain enhanced ROI of the captured image. According to the Figure 10a, the ROI automatically selected by the UE can be marked with the rectangular box, and the cropped image of the ROI can also be seen.

Further, Figure 10b is a schematic diagram 1010 illustrating automatic ROI within the image based on the face detection, according to an embodiment of the present invention. According to the Figure 10b, the processing engine of the UE automatically identifies one or more faces present in the captured image and identifies them as the region of interest (ROI) for enhancement. The ISP processes the ROI present in the image, combines residue of the ROI to obtain enhanced images. According to the Figure 10b, two faces are automatically identified as ROI, further processes to obtain residue for the ROI and combined with the ROI to obtain enhanced ROI image.

Similarly, according to Figure 10c, the schematic diagram 1020 describes automatic ROI selection in the image based on the text present within the image. According to Figure 10c, the UE identifies text content present within the image and selects the portion as the ROI, obtains residue for the selected ROI and combines the residue with the ROI to obtain enhanced ROI image. For example, in an image of a car, the system is allowed to automatically identify the text portion. The number plate of the moving car is automatically identified as ROI, residue for the ROI is obtained and enhanced by combining residue with the ROI to obtain enhanced ROI.

Similarly, according to Figure 10d, the schematic diagram 1030 describes automatic ROI selection in the image based on the focus within the image. According to the Figure 10d, the system automatically identifies the portion of the image with focus as the ROI and discards the remaining portion, obtains residue for the selected ROI and enhances the ROI. For example, in the captured image with the bird, the rest of the portion of the image apart from the bird is blurred, and only the bird is having the focus. Thus, the system identifies the bird as the ROI, obtains residue for the ROI, and enhances the ROI by combining the ROI with the residue to obtain the enhanced image of the bird.

Figure 11 is a schematic diagram 1100 illustrating another use case for automatic ROI selection, according to an embodiment of the present invention. According to Figure 11, the system automatically identifies a plurality of ROIs within the image based on visual content cues such as high frequency, face detection, text, focus, and the like to obtain all possible ROIs, combine them together, obtain residue for the obtained ROIs and enhance the ROIs by combining corresponding residues with the ROIs. For instance, the captured image comprises of faces of a girl and a dog. The system automatically identifies the faces as region of interest. Further, the captured image comprises of the ice cream, wherein the system automatically identifies the ice cream as ROI based on visual content. A segmentation process can be performed on the selected ROI. For the obtained ROIs, residues is obtained, and combined with the obtained ROIs to generate an enhanced image of the ROIs.

Figure 12 is a schematic diagram 1200 illustrating manual selection of ROI within a captured image, according to an example embodiment of the present invention. According to the Figure 12, the user can manually select the region of interest (ROI) for enhancement. For instance, in the image, the user manually selects two birds as region of interest (ROI). The system receives the ROI, obtains residue for the selected ROI and combines the residue with the ROI to obtain enhanced image of the ROI based on manual selection.

Figure 13 is a schematic diagram 1300 illustrating hybrid selection of ROI within a captured image, according to an embodiment of the present invention. According to the Figure 13, the system automatically identifies the ROI for quality enhancement within an image. Further, the user can manually select the ROI among the ROIs which was automatically selected. The selected ROI can be enhanced by obtaining residue for the selected ROI and combining the ROI with the residue to obtain enhanced ROI image. In another embodiment of the present invention, the user can manually select the ROI within an image and the system can automatically select ROI within the manually selected ROI for enhancement. For instance, within a captured image of the two birds, the system automatically selects image of the two birds as ROI. Even within the automatically selected ROI, there are so much of unwanted image data. Therefore, the user manually selects the ROI that he wishes to enhance. The system receives the manually selected ROI, obtains residue for the manually selected ROI, combines the residue with the manually selected ROI to obtain enhanced image of the ROI.

Figures 14a-14b are schematic diagrams illustrating flow for hybrid ROI selection in a captured image, according to an embodiment of the present invention. According to the Figure 14a, in hybrid ROI selection, the user can manually select one or more ROIs by tapping on the portion of the image that he wishes to identify as ROIs. Further, according to the Figure 14b, in hybrid ROI selection, the user can manually select one or more ROIs by manually drawing around the portion of the image that he wishes to select as ROIs.

For instance, according to Figure 14a, in 1402 upon capturing an image, the system automatically identifies three faces and a name plate as region of interest (ROI) and puts a rectangular mark around them to highlight and display them as ROI. Upon identifying the ROIs within the image automatically, in 1404, the user can manually select only the name plate and the face of the mother in the image as ROI by tapping on the ROIs and further process the image to obtain the enhanced image of the ROIs.

In another instance, according to the Figure 14b, in 1412, upon capturing an image, the system automatically identifies three faces and a name plate as region of interest (ROI) and puts a rectangular mark around them to highlight and display them as ROI. Upon identifying the ROIs within the image automatically, in 1414, the user manually selects on the name plate and deselects the faces. Further, in 1416, the user manually selects all the three faces in the image by drawing a circle around all the faces manually, and thereby selecting the plurality of ROIs within the image for further enhancement.

Figure 15 is a schematic diagram 1500 illustrating automatic ROI selection settings in user equipment (UE), according to an embodiment of the present invention. According to the Figure 15, the diagram 1500 clearly depicts user can set on and off options for a plurality of items within an image for automatic ROI selection. For example, according to the diagram 1500, the user has set an option for people, text, object and segmentation as ON, while the user has set OFF for the focus. Therefore, according to the settings, whenever the user captures an image for enhancement, the system can automatically identify plurality of faces, texts, and objects within the image as ROI and segmentation of the ROI within the image can also be done automatically. But the system will not automatically detect the focus in the image as ROI, according to the settings. The person ordinarily skilled in the art can understand that the pictorial representation of the automatic ROI setting is only for the sake of understanding, but the GUI representation and options available during automatic ROI selection can vary depending on user requirement and upgradation, without limiting the scope of the invention.

Figure 16 is a schematic diagram 1600 illustrating alternatives for residual data storage for a captured image, according to an embodiment of the present invention. According to the Figure 16, a plurality of residual data is obtained for the plurality of ROIs identified from the captured image. The plurality of residual data obtained for the ROIs can be of varying zoom ratio. In an embodiment of the present invention, residual data can be stored as a single Jpeg file. Further, the residual data can be of raw form and are stored as application marker data. It is to be observed that there will be no change in the storage format standards. In another embodiment of the present invention, residual data can be stored as a single Jpeg file. Further, the residual data can be compressed using standard jpeg encoding methods and are stored as app marker data. It is to be observed that there will be no change in the storage format standards. In another embodiment of the present invention, residual data can be stored as separate Jpeg files. It is to be observed that there will be no change in the storage format standards.

Figure 17 is a schematic diagram 1700 illustrating a method for enhancing an image quality while zooming by automatic selection of region of interest (ROI) in the image, according to an embodiment of the present invention. According to the diagram 1700, the entire process can be categorized into three phases, capture and residue extraction, storage, and reconstruction during zooming and display. At step 1702, an image of two persons cycling is captured on the UE. The captured image can be encoded using a JPEG encoder 1704, and used later. At step 1706, image segmentation is performed during which one or more regions of interest (ROIs) is automatically selected, wherein the face of the girl is identified as ROI1 and the face of the boy is identified as ROI2. At step 1708, a plurality of residues are obtained for both of the ROIs ROI1 and the ROI2, wherein the residues obtained are of varying zoom ratio and the method of obtaining residues is described in the description herein above and hence not described again to avoid repetition. All the herein above mentioned steps are part of the capture and residue extraction phase.

Further, at step 1710, the image data can be stored in the storage unit. The compressed data comprising ROI and residual data for each ROI can be stored in one part of the storage, wherein the compressed data can be received from the JPEG encoder 1704, which has transmitted the compressed actual captured image, and the residual data for each of the ROIs identified in the captured image. The above step is part of the storage phase.

Further, at step 1712, the user can select the particular ROI which user wishes to enhance and display. The user selects ROI1 for enhancement. At step 1714, the system selects the ROI1, receives the residue for the selected ROI1. At step 1716, bi-cubic upscaling is performed on the selected ROI1. The process of bi-cubic upscaling is described herein in the description and hence not repeated herein again to avoid repetition. At step 1718, bi-cubic upscaler output is added to the residue data for given zoom ratio to obtain an enhanced ROI with better quality for display. The herein above mentioned steps are part of the reconstruction during zooming and display phase.

Figure 18 is a schematic diagram 1800 illustrating a method for enhancing an image quality while zooming by manual selection of region of interest (ROI) in the image, according to an embodiment of the present invention. According to the diagram 1800, the entire process can be categorized into three phases: capture and residue extraction, storage, and reconstruction during zooming and display.

At step 1802, an image of two persons cycling is captured on the UE. The captured image can be encoded using a JPEG encoder 1804, and used later. At step 1806, user manually selects the image of the girl as region of interest (ROI) by encircling around her face. At step 1808, image segmentation is performed, wherein the face of the girl is identified as ROI1. At step 1810 plurality of residues are obtained for each ROI, wherein the residues obtained are of varying zoom ratio and the method of obtaining residues is described in the description herein above, and hence not described again to avoid repetition. All the herein above mentioned steps are part of the capture and residue extraction phase.

Further, at step 1812, the image data can be stored in the storage unit. The compressed data comprising ROI and non-ROI data can be stored in one part of the storage, wherein the compressed data can be received from the JPEG encoder 1804, which has transmitted the compressed actual captured image, and the residual data for each ROI identified in the captured image. The above step is part of the storage phase.

Further, at step 1814, the system zooms into the ROI, receives the residue for the selected ROI. At step 1816, bi-cubic upscaling is performed on the selected ROI and the corresponding residue is being added. The process of bi-cubic upscaling is described herein in the description and hence not repeated herein again to avoid repetition. At step 1818, bi-cubic upscale output when added with corresponding residue for given zoom ratio results in an enhanced ROI with better quality for display. The herein above mentioned steps are part of the reconstruction during zooming and display phase.

In the following detailed description of the embodiments of the invention, reference is made to the accompanying drawings that form a part hereof, and in which are shown by way of illustration specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is defined only by the appended claims.

,CLAIMS:CLAIMS
We claim:

1. A method of rendering captured image with enhanced quality, comprising:
receiving a request to zoom into the captured image;
determining one or more zoom levels for each Region of Interest (ROI) in the captured image from the received zoom request;
displaying an enhanced captured image with different ROIs at respective zoom levels by combining a residual data available for the ROI and the captured image.

2. The method of claim 1, wherein reconstructing the enhanced captured image comprises of:
calculating a super-resolved ROI at a desired zoom level using multiple captured images;
calculating an ROI based residual data between the super-resolved image and the captured image at one or more zoom levels;
compressing and storing the residual data at one point in time; and
decompressing and restoring residual data for reconstructing the super-resolved image at another point in time.

3. The method of claim 1, further comprising:
combining an extrapolated image data with the residual data to reconstruct the super-resolved data.

4. The method of claim 3, wherein extrapolated image data is obtained by using bicubic interpolation for upscaling of the resolution of each of the different ROIs.

5. The method of claim 1, wherein the residue data is extracted using a Region of Interest (ROI) based residue extraction technique during capturing of the image.

6. The method of claim 1, wherein the residue data is stored as at least one of:
metadata of the captured image as a single file;
stored separately as a compressed file; and
stored as a separate file without compression.

7. The method of claim 1, the ROI in the captured image is selected manually or automatically based on visual content related to at least one of:
one or more image segments having a high frequency content;
a scene context;
an image content;
a text content; and
a focus information.

8. The method of claim 6, wherein automatic selection of ROI in the captured image comprises of:
segmenting the captured image to a first ROI and a second ROI;
performing resolution enhancement and residue extraction for the first ROI and the second ROI;
storing the first ROI and the second ROI and the residue of the first ROI and the second ROI in a compressed format.;
displaying the first ROI and the second ROI to the user;
reconstructing a high resolution image of at least one of the first ROI or the second ROI using the associated residue based on user selection; and
displaying the high resolution image to the user.
9. The method of claim 6, wherein manual user selection of ROI in the captured image comprises of:
identifying at least one ROI in the captured image;
performing resolution enhancement and residue extraction of the identified ROI to a plurality of zoom levels;
storing the identified ROI and the extracted residue of the at least one ROI;
displaying the at least one ROI;
zooming the at least one ROI to a desired zoom level by the user;
reconstructing a high resolution image of the at least one ROI using the extracted residue associated with the at least one ROI ; and
displaying the high resolution image to the user.

10. The method of claim 1, wherein displaying an enhanced captured image comprises of:
reconstruction of the image during zoom on a display using the residue data which provides the highest resolution.

11. The method of claim 1, wherein a multi format codec is used for encoding and decoding of the selected ROI and the extracted residue.

12. An apparatus, comprising:
an image signal processor;
a communication module unit for receiving a request to zoom into the captured image;
a processing engine for:
determining at least one of a zoom level and a Region of Interest (ROI) in the captured image for the zoom request; and
reconstruction of the enhanced captured image using a residue data for given ROI at the zoom level; and
a display unit for displaying an enhanced captured image by combining a residual data available for the ROI and the captured image at the zoom level.

13. The apparatus of claim 12, wherein the processing engine is further adapted for:
calculating a super-resolved ROI at a desired zoom level using multiple captured images;
calculating an ROI based residual data between super-resolved and captured image at one or more zoom levels;
compressing and storing the residual data at one point in time; and
decompressing and restoring residual data for reconstructing super-resolved image at another point in time.

14. The apparatus of claim 12, further comprising:
a storage module adapted for storing the residue data of the captured image.

Dated this the 21st day of September 2015

Signature

KEERTHI J S
Patent agent
Agent for the applicant