FIELD OF THE INVENTION
The present invention relates generally to a method for local in-camera image processing.
In particular, it relates to an automatic user controlled selective region-of-interest based
image processing / enhancement. The invention can be applied to the field of digital
camera devices equipped with touch screen interfaces.
BACKGROUND OF THE INVENTION
Devices with digital cameras along with touch screen interfaces are becoming more and
more commonplace, e.g. in smart phones, camcorders etc. The present invention
extends the capability of such devices with novel functionalities.
Currently, many devices have both a digital camera and a touch screen and current trend
suggests that such devices will become mainstream, enjoying widespread penetration in
consumer markets. Moreover the resources (processing power, battery capacity, rich
applications, etc.) these devices command are also expected to increase manifold.

References can be made to the following prior art documents which inspired the inventor
in the development of the present invention and a known good digital camera has most
of the features detailed hereunder:
Document ISBN: 0201180758 on Digital Image Processing by Gonzalez and Woods
discloses an automatic image enhancement using histogram equalization technique. The
histogram equalization technique is automatically applied by the camera to make the
image have a balanced distribution of intensities. The amount of control the user has on
it is usually limited to turning it on or off.
Document ISBN 01308511981 on Computer Vision: A Modern Approach by David A.
Forsyth and Jean Ponce describes face detection in the image. Such method inspires
focus adjusting, using face recognition; system of US Patent Application No. 11/958, 578
for camera having a focus adjusting system and a face recognition system used in US
Patent 7, 362, 368 B2 for perfecting the optics within a digital image acquisition device
using face detection.

Face detection systems have been coupled with optical system of the cameras, and are
being used to focus on faces detected in the image. Faces are usually a prominent
feature of consumer images and as a system which exploits this, cameras with face
detection based focusing have become the standard. The face detected is being used to
optimize exposure and flash output as well.
US Patent Application No. 10/583, 673 by Olivier Le Meur, Dominique Thoreau, Edouard
Francois, Patrick Le Callet, Dominique Barba discloses a device and method for creating a
saliency map of an image.
"Non-Homogenous Content-driven Video Retargeting" by Lior Wolf and Moshe Guttmann
and Daniel Cohen-Or, IEEE International conference on computer vision ICCV / 2007 uses
saliency computation to perform content aware video retargeting for different screen
sizes.
"Predicting Human Gaze Using Low Level Saliency Combined with Face Detector" by M.
Cerf, J. Harel, W. Einhauser, S. Zurich, C. Koch, Advances in Neural Information
Processing System 2008 uses saliency for predicting human gaze.

"A model of Saliency-based Visual Attention for Rapid Scene Analysis" by L. Itti, C. Koch,
E. Niebur, et al. IEEE Transactions on Pattern Analysis and Machine Intelligence 20(11):
1254-1259, 1998.
There are some disadvantages in the current state of the art. Firstly, most of the current
image enhancement tasks, available on consumer cameras, are global. When the camera
enhances the image, it changes the full image. The only control the user has in such
cases is either to accept or reject the global manipulation. Currently the user is unable to
keep local changes in the image with which one may be satisfied with, while discarding
other local changes which one might not find useful. Secondly, in the current devices the
user has very less interaction with the system and user is not able to guide the process of
image enhancement, in-camera, while preparing to take the photograph.
As an example, the following scenario (one of many, perhaps) brings out these two
limitations in more detail. Consider a user hoping to capture an article kept in a Museum,
while there is a bright lamp a few feet away from the object of interest in the image. A
high-end camera would probably give acceptable exposure and auto focus settings to get
the object of interest fairly clear in the image. However, what if the user is not satisfied

with the amount of emphasis given to the lamp while being OK with the object of
interest? In such a case the user has no option but to change the settings for the full
image and settle on a compromise setting, which is satisfactory to the user.
Using automatic / user-controlled color balance settings, the color balance settings can be
adjusted by the user on the camera for the current shot. However, these settings are
global and affect the whole image.
There is therefore a need for a method, which will help the user in such a situation by
allowing him to manipulate the image in a localized fashion, without changing the setting
of the other areas in the image. Such a method can enhance the image locally, either
automatically or in a user controlled fashion, depending on the user's discretion.
SUMMARY OF THE INVENTION
The main object of the present invention is to provide a method for local in-camera
image processing based on user controlled selective regions-of-interest (ROI).

The present invention applies itself to the rapidly growing consumer digital camera
segment. When a user clicks an image, many times unwanted artifacts dominate the
objects of interest in a frame e.g. halo of a light in the background attracting more visual
attention than the actual object of interest in the foreground. This unwanted effect
might be eliminated, either automatically or by user's discretion, using the current
approach.
The invention would work with devices having both, a digital camera and a touch screen
interface. When a user is taking a picture he may use the touch screen to interactively
manipulate the image locally, in-camera before actually taking the picture. Alternatively,
the image may be processed with a saliency detector, which would identify the regions in
the image likely to be visually important, and then the image may be processed in-
camera before taking the picture to enhance the important regions relative to the not so
important ones.

Thus the present invention provides a method for local in-camera image processing,
characterized by the steps of: identifying regions-of-interest (ROI) in the image;
processing for localized enhancement of the ROI portions by manipulating the image
locally in-camera; and taking with the help of the camera a picture with the enhanced
ROI.
BREIF DESCRIPTION OF THE ACCOMPANYING DRAWINGS
The invention can now be described in detail with the help of the figures of the
accompanying drawings in which
Figure 1 is a flow chart for auto focus and optimization of exposure
and flash settings based on face(s) detected in the image.
Figure 2 is a flow chart for histogram equalization for image enhancement.
Figure 3 is a flow chart for applying color balance for image enhancement.

Figure 4 is a flow chart for the present invention
"automatic / user controlled selective region-of-interest
based in-camera image processing / enhancement.
DETAILED DESCRIPTION
Flow charts in Figures 1, 2 and 3 give simplified outlines, which serve the purpose of
current discussion while eliminating complex details of the methods used in known digital
cameras. In these flow charts enhancement of images are performed by auto focusing,
exposure and flash optimization based on detected faces (Figure 1); by performing
histogram equalization on the whole image (Figure 2); and by applying color balance
settings automatically or supplied by the user (Figure 3). In these flowcharts it should be
noted that there is limited or no user control over the process and the global nature of
the processes (either whole image or none).
The present invention standard image enhancement methods are utilized and the
invention provides to enhance an image locally in-camera using a region(s) of interest
(ROI) approach. The region of interest in a consumer image may be detected either

using an automatic saliency detector or may be obtained in a semi-automatic fashion
from the user via a touch screen on the camera (this is practical in the current scenario,
wherein many high end smart phones have both good quality cameras and sufficiently
big touch screens). Once a (salient) region of interest (ROI) is identified, the image may
be processed before taking the picture, in the camera itself. The operations may involve
standard image processing operations, e.g. adaptive histogram equalizations, image
sharpening etc. Even simple relative and localized brightness and contrast increase /
decrease operations could potentially prove beneficial; although a more in-depth survey /
study is called for, to identify the operations most important to the users.
In-camera processing allows at the moment interactions by the principal user itself and
eliminates the use of image editing / processing software on the computer (which is
usually done in a shop by someone other than the principal user). Moreover the input via
the touch screen can be used to derive interactive feedback about the need of the user.
Example a long press indicating the amount of emphasis; positive or negative, depending
on other key(s) pressed, say volume up or down respectively cf. Alt + Tab on a

computer, on the region currently selected. This way multiple regions, in the same
image, may be manipulated by the user. Similarly, tap or drag gestures on the touch
screen (cf. the touch pad on a laptop, notebook, computer or an iPod) may be used to
perform additional functions or fine tunings to the enhancement(s).
Image saliency map computation is a mature subject Olivier Le Meur, Dominique
Thoreau, Edouard Francois, Patrick Le Callet, Dominique Barba. Saliency has been used
for various applications previously such as predicting the human gaze (M. Cerf et al.) and
also to analyze scenes by (L. Itti et al.) Saliency in an image may be computed using
either low level features in the image e.g. gradient, texture etc. (refer to M. Cerf and L.
Itti et al.), for discussions of the various saliency computation methods that may be used
also) or high level information such as recognized objects (although the state of art in
object recognition doesn't allow for high performance at this time, but in future this will
be dominant). Recently a scientific publication (Lior Wolf et al.) was made which used
saliency computation to perform content aware video retargeting for different screen
sizes.

Camera systems with face detection systems have become popular these days. They are
being used to focus the camera on the faces (which are prominent objects of interest in
consumer images) (see US Patent Application No. 11/958, 578 or in similar spirit, for
improving the optics of the camera during run time using face detection (see US Patent
No. 7, 362, 368 B2). However, to the best of our knowledge, they are not doing image
enhancement using the detected face, and certainly not in user controlled way, after the
camera optics / hardware has been optimized and before the picture has been saved
upon click by the user. Also, the idea of region of interest using pixel / region saliency
computations to enhance the quality of the image in-camera is novel.
The flow chart of Figure 4 describes the working of the system. It has two distinct
pathways:
1. For automatic (without user intervention) processing of the image locally based on
automatic visual saliency detection on the image.
2. For an interactive user feedback based local image enhancement, wherein the
processing is driven by subjective user requirements without any interference
from the artificial intelligence on the camera.

In the method of the present invention the Image enhancements Gonzalez and Woods,
e.g. histogram equalizations, sharpening etc., and visual saliency detection (of Lior Wolf
et al and L. Itti et al.)
As has already been mentioned, the present invention is in the new approach of localized
in-camera image processing driven either automatically using visual saliency detection or
by interactive user feedback.
The method of the present invention overcomes the following limitations of the existing
technology:
Existing methods operate on the full image. They enhance / manipulate the full image
and then give the user the option of either keeping the full change or discarding it
altogether. The method of the present invention on the other hand would allow the
enhancement / manipulation of an image at a local level. The user here may enhance
one part of the image while keeping the other(s) intact. It should also be noted that this
processing is happening in-camera, while the user is preparing to take the picture and
before the picture is actually taken.

Also in the known methods there is very limited or no interaction with the user while
preparing the shots. The users cannot selectively choose to enhance or subdue regions
of the image according to their desire. The users only have the proverbial choice of 'take
it or leave it'. The present invention would allow the user to choose which local region(s)
to enhance or subdue.
The present invention is also capable of working automatically, if such is the desire of the
user, by first, automatically finding out the visually salient regions in the image and then
enhancing the detected salient regions relative to the other regions in the image.
The expression in-camera means inside the camera while taking the picture and not on
the computer offline after the picture has been taken using the resources (processing
power, memory, screen etc.) of the device. Localized image enhancement is carried out
either automaticaly or based on user feedback while the picture is being taken and not
after the picture has been taken.
For automatic enhancement image saliency detector is used to enhance / subdue salient
regions of images relative to not so important regions.

For controlled enhancement the touch screen is interactively used to select the region of
interest to be enhanced / subdued by the user. User can define / use gestures such as
tap, drag, pinch etc. as shortcuts to features or to perform fixed special functions for
image enhancement. Using keys on the device and combinations thereof, the user can
select the operation to be applied and the intensity with which it is to be applied.
ROI: Region of interest. A local region in the image, identified automatically or as
defined by the user.
Local image enhancement / manipulations digital processing of a subset of pixels, instead
of the full image pixels, belonging to a small spatial neighborhood of each other.
Visual saliency of a region means the importance of a region in an image as defined by
the visual weight of the content of that region. E.g. in a sunset scene near a beach, the
region containing the sun (and perhaps its reflection) would be more salient than the
regions containing the (homogeneous) sand and ocean water.

WE CLAIM
1. A method for local in-camera image processing, characterized by the steps of:
- identifying regions-of-interest (ROI) in the image;
- processing for localized enhancement of the ROI portions by manipulating
the image locally in-camera; and
- taking with the help of the camera a picture with the enhanced ROI.

2. The method as claimed in claim 1, wherein said localized image enhancement is
carried out when the picture is being taken by the camera.
3. The method as claimed in claims 1 and 2, wherein said local image enhancement /
manipulation comprises digitally processing a subset of pixels instead of processing
the full image pixels.

4. The method as claimed in claim 1, wherein said steps of identifying the ROI in the
image and processing the ROI portions of the image comprise applying a saliency
detector; and performing localized enhancements based on the output of the
saliency detector.
5. The method as claimed in claim 1, wherein said steps of identifying the ROI in the
image and processing the ROI portions of the image comprise obtaining the ROI
from the user via touch screen; and obtaining enhancement / subdue operation
from user via key combination like UP / DOWN volume key, etc.
6. The method as claimed in claim 5, wherein with the help of the touch screen the
user can define gestures like tap, drag and pinch for performing fixed special
functions for image enhancement.
7. The method as claimed in claim 5, wherein with the help of the keys and
combinations thereof, the user can select the operation to be applied and the
intensity with which it is to be applied.

8. A method for local in-camera image processing, substantially as herein described
and illustrated in the figures of the accompanying drawings.

The main object of the present invention is to provide a method for local in-camera
image processing based on user controlled selective regions-of-interest (ROI).
The present invention applies itself to the rapidly growing consumer digital camera
segment. When a user clicks an image, many times unwanted artifacts dominate the objects of interest in a frame e.g. halo of a light in the background attracting more visual attention than the actual object of interest in the foreground. This unwanted effect might be eliminated, either automatically or by user's discretion, using the current approach.
The invention would work with devices having both, a digital camera and a touch screen
interface. When a user is taking a picture he may use the touch screen to interactively
manipulate the image locally, in-camera before actually taking the picture. Alternatively,
the image may be processed with a saliency detector, which would identify the regions in
the image likely to be visually important, and then the image may be processed incamera before taking the picture to enhance the important regions relative to the not so important ones.
Thus the present invention provides a method for local in-camera image processing, characterized by the steps of: identifying regions-of-interest (ROI) in the image; processing for localized enhancement of the ROI portions by manipulating the image locally in-camera; and taking with the help of the camera a picture with the enhanced
ROI.