FIELD OF THE INVENTION
The present invention generally relates to the field of image processing, and more particularly, to methods and systems for geographical identity based image processing.
BACKGROUND 5
This section is intended to provide information relating to the field of the invention and thus any approach/functionality described below should not be assumed to be qualified as prior art merely by its inclusion in this section.
The use of smartphone has grown exponentially over the last decade. In addition to providing voice calling and text messaging features, these smartphones also 10 include a camera for capturing an image or a video.
At present several applications (apps) exist that allows a user to beautify the image or video either on the smartphone or any other computing device. These apps change one or more beautification parameters, for example skin tone, to generate a beautified image of the user. 15
One of the issue with these apps is that they apply the same beautification criteria for beautifying images or videos throughout the world. This is an issue as users belonging to different countries, races, or ethnicity may have different definitions of the beauty. For example, an European user may consider a different skin tone as beautiful compared to an African user. The beautification 20 filter changing the skin tone of both the European and African user to a lighter skin tone may therefore be racially incorrect.
In view of the shortcomings of the existing systems as discussed in the background section, there exists a need for developing methods and systems for geographical identity based beautification of images and videos that beautify 25 images based on geographical identity of user.
3
OBJECTS OF THE INVENTION
This section is intended to introduce certain objects of the disclosed methods and systems in a simplified form, and is not intended to identify the key advantages or features of the present disclosure.
An object of the invention is to provide a method and system for generating one 5 of a beautified image and a beautified video.
The steps include: determining beautification parameter values, corresponding to the beautification parameters, from a captured image or video. A gi of the user included in the captured image or video is then determined. A beautified image or a beautified video corresponding to the captured image or video is 10 finally generated based on the retrieved plurality of beautification parameter values and the gi specific beautification parameter weights corresponding to the determined gi.
BRIEF DESCRIPTION OF THE DRAWINGS
The following description may be best understood when read in conjunction with 15 the following drawings:
Fig. 1 is block diagram illustrating a geographical identity (gi) based beautification system, in accordance with an exemplary embodiment.
FIG. 2 is a block diagram illustrating a process to beautify a captured image of a user based on gi of the user, according to an embodiment. 20
FIG. 3 is a block diagram illustrating a process 300 to update the image generation function, according to an embodiment.
FIG. 4 is an exemplary user interface of a portable electronic device displaying a user image, according to an embodiment.
FIG.5 is an exemplary user interface of a portable electronic device displaying a 25
4
beautification ranges, according to an embodiment.
FIG. 6 is a user interface of a portable electronic device that displays a beautified user image corresponding to the user image of FIG. 4, according to an embodiment.
DESCRIPTION OF THE INVENTION 5
In the following description, for the purposes of explanation, numerous specific details have been set forth in order to provide a description of the invention. It will be apparent however, that the invention may be practiced without these specific details and features.
The present invention includes methods and systems for geographical identity 10 (gi) based beautification of image or video, wherein said system includes a portable electronic device and a beautification parameter update and synchronization server.
Fig. 1 is block diagram illustrating a geographical identity (gi) based beautification system 100, in accordance with an exemplary embodiment. Beautification is the 15 process of improving visual appearance of an image and/or video. In one embodiment, beautification is the process of beautifying a user’s face. The gi based beautification system 100 beautifies a user’s face based on a user’s gi. A user’s gi may be an identity of a user, for example a user’s nationality, race, ethnicity, etc., which groups a user based on at least a few distinct cultural and 20 physical traits.
For example, users with distinct physical traits fair skin and blonde hair may have a gi “European” whereas users with distinct physical traits medium skin tone and black/brown hair may have a gi “Indian”.
Different gi’s, for example different nationalities, races, and ethnicities, may 25 consider different features, including different facial features, as beautiful. For
5
example facial features, fair skin tone and high-bridged nose may be considered beautiful for a user in Japan and facial features round face and bigger eyes may be considered beautiful for a user in India. The gi based beautification system 100 beautifies the image of user with a particular gi based on the facial features considered beautiful for that particular gi. For example, the gi based 5 beautification system 100 beautifies the image of a user with gi “Japanese” by increasing the skin tone fairness and the nose shape to “high bridge” as these features are considered beautiful in Japan.
The gi based beautification system 100 includes a portable electronic device 102. In one embodiment, the portable electronic device may include, for example, a 10 smartphone, a laptop, or a wearable device, for example a smart watch. The portable electronic device 102 includes a camera 104 that captures an image and/or video of a user. The portable electronic device includes a processor 106 to execute the different modules included in the portable electronic device. The captured image and/or video is provided to a beautification parameter value 15 determinator module 108 included in the portable electronic device 102. The beautification parameter value determinator module 108 is configured to determine beautification parameter values for the captured image and/or video. The beautification parameter values may include values corresponding to different beautification parameters. Beautification parameters, for example, 20 “skin tone”, “skin smoothness”, “face shape” and “blemishes”, are a set of parameters that define the beauty of a user’s face. The beautification parameter values, corresponding to these different beautification parameters, for a user’s image is determined by the beautification parameter value determinator 106.
The portable electronic device 102 includes a gi determinator that determines 25 the gi of the user. The gi of the user may be determined based on information provided by the user and/or based on analysing user related data. For example, the gi determinator 110 may determine the gi of the user based on the name,
6
location, etc. of the user. The gi determinator 110 may also include a face recognition analyser that analyses the user’s face included in the captured image or video to determine the gi of the user. The portable electronic device also includes a memory 112 that stores gi specific beautification weights corresponding to the determined gi of the user. A gi specific beautification 5 parameter weight is a factor, depending on the gi of the user, by which a beautification parameter value is modified to beautify a user’s image.
As discussed above, users with different gi’s, for example different nationality, race, etc., have different definition of beauty. A gi specific beautification parameter weight is assigned to a beautification parameter depending on the gi 10 of the user. For example, a “Caucasian American” user may consider a fair skin tone as beautiful whereas an Indian may consider a medium skin tone as beautiful. In this case, the gi specific beautification parameter weight for “skin tone” parameter of a Caucasian American has a higher value compared to the gi specific beautification parameter weight “skin tone” parameter for a user with gi 15 “Indian”. A higher weight ensures that in the beautified user images the skin tone of a user with gi “Caucasian American” is more fairer then a user with gi “Indian”.
The determined beautification parameter values and the gi specific beautification parameter weights are then provided as input to a gi based 20 beautified image generator 114 included in the portable electronic device 102. A beautified image of a user is an image obtained after beautifying an image of the user based on the user’s gi. In one embodiment, the gi based beautified image generator 108 retrieves the gi specific beautification parameter weights corresponding to the different gi specific beautification parameters from the 25 memory 112.
In one embodiment, the gi based beautified image generator 114 execute an image generation function that receives the gi specific beautification parameter
7
weights and beautification parameter values corresponding to the beautification parameters as input. The image generation function then generates the beautified image based on this input. For example, an image generation function for generating a beautified user image is:
I = f (weight1 value1, weight2 value2, ….) 5
Where
I = Generated beautified user image;
f = image generation function;
weight1 = gi specific beautification parameter weight for a beautification parameter 1; 10
value1 = determined beautification parameter value of a user’s image for a beautification parameter 1;
weight2 = gi specific beautification parameter weight for a beautification parameter 2; and
value2 = determined beautification parameter value of a user’s image for 15 a beautification parameter 2.
For example, consider that a user 1 is identified as having a gi “Caucasian American” and a user 2 is identified as having a gi “Indian”. An image 1 and an image 2 may be captured corresponding to user 1 and user 2, respectively. Assume that the beautification parameter value for “skin tone” beautification 20 parameter is determined as 10 for user 1 and 6 for user 2, respectively. A weight of 1.8 may be applied to the beautification parameter value of “Caucasian American” user 1 and a weight of 1.2 may be applied to the beautification parameter value of “Indian” user 2. The weights and the corresponding parameter values are provided as input to the image generation function to 25
8
obtain the beautified user image.
I = f (10*1.8, 1.2*2)
In one embodiment, different image generation functions, along with different beautification parameter weights, may be defined for different gi. For example, an image generation function for gi “Indian” may be different then the image 5 generation function for gi “Caucasian”. These different image generation functions may have different gi specific beautification parameter weights corresponding to different gi. In one embodiment, the different image generation functions corresponding to different gi’s may be stored at the memory 112. 10
The generated beautified user image is then displayed at a user interface 116 of the portable electronic device 102. One of the advantage that this invention provides is that as the process of image generation is being performed entirely at the portable electronic device 102, an image or video may be beautified even when the portable electronic device is offline, i.e., not connected to any other 15 device or system.
In one embodiment, the gi based beautification system 100 also includes a beautification parameters update server 118 for updating the beautification process depending on change in definition of beauty for a particular gi. For updating the beautification process, the beautification parameters update server 20 118 includes a training data receiver 120 that receives a plurality of images or videos corresponding to a particular gi, as training data. For example, the beautification parameters update server 118 may receive a set of images for users with gi “Indian” that are rated as beautiful. After receiving the training data, each new image captured by the user at a particular location is added to 25 the training data. The training data receiver 120 forwards the images received as training data to an artificial intelligence (AI) module 122 included in the
9
beautification parameter update server 118
The AI module 122 provides as input the received training data to neural networks that identify features that are currently considered beautiful for the particular gi from the images received as training data. Based on the identified features considered beautiful in a particular gi by the AI module 122, the 5 beautification parameter update server 118 sends an update to the beautified image generator 114 included in the portable electronic device 102 to modify the image generation function and the beautification parameter weight for the particular gi. In one embodiment, the AI module 122 at the beautification parameter update server 118 and the portable electronic device 102 may 10 communicate using a wired or a wireless communication.
For example, the AI module 122 at the beautification parameter update server 118 may analyse pictures and videos that are currently considered beautiful for users with gi “Indians”. Based on this analysis, the AI may determine that eye size and eyeliner is currently considered beautiful for users with gi “Indian”. The 15 AI module 122 then provides the determined features as input to the beautified image generator 114 that modifies the image generation function for a particular gi. The image generation function may be modified, for example, by changing the beautification parameter weight, by adding a new parameter to the function, or by changing the mathematical operations within the function. 20
In the example described in the previous paragraph, consider that the beautification function includes three parameters skin tone, skin smoothness and eye size. In this case, the beautification parameter update server 118 syncs with the portable electronic device 102 to increase the “eye size” parameter weight to further increase the eye size in beautified user images of user with gi 25 “Indian”. As the image generation function do not have a parameter for eye liner, which has also been determined as a beautification factor for gi “Indian”, a new parameter eye liner may be included in the function so that the beautified
10
user image includes an eye liner.
The invention provides several advantages. Firstly, the gi specific beautification, instead of beautifying images for all users the same way irrespective of their gi, beautifies the user image based on the definition of beauty in a particular gi. This ensures that the beautified images are racially and geographically relevant. The 5 beautification parameter update server also ensures that the beautification parameters are regularly updated and user’s image is beautified based on the latest parameters that are considered beautiful in the particular gi.
FIG. 2 is a block diagram illustrating a process 200 to beautify a captured image of a user based on gi of the user, according to an embodiment. Initially an 10 instruction is received at the portable electronic device to capture an image or a video (202). In one embodiment, the request may be received at an in-app camera that is included within a messaging application (app) executed at the portable electronic device. The in-app camera may capture the image or video of the user automatically based on one-or-more actions performed by the user. In 15 one embodiment, the instruction may be received at a camera application executed at a desktop or a portable electronic device.
In one embodiment, the portable electronic device stores a plurality of gi specific beautification parameter weights corresponding to different beautification parameters. The weights may be pre-determined depending on the 20 beautification criteria for a particular gi. For example, a light skin tone may be considered beautiful for Caucasian American and a medium skin tone may be considered beautiful for a user with gi “Indian”. In this case, the gi specific beautification parameter weight for a Caucasian American would be higher than the gi specific beautification parameter weight for an Indian. 25
Next a user image of a user is identified in the captured image or video (204). A user image may be determined based on different face recognition algorithms
11
that identify a user face or different portions of a user’s body within an image. In case a user image is not identified in the captured image or video, then the portable electronic device may continue to analyse other images to identify a user face. Next beautification parameter values are determined from the identified user image (206). 5
In one embodiment, a user selection of a beautification range for beautifying the captured image is received at the portable electronic device (208). A beautification range is an amount of beautification that a user wants for a captured image. For example, the beautification range may have three ranges: “Natural”, “Medium” and “Maximum”. In one embodiment, a natural range may 10 indicate a minimum beautification for a user’s image, a medium range may indicate a mid-level beautification selection for a user’s image, and a maximum range may indicate a maximum beautification selection for a user’s image. In one embodiment, a correlation may be defined between beautification parameters that are to be modified and beautification ranges. In this case, when a particular 15 beautification range is selected then the beautification parameter values for the beautification parameters correlated to the selected range may be changed whereas the beautification values for other non-correlated beautification parameters may not be changed. For example, a beautification range “Natural” may be pre-defined to be correlated to a beautification parameter “blemishes” 20 whereas a beautification range “maximum” may be pre-defined to be correlated to beautification parameters “skin tone”, “blemishes”, and “face smoothness”. In this case, when the beautification range “Natural” is selected then the beautification parameter values for the beautification parameter “blemishes” is changed resulting in reduction or removal of blemishes. In case, the 25 beautification range “Maximum” is selected then the beautification parameter values for the correlated beautification parameters “skin tone”, “blemishes” and “face smoothness” is changed. In another embodiment, a beautification range may be correlated to a particular parameter weight corresponding to the
12
beautification parameter values. For example, a beautification parameter range “natural” may change the beautification parameter values for the different beautification parameters by 25%, a beautification parameter range “mid-level” may change the beautification parameter values for the different beautification parameters by 50% and a beautification parameter range “Maximum” may 5 change the beautification parameter values for the different beautification parameters by 75%.
Next a gi of a user of the portable electronic device is determined (210). In one embodiment, the gi of the user whose image is included in the captured image is determined. The gi of the user may be determined from different sources 10 including: user provided information or information based on analysing user related data. For example, a gi of a user may be determined based on user’s locale information, user’s name, and user’s mobile location. A locale information of a user is a set of parameters that defines the user's language, region etc.
The user may provide the user’s locale information when using the portable 15 electronic device for the first time or when registering at the messaging app including the in-app camera. This locale information may then be used to determine the user’s gi. For example, when a user provides a locale information as “Nationality: German” and “Language Preference: DE” then the user’s gi may be identified as “German”. A user’s name may also be used to determine the gi 20 of the user. For example, a user may provide the following details when registering at a messaging app: name “Karthik”, nationality “American” and language preference “American-English”. In this case based on user’s nationality and language preference the user’s gi may be identified as “American”. However, based on the user’s name the user may be identified as a person of “Indian” 25 origin. In this case, the gi of the user is determined as “Indo-American”.
In one embodiment, the user’s location information may also be used to determine gi of the user. For example, when the user’s location has been within
13
India for a long period of time, the gi of the user may be identified as “Indian”. In one embodiment, the user’s gi may be identified by using a face recognition algorithm that analyses the user’s image. For example, when the locale information of a user is “American” then a “thin lips” and “light skin tone” may identify the gi of the user as “Caucasian American” and “thick lips” and a “dark 5 skin tone” may identify the gi of the user as “African American”.
In one embodiment, image environment data is retrieved from the captured image (212). An image environment data may include, for example, the brightness of the image, the background of the image, the uniformity of brightness in the image, the age of the user detected from the image, etc. 10
Next, a beautified user image is generated based on the gi specific beautification parameter weight, beautification parameter values, image environment data and the selected beautification range (214). In one embodiment, an image generation function may be used to generate the beautified user image. For example, consider that the gi of a user of the portable electronic device is 15 “Indian”. An exemplary gi specific beautification parameter weight for the gi “Indian” is shown in Table 2.
Skin tone
Skin Smoothness
Nose thickness
1.1
1.025
0.8
Table 2
In this example, consider that the extracted skin tone value, skin smoothness value, nose thickness value, and brightness from the captured image are 8, 2.3, 6 20 and 4, respectively. Further, consider that the beautification range selected by the user is natural that applies a factor of 1.6 to the skin smoothness. In this case, the output image may be obtained based on the following image generation function:
14
Beautified user image = f (1.1 (skin tone weight) *8 (skin tone value), 1.6 (beautification range based factor) * 1.1. (skin smoothness weight) * 2.3 (skin smoothness value), 0.8 (nose thickness weight) *6 (nose thickness value), 4).
Table 3 shows an exemplary generation of beautification parameter 5 values for a beautified user image based on the gi of the user.
Beautification parameters
Country
Indian
Caucasian
Chinese
Face Shape
I1* = I1 (1.6)
I1* = I1 (1.5)
I1* = I1 (1.3)
Skin Tone
I2* = I2 (1.1)
I2* = I2 (1.05)
I2* = I2 (1.8)
Face Smoothness
I3* = I3(1.025)
I3* = I3 (1.2)
I3* = I3 (1.01)
Table 3
Where I1* = beautification parameter value of a beautified user image for beautification parameter “face shape”;
I1 = beautification parameter value of a captured image for beautification 10 parameter “face shape”;
I2* = beautification parameter value of a beautified user image for beautification parameter “skin tone”;
I2 = beautification parameter value of a captured image for beautification parameter “skin tone”; 15
I3* = beautification parameter value of a beautified user image for beautification parameter “face smoothness”;
15
I3 = beautification parameter value of a captured image for beautification parameter “face smoothness”.
In one embodiment, the image generation function generates the beautified user image based on the age of the user. For example, the image generation function may reduce the value of gi specific beautification parameter weight for 5 skin smoothness of a 50 year old compared to a 20 year old. This ensures that the images are beautified appropriately according to the age of the user. The beautified user image is finally displayed at UI of the portable electronic device.
In one embodiment, the image generation function generates the beautified user image based on the gender of the user. For example, the image generation 10 function may increase the value of gi specific beautification parameter weight for skin smoothness of a girl compared to a boy. This ensures that the images are beautified appropriately according to the age of the user. The beautified user image is finally displayed at UI of the portable electronic device.
In one embodiment, a plurality of user images in a group photo may be 15 beautified based on the gi based beautification. A group photo is a photo that has more than one users. In one embodiment, one or more of the users in the group photo may have different gi. For example, a group photo may include 1 user with gi “Indian American”, 3 users with gi “African American”, and 2 users with gi “Caucasian American”. The gi based beautification process described in 20 208-216 may be applied individually on each of the user images in the group image to obtain the beautified user image. In the group photo example, based on analysing the facial features 1 user with gi “Indian American”, 3 users with gi “African American”, and 2 users with gi “Caucasian American” may be identified. An “Indian American” gi based beautification is applied on the user with gi 25 “Indian American”, an “African American” gi based beautification is applied on the user with gi “African American”, and a “Caucasian American” gi based beautification is applied on the user with gi “Caucasian American” to beautify the
16
group photo.
FIG. 3 is a block diagram illustrating a process 300 to update the image generation function, according to an embodiment. Initially a plurality of images corresponding to a particular gi is received as training data at a beautification parameter update server (302). These images may be provided as input to a 5 neural network system.
A neural network system is a system of hardware and/or software patterned after the operation of neurons in the human brain. A neural network may involve a large number of processors operating in parallel and arranged in tiers. The first tier receives the raw input information analogous to optic nerves in human visual 10 processing. Each successive tier receives the output from the tier preceding it, rather than from the raw input. The last tier produces the output of the system. Each processing node has its own small sphere of knowledge, including the data observed by the node and any rules it was originally programmed with or developed for itself. The tiers are interconnected, which means each node 15 in tier n will be connected to many nodes in tier n-1 -- its inputs -- and in tier n+1, which provides input for those nodes. There may be one or multiple nodes in the output layer, from which the answer it produces can be read.
Next one or more features currently considered beautiful are identified from the training data (304). In one embodiment, the features that are currently 20 considered beautiful are identified from the images that are tagged as beautiful, are trending, are liked, or are provided any identification that identifies the image as beautiful. In one embodiment, the one or more features that are considered beautiful are identified by the neural network that constantly learns from the training data to identify these features. 25
Finally, the image generation function is updated by the beautification parameter update server based on these identified features considered beautiful
17
(306). In one embodiment, the image generation function at the portable electronic device is updated to beautify the image based on the identified features considered beautiful. The process is repeated for different gi’s to update the image generation function at the portable electronic device. This ensures that the image generation function generates beautified user image based on 5 features currently considered beautiful for a particular gi.
FIG. 4 is an exemplary user interface 400 of a portable electronic device 402 displaying a user image 404, according to an embodiment. The user image may be captured by the portable electronic device. The user image 404 is an image that has to be beautified. 10
FIG.5 is an exemplary user interface 400 of a portable electronic device 402 displaying beautification ranges 404-408, according to an embodiment. The beautification ranges may be high 404, medium 406, and natural 408. The user may select one of the beautification ranges depending on the amount of beautification that the user wants. For example, the user may select medium 15 range 406 as the beautification range for beautifying the images.
FIG. 6 is a user interface 400 of a portable electronic device 402 that displays a beautified user image 602 corresponding to the user image 404 of FIG. 4, according to an embodiment. The beautified user image 602 is generated based on user’s gi and the selected beautification range from the ranges 502-506 20 shown in FIG. 5. The beautified user image 602 is then displayed at the user interface of the portable electronic device.
While the present invention has been described with reference to certain preferred embodiments and examples thereof, other embodiments, equivalents and modifications are possible and are also encompassed by the scope of the 25 present disclosure.

We claim:
1. A computer implemented method for generating one of a beautified image and a beautified video, the method comprising:
determining, by a processor of the computer, a plurality of beautification parameter values, corresponding to a plurality of beautification 5 parameters, from one of a captured image and a video;
determining, by the processor of the computer, a geographical identity (gi) of a user included in one of the captured image and the video; and
generating one of the beautified image and the beautified video corresponding to the captured one of the image and the video, 10 respectively, based on the determined plurality of beautification parameter values and a plurality of gi specific beautification parameter weights corresponding to the determined gi.
2. The computer implemented method according to claim 1, wherein generating the one of the beautified video and the beautified image 15 include:
retrieving, from a memory of the computer, the gi specific beautification parameter weights;
executing, by the processor of the computer, an image generation function with the determined plurality of beautification parameter values 20 and the plurality of gi specific beautification parameter weights; and
based on the execution, generating, by the processor of the computer, one of the beautified image and the beautified video.
25
19
3. The computer implemented according to claim 1, wherein determining gi of the user includes:
analyzing one or more of a locale, Global Positioning System (GPS) location, name, and other user or user environment related data to 5 determine the gi of the user.
4. The computer implemented method according to claim 1, wherein determining gi of the user includes:
executing, a face analyzing algorithm, to analyze the one of the captured image and the video; and 10
based on the analysis, determining gi of the user.
5. The computer implemented method according to claim 1, further comprising:
determining, by the processor of the computer, a plurality of beautification parameter values, corresponding to a plurality of 15 beautification parameters, for one or more users included in one of a captured group image and a captured group video;
determining, by the processor of the computer, a plurality of geographical identities (gis) of the one or more users included in one of the captured image and the video; and 20
generating one of a beautified group image and a beautified group video corresponding to the captured one of the image and the video, respectively, based on the determined plurality of beautification parameter values and a plurality of gi specific beautification parameter weights corresponding to the determined gis. 25
20
6. The computer implemented method according to claim 1, further comprising:
storing the plurality of gi specific beautification parameter weights, the plurality of gi specific beautification parameters, and an image generation 5 function corresponding to the gi at a portable electronic device.
7. The computer implemented method according to claim 1, further comprising:
receiving a plurality of images corresponding to a gi as training data;
analyzing one or more images in the training data to identify one or more 10 features currently considered beautiful for the gi;
identifying one or more gi specific beautification parameters, gi specific beautification parameter weights, and gi specific beautification function, for the gi, corresponding to identified feature currently considered beautiful; and 15
updating the identified one or more beautification parameters, beautification parameter weights, and beautification function for the gi at the portable electronic device.
8. The computer implemented method according to claim 1, further comprising: 20
capturing the one of the image and the video at a camera included in a messaging application.
21
9. A portable electronic device to generate one of a beautified image and a
beautified video, the method comprising:
a camera to capture one of an image and a video of a user;
a processor to:
5 determine a plurality of beautification parameter values, corresponding
to a plurality of beautification parameters, from one of a captured image
and a video;
determine a geographical identity (gi) of a user included in one of the
captured image and the video; and
10 generate one of the beautified image and the beautified video
corresponding to the captured one of the image and the video,
respectively, based on the determined plurality of beautification
parameter values and a plurality of gi specific beautification parameter
weights corresponding to the determined gi.
15 10. The portable electronic device according to claim 9, further comprising:
a user interface to display the generated one of the beautified image and
the beautified video.