Description:FIELD OF THE INVENTION
This invention relates to Real-Time Facial Expression Analysis for Emotion Recognition Using Image Processing.
BACKGROUND OF THE INVENTION
The importance of image emotion recognition can be employed in the healthcare industry, academic, advertisement, and human-computer interface. The main aim of developing a real time emotion recognition system through the use of deep learning is to predict the emotional state from the facial expressions. This system will extract facial features and this will be done by the use of deep learning and CNNs. This research patient will enhance the interaction between human and machines and overall user experience by enhancing the emotion recognition techniques and their usage in various domains.
Although the field of Emotion Recognition from Images (ERI) and its possible uses in various sectors has been improving rapidly, traditional approaches fail to identify a wide range of emotions on the face and are not versatile enough to work with different groups of users. A stronger, more ethical ERI system is needed to:
1. Enhance the recognition of facial emotions and handling all facial features and emotions.
2. Enlarge the set of emotions to meet the needs of various conditions and cultures.
3. Respect privacy and use facial data in emotion recognition in a right manner.
4. Apply the right emotion recognition to make computers more personal and to understand their users on emotional level.
5. This paper recommends that the emotion recognition algorithm be changed in a way that it does not exclude people of certain gender, color or age.
6. The positive impact of emotion recognition technology application should be enjoyed while adhering to the right use.
This current work proposes to design an improved ERI system that can address these issues in order to enhance the ethical and effective application of emotion recognition technology in health care, learning environments, business, and Human-Computer Interaction.
This job is related to the recognition of the visual emotion. It is because emotion recognition has its application in the health care, education, marketing, and HCI. Recent techniques may fail to provide accurate results, particularly in actual conditions with numerous face expressions and diseases of the environment. The algorithmic size of the emotion recognition algorithmic programs can sometimes pose a challenge to the real time use and hence their practical value.
Therefore, the problem is to create an effective emotion recognition approach that can accurately classify emotions from images in real time, with reasonable computational complexity and applicability to a wide range of situations and environments. For the general use, method optimization, and dataset variation, a more critical issue of algorithm interpretability must be solved. To this end, we aim at presenting a strategy that effectively addresses the needs of users in various contexts and enhances faces’ appearance while creating possibilities for emotion-aware technologies by addressing these limitations.
In [1], the study would focus on “Facial Emotion recognition (FER)”. The emotion recognition system follows the FER system's basic steps: Image capturing, image processing, face recognition, FER extraction, categorization of emotions and music to be played. This research aims at designing a computer based FER system that applies camera feed to recognize stress and offer music therapy in order to reduce it. The basic emotions which have been identified include happiness, sadness, surprise, fear, disgust and anger were used in the study.
In particular, Reference [2] employs correlation modelling to study the link between moral sentiments and helpful action with the use of the “Hidden Markov Models (HMM)” FER algorithm. In order to easily incorporate the HMM architecture, a new deep optimization framework is employed. Moreover, wavelet transform can point to time and frequency characteristics associated with peculiar electric energy oscillations within a given time interval. Therefore, wavelet transform is applied to study the irregular waveforms. Moral emotion modeling and Helping behavior modeling are suitable in the suggested Emotion Analysis Model as indicated [12].
The work [3] employs MFSC images and DL-NN to analyze emotion recognition in the “Romanian language”. The researchers’ 85% accuracy is in line with previous research. Four moods including happy, sad, angry, and neutral were examined on low and high resolution MFSC photos. They compared the DL-CNN and CNN architectures and identified that two-layered DL-NNs with Autoencode and Autoencoder layers and variable neuron count provided the most efficient result [13].In accordance with the reference This paper aims at discussing the problem of distinguishing emotions in visuals. It reviews the related literature on this subject matter and introduces this investigation’s machine and DL approach. In the experiment we utilized photos of humans in 7 different emotional conditions. Many machine and deep learning model characteristics were suggested. To this end, each proposed characteristic was employed to create datasets in order to assess various machine learning algorithms and perceptron architectures. In this paper, classification metrics, confusion matrices of investigated models, and the best f1 score model are provided.
To the best of our knowledge from reference [5] The proposed network consists of an initial classifier stream, an intensity predictor stream and a second classifier stream. The concentration predictor stream is designed upon the feature pyramid network for multilevel feature extraction. The first stream of classifiers is applied to produce the pseudo concentration maps through the class activation mapping. In order to learn emotion intensity, the proposed network is trained with these maps. The last emotion that is extracted is found by incrementing the intensity maps to the classifier stream. the proposed emotion and sentiment classification network is then compared on benchmark datasets [15].
A special but effective manual segmentation process is applied in the FER to divide human facial images for emotions detection. This technique includes checking the nose, mouth and right and left part of the face. A 2D gabor filter is applied to segment facial areas and by down sampling, features are extracted and K-Nearest Neighbors is used to classify facial expressions [16].
According to the reference FER cannot be used in different situations easily. The primary issues include personalized self-compassion and limited material for training specific individuals in large datasets. The current models of emotion prediction may not be effective when dealing with emotion of people who are not part of training data set. To address these issues, this paper introduces a new method based on facial video data and emotion references to predict emotions in a subject without the need for subject examples. The one shot Emotion Score approach to the problem avoids these issues without the need for fine registration. We improve the Inter dataset Experiments classification rates by 23% in MMI and CK+ when training on a baseline system.
FER is an active research field in human- computer interaction. Facial expression based emotion estimation process necessitated the designing of a new image classification model using CNN. This approach necessitates the use of a large number of tagged facial images which in turn will be scarce. To address this issue, we introduce a “Data Augmentation Method Based on StyleGAN2” [18]. Create synthetic expression images of seven emotions for augmenting the performance of the model on the training data set. They also designed an expressive emotion detection model using VGG16 network.
As mentioned in [9], the study presents a new “in the wild” FER method. These sceneries have also affected image FER systems because of variations in “poses, occlusions, lighting, and skin tones”. Each of the 8 emotions was annotated with the “MediaPipe Face Mesh” model resulting in 479 normalized FER+ landmarks from the set of images. These standards are equivalent to 2556 face tessellations applied to incorporate transformer network-based characteristics [13]. This method is the only one that can normalise photos across situations and recover robust 3D face information from cameras. This work proposed an emotion categorization algorithm that achieves an accuracy of 73.7% on the FER+ dataset.
According to [10], the case study used “human thermal image processing” for FER on three human facial states: Normal, Sad, and Happy. It is clear that applying thermal pictures preprocessing and RF for feature selection and extraction assisted in the identification of important features. FFBPN which is the Feed forward Back propagation Neural Network gives its output as binary from the overall features and a particular set of features. Different input photos are grouped into pairs as usual for normal with sad or normal with cheerful to examine feelings. The preprocessing results in a very good classification performance as seen in table 4. [21]
This project demonstrates that the proposed emotion detection approach is capable of identifying and discerning visuals’ sentiment in real-time. The medical treatment, training, marketing, and human-computer interaction can be changed using our method due to its high precision, real-time efficiency, reliability, and defined user input.
SUMMARY OF THE INVENTION
This summary is provided to introduce a selection of concepts, in a simplified format, that are further described in the detailed description of the invention.
This summary is neither intended to identify key or essential inventive concepts of the invention and nor is it intended for determining the scope of the invention.
To further clarify advantages and features of the present invention, a more particular description of the invention will be rendered by reference to specific embodiments thereof, which is illustrated in the appended drawings. It is appreciated that these drawings depict only typical embodiments of the invention and are therefore not to be considered limiting of its scope. The invention will be described and explained with additional specificity and detail with the accompanying drawings.
This research relies on data to train and evaluate machine learning models. Our data collection for this article includes images of anger, disgust, fear, happiness, sadness, shock, and surprise. The model was trained with 1000 member images for these emotions.
BRIEF DESCRIPTION OF THE DRAWINGS
The illustrated embodiments of the subject matter will be understood by reference to the drawings, wherein like parts are designated by like numerals throughout. The following description is intended only by way of example, and simply illustrates certain selected embodiments of devices, systems, and methods that are consistent with the subject matter as claimed herein, wherein:
FIGURE 1: Images of sample input
FIGURE 2: CNN Model
FIGURE 3: Flowchart of proposed Model
The figures depict embodiments of the present subject matter for the purposes of illustration only. A person skilled in the art will easily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the disclosure described herein.
DETAILED DESCRIPTION OF THE INVENTION
The detailed description of various exemplary embodiments of the disclosure is described herein with reference to the accompanying drawings. It should be noted that the embodiments are described herein in such details as to clearly communicate the disclosure. However, the amount of details provided herein is not intended to limit the anticipated variations of embodiments; on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the scope of the present disclosure as defined by the appended claims.
It is also to be understood that various arrangements may be devised that, although not explicitly described or shown herein, embody the principles of the present disclosure. Moreover, all statements herein reciting principles, aspects, and embodiments of the present disclosure, as well as specific examples, are intended to encompass equivalents thereof.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments. As used herein, the singular forms “a",” “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes” and/or “including,” when used herein, specify the presence of stated features, integers, steps, operations, elements and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and/or groups thereof.
It should also be noted that in some alternative implementations, the functions/acts noted may occur out of the order noted in the figures. For example, two figures shown in succession may, in fact, be executed concurrently or may sometimes be executed in the reverse order, depending upon the functionality/acts involved.
In addition, the descriptions of "first", "second", “third”, and the like in the present invention are used for the purpose of description only, and are not to be construed as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Thus, features defining "first" and "second" may include at least one of the features, either explicitly or implicitly.
Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which example embodiments belong. It will be further understood that terms, e.g., those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
Dataset
This research relies on data to train and evaluate machine learning models. Our data collection for this article includes images of anger, disgust, fear, happiness, sadness, shock, and surprise. The model was trained with 1000 member images for these emotions.
This work used different emotions in order to classify the image based on the emotion as depicted below in Figure 1.
Data Pre-Processing
• Data collection: Gather a set of facial images of different people with different expressions. In order to improve the generalization and flexibility of the model, the dataset should be characterized by different people, men and women, people of different ages, and their conditions.
• Removed pictures which were not relevant to the dataset. Delete images which are corrupted or wrong labeled in order to have a good data set.
• Transform the labels into numerical form so that it will be easy to feed to any machine learning model for example one hot encoded. Various numbers should be assigned to various emotions.
• Standardize and Normalize: All picture pixel values should be set to one standard deviation and zero mean. This helps in the quick flattening of gradients and therefore helps in overcoming chaotic training process.
• In order to get more training instances, the set can be made larger by adding translations and rotations, for instance. Overfitting can to some extent be reduced by using data augmentation and also by increasing the size of the data set.
Models used
SVM is a binary class decision algorithm where the algorithm does not depend on probability to produce the output but rather divides the data points by hyper planes. It generates M-1 number hyperplanes in N dimension space for classifying data points with targets and features. SVM is determining the best hyperplane for all classes by working on the maximum margin. As a result of the classification of the tabular data into categories, SVMs are good at predicting diseases. [11]
f(x) = w * x + b (1)
If f(x) is the decision function
f(x) = Σ(α_i * y_i * K(x_i, x)) + b (2)

The decision function f(x) which is used to predict the class was trained with α_ilagrange multipliers. In this context, the kernal function K(x_i, x) the measure of similarity between an input sample x and a training sample x_i is done with the aid of the bias b terms or the intercept.
• CNN: Another name for CNNs is Deep Learning (DL) algorithms, which analyze input images and determine what specifics to look for. It also requires less preprocessing as compared to most other classification algorithms. It means that the filters can be learnt during training process of CNNs and structure of CNNs is based on the association pattern of neurons in human brain particularly in visual cortex. Receptive Fields that encompass the visual field are regions that cause CNN neurons to fire in response to stimulation.[15]
Testing finds errors. Finding all the defects and weaknesses of the product. This procedure tests the effectiveness of several components of the product in order to determine whether the software meets certain requirements and needs of the users without significant defects. Not all tests are the same, they serve different purposes.
The flowchart in Figure 3 shows a technique, algorithm or a paper process in its complete sequence.
A Real-Time Facial Expression Analysis for Emotion Recognition Using Image Processing comprises a data collection module, a data pre-processing module, a machine learning module, a CNN architecture, a SVM model, wherein the data collection module is used for gather a set of facial images of different people with different expressions.
In another embodiment after gathering, the data pre-processing module characterized the data set in order to improve the generalization and flexibility of the model by different people, men and women, people of different ages and their conditions;
Wherein remove pictures which are not relevant to the dataset and delete images which are corrupted or wrong labeled in order to have a good data set
In another embodiment the machine learning module is used to split the preprocessed data for training and testing.
In another embodiment after splitting the data, the training data transfer to the CNN architecture where it analyze input images and determine what specifics to look for;
Wherein the filters learnt during training process of CNNs and structure of CNNs is based on the association pattern of neurons in human brain particularly in visual cortex;
In another embodiment the testing data transfer to the SVM model;
Wherein the SVM is a binary class decision algorithm where the algorithm does not depend on probability to produce the output but rather divides the data points by hyper planes.
REFERENCES
[1] Renuka S. Deshmukh; VandanaJagtap; ShilpaPaygude “Facial emotion recognition system through machine learning approach” 2017 International Conference on Intelligent Computing and Control Systems (ICICCS) pp 2-5 DOI:10.1109/ICCONS.2017.8250725.
[2] Xueting Li; RongLian "Correlation Modeling of Moral Emotion based on Facial Image Emotion Recognition Algorithm" 2022 International Conference on Augmented Intelligence and Sustainable Systems (ICAISS) DOI: 10.1109/ICAISS55157.2022.10011045
[3] Marius Dan Zbancioc; Silvia Monica Feraru "Emotion Recognition for Romanian Language Using MFSC Images with Deep-Learning Neural Networks" 2021 International Conference on e-Health and Bioengineering (EHB) DOI: 10.1109/EHB52898.2021.9657669
[4] Mohammed Ali Shaik, “A Survey on Text Classification methods through Machine Learning Methods”, International Journal of Control and Automation (IJCA), ISSN:2005-4297,Volume-12,Issue-6 (2019), Pp.390-396.
[5] Japleen Kaur; JhalakSaxena; Jayesh Shah; Fahad; Satya Prakash Yadav “Facial Emotion Recognition”2022 International Conference on Computational Intelligence and Sustainable Engineering Solutions (CISES) DOI: 10.1109/CISES54857.2022.9844366
[6] Haimin Zhang; Min Xu "Weakly Supervised Emotion Intensity Prediction for Recognition of Emotions in Images" IEEE Transactions on Multimedia DOI: 10.1109/TMM.2020.3007352
[7] Firoz Mahmud; Bayezid Islam; Arfat Hossain; PushpenBikashGoala “Facial Region Segmentation Based Emotion Recognition Using K-Nearest Neighbours” 2018 International Conference on Innovation in Engineering and Technology (ICIET) DOI: 10.1109/CIET.2018.8660900
[8] Mohammed Ali Shaik, GeethaManoharan, B Prashanth, NuneAkhil, Anumandla Akash and Thudi Raja Shekhar Reddy, (2022), "Prediction of Crop Yield using Machine Learning", International Conference on Research in Sciences, Engineering & Technology, AIP Conf. Proc. 2418, 020072-1–020072-8; https://doi.org/10.1063/5.0081726, Published by AIP Publishing. 978-0-7354-4368-6, pp. 020072-1 to 020072-8
[9] Albert C. Cruz;B. Bhanu;N. S. Thakoor “One shot emotion scores for facial emotion recognition” 2014 IEEE International Conference on Image Processing (ICIP) DOI: 10.1109/ICIP.2014.7025275
[10] Tomoki Kusunose; Xin Kang; Keita Kiuchi; Ryota Nishimura; Manabu Sasayama; Kazuyuki Matsumoto “Facial Expression Emotion Recognition Based on Transfer Learning and Generative Model” 2022 8th International Conference on Systems and Informatics (ICSAI) DOI: 10.1109/ICSAI57119.2022.10005478
[11] Mohammed Ali Shaik, “Time Series Forecasting using Vector quantization”, International Journal of Advanced Science and Technology (IJAST), ISSN:2005-4238,Volume-29,Issue-4 (2020), Pp.169-175.
[12] Junhwan Kwon; Kyeong Teak Oh; Jaesuk Kim; Oyun Kwon; HeeCheol Kang; Sun K. Yoo "Facial Emotion Recognition using Landmark coordinate features" 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) DOI: 10.1109/BIBM58861.2023.10385536
[13] Sorin Pavel; Simona Moldovanu; Dorel Aiordachioaie "Emotion Recognition in Human Thermal Images with Artificial Intelligence Technology" 2023 IEEE 28th International Conference on Emerging Technologies and Factory Automation (ETFA) DOI: 10.1109/ETFA54631.2023.10275377
[14] Meaad Hussein Abdul-Hadi;"Human Speech and Facial Emotion Recognition Technique Using SVM" Jumana Waleed 2020 International Conference on Computer Science and Software Engineering (CSASE) DOI: 10.1109/CSASE48920.2020.9142065
[15] AnushreeBasu; AurobindaRoutray; Suprosanna Shit; AlokKanti Deb "Human emotion recognition from facial thermal image based on fused statistical feature and multi-class SVM" 2015 Annual IEEE India Conference (INDICON) DOI: 10.1109/INDICON.2015.7443712
[16] M. A. Shaik, Y. Sahithi, M. Nishitha, R. Reethika, K. SumanthTeja and P. Reddy, "Comparative Analysis of Emotion Classification using TF-IDF Vector," 2023 International Conference on Self Sustainable Artificial Intelligence Systems (ICSSAS), Erode, India, 2023, pp. 442-447, doi: 10.1109/ICSSAS57918.2023.10331897.
[17] Sabrina Begaj; Ali Osman Topal; Maaruf Ali “Emotion Recognition Based on Facial Expressions Using Convolutional Neural Network (CNN)” 2020 International Conference on Computing, Networking, Telecommunications & Engineering Sciences Applications (CoNTESA) DOI: 10.1109/CoNTESA50436.2020.9302866
[18] Mohammed Ali Shaik and DhanrajVerma, (2022), "Prediction of Heart Disease using Swarm Intelligence based Machine Learning Algorithms", International Conference on Research in Sciences, Engineering & Technology, AIP Conf. Proc. 2418, 020025-1–020025-9; https://doi.org/10.1063/5.0081719, Published by AIP Publishing. 978-0-7354-4368-6, pp. 020025-1 to 020025-9
[19] HimajaAvula; Ranjith R; Anju S Pillai "CNN based Recognition of Emotion and Speech from Gestures and Facial Expressions" 2022 6th International Conference on Electronics, Communication and Aerospace Technology DOI: 10.1109/ICECA55336.2022.10009316
[20] Mohammed Ali Shaik, M. Varshith, S. SriVyshnavi, N. Sanjana and R. Sujith, “Laptop Price Prediction using Machine Learning Algorithms”, 2022 International Conference on Emerging Trends in Engineering and Medical Sciences (ICETEMS), Nagpur, India, 2022, pp. 226-231, doi: 10.1109/ICETEMS56252.2022.10093357
[21] Rishu; Vinay Kukreja; Sahil Chauhan "Analysis of Facial Expression for Emotion Recognition using CNN-SVM" 2023 5th International Conference on Inventive Research in Computing Applications (ICIRCA) DOI: 10.1109/ICIRCA57980.2023.10220858
, C , Claims:1. A method for real-time facial expression analysis for emotion recognition using image processing, comprising the steps of:
Collecting a set of facial images of different individuals displaying various expressions;
Preprocessing the facial images to normalize and standardize pixel values, augment the dataset, and remove irrelevant or corrupted images;
Applying a machine learning model to classify facial expressions into predefined emotions based on the preprocessed data.
2. The method as claimed in claim 1, wherein the machine learning model includes a Convolutional Neural Network (CNN) for analyzing input images, and wherein the CNN filters are learned during the training process.
3. The method as claimed in claim 1, wherein the machine learning model further includes a Support Vector Machine (SVM) for classifying the testing data based on a binary decision algorithm using hyperplane divisions.
4. A system for real-time facial expression analysis for emotion recognition using image processing, comprising:
A data collection module configured to gather facial images of different individuals with various expressions;
A data pre-processing module configured to transform labels into numerical values, standardize and normalize image pixel values, and augment the dataset to enhance model generalization;
A machine learning module for splitting data into training and testing sets.
5. The system as claimed in claim 4, wherein the machine learning module uses a Convolutional Neural Network (CNN) to analyze the input images, and wherein the CNN's structure is based on the visual cortex pattern of neurons.
6. The system as claimed in claim 4, wherein the machine learning module uses a Support Vector Machine (SVM) for classifying facial emotions based on hyperplane divisions in a multi-dimensional space.
7. A method for improving the generalization and flexibility of a facial expression recognition system, comprising:
Collecting facial images from a diverse set of individuals, representing various emotional expressions, gender, age, and conditions;
Preprocessing the facial images to remove irrelevant or corrupted images, standardize the pixel values, and augment the dataset by rotating and translating images to reduce overfitting.
8. The method as claimed in claim 7, wherein the dataset is augmented using a StyleGAN2-based data augmentation method to generate synthetic facial expression images for training purposes.
9. A method for real-time emotion recognition from facial expressions, wherein the method comprises the steps of:
Preprocessing facial images to extract relevant features and remove irrelevant data;
Analyzing the preprocessed images using a deep learning model, wherein the model includes a CNN for feature extraction, followed by an SVM for classification into multiple emotional categories.
10. The method as claimed in claim 9, wherein the emotion categories include anger, disgust, fear, happiness, sadness, shock, and surprise, and the classification results are used for applications in healthcare, education, and human-computer interaction.