1. A system for lip-reading recognition for Hindi learning, comprising: a video input device for capturing video of a speaker's face while they are speaking Hindi; a lip-reading recognition engine configured to analyse the video and generate corresponding Hindi text based on the speaker's lip movements; and a display or output device for presenting the generated Hindi text to the user. 2.The system of claim 1, wherein the lip-reading recognition engine uses machine learning algorithms to improve accuracy and adapt to variations in lip movements across different speakers. 3.The system of claim 1, further comprising a database of common Hindi phrases or vocabulary to aid in recognition and assist with language learning. 4.A method for lip-reading recognition for Hindi learning, comprising: capturing video of a speaker's face while they are speaking Hindi using a video input device; analysing the video using a lip-reading recognition engine to generate corresponding Hindi text based on the speaker's lip movements; presenting the generated Hindi text to the user using a display or output device. The method of claim 4, further comprising providing feedback to the user on their pronunciation or lip movements based on a comparison of the generated Hindi text with the expected or correct pronunciation. 5.The method of claim 4, further comprising allowing the user to customize the lip-reading recognition engine to adapt to their individual lip movements or pronunciation. 6.The method of claim 1, further comprising activating the lip-reading recognition engine uses machine learning algorithms to improve accuracy and adapt to variations in lip movements across different speakers. 7.The system of claim 1, wherein the lip-reading recognition engine utilize a database of common Hindi phrases or vocabulary to aid in recognition and assist with language learning. Lip-reading recognition method for Hindi learning Abstract A system to recognise what a person is saying with their lips in order to help them learn Hindi. A video input device can be arranged to capture footage of the face of a speaker as they are delivering a speech in Hindi. In certain implementations, there is also a lip-reading recognition engine that can analyse the video and generate relevant Hindi text depending on the motions of the speaker's lips. The created Hindi text may be shown to the user through a display or other output device in certain implementations. However, this is not required. In some instances, this could really be the case. , Claims:Claims :

1. A system for lip-reading recognition for Hindi learning, comprising: a video input device for capturing video of a speaker's face while they are speaking Hindi; a lip-reading recognition engine configured to analyse the video and generate corresponding Hindi text based on the speaker's lip movements; and a display or output device for presenting the generated Hindi text to the user. 2.The system of claim 1, wherein the lip-reading recognition engine uses machine learning algorithms to improve accuracy and adapt to variations in lip movements across different speakers. 3.The system of claim 1, further comprising a database of common Hindi phrases or vocabulary to aid in recognition and assist with language learning. 4.A method for lip-reading recognition for Hindi learning, comprising: capturing video of a speaker's face while they are speaking Hindi using a video input device; analysing the video using a lip-reading recognition engine to generate corresponding Hindi text based on the speaker's lip movements; presenting the generated Hindi text to the user using a display or output device. The method of claim 4, further comprising providing feedback to the user on their pronunciation or lip movements based on a comparison of the generated Hindi text with the expected or correct pronunciation. 5.The method of claim 4, further comprising allowing the user to customize the lip-reading recognition engine to adapt to their individual lip movements or pronunciation. 6.The method of claim 1, further comprising activating the lip-reading recognition engine uses machine learning algorithms to improve accuracy and adapt to variations in lip movements across different speakers. 7.The system of claim 1, wherein the lip-reading recognition engine utilize a database of common Hindi phrases or vocabulary to aid in recognition and assist with language learning.

Description:Lip-reading recognition method for Hindi learning
Field of the Invention
[0001] The present invention relates to lip reading and voice recognition technique. More particularly, relates to an image processing technique to combined lip reading for Hindi language learning.
Background
[0002] The background description includes information that may be useful in understanding the present invention. It is not an admission that any of the information provided herein is prior art or relevant to the presently claimed invention, or that any publication specifically or implicitly referenced is prior art.
[0003] The lip-reading recognition technique has a wide range of potential applications, including improving communication for individuals with hearing impairments, speech recognition in noisy environments, and enhancing the accuracy of automatic speech recognition systems. It can also be used in the development of artificial intelligence based virtual assistants and chatbots that can understand and respond to lip movements in addition to audio input.
[0004] Various technological solutions (such as the identity identifying method that a kind of voice and lip reading blend, feature extraction method, lip-reading classification method, device and apparatus, etc.) for lip reading techniques are disclosed in patent literature.
[0005] The WO2016201679A1 relates to a feature extraction method and a lip-reading classification method, device and apparatus, relating to the field of feature recognition, the method comprising: according to a time sequence of frames in a video, dividing the video into M time sub-blocks; dividing a lip portion region of each frame of each time sub-block into N space sub-blocks; assembling a video sub-block from space sub-frames at corresponding identical positions in each frame of the same time sub-block, the video comprising in total M×N video sub-blocks; calculating a lip-reading feature vector of each video sub-block, each video sub-block lip-reading feature vector being an X-dimensional vector; and combining the X-dimensional lip-reading feature vectors of the M×N video sub-blocks of the video to obtain an X×M×N-dimensional feature vector of the video. The present invention fixes the number of feature vector dimensions and does not require dynamic adjustment of the number of feature vector dimensions, simplifying operations and saving training time and classification time.
[0006] The CN110364163A (By- Guangdong xi’an jiaotong university academy, Xian Jiaotong University) relates to the identity identifying methods that a kind of voice and lip reading blend, and are related to information security and mode field of authentication.1) identity identifying method that the voice and lip reading blend is the following steps are included: extract speech cepstrum feature and lid speech characteristic to be certified；2) the speech cepstrum feature and the lid speech characteristic are merged using paralleling tactic, obtains complex vector feature；3) using complex vector feature as the multiple-input and multiple-output model of the plural extreme learning machine of training sample training, judge whether to authenticate successfully according to the distribution of output and its value；If exporting, the maximum value in result vector is consistent with prompt and the value is greater than given threshold, authenticates success；Otherwise, authentication failure.The identity identifying method that the voice and lip reading blend carries out authentication using the multimode biological characteristic that voice and lip reading combine, improves the robustness of identity authorization system.
[0007] The JP2021086274A (By- Kyushu Institute of Technology NUC) relates to a lip reading device and a lip reading method, capable of highly accurately estimating utterance content by performing machine learning in consideration of expression features in addition to lip features of an utterer and further increasing estimation accuracy by considering attributes such as the age and sex of the utterer as necessary.SOLUTION: An image processing unit 14 which subjects a learning target image and an evaluation target image to image processing comprises: face detecting means 22 for detecting a face region of a learning target utterer from the learning target image and a face region of an evaluation target utterer from the evaluation target image; face feature point detecting means 23 for detecting a face feature point from each face region; lip region extracting means 24 for extracting a lip region from a face feature point of each face region; and feature extracting means 25 for extracting expression features and lip features of the learning target utterer serving as learning target data from the face region, face feature point, and lip region of the learning target utterer and extracting expression features and lip features of the evaluation target utterer serving as evaluation target data from the face region, face feature point, and lip region of the evaluation target utterer.
[0008] One of the challenges in lip-reading recognition research is the complexity and variability of lip movements. Lip movements can be influenced by various factors such as facial expressions, speaking style, and cultural differences. Additionally, the variability in the way people speak and the different accents and dialects can make it difficult to develop a universal lip-reading recognition system. Thus, there is need of new research in domain of lip reading for Hindi language.

Summary
[0009] The present invention relates to lip reading and voice recognition technique. More particularly, relates to an image processing technique to combined lip reading for Hindi language learning.
[00010] The following presents a simplified summary of various aspects of this disclosure in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key or critical elements nor delineate the scope of such aspects. Its purpose is to present some concepts of this disclosure in a simplified form as a prelude to the more detailed description that is presented later.
[00011] The following paragraphs provide additional support for the claims of the subject application.
[00012] Embodiments of the present disclosure may include a system for lip-reading recognition for Hindi learning, including a video input device for capturing video of a speaker's face while they may be speaking Hindi. Embodiments may also include a lip-reading recognition engine configured to analyse the video and generate corresponding Hindi text based on the speaker's lip movements. Embodiments may also include a display or output device for presenting the generated Hindi text to the user.
[00013] In some embodiments, the method may include activating the lip-reading recognition engine that uses machine learning algorithms to improve accuracy and adapt to variations in lip movements across different speakers. In some embodiments, the lip-reading recognition engine utilize a database of common Hindi phrases or vocabulary to aid in recognition and assist with language learning.
[00014] Embodiments of the present disclosure may also include a method for lip-reading recognition for Hindi learning, including capturing video of a speaker's face while they may be speaking Hindi using a video input device. Embodiments may also include analysing the video using a lip-reading recognition engine to generate corresponding Hindi text based on the speaker's lip movements.
[00015] Embodiments may also include presenting the generated Hindi text to the user using a display or output device. Feedback can be provided to the user on their pronunciation or lip movements based on a comparison of the generated Hindi text with the expected or correct pronunciation. In some embodiments, the method may include allowing the user to customize the lip-reading recognition engine to adapt to their individual lip movements or pronunciation.
Brief Description of the Drawings
[00016] The features and advantages of the present disclosure would be more clearly understood from the following description taken in conjunction with the accompanying drawings in which:
[00017] FIG. 1 is a block diagram illustrating a system for lip-reading recognition for Hindi learning, according to some embodiments of the present disclosure.
[00018] FIG. 2 is a flowchart illustrating a method for lip-reading recognition for Hindi learning, according to some embodiments of the present disclosure.
Detailed Description
[00019] In the following detailed description of the invention, reference is made to the accompanying drawings that form a part hereof, and in which is shown, by way of illustration, specific embodiments in which the invention may be practiced. In the drawings, like numerals describe substantially similar components throughout the several views. These embodiments are described in sufficient detail to claim those skilled in the art to practice the invention. Other embodiments may be utilized and structural, logical, and electrical changes may be made without departing from the scope of the present invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is defined only by the appended claims and equivalents thereof.
[00020] The use of the terms a and an and the and at least one and similar referents in the context of describing the invention (especially in the context of the following claims) are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The use of the term at least one followed by a list of one or more items (for example, at least one of A and B) is to be construed to mean one item selected from the listed items (A or B) or any combination of two or more of the listed items (A and B), unless otherwise indicated herein or clearly contradicted by context. The terms comprising, having, including, and containing are to be construed as open-ended terms (i.e., meaning including, but not limited to,) unless otherwise noted. Recitation of ranges of values herein are merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein, and each separate value is incorporated into the specification as if it were individually recited herein. All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The use of any and all examples, or exemplary language (e.g., such as) provided herein, is intended merely to better illuminate the invention and does not pose a limitation on the scope of the invention unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the invention.
[00021] The present invention relates to lip reading and voice recognition technique. More particularly, relates to a image process technique to combined lip reading for Hindi language learning.
[00022] FIG. 1 is a block diagram that describes a system 100 for lip-reading recognition for Hindi learning, according to some embodiments of the present disclosure. In some embodiments, the system 100 may include a video input device 110 for capturing video of a speaker's face while he/she may be speaking Hindi, a lip-reading recognition engine 120 configured to analyse the video and generate corresponding Hindi text based on the speaker's lip movements, and a display device 130 (interchangeably referred as output device 130) for presenting the generated Hindi text to the user. In some embodiments, the lip-reading recognition engine 120 may use machine learning algorithms to improve accuracy and adapt to variations in lip movements across different speakers.
[00023] In some embodiments, the system 100 may include a database of common Hindi phrases or vocabulary to aid in recognition and assist with language learning. In some embodiments, the lip-reading recognition engine 120 may use machine learning algorithms to improve accuracy and adapt to variations in lip movements across different speakers. In some embodiments, the lip-reading recognition engine 120 may utilize a database of common Hindi phrases or vocabulary to aid in recognition and assist with language learning.
[00024] FIG. 2 is a flowchart that describes a method for lip-reading recognition for Hindi learning, according to some embodiments of the present disclosure. In some embodiments, at 210, the method may include capturing a video of a speaker's face while he/she may be speaking Hindi using the video input device 110.At 220, the method may include analysing the video using a lip-reading recognition engine 120 to generate corresponding Hindi text based on the speaker's lip movements. At 230, the method may include presenting the generated Hindi text to the user using a display or output device. Feedback can be provided to the user on their pronunciation or lip movements based on a comparison of the generated Hindi text with the expected or correct pronunciation. In some embodiments, the method may include allowing the user to customize the lip-reading recognition engine 120 to adapt to their individual lip movements or pronunciation.
[00025] In an embodiment, the system 100 may comprise the video input device 110 for the purpose of recording video of the speaker's face when he/she might be speaking Hindi. The lip-reading recognition engine 120 that is able of analysing the video and create appropriate Hindi text based on the speaker's lip movements may also be included in embodiments. In some embodiments, the produced Hindi text may be shown to the user through the output device 130. This may be the case in certain embodiments.
[00026] The lip-reading recognition engine 120may, in certain implementations, make use of machine learning methods in order to increase accuracy and adjust to differences in the way different speakers move their lips. A database of popular phrases or vocabulary in Hindi may be included in the system 100 in some embodiments. The database's purpose is to facilitate recognition and learning of the language.
[00027] Activating the lip-reading recognition engine 120 that employs machine learning algorithms to increase accuracy and adjust to changes in lip movements across various speakers is one such step that may be included in certain implementations of this approach. In some implementations, the lip-reading recognition engine 120 makes use of a database that contains popular Hindi phrases or vocabulary in order to facilitate recognition and to help in the process of learning the language.
[00028] The current disclosure may also comprise embodiments of a method for lip-reading recognition for the purpose of learning Hindi. This method may include recording video of a speaker's face while they are speaking Hindi using a video input device. The technique may also involve learning Hindi. In certain embodiments, there is also the possibility of analysing the video using the lip-reading recognition engine 120 in order to create appropriate Hindi text based on the motions of the speaker's lips.
[00029] Displaying the produced Hindi text to the user through the output device 130 is another option that may be included in certain embodiments. Feedback can be provided to the user in response tohis/her pronunciation or lip movements based on a comparison of the generated Hindi text with the expected or correct pronunciation. This feedback may be based on a comparison of the generated Hindi text with the correct pronunciation. In an aspect, the user may be allowed to tailor the lip-reading recognition engine 120 so that it adapts to their specific lip movements or pronunciation may be an element of the approach that is included in some versions of the method.
[00030] Lip-reading recognition is the process of recognizing spoken words by interpreting the movements of a person's lips. It is an important technique for speech recognition in noisy environments and for individuals with hearing impairments. The method has gained significant attention in the field of computer vision and speech processing. The disclosure suggests lip-reading recognition method for Hindi.
[00031] Hindi is one of the most widely spoken languages in India, and it is a complex language with 11 vowels and 33 consonants. The Hindi language has a rich vocabulary, and it is an important language for business, education, and politics in India. The lip-reading recognition method for Hindi can help individuals with hearing impairments to communicate effectively.
[00032] The lip-reading recognition method for Hindi involves three main stages: pre-processing, feature extraction, and classification.
Pre-processing:
[00033] The pre-processing stage involves the following steps:
[00034] a. Video Acquisition: A video of the person speaking Hindi is acquired using a camera.
[00035] b. Face Detection: The first step in lip-reading is to detect the face of the speaker in the video. Various face detection algorithms can be used for this purpose, such as Haar Cascade, Viola-Jones, and HOG.
[00036] c. Lip Segmentation: Once the face is detected, the next step is to segment the lips from the rest of the face. This can be done using color-based segmentation, edge-based segmentation, or model-based segmentation.
Feature Extraction:
[00037] The feature extraction stage involves extracting the relevant features from the segmented lips. The following features are commonly used in lip-reading recognition:
[00038] a. Shape Features: These features describe the shape of the lips and include parameters such as height, width, and thickness.
[00039] b. Texture Features: These features describe the texture of the lips and include parameters such as texture energy, contrast, and homogeneity.
[00040] c. Motion Features: These features describe the motion of the lips over time and include parameters such as velocity, acceleration, and jerk.
[00041] d. Phoneme Features: These features describe the phonemes (distinct units of sound in a language) and include parameters such as phoneme duration, formant frequency, and spectral envelope.
Classification:
[00042] The classification stage involves classifying the extracted features into different categories of spoken words. Various classification algorithms can be used for this purpose, such as Hidden Markov Models (HMM), Artificial Neural Networks (ANN), Support Vector Machines (SVM), and K-Nearest Neighbor (KNN) algorithms.
[00043] The lip-reading recognition method for Hindi has shown promising results in various studies. The accuracy of the lip-reading system depends on the quality of the video, lighting conditions, and the complexity of the spoken words. In a recent study, the lip-reading system achieved an accuracy of 78% for recognizing spoken words in Hindi.
[00044] The lip-reading recognition method for Hindi is a promising technique for speech recognition in noisy environments and for individuals with hearing impairments. The method involves pre-processing, feature extraction, and classification stages. Various algorithms can be used for each stage, and the accuracy of the system depends on the quality of the video and the complexity of the spoken words. Accuracy is improved of the lip-reading system for Hindi.
[00045] Example embodiments herein have been described above with reference to block diagrams and flowchart illustrations of methods and apparatuses. It will be understood that each block of the block diagrams and flowchart illustrations, and combinations of blocks in the block diagrams and flowchart illustrations, respectively, can be implemented by various means including hardware, software, firmware, and a combination thereof. For example, in one embodiment, each block of the block diagrams and flowchart illustrations, and combinations of blocks in the block diagrams and flowchart illustrations can be implemented by computer program instructions. These computer program instructions may be loaded onto a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions which execute on the computer or other programmable data processing apparatus create means for implementing the functions specified in the flowchart block or blocks.
[00046] Throughout the present disclosure, the term ‘processing means’ or ‘microprocessor’ or ‘processor’ or ‘processors’ includes, but is not limited to, a general purpose processor (such as, for example, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a microprocessor implementing other types of instruction sets, or a microprocessor implementing a combination of types of instruction sets) or a specialized processor (such as, for example, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), or a network processor).
[00047] The term non-transitory storage device or storage or memory, as used herein relates to a random access memory, read only memory and variants thereof, in which a computer can store data or software for any duration.
[00048] Operations in accordance with a variety of aspects of the disclosure is described above would not have to be performed in the precise order described. Rather, various steps can be handled in reverse order or simultaneously or not at all.
[00049] While several implementations have been described and illustrated herein, a variety of other means and/or structures for performing the function and/or obtaining the results and/or one or more of the advantages described herein may be utilized, and each of such variations and/or modifications is deemed to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and/or configurations will depend upon the specific application or applications for which the teachings is/are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific implementations described herein. It is, therefore, to be understood that the foregoing implementations are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, implementations may be practiced otherwise than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and/or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and/or methods, if such features, systems, articles, materials, kits, and/or methods are not mutually inconsistent, is included within the scope of the present disclosure.

Claims
I/We Claim:
1. A system for lip-reading recognition for Hindi learning, comprising:
a video input device for capturing video of a speaker's face while they are speaking Hindi;
a lip-reading recognition engine configured to analyse the video and generate corresponding Hindi text based on the speaker's lip movements; and
a display or output device for presenting the generated Hindi text to the user.

2.The system of claim 1, wherein the lip-reading recognition engine uses machine learning algorithms to improve accuracy and adapt to variations in lip movements across different speakers.

3.The system of claim 1, further comprising a database of common Hindi phrases or vocabulary to aid in recognition and assist with language learning.

4.A method for lip-reading recognition for Hindi learning, comprising:

capturing video of a speaker's face while they are speaking Hindi using a video input device;
analysing the video using a lip-reading recognition engine to generate corresponding Hindi text based on the speaker's lip movements;
presenting the generated Hindi text to the user using a display or output device.
The method of claim 4, further comprising providing feedback to the user on their pronunciation or lip movements based on a comparison of the generated Hindi text with the expected or correct pronunciation.

5.The method of claim 4, further comprising allowing the user to customize the lip-reading recognition engine to adapt to their individual lip movements or pronunciation.

6.The method of claim 1, further comprising activating the lip-reading recognition engine uses machine learning algorithms to improve accuracy and adapt to variations in lip movements across different speakers.

7.The system of claim 1, wherein the lip-reading recognition engine utilize a database of common Hindi phrases or vocabulary to aid in recognition and assist with language learning.

Lip-reading recognition method for Hindi learning
Abstract
A system to recognise what a person is saying with their lips in order to help them learn Hindi. A video input device can be arranged to capture footage of the face of a speaker as they are delivering a speech in Hindi. In certain implementations, there is also a lip-reading recognition engine that can analyse the video and generate relevant Hindi text depending on the motions of the speaker's lips. The created Hindi text may be shown to the user through a display or other output device in certain implementations. However, this is not required. In some instances, this could really be the case. , Claims:Claims
I/We Claim:
1. A system for lip-reading recognition for Hindi learning, comprising:
a video input device for capturing video of a speaker's face while they are speaking Hindi;
a lip-reading recognition engine configured to analyse the video and generate corresponding Hindi text based on the speaker's lip movements; and
a display or output device for presenting the generated Hindi text to the user.

2.The system of claim 1, wherein the lip-reading recognition engine uses machine learning algorithms to improve accuracy and adapt to variations in lip movements across different speakers.

3.The system of claim 1, further comprising a database of common Hindi phrases or vocabulary to aid in recognition and assist with language learning.

4.A method for lip-reading recognition for Hindi learning, comprising:

capturing video of a speaker's face while they are speaking Hindi using a video input device;
analysing the video using a lip-reading recognition engine to generate corresponding Hindi text based on the speaker's lip movements;
presenting the generated Hindi text to the user using a display or output device.
The method of claim 4, further comprising providing feedback to the user on their pronunciation or lip movements based on a comparison of the generated Hindi text with the expected or correct pronunciation.

5.The method of claim 4, further comprising allowing the user to customize the lip-reading recognition engine to adapt to their individual lip movements or pronunciation.

6.The method of claim 1, further comprising activating the lip-reading recognition engine uses machine learning algorithms to improve accuracy and adapt to variations in lip movements across different speakers.

7.The system of claim 1, wherein the lip-reading recognition engine utilize a database of common Hindi phrases or vocabulary to aid in recognition and assist with language learning.