Description:Title:

SILENT-SPEAK: A MACHINE LEARNING-BASED MODEL FOR INDIVIDUALS WITH HEARING AND SPEECH IMPAIRMENTS

Field of the Invention

[0001] The present invention falls within the realm of Computer Science and machine learning field.
[0002] This innovation refers to mobile application; facilitate communication for individuals who are deaf and mute. Moreover, it serves as a valuable resource for individuals with normal hearing, aiding in their comprehension of sign language. This development represents a crucial stride towards bridging the communication gap that exists between those with hearing and speech impairments and the general population. The Silent Speech app is equipped with features enabling the conversion of text to sign, image to sign, voice to sign, and sign to text additionally with two more features object detection and language identification.

Background

[0003] The development of our advanced sign language detection model is rooted in a commitment to addressing the profound communication challenges faced by individuals with speech and hearing impairments. In contemporary society, there exists a critical need for innovative solutions that go beyond traditional methods to empower and enhance the communication capabilities of this community.
[0004] The model encapsulates a holistic approach, encompassing six pivotal features designed to revolutionize the landscape of sign language communication. The first two features, sign to text and text to sign, cater to the fluid translation of gestures and written language, facilitating seamless interaction between individuals fluent in sign language and those who are not. Meanwhile, the voice to sign functionality enables users with speech difficulties to express themselves through sign language.
[0005] Adding a layer of visual comprehension, the image to sign feature allows users to interpret textual content within images, such as quotes or captions, by transforming it into equivalent sign language. To broaden the scope of application, the model incorporates object detection, enhancing real-world interactions. Moreover, the inclusion of language identification ensures adaptability to diverse linguistic contexts, making the model versatile and globally applicable.
[0006] By synergizing these features, our model aspires to not only break down communication barriers for those with impairments but also serve as an educational tool, promoting broader awareness and understanding of sign language among the general population. Through this innovative endeavor, we seek to foster inclusivity, enrich user experiences, and contribute significantly to a more accessible and connected world.

Objects of the Invention

[0007] The objectives of the present disclosure include:
• Providing assistance to individuals facing challenges with speaking and hearing, offering features for converting signs to text, signs to images, signs to voices, and signs to text in images.
• Enabling individuals with speech difficulties to easily translate their hand gestures into appropriate text through sign language.
• Offering functionality for speech to sign communication, aiding users in understanding the symbols corresponding to spoken words or sentences.
• Allowing users to comprehend the meaning of text within an image, such as quotes, by transforming it into equivalent sign language.
• Serving as a guide for individuals without impairments to familiarize themselves with sign language and learn its principles.

Summary

[0008] The sign language detection and translation model represents a transformative leap in communication technology, particularly tailored to empower individuals with speech and hearing impairments. Offering a comprehensive suite of features, the model seamlessly translates text, images, and voice into sign language, providing real-time communication enriched with dynamic animations for enhanced comprehension. The integration of object detection extends the model's utility beyond communication scenarios, fostering immersive real-world interactions.
[0009] The model's language adaptability ensures global relevance, catering to diverse linguistic contexts and making it accessible to users worldwide. Simultaneously, its role as an educational resource contributes to broader awareness and inclusivity, allowing individuals without impairments to learn and understand sign language.
[0010] By breaking down communication barriers, our model aligns with the principles of universal design, fostering a more empathetic and accessible society. This innovation stands as a testament to our commitment to advancing technology for the betterment of individuals with special abilities, creating a pathway toward a more inclusive and connected world.

Drawings

Figure 1: Algorithm Process Flowchart

Brief Description of the Drawing

[0011] The figure 1 represents Algorithm Process Flowchart of the working model in the present invention.

Figure 2: Data Flow Diagram

Brief Description of the Drawing

[0012] The figure 2 represents Data Flow Diagram of the working model in the present invention.

Detailed Description of the Drawings

[0013] As shown in the algorithm process flowchart and data flow diagram of the working model, the present invention is developing a mobile application design to cater to the unique needs of individuals with speech and hearing impairments. This innovative model aims to provide assistance by offering features such as text-to-sign, image-to-sign, voice-to-sign, text-to-sign, object detection and Language Identification functionalities. Sign language serves as a crucial mode of communication for individuals facing challenges with hearing or speech. Nevertheless, effective communication can pose difficulties for those unfamiliar with sign language.
[0014] The Technology used in the invention is as follows:
• CNN Model
• Open-CV
• Flask
• Keras
• Scikit-learn
• Firebase ML Kit
• Google Vision API
• Android Studio

[0015] There are a number of potential end users and benefits of the proposed invention which includes:
• Individuals with Speech and Hearing Impairments: Primary beneficiaries using the model for natural expression through sign language. There is seamless communication with both sign language users and those unfamiliar with sign language.
• Individuals with Speech Difficulties: Utilize voice to sign functionality for effective articulation of thoughts and ideas using sign language.
• Users Interacting with Visual Content: Employ image to sign feature to interpret textual content within images, improving understanding of visual information.
• General Users in Real-world Scenarios: Engage with object detection features for enriched interactions beyond communication, applicable in daily life situations.
• Global Audience: Language identification feature adapts to diverse linguistic contexts, making the model accessible and relevant globally.
• Learners and Educational Institutions: Individuals without impairments can use the model as an educational resource to learn and understand sign language. Educational institutions can leverage the model for inclusive learning environments.

Advantages of the Invention

[0016] Here are some of the key advantages of the proposed invention:
1. Enhanced Communication Access: Enables individuals with speech and hearing impairments to communicate effectively through sign language, fostering inclusivity and overcoming communication barriers.
2. Versatile Sign Language Translation: Offers diverse translation options, including text to sign, image to sign, voice to sign, and sign to text, providing flexibility based on user preferences and needs.
3. Real-time Translation and Animation: Facilitates real-time translation of written text into sign language with accompanying animations, enhancing communication speed and aiding in immediate comprehension.
4. Voice-to-Sign Convenience: Allows users to convert spoken words into sign language through the voice-to-sign feature, catering to individuals who may face challenges in typing, thereby improving accessibility.
5. Image-to-Sign Tool: Empowers users to comprehend textual content within images by converting them into sign language, offering an innovative solution for understanding information in visual formats.
6. Object Detection for Interaction: Integrates object detection to enhance real-world interactions, contributing to a more immersive and engaging user experience beyond communication scenarios.
7. Language Adaptability: Utilizes language identification for adaptability to diverse linguistic contexts, making the model globally relevant and accessible to users irrespective of their language preferences.
8. Educational Empowerment: Serves as an educational resource, allowing individuals without impairments to learn and understand sign language, fostering awareness and inclusivity on a broader scale.
9. Empathetic Communication: Contributes to a more empathetic society by promoting understanding between individuals with and without speech and hearing impairments, reducing societal barriers.
10. Inclusive Learning Environments: Enhances inclusivity in educational settings by offering a tool that facilitates communication and learning for individuals with special abilities, creating more inclusive learning environments. , Claims:Following are the claims of the invention:
1. A model designed for individuals with special abilities, facilitating the comprehension and acquisition of signs corresponding to text and voice through the utilization of machine learning.
2. The system of claim 1 utilizes:
• Sign recognition
• Text recognition
• Text extraction from images
• Speech recognition
• Object detection
• Language identification
3. The system in claim 1 allows users to access the web application via a computer or mobile device. Users can choose from four signing options: Text to sign, Image to sign, Voice to sign, or Sign to text.
4. The system in claim 1 is particularly beneficial for users with hearing impairments. It assists in analyzing written text and generating an animation of signs corresponding to words. This animation can be translation.
5. The system in claim 1 includes a voice to sign feature, allowing users to record their voice and convert it into sign language. This feature is especially useful for individuals who are unable to type. It operates by analyzing spoken words and generating an animation of a person signing the words, which can be displayed on a screen.
6. The system in claim 1 enables users to upload photographs and convert them into sign language using the image-to-sign tool. The model first extracts text from the image, providing assistance to those seeking to comprehend textual content within images.
7. The system in claim 1 empowers users to translate their signs into appropriate text, displayed on the screen. This functionality enhances the understanding of the connection between signs and text for individuals with disabilities.