Description: Field of Invention
The present invention relates to mechanical keyboards and mouse used in computers such as qwerty keyboards, using Computer Vision and Machine learning we can develop a virtual keyboard and mouse using this we can implement more language support as well as various layout of keyboards and we can control the mouse using finger gestures.
The Objectives of this Invention
The purpose of the invention is to replace the commonly used mechanical keyboards and mouse used in computers. Users can use simple finger gestures to control or interact with the keyboard or mouse without physically touching them. This improves the productivity while using projector screens for explanation as we can draw, write and type with our fingers there is no need of going to keyboard and mouse multiple times in the middle of explanation.
Background of the Invention
First, type of technique has been introduced in (CN2003/1275167C), it pertains to a controller for a virtual input device that can be utilized in a distributed environment to manage a computer's far end remotely. An input interface, a conversion device, a first solutions based, and a second output functionality make up a converter, with the input interface processing internet traffic sent across a connection by an even more end system. Another type of technique is (US2009/8176324B1), A secure virtual keyboard, portrayed as an electronic keyboard, with many degrees of protection is provided through structures and procedures. The secure virtual keyboard has keys which are a mix of numbers, letters, and special characters. Randomly modifying the key layout, offering an alternative for transgenic mice inputs, by using a secured communication.
In (US2010/8462131B2), A building, a capacitive touch screen handled by the residences, as well as a CPU transported by the accommodation and collaborating with both the display panel are all possible components of an electrical gadget. The CPU could be used to control a physical keyboard on a smartphone in order to receive linguistic user input. Based on earlier textual input, the processor may assign a predicted key value to the at least one remappable key selectively. In (WO2004/006080A3), The invention pertains to a data computer processor (100) that allows a user to input symbols and includes a tiny bit member (200) that may be used as a virtual keyboard and probably includes one touchscreen interface (160) for recognizing contacted regions on the membership. Another inventions (US2006/0232558A1), virtual keyboards with a simulated territory, at least one or two vibrations detectors, a storage unit, as well as a peripheral device is disclosed in the biosensor. The electronic sector is made up of a number of synthetic elements and is built on a substrate. On the surface, vibrating sensors are deployed. The storage facility is being used to save a range of sources, each of which records the physical address and meaning from each of the imaginary keys.
Summary of the Invention
The proposed invention is to replace the commonly used mechanical keyboards and mouse used in computers. Users can use simple finger gestures to control or interact with the keyboard or mouse without physically touching them.
Detailed Description of the Invention
Throughout this document the Fig 1 is considered as for referencing. The preferable implementation will be detailed as employed for detecting hand, hand-knuckles and a type-able keyboard and gesture-controlled mouse using computer vision and media pipe modules in the proposed invention. The components used in this proposed invention are laptop with camera enabled.
The Computer Vision and Media pipe Hand Tracking modules are used in identifying the hand-knuckles. The important attributes of this project are displaying the virtual keyboard, identifying the hand-knuckles, distance between hand-knuckles to ensure the clicks. The entire task is based on python programming language. Then comes to efficiency of this software, up to now we will get around 40% to 60% of accuracy and efficiency. The detailed working process of this invention is discussed below. We should have a device with a camera and OpenCV, Media pipe, pynput, autopy libraries and python installed in it. Once the setting is done, we display the virtual key-board on screen which is now ready to detect the keys. A proper click gesture is made by the user, i.e., click operation is performed with the help of index finger tip and middle finger tip.
We select the key using index fingertip by moving fingertip among the keys, to print that key we have to minimize the distance between index finger tip and middle fingertip. The two consecutive frames are captured by the camera and only if the difference in the two frames is noticeable the gesture is recognized as a click. Once a proper click is made on any key the center of that key is marked with green color for a millisecond. The difference in the area of the click and that of the center is calculated to make sure that the key whose co-ordinates encompasses the click is detected and displayed.
Now, coming to mouse, it also uses the similar pattern with small modifications to recognize the gestures of fingers. We can move the cursor on screen using index and a click operation is performed with the help of middle finger. Small smothering algorithm is used to optimize the movement of cursor. Autopy library is helpful in identifying and inputs - performed mouse operations to the computer system.
5 Claims & 1 Figure
Brief description of Drawing
In the figure which are illustrate exemplary embodiments of the invention.
Figure 1, the Process of Virtual Keyboard and Mouse , Claims: The scope of the invention is defined by the following claims:

Claim:
1. A system/method for image processing based virtual keyboard and mouse, said system/method comprising the steps of:
a) The system starts with application camera (2), then the users (1) have to moisture the action (3).
b) Based on the gesture detection, the palm location can be detected (4), then finger tracking is identified (5) from the gesture recognition (6) and templates (7).
c) If it is gesture recognized then interaction operation (8) is initiated, then we can operate the mouse/keyboard (9).
d) If gesture is not detected (10), then we can rematch the images from (3).
2. As mentioned in claim 1, the invented system gets the user inputs and capture via the cameras.
3. According to claim 1, if it gesture is detected, first detect the palm location and then finger tracking.
4. According to claim 1, after that, the gesture is analyzed from the gesture templates, according to this templates, we can easily operate the virtual key board and mouse.
5. As per claim 1, if it gesture is not detected, then restart the whole process from image capturing process.