DESC:FIELD OF THE INVENTION
Embodiment of the present invention relates to a system and method for providing haptic feedback to user in a virtual reality environment via a sole wearable and more particularly for providing enriched haptic feedback to a user via a sole configured with sensors to experience varied terrains, and assist user with foot reflexology.
BACKGROUND OF THE INVENTION
For the virtual reality world, creating realistic experiences with virtual objects has been a constant area of research. Various organizations, research groups and scientific communities are intensely exploring options of generating real world experience via body wearables. Many have been developed in the course of this research, such as suits and armors, for transmitting various contextual bodily sensations to the wearer. However, majority of development has focused on creating haptic feedback via hand or gloves or other wearables that generates touch sensation in upper half of body.
Though hands are the most widely and effectively used tool by humans for experiencing and manipulating the world around them, the role and significance of sensations perceived through foot cannot be overlooked. User feet are the only parts of bodies that is in constant contact with the surrounding environment. Hence, this is one area that has much room for improvement for much realistic, interactive locomotion in virtual reality spaces. Especially in a virtual reality world, where the user is looking for most engaging and high fidelity immersive experience, feedback from lower half of body and particularly the foot of the user becomes extremely important as it may enhance a perception of tangibility of the virtual environment. By synergistically combining visual and tactile perception, users can achieve efficient and convenient interaction within virtual environments.
With high performance of hardware interfaces and tracking technologies, people are now engaged in more dynamic, whole body activities in a large space such as walking, dancing, and jumping. In this regard, the foot of wearer provides vital information to human body especially when one intends to feel the ground, terrain or topography of place to where the user is transferred in virtual world.
For example, a virtual reality user exploring Rocky Mountains, enjoying green grass walk, struggling with muddy waters, walking bare foot on beach side wet sand or grainy sand of deserts would want that he can sense the same pressure and touch of underlying surface as if in real world. Besides experiencing different types of terrain, the sensations on bare foot are vital from perspective of reflexology. In terms of reflexology, right foot is associated with the right part of the body and left foot is associated with the left side of the body. The stomach, for example, is primarily located on the left side of the body so massaging and applying pressure to the left foot can treat stomach ailments.
Many prior art have explored electrotactile stimulators for achieving real time haptic; however when used in isolation, they fail to provide rich textural rendering besides requiring direct interfacing of electrodes with the skin and other calibrations. Further, there are limited options of exploring attachments to toes as they are joints and critical for walking movement. Body weight is another limiting and constraining factor as vibrations are better perceived for lighter users. Also, need lot of customization to suit each user’s foot size as deviations smaller than even a centimetre can cause misalignment between the shape of the foot and the placement of the actuators.
This sets need for a one size fit all solution which is more generalized. Since, not much has been accomplished from the perspective of devising a solution that provides an enriched and complete immersive experience when travelling in a virtual reality realm, there exists a need for a body wearable that provides true haptic feedback to sole of user so that the bottom surface of the virtual reality environment can be reproduced on the user's sole as realistically as possible.
In the background of foregoing limitations, a ready, handy, light and highly effective sole wearable that can help user fully immerse in a virtual reality space is needed. The present disclosure sets forth system and method for providing tactile feedback to user sole such that a very real and lifelike experience of underlying surface can be sensed by the user. This disclosure embodies advantageous alternatives and improvements to existing haptic feedback systems and methods, which may address one or more of the challenges or needs mentioned herein, as well as provide other benefits and advantages.
OBJECT OF THE INVENTION
An object of the present invention is to provide a system and method for enabling haptic feedback to user via an interactive sole wearable for full immersive experience.
Another object of the present invention is to provide a sensor based wearable sole that can be worn by the user to have a high fidelity virtual experience of virtual terrain features or surface.
Yet another object of the present invention is to provide a system and method that provides realistic features of textures, small bumps, rocks, and other fine terrain features along with shape and form of virtual objects encountered in the real world with sufficiently low latency.
Yet another object of the present invention is to provide a system and method for enabling the user immersively experience the virtual world through their feet via a combination of fibrous bioelectronics and ferromagnetic sensors embedded within sole wearable.

In yet another object of the present invention, a system and method for selectively triggering the sensors is provided based on type of mechanoreceptors in skin to emulate experience of real world.
SUMMARY
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.
In first aspect, a system for generating a haptic feedback for a user in a virtual environment via a head mounted device and a wearable sole worn by the user is disclosed. The system comprises of the head mounted device configured to capture an omnidirectional view of the virtual environment. The system further comprises of wearable sole that further includes a sensor arrangement for generating haptic feedback from a first layer comprising of fibrous bioelectronics and a second layer comprising of a ferromagnetic fluid. Finally, the system includes a processing unit configured to receive and analyze the omnidirectional view of the virtual world to obtain characteristic features therefrom, and selectively trigger at least a portion of the first layer, at least a portion of the second layer or a combination thereof in response to the analyzed view such that the haptic feedback is generated in consonance with the characteristic features of the virtual world.

In second aspect, a method for generating a haptic feedback for a user in a virtual environment via a head mounted device and a wearable sole worn by the user is disclosed. The method comprising obtaining an omnidirectional view of the virtual environment via the head mounted device, relaying the omnidirectional view to a processing unit to obtain characteristic features therefrom; and selectively triggering a sensor arrangement of the wearable sole using: a first layer comprising of fibrous bioelectronics; and a second layer comprising of a ferromagnetic fluid; or a combination thereof such that the haptic feedback is generated in consonance with the characteristic features of the virtual environment.

BRIEF DESCRIPTION OF THE DRAWINGS
So that the manner in which the above recited features of the present invention can be understood in detail, a more particular to the description of the invention, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of this invention and are therefore not to be considered limiting of its scope, the invention may admit to other equally effective embodiments.
These and other features, benefits and advantages of the present invention will become apparent by reference to the following text figure, with like reference numbers referring to like structures across the views, wherein:
Fig. 1 is a block diagram illustrating an exemplary environment of virtual world and interaction thereof, in accordance with an embodiment of the present invention;
Fig. 2 illustrates a wearable sole configured with sensor arrangement, in accordance with an embodiment of the present invention;
Fig. 3 shows the sensor arrangement distributed through user sole, in accordance with an embodiment of the present invention; and
Fig. 4 shows the distribution of various kinds of cutaneous afferents within user sole, in accordance with an embodiment of the present invention.
DETAILED DESCRIPTION
While the present invention is described herein by way of example using embodiments and illustrative drawings, those skilled in the art will recognize that the invention is not limited to the embodiments of drawing or drawings described and are not intended to represent the scale of the various components. Further, some components that may form a part of the invention may not be illustrated in certain figures, for ease of illustration, and such omissions do not limit the embodiments outlined in any way. It should be understood that the drawings and detailed description thereto are not intended to limit the invention to the particular form disclosed, but on the contrary, the invention is to cover all modifications, equivalents, and alternatives falling within the scope of the present invention as defined by the appended claims. As used throughout this description, the word "may" be used in a permissive sense (i.e., meaning having the potential to), rather than the mandatory sense, (i.e., meaning must). Further, the words "a" or "an" mean "at least one” and the word “plurality” means “one or more” unless otherwise mentioned. Furthermore, the terminology and phraseology used herein is solely used for descriptive purposes and should not be construed as limiting in scope. Language such as "including," "comprising," "having," "containing," or "involving," and variations thereof, is intended to be broad and encompass the subject matter listed thereafter, equivalents, and additional subject matter not recited, and is not intended to exclude other additives, components, integers or steps. Likewise, the term "comprising" is considered synonymous with the terms "including" or "containing" for applicable legal purposes. Any discussion of documents, acts, materials, devices, articles, and the like are included in the specification solely for the purpose of providing a context for the present invention. It is not suggested or represented that any or all of these matters form part of the prior art base or were common general knowledge in the field relevant to the present invention.
In this disclosure, whenever a composition or an element or a group of elements is preceded with the transitional phrase “comprising”, it is understood that we also contemplate the same composition, element or group of elements with transitional phrases “consisting of”, “consisting”, “selected from the group of consisting of, “including”, or “is” preceding the recitation of the composition, element or group of elements and vice versa.
The present invention is described hereinafter by various embodiments with reference to the accompanying drawings, wherein reference numerals used in the accompanying drawing correspond to the like elements throughout the description. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiment set forth herein. Rather, the embodiment is provided so that this disclosure will be thorough and complete and will fully convey the scope of the invention to those skilled in the art. In the following detailed description, numeric values and ranges are provided for various aspects of the implementations described. These values and ranges are to be treated as examples only and are not intended to limit the scope of the claims. In addition, a number of materials are identified as suitable for various facets of the implementations. These materials are to be treated as exemplary and are not intended to limit the scope of the invention.
Fig. 1 illustrates an environment for detecting a user's presence and recording the user's activity in a virtual environment 100 according to certain embodiments of the disclosed subject matter. As shown in Fig. 1, the virtual environment 1000 comprises of a head mounted device 100, a wearable sole 200, a communication network 300, a server 400, a haptic sensor arrangement 250, a storage medium 500, a wireless network 600, and a processing unit 700. The term “virtual environment” has been interchangeably used with terms like “virtual world”, “virtual space”, “virtual reality”, “simulated world” etc. and shall be treated as same.
In accordance with one general embodiment of present disclosure, the present system and method enables the user to have a real life experience in virtual world 1000, precisely enabled by means of a wearable sole 200 constituting the sensor arrangement 250 and a head mounted device HMD 100 worn by the user. For evoking and recreating the tactile sensations, appropriate haptic feedback is generated from suitably selected and optimally positioned sensors in sensor arrangement 250. Haptic feedback, a reverse process of tactile sensing, is to stimulate user skin to evoke the tactile receptors and generate tactile sensations via feedback devices of the sensor arrangement 250.
In one specific embodiment, when the user equips himself with a head mounted device 100, he enters the virtual environment where he is presented with fantasies of fictional world, which to him should ideally appear as impressive as real world. As he visualizes the virtual environment via his head mounted device 100, the real thrill is packed in sensing and feeling the figures, figurines, objects and material of the simulated world 1000. In the present disclosure, the user enhanced vision capability is being enhanced with a compact, light-weight and flexible wearable sole 200 that helps him feel the ground surface or terrain in real time exactly in a manner as in real world.
In one example embodiment, the head mounted device 100 is configured with a multi-spectrum camera or image processing unit or imaging unit or one or more motion sensors to capture omnidirectional view of the virtual world 1000. Other user specific parameters and poses may be captured via different kinds of sensors embedded on head mounted device 100. Sensors include any electronic medium through which the device is operable to receive and record an input and use as feed for extracting relevant features therefrom (as discussed in detail later).
In a first embodiment, the head mounted device (HMD) 100 is worn in a manner similar to glasses, goggles, or a helmet, and is configured to display a virtual gaming, training or other content to the user. The HMD 100 provides a very immersive experience to the user by virtue of its provision of display mechanisms in close proximity to the user's eyes. Thus, the HMD 100 can provide display regions to each of the user's eyes which occupy large portions or even the entirety of the field of view of the user.
Next, as shown in Fig. 2, the system comprises of a wearable sole 200 that is further constituted of miniaturized sensor arrangement 250 and electronics 260 that essentially includes at least a) a first layer 260 comprising of flexible bioelectronic fibrous sensors 265 contained within a thin membrane 262; and b) a second layer 270 of ferromagnetic fluid 275 encapsulated within a thin film 272 that readily integrates and interacts with wearable sole 200 that can be worn, sewn or knitted with shoe or socks or can be directly worn at the bottom of user foot.
Following from above description, the wearable sole 200 comprises of a a) first layer 260 is formed of a thin flexible film or membrane 262 (that resembles feel of skin. eg. silicon) encapsulating therewithin a highly sensitive, responsive and spatially flexible fibrous bioelectronics 265; and b) a second layer 270 formed of a thin film/membrane 272 like that of first layer 260 containing therewithin a ferromagnetic fluid that runs atop the first layer 260, which is positioned in proximity to user foot, as shown in Fig. 2. The first layer membrane 262 and the second layer membrane 272 runs through the entire sole, to stimulate foot skin region, evoke the tactile receptor and generate tactile sensations in consonance with virtual scene 1000 displayed over the head mounted device 100. In another alternate embodiment, the first layer 260 and the second layer 270 may be glued together to form small patches that may be located all through the sole of user, especially in skin regions rich and dense with mechanoreceptors, as shown in Fig. 3.
The attempt is to match intensity of the haptic actuators in proportion to the pressure measured by a specific combination of sensors within the sensor arrangement 250. In one other embodiment, electrical stimulations provided to first layer membrane 262 of the sensor arrangement 250 helps in generating light, subtle and slightly vibratory tactile feedback for the user in order to replicate intended action of user or to correspond to various ground conditions in which the user is present within a virtual reality environment 1000. Likewise, the second layer membrane 272 configured with a ferromagnetic fluid 275 is more suited when the user intends to experience shape or varying forms, orientation of objects/surface within the virtual environment 1000.
Suppose that a user walks the sand of a beach in virtual reality 1000. Sand gives a soft feel compared to plain flat floor. Accordingly, the scene of the current virtual reality environment 1000 is a sandy beach environment, and should aptly generate a haptic signal that allows a similar feeling to be transmitted to the user's soles through the combination of haptic actuators provided in the wearable sole 200. Now, suppose that the user walks through the forest in virtual world and experiences "rustling" below his feet. Thus, the scene of the current virtual reality environment being recognized as a forest environment, should generate a haptic signal that causes the user to feel "rustling" when walking on the forest floor.
Likewise, the user navigating the snowfield in virtual reality finds the terrain covered with flaky and slippery snow. When a user walks in a snowy terrain, it gives a feeling of "puffiness". Accordingly, the scene of the current virtual reality environment is a snowy field environment, and generates a haptic signal to give a feeling of "puffy" distance. Thus, the present disclosure generates a haptic signal representing various treads (hardness, softness, roughness, etc.) through various scenarios.
Accordingly, as the user experiences small and fine surface features that are generally perceivable as a change in pressure or weight, velocity, pressure distribution, centre of gravity, acceleration, direction, speed of touch, proximity, temperature, etc. with changing surfaces, he is expected to immerse in virtual world only when replicated with similar feeling in virtual environment. Such a haptic terrain experience enablement can be used for multiple applications for virtual reality including video gaming, entertainment, military training simulations, skill training simulations, fire fighter training simulations, space simulations, flight simulations, science education simulations, and various medical, architectural and design applications.
To literally feel the terrain and bask in the offerings of virtual reality, it is important that the system derives meaningful comprehension of feedback from cutaneous afferents within human skin that are associated with balance, posture, gait control, locomotion etc. Cutaneous afferents are a combination of sensory nerve and mechanoreceptor endings that are vital for sensing and feeling uneven terrains and other ground obstacles like stairs and also standing balance as they transmit strong tactile and proprioceptive feedback to the central nervous system (CNS). These cutaneous afferents are classified based on receptive field characteristics that corresponds to the area of skin that elicits a response to a given stimulation.
In particular, different receptive regions on the foot sole are responsible for uniquely perceiving the ground/floor underneath due to differing classes of cutaneous mechanoreceptors (the receptors that transduce mechanical stimuli in nervous signals) innervating the most responsive glabrous skin of foot sole. For example, stimulation needed at the heel is twice needed in any other perceptible area of sole to obtain high tactile feedback of same resolution. Accordingly, when pressure, vibration or electrical stimulation is applied to foot sole, mechanical deformations are transmitted through the tissue to the cutaneous afferent mechanoreceptor endings that are specialized in perceiving different mechanical stimulus.
Broadly, these mechanoreceptors are classified as Fast adapting (FA) type 1 and type 2 (FA 1 or FA2) or slow adapting (SA) type 1 and type 2 (SA 1 or SA2). While FA fires throughout the dynamic phase of an indentation, it cease to fire when indentation is sustained; On the other hand, the SA continues to fire even during sustained indentations. Thus, in situations of accidental slip, motion pace change, or coarse vibrations, FA responds more affirmatively to provide a tactile feedback. All of these critical physiological and biochemical factors and technical considerations have been taken into account while designing the wearable sole 200 of present disclosure for optimal selection and placement of membrane layers forming the sensor arrangement 250 to achieve enhanced tactile rendering and increased awareness.
Now, to elaborate further on these cutaneous afferents, it is to be noted that most of FA Type 1 afferents end in Meissner corpuscles while SA1 terminate in Merkel cells in basal layer. This makes them having small receptive fields with distinct borders and multiple hotspots. On the other hand, Type 2 afferents (for both FA and SA) do not branch within the skin and innervate a single, relatively large mechanoreceptor in the dermis and subcutaneous tissues. FA2 afferents terminate in Pacinian corpuscles, and SA2 afferents terminate in Ruffini endings. This makes them having large receptive fields with undistinguished borders and concentrated regions of maximal sensitivity instead of multiple hotspots. Further, SA2 afferents are more susceptible to stretches while FA2 is most responsive to high frequency vibrations.
Due to a high concentration of receptors such as Meissner’s corpuscles, Merkel disk receptors, Ruffini endings, and Pacinian corpuscles, the glaborous skin of the plantar foot, which essentially is uniformly distributed across the sole responds more sensitively to tactile input. Also, the distribution of these afferents is found to increase from heels to toe, with smaller receptive fields enabling the potential for greater resolution of tactile feedback.
As shown in Table 1 below, the responsive state of cutaneous afferents have been summarized for reference purposes.
Category Meissner Corpuscles Pacinian Corpuscle Merkel receptors Ruffini Endings
Type FA1 FA2 SA1 SA2
Adaptation rate Fast Fast Slow Slow
Location Shallow Deep Shallow Deep
Stimuli Frequency (Hz) 10-200 70-100 0.4-100 0.4-100
Density (units/cm2) 140 30 70 10
Functional range Light contact, texture High frequency, vibrations Static forces with high resolution Tension deep in skin
Receptive Field Small and sharp Very large and diffuse Small and sharp Large and diffuse
Characteristic Capture slippage information, temporal changes in skin deformation; spatial deformation Detect vibrations of encountered object; temporal changes in skin deformation Detect edges, corners and curvatures; sustained pressure; maximal sensitive to low frequency Detect skin stretch, motion perception; sustained downward pressure, lateral skin stretch, low dynamic sensitivity
Besides, considering physiological considerations, it needs to be understood that a wearable which is more intimately connected with skin, is of micro scale dimension and is mechanically imperceptible to human skin to ensure that sensory receptors and topographical features in skin are not concealed like that in conventional sensors (like that of vibratory, pressure, electric, pneumatic and the like).
In one example embodiment, the high definition haptic feedback of terrain features may be received as an electrotactile feedback that delivers the pressure and vibrations to precisely sense and feel physical texture of underlying surface. Conventionally, vibrotactile feedback is provided primarily using Linear Resonant Actuators (LRAs), Eccentric Rotating Masses (ERMs) and piezoelectric actuators are used. However, these mechanical actuators are usually focused on creating localized haptic cues and have limited capability in providing rich feel of varying texture in virtual world. To achieve adequate experience of ground deformation and wide gamut of textual surfaces under feet, the present disclosure has designed a specialized sensor arrangement 250 combining capabilities and synergies of highly flexible, imperceptible and conducting fibrous bioelectronics 265 with bumpy, corrugated and spiky pattern formation capability of ferromagnetic fluid 275.
In one preferred embodiment, stretchable electronics enabled sensors are selected to complement and not perturb inherent sensations and physiological changes in the user besides preserving user’s imperceptibility to such wearable devices. The combination of bioelectronic sensors 265 and ferromagnetic fluid 275 is more compatible, sensitive and flexible when a larger area (such as human foot) needs to be covered with implantable or wearable electronics to transmit/receive signals from epidermis/dermis layers.
In one significant embodiment, an array of micro/nano scale, flexible and stretchable polymer based fibre network is proposed to create on-skin electric stimulation and signal measurement interface without curbing body movement and inherent sensitivity. In accordance with one working embodiment, the fibre number density (number of fibres over width of fibre array), orientation and modalities are tunable as per the receptive field characteristics of cutaneous afferents identified in aforementioned paragraphs.
Accordingly, any known technique for fabricating sub-millimetre level resolution and small diameter fibrous bioelectronics may be opted as they are known to have higher surface-to-area volume ratio, which is important to obtain enhanced sensitivity in wearable soles. Techniques such as microfabrication or additive manufacturing employing conjugate polymers (preferably poly (3,4-ethylenedioxythiophene):polystyrene sulfonate) may be opted to achieve better electronic conductivity. Alternately any other micro/nano scale fabrication technique that can provide for excellent alignment and controlled morphologies of composite fibres will be preferred. It is pertinent to note that these bioelectronics sensors 265 may be encapsulated within a thin film 262 to apply the sensations in accordance with various interactive contexts in order to feel virtual surfaces.
The attempt is to distribute the tactile experience across the entire user foot at higher number of regions receptive to cutaneous afferents compared to the localized cues provided in earlier endeavours. Also, besides experiencing varying texture, certain static information and shapes are also required to be felt by the user navigating the virtual world. For example, the user may be stepping on a large stone or a big rock having a large surface area within a rocky terrain. In such situations, actuating the large array of vibrators simultaneously may posit the fundamental problem of inter-vibrator cross talk, which will make the entire experience more vibratory and insufficient for the user to experience the shape and feel of huge rock/stone.
The density of bioelectronics fibres 265 over the user foot sole may be varied based on spatial resolution needed for sensor placement on user feet. Since the skin is heterogeneous it is not ideal to localize the tactile experience. In accordance with Transcutaneous Electric Nerve Stimulation (TENS) theory, the current injected into the human tissue causes the depolarization of the semi-permeable membrane of the skin’s receptors and therefore the generation of action potentials generates a real haptic feedback. The contact impedance levels achieved by bioelectronics sensors 265 helps in creating such electrical conductivity in wearable sole as the user skin acts as electrolyte thereby allowing the charge exchange with low intensity electrical current, and elicit tactile sensations and delivery of electrical signals to happen to stimulate local afferent nerves.
Accordingly, the wearable sole 250 is provided with a first layer of bioelectronics fibre sensors 265 encapsulated within a thin, flexible and minimal form factor skin-like casing/film/membrane 262 that runs densely through the Meissner’s corpuscles, Merkel disk receptors, Ruffini endings, and Pacinian corpuscles in the glaborous skin of the plantar foot with increasing density from heels towards the toes, as shown in Fig. 4. The fibrous bioelectronic sensor layer 260 is capable of providing sensations of varying magnitude including a wide frequency range and even vibrations of amplitude smaller than 10 nanometers with utmost precision.
Next, atop the first fibrous bioelectronic layer 260 runs the second layer 270 formed of thin, flexible silicon like casing 272 containing therewithin ferromagnetic fluid 275 that enables generating a reaction force when a virtual object characteristic of it’s shape or form is encountered over a virtual terrain. Ferromagnetic fluids are colloidal suspensions of paramagnetic metal nanoparticles in a carrier liquid that responds by way of movement to applied electric/magnetic field. The particles are held in suspension by the effects of Brownian motion, and do not settle. The ferromagnetic particles are coated with a surfactant to prevent their agglomeration. In another example, the ferromagnetic fluid layer can contain magnetorheological fluid.
In one preferable embodiment, the second layer 270 enclosing the ferromagnetic fluid (FF) offers millimeter of displacement when stimulated which is enough to create dynamic, small scale interaction forces and tactile feedback to feel immersed in virtual environment 1000. Particularly, with ferromagnetic fluid being meticulously actuated, one can hold control over experiencing of varied shapes and physical properties of virtual objects the user may be stepping over. By varying viscosity of ferromagentic fluid via electric stimulation will enable real life sensations of various ground surface deformations. Since ferromagnetic fluid either increases or decreases its viscosity with respect to the magnetic field derived from input current, the stiffness of fluid is changed accordingly, delivering a variety of tactile sensations and high fidelity walking experience as the fluid moves and get pressed by the foot.
Furthermore, in another example, the ferromagnetic fluid layer can be divided up into a number of cells or “pockets”, each holding a quantity of ferromagnetic fluid. This limits the gross movement of the ferromagnetic fluid within the ferromagnetic fluid layer 270, and can enable the ferromagnetic fluid 275 to selectively sink/raise the cell/pocket based on directed electric current. The nuanced feature of enabling varying shape and texture feel of terrains of varying complexity by ferromagnetic fluid 275 complements the subtle, very fine and light stimulations created by use of fibrous bioelectronic 265 constituted first layer 260.
Apparently, the electric current selectively directed towards sections filled with ferromagnetic fluid 275 allows control over nerve afferents that needs to be charged for emulating the real world feel. The high strain deformations brought by spiking ferromagnetic material 275 within the fluid enables realization of shape changing terrains, angles of curvature and objects of various form, orientation, volume, texture, viscosity, spatiality and the like encountered in the virtual world. The ferromagnetic fluid 275 is electrically manipulated and controlled by inducing current that causes the encapsulating flexible membrane 272 to deform with changing shape and direction of ferromagnetic fluid 275.
Now, based on the kind of perception the user needs to experience, the FA1, FA2, SA1 or SA2 region centric bioelectronic fibre sensors 265 and ferromagnetic fluid 275 is stimulated to generate resolution comparable with receptors acuity. For example, when dynamic events are happening within the scene the bioelectronics fibre positioned on regions of FA afferents are made to respond more rapidly. On the contrary, for a static scene information, sensors positioned at SA afferents affiliated regions are more strongly triggered for higher spatial resolution and enhanced proprioceptive haptic feedback.
In order to realize the endeavours of present disclosure, a method for generating haptic feedback using a head mounted device 100 along with a wearable sole 200 is discussed. In accordance with one preferred embodiment, the first layer 260 and second layer 270 of sensor arrangement 250 are electrically stimulated in accordance with the scene dynamically changing scene in virtual reality environment 1000. In one example embodiment, the sensor arrangement 250 may be distributed around a toe, a heel and at opposing sides of the bottom of the insole for direct communication with user foot.
In one method embodiment, the virtual scene is spatially captured by imaging unit of HMD 100 to render the obtained omnidirectional view on HMD display. The omnidirectional image/video data is relayed to the processing unit 700 where it is analyzed for target dispensing of haptic effect. The present method utilizes computer vision technology to extract the geometric features in image/video such as shaped, size, edges, orientation, motion or other material characteristics of the virtual objects/surface in the virtual scene that has been captured from all different directions.
In another alternate embodiment, the high level image understanding along with the contextual understanding of scene is obtained using machine learning algorithms such as convolutional neural network (CNN) that suitably performs tasks of classification, regression, clustering and dimension reduction. The attempt is to generate perceptually realistic simulations of ground areas, by modelling their physically interactive properties.
The machine learning model is trained with a series of data acquisition on terrains of different texture, material, resistance and other surface properties. The data is recorded, normalized and then used to train the model to trigger first layer 260 or second layer 270 in response to sensations required to be generated from mechanoreceptors. This selective triggering of first layer 260 and second layer 270 or a combination thereof is primarily based on primary parameters such as adaptation rate, frequency, and receptive field of mechanoreceptors that generally respond to similar stimuli in real world.
In one preferred embodiment, machine learning model is augmented with differentials of labels and weights with respect to the identified parameters to generalize the learned features for future application. In accordance with one exemplary embodiment, weights are assigned to parameters adaptation rate, frequency and receptive field, where the highest weight may be assigned to adaptation rate, then to stimuli frequency followed by receptive field. Receptive field is allotted a least weight for reasons of skin being a bit more homogeneous in user sole compared to hands. These weights will set the standards for neuron’s signal strength which will provide that extra information the model will need to efficiently propagate forward and accurately analyze the unidentified features.
Thus, based on differential weights of adaptation rate, frequency, texture and receptive field, in an event SA1 afferent needs to be stimulated, first layer comprising of fibrous bioelectronics 265 is actuated. Likewise for afferents FA1, FA2 and SA2, second layer, second layer and first layer is respectively actuated. For example, if the virtual scene comprises of a user training in a military environment and the underlying terrain is a rocky terrain, the virtual military scenario is analyzed for feature extraction such as hill, ridge, valley, saddle, depression. The forest terrain is analyzed by the machine learning algorithm for characteristic features such as hardness, roughness, shape, form, volume etc. indicative of rugged forest terrain. Once the scene is contextually comprehended, the user is instructed over the head mounted device 110 to make specific motion or pose or combination of poses. This expands the sense of realism and immersion of user in virtual space.
For example, if the user is trying to make his way out of muddy waters, he may be instructed by way of an image or small video displayed over his head mounted device 100, to make certain poses or acquire a certain position using combination of his both hands and particularly feet. Likewise, if the user is walking on grainy desert sand, he may be asked to bend his knees and push the feet backwards. In one example embodiment, the system comprises of a database of varying topography, associated terrains and surrounding landscape to help gain contextual understanding of scene. The varied terrains are mapped to corresponding poses that are registered and stored in system for presentation to user whenever he is experiencing that particular surface/terrain using CNN model.
In event no body posture is suggested, the image features fetched are mapped with pre-recorded images that are used to train the model and used as input for sensor arrangement 250 actuation. Thus, these body movements or pre-trained images are learned and used as input feed to extract characteristic features to elicit sensor arrangement 250 based on primary parameters-adaptation rate, frequency and receptive field. For instance, the dynamic variations in forest terrain and feel of experiencing varied twigs, leaves, rocks, pebbles and the like shall sequentially actuate Layer 2, layer 1, layer 1 and layer 2 to stimulate FA2, SA1, SA2 and FA1 respectively. The combinations for stimulation can be learned as the model is trained on various terrains to sense activity of the user, pressure, force, vibrations etc. experienced on a particular zone or region of the sole or movement of the sole and the like.
For instance, the convolutional neural network is trained to extract object/surface features by applying a set of learnable filters/kernels to input image. The object/surface feature map obtained as an output are recorded for training the model to forthcoming dynamic dataset consisting of virtual scenes. Meanwhile, the Rectified Linear Unit (ReLU) is used as the activation function of the convolutional layer and the regularization enhances the general ability of CNN to avoid overfitting. Further, the method classifies and identifies the sensation associated with surface features (obtained as output) based on Table 1 (listed above). The identified sensation will be used to trigger skin mechanoreceptors precisely in designated foot regions to reconstruct the sense of touch and help user differentiate between the harness and texture of different surfaces.
In one exemplary embodiment, the stimulation pattern is preferably determined based on the value of the one or more parameters, namely adaptation rate, frequency and receptive field of the mechanoreceptors, as stated above. The characteristic selection and configuration of sensor arrangement 250 is thus activated separately and selectively to augment virtual reality experience for instance, the front or back activation, to strike a balance, or to realize other effects such as modulating different, back-front signals thereby making the user experience force, load, tension, compression forces, strain, shapes, forms, orientations and other. As the section/segment of first layer 260 or second layer 270 gets activated based on selective or sequential electrical signal received for target layer, the other sections/segments remain deactivated (e.g. to conserve power and/or reduce the amount of data processing required).
In one working embodiment, a set of electrically controlled switches may help in selection of membrane segments to be selectively actuated based on classified surface/terrain features and associated mechanoreceptors that are receptive to sensation corresponding to the terrain. In one other embodiment, the sole may be optionally provided by way of providing thermal sensors that may be perceived as rising temperature on certain foot segments that may be used to enhance immersion in VR. Particularly, if the user wishes to train in a military environment and is profusely bleeding, then the sensors can be triggered for heat increase that may cause discomfort to users. This scenario can also be replicated for desert areas to make the user experience ground heat.
In another example embodiment, the visual footprints are also created along with the appropriate auditory feedback (i.e. sound of snow crunching) to maximize the realistic experiences with multimodal cues in virtual reality. In one alternate embodiment, the system may be pre-loaded with discrete sounds that may correspond to different scenarios portrayed in virtual world. For example, a user travelling a road covered with dried leaves in the virtual world is given a sense of realism as the sound of dried leaves or gravel shifting are synced along with the haptic feedback given to user sole with pressing of virtual dried leaves.
In one embodiment, the wearable sole 200 can be a shoe or a sock, or can be other types of footwear, such as a sandal, a flip-flop, a boot, an insole, sock shoe, removable shoe insert or even a foot wrap, such as might be made out of a stretchy material such as neoprene. In one exemplary embodiment, such soles may be secured to shoe by any known shoe-to-sole securement methods such as adhesion, molding, stitching, using zippers, snaps, buckles, buttons, Velcro, other attachments, etc., to secure the wearable article to the user. The haptic signals received from sole may be transmitted to a processing unit 700 to process the signals and generate a feedback to produce an electrical signal representing the feedback generated by each of the first or second layer 260, 270 of the sensor arrangement 250. The configuration of sensors is utilized and operated in specific, predetermined ways, which can provide dynamic, instantaneous and distinct high performance haptic sensing.
In accordance with an embodiment, the machine-readable instructions may be loaded into the memory unit from a non-transitory machine-readable medium, such as, but not limited to, CD-ROMs, DVD-ROMs and Flash Drives. Alternately, the machine-readable instructions may be loaded in a form of a computer software program into the memory unit. The memory unit in that manner may be selected from a group comprising EPROM, EEPROM and Flash memory. Further, the micro controller is operably connected with the memory unit. In various embodiments, the micro controller is one of, but not limited to, a general-purpose processor, an application specific integrated circuit (ASIC) and a field-programmable gate array (FPGA). In general, the word “module/unit,” as used herein, refers to logic embodied in hardware or firmware, or to a collection of software instructions, written in a programming language, such as, for example, Java, C, or assembly. One or more software instructions in the modules may be embedded in firmware, such as an EPROM. It will be appreciated that modules may comprised connected logic units, such as gates and flip-flops, and may comprise programmable units, such as programmable gate arrays or processors.
Further, while one or more operations have been described as being performed by or otherwise related to certain modules, devices or entities, the operations may be performed by or otherwise related to any module, device or entity. As such, any function or operation that has been described as being performed by a module could alternatively be performed by a different server, by the cloud computing platform, or a combination thereof. It should be understood that the techniques of the present disclosure might be implemented using a variety of technologies. For example, the methods described herein may be implemented by a series of computer executable instructions residing on a suitable computer readable medium. Suitable computer readable media may include volatile (e.g., RAM) and/or non-volatile (e.g., ROM, disk) memory, carrier waves and transmission media. Exemplary carrier waves may take the form of electrical, electromagnetic or optical signals conveying digital data steams along a local network or a publicly accessible network such as the Internet. It should also be understood that, unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as "controlling" or "obtaining" or "computing" or "storing" or "receiving" or "determining" or the like, refer to the action and processes of a computer system, or similar electronic computing device, that processes and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.
Various modifications to these embodiments are apparent to those skilled in the art from the description and the accompanying drawings. The principles associated with the various embodiments described herein may be applied to other embodiments. Therefore, the description is not intended to be limited to the embodiments shown along with the accompanying drawings but is to be providing broadest scope of consistent with the principles and the novel and inventive features disclosed or suggested herein. Accordingly, the invention is anticipated to hold on to all other such alternatives, modifications, and variations that fall within the scope of the present invention.
,CLAIMS:We Claim:

1) A system for generating a haptic feedback for a user in a virtual environment (1000) via a head mounted device (100) and a wearable sole (200) worn by the user, wherein:
the head mounted device (100) is configured to capture an omnidirectional view of the virtual environment (1000);
the wearable sole (200) comprises of a sensor arrangement (250) for generating haptic feedback from:
a first layer (260) comprising of fibrous bioelectronics (265); and
a second layer (270) comprising of a ferromagnetic fluid (270); and
a processing unit (700) configured to:
receive and analyze the omnidirectional view of the virtual world (1000) to obtain characteristic features therefrom; and
selectively trigger at least a portion of the first layer (260), at least a portion of the second layer (270) or a combination thereof in response to the analyzed view such that the haptic feedback is generated in consonance with the characteristic features of the virtual environment (1000).

2) The system for generating a haptic feedback, as claimed in claim 1, wherein the first layer (260) and the second layer (270) are encapsulated within a skin-like thin film or membranes (262, 272).

3) The system for generating a haptic feedback, as claimed in claim 1, wherein the first layer (260) is arranged in proximity to sole of user and the second layer (270) is arranged atop the first layer (260).

4) The system for generating a haptic feedback, as claimed in claim 3, wherein the first layer (260) and the second layer (270) runs through the sole of the user in entirety.

5) The system for generating a haptic feedback, as claimed in claim 3, wherein the first layer (260) and the second layer (270) is positioned as plurality of patches in regions of the user sole dense in mechanoreceptors.

6) The system for generating a haptic feedback, as claimed in claim 1, wherein the processing unit (700) is configured to generate haptic feedback by selectively triggering the first layer (260) or the second layer (270) or the combination thereof of the sensor arrangement (250) based on type of cutaneous afferents present in user sole.

7) The system for generating a haptic feedback, as claimed in claim 6, wherein the type of cutaneous afferents include Fast adapting (FA) 1, Fast adapting (FA) 2, Slow adapting (SA) 1 and Slow adapting (SA) 2.

8) The system for generating a haptic feedback, as claimed in claim 1, wherein the fibrous bioelectronics is formed of poly (3,4-ethylenedioxythiophene):polystyrene sulfonate.

9) The system for generating a haptic feedback, as claimed in claim 1, wherein the second layer (270) is segmented to form plurality of pockets containing the ferromagnetic fluid (275) that sink or raise to form shapes based on haptic feedback required in consonance with the characteristic features of the virtual environment (1000).

10) The system for generating a haptic feedback, as claimed in claim 6, wherein the processing unit (700) analyze the omnidirectional view of the virtual world (1000) using convolutional neural network (CNN) model, wherein differential weights are assigned to parameters: adaptation rate, frequency and receptive field of the cutaneous afferents for training the CNN model.

11) The system for generating a haptic feedback, as claimed in claim 6, wherein highest weight is assigned to the adaptation rate, followed by the frequency and the receptive field.

12) The system for generating a haptic feedback, as claimed in claim 6, wherein the user is optionally instructed via the head mounted device (100) to make bodily postures and move his feet in accordance with the characteristic features of the virtual environment (1000).

13) The system for generating a haptic feedback, as claimed in claim 7, wherein:
in an event SA1 is required for stimulation, the first layer 260 is actuated,
in an event FA1 is required for stimulation, the second layer 270 is actuated,
in an event FA2 is required for stimulation, the second layer 270 is actuated, and
in an event SA2 is required for stimulation, the first layer 260 is actuated.

14) A method for generating a haptic feedback for a user in a virtual environment (1000) via a head mounted device (100) and a wearable sole (200) worn by the user, comprising:
obtaining an omnidirectional view of the virtual environment (1000) via the head mounted device (100);
relaying the omnidirectional view to a processing unit (700) to obtain characteristic features therefrom; and
selectively triggering a sensor arrangement (250) of the wearable sole (200) using:
a first layer (260) comprising of fibrous bioelectronics (265); and
a second layer (270) comprising of a ferromagnetic fluid (270);
or a combination thereof such that the haptic feedback is generated in consonance with the characteristic features of the virtual environment (1000).

15) The method for generating a haptic feedback, as claimed in claim 14, wherein the haptic feedback is generated by selectively triggering the first layer (260) or the second layer (270) or the combination thereof of the sensor arrangement (250) based on type of cutaneous afferents present in user sole.

16) The method for generating a haptic feedback, as claimed in claim 15, wherein the type of cutaneous afferents include Fast adapting (FA) 1, Fast adapting (FA) 2, Slow adapting (SA) 1 and Slow adapting (SA) 2.

17) The method for generating a haptic feedback, as claimed in claim 15, wherein the omnidirectional view of the virtual world (1000) is analyzed using convolutional neural network (CNN) model, wherein differential weights are assigned to parameters: adaptation rate, frequency and receptive field of the cutaneous afferents for training the CNN model.

18) The method for generating a haptic feedback, as claimed in claim 17, wherein highest weight is assigned to the adaptation rate, followed by the frequency and the receptive field.

19) The method for generating a haptic feedback, as claimed in claim 16, wherein in an event SA1 is required for stimulation, the first layer 260 is actuated,
in an event FA1 is required for stimulation, the second layer 270 is actuated,
in an event FA2 is required for stimulation, the second layer 270 is actuated, and
in an event SA2 is required for stimulation, the first layer 260 is actuated.

20) The method for generating a haptic feedback, as claimed in claim 19, wherein the user is optionally instructed via the head mounted device (100) to make bodily postures and move his feet in accordance with the characteristic features of the virtual environment (1000).