Touching the Future: Exploring Haptics and Multisensory Experiences in Virtual Reality

In real life environments, the brain associates and transfers information, crossmodally, from one sense to another. It integrates and processes information from multiple senses. And emotional perceptions too. (See our blog for the crossmodal correspondences between the senses, crossmodal brain plasticity, multisensory processing, and emmotional perceptions). But what happens in Virtual Realities? Virtual Realities are created to trick us into believing something is real when it is not. They can be all visual, auditory, or tactile – and even multisensory


I have invited Associate Professor Mounia Ziat, Bentley University to write about the sense of touch in multisensory virtual realities. That is, on haptic technologies that simulate the tactile and kinaesthetic sensations we feel when interacting with the real world. Mounia Ziat has published extensively on perception and human interaction with natural and artificial environments. And, she has been awarded numerous prizes and grants for her work (e.g., from the EuroHaptics Society, National Science Foundation,  America’s Seed Fund, and Google Research). In this blog post, Mounia explores the transformative potential of haptics in virtual reality, with applications that enrich accessibility, emotional well-being, rehabilitation, and sensory understanding.


The sense of touch, including its interplay with other sensory modalities, is essential to how we experience and navigate the world. In virtual reality (VR), haptic technologies are unlocking new dimensions of sensory engagement, from emotional resonance to crossmodal integration with temperature, sound, and vision.


Multisensory Integration: The Role of Touch and Temperature

Touch and temperature are deeply intertwined in our perception of the world. Studies on the hue-heat hypothesis, for instance, show how color can influence temperature perception: blue hues can make hot objects feel cooler, while red hues can intensify the sensation of cold​. These crossmodal interactions highlight the importance of synchronizing sensory inputs for a coherent and meaningful experience. In VR, combining haptics with temperature modulation can create more immersive and realistic interactions. For example, a VR system could use haptic feedback and visual cues to replicate the warmth of a sunny beach or the chill of a snowstorm, enhancing the user’s sense of presence.


Haptics in Emotional and Interpersonal Experiences

Touch isn’t just functional—it’s deeply emotional. Haptic sensations in VR can evoke feelings of comfort, fear, or excitement, depending on how they are designed. Research on the cutaneous rabbit illusion, where participants feel “hops” on their arm, shows how tactile feedback can influence emotions like arousal and valence.


Wearable haptic systems, such as gloves, smart clothing, and vests, are being developed to provide tactile feedback that carries emotional meaning. These devices can simulate caresses, tickling sensations, or even the comforting pressure of a hug, paving the way for emotionally expressive communication in virtual and augmented realities.


However, existing haptic stimuli often lack the ability to fully capture the emotional nuances of real-world touch. To unlock the full potential of haptics, researchers should design stimuli that evoke emotions, identify socially acceptable touchpoints, and improve the integration of tactile feedback into eXtended Reality (XR) systems. These advancements could transform how people connect and communicate, especially in mediated or virtual environments.


Applications Across Fields

Haptics is already making waves across diverse fields:

  • Healthcare and Rehabilitation: Haptic feedback in VR has been instrumental in neurorehabilitation for individuals with upper limb paralysis. Devices like robotic exoskeletons and haptic gloves provide tactile stimulation during therapy, promoting motor and sensory recovery while engaging patients in interactive exercises. These technologies not only improve physical outcomes but also enhance patient motivation by integrating gamified elements into therapy. Mid-air Haptics has similarly been used to reduce anxiety during medical procedures, demonstrating the versatility of haptic technology in healthcare.
  • Art and Immersion: In artistic VR installations, passive haptics—like vibrations underfoot when “walking” on virtual paintings—can be paired with temperature shifts to simulate the feel of stepping on different materials.
  • Accessibility: For individuals with sensory challenges, haptics can provide more nuanced and informative feedback, bridging gaps in sensory perception.

These applications demonstrate how haptics can enrich both functional and creative experiences.


Future Challenges and Opportunities

As promising as haptic technology is, challenges remain. Designing devices that seamlessly integrate touch feedback is technically complex. Moreover, creating socially acceptable and emotionally expressive tactile stimuli requires careful consideration of cultural and personal differences. Future research will likely explore these intersections, advancing haptic systems that are not only precise and realistic but also adaptable and inclusive.


Conclusion

Haptics is at the frontier of sensory innovation, transforming virtual reality into a multisensory experience that engages touch, vision, audition, and emotion. By harnessing these technologies, we can create inclusive, immersive environments that redefine how we interact with both the virtual and physical worlds.


As we move forward, the integration of haptics in neurorehabilitation, art, and accessibility offers exciting possibilities—not just for technology, but for human connection and understanding.


See our blog for Activities; especially 55-57.


Some suggestions for further listening and watching

Emergence Gallery: Virtual Walking

Haptic gloves help blind people ‘see’ art

Is That my Real Hand?

Smart Clothing

The Predictive Perception of Dynamic Vibrotactile Stimuli Applied to the Fingertip

The VR Dilemma: How AR and VR redefine our reality

Understanding Affective Touch for Better VR Experience

Virtual reality: how technology can help amputees

Virtual Reality Used To Treat Mental Health Problems


And reading

Haptics for Human-Computer Interaction: From the Skin to the Brain

Interpersonal Haptic Communication: Review and Directions for the Future

The Effect of Multimodal Virtual Reality Experience on the Emotional Responses Related to Injections

Walking on Paintings: Assessment of passive haptic feedback to enhance the immersive experience

What the Mind Can Comprehend from a Single Touch

Visual memories and sensory experiences

It seems people who lose vision use information that transfers between the senses to retrieve visual memories. And that their visual memories decay when they have reached a certain level of experience in the other senses. (See our blog for the scientific approach, the crossmodal correspondences between the senses, and Decay and maintenance of sensory memories). But what happens when people who are born blind gain vision and then lose it again?


I have invited P.B. to share his experiences of visual memories and sensory information when going from blind to partially sighted and back to being blind again. P.B. was born blind with some light perception in one eye. After surgery and other medical treatments, he had gained about 7% vision in the other eye at three and a half years old. P.B. does not remember this shift from being blind to being partially sighted. Then, in his early twenties, P.B. had an accident that rendered him totally blind in both eyes. P.B. thinks of himself as a sighted person who happens not to see. He visualises when making plans. And his dreams are always visual. P.B. approved this text before we posted it on our blog.


P.B. has several visual memories from when he was little. From the house he grew up in, for example, he remembers the light from a chandelier hanging over the coffee table in the living room, the patterned wallpaper in his room, and the countries on the world map hanging on the wall. And he remembers the colours on the outside walls, the doors, and the window frames.


Today, when he has no new visual input, P.B. creates visual images based on his memories and knowledge. For example, that of a White Swiss Shepherd Dog working as a guide dog from his memories of both the white colour and the German Shepherd Dog. As well as his knowledge of guide dogs.


His visual memories are typically triggered by somebody describing how something looks, like a bright red sunset – not by information from the other senses.


P.B.’s visual memories “flash up almost like the flavour when eating or drinking”.


He has to decide not to focus on visual memories and images when talking to people: the memories are now 20 years old and people have changed. And his created visual images of what new people look like may be very wrong.


P.B. has to actively suppress his urge to retrieve visual memories or create visual images of people based on their voices.


He describes not having visual memories and images of spatial relations and distances. P.B. rather remembers them through his body. He walks around in the city centre with no vision, a podcast or some music in his ears, and shoes with a thick curved sole (which makes it difficult to feel the surfaces on pavements, streets, steps, floors, etc.). He does not count steps or any of that and pauses the podcast or music in his ears only when he feels he might get lost.


For P.B., sensory experiences include a variety of simultaneous sensations as well as visual memories and created images – they are multisensory. The forest, for example, is the sound of wind and trees, the texture of surfaces under his feet, scents, and the memories of colours. He zooms in on whatever attracts his attention: indeed, not on one piece of sensory information after the other in a certain order. For example, zooming in on a sudden shift from soft to hard texture under his feet.


An interesting sensory experience, according to P.B., is that of new flavours, the texture of the food, and the sound of crunch when chewing, He describes a cloudberry cream dessert as “not very sweet, but also not sour, light orange colour, and creamy texture”. But a redcurrant jelly only as “something wobbling on the plate”. He does not describe the flavour, scent, and/or feel in his mouth – merely how the jelly looks on the plate.


Was he not able to suppress his visual memories of jelly – did they take over?


When P.B. is sensory tired, he relaxes his senses by listening to “concentration music, like opera or prog rock where I can immerse myself and get absorbed by a different universe”.


It seems P.B. can maintain his visual memories and ability to create visual images through people’s descriptions of what something looks like. But, at the same time, these memories and images are distracting him from focusing on sensory information that is more relevant to him today with no vision, for example, people’s voices and the feel of wobbling jelly in his mouth, So is this, in fact, a negative spiral – that maintaining visual memories prevents experiences in the other senses, which in turn helps nurture the visual memories?


See our blog for Activities; especially 49-51.