Touching the Future: Exploring Haptics and Multisensory Experiences in Virtual Reality

In real life environments, the brain associates and transfers information, crossmodally, from one sense to another. It integrates and processes information from multiple senses. And emotional perceptions too. (See our blog for the crossmodal correspondences between the senses, crossmodal brain plasticity, multisensory processing, and emmotional perceptions). But what happens in Virtual Realities? Virtual Realities are created to trick us into believing something is real when it is not. They can be all visual, auditory, or tactile – and even multisensory


I have invited Associate Professor Mounia Ziat, Bentley University to write about the sense of touch in multisensory virtual realities. That is, on haptic technologies that simulate the tactile and kinaesthetic sensations we feel when interacting with the real world. Mounia Ziat has published extensively on perception and human interaction with natural and artificial environments. And, she has been awarded numerous prizes and grants for her work (e.g., from the EuroHaptics Society, National Science Foundation,  America’s Seed Fund, and Google Research). In this blog post, Mounia explores the transformative potential of haptics in virtual reality, with applications that enrich accessibility, emotional well-being, rehabilitation, and sensory understanding.


The sense of touch, including its interplay with other sensory modalities, is essential to how we experience and navigate the world. In virtual reality (VR), haptic technologies are unlocking new dimensions of sensory engagement, from emotional resonance to crossmodal integration with temperature, sound, and vision.


Multisensory Integration: The Role of Touch and Temperature

Touch and temperature are deeply intertwined in our perception of the world. Studies on the hue-heat hypothesis, for instance, show how color can influence temperature perception: blue hues can make hot objects feel cooler, while red hues can intensify the sensation of cold​. These crossmodal interactions highlight the importance of synchronizing sensory inputs for a coherent and meaningful experience. In VR, combining haptics with temperature modulation can create more immersive and realistic interactions. For example, a VR system could use haptic feedback and visual cues to replicate the warmth of a sunny beach or the chill of a snowstorm, enhancing the user’s sense of presence.


Haptics in Emotional and Interpersonal Experiences

Touch isn’t just functional—it’s deeply emotional. Haptic sensations in VR can evoke feelings of comfort, fear, or excitement, depending on how they are designed. Research on the cutaneous rabbit illusion, where participants feel “hops” on their arm, shows how tactile feedback can influence emotions like arousal and valence.


Wearable haptic systems, such as gloves, smart clothing, and vests, are being developed to provide tactile feedback that carries emotional meaning. These devices can simulate caresses, tickling sensations, or even the comforting pressure of a hug, paving the way for emotionally expressive communication in virtual and augmented realities.


However, existing haptic stimuli often lack the ability to fully capture the emotional nuances of real-world touch. To unlock the full potential of haptics, researchers should design stimuli that evoke emotions, identify socially acceptable touchpoints, and improve the integration of tactile feedback into eXtended Reality (XR) systems. These advancements could transform how people connect and communicate, especially in mediated or virtual environments.


Applications Across Fields

Haptics is already making waves across diverse fields:

  • Healthcare and Rehabilitation: Haptic feedback in VR has been instrumental in neurorehabilitation for individuals with upper limb paralysis. Devices like robotic exoskeletons and haptic gloves provide tactile stimulation during therapy, promoting motor and sensory recovery while engaging patients in interactive exercises. These technologies not only improve physical outcomes but also enhance patient motivation by integrating gamified elements into therapy. Mid-air Haptics has similarly been used to reduce anxiety during medical procedures, demonstrating the versatility of haptic technology in healthcare.
  • Art and Immersion: In artistic VR installations, passive haptics—like vibrations underfoot when “walking” on virtual paintings—can be paired with temperature shifts to simulate the feel of stepping on different materials.
  • Accessibility: For individuals with sensory challenges, haptics can provide more nuanced and informative feedback, bridging gaps in sensory perception.

These applications demonstrate how haptics can enrich both functional and creative experiences.


Future Challenges and Opportunities

As promising as haptic technology is, challenges remain. Designing devices that seamlessly integrate touch feedback is technically complex. Moreover, creating socially acceptable and emotionally expressive tactile stimuli requires careful consideration of cultural and personal differences. Future research will likely explore these intersections, advancing haptic systems that are not only precise and realistic but also adaptable and inclusive.


Conclusion

Haptics is at the frontier of sensory innovation, transforming virtual reality into a multisensory experience that engages touch, vision, audition, and emotion. By harnessing these technologies, we can create inclusive, immersive environments that redefine how we interact with both the virtual and physical worlds.


As we move forward, the integration of haptics in neurorehabilitation, art, and accessibility offers exciting possibilities—not just for technology, but for human connection and understanding.


See our blog for Activities; especially 55-57.


Some suggestions for further listening and watching

Emergence Gallery: Virtual Walking

Haptic gloves help blind people ‘see’ art

Is That my Real Hand?

Smart Clothing

The Predictive Perception of Dynamic Vibrotactile Stimuli Applied to the Fingertip

The VR Dilemma: How AR and VR redefine our reality

Understanding Affective Touch for Better VR Experience

Virtual reality: how technology can help amputees

Virtual Reality Used To Treat Mental Health Problems


And reading

Haptics for Human-Computer Interaction: From the Skin to the Brain

Interpersonal Haptic Communication: Review and Directions for the Future

The Effect of Multimodal Virtual Reality Experience on the Emotional Responses Related to Injections

Walking on Paintings: Assessment of passive haptic feedback to enhance the immersive experience

What the Mind Can Comprehend from a Single Touch

Colours without vision

Colours are perceived solely through vision. Indeed, when S.B. gained sight after more than 50 years of blindness, he saw the world for the first time in colours. (See our blog for the scientific approach and Vision, haptic touch, and hearing.)


So, what is colour to people who are blind? This post presents some lived experiences.


The eye perceives colours when light reflects off an object. It detects their hue (e.g., red or orange), lightness or darkness (e.g., pink or red), and intensity (e.g., bright or dull yellow). Various devices, smartphone apps, and a braille-like alphabet have been developed to convey this information to people who are blind. While sighted adults and children rather describe:

– the colours through their other senses. For example, that “pink is fangirls screaming”

– what colours do. For example, that red attracts their attention

– how colours make them feel. For example, that blue relaxes them

 

“So, being blind since birth, I’ve never seen color. I don’t have any concept of what it is (…) there’s this whole part of vocabulary of language that doesn’t mean anything to me. Over the years people have tried and tried to explain color to me, and I just don’t understand it.”

“Because I have never seen colors, I don’t think I really understand what they are. However, I do know the color of many things: crows are black, leaves are green, and hair may be blonde, brown, red, black, gray, white — or dyed any color a person desires. I also know colors are highly symbolic in our culture: “I was so angry, I saw red.” Or, ‘I was tickled pink.’”


In fact, the concept of colour is just as abstract for people who are born blind as the concept of justice. Both are learnt through language, by reading and hearing about them.

 

“Ocean green smells so nice, it’s my favourite colour.” (M.O.)

“(…) right now my favorite color is purple, and that is because of my musical hero Prince. Everything he had and did, purple was his thing so I just figured why not. I like Prince, therefore I like purple. I’m not quite sure what I understand about the color purple. I mean it’s not as popular as red, green, orange, or blue or any of those, right? I hear about those things all the time, but I don’t hear about purple that much. As far as purple stuff? I’m not quite sure. Grapes are purple. I know that. I’m not quite sure what else in nature is purple, though? I don’t know. It feels like it’s sort of a strange color. It’s not in the rainbow. What else is purple?


For sighted people, the concept of colour has sensory qualities and is, therefore, represented in a different part of the brain than the concept of justice.


But what if vision is lost later in life?

 

“So, to me, blue is swimming in a pool, or running my hands under water. Yellow is the heat from the sun (…). When I think of purple, I think of feeling peaceful and calm, or the texture of velvet. When I think of black, I think of a fluffy cat. When I think of white I think of cotton balls, overly soft blankets. And green is the smell of fresh cut grass or walking barefoot in the summer.”

“I group colours into large temperature groups, and narrow it down from there using my other senses. Is it warm like pink, red, orange or yellow? (…) Is it a combination colour like peach which I conceptualize as pink/orange? (…) I keep the relationships memorized. If it’s two-colour combination of warm, a cool or a neutral, I’m usually OK. Then I group colours into three main shade categories. (…) Light/pastel, medium, and dark. I also love when colours are compared to my other senses that don’t involve vision. Like saying something is green like lush summer grass or saying something is brown like a piece of rich dark chocolate.”


B.T., who lost his vision more than 20 years ago, explained that his memory of colours is decaying. Although he remembers them, their hue, their lightness or darkness, and their intensity are no longer as clear as they once were. But not in his dreams. When B.T. is fast asleep and dreaming, the colours are still both nuanced and vivid. (See our blog for the Decay and maintenance of sensory memories and the Visual memories and sensory experiences.)


When vision is absent from birth, colour is just a sum of learnt information. Even black – people who are blind do also not see the colour black. To them, colour is an abstract concept. And, for those who lose vision later in life, there are no crossmodal correspondences that transfers information from the sense of hearing, smell, taste, or touch to vision: colour is solely visual. (See our blog for the crossmodal correspondences between the senses.) It seems they have to actively create and learn associations between their memory of each colour and their other senses, as well as systems for memorising them. Or else their memories will decay.


See our blog for Activities; especially 52-54.