Touching the Future: Exploring Haptics and Multisensory Experiences in Virtual Reality

In real life environments, the brain associates and transfers information, crossmodally, from one sense to another. It integrates and processes information from multiple senses. And emotional perceptions too. (See our blog for the crossmodal correspondences between the senses, crossmodal brain plasticity, multisensory processing, and emmotional perceptions). But what happens in Virtual Realities? Virtual Realities are created to trick us into believing something is real when it is not. They can be all visual, auditory, or tactile – and even multisensory


I have invited Associate Professor Mounia Ziat, Bentley University to write about the sense of touch in multisensory virtual realities. That is, on haptic technologies that simulate the tactile and kinaesthetic sensations we feel when interacting with the real world. Mounia Ziat has published extensively on perception and human interaction with natural and artificial environments. And, she has been awarded numerous prizes and grants for her work (e.g., from the EuroHaptics Society, National Science Foundation,  America’s Seed Fund, and Google Research). In this blog post, Mounia explores the transformative potential of haptics in virtual reality, with applications that enrich accessibility, emotional well-being, rehabilitation, and sensory understanding.


The sense of touch, including its interplay with other sensory modalities, is essential to how we experience and navigate the world. In virtual reality (VR), haptic technologies are unlocking new dimensions of sensory engagement, from emotional resonance to crossmodal integration with temperature, sound, and vision.


Multisensory Integration: The Role of Touch and Temperature

Touch and temperature are deeply intertwined in our perception of the world. Studies on the hue-heat hypothesis, for instance, show how color can influence temperature perception: blue hues can make hot objects feel cooler, while red hues can intensify the sensation of cold​. These crossmodal interactions highlight the importance of synchronizing sensory inputs for a coherent and meaningful experience. In VR, combining haptics with temperature modulation can create more immersive and realistic interactions. For example, a VR system could use haptic feedback and visual cues to replicate the warmth of a sunny beach or the chill of a snowstorm, enhancing the user’s sense of presence.


Haptics in Emotional and Interpersonal Experiences

Touch isn’t just functional—it’s deeply emotional. Haptic sensations in VR can evoke feelings of comfort, fear, or excitement, depending on how they are designed. Research on the cutaneous rabbit illusion, where participants feel “hops” on their arm, shows how tactile feedback can influence emotions like arousal and valence.


Wearable haptic systems, such as gloves, smart clothing, and vests, are being developed to provide tactile feedback that carries emotional meaning. These devices can simulate caresses, tickling sensations, or even the comforting pressure of a hug, paving the way for emotionally expressive communication in virtual and augmented realities.


However, existing haptic stimuli often lack the ability to fully capture the emotional nuances of real-world touch. To unlock the full potential of haptics, researchers should design stimuli that evoke emotions, identify socially acceptable touchpoints, and improve the integration of tactile feedback into eXtended Reality (XR) systems. These advancements could transform how people connect and communicate, especially in mediated or virtual environments.


Applications Across Fields

Haptics is already making waves across diverse fields:

  • Healthcare and Rehabilitation: Haptic feedback in VR has been instrumental in neurorehabilitation for individuals with upper limb paralysis. Devices like robotic exoskeletons and haptic gloves provide tactile stimulation during therapy, promoting motor and sensory recovery while engaging patients in interactive exercises. These technologies not only improve physical outcomes but also enhance patient motivation by integrating gamified elements into therapy. Mid-air Haptics has similarly been used to reduce anxiety during medical procedures, demonstrating the versatility of haptic technology in healthcare.
  • Art and Immersion: In artistic VR installations, passive haptics—like vibrations underfoot when “walking” on virtual paintings—can be paired with temperature shifts to simulate the feel of stepping on different materials.
  • Accessibility: For individuals with sensory challenges, haptics can provide more nuanced and informative feedback, bridging gaps in sensory perception.

These applications demonstrate how haptics can enrich both functional and creative experiences.


Future Challenges and Opportunities

As promising as haptic technology is, challenges remain. Designing devices that seamlessly integrate touch feedback is technically complex. Moreover, creating socially acceptable and emotionally expressive tactile stimuli requires careful consideration of cultural and personal differences. Future research will likely explore these intersections, advancing haptic systems that are not only precise and realistic but also adaptable and inclusive.


Conclusion

Haptics is at the frontier of sensory innovation, transforming virtual reality into a multisensory experience that engages touch, vision, audition, and emotion. By harnessing these technologies, we can create inclusive, immersive environments that redefine how we interact with both the virtual and physical worlds.


As we move forward, the integration of haptics in neurorehabilitation, art, and accessibility offers exciting possibilities—not just for technology, but for human connection and understanding.


See our blog for Activities; especially 55-57.


Some suggestions for further listening and watching

Emergence Gallery: Virtual Walking

Haptic gloves help blind people ‘see’ art

Is That my Real Hand?

Smart Clothing

The Predictive Perception of Dynamic Vibrotactile Stimuli Applied to the Fingertip

The VR Dilemma: How AR and VR redefine our reality

Understanding Affective Touch for Better VR Experience

Virtual reality: how technology can help amputees

Virtual Reality Used To Treat Mental Health Problems


And reading

Haptics for Human-Computer Interaction: From the Skin to the Brain

Interpersonal Haptic Communication: Review and Directions for the Future

The Effect of Multimodal Virtual Reality Experience on the Emotional Responses Related to Injections

Walking on Paintings: Assessment of passive haptic feedback to enhance the immersive experience

What the Mind Can Comprehend from a Single Touch

Emotional perceptions associated with sound environments

People pay more attention to sounds when vision is absent than when it is present. Typically identifying individual sounds of people and objects; activities and scenes; locations and distances. They listen more to sounds for pleasure. For example, to trees whistling, cracking, and hissing in the winter; echoes in narrow places; constantly varying soft sounds; birds singing and the sea. (See our blog for Sensory mismatch, Decay and maintenance of sensory memories, and Using the senses when vision and hearing are impaired). And they describe more emotional perceptions too.


I have invited Professor Christina Eviutami Mediastika, School of Creative Industry, Universitas Ciputra to write this blog post on the emotional perceptions people who are visually impaired and fully sighted describe in soundscapes and soundwalks. Christina E. Mediastika has conducted and published a series of studies on emotional perceptions in parks, on pavements, and inside shopping centres in Surabaya and Yogyakarta, Indonesia. And one study in cinemas.


According to United Nations Development Programme data from 2017, Indonesia has the second largest population of people who are visually impaired (after India), with 1.5% of its population having a visual impairment. The pavements are narrow, damaged, contoured or uneven, and obstructed; the parks are most often located next to major streets, even within roundabouts. And the streets are noisy, polluted, hazardous, and unpleasant. The shopping malls are busy housing a wide variety of offices, and clinics as well as retail.


In our studies, people with visual impairments were encouraged to explore these public facilities and data about their emotional perceptions of each surrounding sound environment was collected. Our studies were based on the ISO 2014 and 2018 Acoustics – Soundscape.


People who are visually impaired describe more emotional perceptions of soundscapes than the fully sighted, especially when outdoors. For example, those with a visual impairment used 56 words and the fully sighted 32 words in urban parks; with “eventfulness” and “pleasantness”, respectively, as the most important emotional perceptions. This finding indicates that engagement with the sonic environment is higher for people who are visually impaired than for those who are fully sighted.

 


More specifically, for people who are visually impaired the most important aspect is how much data the outdoor sound environment provides, which impacts both their ability to navigate and their feeling of being safe (related to “eventfulness”). In contrast, for the fully sighted the most critical aspect is whether the sonic environment makes them feel comfortable (related to “pleasantness”).


Furthermore, people who are visually impaired associate more words to each perceived emotion than the fully sighted. On pavements, for example, “pleasantness” is the most important emotional perception for the fully sighted and “pleasantness-direction-safety” for those with a visual impairment. The fully sighted associate five words to “pleasantness” – comfort, fun, safe, easy, and clear route. And those with a visual impairment three more to “pleasantness-direction-safety” – clear direction, know the location, and soothing: they find pleasantness when they believe in their direction and feel safe.


Regarding indoor sound environments. Both groups perceive “pleasantness” as their dominant emotion inside shopping malls, using words like happy, comfortable, luxurious, and modern.


Moreover, when people who are visually impaired watch movies with a fully sighted narrator guide, their three most significant emotional perceptions are pleasure, storyline, and sound dynamic. Pleasure is associated with comfort, clear sound, and interest in the story. Storyline dimension is the extent to which the storyline was understood. Sound dynamic is connected to intensity and quality of the soundtrack.


There is a misconception that people who are visually impaired have more sensitive hearing than sighted individuals. Both groups have equally sensitive hearing. However, people who are visually impaired rely on sound for many aspects of their daily lives because they are forced to do so, which trains their hearing to be used to its fullest potential. This includes being aware of their surroundings, understanding the direction, recognising safety or danger, and experiencing enjoyment using their surrounding sounds.


See our blog for Activities; especially 46-48.


Some suggestions for further listening, reading, and watching:

Emotional responses to music

From Perception to Pleasure

How the sound in your home affects your mood

Soundspace: A Manifesto

The Psychology of Sound and Emotions

Urban noise: Why cities need to turn down the volume