Touching the Future: Exploring Haptics and Multisensory Experiences in Virtual Reality

In real life environments, the brain associates and transfers information, crossmodally, from one sense to another. It integrates and processes information from multiple senses. And emotional perceptions too. (See our blog for the crossmodal correspondences between the senses, crossmodal brain plasticity, multisensory processing, and emmotional perceptions). But what happens in Virtual Realities? Virtual Realities are created to trick us into believing something is real when it is not. They can be all visual, auditory, or tactile – and even multisensory


I have invited Associate Professor Mounia Ziat, Bentley University to write about the sense of touch in multisensory virtual realities. That is, on haptic technologies that simulate the tactile and kinaesthetic sensations we feel when interacting with the real world. Mounia Ziat has published extensively on perception and human interaction with natural and artificial environments. And, she has been awarded numerous prizes and grants for her work (e.g., from the EuroHaptics Society, National Science Foundation,  America’s Seed Fund, and Google Research). In this blog post, Mounia explores the transformative potential of haptics in virtual reality, with applications that enrich accessibility, emotional well-being, rehabilitation, and sensory understanding.


The sense of touch, including its interplay with other sensory modalities, is essential to how we experience and navigate the world. In virtual reality (VR), haptic technologies are unlocking new dimensions of sensory engagement, from emotional resonance to crossmodal integration with temperature, sound, and vision.


Multisensory Integration: The Role of Touch and Temperature

Touch and temperature are deeply intertwined in our perception of the world. Studies on the hue-heat hypothesis, for instance, show how color can influence temperature perception: blue hues can make hot objects feel cooler, while red hues can intensify the sensation of cold​. These crossmodal interactions highlight the importance of synchronizing sensory inputs for a coherent and meaningful experience. In VR, combining haptics with temperature modulation can create more immersive and realistic interactions. For example, a VR system could use haptic feedback and visual cues to replicate the warmth of a sunny beach or the chill of a snowstorm, enhancing the user’s sense of presence.


Haptics in Emotional and Interpersonal Experiences

Touch isn’t just functional—it’s deeply emotional. Haptic sensations in VR can evoke feelings of comfort, fear, or excitement, depending on how they are designed. Research on the cutaneous rabbit illusion, where participants feel “hops” on their arm, shows how tactile feedback can influence emotions like arousal and valence.


Wearable haptic systems, such as gloves, smart clothing, and vests, are being developed to provide tactile feedback that carries emotional meaning. These devices can simulate caresses, tickling sensations, or even the comforting pressure of a hug, paving the way for emotionally expressive communication in virtual and augmented realities.


However, existing haptic stimuli often lack the ability to fully capture the emotional nuances of real-world touch. To unlock the full potential of haptics, researchers should design stimuli that evoke emotions, identify socially acceptable touchpoints, and improve the integration of tactile feedback into eXtended Reality (XR) systems. These advancements could transform how people connect and communicate, especially in mediated or virtual environments.


Applications Across Fields

Haptics is already making waves across diverse fields:

  • Healthcare and Rehabilitation: Haptic feedback in VR has been instrumental in neurorehabilitation for individuals with upper limb paralysis. Devices like robotic exoskeletons and haptic gloves provide tactile stimulation during therapy, promoting motor and sensory recovery while engaging patients in interactive exercises. These technologies not only improve physical outcomes but also enhance patient motivation by integrating gamified elements into therapy. Mid-air Haptics has similarly been used to reduce anxiety during medical procedures, demonstrating the versatility of haptic technology in healthcare.
  • Art and Immersion: In artistic VR installations, passive haptics—like vibrations underfoot when “walking” on virtual paintings—can be paired with temperature shifts to simulate the feel of stepping on different materials.
  • Accessibility: For individuals with sensory challenges, haptics can provide more nuanced and informative feedback, bridging gaps in sensory perception.

These applications demonstrate how haptics can enrich both functional and creative experiences.


Future Challenges and Opportunities

As promising as haptic technology is, challenges remain. Designing devices that seamlessly integrate touch feedback is technically complex. Moreover, creating socially acceptable and emotionally expressive tactile stimuli requires careful consideration of cultural and personal differences. Future research will likely explore these intersections, advancing haptic systems that are not only precise and realistic but also adaptable and inclusive.


Conclusion

Haptics is at the frontier of sensory innovation, transforming virtual reality into a multisensory experience that engages touch, vision, audition, and emotion. By harnessing these technologies, we can create inclusive, immersive environments that redefine how we interact with both the virtual and physical worlds.


As we move forward, the integration of haptics in neurorehabilitation, art, and accessibility offers exciting possibilities—not just for technology, but for human connection and understanding.


See our blog for Activities; especially 55-57.


Some suggestions for further listening and watching

Emergence Gallery: Virtual Walking

Haptic gloves help blind people ‘see’ art

Is That my Real Hand?

Smart Clothing

The Predictive Perception of Dynamic Vibrotactile Stimuli Applied to the Fingertip

The VR Dilemma: How AR and VR redefine our reality

Understanding Affective Touch for Better VR Experience

Virtual reality: how technology can help amputees

Virtual Reality Used To Treat Mental Health Problems


And reading

Haptics for Human-Computer Interaction: From the Skin to the Brain

Interpersonal Haptic Communication: Review and Directions for the Future

The Effect of Multimodal Virtual Reality Experience on the Emotional Responses Related to Injections

Walking on Paintings: Assessment of passive haptic feedback to enhance the immersive experience

What the Mind Can Comprehend from a Single Touch

AI and the Aesthetic Enjoyment of Visual Art without Vision

Certain information, objects, and shapes are recognised through crossmodal correspondences. For example, when people who have just lost vision correctly identify cubes and squares of the same metal by touch alone. (See our blog for the scientific approach and the crossmodal correspondences between the senses,) But, what about the aesthetic enjoyment of perceiving visual art through audio-descriptions, tactile pictures, or both? (See our blog for Drawing pictures with and without vision, A Feel for Art, and On the intriguing association between sounds and colours.) New research suggests Artificial Intelligence can help improve aesthetic enjoyment through generating personalised descriptions: for example, of colours, shapes, and emotions and, thus, kindle people’s imagination and/or visual memories.


I invited the researchers behind Understanding Visual Arts Experiences of Blind People to shed some light on the aesthetic enjoyment of visual art without vision. Together, these researchers aim to understand better ways of supporting the experience of visual art of people who are blind. This blog post is written by Lotus Zhang, University of Washington, Franklin Mingzhe Li, Carnegie Mellon University, and Associate Professor Patrick Carrington, Human-Computer Interaction Institute, School of Computer Science, Carnegie Mellon University.


Art allows the expression of important ideas, emotions, and beliefs in a multitude of forms, and profoundly influences human society. However, most public art exhibits are experienced visually (e.g., photography, drawing, painting, sculpture) and thus pose access barriers for over 2.2 billion people in the world who have vision impairments. Although art museums and galleries increasingly offer accessible tours, these are still limited to a small number of venues and are far from comparably enjoyable to what is offered to sighted visitors.


In the quest for art appreciation, blind enthusiasts face a canvas of challenges. While guided tours offer a glimpse into the art world, they depend heavily on the descriptive skills of companions, often leading to a fragmented understanding of the artwork’s essence. Tactile graphics, though a bridge to the visual, are scarce and demand patience, turning a quick visit into a lengthy exploration. Smart devices promise a solution but falter, lacking the nuanced comprehension of the art’s depth. The digital realm offers remote tours, yet these can’t replicate the profound connection felt when standing before a masterpiece. As they navigate these barriers, blind patrons seek not just access, but a richer, more textured experience of art, where every shade and shape is felt, not just described.


For example, many blind patrons experience difficulties when sighted people describe form information of visual arts (e.g., shape, line, color), as we as a society do not share a standard for describing visual arts in accessible languages (e.g., vocabulary, grammar, visual references):


“I found sighted people always have difficulties explaining color and shape information in detail, which includes the shade of the color, contour of the objects.” (Female, 38, with congenital blindness. See Understanding Visual Arts Experiences of Blind People, section 6.5 Establish Shared Art Vocabulary and Grammar.)


Also, visual descriptions provided by sighted friends, family, and docents can be heavily subjective, making it difficult for blind individuals to form individual interpretations of the artwork:


“(…) I do not want to hear personal comments from people, just like this or that painting is so pretty and meaningful, all I need is what color they used, the contours of the lines, and what kinds of objects present in the painting.” (Female, 29, with acquired blindness at the age of six. See Understanding Visual Arts Experiences of Blind People, section 6.4 Enhance Objective Interpretation.)


Visual imagination and physical connection

For those who acquire blindness later in life, they are able to use their existing visual knowledge alongside conversation with sighted peers to use their imagination to envision and enjoy the artwork:


“I used to have vision when I was young, and I currently enjoy art by imagining from the information I know, such as people, activity, and the environment. I then think about what type of color they might use, or the facial expressions, I imagine everything that I am not told. The magic part is confirming my imagination with sighted friends or family members. And it is totally fine if I am wrong, I still like my imagination on how this artwork should be.” (Male, 25, with acquired blindness at the age of 20. See Understanding Visual Arts Experiences of Blind People, section 5.2 Cognition of Perceiving Visual Arts.)


In contrast, art enthusiasts who are congenitally blind establish their enjoyable experiences through tactile means; engaging with textures, shapes, figures, and paths. The experience of the same artwork through imagined visual details compared with tactile methods evokes different feelings as the reproduction in a more tactile format is fundamentally different from the original piece. This draws attention to personal experience (e.g., visual memory), motivation, as well as the presentation mode of visual arts as key factors for aesthetic enjoyment without vision.


AI augmenting existing descriptions

From the perspective of improving visual art access technologies, we envision significant changes from emerging AI development. For example, by generating vivid descriptions that engage all senses, generative AI can help blind people create mental images or sensations related to the artwork. Descriptions can include the imagined texture of brush strokes, the atmosphere that a scene depicts, or the emotions that the artwork is intended to evoke. Future art access technologies can also consider using generative AI to transform the description of artwork into a dynamic story, making the experience more immersive for blind individuals. Users can ask questions, and generative AI tools can adapt the narrative to focus on aspects that interest the user most, such as the symbolism behind certain elements or the techniques used by the artist. We encourage professionals to explore ways to utilize recent AI development and avoid potential harms.


Lotus, Franklin, and Patrick have also very kindly suggested some very interesting papers and books for us to read:

Asakawa, S., Guerreiro, J., Sato, D., Takagi, H., Ahmetovic, D., Gonzalez, D., Sato, D., Takagi, H., Ahmetovic, D., Gonzales, D., Kitani, K. M., & Asakawa, C. (2019). An independent and interactive museum experience for blind people. Proceedings of the 16th International Web for All Conference, May(30), 1-9.


Axel, E. S., & Levent, N. S. (2003). Art beyond sight: a resource guide to art, creativity, and visual impairment. New York: AFB Press.


Bernardi, R., Cakici, R., Elliott, D., Erdem, A., Erdem, E., Ikizler-Cinbis, N., Keller, F., Muscat, A., & Plank, B. (2016). Automatic description generation from images: A survey of models, datasets, and evaluation measures. Journal of Artificial Intelligence Research, 55(02), 409-442.


Bieber, R., & Rae, J. (2013). From the Mind’s Eye: Museum and Art Gallery Appreciation for the Blind–Canadian Perspectives. Disability Studies Quarterly, 33(3).


Hayhoe, S. (2013). Expanding our vision of museum education and perception: An analysis of three case studies of independent blind arts learners. Harvard Educational Review, 83(1), 67-86.


And some further listening and watching:

Harnessing the power of AI to make art accessible to all

“I hear colour” says colour blind artist with antenna on attached on his skull

Incredible art by visually impaired artists!

Will AI Create New Forms of Art for Blind People?

 


See our blog for Activities; especially 28-30.