Category: Haptic touch

Association strategies in crossmodal metaphors

smackenz/ / Crossmodal correspondences, Flavour, Haptic touch, Hearing, Metaphor, Multisensory, Posts, Smell, Taste, Vision

Several correspondences between the senses exist. For example, transferring information about shape between touch and vision. Associating the sound of spoken words and visual shapes (as in the Bouba/Kiki-effect).

Rounded blob and spiky blob
(Bouba [left] and Kiki [right])
And, subjectively, the scent of a specific perfume with the feel of velvet fabrics. (See our blog for the scientific approach, Crossmodal correspondences between the senses, On the intriguing association between sounds and colours, and Multisensory processing.)

These correspondences are visible in crossmodal metaphors too. That is, when people are using words and phrases related to one sense to describe an experience from another sense. Like when they label visual colours, through words that are specific to the sense of hearing, calling them “loud” and “mute”. And define a sound through the sense of touch, as with “a smooth voice”.

 

I have invited researchers connected with the Diverse-ability Interaction Lab to write this post on how people generate and interpret crossmodal metaphors. These researchers have identified seven association strategies. The Diverse-ability Interaction Lab aims to change the design of interactive technologies in ways that make them inclusive, both for people who are disabled and people who are non-disabled. This post is written by Tegan Roberts-Morgan, University of Bristol.

 

“Blue tastes like salt, it just does”. That is what one participant told me when I asked them what blue might taste like. We all make connections between our senses. A citrus smell may be sharp; someone may have a sweet voice, or red might remind you of anger. We call these cross-sensory metaphors, as they use words from one sense to describe something which is typically associated with another sense. As a HCI researcher in sensory technologies, this is important, as understanding how these metaphors are created can give us an insight into the methods behind our sensory thinking, supporting us to hopefully design better sensory technologies.

 

We use association strategies to represent the different methods people use to create connections between different senses. These strategies help us to begin to understand the reasons behind why we make the cross-sensory metaphors that we do. If we can understand why the connections are made, then this can be leveraged in the design of technologies that support communication. To explore these strategies, we designed tasks that encourage participants to think in cross-sensory terms. For example, in Sense-O-Nary, participants are given an item related to a specific sense (e.g. the colour red, a pyramid, or a lemon scent) and asked to describe it using a sense that is not typically associated with it (e.g. what does red smell like, or what does a pyramid sound like?). They then share their cross-sensory metaphor with another team, who must guess which item is being described. This task, along with others we used, helped us to identify the 7 different strategies people use when creating cross-sensory metaphors.

  • Participants used personal stories and memories, and we labelled these as the personal connection strategy. One participant, for example, said that the lemon scent reminded them about when they went “on holiday to the Mediterranean” or “this reminds me of my friend”.
  • Participants also created cross-sensory metaphors using the familiar experience strategy. This is when the metaphor created uses a common object, emotion, texture etc. “This smells like a banana smoothie” or “this reminds me of a marshmallow” and even “this tastes like soy sauce”.
  • Some participants rely on some basic primitives to make an association, which we labelled as the sensory features strategy. This includes words like “sharp”, “smooth”, “soft” “bitter” and “sweet”.
  • Participants also used the valence strategy, using negative or positive words in the description, for example “I like this”, “I love this“ and “this would taste horrible”.
  • Another approach was using vocalisation. This involved participants using a sound or noise as
  • opposed to words to describe an item like “this sounds like Krrrr and tssssss”, “boooom” or when one child just screamed to describe what red may sound like.
  • Some participants chose not to use the sense that we originally asked them to use; they would instead use words from a different sense. We called this grasping for another sense. In one study we asked participants to describe how red would taste and they said, “this tastes strong”.
  • Finally, some participants did not only use their words to communicate their connection, but they also used their body. When they did this, they used the embodied action strategy. An example includes when one participant said green “feels like this” and then stroked the floor back and forth.

 

We believe that understanding and using these strategies can support designers, educators, and researchers in creating experiences that align with how people naturally relate the senses. For instance, we found that most adults used personal connections when describing how something would sound, so incorporating prompts or features that relate to a memory the person may have could support their communication.

 

We have found that age plays a vital role in what association strategies a person uses. Children tend to use familiar experiences the majority of the time, describing the item using something common. Whereas young adults (18-25 years olds) also used familiar experiences, but used personal connections, additionally, to create their metaphor. And finally older adults (65-80 years old) used a much wider range of association strategies, with sensory features being used more often.

 

These association strategies can be applied in any context that involves multisensory interactions, from educational devices that support children learning about their phonics by using shapes and audio, boards that can help children explain how their pain feels by using scents, shapes, colours etc., and accessible technology to support communication between children who are sighted and children who are visually impaired. Ultimately, association strategies give us a window into how people construct meaning across their senses. By recognising and applying these strategies, we can potentially design experiences that resonate more deeply, communicate more clearly, and build richer, more inclusive multisensory worlds.

 

See our blog for Activities; especially 70-72.

Magic without vision: Can you fool the ear like you fool the eye?

smackenz/ / Haptic touch, Hearing, Illusions, Magic tricks, Posts, Vision

Sometimes the brain gets it all wrong because it misperceives conflicting information from one or more of the senses (see our blog for Sensory illusions before and after vision). And sometimes because it is tricked by a magician. Typically, through vision.

So, are magic tricks possible without vision?

“When it comes right to it, it’s a trick for the eye. I mean, it’s a magic trick, right? So, they trick your eyes. (…) So, if you’re blind, I don’t think that’s really going to work for you. You’re never going to know what just happened. You can hear about it, but that’s not going to be a big deal for you” (according to Tommy Edison, who is born blind).

However, the brain can be tricked through the sense of touch too. For example, to feel three human fingerbones move in the palm of the hand, and two pieces of rope go straight through the neck.

“So you could show me something and show me how it changes or whatever but at any point, if I’m not touching the thing. You can just switch it just like that. And I wouldn’t know wiser” (explained by Tommy Edison, who is born blind).

But what about the other senses?

This time, Associate Professor Gustav Kuhn, University of Plymouth sheds some light on tricking the brain through the sense of hearing. Anchored in both magic and science, Gustav Kuhn has established Science of Magic, he is the Director of the MAGIC-Lab, and he is the funding member and President of The Science of Magic Association. Furthermore, Gustav has published numerous academic articles and books. And his work has been featured in the national media (e.g., by the BBC and the Guardian).

Magic is an artform that allows us to experience the impossible. At its core lies a powerful contradiction between what we see and what we believe to be possible. Most magic tricks exploit our visual perception: we see a coin vanish or a rabbit appear in a hat. But what happens when we remove sight from the equation? Can magic still work if it’s entirely auditory? And if not—why?

Investigating how magic might operate without vision offers more than practical benefits—it opens the door to fundamental questions about how different senses process information, and how much we trust each of them. For centuries, artists and scientists alike have tried to translate between the senses, asking whether sound can be “seen” or images can be “heard.” This research can support the development of sensory substitution tools—but it also highlights important limits in how different senses convey information.

Auditory-only magic tricks

In 2024, we launched a bold experiment: an international competition to create the world’s first auditory-only magic trick. The rules were strict—no visuals, no speech, no language. Just sound. Could the ear be fooled as convincingly as the eye?  Magic is full of non-verbal visual illusions—a vanishing coin, a colour-changing silk—but could pure sound evoke that same sense of wonder?

The challenge captured global attention. Magicians, sound artists, and psychologists all stepped up with creative and clever ideas. One trick stood out for its clever use of an auditory illusion. In this “trick” the magician walks around the blindfolded spectator, making a sound, and the spectator tracks their position (link to more information about the competition)—until, suddenly, the sound shifts and the magician appears to “teleport” to another spot, creating a surprising and disorienting effect. The trick was clever, but since it relied on language, it did not fully meet the competition criteria. Despite our best efforts, no submission delivered a powerful, language-free illusion. We awarded a prize for ingenuity, but the core goal remained unmet. Still, this wasn’t failure—it was insightful and what we predicted

Nearly all well-known magic tricks are visual. As a result, people with visual impairments are often excluded from fully engaging with magic performances. This isn’t just an issue of entertainment; magic can foster joy, curiosity, and even wellbeing. Ensuring access to those experiences is an important step toward inclusivity.

Both our eyes and ears are easily fooled by perceptual illusions, though we rarely notice them. For example, when listening to music on headphones, sounds can seem to come from different directions. This illusion is created by adjusting the volume and timing of sounds in each ear, and there are lots of other auditory illusions that are just as powerful as their visual counterparts. So, the rarity of auditory magic tricks isn’t due to a lack of auditory illusions.

One possibility is for the lack of auditory magic tricks is that auditory perception simply |doesn’t produce the same kind of cognitive surprise as vision. We know our ears can miss things. In the classic cocktail party effect, for instance, we often fail to notice conversations around us until we hear our name. Unlike visual failures, these auditory lapses don’t feel shocking. We accept them as normal. That means there’s less room for magic to create the emotional jolt that makes a trick truly effective. Both our eyes and ears are vulnerable to deception, but we seem to trust our vision more. When something vanishes from sight, it feels like it’s gone. But when a sound disappears, we simply assume the source has stopped making noise—not that the object has vanished. Many magic tricks revolve around manipulating physical objects—making them appear, disappear, or transform. These feats are inherently more compatible with how we interpret visual information.

The competition revealed just how much magic depends on vision, and how deeply we trust what we see compared to what we hear. In trying to trick the ear, we uncovered something fundamental about how we experience the world—and the unique role vision plays in shaping belief.

Toward a New Kind of Magic

We believe magic without sight is a frontier worth exploring—not just for theoretical insight, but for practical innovation. Developing magic for people with visual impairments is about more than accessibility; it’s about reimagining what magic can be. It invites us to consider whether the ear and other senses such as touch and smell, too, can be deceived in ways that are joyful, meaningful, and profound. To do this, we’ll need to understand how people with blindness experience magic—something that can only be achieved through close collaboration between psychologists, magicians, and individuals with visual impairments. In doing so, we not only open the world of magic to those who experience the world differently—we also expand the very boundaries of what magic, as an art and a science, can do.

Gustav has also very kindly suggested some very interesting materials for further listening, reading, and watching:

See our blog for Activities; especially 67-69.