Category: Crossmodal correspondences

Association strategies in crossmodal metaphors

smackenz/ / Crossmodal correspondences, Flavour, Haptic touch, Hearing, Metaphor, Multisensory, Posts, Smell, Taste, Vision

Several correspondences between the senses exist. For example, transferring information about shape between touch and vision. Associating the sound of spoken words and visual shapes (as in the Bouba/Kiki-effect).

Rounded blob and spiky blob
(Bouba [left] and Kiki [right])
And, subjectively, the scent of a specific perfume with the feel of velvet fabrics. (See our blog for the scientific approach, Crossmodal correspondences between the senses, On the intriguing association between sounds and colours, and Multisensory processing.)

These correspondences are visible in crossmodal metaphors too. That is, when people are using words and phrases related to one sense to describe an experience from another sense. Like when they label visual colours, through words that are specific to the sense of hearing, calling them “loud” and “mute”. And define a sound through the sense of touch, as with “a smooth voice”.

 

I have invited researchers connected with the Diverse-ability Interaction Lab to write this post on how people generate and interpret crossmodal metaphors. These researchers have identified seven association strategies. The Diverse-ability Interaction Lab aims to change the design of interactive technologies in ways that make them inclusive, both for people who are disabled and people who are non-disabled. This post is written by Tegan Roberts-Morgan, University of Bristol.

 

“Blue tastes like salt, it just does”. That is what one participant told me when I asked them what blue might taste like. We all make connections between our senses. A citrus smell may be sharp; someone may have a sweet voice, or red might remind you of anger. We call these cross-sensory metaphors, as they use words from one sense to describe something which is typically associated with another sense. As a HCI researcher in sensory technologies, this is important, as understanding how these metaphors are created can give us an insight into the methods behind our sensory thinking, supporting us to hopefully design better sensory technologies.

 

We use association strategies to represent the different methods people use to create connections between different senses. These strategies help us to begin to understand the reasons behind why we make the cross-sensory metaphors that we do. If we can understand why the connections are made, then this can be leveraged in the design of technologies that support communication. To explore these strategies, we designed tasks that encourage participants to think in cross-sensory terms. For example, in Sense-O-Nary, participants are given an item related to a specific sense (e.g. the colour red, a pyramid, or a lemon scent) and asked to describe it using a sense that is not typically associated with it (e.g. what does red smell like, or what does a pyramid sound like?). They then share their cross-sensory metaphor with another team, who must guess which item is being described. This task, along with others we used, helped us to identify the 7 different strategies people use when creating cross-sensory metaphors.

  • Participants used personal stories and memories, and we labelled these as the personal connection strategy. One participant, for example, said that the lemon scent reminded them about when they went “on holiday to the Mediterranean” or “this reminds me of my friend”.
  • Participants also created cross-sensory metaphors using the familiar experience strategy. This is when the metaphor created uses a common object, emotion, texture etc. “This smells like a banana smoothie” or “this reminds me of a marshmallow” and even “this tastes like soy sauce”.
  • Some participants rely on some basic primitives to make an association, which we labelled as the sensory features strategy. This includes words like “sharp”, “smooth”, “soft” “bitter” and “sweet”.
  • Participants also used the valence strategy, using negative or positive words in the description, for example “I like this”, “I love this“ and “this would taste horrible”.
  • Another approach was using vocalisation. This involved participants using a sound or noise as
  • opposed to words to describe an item like “this sounds like Krrrr and tssssss”, “boooom” or when one child just screamed to describe what red may sound like.
  • Some participants chose not to use the sense that we originally asked them to use; they would instead use words from a different sense. We called this grasping for another sense. In one study we asked participants to describe how red would taste and they said, “this tastes strong”.
  • Finally, some participants did not only use their words to communicate their connection, but they also used their body. When they did this, they used the embodied action strategy. An example includes when one participant said green “feels like this” and then stroked the floor back and forth.

 

We believe that understanding and using these strategies can support designers, educators, and researchers in creating experiences that align with how people naturally relate the senses. For instance, we found that most adults used personal connections when describing how something would sound, so incorporating prompts or features that relate to a memory the person may have could support their communication.

 

We have found that age plays a vital role in what association strategies a person uses. Children tend to use familiar experiences the majority of the time, describing the item using something common. Whereas young adults (18-25 years olds) also used familiar experiences, but used personal connections, additionally, to create their metaphor. And finally older adults (65-80 years old) used a much wider range of association strategies, with sensory features being used more often.

 

These association strategies can be applied in any context that involves multisensory interactions, from educational devices that support children learning about their phonics by using shapes and audio, boards that can help children explain how their pain feels by using scents, shapes, colours etc., and accessible technology to support communication between children who are sighted and children who are visually impaired. Ultimately, association strategies give us a window into how people construct meaning across their senses. By recognising and applying these strategies, we can potentially design experiences that resonate more deeply, communicate more clearly, and build richer, more inclusive multisensory worlds.

 

See our blog for Activities; especially 70-72.

Touch and Nature Learning

tgraven/ / Crossmodal correspondences, Haptic touch, Multisensory, Posts, Vision

Some information is sensory specific, for example, colour to vision and temperature to touch. But most is perceived by several senses. An example is shape. Shape is perceived by vision, touch, and hearing. It transfers between the senses. And when used together, the senses perceive supplementary and overlapping information. (See our blog for Crossmodal correspondences between the senses, Vision, haptic touch, and hearing, and Multisensory processing).

In this post, I have invited the research team behind TOUCH to shed some light on how sighted people use their sense of touch when exploring real objects. The blog post is written by Dr Lisa Bowers, Open University, Professor Andrew Manches, University of Edinburgh, and Professor Laura Colucci-Gray, University of Edinburgh. Here, they present their previous research, leading up to TOUCH, on the role of touch when sighted children learn about nature.

Given the worsening state of our planet, an important challenge for education is how to support and promote a renovated relationship between humans and the natural world. Whilst schools offer an ideal context for this challenge, current teaching approaches tend to emphasise children’s visual experiences, exaggerated through increased engagement with screen devices: the world is presented as if ‘at a distance’, disconnected from everyday experiences and the matters we deeply care about. Although an education through the senses is predicated to support various hands-on initiatives, including outdoor and play-based experiences, its role is typically conceived as a means to actively explore visual information, rather than holding any intrinsic value in its own right.

Our research team has examined the role of touch in how children interacted with and described a selection of nature objects (e.g., a leaf, a shell, and a feather). We have found three key dimensions:

  • Propensity to Touch. Children differ in how much prompting they need to physically pick up objects – ranging from those who immediately touch objects to those who are reluctant even with prompting.
  • Touch interaction. Children differ in the
    © University of Edinburgh

    richness of their touch interaction – from simple tapping to rich exploration using both hands.

  • Touch communication. Children differ in the extent to which they use tactile language in their description of objects, as well as how they simulate touch through gesture when describing what they see.

 

As well as revealing intriguing similarities and differences between children, our research has highlighted the way that touch wasn’t simply a means for children to visually explore objects (although this was important, e.g., manipulating objects to inspect them from all angles) but offered much value as a unique mode of interaction. Indeed, many children would look away from objects whilst exploring them through touch. For example, a child might rub the inside and outside of the shell whilst reflecting on why one was smooth and the other rough and this process would recall prior experiences in other places or in the company of significant others – often positive emotional recollections.

Our research also revealed challenges children often had in describing tactile properties of objects – often drawing on analogies with other touch experiences when they lacked vocabulary. The identification of gestures simulating touch was particularly powerful in revealing how children had internalised touch experiences in their concepts of objects as they described them. This internalisation of sensory experience in conceptual thinking is a key claim of embodied theories of cognition. However, the benefits of tactile learning extend beyond basic sensory exploration. Somatosensory perception is central to early human development and the explicit incorporation of touch as a way to feel, interact, and communicate through the body can enhance learning, with increased engagement and memorability.

Somatosensory Cortex Stimulation by Touch: Somatosensory Cortex - Nerve impulse - Touch
Created by Lisa Bowers, 2025

The foundational work of research studies such as the one described above has highlighted various possible avenues for design – including ways to encourage children’s propensity to touch, ways to encourage children’s exploration of tactile properties in greater depth, or ways to develop children’s tactile language (and possibly gestures) as a means to communicate tactile properties. We have explored some of these opportunities through classroom and outdoor activities, as well as the development of haptic prototype designs where children receive haptic feedback from a special stylus moved over a screen device. For these haptic designs, tactile feedback not only provides a means for children to tactilely explore ‘less accessible’ objects like a bee or unfamiliar plant, but can prime children to be more curious about the feel of real-world natural objects. We are also exploring the potential of this work to quantify the role of touch in order to examine how different factors influence children’s touch interaction and communication (e.g., their age, confidence or nature experiences), and to evaluate how, and if, our interventions influence the role of children’s touch in children’ science learning.

In our future research, touch is conceived of as a medium that builds on human capacities to feel and perceive what might be distant and inaccessible, but also to pay attention to aspects of the world that may go unnoticed or taken for granted. We are now asking ourselves and all we are engaging with a key question – what is the role of touch in what it means to be curious about, explore, think, feel, communicate, and be connected to our natural world?

See our blog for Activities; especially 62-64.