Category: Taste

Association strategies in crossmodal metaphors

smackenz/ / Crossmodal correspondences, Flavour, Haptic touch, Hearing, Metaphor, Multisensory, Posts, Smell, Taste, Vision

Several correspondences between the senses exist. For example, transferring information about shape between touch and vision. Associating the sound of spoken words and visual shapes (as in the Bouba/Kiki-effect).

Rounded blob and spiky blob
(Bouba [left] and Kiki [right])
And, subjectively, the scent of a specific perfume with the feel of velvet fabrics. (See our blog for the scientific approach, Crossmodal correspondences between the senses, On the intriguing association between sounds and colours, and Multisensory processing.)

These correspondences are visible in crossmodal metaphors too. That is, when people are using words and phrases related to one sense to describe an experience from another sense. Like when they label visual colours, through words that are specific to the sense of hearing, calling them “loud” and “mute”. And define a sound through the sense of touch, as with “a smooth voice”.

 

I have invited researchers connected with the Diverse-ability Interaction Lab to write this post on how people generate and interpret crossmodal metaphors. These researchers have identified seven association strategies. The Diverse-ability Interaction Lab aims to change the design of interactive technologies in ways that make them inclusive, both for people who are disabled and people who are non-disabled. This post is written by Tegan Roberts-Morgan, University of Bristol.

 

“Blue tastes like salt, it just does”. That is what one participant told me when I asked them what blue might taste like. We all make connections between our senses. A citrus smell may be sharp; someone may have a sweet voice, or red might remind you of anger. We call these cross-sensory metaphors, as they use words from one sense to describe something which is typically associated with another sense. As a HCI researcher in sensory technologies, this is important, as understanding how these metaphors are created can give us an insight into the methods behind our sensory thinking, supporting us to hopefully design better sensory technologies.

 

We use association strategies to represent the different methods people use to create connections between different senses. These strategies help us to begin to understand the reasons behind why we make the cross-sensory metaphors that we do. If we can understand why the connections are made, then this can be leveraged in the design of technologies that support communication. To explore these strategies, we designed tasks that encourage participants to think in cross-sensory terms. For example, in Sense-O-Nary, participants are given an item related to a specific sense (e.g. the colour red, a pyramid, or a lemon scent) and asked to describe it using a sense that is not typically associated with it (e.g. what does red smell like, or what does a pyramid sound like?). They then share their cross-sensory metaphor with another team, who must guess which item is being described. This task, along with others we used, helped us to identify the 7 different strategies people use when creating cross-sensory metaphors.

  • Participants used personal stories and memories, and we labelled these as the personal connection strategy. One participant, for example, said that the lemon scent reminded them about when they went “on holiday to the Mediterranean” or “this reminds me of my friend”.
  • Participants also created cross-sensory metaphors using the familiar experience strategy. This is when the metaphor created uses a common object, emotion, texture etc. “This smells like a banana smoothie” or “this reminds me of a marshmallow” and even “this tastes like soy sauce”.
  • Some participants rely on some basic primitives to make an association, which we labelled as the sensory features strategy. This includes words like “sharp”, “smooth”, “soft” “bitter” and “sweet”.
  • Participants also used the valence strategy, using negative or positive words in the description, for example “I like this”, “I love this“ and “this would taste horrible”.
  • Another approach was using vocalisation. This involved participants using a sound or noise as
  • opposed to words to describe an item like “this sounds like Krrrr and tssssss”, “boooom” or when one child just screamed to describe what red may sound like.
  • Some participants chose not to use the sense that we originally asked them to use; they would instead use words from a different sense. We called this grasping for another sense. In one study we asked participants to describe how red would taste and they said, “this tastes strong”.
  • Finally, some participants did not only use their words to communicate their connection, but they also used their body. When they did this, they used the embodied action strategy. An example includes when one participant said green “feels like this” and then stroked the floor back and forth.

 

We believe that understanding and using these strategies can support designers, educators, and researchers in creating experiences that align with how people naturally relate the senses. For instance, we found that most adults used personal connections when describing how something would sound, so incorporating prompts or features that relate to a memory the person may have could support their communication.

 

We have found that age plays a vital role in what association strategies a person uses. Children tend to use familiar experiences the majority of the time, describing the item using something common. Whereas young adults (18-25 years olds) also used familiar experiences, but used personal connections, additionally, to create their metaphor. And finally older adults (65-80 years old) used a much wider range of association strategies, with sensory features being used more often.

 

These association strategies can be applied in any context that involves multisensory interactions, from educational devices that support children learning about their phonics by using shapes and audio, boards that can help children explain how their pain feels by using scents, shapes, colours etc., and accessible technology to support communication between children who are sighted and children who are visually impaired. Ultimately, association strategies give us a window into how people construct meaning across their senses. By recognising and applying these strategies, we can potentially design experiences that resonate more deeply, communicate more clearly, and build richer, more inclusive multisensory worlds.

 

See our blog for Activities; especially 70-72.

Visual memories and sensory experiences

tgraven/ / Crossmodal brain plasticity, Crossmodal correspondences, Haptic touch, Hearing, Multisensory, Posts, Smell, Taste, Vision

It seems people who lose vision use information that transfers between the senses to retrieve visual memories. And that their visual memories decay when they have reached a certain level of experience in the other senses. (See our blog for the scientific approach, the crossmodal correspondences between the senses, and Decay and maintenance of sensory memories). But what happens when people who are born blind gain vision and then lose it again?

I have invited P.B. to share his experiences of visual memories and sensory information when going from blind to partially sighted and back to being blind again. P.B. was born blind with some light perception in one eye. After surgery and other medical treatments, he had gained about 7% vision in the other eye at three and a half years old. P.B. does not remember this shift from being blind to being partially sighted. Then, in his early twenties, P.B. had an accident that rendered him totally blind in both eyes. P.B. thinks of himself as a sighted person who happens not to see. He visualises when making plans. And his dreams are always visual. P.B. approved this text before we posted it on our blog.

P.B. has several visual memories from when he was little. From the house he grew up in, for example, he remembers the light from a chandelier hanging over the coffee table in the living room, the patterned wallpaper in his room, and the countries on the world map hanging on the wall. And he remembers the colours on the outside walls, the doors, and the window frames.

Today, when he has no new visual input, P.B. creates visual images based on his memories and knowledge. For example, that of a White Swiss Shepherd Dog working as a guide dog from his memories of both the white colour and the German Shepherd Dog. As well as his knowledge of guide dogs.

His visual memories are typically triggered by somebody describing how something looks, like a bright red sunset – not by information from the other senses.

P.B.’s visual memories “flash up almost like the flavour when eating or drinking”.

He has to decide not to focus on visual memories and images when talking to people: the memories are now 20 years old and people have changed. And his created visual images of what new people look like may be very wrong.

P.B. has to actively suppress his urge to retrieve visual memories or create visual images of people based on their voices.

He describes not having visual memories and images of spatial relations and distances. P.B. rather remembers them through his body. He walks around in the city centre with no vision, a podcast or some music in his ears, and shoes with a thick curved sole (which makes it difficult to feel the surfaces on pavements, streets, steps, floors, etc.). He does not count steps or any of that and pauses the podcast or music in his ears only when he feels he might get lost.

For P.B., sensory experiences include a variety of simultaneous sensations as well as visual memories and created images – they are multisensory. The forest, for example, is the sound of wind and trees, the texture of surfaces under his feet, scents, and the memories of colours. He zooms in on whatever attracts his attention: indeed, not on one piece of sensory information after the other in a certain order. For example, zooming in on a sudden shift from soft to hard texture under his feet.

An interesting sensory experience, according to P.B., is that of new flavours, the texture of the food, and the sound of crunch when chewing, He describes a cloudberry cream dessert as “not very sweet, but also not sour, light orange colour, and creamy texture”. But a redcurrant jelly only as “something wobbling on the plate”. He does not describe the flavour, scent, and/or feel in his mouth – merely how the jelly looks on the plate.

Was he not able to suppress his visual memories of jelly – did they take over?

When P.B. is sensory tired, he relaxes his senses by listening to “concentration music, like opera or prog rock where I can immerse myself and get absorbed by a different universe”.

It seems P.B. can maintain his visual memories and ability to create visual images through people’s descriptions of what something looks like. But, at the same time, these memories and images are distracting him from focusing on sensory information that is more relevant to him today with no vision, for example, people’s voices and the feel of wobbling jelly in his mouth, So is this, in fact, a negative spiral – that maintaining visual memories prevents experiences in the other senses, which in turn helps nurture the visual memories?

See our blog for Activities; especially 49-51.