Category: Multisensory

Association strategies in crossmodal metaphors

smackenz/ / Crossmodal correspondences, Flavour, Haptic touch, Hearing, Metaphor, Multisensory, Posts, Smell, Taste, Vision

Several correspondences between the senses exist. For example, transferring information about shape between touch and vision. Associating the sound of spoken words and visual shapes (as in the Bouba/Kiki-effect).

Rounded blob and spiky blob
(Bouba [left] and Kiki [right])
And, subjectively, the scent of a specific perfume with the feel of velvet fabrics. (See our blog for the scientific approach, Crossmodal correspondences between the senses, On the intriguing association between sounds and colours, and Multisensory processing.)

These correspondences are visible in crossmodal metaphors too. That is, when people are using words and phrases related to one sense to describe an experience from another sense. Like when they label visual colours, through words that are specific to the sense of hearing, calling them “loud” and “mute”. And define a sound through the sense of touch, as with “a smooth voice”.

 

I have invited researchers connected with the Diverse-ability Interaction Lab to write this post on how people generate and interpret crossmodal metaphors. These researchers have identified seven association strategies. The Diverse-ability Interaction Lab aims to change the design of interactive technologies in ways that make them inclusive, both for people who are disabled and people who are non-disabled. This post is written by Tegan Roberts-Morgan, University of Bristol.

 

“Blue tastes like salt, it just does”. That is what one participant told me when I asked them what blue might taste like. We all make connections between our senses. A citrus smell may be sharp; someone may have a sweet voice, or red might remind you of anger. We call these cross-sensory metaphors, as they use words from one sense to describe something which is typically associated with another sense. As a HCI researcher in sensory technologies, this is important, as understanding how these metaphors are created can give us an insight into the methods behind our sensory thinking, supporting us to hopefully design better sensory technologies.

 

We use association strategies to represent the different methods people use to create connections between different senses. These strategies help us to begin to understand the reasons behind why we make the cross-sensory metaphors that we do. If we can understand why the connections are made, then this can be leveraged in the design of technologies that support communication. To explore these strategies, we designed tasks that encourage participants to think in cross-sensory terms. For example, in Sense-O-Nary, participants are given an item related to a specific sense (e.g. the colour red, a pyramid, or a lemon scent) and asked to describe it using a sense that is not typically associated with it (e.g. what does red smell like, or what does a pyramid sound like?). They then share their cross-sensory metaphor with another team, who must guess which item is being described. This task, along with others we used, helped us to identify the 7 different strategies people use when creating cross-sensory metaphors.

  • Participants used personal stories and memories, and we labelled these as the personal connection strategy. One participant, for example, said that the lemon scent reminded them about when they went “on holiday to the Mediterranean” or “this reminds me of my friend”.
  • Participants also created cross-sensory metaphors using the familiar experience strategy. This is when the metaphor created uses a common object, emotion, texture etc. “This smells like a banana smoothie” or “this reminds me of a marshmallow” and even “this tastes like soy sauce”.
  • Some participants rely on some basic primitives to make an association, which we labelled as the sensory features strategy. This includes words like “sharp”, “smooth”, “soft” “bitter” and “sweet”.
  • Participants also used the valence strategy, using negative or positive words in the description, for example “I like this”, “I love this“ and “this would taste horrible”.
  • Another approach was using vocalisation. This involved participants using a sound or noise as
  • opposed to words to describe an item like “this sounds like Krrrr and tssssss”, “boooom” or when one child just screamed to describe what red may sound like.
  • Some participants chose not to use the sense that we originally asked them to use; they would instead use words from a different sense. We called this grasping for another sense. In one study we asked participants to describe how red would taste and they said, “this tastes strong”.
  • Finally, some participants did not only use their words to communicate their connection, but they also used their body. When they did this, they used the embodied action strategy. An example includes when one participant said green “feels like this” and then stroked the floor back and forth.

 

We believe that understanding and using these strategies can support designers, educators, and researchers in creating experiences that align with how people naturally relate the senses. For instance, we found that most adults used personal connections when describing how something would sound, so incorporating prompts or features that relate to a memory the person may have could support their communication.

 

We have found that age plays a vital role in what association strategies a person uses. Children tend to use familiar experiences the majority of the time, describing the item using something common. Whereas young adults (18-25 years olds) also used familiar experiences, but used personal connections, additionally, to create their metaphor. And finally older adults (65-80 years old) used a much wider range of association strategies, with sensory features being used more often.

 

These association strategies can be applied in any context that involves multisensory interactions, from educational devices that support children learning about their phonics by using shapes and audio, boards that can help children explain how their pain feels by using scents, shapes, colours etc., and accessible technology to support communication between children who are sighted and children who are visually impaired. Ultimately, association strategies give us a window into how people construct meaning across their senses. By recognising and applying these strategies, we can potentially design experiences that resonate more deeply, communicate more clearly, and build richer, more inclusive multisensory worlds.

 

See our blog for Activities; especially 70-72.

Sensory illusions before and after vision

smackenz/ / Haptic touch, Hearing, Illusions, Multisensory, Vision

Sometimes the brain gets it all wrong. It misinterprets the information from one or more of the senses. This phenomenon is commonly known as sensory illusions.

Revisiting S.B., who regained his eyesight after more than 50 years of being blind. Using vision, he now recognised simple shapes and ordinary objects as well as their size. But he closed his eyes in traffic. Perhaps more complex visual information overwhelmed him. Perhaps it did not match his memories from when he was still blind. Or perhaps both. (See our blog for the scientific approach, Vision, haptic touch, and hearing and Sensory mismatch.) A related issue is that of conflicting information within and between the senses. Did S.B. show an effect on sensory illusions based on or including visual information?

When S.B. was still blind, he would have been familiar with both tactile and auditory illusions.

But what about visual illusions?

Visual experience is not necessary to show an effect on all visual illusions1. Indeed, S.B. would have encountered some of them when he was still blind. Simply because certain illusions are both visual and tactile. And S.B. would, therefore, have shown an effect on these illusions immediately after he had started using vision. For example, on the Müller-Lyer Illusion2,3.

Visual illusion: 2 lines of equal length appear unequal when the ends have arrow shapes attached.
(Müller-Lyer Illusion, retrieved from elevers.us)

The Müller-Lyer Illusion consist of two horizontal lines that are identical in length: one with inwards-pointing and one with outward-pointing fins. People who show an effect on this illusion, perceive the line with the outwards-pointing fins as longer than the other line. The Müller-Lyer Illusion is found both in people who are born fully sighted and in people who are born blind. As well as in children (born with very low or no vision; 8–16 years old) after only 48 hours of seeing4. But this was not the case for S.B., who regained his eyesight at the age of 525.

S.B. showed a very weak effect on the visual Müller-Lyer Illusion.

For other visual illusions, visual experience is sometimes necessary and sometimes not. An example is the Ponzo Illusion. The Ponzo Illusion consists of two parallel lines that are converging. These two lines are crossed by several horizontal lines that are identical in length. Almost like a railway track that disappears into the distance. People who show the Ponzo Illusion perceive the crossing lines as becoming shorter and shorter the more the vertical lines converge.

Visual Illusion, perspective of the train tracks makes the 2 yellow lines appear different sizes
(Ponzo Illusion, retrieved from illusionsindex.org)

The Ponzo Illusion does not show and effect in people who rely on their sense of touch. And prior visual experience does not change that6. This illusion is not tactile. At the same time, the visual Ponzo Illusion is found in children (born with very low or no vision; 8–16 years old) after only 48 hours of seeing4. The illusion is visual, but prior visual experience is not necessary. In a parallel vein, the Ponzo Illusion has been translated into an auditory format. This auditory version of the illusion occurs in people who are fully sighed and wearing a blindfold. But not in people who have been blind since before they were 20 months old7. The Ponzo Illusion is not auditory without prior visual experience. S.B. who had been visually impaired from before he was two years old should, therefore, have shown an effect on the visual Ponzo Illusion immediately after he had regained vision. Or on a similar illusion.

Visual illusion using perspective to make figures appear larger.
(retrieved from richardgregory.org)

Instead of judging the length of two lines as in the Ponzo Illusion, S.B. was asked to describe the relative sizes of four men. People who have been fully sighted since birth typically perceive the men as increasing in height. S.B. described: “They don’t look far away, it’s just as though the men were standing underneath (? the buildings). The first man looks smaller, but the last three look the same.” 5 S.B. showed a very weak effect on the visual Perspective Size Changes Illusion. (Gregory & Wallace, 1969, p. 22)

After having regained vision, S.B. would also encounter multisensory illusions that include visual information. These illusions consist of conflicting information from vision, touch, hearing, smell, and/or taste. The brain now has to decide how to deal with this. It most often turns to previous learning. An alternative would be to ignore the visual information. Indeed, multisensory illusions that include visual information do not exist without vision. And also not if the visual information is not associated with the other sensory information in a certain way, for example, the lip movements and the sound of spoken words. Prior visual experience is necessary.

Immediately after having regained vision, S.B. would not show an effect on multisensory illusions that included visual information. But his susceptibility to them would probably increase as he learnt to associate and integrate visual information with other sensory information. (See our blog for Crossmodal correspondences between the senses and Multisensory processing.) That is, if he did not close his eyes.

Now, challenge your senses.

Tactile illusions:

 

Auditory illusions:

 

Visual illusions:

 

See our blog for Activities; especially 65-67.

Blog post author: Dr Torø Graven

 

1. Bean, C H (1938) The blind have “optical illusions.” Journal of Experimental Psychology, 22(3), 283–289. https://doi.org/10.1037/h0061244

2. Heller, M A, … [et al.] (2002). The haptic Müller-Lyer illusion in sighted and blind people. Perception, 31(10), 1263-1274. https://doi.org/10.1068/p3340

3. Millar, S, & Al-Attar, Z (2002) The Müller-Lyer illusion in touch and vision: Implications for multisensory processes. Perception & Psychophysics 64(April), 353–365. https://doi.org/10.3758/BF03194709

4. Gandhi, T, Kali, A, Ganesh, S, & Sinha, P (2016) Immediate susceptibility to visual illusions after sight onset. Current Biology, 25(9), R358-R359. https://doi.org/10.1016/j.cub.2015.03.005

5. Gregory, R L, & Wallace, J G (1969) Recovery from Early Blindness A Case Study. Experimental Psychology Society Monograph, No. 2. https://www.richardgregory.org/papers/recovery_blind/recovery-from-early-blindness.pdf

6. Heller, M A, & Ballesteros, S (2012) Visually-impaired touch. Scholarpedia, 7(11), 8240. http://www.scholarpedia.org/article/Visually-impaired_touch

7. Renier, L, …  [et al.] (2005) The Ponzo Illusion with Auditory Substitution of Vision in Sighted and Early-Blind Subjects. Perception, 34(7), 857-867. https://doi.org/10.1068/p5219