Category: Crossmodal correspondences

How the internal senses may connect sight and sound

smackenz/ / Art, Crossmodal correspondences, Emotion, Hearing, Interoception, Movement, Posts, Proprioception, Vestibular, Vision

Certain information is associated across the senses. Some of these crossmodal associations are shared by most people. For example, in the Bouba/Kiki-effect, more than 95% of people around the world match the spoken word “Bouba” with a rounded shape and the spoken word “Kiki” with an angular shape. Other crossmodal associations are subjective; while some people see colours when hearing music, others read braille in colour. And it seems these subjective associations between the external senses may be closely related to the internal senses. (See our blog for How anxious individuals perceive odours, Emotional perceptions associated with sound environments, and Growing into one’s own body.)

I have invited Dr Marina Iosifian, School of Divinity, University of St. Andrews to write this post about crossmodal associations between visual paintings and sounds. Dr Iosifian has contributed to several scientific papers and public outreach events on how the internal senses might create crossmodal associations between vision and hearing.

Have you ever noticed that certain colours seem to “fit” certain sounds? For example, dark red might feel like it matches a low, deep voice, while pink feels more like a high, light voice. These kinds of connections between different senses—such as sight and hearing—are called cross-modal associations. Researchers study them to understand how our brain brings together information from different senses to form a unified picture of the world, even though each sense works separately (our eyes only see, our ears only hear).

Why do these associations happen? One possible explanation involves emotion. For instance, dark red and a low voice might both feel connected to sadness, while pink and a high voice might both be linked to happiness or playfulness.

But emotions aren’t the only reason. Another explanation has to do with the body’s movements and sensations. For example, when people are asked to name two round tables—one large and one small—they often call the large one “mal” and the small one “mil.” This may be because of how our mouths move when saying these sounds: “mal” requires a wider, more open mouth shape, similar to something large, while “mil” involves smaller, tighter movements, like something small.

Girl in garden scene with a cat and a dog
The Garden Walk, by Emile Friant. Retrieved from WIKIART

In our study, we explored these bodily mechanisms—the ways our physical sensations and actions might shape how we connect sights and sounds—to better understand how cross-modal associations arise.

To explore these associations, we collected a set of sounds produced by the human body, such as the sound of someone drinking. We called these embodied sounds. To provide a contrast, we also included sounds that cannot be produced by the human body, such as electronic or synthesized sounds, which we called synthetic sounds.

Because we were interested in how sounds are connected with visual experiences, we also gathered a collection of images. These included two types of paintings: figurative paintings, which show recognizable subjects like people or objects (eg, The Garden Walk by Emile Friant), and abstract paintings, which do not represent specific things (eg, Sky above clouds by Georgia O’Keefe). We then paired the paintings with the sounds and asked our participants a simple question: “Does this sound and this painting fit together?”

Glowing horizon with fluffy white clouds below
Sky above clouds, by Georgia O’Keefe. Retrieved from Custom Prints, Georgia O’Keefe Museum

We found that embodied sounds were more often associated with figurative paintings, while synthetic sounds were more often linked with abstract paintings. This suggests that the body—and the way we experience sensations physically—plays an important role in how people connect what they see with what they hear.

Why might these associations occur? One possible explanation lies in the difference between concrete and abstract ways of thinking. Figurative paintings depict familiar, tangible things—people, objects, and scenes—so they may evoke more concrete thinking. Abstract paintings, on the other hand, invite a more imaginative or distant mindset.

Interestingly, previous research has shown that people tend to associate abstract art with more distant situations—whether in time or space—compared to figurative art. This idea is related to the psychological concept of psychological distance, where concrete things feel close to us and abstract things feel farther away. Our results suggest that this distinction between the concrete and the abstract may also shape how we connect sights and sounds.

Some researchers believe that psychological distance is one of the main concepts which can help us understand how the mind works. They developed the Construal Level Theory or CLT – which explains how our mental distance from things – called psychological distance – affect the way we think about them. Psychological distance can take many forms: something can feel distant in time (happening in the future or past), in space (far away), in social distance (involving people unlike us), or in hypotheticality (something uncertain or imaginary). It is suggested that people think about things that feel close to them—such as events happening soon or nearby—in a more concrete and detailed way. In contrast, things that feel distant in time or space, are understood in a more abstract and general way.

If abstract thinking is linked to distant, less embodied experiences, and concrete thinking to close, bodily ones, then the way we perceive and connect sounds and images may depend on how “distant” or “close” they feel to us psychologically. In other words, our sense of distance—both mental and sensory—may shape how we integrate what we see and hear.

Thus, the concept of abstraction offers valuable insight into how people interpret and understand the world around them. Art, in particular, provides a powerful way to explore these processes. Recent research suggests that engaging with beauty in art can encourage people to think in more abstract ways, making art an especially meaningful tool for studying perception and the connections between our senses.

See our blog for Activities; especially 85-87.

Some suggestions for further reading, listening, and watching:

Applying Bodily Sensation Maps to Art-Elicited Emotions

From Perception to Pleasure

From Sensory to Movement

How Does Your Body React to Art?

How Music Changes Your Mind

Processing Internal Sensory Messages

See What Your Brain Does When You Look at Art

Association strategies in crossmodal metaphors

smackenz/ / Crossmodal correspondences, Flavour, Haptic touch, Hearing, Metaphor, Multisensory, Posts, Smell, Taste, Vision

Several correspondences between the senses exist. For example, transferring information about shape between touch and vision. Associating the sound of spoken words and visual shapes (as in the Bouba/Kiki-effect).

Rounded blob and spiky blob
(Bouba [left] and Kiki [right])
And, subjectively, the scent of a specific perfume with the feel of velvet fabrics. (See our blog for the scientific approach, Crossmodal correspondences between the senses, On the intriguing association between sounds and colours, and Multisensory processing.)

These correspondences are visible in crossmodal metaphors too. That is, when people are using words and phrases related to one sense to describe an experience from another sense. Like when they label visual colours, through words that are specific to the sense of hearing, calling them “loud” and “mute”. And define a sound through the sense of touch, as with “a smooth voice”.

 

I have invited researchers connected with the Diverse-ability Interaction Lab to write this post on how people generate and interpret crossmodal metaphors. These researchers have identified seven association strategies. The Diverse-ability Interaction Lab aims to change the design of interactive technologies in ways that make them inclusive, both for people who are disabled and people who are non-disabled. This post is written by Tegan Roberts-Morgan, University of Bristol.

 

“Blue tastes like salt, it just does”. That is what one participant told me when I asked them what blue might taste like. We all make connections between our senses. A citrus smell may be sharp; someone may have a sweet voice, or red might remind you of anger. We call these cross-sensory metaphors, as they use words from one sense to describe something which is typically associated with another sense. As a HCI researcher in sensory technologies, this is important, as understanding how these metaphors are created can give us an insight into the methods behind our sensory thinking, supporting us to hopefully design better sensory technologies.

 

We use association strategies to represent the different methods people use to create connections between different senses. These strategies help us to begin to understand the reasons behind why we make the cross-sensory metaphors that we do. If we can understand why the connections are made, then this can be leveraged in the design of technologies that support communication. To explore these strategies, we designed tasks that encourage participants to think in cross-sensory terms. For example, in Sense-O-Nary, participants are given an item related to a specific sense (e.g. the colour red, a pyramid, or a lemon scent) and asked to describe it using a sense that is not typically associated with it (e.g. what does red smell like, or what does a pyramid sound like?). They then share their cross-sensory metaphor with another team, who must guess which item is being described. This task, along with others we used, helped us to identify the 7 different strategies people use when creating cross-sensory metaphors.

  • Participants used personal stories and memories, and we labelled these as the personal connection strategy. One participant, for example, said that the lemon scent reminded them about when they went “on holiday to the Mediterranean” or “this reminds me of my friend”.
  • Participants also created cross-sensory metaphors using the familiar experience strategy. This is when the metaphor created uses a common object, emotion, texture etc. “This smells like a banana smoothie” or “this reminds me of a marshmallow” and even “this tastes like soy sauce”.
  • Some participants rely on some basic primitives to make an association, which we labelled as the sensory features strategy. This includes words like “sharp”, “smooth”, “soft” “bitter” and “sweet”.
  • Participants also used the valence strategy, using negative or positive words in the description, for example “I like this”, “I love this“ and “this would taste horrible”.
  • Another approach was using vocalisation. This involved participants using a sound or noise as
  • opposed to words to describe an item like “this sounds like Krrrr and tssssss”, “boooom” or when one child just screamed to describe what red may sound like.
  • Some participants chose not to use the sense that we originally asked them to use; they would instead use words from a different sense. We called this grasping for another sense. In one study we asked participants to describe how red would taste and they said, “this tastes strong”.
  • Finally, some participants did not only use their words to communicate their connection, but they also used their body. When they did this, they used the embodied action strategy. An example includes when one participant said green “feels like this” and then stroked the floor back and forth.

 

We believe that understanding and using these strategies can support designers, educators, and researchers in creating experiences that align with how people naturally relate the senses. For instance, we found that most adults used personal connections when describing how something would sound, so incorporating prompts or features that relate to a memory the person may have could support their communication.

 

We have found that age plays a vital role in what association strategies a person uses. Children tend to use familiar experiences the majority of the time, describing the item using something common. Whereas young adults (18-25 years olds) also used familiar experiences, but used personal connections, additionally, to create their metaphor. And finally older adults (65-80 years old) used a much wider range of association strategies, with sensory features being used more often.

 

These association strategies can be applied in any context that involves multisensory interactions, from educational devices that support children learning about their phonics by using shapes and audio, boards that can help children explain how their pain feels by using scents, shapes, colours etc., and accessible technology to support communication between children who are sighted and children who are visually impaired. Ultimately, association strategies give us a window into how people construct meaning across their senses. By recognising and applying these strategies, we can potentially design experiences that resonate more deeply, communicate more clearly, and build richer, more inclusive multisensory worlds.

 

See our blog for Activities; especially 70-72.