Category: Vision

How the internal senses may connect sight and sound

smackenz/ / Art, Crossmodal correspondences, Emotion, Hearing, Interoception, Movement, Posts, Proprioception, Vestibular, Vision

Certain information is associated across the senses. Some of these crossmodal associations are shared by most people. For example, in the Bouba/Kiki-effect, more than 95% of people around the world match the spoken word “Bouba” with a rounded shape and the spoken word “Kiki” with an angular shape. Other crossmodal associations are subjective; while some people see colours when hearing music, others read braille in colour. And it seems these subjective associations between the external senses may be closely related to the internal senses. (See our blog for How anxious individuals perceive odours, Emotional perceptions associated with sound environments, and Growing into one’s own body.)

I have invited Dr Marina Iosifian, School of Divinity, University of St. Andrews to write this post about crossmodal associations between visual paintings and sounds. Dr Iosifian has contributed to several scientific papers and public outreach events on how the internal senses might create crossmodal associations between vision and hearing.

Have you ever noticed that certain colours seem to “fit” certain sounds? For example, dark red might feel like it matches a low, deep voice, while pink feels more like a high, light voice. These kinds of connections between different senses—such as sight and hearing—are called cross-modal associations. Researchers study them to understand how our brain brings together information from different senses to form a unified picture of the world, even though each sense works separately (our eyes only see, our ears only hear).

Why do these associations happen? One possible explanation involves emotion. For instance, dark red and a low voice might both feel connected to sadness, while pink and a high voice might both be linked to happiness or playfulness.

But emotions aren’t the only reason. Another explanation has to do with the body’s movements and sensations. For example, when people are asked to name two round tables—one large and one small—they often call the large one “mal” and the small one “mil.” This may be because of how our mouths move when saying these sounds: “mal” requires a wider, more open mouth shape, similar to something large, while “mil” involves smaller, tighter movements, like something small.

Girl in garden scene with a cat and a dog
The Garden Walk, by Emile Friant. Retrieved from WIKIART

In our study, we explored these bodily mechanisms—the ways our physical sensations and actions might shape how we connect sights and sounds—to better understand how cross-modal associations arise.

To explore these associations, we collected a set of sounds produced by the human body, such as the sound of someone drinking. We called these embodied sounds. To provide a contrast, we also included sounds that cannot be produced by the human body, such as electronic or synthesized sounds, which we called synthetic sounds.

Because we were interested in how sounds are connected with visual experiences, we also gathered a collection of images. These included two types of paintings: figurative paintings, which show recognizable subjects like people or objects (eg, The Garden Walk by Emile Friant), and abstract paintings, which do not represent specific things (eg, Sky above clouds by Georgia O’Keefe). We then paired the paintings with the sounds and asked our participants a simple question: “Does this sound and this painting fit together?”

Glowing horizon with fluffy white clouds below
Sky above clouds, by Georgia O’Keefe. Retrieved from Custom Prints, Georgia O’Keefe Museum

We found that embodied sounds were more often associated with figurative paintings, while synthetic sounds were more often linked with abstract paintings. This suggests that the body—and the way we experience sensations physically—plays an important role in how people connect what they see with what they hear.

Why might these associations occur? One possible explanation lies in the difference between concrete and abstract ways of thinking. Figurative paintings depict familiar, tangible things—people, objects, and scenes—so they may evoke more concrete thinking. Abstract paintings, on the other hand, invite a more imaginative or distant mindset.

Interestingly, previous research has shown that people tend to associate abstract art with more distant situations—whether in time or space—compared to figurative art. This idea is related to the psychological concept of psychological distance, where concrete things feel close to us and abstract things feel farther away. Our results suggest that this distinction between the concrete and the abstract may also shape how we connect sights and sounds.

Some researchers believe that psychological distance is one of the main concepts which can help us understand how the mind works. They developed the Construal Level Theory or CLT – which explains how our mental distance from things – called psychological distance – affect the way we think about them. Psychological distance can take many forms: something can feel distant in time (happening in the future or past), in space (far away), in social distance (involving people unlike us), or in hypotheticality (something uncertain or imaginary). It is suggested that people think about things that feel close to them—such as events happening soon or nearby—in a more concrete and detailed way. In contrast, things that feel distant in time or space, are understood in a more abstract and general way.

If abstract thinking is linked to distant, less embodied experiences, and concrete thinking to close, bodily ones, then the way we perceive and connect sounds and images may depend on how “distant” or “close” they feel to us psychologically. In other words, our sense of distance—both mental and sensory—may shape how we integrate what we see and hear.

Thus, the concept of abstraction offers valuable insight into how people interpret and understand the world around them. Art, in particular, provides a powerful way to explore these processes. Recent research suggests that engaging with beauty in art can encourage people to think in more abstract ways, making art an especially meaningful tool for studying perception and the connections between our senses.

See our blog for Activities; especially 85-87.

Some suggestions for further reading, listening, and watching:

Applying Bodily Sensation Maps to Art-Elicited Emotions

From Perception to Pleasure

From Sensory to Movement

How Does Your Body React to Art?

How Music Changes Your Mind

Processing Internal Sensory Messages

See What Your Brain Does When You Look at Art

Choosing food

smackenz/ / Emotion, Flavour, Haptic touch, Hearing, Multisensory, Posts, Smell, Taste, Vision

When it comes to eating, “The first bite is with the eyes”. Feeling the texture stimulates the appetite. 75-95% of the flavour comes from smell and the flavour is enhanced by “sonic seasoning“. Eating is a truly multisensory experience (see our blog for Multisensory processing and Food for thought: taste, smell and flavour.) And it seems the senses also help us decide what food is familiar and, thus, whether to eat it or not.

In this post, I have invited Associate Professor Suzanna Forwood, Anglia Ruskin University to reflect on how the senses affect what food we decide to eat or not to eat. Suzanna Forwood conducts research on the factors that determine our food choices, including available tools for healthy choices.

When offered a menu, most people do not seek out the least familiar dish for their dinner.  This is because most of us need food to be familiar for it to be appealing, and there are good reasons for this. From an evolutionary perspective, familiar food eaten in the past without any ill-effects is more likely to be safe this time, and safe food is essential for survival. From a psychological perspective, familiar food is food that we know about how filling or tasty it might be, and we need this information when choosing something to eat so we can match our current appetite.

When you reflect on it, eating is a profoundly unusual sensory experience. On the one hand, exploring food is necessary to gain the sensory information that makes it familiar: we don’t know whether we like it or want to eat it until it is familiar. On the other hand, eating food is bound up in a social contract: there is an expectation that we know what food we like and that we eat food we are served. This tension is particularly problematic for children who are still learning about the world and find it hard to express what they like. Their reluctance to like and eat less familiar foods can look like picky eating.

Sensory Education activities are designed for children and break this tension: food is not a meal but a game or classroom activity. The philosophy, originating in France and Scandinavia in the Sapere movement, is to offer children the chance to explore food in a structured and non-judgemental activity away from mealtimes. Children are provided with samples of foods, typically fruits or vegetables, to explore using all their senses. Golden rules for these activities are that no one must try or like any of the foods. Activities include variations of a single food or focus on specific senses. Food is discussed in terms of its sensory properties as experienced by the child with no expectation that the child has a preference. Sensory education therefore supports children by growing their familiarity with novel foods, as well as their vocabulary for talking about their sensory experience and communicating their preferences1,2.

The need for familiarity presents challenges when an adult loses part of their sensory world.  Eating is fully multisensory activity: we eat with our eyes, our hands, our mouths, our noses and our ears, and our experience of food merges senses. For example, what we experience as flavour combines information from tastebuds in the mouth and smell receptors in the nose, and what we experience as texture combines information from touch receptors in the mouth and sound receptors in the ear. Simply removing one sensory domain can alter how food is experienced. You can explore this for yourself by tasting a food while holding your nose to block smell or wearing ear defenders to block sound. Doing either of these will change the holistic sensory experience of eating the food: the food will no longer be quite so familiar and there may be a change in how much you like or dislike it.

It’s complicated to adjust to a radical change in sensory or motor function for many reasons but retaining dietary variety and pleasure in eating remains important for health and wellbeing. At a very practical level, then, Sensory Education might offer a structured method for supporting a process of re-learning foods in the new sensory world – re-experiencing foods from an altered multisensory perspective and re-evaluating what is familiar and liked.  Research has not yet explored whether Sensory Education can support adults experiencing sensory difficulty with their diets. We have tried co-developing Sensory Education activities with young adults, and the activities were enjoyed. The next step is to explore whether similar activities can be used with adults adjusting to sensory difficulty, such as visual impairment, or motor difficulty, such as recovery following stroke.

My father is that rare person who chooses unfamiliar foods – I think he enjoys the excitement when on holiday or somewhere new. And I remember thinking this was brave – like most children, I preferred familiar food and was amazed at someone who chose to eat something unknown. As it happened, when the food arrived, he would be presented with a regional dish or a local speciality, and when I tasted it, I learned that unfamiliar foods can be delicious and, in time, familiar favourites. It requires a kind of bravery to explore the unknown.

 

See our blog for Activities; especially 76-78.

 

Some suggestions for further listening, reading, and watching:

Dining in the dark

Eating for children with Sensory Difficulties

5 Sensory Tips for Picky Eaters

How to get your taste and smell back after Covid

I Can’t Taste Anything

Sapere

_______________

1Mustonen, S., Rantanen, R, & Tuorila, H. (2009). Effect of sensory education on school

children’s food perception: A 2-year follow-up study. Food Quality and Preference, 20(3), 230-240. DOI: https://doi.org/10.1016/j.foodqual.2008.10.003

2Reverdy, C. (2011). Sensory Education: French Perspectives. In V. R. Preedy, R. R. Watson,

and C. R. (Eds.) Handbook of Behavior, Food and Nutrition (pp. 143-157) New York: Springer. DOI: https://doi.org/10.1007/978-0-387-92271-3_11