Category: Vestibular

How the internal senses may connect sight and sound

smackenz/ / Art, Crossmodal correspondences, Emotion, Hearing, Interoception, Movement, Posts, Proprioception, Vestibular, Vision

Certain information is associated across the senses. Some of these crossmodal associations are shared by most people. For example, in the Bouba/Kiki-effect, more than 95% of people around the world match the spoken word “Bouba” with a rounded shape and the spoken word “Kiki” with an angular shape. Other crossmodal associations are subjective; while some people see colours when hearing music, others read braille in colour. And it seems these subjective associations between the external senses may be closely related to the internal senses. (See our blog for How anxious individuals perceive odours, Emotional perceptions associated with sound environments, and Growing into one’s own body.)

I have invited Dr Marina Iosifian, School of Divinity, University of St. Andrews to write this post about crossmodal associations between visual paintings and sounds. Dr Iosifian has contributed to several scientific papers and public outreach events on how the internal senses might create crossmodal associations between vision and hearing.

Have you ever noticed that certain colours seem to “fit” certain sounds? For example, dark red might feel like it matches a low, deep voice, while pink feels more like a high, light voice. These kinds of connections between different senses—such as sight and hearing—are called cross-modal associations. Researchers study them to understand how our brain brings together information from different senses to form a unified picture of the world, even though each sense works separately (our eyes only see, our ears only hear).

Why do these associations happen? One possible explanation involves emotion. For instance, dark red and a low voice might both feel connected to sadness, while pink and a high voice might both be linked to happiness or playfulness.

But emotions aren’t the only reason. Another explanation has to do with the body’s movements and sensations. For example, when people are asked to name two round tables—one large and one small—they often call the large one “mal” and the small one “mil.” This may be because of how our mouths move when saying these sounds: “mal” requires a wider, more open mouth shape, similar to something large, while “mil” involves smaller, tighter movements, like something small.

Girl in garden scene with a cat and a dog
The Garden Walk, by Emile Friant. Retrieved from WIKIART

In our study, we explored these bodily mechanisms—the ways our physical sensations and actions might shape how we connect sights and sounds—to better understand how cross-modal associations arise.

To explore these associations, we collected a set of sounds produced by the human body, such as the sound of someone drinking. We called these embodied sounds. To provide a contrast, we also included sounds that cannot be produced by the human body, such as electronic or synthesized sounds, which we called synthetic sounds.

Because we were interested in how sounds are connected with visual experiences, we also gathered a collection of images. These included two types of paintings: figurative paintings, which show recognizable subjects like people or objects (eg, The Garden Walk by Emile Friant), and abstract paintings, which do not represent specific things (eg, Sky above clouds by Georgia O’Keefe). We then paired the paintings with the sounds and asked our participants a simple question: “Does this sound and this painting fit together?”

Glowing horizon with fluffy white clouds below
Sky above clouds, by Georgia O’Keefe. Retrieved from Custom Prints, Georgia O’Keefe Museum

We found that embodied sounds were more often associated with figurative paintings, while synthetic sounds were more often linked with abstract paintings. This suggests that the body—and the way we experience sensations physically—plays an important role in how people connect what they see with what they hear.

Why might these associations occur? One possible explanation lies in the difference between concrete and abstract ways of thinking. Figurative paintings depict familiar, tangible things—people, objects, and scenes—so they may evoke more concrete thinking. Abstract paintings, on the other hand, invite a more imaginative or distant mindset.

Interestingly, previous research has shown that people tend to associate abstract art with more distant situations—whether in time or space—compared to figurative art. This idea is related to the psychological concept of psychological distance, where concrete things feel close to us and abstract things feel farther away. Our results suggest that this distinction between the concrete and the abstract may also shape how we connect sights and sounds.

Some researchers believe that psychological distance is one of the main concepts which can help us understand how the mind works. They developed the Construal Level Theory or CLT – which explains how our mental distance from things – called psychological distance – affect the way we think about them. Psychological distance can take many forms: something can feel distant in time (happening in the future or past), in space (far away), in social distance (involving people unlike us), or in hypotheticality (something uncertain or imaginary). It is suggested that people think about things that feel close to them—such as events happening soon or nearby—in a more concrete and detailed way. In contrast, things that feel distant in time or space, are understood in a more abstract and general way.

If abstract thinking is linked to distant, less embodied experiences, and concrete thinking to close, bodily ones, then the way we perceive and connect sounds and images may depend on how “distant” or “close” they feel to us psychologically. In other words, our sense of distance—both mental and sensory—may shape how we integrate what we see and hear.

Thus, the concept of abstraction offers valuable insight into how people interpret and understand the world around them. Art, in particular, provides a powerful way to explore these processes. Recent research suggests that engaging with beauty in art can encourage people to think in more abstract ways, making art an especially meaningful tool for studying perception and the connections between our senses.

See our blog for Activities; especially 85-87.

Some suggestions for further reading, listening, and watching:

Applying Bodily Sensation Maps to Art-Elicited Emotions

From Perception to Pleasure

From Sensory to Movement

How Does Your Body React to Art?

How Music Changes Your Mind

Processing Internal Sensory Messages

See What Your Brain Does When You Look at Art

Growing into one’s own body

smackenz/ / Interoception, Posts, Proprioception, Vestibular

The brain integrates information not only from the five external senses – hearing, smell, taste, touch, and vision – but also from the three internal senses: interoception, proprioception, and the vestibular sense. (See our blog for the crossmodal correspondences between the senses, Vision, haptic touch, and hearing, Multisensory processing, and Food for thought: taste, smell and flavour). Interoception perceives the body’s internal state. Like hunger, thirst, and muscle fatigue. Proprioception perceives the body’s position. And the vestibular sense perceives the body’s balance, movement, and spatial orientation. However, the brain’s models of “what the body is” and “what the body can do” – known as “body representation” – appears to develop at different rates.

I have invited Lara Coelho, Unit for Visually Impaired People (the UVIP Lab), the Italian Institute of Technology, to write this post about children and their body representations. Lara has conducted multiple studies on how accurately people with and without vision estimate the size of their own bodies, including their feet and hands, and even the length of their own arm reach. The UVIP Lab is led by Dr Monica Gori.

What is body representation?

Have you ever closed your eyes and tried to touch your nose, only to miss slightly? Or remember the shock of seeing yourself in a mirror after a big growth spurt as a child? These are classic examples that demonstrate body representation knowledge is not as straightforward as it seems. Our brains constantly build internal models of the body’s size, shape, and capabilities. These models are what allow us to move gracefully, reach accurately, and feel that our body belongs to us.

Scientists call this body representation, and it underpins nearly everything we do: picking up a cup of coffee, navigating a crowded area, or even imagining ourselves in another person’s position. If this representation becomes inaccurate, whether after rapid growth, injury, or neurological changes, our perception of space and action can become distorted.

For children, whose bodies are rapidly changing, keeping track of “where my body ends” and “what I can do with it” is a constant learning process. Yet, we still do not fully understand how these internal maps develop, or whether different kinds of body knowledge grow at different rates.

In our new study, we explored exactly this question. We discovered that children can simultaneously underestimate the size of their hands and overestimate how far they can reach. This is a fascinating mismatch that reveals just how complex growing into one’s own body really is. The finding adds to the evidence that multiple “body maps” coexist in the brain, serving different functions. In children, these maps are still under construction.

Two ways of knowing your body
In our experiment, we asked 84 children aged 6 to 10 to complete two tasks. The first was a structural body representation task: children were blindfolded and asked to localize points on their own arm: the elbow, wrist, and fingertip, after being lightly touched by the researcher. This gave us a sense of how they internally mapped their arm’s proportions using touch and proprioception alone.

The second was a functional representation task. Here, children sat facing a table where small objects appeared at different distances. The children were no longer blindfolded.

Without moving, they had to decide whether they could reach each object if their arm was fully extended. This required them to judge their action capabilities; the “functional” length of their arm.

Illustration of the two tasks: (A) structural body localization, (B) functional reachability judgment.

By comparing performance across tasks, we could see how these two types of body knowledge relate, or, as it turns out, how they diverge.

The structural underestimation of hand size. Note, how the negative shift only occurs between wrist and fingertip (underestimation of hand length)

Children showed an intriguing pattern. When asked to locate their own body landmarks (the structural task), they consistently underestimated the length of their hand (the distance between the wrist and the fingertip). The forearm, however, was represented quite accurately. Put together, this meant the total arm was slightly underestimated.

In contrast, when asked how far they could reach (the functional task), children overestimated. They believed they could reach objects placed further away than their actual reach allowed. These two biases, underestimation of hand length and overestimation of reach, coexisted in the same children but were not correlated. In other words, children who thought their hands were short were not necessarily the ones who thought their reach was long.

Estimation of reaching ability across age groups. All age groups overestimated their reaching abilities.

This tells us that the brain maintains distinct internal models for “what the body is” and “what the body can do.”

 

Figures reprinted with permission: Cardinali, L., Coelho, L., Becchio, C., & Gori, M. (2025) Underestimation and Overestimation of Hand and Arm Length Coexist in Children. Developmental Science, 28(4):e70035, doi:10.1111/desc.70035

 

Why would the brain keep two different maps?
During childhood, the body changes rapidly, and sensory systems are still calibrating. The brain may separate these representations to allow flexible learning: one system keeps track of the body’s structure; another tracks its functional capabilities. They may update at different speeds as children grow and gain motor experience.

For example, the structural map (used to localize body parts) might depend more on proprioceptive and tactile cues, which can be noisy and slow to mature. The functional map, by contrast, is linked to action prediction and could be influenced by visual feedback and motor exploration, systems that reward overestimation, since trying to reach too far can promote learning.

Having two partially independent maps might therefore be an adaptive feature rather than a flaw. It allows children to explore the boundaries of what their bodies can do while still maintaining a stable sense of their physical form.

The bigger picture
This study builds on a broader effort to understand how the sense of body ownership develops. Our bodies are the lens through which we perceive and act upon the world. By studying how this lens changes during growth, we can better understand not only motor control but also self-awareness, perception, and even social cognition.

 

See our blog for Activities; especially 82-84.

Some suggestions for further listening and watching: