Month: October 2023

Crossmodal correspondences between the senses

tgraven/ / Crossmodal correspondences

As with the case of S.B., most people transfer information between their sense of touch and vision. And they associate certain information across the senses too, for example, a rounded visual shape with the spoken word “Bouba” and an angular visual shape with the spoken word “Kiki”. (See our blog for the scientific approach.) Moreover, some subjective crossmodal correspondences also exist, as with a friend who associates the floral fragrance of her favourite perfume with the colour of red wine and the feel of velvet fabrics. New research has, however, started to question the appropriateness of some of the previously accepted crossmodal correspondences, like that between hue and pitch.

In this blog post, I have asked Professor Charles Spence, University of Oxford to explain the link between the crossmodal transfer of information and the crossmodal correspondences. Charles Spence has published over 1,100 academic articles and books, and he has been awarded numerous prizes for his work (e.g., the 10th Experimental Psychology Society Prize, the British Psychology Society: Cognitive Section Award, and the 2008 IG Nobel prize for nutrition). He established the Crossmodal Research Laboratory at the University of Oxford in 1997.

Artists, scientists, and many others have long been curious about the link between the senses, such as between sight and sound. One of the fundamental questions for many of those who are interested in this area is whether it is possible to translate between one sense and another. The majority of interest in this question has tended to revolve around whether it is possible to translate (or how best to transfer information) between vision and audition. In fact, ever since Isaac Newton, researchers and music practitioners/artists have been suggesting their own idiosyncratic mappings between hue and pitch.

Such crossmodal mappings have often been inspired by the phenomenon of synaesthesia. This is the name given to the idiosyncratic mappings (often between the senses; e.g., as in the case of coloured music synaesthesia) that are experienced by a small minority of the population, and may perhaps be overrepresented amongst artists (such as composers Rimski-Kórsakov and Scriabin). However, it is important to note that the nature of the synaesthete’s mapping between hue and pitch tends to be idiosyncratic. As such, a much more fruitful/robust basis for the crossmodal mapping, or transfer, of information between the senses is actually to base it on the crossmodal correspondences that are common to us all (or at least are shared by the majority of the population).

Crossmodal correspondences go by many different names in the literature, including crossmodal associations, crossmodal equivalences, synaesthetic correspondences, metaphorical crossmodal mappings… These terms are all used to refer to a shared mapping between basic features, attributes, or dimensions between the senses, either when physically presented, or else merely imagined. So, for example, most people tend to associate sweetness with roundness, and angularity with sourness/bitterness; similarly, they associate a pinkish-red food with a sweet taste, and famously, many commentators over the centuries have associated the colour scarlet with the sound of the trumpet.

Given that such crossmodal correspondences tend to be shared across the population, they would appear to represent a sensible basis for those wanting to transfer information from one sense to another, such as those interested in the development of sensory substation systems for the blind. However, one of the challenges for those working in this area is to figure out which of the many possible correspondences will be dominant in any given situation or context. So, for example, consider only how the majority of people will associate a high-pitched sound with a small, light, bright object situated high in space.

Another challenge highlighted by our recent research concerns the question of why so many of those interested in this area have chosen to map hue onto pitch1. In such cases, it would appear that the intuition is that dimensions of experience that are similarly structured must map naturally on to one another. So, for example, both colour and pitch can be represented as circular dimensions, and can potentially also be divided into the same number of basic categories. And yet, the fact that everyone who has put forward a crossmodal mapping between hue and pitch seems to have gone for a slightly different solution raises the question of whether this is really the most appropriate correspondence. Nowadays, some are starting to consider whether a more natural crossmodal mapping (or correspondence) for hue might be timbre instead. Resolving these questions will undoubtedly help those wishing to transfer information between the senses.

Charles now challenges and inspires how we taste and smell what we eat and drink:

Sensploration (FoST 2016)

Multisensory Experience and Coffee (Talks at Google)

Neurocuisine (The Guardian)

The Perfect Meal (RecoSymposium)

See our blog for Activities; especially 7-9.

_______________

1Spence, C., & Di Stefano, N. (2023). Sensory translation between audition and vision.  Psychonomic Bulletin & Review.

The senses and the relationship between them – the scientific approach

tgraven/ / Scientific Approach

It all started with Molyneux asking Locke about seeing after “recovery from blindness”:

“Suppose a man born blind, and now adult, and taught by his touch to distinguish between a cube and a sphere of the same metal. Suppose then the cube and sphere were placed on a table, and the blind man made to see: query, whether by his sight, before he touched them, could he distinguish and tell which was the globe and which the cube?”

And the scientific study of S.B. who, after more than fifty years of blindness, regained vision1. (See our blog for the philosophical approach.)

So, do people who are born blind and who gain vision later in life recognise the world around them?

Yes and no. Gregory and Wallace found that S.B. recognised simple shapes, like circles and squares, as well as ordinary objects, such as chairs and tables, and their size. S.B. read print numbers, print capital letters, and the time on a large clock hanging on the wall. He struggled with complex information: for example, faces and facial expressions meant nothing to him if he did not hear the person’s voice. S.B. could not read the small print letters, and he did not recognise depth and movement. In traffic, he closed his eyes and functioned as if he were still blind; he was surprised by how the moon looked, and he was fascinated by reflections.

It was known long before S.B. that correspondences between the senses exist, but not that information transfers between them in shape and object recognition2.

Gregory and Wallace found that S.B. recognised by vision information that he already knew by touch – indeed, for the first time, demonstrating that information transfers from touch to vision.

Following Gregory and Wallace’s first scientific study on crossmodal recognition, researchers continue to investigate what information is and is not transferred and between which senses. Revisiting Molyneux, they typically measure with what accuracy, speed, or both people recognise information already familiar by touch and now explored by vision. An example of information that transfers between the senses is shape however information that is retained in one sense only is colour. Researchers are also investigating crossmodal plasticity in the brain. They typically scan what brain areas are active in people with one sense missing and people with all senses intact. An example of brain plasticity in people who are born blind is auditory information that activates both the auditory and visual areas (compared with only the auditory in the fully sighted).

When Gregory revisited the historical cases of people who have gained vision, he found that all, perhaps except one, had sad endings3. However, our knowledge from scientific studies allows us to change all that: for example, for people who lose vision later in life and who do not have the same option as S.B. – to function in traffic as if they were still fully sighted. To ensure that people who are partially sighted successfully combine information from low vision and touch. And to ensure that children with and without sensory disabilities learn well together. For an example of sharing sensory experiences in art, please see “Shaping Shapes” and “Please Touch the Art“.

See our blog for Activities; especially 4-6.

_______________

1Gregory, R. L., & Wallace, J. G. (1969). Recovery from Early Blindness A Case Study. Experimental Psychology Society Monograph, No. 2. See: https://www.richardgregory.org/papers/recovery_blind/recovery-from-early-blindness.pdf

2An example of a correspondence between the senses is the sound of spoken words and visualshapes: later known as the “bouba/kiki-effect”.

3Gregory, R. L. (2003). Seeing after blindness. Nature Neuroscience, 6(9), 909-910. See:  https://faculty.washington.edu/gboynton/publications/gregory-newsandviews03.pdf