As with the case of S.B., most people transfer information between their sense of touch and vision. And they associate certain information across the senses too, for example, a rounded visual shape with the spoken word “Bouba” and an angular visual shape with the spoken word “Kiki”. (See our blog for the scientific approach.) Moreover, some subjective crossmodal correspondences also exist, as with a friend who associates the floral fragrance of her favourite perfume with the colour of red wine and the feel of velvet fabrics. New research has, however, started to question the appropriateness of some of the previously accepted crossmodal correspondences, like that between hue and pitch.
In this blog post, I have asked Professor Charles Spence, University of Oxford to explain the link between the crossmodal transfer of information and the crossmodal correspondences. Charles Spence has published over 1,100 academic articles and books, and he has been awarded numerous prizes for his work (e.g., the 10th Experimental Psychology Society Prize, the British Psychology Society: Cognitive Section Award, and the 2008 IG Nobel prize for nutrition). He established the Crossmodal Research Laboratory at the University of Oxford in 1997.
Artists, scientists, and many others have long been curious about the link between the senses, such as between sight and sound. One of the fundamental questions for many of those who are interested in this area is whether it is possible to translate between one sense and another. The majority of interest in this question has tended to revolve around whether it is possible to translate (or how best to transfer information) between vision and audition. In fact, ever since Isaac Newton, researchers and music practitioners/artists have been suggesting their own idiosyncratic mappings between hue and pitch.
Such crossmodal mappings have often been inspired by the phenomenon of synaesthesia. This is the name given to the idiosyncratic mappings (often between the senses; e.g., as in the case of coloured music synaesthesia) that are experienced by a small minority of the population, and may perhaps be overrepresented amongst artists (such as composers Rimski-Kórsakov and Scriabin). However, it is important to note that the nature of the synaesthete’s mapping between hue and pitch tends to be idiosyncratic. As such, a much more fruitful/robust basis for the crossmodal mapping, or transfer, of information between the senses is actually to base it on the crossmodal correspondences that are common to us all (or at least are shared by the majority of the population).
Crossmodal correspondences go by many different names in the literature, including crossmodal associations, crossmodal equivalences, synaesthetic correspondences, metaphorical crossmodal mappings… These terms are all used to refer to a shared mapping between basic features, attributes, or dimensions between the senses, either when physically presented, or else merely imagined. So, for example, most people tend to associate sweetness with roundness, and angularity with sourness/bitterness; similarly, they associate a pinkish-red food with a sweet taste, and famously, many commentators over the centuries have associated the colour scarlet with the sound of the trumpet.
Given that such crossmodal correspondences tend to be shared across the population, they would appear to represent a sensible basis for those wanting to transfer information from one sense to another, such as those interested in the development of sensory substation systems for the blind. However, one of the challenges for those working in this area is to figure out which of the many possible correspondences will be dominant in any given situation or context. So, for example, consider only how the majority of people will associate a high-pitched sound with a small, light, bright object situated high in space.
Another challenge highlighted by our recent research concerns the question of why so many of those interested in this area have chosen to map hue onto pitch1. In such cases, it would appear that the intuition is that dimensions of experience that are similarly structured must map naturally on to one another. So, for example, both colour and pitch can be represented as circular dimensions, and can potentially also be divided into the same number of basic categories. And yet, the fact that everyone who has put forward a crossmodal mapping between hue and pitch seems to have gone for a slightly different solution raises the question of whether this is really the most appropriate correspondence. Nowadays, some are starting to consider whether a more natural crossmodal mapping (or correspondence) for hue might be timbre instead. Resolving these questions will undoubtedly help those wishing to transfer information between the senses.
Charles now challenges and inspires how we taste and smell what we eat and drink:
Sensploration (FoST 2016)
Multisensory Experience and Coffee (Talks at Google)
Neurocuisine (The Guardian)
The Perfect Meal (RecoSymposium)
See our blog for Activities; especially 7-9.
_______________
1Spence, C., & Di Stefano, N. (2023). Sensory translation between audition and vision. Psychonomic Bulletin & Review.
