Category: Posts

Vision, haptic touch, and hearing

tgraven/ / Haptic touch, Hearing, Posts, Vision

S.B. after more than fifty years of blindness regained his eyesight. When he recognised simple shapes and ordinary objects as well as their size by vision, he was helped by his tactile memories. Indeed, memories based on one sense influence the perception of another. (See our blog for the scientific approach and the crossmodal correspondences between the senses.)

S.B. matched his new visual information to his memories, stored when he was still blind. When vision is absent, memories about simple shapes and ordinary objects, including size, are based mainly on tactile and auditory information.

Vision perceives a lot of different information simultaneously and it fills in any missing information. It starts with an overview and, then, focuses on details. Visual attention is drawn to colours that stand out from the background, sudden movements, and any changes. Vision perceives information about colour, location, movement, quantity, shape, size, and texture.

Haptic touch – the combination of touch and movement – perceives information serially, piece by piece by piece. It starts with the details and, then, builds up an understanding of the whole information (e.g., a picture, illustration, or scene). Attention seems to be drawn to differences in surfaces and textures. Haptic touch perceives information about hardness, location, movement, quantity, shape, size, texture, and weight1.

Hearing perceives a lot of different information simultaneously. It starts with an overview and, then, focuses on details. Auditory attention is drawn to sudden sounds that stand out from the background. Hearing perceives information about location, movement, quantity, shape, size, sound, and texture.

Vision, haptic touch, and hearing perceive much of the same information, but S.B. was still not able to recognise visual information that he had not experienced by haptic touch, hearing, or both.

Very little information is perceived by merely one of the senses: colour by vision, hardness and weight by haptic touch, and sound by hearing. And correspondences exist between this sensory-specific information, for example, hue and pitch (or timbre). Shape is an example of information that is perceived by all three senses. It transfers between them, and when they work together all sensory information is integrated to form or retrieve memories of it.

Only when the visual information matched well with his memories, based mainly on tactile and auditory information, did S.B. recognise the world around him. It seems the match is best with information that is perceived by two (or all) senses: S.B. recognised simple shapes and ordinary objects as well as their size.

The brain adapts rapidly when using a different sense, but it needs time to permanently change how it processes the perceived information. During this phase, it would be beneficial to use the senses for perceiving information that they are naturally attentive to and therefore familiar with already, for example, texture for haptic touch. Research has found that it also helps to focus on information that transfers between the senses, for example, shape. Both would serve as warm-up activities for the brain, preparing it for more complex information to come, such as traffic. To hear a series of favourite city soundscapes, please listen to “The Oxford Sound Album“.

See our blog Activities; especially 10-12.

_______________

1The sense of touch, without movement, perceives information about temperature, pain, and pleasure.

Crossmodal correspondences between the senses

tgraven/ / Crossmodal correspondences

As with the case of S.B., most people transfer information between their sense of touch and vision. And they associate certain information across the senses too, for example, a rounded visual shape with the spoken word “Bouba” and an angular visual shape with the spoken word “Kiki”. (See our blog for the scientific approach.) Moreover, some subjective crossmodal correspondences also exist, as with a friend who associates the floral fragrance of her favourite perfume with the colour of red wine and the feel of velvet fabrics. New research has, however, started to question the appropriateness of some of the previously accepted crossmodal correspondences, like that between hue and pitch.

In this blog post, I have asked Professor Charles Spence, University of Oxford to explain the link between the crossmodal transfer of information and the crossmodal correspondences. Charles Spence has published over 1,100 academic articles and books, and he has been awarded numerous prizes for his work (e.g., the 10th Experimental Psychology Society Prize, the British Psychology Society: Cognitive Section Award, and the 2008 IG Nobel prize for nutrition). He established the Crossmodal Research Laboratory at the University of Oxford in 1997.

Artists, scientists, and many others have long been curious about the link between the senses, such as between sight and sound. One of the fundamental questions for many of those who are interested in this area is whether it is possible to translate between one sense and another. The majority of interest in this question has tended to revolve around whether it is possible to translate (or how best to transfer information) between vision and audition. In fact, ever since Isaac Newton, researchers and music practitioners/artists have been suggesting their own idiosyncratic mappings between hue and pitch.

Such crossmodal mappings have often been inspired by the phenomenon of synaesthesia. This is the name given to the idiosyncratic mappings (often between the senses; e.g., as in the case of coloured music synaesthesia) that are experienced by a small minority of the population, and may perhaps be overrepresented amongst artists (such as composers Rimski-Kórsakov and Scriabin). However, it is important to note that the nature of the synaesthete’s mapping between hue and pitch tends to be idiosyncratic. As such, a much more fruitful/robust basis for the crossmodal mapping, or transfer, of information between the senses is actually to base it on the crossmodal correspondences that are common to us all (or at least are shared by the majority of the population).

Crossmodal correspondences go by many different names in the literature, including crossmodal associations, crossmodal equivalences, synaesthetic correspondences, metaphorical crossmodal mappings… These terms are all used to refer to a shared mapping between basic features, attributes, or dimensions between the senses, either when physically presented, or else merely imagined. So, for example, most people tend to associate sweetness with roundness, and angularity with sourness/bitterness; similarly, they associate a pinkish-red food with a sweet taste, and famously, many commentators over the centuries have associated the colour scarlet with the sound of the trumpet.

Given that such crossmodal correspondences tend to be shared across the population, they would appear to represent a sensible basis for those wanting to transfer information from one sense to another, such as those interested in the development of sensory substation systems for the blind. However, one of the challenges for those working in this area is to figure out which of the many possible correspondences will be dominant in any given situation or context. So, for example, consider only how the majority of people will associate a high-pitched sound with a small, light, bright object situated high in space.

Another challenge highlighted by our recent research concerns the question of why so many of those interested in this area have chosen to map hue onto pitch1. In such cases, it would appear that the intuition is that dimensions of experience that are similarly structured must map naturally on to one another. So, for example, both colour and pitch can be represented as circular dimensions, and can potentially also be divided into the same number of basic categories. And yet, the fact that everyone who has put forward a crossmodal mapping between hue and pitch seems to have gone for a slightly different solution raises the question of whether this is really the most appropriate correspondence. Nowadays, some are starting to consider whether a more natural crossmodal mapping (or correspondence) for hue might be timbre instead. Resolving these questions will undoubtedly help those wishing to transfer information between the senses.

Charles now challenges and inspires how we taste and smell what we eat and drink:

Sensploration (FoST 2016)

Multisensory Experience and Coffee (Talks at Google)

Neurocuisine (The Guardian)

The Perfect Meal (RecoSymposium)

See our blog for Activities; especially 7-9.

_______________

1Spence, C., & Di Stefano, N. (2023). Sensory translation between audition and vision.  Psychonomic Bulletin & Review.