Author: tgraven

Sensory mismatch

tgraven/ / Hearing, Posts, Sensory mismatch, Vision

Even though S.B. had regained his vision, after more than fifty years of blindness, he always closed his eyes in traffic. He continued to function as if he was still blind.

But why? Does information about traffic not transfer from one sense to another? (See our blog for the scientific approach and the crossmodal correspondences between the senses.)

Without vision, S.B. would be used to listening for human activity and unique spatial characteristics when identifying an area, such as conversations about fish and the openness of the sea. He would be helped by echoes of his own sounds and sounds that stand out from the background, like a fan in a quiet street. (See how The Oxford Sound Album started.) S.B. would have listened for contrasting sounds. And enjoy echoes in narrow places; Walking up Queen’s Lane and New College Lane, constantly varying soft sounds; on Radcliffe Square on an early summer morning, and quiet but informative places; Walking inside the University Church of St. Mary the Virgin. Nonetheless, he may have been inspired by the chaotic sounds while Walking through Clarendon Centre. Rain and snow mask information: making it hard, for example, to determine the speed of cars and the distance to a building. To be able to navigate in chaotic and masked sounds, S.B. would have had to tap harder with his white cane to create a better echo against walls, bushes, and poles at pedestrian crossings. But the rain would also help him hear silent objects, like parked bikes and street furniture.

S.B. was experienced in listening for, paying attention to, and intervening with sounds in traffic, and he would have added tactile information too (typically from his white cane, face, and feet) about surfaces, obstructions, and gradients. So, what changes with vision?

Had S.B. started using vision alone, then he would constantly be looking for shapes, for example, in the city skyline, colours, and textures on the Royal Mile. He would have enjoyed how the sunlight enhanced and changed the shapes. As well as three-dimensional architecture and movement, like bay windows and passersby. He would have received information about human activities and danger from signs only.

Indeed, using vision alone, S.B. would be looking for and paying attention to some of the same information about traffic as when he was blind, namely shapes and surfaces/textures; however, not intervene with it. So, again, why did S.B. after having regained vision continue to function in traffic as when he was blind?

There seem to be two possible explanations. One is that when people combine vision and hearing, paying attention to visual shapes reduces their attention to sound and vice versa. S.B. then, who was so very experienced in paying attention to the sounds in traffic, would not have focused on any visual information regardless of whether his eyes were open or closed. Perhaps not even knowing they were closed. Or perhaps getting so confused by the visual information that he closed them. The other explanation is that when people combine visual and auditory information, they tend not to pay attention to either unless the two match. For S.B. a match may never have been possible: he was too experienced in hearing.

Now challenging and inspiring us to use hearing alone, vision alone, and both senses together:

Hearing Princes Street – St David Street

Seeing Princes Street – St David Street

Hearing and seeing Princes Street – St David Street

 

Hearing inside a coffee shop

Seeing inside a coffee shop

Hearing and seeing inside a coffee shop

 

Hearing Princes Street

Seeing Princes Street

Hearing and seeing Princes Street

 

Hearing Victoria Street

Seeing Victoria Street

Hearing and seeing Victoria Street

See our blog for Activities; especially 13-15.

         Some suggestions for further listening and reading:

Herssens, J., Roelants, L., Rychtáriková, M., & Heylighen, A. (2011). Listening in the Absence of Sight: The Sound of Inclusive Environments. Proceedings of Include 2011, 10-17.

Sabourine, C. J., Merrikhi, Y., & Lomber, S. G. (2022). Do blind people hear better? Trends in Cognitive Sciences, 26(11), 999-1012.

Schafer, R. M. (1994). The Soundscape: Our Sonic Environment and the Tuning of the World. Rochester, Vermont: Destiny Books.

Southworth, M. (1969). The Sonic Environment of Cities. Environment and Behavior, 1(1), 49-70.

Stein, B. E., & Meredith, M. A. (1993). The merging of the senses. Cambridge: The MIT Press.

The Oxford Sound Album

The World Soundscape Project

 

Vision, haptic touch, and hearing

tgraven/ / Haptic touch, Hearing, Posts, Vision

S.B. after more than fifty years of blindness regained his eyesight. When he recognised simple shapes and ordinary objects as well as their size by vision, he was helped by his tactile memories. Indeed, memories based on one sense influence the perception of another. (See our blog for the scientific approach and the crossmodal correspondences between the senses.)

S.B. matched his new visual information to his memories, stored when he was still blind. When vision is absent, memories about simple shapes and ordinary objects, including size, are based mainly on tactile and auditory information.

Vision perceives a lot of different information simultaneously and it fills in any missing information. It starts with an overview and, then, focuses on details. Visual attention is drawn to colours that stand out from the background, sudden movements, and any changes. Vision perceives information about colour, location, movement, quantity, shape, size, and texture.

Haptic touch – the combination of touch and movement – perceives information serially, piece by piece by piece. It starts with the details and, then, builds up an understanding of the whole information (e.g., a picture, illustration, or scene). Attention seems to be drawn to differences in surfaces and textures. Haptic touch perceives information about hardness, location, movement, quantity, shape, size, texture, and weight1.

Hearing perceives a lot of different information simultaneously. It starts with an overview and, then, focuses on details. Auditory attention is drawn to sudden sounds that stand out from the background. Hearing perceives information about location, movement, quantity, shape, size, sound, and texture.

Vision, haptic touch, and hearing perceive much of the same information, but S.B. was still not able to recognise visual information that he had not experienced by haptic touch, hearing, or both.

Very little information is perceived by merely one of the senses: colour by vision, hardness and weight by haptic touch, and sound by hearing. And correspondences exist between this sensory-specific information, for example, hue and pitch (or timbre). Shape is an example of information that is perceived by all three senses. It transfers between them, and when they work together all sensory information is integrated to form or retrieve memories of it.

Only when the visual information matched well with his memories, based mainly on tactile and auditory information, did S.B. recognise the world around him. It seems the match is best with information that is perceived by two (or all) senses: S.B. recognised simple shapes and ordinary objects as well as their size.

The brain adapts rapidly when using a different sense, but it needs time to permanently change how it processes the perceived information. During this phase, it would be beneficial to use the senses for perceiving information that they are naturally attentive to and therefore familiar with already, for example, texture for haptic touch. Research has found that it also helps to focus on information that transfers between the senses, for example, shape. Both would serve as warm-up activities for the brain, preparing it for more complex information to come, such as traffic. To hear a series of favourite city soundscapes, please listen to “The Oxford Sound Album“.

See our blog Activities; especially 10-12.

_______________

1The sense of touch, without movement, perceives information about temperature, pain, and pleasure.