Category: Crossmodal brain plasticity

Visual memories and sensory experiences

tgraven/ / Crossmodal brain plasticity, Crossmodal correspondences, Haptic touch, Hearing, Multisensory, Posts, Smell, Taste, Vision

It seems people who lose vision use information that transfers between the senses to retrieve visual memories. And that their visual memories decay when they have reached a certain level of experience in the other senses. (See our blog for the scientific approach, the crossmodal correspondences between the senses, and Decay and maintenance of sensory memories). But what happens when people who are born blind gain vision and then lose it again?

I have invited P.B. to share his experiences of visual memories and sensory information when going from blind to partially sighted and back to being blind again. P.B. was born blind with some light perception in one eye. After surgery and other medical treatments, he had gained about 7% vision in the other eye at three and a half years old. P.B. does not remember this shift from being blind to being partially sighted. Then, in his early twenties, P.B. had an accident that rendered him totally blind in both eyes. P.B. thinks of himself as a sighted person who happens not to see. He visualises when making plans. And his dreams are always visual. P.B. approved this text before we posted it on our blog.

P.B. has several visual memories from when he was little. From the house he grew up in, for example, he remembers the light from a chandelier hanging over the coffee table in the living room, the patterned wallpaper in his room, and the countries on the world map hanging on the wall. And he remembers the colours on the outside walls, the doors, and the window frames.

Today, when he has no new visual input, P.B. creates visual images based on his memories and knowledge. For example, that of a White Swiss Shepherd Dog working as a guide dog from his memories of both the white colour and the German Shepherd Dog. As well as his knowledge of guide dogs.

His visual memories are typically triggered by somebody describing how something looks, like a bright red sunset – not by information from the other senses.

P.B.’s visual memories “flash up almost like the flavour when eating or drinking”.

He has to decide not to focus on visual memories and images when talking to people: the memories are now 20 years old and people have changed. And his created visual images of what new people look like may be very wrong.

P.B. has to actively suppress his urge to retrieve visual memories or create visual images of people based on their voices.

He describes not having visual memories and images of spatial relations and distances. P.B. rather remembers them through his body. He walks around in the city centre with no vision, a podcast or some music in his ears, and shoes with a thick curved sole (which makes it difficult to feel the surfaces on pavements, streets, steps, floors, etc.). He does not count steps or any of that and pauses the podcast or music in his ears only when he feels he might get lost.

For P.B., sensory experiences include a variety of simultaneous sensations as well as visual memories and created images – they are multisensory. The forest, for example, is the sound of wind and trees, the texture of surfaces under his feet, scents, and the memories of colours. He zooms in on whatever attracts his attention: indeed, not on one piece of sensory information after the other in a certain order. For example, zooming in on a sudden shift from soft to hard texture under his feet.

An interesting sensory experience, according to P.B., is that of new flavours, the texture of the food, and the sound of crunch when chewing, He describes a cloudberry cream dessert as “not very sweet, but also not sour, light orange colour, and creamy texture”. But a redcurrant jelly only as “something wobbling on the plate”. He does not describe the flavour, scent, and/or feel in his mouth – merely how the jelly looks on the plate.

Was he not able to suppress his visual memories of jelly – did they take over?

When P.B. is sensory tired, he relaxes his senses by listening to “concentration music, like opera or prog rock where I can immerse myself and get absorbed by a different universe”.

It seems P.B. can maintain his visual memories and ability to create visual images through people’s descriptions of what something looks like. But, at the same time, these memories and images are distracting him from focusing on sensory information that is more relevant to him today with no vision, for example, people’s voices and the feel of wobbling jelly in his mouth, So is this, in fact, a negative spiral – that maintaining visual memories prevents experiences in the other senses, which in turn helps nurture the visual memories?

See our blog for Activities; especially 49-51.

Using the senses when vision and hearing are impaired

tgraven/ / Crossmodal brain plasticity, Crossmodal correspondences, Haptic touch, Hearing, Multisensory, Posts, Vision

The brain integrates simultaneous information from several senses. It transfers information from one sense to another. And, it reorganises itself when one of the senses is impaired. All of this improves our reaction time, precision, and recognition accuracy (see our blog for Multisensory processing, crossmodal correspondences between the senses, and crossmodal brain plasticity and empowering of sensory abilities). But what happens when more senses are impaired?

I have invited K.H. to share her experiences of living with deafblindness. K.H. was born with about 10% vision – a severe visual impairment. She has had periods of losing and regaining vision: currently with about 0.5% visual acuity, which corresponds to being blind. In addition, K.H. was born with moderate to severe hearing loss. Her hearing loss is most severe in the middle of the pitch range, which is where people’s voices typically lie when talking. She struggles to recognise voices, where a sound comes from, and how far away they are. K.H. approved this text before we posted it on our blog.

So, how does K.H. use her senses on their own and together?

“It’s like working on a jigsaw puzzle where a lot of the pieces are missing: I have to search for as many pieces as I possibly can using vision, hearing, and touch and, then, piece them all together to get the full picture. I have to guess a lot, which is exhausting, and I often get it wrong because I’ve not recognised or found a piece. I’m constantly in a state of high alert and fear of making a fool of myself.”

K.H. describes searching for specific sensory information, that what she searches for is determined by what she is doing, and where one sense fails other senses step in. In traffic, for example, she describes using vision alone and hearing alone to recognise separate landmarks, like a red house by vision and an open area by hearing. She uses hearing and vision to determine if cars and people are moving, and in what direction – first hearing, then vision to double-check. K.H. describes using hearing alone to judge the distance to walls, other people, etc., and her feet to check tactile information about kerbs and steps. And both hearing and touch when crossing the street – listening for the sound on a signal-controlled crossing first, then touching the spinning cone to double-check.

I have become experienced in knowing what sensory information I need, for example, from vision to support hearing.

Is K.H. ever not in a state of high alert?

She describes dropping her guard when she is home alone because nobody might talk to her, show her something, etc. Another example is at a concert together with a guide she trusts and who is also skilled in using social haptic communication. Certainly, she takes every opportunity to have a “sense break”: relaxing her vision either by playing a game on her mobile phone with easy-to-see colours and lots of repetitions or by lying down in a dark room. She relaxes her hearing by listening to music (the melody, not the lyrics) or a podcast that she does not have to pay attention to. K.H. sometimes relaxes both senses and sometimes one sense while focusing on the other; for example, relaxing vision by playing a game on her mobile phone or touching the texture of her clothes while focusing her hearing on the lecturer.

I feel I hear them better when vision doesn’t have to work so hard or is completely “closed off”.

Also, physical exercise in non-demanding sensory environments gives her brain some time off.

Does K.H. enjoy a sensory experience – do they give her pleasure?

K.H. describes enjoying birdsong and music. And bright colours, like the blue sky or sea, green trees or hills, and the reds and yellows in a sunset or flower: especially, in combination with the smell and sound of the sea or forest. But she has to be either with a guide or on her own and standing still or sitting down. As soon as she knows somebody other than her guide might seek her attention or she starts moving, she goes right back to high alert mode.

It seems K.H. has a plan for what to search for both in the preferred and the supplementing sense, often used to double-check the correctness of the favoured sense. K.H. does not describe focusing on any of the information that research has found to transfer from one sense to another, like shape. Perhaps she cannot see or hear it? It is almost as if K.H. perceives the world linearly – in a string of well-organised sensory information. Only when she is enjoying a sensory experience, does K.H. describe appreciating a scene of multisensory information, for example, the birdsong, colour, and smell of the forest.

See our blog for Activities; especially 37-39.