Author: tgraven

AI and the Aesthetic Enjoyment of Visual Art without Vision

tgraven/ / Art, Artificial Intelligence, Colour, Haptic touch, Hearing, Posts, Vision

Certain information, objects, and shapes are recognised through crossmodal correspondences. For example, when people who have just lost vision correctly identify cubes and squares of the same metal by touch alone. (See our blog for the scientific approach and the crossmodal correspondences between the senses,) But, what about the aesthetic enjoyment of perceiving visual art through audio-descriptions, tactile pictures, or both? (See our blog for Drawing pictures with and without vision, A Feel for Art, and On the intriguing association between sounds and colours.) New research suggests Artificial Intelligence can help improve aesthetic enjoyment through generating personalised descriptions: for example, of colours, shapes, and emotions and, thus, kindle people’s imagination and/or visual memories.

I invited the researchers behind Understanding Visual Arts Experiences of Blind People to shed some light on the aesthetic enjoyment of visual art without vision. Together, these researchers aim to understand better ways of supporting the experience of visual art of people who are blind. This blog post is written by Lotus Zhang, University of Washington, Franklin Mingzhe Li, Carnegie Mellon University, and Associate Professor Patrick Carrington, Human-Computer Interaction Institute, School of Computer Science, Carnegie Mellon University.

Art allows the expression of important ideas, emotions, and beliefs in a multitude of forms, and profoundly influences human society. However, most public art exhibits are experienced visually (e.g., photography, drawing, painting, sculpture) and thus pose access barriers for over 2.2 billion people in the world who have vision impairments. Although art museums and galleries increasingly offer accessible tours, these are still limited to a small number of venues and are far from comparably enjoyable to what is offered to sighted visitors.

In the quest for art appreciation, blind enthusiasts face a canvas of challenges. While guided tours offer a glimpse into the art world, they depend heavily on the descriptive skills of companions, often leading to a fragmented understanding of the artwork’s essence. Tactile graphics, though a bridge to the visual, are scarce and demand patience, turning a quick visit into a lengthy exploration. Smart devices promise a solution but falter, lacking the nuanced comprehension of the art’s depth. The digital realm offers remote tours, yet these can’t replicate the profound connection felt when standing before a masterpiece. As they navigate these barriers, blind patrons seek not just access, but a richer, more textured experience of art, where every shade and shape is felt, not just described.

For example, many blind patrons experience difficulties when sighted people describe form information of visual arts (e.g., shape, line, color), as we as a society do not share a standard for describing visual arts in accessible languages (e.g., vocabulary, grammar, visual references):

“I found sighted people always have difficulties explaining color and shape information in detail, which includes the shade of the color, contour of the objects.” (Female, 38, with congenital blindness. See Understanding Visual Arts Experiences of Blind People, section 6.5 Establish Shared Art Vocabulary and Grammar.)

Also, visual descriptions provided by sighted friends, family, and docents can be heavily subjective, making it difficult for blind individuals to form individual interpretations of the artwork:

“(…) I do not want to hear personal comments from people, just like this or that painting is so pretty and meaningful, all I need is what color they used, the contours of the lines, and what kinds of objects present in the painting.” (Female, 29, with acquired blindness at the age of six. See Understanding Visual Arts Experiences of Blind People, section 6.4 Enhance Objective Interpretation.)

Visual imagination and physical connection

For those who acquire blindness later in life, they are able to use their existing visual knowledge alongside conversation with sighted peers to use their imagination to envision and enjoy the artwork:

“I used to have vision when I was young, and I currently enjoy art by imagining from the information I know, such as people, activity, and the environment. I then think about what type of color they might use, or the facial expressions, I imagine everything that I am not told. The magic part is confirming my imagination with sighted friends or family members. And it is totally fine if I am wrong, I still like my imagination on how this artwork should be.” (Male, 25, with acquired blindness at the age of 20. See Understanding Visual Arts Experiences of Blind People, section 5.2 Cognition of Perceiving Visual Arts.)

In contrast, art enthusiasts who are congenitally blind establish their enjoyable experiences through tactile means; engaging with textures, shapes, figures, and paths. The experience of the same artwork through imagined visual details compared with tactile methods evokes different feelings as the reproduction in a more tactile format is fundamentally different from the original piece. This draws attention to personal experience (e.g., visual memory), motivation, as well as the presentation mode of visual arts as key factors for aesthetic enjoyment without vision.

AI augmenting existing descriptions

From the perspective of improving visual art access technologies, we envision significant changes from emerging AI development. For example, by generating vivid descriptions that engage all senses, generative AI can help blind people create mental images or sensations related to the artwork. Descriptions can include the imagined texture of brush strokes, the atmosphere that a scene depicts, or the emotions that the artwork is intended to evoke. Future art access technologies can also consider using generative AI to transform the description of artwork into a dynamic story, making the experience more immersive for blind individuals. Users can ask questions, and generative AI tools can adapt the narrative to focus on aspects that interest the user most, such as the symbolism behind certain elements or the techniques used by the artist. We encourage professionals to explore ways to utilize recent AI development and avoid potential harms.

Lotus, Franklin, and Patrick have also very kindly suggested some very interesting papers and books for us to read:

Asakawa, S., Guerreiro, J., Sato, D., Takagi, H., Ahmetovic, D., Gonzalez, D., Sato, D., Takagi, H., Ahmetovic, D., Gonzales, D., Kitani, K. M., & Asakawa, C. (2019). An independent and interactive museum experience for blind people. Proceedings of the 16th International Web for All Conference, May(30), 1-9.

Axel, E. S., & Levent, N. S. (2003). Art beyond sight: a resource guide to art, creativity, and visual impairment. New York: AFB Press.

Bernardi, R., Cakici, R., Elliott, D., Erdem, A., Erdem, E., Ikizler-Cinbis, N., Keller, F., Muscat, A., & Plank, B. (2016). Automatic description generation from images: A survey of models, datasets, and evaluation measures. Journal of Artificial Intelligence Research, 55(02), 409-442.

Bieber, R., & Rae, J. (2013). From the Mind’s Eye: Museum and Art Gallery Appreciation for the Blind–Canadian Perspectives. Disability Studies Quarterly, 33(3).

Hayhoe, S. (2013). Expanding our vision of museum education and perception: An analysis of three case studies of independent blind arts learners. Harvard Educational Review, 83(1), 67-86.

And some further listening and watching:

Harnessing the power of AI to make art accessible to all

“I hear colour” says colour blind artist with antenna on attached on his skull

Incredible art by visually impaired artists!

Will AI Create New Forms of Art for Blind People?

 

See our blog for Activities; especially 28-30.

Crossmodal brain plasticity and empowering of sensory abilities

tgraven/ / Crossmodal brain plasticity, Crossmodal correspondences, Hearing, Posts, Vision

Research on crossmodal brain plasticity has not only found that the brain compensates for sensory impairments. For example, so that people who are born blind process auditory information in both the auditory and the visual areas – not just the auditory like the fully sighted. (See our blog for the scientific approach.) It has also shown that the brain adapts to artificial input restoring the impaired senses and computer algorithms translating information from one sense to another. However, while people automatically recognise information processed through their natural crossmodal correspondences (see our blog for the crossmodal correspondences between the senses), they have to learn to interpret the sensations from both brain implants for hearing and sensory substitution devices from vision to hearing (see our blog for A Feel for Art.)

I have invited Carina Sabourin, Dr Yaser Merrikhi, and Professor Stephen G. Lomber, Cerebral Systems Laboratory, McGill University to write this blog post about crossmodal brain plasticity and empowering of sensory abilities. Carina Sabourin, Yaser Merrikhi, and Stephen G. Lomber investigate cortical plasticity in the auditory and visual cortices following hearing loss and the initiation of hearing with cochlear prosthetics. And, recently, in a comprehensive review study, they addressed the question “Do the blind hear better?”.

The idea that blind people can compensate for their lack of vision with enhanced hearing or other abilities, has been around for millennia. Many of the most acclaimed artists from the 8th century BC Greek poet, Homer, to the great jazz musician, Stevie Wonder, lost their vision. Recently, researchers have investigated these anecdotes and confirmed that there is now over-whelming evidence the blind have specific super hearing abilities compared to the sighted1. More excitingly, the brains of blind individuals recruit neural areas that typically handle vision to process auditory information along with hearing brain areas. The ability of typically visual areas to adapt to auditory input is called crossmodal plasticity. The extra brain power crossmodal plasticity provides gives blind individuals their superhuman hearing abilities.

Crossmodal plasticity can occur for other senses beyond hearing too. Deaf individuals recruit hearing brain areas to improve their vision2. Beyond other senses taking advantage of the freed-up brain power, brain plasticity can help the brain adapt to artificial input from brain implants restoring the lost sense. One example is cochlear implants which bypass the inner ear and directly stimulate the auditory nerve giving people with certain types of hearing loss access to sounds. Crossmodal plasticity is thought to help visual and hearing brain areas work together to better process speech. The more teamwork between visual and hearing brain areas, the better cochlear implant users can understand speech3. Similarly, researchers and engineers developing tools for blind people can leverage brain plasticity as well as the specific super hearing abilities of the blind.

One such attempt is sensory substitution devices (SSD) which translate information from one sensory modality into another. Audio-to-visual SSDs convert visual scenes captured by a camera into soundscapes. These devices exploit the improved pitch discrimination4-5 and sound localization abilities6-8 of blind people to convey information about visual environments as the frequency and movement of sounds. SSDs can even use the available brain space in the visual cortex. The part of the visual cortex that recognizes human bodies and tracks their movement was recruited to localize body movement conveyed by an SSD9. The visual reading brain area was even activated by SSDs to enable blind individuals to read with sounds10. Even years after getting their vision back, the visual cortex of individuals who sight was restored through gene therapy was still helping hearing brain areas process sounds11. Some concern exists that crossmodal plasticity may hinder sight restoration from visual brain implants. However, brain plasticity may help the visual and hearing brain areas work together to improve vision outcomes for visual brain implant users, just like it improved the ability of cochlear implant users to understand speech3.

The additional brain power provided by crossmodal plasticity empowers blind individuals with their extraordinary hearing abilities. Researchers and engineers creating tools for the blind can leverage both brain plasticity and their remarkable auditory skills to improve how blind individuals navigate and interact with the world around them.

See our blog for Activities; especially 22-24.

Some suggestions for further listening and watching:

Healing the brain via multisensory technologies and using these to better understand the brain

Losing and recovering sight

Neuroplasticity Animation

The Brain

What Does Blindness or Deafness Tell Us About Brain Development?

What is the function of auditory cortex when it develops in the absence of acoustic input?

 

_______________

1Sabourin, C. J., Merrikhi, Y., & Lomber, S. G. (2022). Do blind people hear better? Trends in Cognitive Sciences, 26(11), 999-1012. https://doi.org/10.1016/j.tics.2022.08.016

2Bavelier, D., Dye, M. W. G., & Hauser, P. C. (2006). Do deaf individuals see better? Trends in Cognitive Sciences, 10(11), 512-518. https://doi.org/10.1016/j.tics.2006.09.006

3Anderson, C. A., Wiggins, I. M., Kitterick, P. T., & Hartley, D. E. H. (2017). Adaptive benefit of cross-modal plasticity following cochlear implantation in deaf adults. Proceedings of the National Academy of Sciences of the United States of America, 114(38), 10256-10261. https://doi.org/10.1073/pnas.1704785114

4Collignon, O., Dormal, G., Albouy, G., Vandewalle, G., Voss, P., Phillips, C., & Lepore, F. (2013). Impact of blindness onset on the functional organization and the connectivity of the occipital cortex. Brain, 136(9), 2769-2783. https://doi.org/10.1093/brain/awt176

5Rokem, A., & Ahissar, M. (2009). Interactions of cognitive and auditory abilities in congenitally blind individuals. Neuropsychologia, 47(3), 843-848. https://doi.org/10.1016/j.neuropsychologia.2008.12.017

6Chen, Q., Zhang, M., & Zhou, X. (2006). Spatial and nonspatial peripheral auditory processing in congenitally blind people. NeuroReport, 17(13), 1449-1452. https://doi.org/10.1097/01.wnr.0000233103.51149.52

7Lewald, J. (2013). Exceptional ability of blind humans to hear sound motion: Implications for the emergence of auditory space. Neuropsychologia, 51(1), 181-186.

https://doi.org/10.1016/j.neuropsychologia.2012.11.017

8Röder, B., Teder-Salejarvi, W., Sterr, A., Rosler, F., Hillyard, S. A., & Neville, H. J. (1999). Improved auditory spatial tuning in blind humans. Nature, 400(6740), 163-166.

9Striem-Amit, E., & Amedi, A. (2014). Visual Cortex Extrastriate Body-Selective Area Activation in Congenitally Blind People “Seeing” by Using Sounds. Current Biology, 24(6), 687-692. https://doi.org/10.1016/j.cub.2014.02.010

10Striem-Amit, E., Cohen, L., Dehaene, S., & Amedi, A. (2012). Reading with Sounds: Sensory Substitution Selectively Activates the Visual Word Form Area in the Blind. Neuron, 76(3), 640-652. https://doi.org/10.1016/j.neuron.2012.08.026

11Mowad, T. G., Willett, A. E., Mahmoudian, M., Lipin, M., Heinecke, A., Maguire, A. M., Bennett, J., & Ashtari, M. (2020). Compensatory Cross-Modal Plasticity Persists After Sight Restoration. Frontiers in Neuroscience, 14(12 May). https://doi.org/10.3389/fnins.2020.00291