Category: Hearing

AI and the Aesthetic Enjoyment of Visual Art without Vision

tgraven/ / Art, Artificial Intelligence, Colour, Haptic touch, Hearing, Posts, Vision

Certain information, objects, and shapes are recognised through crossmodal correspondences. For example, when people who have just lost vision correctly identify cubes and squares of the same metal by touch alone. (See our blog for the scientific approach and the crossmodal correspondences between the senses,) But, what about the aesthetic enjoyment of perceiving visual art through audio-descriptions, tactile pictures, or both? (See our blog for Drawing pictures with and without vision, A Feel for Art, and On the intriguing association between sounds and colours.) New research suggests Artificial Intelligence can help improve aesthetic enjoyment through generating personalised descriptions: for example, of colours, shapes, and emotions and, thus, kindle people’s imagination and/or visual memories.

I invited the researchers behind Understanding Visual Arts Experiences of Blind People to shed some light on the aesthetic enjoyment of visual art without vision. Together, these researchers aim to understand better ways of supporting the experience of visual art of people who are blind. This blog post is written by Lotus Zhang, University of Washington, Franklin Mingzhe Li, Carnegie Mellon University, and Associate Professor Patrick Carrington, Human-Computer Interaction Institute, School of Computer Science, Carnegie Mellon University.

Art allows the expression of important ideas, emotions, and beliefs in a multitude of forms, and profoundly influences human society. However, most public art exhibits are experienced visually (e.g., photography, drawing, painting, sculpture) and thus pose access barriers for over 2.2 billion people in the world who have vision impairments. Although art museums and galleries increasingly offer accessible tours, these are still limited to a small number of venues and are far from comparably enjoyable to what is offered to sighted visitors.

In the quest for art appreciation, blind enthusiasts face a canvas of challenges. While guided tours offer a glimpse into the art world, they depend heavily on the descriptive skills of companions, often leading to a fragmented understanding of the artwork’s essence. Tactile graphics, though a bridge to the visual, are scarce and demand patience, turning a quick visit into a lengthy exploration. Smart devices promise a solution but falter, lacking the nuanced comprehension of the art’s depth. The digital realm offers remote tours, yet these can’t replicate the profound connection felt when standing before a masterpiece. As they navigate these barriers, blind patrons seek not just access, but a richer, more textured experience of art, where every shade and shape is felt, not just described.

For example, many blind patrons experience difficulties when sighted people describe form information of visual arts (e.g., shape, line, color), as we as a society do not share a standard for describing visual arts in accessible languages (e.g., vocabulary, grammar, visual references):

“I found sighted people always have difficulties explaining color and shape information in detail, which includes the shade of the color, contour of the objects.” (Female, 38, with congenital blindness. See Understanding Visual Arts Experiences of Blind People, section 6.5 Establish Shared Art Vocabulary and Grammar.)

Also, visual descriptions provided by sighted friends, family, and docents can be heavily subjective, making it difficult for blind individuals to form individual interpretations of the artwork:

“(…) I do not want to hear personal comments from people, just like this or that painting is so pretty and meaningful, all I need is what color they used, the contours of the lines, and what kinds of objects present in the painting.” (Female, 29, with acquired blindness at the age of six. See Understanding Visual Arts Experiences of Blind People, section 6.4 Enhance Objective Interpretation.)

Visual imagination and physical connection

For those who acquire blindness later in life, they are able to use their existing visual knowledge alongside conversation with sighted peers to use their imagination to envision and enjoy the artwork:

“I used to have vision when I was young, and I currently enjoy art by imagining from the information I know, such as people, activity, and the environment. I then think about what type of color they might use, or the facial expressions, I imagine everything that I am not told. The magic part is confirming my imagination with sighted friends or family members. And it is totally fine if I am wrong, I still like my imagination on how this artwork should be.” (Male, 25, with acquired blindness at the age of 20. See Understanding Visual Arts Experiences of Blind People, section 5.2 Cognition of Perceiving Visual Arts.)

In contrast, art enthusiasts who are congenitally blind establish their enjoyable experiences through tactile means; engaging with textures, shapes, figures, and paths. The experience of the same artwork through imagined visual details compared with tactile methods evokes different feelings as the reproduction in a more tactile format is fundamentally different from the original piece. This draws attention to personal experience (e.g., visual memory), motivation, as well as the presentation mode of visual arts as key factors for aesthetic enjoyment without vision.

AI augmenting existing descriptions

From the perspective of improving visual art access technologies, we envision significant changes from emerging AI development. For example, by generating vivid descriptions that engage all senses, generative AI can help blind people create mental images or sensations related to the artwork. Descriptions can include the imagined texture of brush strokes, the atmosphere that a scene depicts, or the emotions that the artwork is intended to evoke. Future art access technologies can also consider using generative AI to transform the description of artwork into a dynamic story, making the experience more immersive for blind individuals. Users can ask questions, and generative AI tools can adapt the narrative to focus on aspects that interest the user most, such as the symbolism behind certain elements or the techniques used by the artist. We encourage professionals to explore ways to utilize recent AI development and avoid potential harms.

Lotus, Franklin, and Patrick have also very kindly suggested some very interesting papers and books for us to read:

Asakawa, S., Guerreiro, J., Sato, D., Takagi, H., Ahmetovic, D., Gonzalez, D., Sato, D., Takagi, H., Ahmetovic, D., Gonzales, D., Kitani, K. M., & Asakawa, C. (2019). An independent and interactive museum experience for blind people. Proceedings of the 16th International Web for All Conference, May(30), 1-9.

Axel, E. S., & Levent, N. S. (2003). Art beyond sight: a resource guide to art, creativity, and visual impairment. New York: AFB Press.

Bernardi, R., Cakici, R., Elliott, D., Erdem, A., Erdem, E., Ikizler-Cinbis, N., Keller, F., Muscat, A., & Plank, B. (2016). Automatic description generation from images: A survey of models, datasets, and evaluation measures. Journal of Artificial Intelligence Research, 55(02), 409-442.

Bieber, R., & Rae, J. (2013). From the Mind’s Eye: Museum and Art Gallery Appreciation for the Blind–Canadian Perspectives. Disability Studies Quarterly, 33(3).

Hayhoe, S. (2013). Expanding our vision of museum education and perception: An analysis of three case studies of independent blind arts learners. Harvard Educational Review, 83(1), 67-86.

And some further listening and watching:

Harnessing the power of AI to make art accessible to all

“I hear colour” says colour blind artist with antenna on attached on his skull

Incredible art by visually impaired artists!

Will AI Create New Forms of Art for Blind People?

 

See our blog for Activities; especially 28-30.

On the intriguing association between sounds and colours

smackenz/ / Art, Colour, Crossmodal correspondences, Hearing, Music, Philosophical Approach, Posts, Vision

It seems three main types of crossmodal correspondences between the senses exist: transfer of information, shared associations, and subjective associations (see our blog for the crossmodal correspondences between the senses).

In this blog post, I have invited Researcher Nicola Di Stefano, Institute of Cognitive Sciences and Technologies, National Research Council of Italy to explain the subjective associations between music and colour. Nicola Di Stefano has contributed numerous publications on both the philosophy and psychology of perception and the aesthetics and psychology of music.

Sounds and colours are two distinct sensory experiences that convey different information about the environment we inhabit. While we typically attribute a colour to every object we perceive, we wouldn’t assert that each object possesses or is inherently associated with a particular sound. Of course, musical instruments produce sounds, and various objects can emit sounds, like hammers, rocks, and sticks, but sound seems to be an ontologically different, namely less foundational, feature of objects compared to colours.

Interestingly, however, intellectuals, researchers, artists, and composers have been long fascinated by the association between those two seemingly radically different sensory experiences. Their idiosyncratic association is evident in sound-colour synaesthesia, one of the most prevalent forms of synaesthesia, a rare neurological phenomenon where stimulation of one sensory (or cognitive) pathway leads to automatic, involuntary experiences in another1. This cross-wiring allows individuals with synaesthesia to experience a unique blending of sensations, such as seeing colours in response to musical notes or chords. Sound-colour synaesthesia has inspired several artworks, including the musical paintings by Kandinsky and Scriabin’s Prometheus, a composition based on the combination of coloured lights and music.

Coloured lights on a board in Scriabin's house

 

Photo retrieved from: Scriabin and the Possible

Psychologists have also explored the mechanisms underlying the consistent association between certain features of sounds and colours in non-synaesthetes. The concept of “crossmodal correspondence” suggests that certain sensory attributes share underlying perceptual or cognitive processes, leading to associations between them2. For instance, studies have revealed that people tend to associate high-pitched sounds with light or bright colours, while low-pitched sounds are often linked to dark colours3. These associations may arise from shared perceptual features, such as the frequency or intensity of auditory and visual stimuli.

One of the most intuitive ways to explain sound-colour correspondences is psychophysical, suggesting that both sounds and colours are vibratory phenomena. However, the sensory systems that process the two signals are quite different, making it challenging to establish a link between sounds and colours based solely on alleged psychophysical similarity. Additionally, an important distinction lies in the octave similarity in music, where sounds at different frequencies (integer multiples of the same fundamental frequency) share the same pitch class (e.g., “D”), whereas in the domain of colour, there is no equivalent octave repetition.

Furthermore, philosophers grapple with the metaphysical implications of the interplay between music and colour. Music, often described as the “language of the emotions” elicits powerful affective responses in listeners, shaping their emotional landscapes4-5. Similarly, colour possesses symbolic and emotional resonance, evoking mood and atmosphere in visual art and design. The intentional combination of music and colour in multimedia art forms, such as film and digital media, underscores the transformative potential of blending sensory modalities to create immersive experiences.

Whether through the lens of synaesthesia, crossmodal correspondence, or aesthetic inquiry, the convergence of music and colour illuminates the intricate interplay between sensory perception, cognition, and emotion. By unravelling the mysteries of this symbiotic relationship, researchers, artists, and practitioners aim to gain deeper insights into the nature of human experience and the profound ways in which art shapes our understanding of the world.

See our blog for Activities; especially 25-27.

Some suggestions for further listening and watching:

Artists use synesthesia to expand their creative limits

Elements of Music

Introduction to Color

Is Your Red The Same as My Red?

Light Organ (Clavière a lumiére) – Scriabin op 65 no 2

Seeing Sound: How Synesthesia Can Change Our Thinking

Seeing song through the ears of a synesthete

Synesthesia & creating your own score

 

_______________

1Ramachandran, V. S., & Hubbard, E. M. (2001). Synaesthesia–a window into perception, thought and language. Journal of consciousness studies, 8(12), 3-34.

2Spence, C. (2011). Crossmodal correspondences: A tutorial review. Attention, Perception, & Psychophysics, 73(4), 971-995.

3Spence, C., & Di Stefano, N. (2022). Coloured hearing, colour music, colour organs, and the search for perceptually meaningful correspondences between colour and sound. i-Perception, 13(3), https://doi.org/10.1177/20416695221092802

4Cooke, D. (1959). The language of music. London: OUP.

5Juslin, P. N., & Sloboda, J. (2011). Handbook of music and emotion: Theory, research, applications. Oxford: Oxford University Press.