Category: Haptic touch

AI and the Aesthetic Enjoyment of Visual Art without Vision

tgraven/ / Art, Artificial Intelligence, Colour, Haptic touch, Hearing, Posts, Vision

Certain information, objects, and shapes are recognised through crossmodal correspondences. For example, when people who have just lost vision correctly identify cubes and squares of the same metal by touch alone. (See our blog for the scientific approach and the crossmodal correspondences between the senses,) But, what about the aesthetic enjoyment of perceiving visual art through audio-descriptions, tactile pictures, or both? (See our blog for Drawing pictures with and without vision, A Feel for Art, and On the intriguing association between sounds and colours.) New research suggests Artificial Intelligence can help improve aesthetic enjoyment through generating personalised descriptions: for example, of colours, shapes, and emotions and, thus, kindle people’s imagination and/or visual memories.

I invited the researchers behind Understanding Visual Arts Experiences of Blind People to shed some light on the aesthetic enjoyment of visual art without vision. Together, these researchers aim to understand better ways of supporting the experience of visual art of people who are blind. This blog post is written by Lotus Zhang, University of Washington, Franklin Mingzhe Li, Carnegie Mellon University, and Associate Professor Patrick Carrington, Human-Computer Interaction Institute, School of Computer Science, Carnegie Mellon University.

Art allows the expression of important ideas, emotions, and beliefs in a multitude of forms, and profoundly influences human society. However, most public art exhibits are experienced visually (e.g., photography, drawing, painting, sculpture) and thus pose access barriers for over 2.2 billion people in the world who have vision impairments. Although art museums and galleries increasingly offer accessible tours, these are still limited to a small number of venues and are far from comparably enjoyable to what is offered to sighted visitors.

In the quest for art appreciation, blind enthusiasts face a canvas of challenges. While guided tours offer a glimpse into the art world, they depend heavily on the descriptive skills of companions, often leading to a fragmented understanding of the artwork’s essence. Tactile graphics, though a bridge to the visual, are scarce and demand patience, turning a quick visit into a lengthy exploration. Smart devices promise a solution but falter, lacking the nuanced comprehension of the art’s depth. The digital realm offers remote tours, yet these can’t replicate the profound connection felt when standing before a masterpiece. As they navigate these barriers, blind patrons seek not just access, but a richer, more textured experience of art, where every shade and shape is felt, not just described.

For example, many blind patrons experience difficulties when sighted people describe form information of visual arts (e.g., shape, line, color), as we as a society do not share a standard for describing visual arts in accessible languages (e.g., vocabulary, grammar, visual references):

“I found sighted people always have difficulties explaining color and shape information in detail, which includes the shade of the color, contour of the objects.” (Female, 38, with congenital blindness. See Understanding Visual Arts Experiences of Blind People, section 6.5 Establish Shared Art Vocabulary and Grammar.)

Also, visual descriptions provided by sighted friends, family, and docents can be heavily subjective, making it difficult for blind individuals to form individual interpretations of the artwork:

“(…) I do not want to hear personal comments from people, just like this or that painting is so pretty and meaningful, all I need is what color they used, the contours of the lines, and what kinds of objects present in the painting.” (Female, 29, with acquired blindness at the age of six. See Understanding Visual Arts Experiences of Blind People, section 6.4 Enhance Objective Interpretation.)

Visual imagination and physical connection

For those who acquire blindness later in life, they are able to use their existing visual knowledge alongside conversation with sighted peers to use their imagination to envision and enjoy the artwork:

“I used to have vision when I was young, and I currently enjoy art by imagining from the information I know, such as people, activity, and the environment. I then think about what type of color they might use, or the facial expressions, I imagine everything that I am not told. The magic part is confirming my imagination with sighted friends or family members. And it is totally fine if I am wrong, I still like my imagination on how this artwork should be.” (Male, 25, with acquired blindness at the age of 20. See Understanding Visual Arts Experiences of Blind People, section 5.2 Cognition of Perceiving Visual Arts.)

In contrast, art enthusiasts who are congenitally blind establish their enjoyable experiences through tactile means; engaging with textures, shapes, figures, and paths. The experience of the same artwork through imagined visual details compared with tactile methods evokes different feelings as the reproduction in a more tactile format is fundamentally different from the original piece. This draws attention to personal experience (e.g., visual memory), motivation, as well as the presentation mode of visual arts as key factors for aesthetic enjoyment without vision.

AI augmenting existing descriptions

From the perspective of improving visual art access technologies, we envision significant changes from emerging AI development. For example, by generating vivid descriptions that engage all senses, generative AI can help blind people create mental images or sensations related to the artwork. Descriptions can include the imagined texture of brush strokes, the atmosphere that a scene depicts, or the emotions that the artwork is intended to evoke. Future art access technologies can also consider using generative AI to transform the description of artwork into a dynamic story, making the experience more immersive for blind individuals. Users can ask questions, and generative AI tools can adapt the narrative to focus on aspects that interest the user most, such as the symbolism behind certain elements or the techniques used by the artist. We encourage professionals to explore ways to utilize recent AI development and avoid potential harms.

Lotus, Franklin, and Patrick have also very kindly suggested some very interesting papers and books for us to read:

Asakawa, S., Guerreiro, J., Sato, D., Takagi, H., Ahmetovic, D., Gonzalez, D., Sato, D., Takagi, H., Ahmetovic, D., Gonzales, D., Kitani, K. M., & Asakawa, C. (2019). An independent and interactive museum experience for blind people. Proceedings of the 16th International Web for All Conference, May(30), 1-9.

Axel, E. S., & Levent, N. S. (2003). Art beyond sight: a resource guide to art, creativity, and visual impairment. New York: AFB Press.

Bernardi, R., Cakici, R., Elliott, D., Erdem, A., Erdem, E., Ikizler-Cinbis, N., Keller, F., Muscat, A., & Plank, B. (2016). Automatic description generation from images: A survey of models, datasets, and evaluation measures. Journal of Artificial Intelligence Research, 55(02), 409-442.

Bieber, R., & Rae, J. (2013). From the Mind’s Eye: Museum and Art Gallery Appreciation for the Blind–Canadian Perspectives. Disability Studies Quarterly, 33(3).

Hayhoe, S. (2013). Expanding our vision of museum education and perception: An analysis of three case studies of independent blind arts learners. Harvard Educational Review, 83(1), 67-86.

And some further listening and watching:

Harnessing the power of AI to make art accessible to all

“I hear colour” says colour blind artist with antenna on attached on his skull

Incredible art by visually impaired artists!

Will AI Create New Forms of Art for Blind People?

 

See our blog for Activities; especially 28-30.

A Feel for Art

tgraven/ / Art, Colour, Haptic touch, Posts, Vision

There is more to touching art than merely using the hands and fingers to recognise depictions of landscapes and victorious kings on horses; ornate chairs and tables with carved and inlaid decorations; geometric shapes and stylised forms of flowers; figurative and abstract fruits and animals. (See our blog on Vision, haptic touch, and hearing.)

In this blog post, I have invited Professor Georgina Kleege, University of California, Berkeley to write about touching art: why she likes to do it, how she does it, and why she thinks others would get something out of the experience. Georgina Kleege has published numerous books and papers, and been awarded the Distinguished Teaching Award twice. She is an art lover and is well known for advocating for more tactile opportunities for all museum visitors, including those who are fully sighted.

As a blind person, I enjoy the incredible privilege of touch access to art works at museums around the world who offer such opportunities. Over the years I have accumulated many observations and insights that inspire me to dispel common myths and misconceptions about touch perception, specifically in an artistic context. For one thing, I find that the notion of sensory translation or substitution can be misleading. People seem to want to make an analogy between the two eyes of the sighted and the two hands of the blind, as if simply laying a hand on the art object will summon a detailed image to the blind person’s mind’s eye. There’s a mistaken impression that the point of touching the art object is merely to determine what it’s representing visually—what it looks like to people who can see. This implies that sighted people don’t need to touch the art because they can see it, when in fact, determining what the art object might be depicting is not always, I’d say rarely, the most interesting thing about the experience.

When touch is merely an exercise to identify a shape, one uses a minimum of the tactile apparatus available. One traces the outline of the object in a manner analogous to the way the visual system separates the object from its background. One uses mainly the fingertips which also deliver some information about the surface texture and temperature. They can also discover fine details in carving, or the seams, joints and welds that hold the thing together, and even signs of past damage and repairs which may not be available to the eyes alone. But to grasp the objects three dimensionality one must grasp: wrap one’s fingers and palms around volumes, drape the whole hand around contours. The action of the hands and the skin of the palms delivers more information. The movement of the hands inspires other movement, of the whole arm, of the spine, as one reaches, stretches, bends and extends to take in the form in its entirety. There’s no point sticking to one place—the vantage point for sighted people. One is better off moving around doing what I have come to call dancing with the sculpture, circumnavigating the object while maintaining light contact with one hand. This action can convey a sense of composition, of symmetry and dynamism. These techniques work equally well whether the sculpture is figurative or abstract.

I used to avoid referencing my emotional response to touching art. English, like other languages, conflates touch sensation with the emotions. We find a work of art touching and it makes us feel happy or sad. This conflation is problematic for blind people who rely more on touch than our sighted peers because touch is often considered to be a lesser way of knowing the world, more animalistic or infantile. Babies rely on touch before their visual perception fully develops. But now I lean into this connection. I actively scan my emotions for a response to what I’m touching. Touch can be intuitional; the sculpture tells me how to touch it. It reveals aspects of itself sequentially, as I experiment with different methods and repeat or reject actions according to what feels most generative. It is an accretive process requiring attention, sensitivity sometimes even playfulness. And the emotions this touching summons in me may have something to do with what the artist hoped to convey. If nothing else, my hands and body replicate the artist’s own gestures and movements which in turn may link me to the ideas and emotions that went into the sculpture’s creation.

Recently, I’ve been advocating for more tactile opportunities for all museum visitors, including sighted people who are typically excluded from this form of access. For now, conservation and crowd management concerns make it unlikely that everyone will be allowed to get their hands on art. So instead, I endeavor to describe my experiences, as here, in the hope that sighted art lovers can profit, if only vicariously. (See also Nydia, the Blind Flower Girl of Pompeii.)

See our blog for Activities; especially 19-21.

Some suggestions for further listening, reading, and watching:

A Conversation about Blindness and Art

Art Beyond Sight

Best Things To Touch As A Blind Person

Do Touch The Artwork At Prado’s Exhibit For the Blind

Some Touching Thoughts and wishful Thinking

The Gravity, The Levity: Let Us Speak of Tactile Encounters

Touch and See

Worst Things To Touch As A Blind Person