Category: Crossmodal correspondences

Drawing pictures with and without vision

tgraven/ / Art, Colour, Crossmodal correspondences, Drawing, Haptic touch, Posts, Vision

Soon after S.B. had regained vision, after more than five decades of blindness, his wife gave him his first ballpoint pen. He wrote his name and drew a cobbler’s chipping hammer. Whether drawing his own house from memory, a bus he had just seen, or a farmhouse from imagination; S.B. incorporated features that he knew from haptic touch. For example, cartwheels on the bus instead of hub wheels: S.B. knew the feel of cartwheels, not of hub wheels. However, six months later, and with gained visual experience, S.B. replaced his cartwheels. (See Gregory & Wallace, 1969, Figures 9-14.) Surely, S.B. transferred information from haptic touch to vision when he was drawing pictures: combining circles, rectangles, and squares in different sizes. (See our blog for the scientific approach and the crossmodal correspondences between the senses).

I have invited Professor John M. Kennedy, University of Toronto to write this blog post. John M. Kennedy has written numerous books and articles on pictures in vision and touch: he argues that lines are the “nuts and bolts of universals in perception”.  

But first, John challenges and inspires us to try drawing without vision.

 

Try this in touch: Drawing

– Perhaps the ice age people who made the very first drawings said “We didn’t know we could do this!”

– For sure, some blind people I asked to draw for me said “I don’t think I can do this.”

– I said, in effect, “Take up thy pen and draw.” And they did. And said “I didn’t know I could do this!”

They were drawing on plastic sheets with ball point pens. The sheets rested on rubber-coated pads, like mouse pads. The pens made rough lines they could feel.

They drew hands, cups, wine glasses and chairs. All were tactile pictures, made of raised lines.

Ambitious people drew a man from in front, an insect from above and a dog from the side. Just to be clear – these were tactile pictures.

Let’s consider a challenge. Imagine taking up the ball-point and the pad, and, without any vision whatsoever, try this just purely from your mind’s own vantage point.

– Draw a singer pouring out love and radiating glory.

– Draw a monster that has emerged from a lake, horns and claws and spiny backbone dripping wet.

– Draw a cup so filled with memories it bubbles and pops and spills out irrepressibly.

– Draw a car with its brakes pulling it screeching to a halt so abruptly the tyres deform.

If your imagination succeeds at some or all of these drawing topics, then know this. Blind adults and children who grew up without drawing and took to sketching later in life, and sometimes only shortly before tackling these imaginary scenes, invented pictures – tactile pictures — that fit the bill. (See Kennedy, 2014, Figure 2 – singer pouring out love and radiating glory – and Figure 1 – cups so filled with memories it bobbles and pops and spills out irrepressibly.)

It is only recently that we realized pictures are not just for vision and the people who are fully sighted, as tactile pictures they are also for touch and for those who were born totally blind. Drawing is for the eye and the hand.

 

Learning lines: Shape and outline

Skill at line drawing develops in pretty similar ways in people who are sighted and people who are blind. Lines are tangible as well as visual of course, but so too are the blocks and creatures we like to draw. We can feel a cat or a dog or a hamster. At first, we have to tell people what we have sketched, but clear unmistakeable shapes do emerge eventually if we keep at it. Consider the path drawing development usually takes for the hand and the eye alike.

At first, drawing novices, feeling their work or looking at it, often learn to make tactile and optical marks by scribbling, and then they may very shortly begin controlling straight and curved lines and enclosures such as circles. Perhaps soon after, scenes with flat objects like hands or knives, or long slim sticks like bananas and bulky objects like oranges and heads and houses become favoured topics. The timeline between scribbles and recognizable forms for young children – a year or two or three — is highly compressed for older beginners – sometimes mere minutes or an hour perhaps.

Soon after mastering the basics of scribbling, blind and sighted novices put to work some lines actually representing something. A complete beginner may simply make a random mark and announce what it stands for: “It’s Grandpa!”

According to John Willats, circles are drawn to mean solid objects three-dimensional like balls and boxes, roughly symmetrically extended in space. Snakes and rods and branches that are long and slim are what fairly straight lines mean to this novice.

Next in development, once the shape of objects like books and chocolate bars can be shown, things mostly extended in two dimensions, squares and rectangles and triangles and the like are used. A cube would be shown as a square, a single face, by these early artists.

Faces on the sides of objects begin to be added next. Learning to pick a side that faces the observer brings the issue of the vantage point on the object to mind. This means perspective, and questions of many kinds, solved in lots of ways by eager minds.

Outline drawing is a skill that ice-age artists manifested. We usually think of these as visual works. But lines in outline drawings mean the same thing to the blind as they do to the sighted. Drawings made in the flickering half-light of caves may have meant a great deal to the hands of the ancient art workers, not just to the eyes. Lines are tools to make shapes. The toolkit of outlines is tiny. Ice age artists discovered the kit. Its results are infinite. Its combinations picture the many, many objects of our world.

Ice-age artists drew lines standing for borders of surfaces. They showed occlusion and foreground surfaces ending with a background. Aboriginal artists in Australia used lines similarly. Outline is universally understood. It is a birthright of our species, and others besides.

The birthright of humanity is outline in vision, but it is also outline in touch. Lines in outline drawings mean the same edges to the eye and the hand, the psychology of vision and the psychology of touch. We discovered their powers and capabilities in ice-age times for vision, it may be, but in this generation we are uncovering their functions are also just as deliberate in the psychology of touch.

John has also very kindly suggested some useful readings for us:

Carboni, S., & Kennedy, J. M. (2020). Eyes outside a boundary line: an example of the Willats region-drawing theory? Perception, 49(7), 793-795, https://doi.org/10.1177/0301006620929473

Eriksson, Y. (1998). Tactile pictures. Lund: Lund University Press.

Fabris, M., Lange-Kuettner, C., Shiakou, M., & Longobardi, C. (2023). Editorial: Children’s drawings: evidence-based research and practice. Frontiers of Psychology. Section Educational Psychology, 14, https://doi.org/10.3389/fpsyg.2023.1250556

Kennedy, J. M. (1993). Drawing and the blind: Pictures to touch. New Haven: Yale Press.

Rubin, E. (1915). Visually experienced figures [Synsoplevede figurer]. Copenhagen: Gyldendal, Nordisk Forlag.

Willats, J. (2005). Making sense of children’s drawings. London: Erlbaum.

See our blog for Activities; especially 16-18.

Crossmodal correspondences between the senses

tgraven/ / Crossmodal correspondences

As with the case of S.B., most people transfer information between their sense of touch and vision. And they associate certain information across the senses too, for example, a rounded visual shape with the spoken word “Bouba” and an angular visual shape with the spoken word “Kiki”. (See our blog for the scientific approach.) Moreover, some subjective crossmodal correspondences also exist, as with a friend who associates the floral fragrance of her favourite perfume with the colour of red wine and the feel of velvet fabrics. New research has, however, started to question the appropriateness of some of the previously accepted crossmodal correspondences, like that between hue and pitch.

In this blog post, I have asked Professor Charles Spence, University of Oxford to explain the link between the crossmodal transfer of information and the crossmodal correspondences. Charles Spence has published over 1,100 academic articles and books, and he has been awarded numerous prizes for his work (e.g., the 10th Experimental Psychology Society Prize, the British Psychology Society: Cognitive Section Award, and the 2008 IG Nobel prize for nutrition). He established the Crossmodal Research Laboratory at the University of Oxford in 1997.

Artists, scientists, and many others have long been curious about the link between the senses, such as between sight and sound. One of the fundamental questions for many of those who are interested in this area is whether it is possible to translate between one sense and another. The majority of interest in this question has tended to revolve around whether it is possible to translate (or how best to transfer information) between vision and audition. In fact, ever since Isaac Newton, researchers and music practitioners/artists have been suggesting their own idiosyncratic mappings between hue and pitch.

Such crossmodal mappings have often been inspired by the phenomenon of synaesthesia. This is the name given to the idiosyncratic mappings (often between the senses; e.g., as in the case of coloured music synaesthesia) that are experienced by a small minority of the population, and may perhaps be overrepresented amongst artists (such as composers Rimski-Kórsakov and Scriabin). However, it is important to note that the nature of the synaesthete’s mapping between hue and pitch tends to be idiosyncratic. As such, a much more fruitful/robust basis for the crossmodal mapping, or transfer, of information between the senses is actually to base it on the crossmodal correspondences that are common to us all (or at least are shared by the majority of the population).

Crossmodal correspondences go by many different names in the literature, including crossmodal associations, crossmodal equivalences, synaesthetic correspondences, metaphorical crossmodal mappings… These terms are all used to refer to a shared mapping between basic features, attributes, or dimensions between the senses, either when physically presented, or else merely imagined. So, for example, most people tend to associate sweetness with roundness, and angularity with sourness/bitterness; similarly, they associate a pinkish-red food with a sweet taste, and famously, many commentators over the centuries have associated the colour scarlet with the sound of the trumpet.

Given that such crossmodal correspondences tend to be shared across the population, they would appear to represent a sensible basis for those wanting to transfer information from one sense to another, such as those interested in the development of sensory substation systems for the blind. However, one of the challenges for those working in this area is to figure out which of the many possible correspondences will be dominant in any given situation or context. So, for example, consider only how the majority of people will associate a high-pitched sound with a small, light, bright object situated high in space.

Another challenge highlighted by our recent research concerns the question of why so many of those interested in this area have chosen to map hue onto pitch1. In such cases, it would appear that the intuition is that dimensions of experience that are similarly structured must map naturally on to one another. So, for example, both colour and pitch can be represented as circular dimensions, and can potentially also be divided into the same number of basic categories. And yet, the fact that everyone who has put forward a crossmodal mapping between hue and pitch seems to have gone for a slightly different solution raises the question of whether this is really the most appropriate correspondence. Nowadays, some are starting to consider whether a more natural crossmodal mapping (or correspondence) for hue might be timbre instead. Resolving these questions will undoubtedly help those wishing to transfer information between the senses.

Charles now challenges and inspires how we taste and smell what we eat and drink:

Sensploration (FoST 2016)

Multisensory Experience and Coffee (Talks at Google)

Neurocuisine (The Guardian)

The Perfect Meal (RecoSymposium)

See our blog for Activities; especially 7-9.

_______________

1Spence, C., & Di Stefano, N. (2023). Sensory translation between audition and vision.  Psychonomic Bulletin & Review.