Category: Flavour

Association strategies in crossmodal metaphors

smackenz/ / Crossmodal correspondences, Flavour, Haptic touch, Hearing, Metaphor, Multisensory, Posts, Smell, Taste, Vision

Several correspondences between the senses exist. For example, transferring information about shape between touch and vision. Associating the sound of spoken words and visual shapes (as in the Bouba/Kiki-effect).

Rounded blob and spiky blob
(Bouba [left] and Kiki [right])
And, subjectively, the scent of a specific perfume with the feel of velvet fabrics. (See our blog for the scientific approach, Crossmodal correspondences between the senses, On the intriguing association between sounds and colours, and Multisensory processing.)

These correspondences are visible in crossmodal metaphors too. That is, when people are using words and phrases related to one sense to describe an experience from another sense. Like when they label visual colours, through words that are specific to the sense of hearing, calling them “loud” and “mute”. And define a sound through the sense of touch, as with “a smooth voice”.

 

I have invited researchers connected with the Diverse-ability Interaction Lab to write this post on how people generate and interpret crossmodal metaphors. These researchers have identified seven association strategies. The Diverse-ability Interaction Lab aims to change the design of interactive technologies in ways that make them inclusive, both for people who are disabled and people who are non-disabled. This post is written by Tegan Roberts-Morgan, University of Bristol.

 

“Blue tastes like salt, it just does”. That is what one participant told me when I asked them what blue might taste like. We all make connections between our senses. A citrus smell may be sharp; someone may have a sweet voice, or red might remind you of anger. We call these cross-sensory metaphors, as they use words from one sense to describe something which is typically associated with another sense. As a HCI researcher in sensory technologies, this is important, as understanding how these metaphors are created can give us an insight into the methods behind our sensory thinking, supporting us to hopefully design better sensory technologies.

 

We use association strategies to represent the different methods people use to create connections between different senses. These strategies help us to begin to understand the reasons behind why we make the cross-sensory metaphors that we do. If we can understand why the connections are made, then this can be leveraged in the design of technologies that support communication. To explore these strategies, we designed tasks that encourage participants to think in cross-sensory terms. For example, in Sense-O-Nary, participants are given an item related to a specific sense (e.g. the colour red, a pyramid, or a lemon scent) and asked to describe it using a sense that is not typically associated with it (e.g. what does red smell like, or what does a pyramid sound like?). They then share their cross-sensory metaphor with another team, who must guess which item is being described. This task, along with others we used, helped us to identify the 7 different strategies people use when creating cross-sensory metaphors.

  • Participants used personal stories and memories, and we labelled these as the personal connection strategy. One participant, for example, said that the lemon scent reminded them about when they went “on holiday to the Mediterranean” or “this reminds me of my friend”.
  • Participants also created cross-sensory metaphors using the familiar experience strategy. This is when the metaphor created uses a common object, emotion, texture etc. “This smells like a banana smoothie” or “this reminds me of a marshmallow” and even “this tastes like soy sauce”.
  • Some participants rely on some basic primitives to make an association, which we labelled as the sensory features strategy. This includes words like “sharp”, “smooth”, “soft” “bitter” and “sweet”.
  • Participants also used the valence strategy, using negative or positive words in the description, for example “I like this”, “I love this“ and “this would taste horrible”.
  • Another approach was using vocalisation. This involved participants using a sound or noise as
  • opposed to words to describe an item like “this sounds like Krrrr and tssssss”, “boooom” or when one child just screamed to describe what red may sound like.
  • Some participants chose not to use the sense that we originally asked them to use; they would instead use words from a different sense. We called this grasping for another sense. In one study we asked participants to describe how red would taste and they said, “this tastes strong”.
  • Finally, some participants did not only use their words to communicate their connection, but they also used their body. When they did this, they used the embodied action strategy. An example includes when one participant said green “feels like this” and then stroked the floor back and forth.

 

We believe that understanding and using these strategies can support designers, educators, and researchers in creating experiences that align with how people naturally relate the senses. For instance, we found that most adults used personal connections when describing how something would sound, so incorporating prompts or features that relate to a memory the person may have could support their communication.

 

We have found that age plays a vital role in what association strategies a person uses. Children tend to use familiar experiences the majority of the time, describing the item using something common. Whereas young adults (18-25 years olds) also used familiar experiences, but used personal connections, additionally, to create their metaphor. And finally older adults (65-80 years old) used a much wider range of association strategies, with sensory features being used more often.

 

These association strategies can be applied in any context that involves multisensory interactions, from educational devices that support children learning about their phonics by using shapes and audio, boards that can help children explain how their pain feels by using scents, shapes, colours etc., and accessible technology to support communication between children who are sighted and children who are visually impaired. Ultimately, association strategies give us a window into how people construct meaning across their senses. By recognising and applying these strategies, we can potentially design experiences that resonate more deeply, communicate more clearly, and build richer, more inclusive multisensory worlds.

 

See our blog for Activities; especially 70-72.

Food for thought: taste, smell and flavour

tgraven/ / Flavour, Haptic touch, Hearing, Multisensory, Posts, Smell, Taste, Vision

Eating and drinking are a truly multisensory experience – flavour occurs when input from all senses is combined. Not just from two or three senses; like when admiring the sea or the botanical garden; when skiing on a cold winter day; or when playing with a kitten or a puppy. (See our blog for Multisensory processing.)

In this blog post, I have invited Dr Xinni Xua,b and Professor Thomas Hummela to explain the relationship between smell and taste (known as chemical senses), and the other senses too. Dr Xinni Xu specialises in ear, nose, and throat conditions, both as a clinician and a researcher. Professor Thomas Hummel’s research focuses on the diagnostics and treatment of olfactory / gustatory loss, the mechanisms involved in irritation of the upper airways, the olfaction in neurodegenerative disorders, and the interactions between the olfactory, trigeminal, and gustatory systems. Between them, they have published over 800 articles, books, book chapters, and abstracts. And received several awards for their work (e.g., the Young Surgeon Award and the European Chemoreception Research Organization (ECRO) for “Excellence in Chemosensory Research”).

aSmell & Taste Clinic, Department of Otorhinolaryngology, Universitätsklinik Dresden. bDepartment of Otolaryngology – Head & Neck Surgery, National University Hospital, Singapore.

Let’s start with a simple exercise. Think about one of your favourite foods. Why is it your favourite?

Perhaps some of the immediate things that came to mind were that you like the way it tastes, or that its flavour appeals to you. But are taste and flavour the same thing? These terms are often used interchangeably in common language. Biologically speaking, they mean different things.

Not just a matter of taste

Humans can recognise five fundamental tastes: sweet, sour, salty, bitter and umami (savoury). In the last decade, the taste of fat (oleogustus)1 and even water2 have been added to this list. The taste buds containing taste receptor cells that are responsible for detecting these tastes are located mainly on the tongue, but they can also be found in other areas such as the soft palate, epiglottis and upper esophagus.

The scientific meaning of taste, of gustatory function, is therefore rather narrow. Food is much more than just these few categories. For example, vanilla and chocolate ice cream both taste sweet, but at the same time, you would not say that they “taste” the same. Your ability to smell is what distinguishes between the two flavours.

The nose knows

Diagram of how flavour is synthesised by the brain, mouth, nose and ears
Figure 1: The senses in flavour (Xu & Hummel)

There are two ways that odour molecules reach the olfactory receptors high up in the nose. The first is by sniffing through the nose, which is called orthonasal olfaction. The second way is from the back of the mouth and through the back of the nose. This is termed retronasal olfaction. Retronasal olfaction plays a bigger role in flavour perception than orthonasal olfaction3 . The next time you eat, for example something fruity, try chewing first with your nose pinched and focus on the taste. Then, after some 15-20 seconds, release your nose. With the first part of the experiment, the experience may have come across as somewhat flat. In the second part of the experiment, the added dimension of smell from retronasal olfaction creates the perception of flavour. One reason why the sense of smell adds body to flavour is because humans can detect thousands of distinct odours, if not more, compared to our ability to detect only a few types of taste4. We have about 400 different receptor types for smell5. In addition, combinatorial encoding occurs in olfaction, in which each odourant is recognised by varying combinations of olfactory receptors6. This makes our olfactory sense significantly more intricate than the gustatory sense.

Another aspect of smell is that it is directly linked with emotions – neurologically speaking. The olfactory bulb, a structure at the base of the brain, receives olfactory signals from the nasal olfactory receptors. The olfactory bulb projects neurons directly to the limbic system, including the amygdala and hippocampus, which are responsible for regulating emotions, memory and behaviour. This is why odours can involuntarily evoke memories of events past and feelings associated with them. This experience has a name: the Proustian effect, coined after the French author Marcel Proust. In his tome “À la recherche du temps perdu (Remembrance of things past)”, he famously reflected on how a morsel of madeleine soaked in tea brought forth a heady rush of vivid memories from the past and whimsical nostalgia.

“Mouth-feel”

A big part of food experience is how food feels in the mouth. This is known as somato-sensation. Receptors from the trigeminal nerve in the oral cavity are responsible for this, and they include mechanoreceptors to detect texture and tactile stimulation, thermoreceptors to detect temperature and nociceptors detect noxious stimuli. Mouth-feel is what allows us to savour the creaminess of a pudding, the crunch of crisps, the prickle of carbonated drinks and the spiciness of curries. These sensations can influence our preference or rejection for foods of certain temperatures or textures. Somato-sensation becomes even more important in improving the hedonic quality of a meal for people who have lost their sense of smell or taste.

In short, flavour is the synthesis of individual tastes, smells and feels into an overall perception, modified by mastication sounds during eating, the appearance of the food and numerous environmental factors7. Yet at the same time, flavour is more than the sum of its parts, because it is shaped by our individual experiences, cultures and preferences. Hence the reason why the flavour of your favourite food appeals to you is really a matter of personal taste.

See our blog for Activities; especially 40-42.

Some suggestions for further listening, reading, and watching:

Losing the sense of taste

Madeleines and memories – the Proust Effect

Overview of Smell and Taste Disorders

This is What it’s Like to Live in a World Without Smell

Taste Test

What is life like without smells

_______________

1Running, C. A., Craig, B. A., & Mattes, R. D. (2015). Oleogustus: The Unique Taste of Fat. Chemical Senses, 40(7), 507-516. DOI: 10.1093/chemse/bjv036

2Zocchi, D., Wennemuth, G., & Oka, Y. (2017). The cellular mechanism for water detection in the mammalian taste system. Nature Neuroscience, 20(7), 927-933. DOI: 10.1038/nn.4575

3Hummel, T., & Seok, H. S. (2016). Orthonasal and retronasal perception. In: E. Guichard, C., Salles, M. Morzel, and A.-M. Le Bon (eds). Flavour: From food to perception [pp 310-318]. Oxford: Wiley Blackwell. DOI: 10.1002/9781118929384.ch13

4Dunkel, A., Steinhaus, M., Kotthoff, M., Nowak, B., Krautwurst, D., Schieberle, P., & Hofmann, T. (2014). Nature’s Chemical Signatures in Human Olfaction: A Foodborne Perspective for Future Biotechnology. Angewante Chemie International Edition, 53(28), 7124-7143. DOI: 10.1002/anie.201309508

5Mainland, J. D., Keller, A., Li, Y. R., Zhou, T., Trimmer, C., Snyder, L. L., Moberly, A. H., Adipietro, K. A., Liu, W. L. L., Zhuang, H., Zhan, S., Lee. S. S., Lin, A., & Matsunami, H. (2014). The missense of smell: functional variability in the human odorant receptor repertoire. Nature Neuroscience, 17(1), 114-20. DOI: 10.1038/nn.3598

6Firestein, S. (2001). How the olfactory system makes sense of scents. Nature, 413(6852), 211-218. DOI: 10.1038/35093026

7Spence, C. (2020). Multisensory Flavour Perception: Blending, Mixing, Fusion, and Pairing Within and Between the Senses. Foods, 9(4), 407. DOI: 10.3390/foods9040407