Any views expressed within media held on this service are those of the contributors, should not be taken as approved or endorsed by the University, and do not necessarily reflect the views of the University in respect of any particular issue.
In the third stage of the performance, the performer interacts with the ‘shadow’ in the video, and part of the sound needed to match the qualities of the shadow in the visual element, so it was created by combining the recorded sound of fabric and thread rubbing, and by overlaying the frequencies of several sounds after equalising them, to make some sound effects with the effect of shadows moving.
Because the shadow contains a stressful aspect that the daughter used to wallow in, whispers to the left and right, symbolising the droning and accusations from the mother will accompany the shadow. To these are added depressing synth tones and fast-paced sound effects echoing the shadow’s movements. The actual actor stands out through the chanting vocals, which portray her self-finding process.
Music
There are three main stages in the development of the music; the first stage is the two figures pulling and searching for each other, the second stage is a return to the depressingly gloomy atmosphere of the first performance section, and the third stage is a melodic narrative of the two figures merging into one.
The base of the entire piece is set by washy synth sounds, which are joined by a morphing wind section with a sense of refrain as it moves into the second phase, and a choral effect piano sound is applied to externalise the merging process in the final melodic section.
In this project, I was mainly responsible for the design and composition of the sound part, including recording sound, designing sound effects, composing music, and also exploring the techniques of sound-painting interaction, helping the group to realise a live performance using M5Stick sensors and Max to follow the actors’ movements in real-time to generate semi-automated music. This was also my first attempt at composing music that matched the atmospheric settings of the three stages of the project through midi, designed sound effects. I also designed the audio system to fit the performance area, ensuring that the audience could hear the abundant sound details during the performance live.
During the performance I was involved as the sound performer, controlling all the sound output effects, adjusting the balance between the multiple sound elements and mixing the sound for the scene in real time.
Technical Approach
For the recording, I recorded the sounds of many common household objects, such as floating curtains, torn fabric, squashed balloons, twisted threads and so on. These subtle sounds in the family life are often amplified in a depressingly intimate setting, and I used these sounds, either by equalising, layering, or adding effects such as delayed reverb, to create a soundscape that could be used alongside whispered and chanted vocals to build the character’s inner atmosphere.
When composing the three parts, I focus on using some hollow synth sounds as a backdrop and distorted piano as a melody to tell the story. In writing the second part of the dance, some powerful drums and crisp percussion build a disorienting and tense musical base.
Considering the human-sound interaction part of the performance, I changed part of the sound input of the first part of the performance to come from the performer in the scene, some teaching and reprimanding words became part of the sound performance, and these sounds affected the image, thus distorting it.
In the second part of the performance, many components of the music are derived from the generation of the performer’s movements, as she touches the strings around her to drive the bells to ring, using the M5Stick to capture her hand movements and compose the music in real time. The M5Stick is connected to Max under the same LAN using Arduino, and by receiving real-time yaw, roll, pitch, etc. data from the M5, scale them and trigger changing timbres, pitches and rhythms.
Wifi Setup
M5Stick and Max Connection
Reflections
The techniques I utilised were limited in sound composition, as it was my first time composing, and I needed to gain experience, making it challenging to write melodies with layers of evolving feeling. And when using Max for semi-algorithmic compositions, not adding more sound elements that fit the intense action of the characters, many objects still needed to be mastered. In the second stage of sound and picture interaction exploration, the attempt to use Max as a sender of output volume data caused the parameters in Processing to change, thus making the effect of visual particle dispersion not possible. In the future, I will focus more on learning techniques and exploring more possibilities for sound interaction.
Documents
Part 1 Mixing https://drive.google.com/file/d/1mFLVemaLbHSIs_EXReBWH9cNFqvpOlIB/view?usp=share_link
Part 2 Composing https://drive.google.com/file/d/1hJ__2rSRBj7yTwlH3oCR8tdIHqadInnu/view?usp=share_link
Part 2 Max Patchers https://drive.google.com/file/d/1xCIy2S-RgqypvWlqYt3FvaF5v551XoO7/view?usp=share_link
Part 3 Mixing https://drive.google.com/file/d/1UBbC0qdSQUK-sPXm9zSnHK_XTKBExe_W/view?usp=share_link
In the second part of the performance, the music aims to connect with the feelings of the ‘daughter’, to express her emotional process from confusion to fear to struggle when she is in a repressed inner world. When conceiving the sound, I thought more about how to leave more of the sound creation to the performer.
Depending on the design of the performance setting, we set up strings with bells of different sizes tied around the performers, which, when touched by the performers as she is dancing, produce a random sound of bells that become part of the sound source that will form the music.
I intend to make part of the sound follow the dance movements of ‘daughter’ by capturing data on the locations of the hand movements in space, sonifying the data and using a smooth and flowing timbre to express the changes in the daughter’s state of mind.
In addition, a piece of background music is needed to set the mood for the whole performance, to underline the three internal states of the daughter and to provide a guide for the emotional development of the performance.
Composition process
Background Music
Composition project file in Reaper
Using crunchy piano to sketch out melodies, they go towards rendering a sense of perplexity. A single note with a soprano timbre is added to the piano, appearing every other bar to reflect the flowing piano sound.
On the bass side, three percussions form the basic rhythm, with the low, harmonic overtones playing the role of the kick in the drum kit, which, together with the tuned drums, sets the overall disorienting and tense atmosphere. Powerful drumming joins in as it moves into the second section, matching the daughter’s physical movements as she tries to break through the invisible wall of pressure.
In order to give the overall music a more oppressive, hollow sense, ethereal synthesizer sounds are used to play the harmonies.
Interactive sound
The M5stick is attached to the performer’s wrist and receives real-time data changes in Max in terms of yaw, roll, pitch, etc.
By limiting the threshold to the data collected, the range of variation is restricted to a usable interval using scale. When the data changes, the corresponding pitch and timbre are actuated and the rhythm of the sound is changed at the same time.
The actor can promote two states of sound, a detuned piano with a sense of dropping, and a sharp ripping sound.
The first part of the project is the introduction to the scene in terms of sound that both fits the specific movements of the characters and conveys the depressingly low atmosphere contained in the images. The two main parts of the recorded sound samples are the two phases of the human voice, the multiple whispers and the single chant, and the recording of sounds with a sense of movement and rhythm with the objects used for filming.
Recording
Objects used: balloons, cloth curtains, woollen balls, piano, etc.
After sorting through all the recorded sounds, adding filters, reverbs, delays, and other effects makes the sounds more in tune with the narrative qualities.
from cloth curtains
double voices
voice with oVox
Mixing
Writing the musical parts, adding musical elements such as synth, violin, bass and piano to build the plot development of the images.
Interactive sound
In the first stage of the performance, an actor will play the role of a mother on set, feeding in an angry accusatory voice against her daughter in the picture in real time, the vocal will affect the picture and give it a distorted effect.
