Invisible strings- Submission 2 – dmsp-performance23

Our Final documentary：https://youtu.be/5wdCRBJSlR4

Part1 Video：https://youtu.be/6fKzaTzt7N0

Part2 Video: https://youtu.be/5MQRS2qN5ck

Part3 Video: https://youtu.be/ltNtx1xkXow

Our project’s own source files

Processing
https://drive.google.com/drive/folders/1oyMpaftdqVVoR2Yg1Jd-NMoTqF9XqOA1?usp=sharing

Touch Designer
https://drive.google.com/drive/folders/1mTUTHKC1a6cvSnVqj6IKhykOgJr1NG63?usp=sharing

Processing Test (Fist Version)
https://drive.google.com/drive/folders/1b57YIC2tuZJ0PA9p-rgbjUhLvx3UEDa5?usp=sharing

Part 1 Mixing https://drive.google.com/file/d/1mFLVemaLbHSIs_EXReBWH9cNFqvpOlIB/view?usp=share_link

Part 2 Composing https://drive.google.com/file/d/1hJ__2rSRBj7yTwlH3oCR8tdIHqadInnu/view?usp=share_link

Part 2 Max Patchers

https://drive.google.com/file/d/1xCIy2S-RgqypvWlqYt3FvaF5v551XoO7/view?usp=share_link

Part 3 Mixing

https://drive.google.com/file/d/1UBbC0qdSQUK-sPXm9zSnHK_XTKBExe_W/view?usp=share_link

Background

The creation of Invisible String originated from the personal experience of one of the group members, Shutong. She had a habit of biting her nails, and whenever she felt nervous, she would start biting them. So she started to trace the origin of this habit, and when she saw the film Autumn Sonata, she remembered that this habit was a stressful reaction to her mother’s demands that made her feel nervous during her piano lessons.

As Shutong told us about this experience, we realised that such encounters did not only happen to her alone but that in different ways, we were bound by our parents in one form or another. Many people grow up with high expectations from their families and too much pressure on their children. We, therefore, decided to use the film Autumn Sonata as a backdrop for our performance, creating an interactive performance piece, invisible strings, which uses the film as a basis for exploring the relationship between children and parents, about expectations, bondage and the process of self-reconciliation.

Design goals

Our group set the following goals for our performance
– To explore in depth the relationship between children and parents
– A multi-sensory, multi-technical performance
Our group set the following goals for our group
-Good teamwork
-A clear division of labour, each with their own strengths
-planning the allocation of time

Refine after submission1

After Submission 1, we got some design feedback. In it, we summarise effective information, including consideration of venue constraints, audience experience, and the presentation of increased visual content.

In our original concept, we designed a lot of large-scale installations, which made us need a very large space to support the performance. But the space we can get is limited, so we modified our proposal to remove these large art installations.

The second is the modification of the visual content. In the original design plan, our visual design was relatively boring, more like a display of installation art than a performance. Therefore, we redesigned the visual expression of each part in the later stage and integrated the concept of the original large-scale installation into the visual expression. For example, the first part is the high pressure that the daughter feels under the verbal accusation of the mother. The original scheme of projecting abstract growing particles onto the ground from top to bottom with a projector was changed to a real shot. Videos convey the possible impact of parents’ verbal and behavioural suppression on their children. This will serve as the beginning of the story and the prelude to the whole performance.

Looking at our original design scheme from the audience’s perspective, we realised that the expression of the content might be too abstract. So we revisit the proposal. Present the story of Autumn Sonate to the audience step by step as a storyline. This will enable the audience to understand the story more clearly and feel the emotions that our works want to convey.

Research and Critical thinking

1. Performance and Installation

During the creative design phase of this project, we searched extensively online for reference projects. Ultimately, we selected some of them and developed our design based on them.

Image References about Performance

In these reference images, the performer and the video presentation seem to be two independent entities. However, visually appealing effects may help the performer to better convey the musical presentation to the audience, for example, by using video to help viewers understand the expression of the music. Therefore, in our performance, the presentation of the video content may be abstract, but the interaction and musical presentation can be coordinated with the image to complete the story narration.

In reference image 2, the performer’s hand and the video image or hand shadow in the image correspond to each other, bringing some stunning visual experiences to the audience. We drew inspiration from this short experimental film and realised that when many objects are repeatedly stacked in the image, it can give the audience a sense of oppression. This is similar to what we want to present, where the mother pressures her daughter. At the same time, through the shadows in this work, we came up with the idea of a simple interaction between the actor and the video material. The person’s shadow in the video and the actual performer exists in two different spaces but are connected. Based on these ideas, we completed the ideation process for the third part.

Video References about Interaction

In addition to image references, we also looked for related interactive installation art videos as our references.

1. Tomás Saraceno

Firstly, we looked at the works of Tomás Saraceno. In this work, Tomas used lines to form spider-web shapes, and when the audience touches the lines, they create sound.

We believe that lines can also be represented as the strings of a piano. In our performance, the mother and daughter are connected by an invisible bond, visualised by a red thread. If a piano wants to make a sound, it needs the vibration of the strings, which is also the vibration of the lines. Therefore, in our design, in the second part of the work, we used black and white lines to represent the piano strings. However, unlike Tomas, it is difficult for us to make the strings produce sound. Therefore, we chose to tie solid bells to the lines in this part. When the audience interacts with the lines, the sound of the bells will appear. At the same time, the positional relationship of the actors also generates random sounds through Max as a supplement to the sound of this part.

2. Particle References

We have found some reference videos on motion capture and particle effects. The interaction between dance and particle effects can bring great visual impact. However, if the visual of particles dominates the visual presentation, it may make the performer lose its functionality in our performance. Also, the interactive effect may not be better displayed, only playing a video made by the particle system for the audience. In our design, the main visual colour is black and white to match the piano theme. To simplify the related visual effects, we designed the colour of the character’s particles as white, reducing the degree of dispersion. We also add sound control to the particles but only make them dissipate when the sound reaches a fixed range.

3. Touch Designer

In most Touch Designer cases, we found online imported pre-produced music files and analysed them to generate video changes. For example, in our reference case, it used a chop to the controller and noise controller to simulate the feeling of blood spreading from a paper by analysing the emotion and style of the music. However, according to our idea, we hope the sound can generate real-time video effects changes. Therefore, we use the Device in the microphone module instead of audio to achieve real-time sound analysis and video changes.

Sketch of the performance venue

Taking into account the audience’s walking path and the placement of equipment such as projectors, we created a layout diagram for the venue.

Overhead view of the performance venue

Technology (visual; sound) flow chart

Performance

Part 1

Performance Part1-Visual design

The first part of the main interaction design is the impact and change of the live real-time human voice input on the video screen. The visual design is dominated by the video projected on the curtain. The main content of the video is the creative concept of abstract plot expression using live filming methods, distilling elements such as the mother’s reprimand and facial movements, fusing them with abstract editing expressions to present the audience with a conceptual video of the tense and oppressive relationship between mother and daughter. As the first act of the performance, the video serves as an introduction to the story and sets the emotional tone of the story, allowing the audience to feel both the oppressive relationship between mother and daughter and the flowing, uninhibited changes in their own emotions as they watch the first act.

The video’s focus on the piano-practising daughter constantly becoming struggling and in pain under her mother’s oppressive scolding and gaze. In the design of the scripted content of the video, the visuals use metaphors to assist in the expression of abstract emotions – the ‘hand’ as a metaphorical element to express emotions: a red thread tied to a finger is used as a clue and the beginning of the story, followed by a large The two hands pulling on each other, the struggling hands behind the gauze, etc., all express the emotions of the mother’s constant manipulation and restraint of her daughter. The close-ups of the “mouth” and “eyes” are direct expressions of the mother’s attitude, and the close-ups highlight the intensity of the emotions.

In the filming of the video, the props and the make-up of the actors are relatively plain in order to create a sense of storytelling, and a clear distinction is made between the images of the ‘mother’ and the ‘daughter’. In the editing and processing of the video, the overall tone of the original video was reduced to black and white to convey the depressing and suffocating nature of the relationship between mother and daughter. The use of “flashback” footage as a transitions to express the mother’s control over her daughter is a subconscious and irregular control and bondage, and the overlapping of the two scenes also expresses the struggle between the daughter’s psychology and reality.

Performance Part1-Touch Designer

In the conception of Part 1, we hoped to present the beginning of the story of Autumn Sonata to the audience clearly according to the development of the storyline. The team believed that a short experimental film could help us to tell the story, so we shot an experimental film to showcase the fear and anxiety of the girl towards piano practice under the excessive expectations of her pianist mother. After completing the filming and editing of the short film, we imported the video into Touch Designer and searched for corresponding tutorials on the website to help us achieve the effect of garbled characters appearing on the screen according to changes in sound. The final result was the successful implementation of showing distorted images of the daughter in the video based on changes in sound, which conveyed the daughter’s anxiety under the excessive expectations of her mother.

Processing of learning Touch designer

We found some video and text tutorials online. The main references are:
https://www.youtube.com/watch?v=IFegKFjtj80
https://www.youtube.com/watch?v=rvAB3Rzh7CI

Here are three parts: Sound Analysis, Video Changes and Background Music.

1) Sound Analysis

We used the Audio Device In controller as the source of the sound input, which is different from the video tutorials we referred to. Then we analyzed the relevant parameters of the sound according to the tutorials.

2) Video Changes

We divided the video editing into two identical parts: the main video changes and the daughter’s separate video in the mother-daughter frame. This was done so that we could later overlay the daughter’s video onto the right side of the original video to achieve the effect of the daughter’s part changes separately.

3) Background Music

We imported the background music file and added the Audio Device Out controller so that we could connect to the Audio Interface and provide the audience with better sound effects.

In addition to these three parts, we added controllers to achieve the changes in certain video parts better. We used code to set the controllers only to achieve the garbled effect in specific keyframe areas. We also set the reload shortcut code for the video and audio so that when we hit the “1” key on the keyboard, the sound and video will restart playing simultaneously.

Performance Part1-Sound Design

The first part of the project is the introduction to the scene in terms of sound that both fits the specific movements of the characters and conveys the depressingly low atmosphere in the images. The two main parts of the recorded sound samples are the two phases of the voice, the multiple whispers and the single chant, and the recording of sounds with a sense of movement and rhythm with the objects （balloons, cloth curtains, woollen balls ）used for filming.

After sorting through all the recorded sounds, adding filters, reverbs, delays, and other effects make the sounds more in tune with the narrative qualities. To shape the actual piano performance episode, we recorded sound clips of the group member playing the piano piece Pour Le Piano (Achille-Claude Debussy, 1901). Writing the musical parts, adding musical elements such as synth, violin, bass and piano to build the plot development of the images.

In the first stage of the performance, an actor will play the role of a mother on set, feeding in an angry, accusatory voice against her daughter in the picture in real time. The vocal will affect the picture and give it a distorted effect.

Part 2

Performance Part2-Visual Design

In terms of visual considerations, we felt that Part 2 was the most important part of the performance, a transition between the struggle of the daughter’s self-consciousness in Part 2 and her reconciliation in Part 3. We chose a member of the group to play the role of the “daughter”, and we used particles to form a portrait, using an abstract means to encapsulate the image of the daughter because the role of the “daughter” in the film exists not only in the film but also in real life. It is a microcosm of the countless children who are oppressed by their parents.
(as shown in fig1.particles below)

fig1.particles

The square shape in the background was also designed to be a metaphor for the invisible walls of a mother’s bondage to her children, who are trapped within them.
Initially, we planned to use black and white for the visual design, but on 27/3/2023, following feedback from Jules, who had seen the group’s performance and suggested we add colour to the visuals, we ended up changing the cube in the background to a bright red colour to add visual impact to the image between the black and white blocks.
Finally, we choreographed the movement of the actor playing the daughter, who wakes up from the darkness in a world of her own inner self, covered with threads created by her mother’s expectations of her, and begins to panic, trying to break free but having to be pulled and twisted like a puppet entangled.

Performance Part2-Processing

Background

In the second part of our performance, we aimed to portray the inner conflict and struggle of the daughter for the audience. Therefore, we wanted to capture the actor’s movements and project them into a 3D particle space to depict the daughter’s inner world. Eventually, we chose the Kinect V2 sensor for this purpose. As the Kinect is an RGBD sensor, it can recognize the human body and automatically calculate the motion data of each joint based on the person’s proportion in real-time, capturing the person’s outline.

After selecting the sensor, we attempted to use Touch Designer to achieve the motion particle presentation. However, as some team members had more experience using Processing, we decided to use Processing to present the motion data captured by the Kinect for better visual effects.

Process

In the Processing coding, there are mainly three parts: Particle from Kinect capture, particle clustering and dispersion influenced by sound changes, and movement of a red wireframe.

Firstly, the setup() function initializes the libraries being called, sets up the screen, audio input, and particle array.

1）Kinect

We used the KinectPV2 and peasy libraries to access the Kinect camera and create an interactive camera controller. We called the Kinect data by defining “kinect = new KinectPV2(this)”. In the code, after initializing the Kinect sensor, It creates arrays and ArrayLists to store particle coordinates, image data, and particle size information. Using If statement, Processing checks whether the Kinect detects any human bodies. If the actress is detected, it iterates through the pixels of the body tracking image and looks for pixels that correspond to the user’s body. For each pixel found, it creates a particle-like effect at the corresponding location and adds it to the ArrayList of particles. If no actress is detected, it randomly generates new particles. The particle movement, dissipation, and iteration are also set up at the same time.

2）Sound

We also used the processing.sound library to capture audio input from the computer and extract volume information. By using “float volume = loudness.analyze()*5”, we can quickly adjust the sensitivity of how sound affects particle dissipation according to the impact of environmental noise during live performances.

3）Red Box

We used P3D to draw a rotating 3D cube in space, controlled by “rotateY(-sin(frameCountspeedR));rotateX(-cos(frameCountspeedR));” to adjust its rotation speed. In the initial design, we used a white wireframe, but later changed the stroke color to red. During live testing, we found that the wireframe was not clear enough, so we adjusted the strokeweight() data to make the wireframe thicker, making it easier for the audience to see.

Max and Processing Connection

We intended to output the sound data generated by the performer in real time to the processing via Max, thus causing a diffusion of the visual particle effect to occur. The LAN is set up in the Arduino and Max is connected under the same port.

Using Max as the emitting device, the volume data is sent to the ip address where the processing is located, setting the data to vary between 0 and 1.

In the Processing part, we conducted tests on data reception using the test code from the OSC database based on relevant references we found online.

After modifying our own IP address and receiving port, we carried out sending and receiving simultaneously.

However, in the end, we did not receive any data from Max on the Processing side. Instead, Max received ‘123’ sent from Processing on the Max side. So we gave up this plan and still use environment sound to affect the particles.

Performance Part2-sound design

Composition Creation

In the second part of the performance, the music aims to connect with the feelings of the ‘daughter’, to express her emotional process from confusion to fear to struggle when she is in a repressed inner world. When conceiving the sound, we thought more about how to leave more of the sound creation to the performer.

Depending on the design of the performance setting, we set up strings with bells of different sizes tied around the performers, which, when touched by the performers as she is dancing, produce a random sound of bells that become part of the sound source that will form the music.

we intend to make part of the sound follow the dance movements of ‘daughter’ by capturing data on the locations of the hand movements in space, sonifying the data and using a smooth and flowing timbre to express the changes in the daughter’s state of mind.

In addition, a piece of background music is needed to set the mood for the whole performance, to underline the three internal states of the daughter and to provide a guide for the emotional development of the performance.

Composition project file in Reaper

Interactive sound

The M5stick is attached to the performer’s wrist and receives real-time data changes in Max in terms of yaw, roll, pitch, etc.

The LAN set up in Ardunio allows Max to receive data messages sent by the M5Stick.

Limiting the threshold to the data collected limits the range of variation to a usable interval using scale. When the data changes, the corresponding pitch and timbre are actuated and the rhythm of the sound is changed at the same time.

The actor can promote two states of sound, a detuned piano with a sense of dropping and a sharp ripping sound.

Performance Part3-Visual design

The third part of the performance is the interaction between the performer and the video through the collaboration of the performer, the projection and the video. To ensure that the performance is effective in a dark environment, the visual design of the projection in the performance is based on the video projected on the gauze, with the performers docking behind the gauze and the projection in the air to complete the performance. The main content of the video is shot in real life and post-processed into the form of a shadow. The shadow in the video represents the inner character of the daughter, and the performer represents the ‘daughter’, with the daughter and the shadow pushing each other away and searching for each other. As the final scene and conclusion of the video, the projected image represents the daughter’s struggle to find her inner self and to reconcile with herself.

In the design of the scripted content of the video, the shadow representing the daughter’s inner self goes through trials, fears and confusion as she explores her inner self and eventually finds a balance between herself. The red residue behind the shadow echoes the red line, the dissipation of the particles of the characters echoes the particles of the second act, and the appearance of the mouth and eyes in the image is the climax of the third act, where the daughter finally finds herself in the darkness of confusion and embraces “herself”.

Performance Part3-sound design

Sound Quality

In the third stage of the performance, the performer interacts with the ‘shadow’ in the video, and part of the sound needed to match the qualities of the shadow in the visual element, so it was created by combining the recorded sound of fabric and thread rubbing, and by overlaying the frequencies of several sounds after equalising them, to make some sound effects with the effect of shadows moving.

Because the shadow contains a stressful aspect that the daughter used to wallow in, whispers to the left and right, symbolising the droning and accusations from the mother will accompany the shadow. To these are added depressing synth tones and fast-paced sound effects echoing the shadow’s movements. The actual actor stands out through the chanting vocals, which portray her self-finding process.

Music

There are three main stages in the development of the music; the first stage is the two figures pulling and searching for each other, the second stage is a return to the depressingly gloomy atmosphere of the first performance section, and the third stage is a melodic narrative of the two figures merging into one.

The base of the entire piece is set by washy synth sounds, which are joined by a morphing wind section with a sense of refrain as it moves into the second phase, and a choral effect piano sound is applied to externalise the merging process in the final melodic section.

Reflection

From left to right are Crystal Wu, Lerong Qi, Shutong Liu and Xiaozhuang Gao

What is good in the process？

Throughout the project, our team communicated and divided up the functions so that everyone could contribute their skills to the project to the best of their ability.

Lerong Qi was responsible for the sound design, Shutong Liu for the visual design, Crystal Wu for the technical support and Xiaozhuang Gao for the visual design and technical support for the code, and he is also Project Coordinator and controlling the overall project finalisation concept. Once we had confirmed the division of labour between each person in the team, every communication in the group could lead to an effective conclusion. Each person can present their own suggestions and ideas for different parts of the project, and the person in charge of each part makes the final decision. At the end of the day, the team member responsible for coordinating the whole project ensures and follows up on the visual, sound and technical outputs in a consistent style and that each part goes smoothly and delivers a complete story and effective output.

What is difficult in the process？

When we produced the installation for the second part of the performance, we faced unexpected difficulties. Our team hopes to build an installation about the string and combine bells to interact with the performers and make sounds to express the emotions of the “daughter” growing up under the oppression of her mother. To make the set-up and dismantling of the scene easier to manage, we asked the advice of the instructor Andrew. He suggested that we use wooden sticks to pre-wire the set-up so that during the performance, we only need to fix the position of the sticks in the venue, and the installation of the device can be completed quickly. However, when we took this approach and spent at least five days surveying the site, purchasing materials, and fully tying up all the wires, we found that the number of wires we used was far more than we expected. Whenever we lay out the sticks and rope, we spend three to four hours untying the knots. With the addition of bells, the knotting situation is even more serious. On the day of the test, Wednesday, we brought the lined-up lines to the venue, but we couldn’t untie them. As a result, we had to modify the setup plan of the device the day before the show and tie the strings directly to the wooden sticks on site. This also means that we must spend 1-2 hours completing the construction of all the lines before the official performance, which brings us great pressure.

Secondly the second difficulty is that our team is short of manpower, and our project is relatively large. For a team of four, the project is not too easy and smooth. In addition to our own design work, we also need to rent equipment and measure venues, repeat lottery tickets, etc. The four of us often need to carry sound equipment and filming equipment back and forth between Alison House and Booket Centre.

What can we do better in the future？

In this project, we had a lot of discussions on the concept and content of each part at the beginning, which took up a lot of our time, so we started to implement all the plans later, and the time to complete each part was compressed. Although the clear concept and design allowed us to achieve the ideal presentation effect, we can do better in this part. By figuring out everything early on and starting the actual build, we might have more time to rehearse and troubleshoot tangled wires. It can bring better presentation effects to the audience during the actual performance.

But in general, we realised 90% of our vision for the whole project, which can be said to be the result of our very successful teamwork. We would like to thank every team member for their participation and dedication.