Any views expressed within media held on this service are those of the contributors, should not be taken as approved or endorsed by the University, and do not necessarily reflect the views of the University in respect of any particular issue.
In the third stage of the performance, the performer interacts with the ‘shadow’ in the video, and part of the sound needed to match the qualities of the shadow in the visual element, so it was created by combining the recorded sound of fabric and thread rubbing, and by overlaying the frequencies of several sounds after equalising them, to make some sound effects with the effect of shadows moving.
Because the shadow contains a stressful aspect that the daughter used to wallow in, whispers to the left and right, symbolising the droning and accusations from the mother will accompany the shadow. To these are added depressing synth tones and fast-paced sound effects echoing the shadow’s movements. The actual actor stands out through the chanting vocals, which portray her self-finding process.
Music
There are three main stages in the development of the music; the first stage is the two figures pulling and searching for each other, the second stage is a return to the depressingly gloomy atmosphere of the first performance section, and the third stage is a melodic narrative of the two figures merging into one.
The base of the entire piece is set by washy synth sounds, which are joined by a morphing wind section with a sense of refrain as it moves into the second phase, and a choral effect piano sound is applied to externalise the merging process in the final melodic section.
The third part is achieved through the cooperation of the performer, projection, and video. In order to ensure the effect in the dark performance environment, the visual design of the projection in the performance is based on the video projected on the gauze, and the performer is docked with the projection behind the gauze to complete the performance. The main content of the video was filmed in real life and post-processed into the form of a shadow. The shadow in the video represents the inner part of the daughter and the performer represents the “daughter”, the daughter and the shadow push each other away and search for each other, and finally when the physical daughter and the shadow become one, it represents the daughter finding the balance between herself. As the last scene and the end of the video, the projected image represents the daughter going through self-discovery and struggle, and finally finding her inner self and reconciling with herself.
In the design of the video script content, in the process of the daughter exploring her inner self, the shadow representing the daughter’s inner self goes through trial and error, fear, confusion, and finally finds the balance between the self. The red residual shadow behind the shadow and the red line echo each other, the particle dissipation effect of the characters echoes the particles in the second act, the appearance of the mouth and eyes in the image is the third act to push the emotion of the third act to the climax, the daughter finally found herself in the dark confusion, and embraced with “herself”.
Our team took field measurements of the Alsion house, such as the height of the central iron frame and the length, width and height of the venue, to determine the floor space and curtain size required for each performance.
Based on the scale of the site, I made a sketch. Considering the placement of audio equipment, the audience’s sense of hearing and immersion, we finally divided the venue into the following picture. The screens of the first part cover part2 and part3 first to create a sense of mystery. We need to wait for the audience to complete part1. After watching, follow the signs on the ground to move the viewing positions of part2 and part3. Therefore, our performance is carried out in a progressive narrative mode.
In “Invisible String”, I (Xiaozhuang Gao) was responsible for the programming techniques and visual design of part 2, as well as taking on the role of coordinating the team. I needed to make creative decisions regarding the core concepts and forms of the performance, in order to drive the project forward and unify team members’ ideas. Therefore, in this reflection, I will introduce the work I did in the group and what I learned in this course.
Reflection- technology support
Firstly, in the early stages of the project, I was responsible for the programming part of Processing and Kinect. After Submission1, as we believed that Part1 of the performance had already involved abstract particle graphics, we wanted to control the growth speed of the particles through sound interaction. Based on this idea, I began to write the particle code, and the initial particle generation was designed in Touch designer. As I had prior experience using Touch designer, I believed that it was a great software for sound and visual interaction, and that we could control particle changes through functions such as Life Expect, Drag, and Life Variance.
Fig 1. First particle testing on TD
During the technical implementation, I found that the particles in Touch Designer were essentially composed of individual pixels and could not be resized to the expected size. Therefore, I switched to using Processing to achieve the particle effect.
As the project progressed, we decided to combine the Processing particle section with Kinect motion capture for Part 2 of the performance instead of Part 1. After discussing with my teammates, I believed that the figure captured by Kinect V2 should be trapped in an invisible space, metaphorically representing the cage formed by a mother’s expectations for her children. Therefore, when drawing the graphics, I utilized the built-in P3D mode in Processing, adding a parameter for depth. To set different origins and rotations for different objects, I used the pair of matrix statements pushMatrix() and popMatrix(). With these commands, I was able to create 3D graphics while maintaining the rotation of the cube. I used the code instruction rotateY(-sin(frameCountspeedR)); rotateX(-cos(frameCountspeedR)); to control the rotation speed on the X and Y axes, thereby expressing the concept of the daughter being trapped in bondage and unable to escape. (fig2.Coding)
Fig2.Coding
And, based on the concept of “confinement”, I came up with an idea: Can we calculate the position of the vertices of the cube to prevent the particles from escaping during diffusion? After testing, I found that because the cube is constantly rotating randomly, it is impossible to calculate the exact vertices, which also means that the particles cannot be completely confined within the cube. However, after consulting tutorials, I adopted a compromise by controlling the left and right, and top and bottom boundaries of particle movement. Additionally, while testing the kinect, I attempted to test the movement of particles in a 3D environment but found that the effect was not satisfactory, so I did not enable the Z-axis depth value.
In fact, there are still many areas for improvement in the visual code presentation of Part 2. For example, on March 27th, 2023, during a tutorial, the instructor pointed out that the visual richness of Part 2’s particle display was somewhat disconnected from Part 1’s key imagery. Although I eventually changed the background cube to red, the problem still persisted. In my opinion, in future project development, the color of the particles could reflect the color of the clothing captured by the Kinect, rather than just being white.
At the same time, because of the interaction with sound, the speed of particle diffusion and disappearance was not well controlled, sometimes resulting in a too-scattered state. This part of the code still has room for improvement.
Reflection- Visual design
In terms of visuals, Shutong Liu and I were responsible for the visual design of the project. At the beginning of the project, our group’s visuals were monotonous and abstract. Following feedback from our submission 1 mentor, we were encouraged to increase the feedback of the visual projection. Drawing from my previous experience in filming and producing key images, I suggested to the group that we use key images to showcase our visuals, and use narrative techniques to present our performance. Subsequently, I drew the storyboards for part 1 video and the video was filmed based on this. Meanwhile, our group discussed many versions of the performance venue layout, and I combined the opinions of the group members to design the performance venue (Fig3. Performance venue). I also designed the poster for our final performance (Fig4. poster), and our group worked together to build the line installation for part 2.
Fig3.Performance venue
Fig4. poster
In my opinion, the biggest visual challenge was the planning of the performance venue and the construction of the installation. The planning of the venue took too much time and we discarded many ideas and sketches throughout the process. Although the planning of the venue needed to consider the specific performance format, the audience, and the performance area for the performers, due to too many considerations in the early stage, we could not determine the specific performance format for each aspect. This resulted in our group wasting too much time on the visual planning of the venue, which delayed the normal progress of other aspects of the work.
Furthermore, the construction of the installation was also a challenge. We spent a lot of time trying the wrong way to fix the wires, which resulted in all the wires getting tangled together the day before the performance. We had to start over and use a different approach. If we had discovered the flaw in the wire-fixing method earlier, we would have had more time to create the visual installation.（Fig5. Installation processing)
Fig5. Installation processing
Reflection-Group coordination
Finally, I also served as the overall coordinator within the group. For video filming, I took full responsibility for the shooting of the film (Fig6.recording), recording the process of the group’s work on the lighting arrangement at the scene and other related matters. At the same time, I was also the voice performer for the part 1 performance (Fig7.voice performer)and participated in the choreography of the dance movements for part 3.
Fig6.recording
Fig7.voice performer
Conclusion
In general, this was my first time putting interactive performance into practice. Although our group consisted of only four people, we often felt that we were short-staffed. However, it was precisely because of this that our group’s work was not disjointed, and each member participated in every aspect of the work, rather than working separately. Of course, there are still many areas in our performance that need improvement, as pointed out by our mentor in the feedback given after our last formal performance. Specifically, the transitions between each stage of our performance need to be more closely linked in order to provide a better experience for the audience.
This was my first group assignment in my graduate studies. Difficulties and challenges kept arising during the process, and the group members worked together again and again to solve them successfully, and the final work was presented to everyone’s satisfaction. This group work has benefited me a lot, not only making us pay attention to integrating various resources, exchanging our ideas and organizing group members to share the work together, but also making me realize that hands-on practice, multidisciplinary exploration and cooperative communication are one of the important ways to learn, create and promote each other. In the following, I will share and elaborate my thoughts and reflection on this group work from my personal perspective.
My Role
This topic comes from my personal experience and I am happy to be able to resonate with you and make it happen together. In the invisible strings teamwork, I was mainly responsible for the video creation and editing of part1 and part3 of the performance, the installation design of part2, and the final video design of the performance. Since this is a large volume interactive performance, and we divided it into three parts to interpret it step by step, this requires the three parts to be visually unified as a whole. Because of the different materials, media and forms of visual presentation, and I have never taken over a project of such a large volume, it was difficult to unify the three parts visually. After a lot of case studies and expansion of ideas, we finally decided to use black and white as the main color, with red as the accent color, and also particles as one of our elements throughout the show.
In the shooting of the part1 and part3 videos, our group members again served as actors, directors, scenarists and action directors, which was a new challenge and experience for each of us. In the creation and editing of the video, since the design of the video affects the rhythm and mood of the whole performance, and also affects the subsequent design of the sound part, in this part of the creation process, I edited out the general mood of the video and provided it to my teammate Lerong, who composed the music according to the atmosphere and mood of the video, and then I continued to modify the details of the video according to the tune. It was an unforgettable and interesting experience for me to work on the same piece of work as two majors with the same goal in mind, from scattered footage to a complete video that could convey the concept of the project.
In part 2 of the installation, as the video footage shows, we encountered a lot of difficulties – there were so many ‘strings’ and bells that the installation was easily tied up in knots, even though we had spent a lot of time working on the installation. When it got so knotted that we couldn’t set up, we had to be ruthless and start from scratch, which was not only a test of physical strength but also of patience, but after the difficult ordeal, the moment when the installation was built was more than happy. Due to a temporary change in the construction method, the number of lines presented in the performance was half of what was originally planned, and in order to show the effect of the “daughter” being trapped in a cage of lines, more lines were needed to surround and submerge the actor, but the final presentation was much thinner and it was difficult to convey the feeling of the daughter struggling under high pressure, which was the most regrettable This is the most unfortunate aspect of the production.
For the final video edit, after discussion with my teammate Xiaozhuang Gao, I decided to narrate our entire project in the form of a group presentation, with documented images of the process throughout, and finally showing the full performance and audience feedback, so that the logic of the video would show both our thinking and inspiration and the creative project. As the process was very fragmented, it took a lot of time to select usable video footage, which made editing tricky, but in the end, through editing and tweaking, a complete video of the performance project emerged.
Conclusion
Overall, I am satisfied with the completion of the work, but what is more valuable is the process of continuous polishing over the semester, which will be a valuable asset for future creation and learning.
Finally, I would like to thank my three teammates for working together to produce this piece of work.
In this project, I was mainly responsible for the design and composition of the sound part, including recording sound, designing sound effects, composing music, and also exploring the techniques of sound-painting interaction, helping the group to realise a live performance using M5Stick sensors and Max to follow the actors’ movements in real-time to generate semi-automated music. This was also my first attempt at composing music that matched the atmospheric settings of the three stages of the project through midi, designed sound effects. I also designed the audio system to fit the performance area, ensuring that the audience could hear the abundant sound details during the performance live.
During the performance I was involved as the sound performer, controlling all the sound output effects, adjusting the balance between the multiple sound elements and mixing the sound for the scene in real time.
Technical Approach
For the recording, I recorded the sounds of many common household objects, such as floating curtains, torn fabric, squashed balloons, twisted threads and so on. These subtle sounds in the family life are often amplified in a depressingly intimate setting, and I used these sounds, either by equalising, layering, or adding effects such as delayed reverb, to create a soundscape that could be used alongside whispered and chanted vocals to build the character’s inner atmosphere.
When composing the three parts, I focus on using some hollow synth sounds as a backdrop and distorted piano as a melody to tell the story. In writing the second part of the dance, some powerful drums and crisp percussion build a disorienting and tense musical base.
Considering the human-sound interaction part of the performance, I changed part of the sound input of the first part of the performance to come from the performer in the scene, some teaching and reprimanding words became part of the sound performance, and these sounds affected the image, thus distorting it.
In the second part of the performance, many components of the music are derived from the generation of the performer’s movements, as she touches the strings around her to drive the bells to ring, using the M5Stick to capture her hand movements and compose the music in real time. The M5Stick is connected to Max under the same LAN using Arduino, and by receiving real-time yaw, roll, pitch, etc. data from the M5, scale them and trigger changing timbres, pitches and rhythms.
Wifi Setup
M5Stick and Max Connection
Reflections
The techniques I utilised were limited in sound composition, as it was my first time composing, and I needed to gain experience, making it challenging to write melodies with layers of evolving feeling. And when using Max for semi-algorithmic compositions, not adding more sound elements that fit the intense action of the characters, many objects still needed to be mastered. In the second stage of sound and picture interaction exploration, the attempt to use Max as a sender of output volume data caused the parameters in Processing to change, thus making the effect of visual particle dispersion not possible. In the future, I will focus more on learning techniques and exploring more possibilities for sound interaction.
Documents
Part 1 Mixing https://drive.google.com/file/d/1mFLVemaLbHSIs_EXReBWH9cNFqvpOlIB/view?usp=share_link
Part 2 Composing https://drive.google.com/file/d/1hJ__2rSRBj7yTwlH3oCR8tdIHqadInnu/view?usp=share_link
Part 2 Max Patchers https://drive.google.com/file/d/1xCIy2S-RgqypvWlqYt3FvaF5v551XoO7/view?usp=share_link
Part 3 Mixing https://drive.google.com/file/d/1UBbC0qdSQUK-sPXm9zSnHK_XTKBExe_W/view?usp=share_link
In this project, my main responsibility was the technical aspect of it. I completed the interactive design using Touch Designer and made adjustments to the Processing part while also experimenting with connecting Processing and Max via OSC. Also, I worked as the actress in our live performance.
In addition to my technical support role, I also acted in the experimental film as one of the daughter actress, playing the piano and providing hand acting for the mother’s hands. During the music recording, I played the piano and contributed some vocal humming sounds for the group.
Reflection
Throughout this course, I have been thrilled to collaborate with classmates from diverse backgrounds to create a multimedia interactive art piece. In my past creative endeavors, I’ve attempted to collaborate on an interactive lighting device based on Arduino. Unfortunately, due to the absence of team members with an art background, the final visual result fell short of my expectations. That’s why working on Invisible Strings has been so rewarding for me, as it has allowed me to finally bridge the gap between technology and art. I am grateful to my partners for providing me with a top-tier collaborative experience.
Of course, the creation of this project was not without its difficulties, and I encountered many challenges along the way.
At the beginning of the project, we spent a lot of time on the presentation of the project content and the expression of the theme. Although this made it possible for us to achieve a good visual effect, it also compressed the time we had to work on the technical aspects. Therefore, we had very little time to study the connection issues between Processing and Max. The night before the test, we spent six hours trying to transmit data from Max to Processing, but ultimately we only succeeded in transmitting data in the opposite direction. I think if we had more time, we might have been able to solve this problem.
Due to the shortage of manpower, I also had to simultaneously debug the Touch Designer, Processing, Kinect, and projector during the testing and actual performance. This meant that I did not have enough time to rehearse my dance before the performance, and as a result, the dance performance did not turn out as well as I had hoped.
Despite these challenges, the project’s theme of the piano resonated deeply with me. As someone who has studied piano for nearly ten years, I have had many arguments with my mother and have tried to avoid practicing. However, in this project, my past piano learning experience was valuable. The experimental video and sound design incorporated many of the pieces that I had practiced, causing me to reflect on my mother’s insistence on my learning and practice experience in the past.
I also tried using Touch Designer for the first time. Although we encountered many difficulties along the way, I learned a lot of knowledge. I hope to have the opportunity to continue exploring the use of Touch Designer in the future and create more projects.
During the creative design phase of this project, we searched extensively online for reference projects. In the end, we selected some of them and developed our own design work based on them.
Image References about Performance
In these reference images, the performer and the video presentation seem to be two independent entities. However, visually appealing effects may help the performer to better convey the musical presentation to the audience, for example, by using video to help viewers understand the expression of the music. Therefore, in our performance, the presentation of the video content may be abstract, but the interaction and musical presentation can be coordinated with the image to complete the story narration.
In reference image 2, the performer’s hand and the video image or hand shadow in the image correspond to each other, bringing some stunning visual experiences to the audience. We drew inspiration from this short experimental film and realized that when many objects are repeatedly stacked in the image, it can give the audience a sense of oppression. This is similar to what we want to present, where the mother gives pressure to her daughter. At the same time, through the shadows in this work, we came up with the idea of a simple interaction between the actor and the video material. The person’s shadow in the video and the actual performer exist in two different spaces but are connected. Based on these ideas, we completed the ideation process for the third part.
Video References about Interaction
In addition to image references, we also looked for related interactive installation art videos as our references.
Firstly, we looked at the works of Tomás Saraceno. In this work, Tomas used lines to form spider-web shapes, and when the audience touches the lines, they create sound.
We believe that lines can also be represented as strings of a piano. In our performance, the mother and daughter are connected by an invisible bond, which is visualized by a red thread. If a piano wants to make a sound, it needs the vibration of the strings, which is also the vibration of the lines. Therefore, in our design, in the second part of the work, we used black and white lines to represent the piano strings. However, unlike Tomas, it is difficult for us to make the strings produce sound. Therefore, we chose to tie solid bells to the lines in this part. When the audience interacts with the lines, the sound of the bells will appear. At the same time, the positional relationship of the actors also generates random sounds through Max as a supplement to the sound of this part.
We have found some reference videos on motion capture and particle effects. The interaction between dance and particle effects can bring great visual impact. However, if the visual of particles dominates the visual presentation, it may make the performer lose its functionality in our performance. Also, the interactive effect may not be better displayed, only playing a video made by particle system for the audience. In our design, to match the theme of the piano, the main visual color is black and white. In order to simplify the related visual effects, we design the color of the character’s particles as white, reducing the degree of dispersion. We also add sound control to the particles, but only make them dissipate when the sound reaches a fixed range.
We found that most of the Touch Designer cases we found online imported pre-produced music files and analyzed them to generate video changes. For example, in our reference case, it used chop to controller and noise controller to simulate the feeling of blood spreading from paper by analyzing the emotion and style of the music. However, according to our idea, we hope that the sound can generate real-time video effects changes. Therefore, we use the Device in module of the microphone instead of audio to achieve real-time sound analysis and video changes.
In the conception of Part 1, we hoped to present the beginning of the story of Autumn Sonata to the audience in a clear manner according to the development of the storyline. The team believed that short experimental film could help us to tell the story, so we shot an experimental film to showcase the fear and anxiety of the girl towards piano practice under the excessive expectations of her pianist mother. After completing the filming and editing of the short film, we imported the video into Touch Designer and searched for corresponding tutorials on the website to help us achieve the effect of garbled characters appearing on the screen according to changes in sound. The final result was the successful implementation of showing distorted images of the daughter in the video based on changes in sound, which conveyed the daughter’s anxiety under the excessive expectations of her mother.
Here we have three parts: Sound Analysis, Video Changes and Background Music
1) Sound Analysis
We used the Audio Device In controller as the source of the sound input, which is different from the video tutorials we referred to. Then we analyzed the relevant parameters of the sound according to the tutorials.
2) Video Changes
We divided the video editing into two identical parts: the main video changes and the daughter’s separate video in the mother-daughter frame. This was done so that we could later overlay the daughter’s video onto the right side of the original video to achieve the effect of the daughter’s part changes separately.
3) Background Music
We imported the background music file and added the Audio Device Out controller so that we could connect to the Audio Interface and provide the audience with better sound effects.
In addition to these three parts, we added controllers to better achieve the changes in certain parts of the video. We used code to set the controllers to only achieve the garbled effect in specific keyframe areas. We also set the reload shortcut code for the video and audio so that when we hit the “1” key on the keyboard, the sound and video will restart playing simultaneously.
In the second part of the performance, the music aims to connect with the feelings of the ‘daughter’, to express her emotional process from confusion to fear to struggle when she is in a repressed inner world. When conceiving the sound, I thought more about how to leave more of the sound creation to the performer.
Depending on the design of the performance setting, we set up strings with bells of different sizes tied around the performers, which, when touched by the performers as she is dancing, produce a random sound of bells that become part of the sound source that will form the music.
I intend to make part of the sound follow the dance movements of ‘daughter’ by capturing data on the locations of the hand movements in space, sonifying the data and using a smooth and flowing timbre to express the changes in the daughter’s state of mind.
In addition, a piece of background music is needed to set the mood for the whole performance, to underline the three internal states of the daughter and to provide a guide for the emotional development of the performance.
Composition process
Background Music
Composition project file in Reaper
Using crunchy piano to sketch out melodies, they go towards rendering a sense of perplexity. A single note with a soprano timbre is added to the piano, appearing every other bar to reflect the flowing piano sound.
On the bass side, three percussions form the basic rhythm, with the low, harmonic overtones playing the role of the kick in the drum kit, which, together with the tuned drums, sets the overall disorienting and tense atmosphere. Powerful drumming joins in as it moves into the second section, matching the daughter’s physical movements as she tries to break through the invisible wall of pressure.
In order to give the overall music a more oppressive, hollow sense, ethereal synthesizer sounds are used to play the harmonies.
Interactive sound
The M5stick is attached to the performer’s wrist and receives real-time data changes in Max in terms of yaw, roll, pitch, etc.
By limiting the threshold to the data collected, the range of variation is restricted to a usable interval using scale. When the data changes, the corresponding pitch and timbre are actuated and the rhythm of the sound is changed at the same time.
The actor can promote two states of sound, a detuned piano with a sense of dropping, and a sharp ripping sound.
Fig5. Installation processing
