Contribution

In this project, my main responsibility was the technical aspect of it. I completed the interactive design using Touch Designer and made adjustments to the Processing part while also experimenting with connecting Processing and Max via OSC. Also, I worked as the actress in our live performance.

In addition to my technical support role, I also acted in the experimental film as one of the daughter actress, playing the piano and providing hand acting for the mother’s hands. During the music recording, I played the piano and contributed some vocal humming sounds for the group.

Reflection

Throughout this course, I have been thrilled to collaborate with classmates from diverse backgrounds to create a multimedia interactive art piece. In my past creative endeavors, I’ve attempted to collaborate on an interactive lighting device based on Arduino. Unfortunately, due to the absence of team members with an art background, the final visual result fell short of my expectations. That’s why working on Invisible Strings has been so rewarding for me, as it has allowed me to finally bridge the gap between technology and art. I am grateful to my partners for providing me with a top-tier collaborative experience.

Of course, the creation of this project was not without its difficulties, and I encountered many challenges along the way.

At the beginning of the project, we spent a lot of time on the presentation of the project content and the expression of the theme. Although this made it possible for us to achieve a good visual effect, it also compressed the time we had to work on the technical aspects. Therefore, we had very little time to study the connection issues between Processing and Max. The night before the test, we spent six hours trying to transmit data from Max to Processing, but ultimately we only succeeded in transmitting data in the opposite direction. I think if we had more time, we might have been able to solve this problem.

Due to the shortage of manpower, I also had to simultaneously debug the Touch Designer, Processing, Kinect, and projector during the testing and actual performance. This meant that I did not have enough time to rehearse my dance before the performance, and as a result, the dance performance did not turn out as well as I had hoped.

Despite these challenges, the project’s theme of the piano resonated deeply with me. As someone who has studied piano for nearly ten years, I have had many arguments with my mother and have tried to avoid practicing. However, in this project, my past piano learning experience was valuable. The experimental video and sound design incorporated many of the pieces that I had practiced, causing me to reflect on my mother’s insistence on my learning and practice experience in the past.

I also tried using Touch Designer for the first time. Although we encountered many difficulties along the way, I learned a lot of knowledge. I hope to have the opportunity to continue exploring the use of Touch Designer in the future and create more projects.

Project Source File

Processing 

https://drive.google.com/drive/folders/1oyMpaftdqVVoR2Yg1Jd-NMoTqF9XqOA1?usp=sharing 

Touch Designer 

https://drive.google.com/drive/folders/1mTUTHKC1a6cvSnVqj6IKhykOgJr1NG63?usp=sharing 

Processing Test (Old Version) 

https://drive.google.com/drive/folders/1b57YIC2tuZJ0PA9p-rgbjUhLvx3UEDa5?usp=sharing 

Apr 26, 2023

During the creative design phase of this project, we searched extensively online for reference projects. In the end, we selected some of them and developed our own design work based on them.

Image References about Performance

In these reference images, the performer and the video presentation seem to be two independent entities. However, visually appealing effects may help the performer to better convey the musical presentation to the audience, for example, by using video to help viewers understand the expression of the music. Therefore, in our performance, the presentation of the video content may be abstract, but the interaction and musical presentation can be coordinated with the image to complete the story narration.

In reference image 2, the performer’s hand and the video image or hand shadow in the image correspond to each other, bringing some stunning visual experiences to the audience. We drew inspiration from this short experimental film and realized that when many objects are repeatedly stacked in the image, it can give the audience a sense of oppression. This is similar to what we want to present, where the mother gives pressure to her daughter. At the same time, through the shadows in this work, we came up with the idea of a simple interaction between the actor and the video material. The person’s shadow in the video and the actual performer exist in two different spaces but are connected. Based on these ideas, we completed the ideation process for the third part.

Video References about Interaction

In addition to image references, we also looked for related interactive installation art videos as our references.

1. Tomás Saraceno

https://www.xiaohongshu.com/discovery/item/63e603cb000000001303ec8b?app_platform=ios&app_version=7.75.2&share_from_user_hidden=true&type=video&xhsshare=WeixinSession&appuid=5c32b6ff0000000007022abe&apptime=1677365325

Firstly, we looked at the works of Tomás Saraceno. In this work, Tomas used lines to form spider-web shapes, and when the audience touches the lines, they create sound.

We believe that lines can also be represented as strings of a piano. In our performance, the mother and daughter are connected by an invisible bond, which is visualized by a red thread. If a piano wants to make a sound, it needs the vibration of the strings, which is also the vibration of the lines. Therefore, in our design, in the second part of the work, we used black and white lines to represent the piano strings. However, unlike Tomas, it is difficult for us to make the strings produce sound. Therefore, we chose to tie solid bells to the lines in this part. When the audience interacts with the lines, the sound of the bells will appear. At the same time, the positional relationship of the actors also generates random sounds through Max as a supplement to the sound of this part.

2. Particle References

https://www.xiaohongshu.com/explore/63ecca840000000013010084?app_platform=ios&app_version=7.84.1&share_from_user_hidden=true&type=video&xhsshare=WeixinSession&appuid=5f1490f0000000000101d9d5&apptime=1682533865

https://www.xiaohongshu.com/explore/630ee2470000000013038d05?app_platform=ios&app_version=7.84.1&share_from_user_hidden=true&type=video&xhsshare=WeixinSession&appuid=5f1490f0000000000101d9d5&apptime=1682533893

We have found some reference videos on motion capture and particle effects. The interaction between dance and particle effects can bring great visual impact. However, if the visual of particles dominates the visual presentation, it may make the performer lose its functionality in our performance. Also, the interactive effect may not be better displayed, only playing a video made by particle system for the audience. In our design, to match the theme of the piano, the main visual color is black and white. In order to simplify the related visual effects, we design the color of the character’s particles as white, reducing the degree of dispersion. We also add sound control to the particles, but only make them dissipate when the sound reaches a fixed range.

3. Touch Designer

https://www.xiaohongshu.com/explore/618bb7210000000001024964?app_platform=ios&app_version=7.84.1&share_from_user_hidden=true&type=video&xhsshare=WeixinSession&appuid=5f1490f0000000000101d9d5&apptime=1682533959

We found that most of the Touch Designer cases we found online imported pre-produced music files and analyzed them to generate video changes. For example, in our reference case, it used chop to controller and noise controller to simulate the feeling of blood spreading from paper by analyzing the emotion and style of the music. However, according to our idea, we hope that the sound can generate real-time video effects changes. Therefore, we use the Device in module of the microphone instead of audio to achieve real-time sound analysis and video changes.

Apr 26, 2023

Touch Designer

Background

In the conception of Part 1, we hoped to present the beginning of the story of Autumn Sonata to the audience in a clear manner according to the development of the storyline. The team believed that short experimental film could help us to tell the story, so we shot an experimental film to showcase the fear and anxiety of the girl towards piano practice under the excessive expectations of her pianist mother. After completing the filming and editing of the short film, we imported the video into Touch Designer and searched for corresponding tutorials on the website to help us achieve the effect of garbled characters appearing on the screen according to changes in sound. The final result was the successful implementation of showing distorted images of the daughter in the video based on changes in sound, which conveyed the daughter’s anxiety under the excessive expectations of her mother.

Process

We found some video and text tutorials online, the main references are:
https://www.youtube.com/watch?v=IFegKFjtj80
https://www.youtube.com/watch?v=rvAB3Rzh7CI

Here we have three parts: Sound Analysis, Video Changes and Background Music

1) Sound Analysis

We used the Audio Device In controller as the source of the sound input, which is different from the video tutorials we referred to. Then we analyzed the relevant parameters of the sound according to the tutorials.

2) Video Changes

We divided the video editing into two identical parts: the main video changes and the daughter’s separate video in the mother-daughter frame. This was done so that we could later overlay the daughter’s video onto the right side of the original video to achieve the effect of the daughter’s part changes separately.

3) Background Music

We imported the background music file and added the Audio Device Out controller so that we could connect to the Audio Interface and provide the audience with better sound effects.

In addition to these three parts, we added controllers to better achieve the changes in certain parts of the video. We used code to set the controllers to only achieve the garbled effect in specific keyframe areas. We also set the reload shortcut code for the video and audio so that when we hit the “1” key on the keyboard, the sound and video will restart playing simultaneously.

Apr 26, 2023