Any views expressed within media held on this service are those of the contributors, should not be taken as approved or endorsed by the University, and do not necessarily reflect the views of the University in respect of any particular issue.
The final performance went smoothly as a whole. We arranged the venue and set up the system one day in advance. On the day of the performance, we quickly completed the remaining layout work and then carried out three overall rehearsals, optimizing the details in the process of each rehearsal. For example, the approach time and scheduling of the fish, adjusting the audio pressure level to a reasonable range, and most importantly, we re-fixed our main projector. In the process of rehearsal, the small fish actor accidentally stepped on the line of the main projector when going up and down the stage, which caused the projector to be disconnected. We immediately reinforced the support and connection line placed on the projector to ensure that there would be no problems during the performance.
For the part I made, one of them was the touchdesigner line effect displayed on the side screen. My original plan was to use this style of screen display in the demon whispering part and the big fish leaping over the gantry to reflect the sense of terror, but I ignored the harmony between this screen and the screen displayed on the main screen, and their colors did not match well. If I want the color of the screen to be related to the home screen, the improvements I think about are: Install a blue atmosphere light at the bottom of the screen, because I want to make some live performance images on this screen rather than set videos in advance that can be played directly. However, I am not very familiar with the color display of touch designer, so this is the solution I can think of at present.
In order to enrich the main stage, I plan to add a secondary screen on the left side of the stage, and use the unique visual effect created by touch designer on the secondary screen to assist the main stage performance. The original plan was to set up a camera in front of the main stage, which could display the movement and outline of the dancer’s body on the screen when the dancer appeared in the dance. However, during the test, I found that the light on the main stage was not bright enough, and TouchDesigner could not recognize the characters on the stage, so we planned to set up LED lights on both sides of the stage. The light was heavy and we were worried about the safety risks of not hanging well, so I ended up using this interactive part for the main stage with the devil whispering part and the fish struggling as they jumped the gantry.
Since this idea was planned at a later stage, it was too late to prepare the Kinect camera. Instead, I sought ZigSim, an app on the mobile phone, which can use the mobile phone as a sensor and peripheral camera to transmit video signals from the mobile phone to TouchDesigner through the network.
The cache is used inTouchDesigner to save frames in the video and to obtain motion tracks through difference detection. Next, the motion contours are enhanced by adjusting brightness, contrast, and edge detection. To eliminate blemishes in the image, use a high filter size for blurring. Adjusting the minimum value of the Chroma Key can reduce the number of black pixels in the image and optimize the Settings for different environments.
For the subtitles in the video of the first scene, I used touch designer to make dynamic art fonts. Compared with ordinary traditional subtitles display, such a production method makes the subtitles more flexible, interesting and artistic. I will explain a little about my production process and engineering.
Key steps
Font making: First create the text, adjust the font and position. Effect production:
Use the “creep” tool to connect to a grid or sphere to achieve dynamic font effects.
Use “noise” to add noise, modify its parameters, and confirm the effect through “reset”. Materials and colors: Add materials and colors (such as pink) and adjust the background materials. Set the font and box background to black, adjust the resolution and camera position. Dynamic adjustment: Use “attribute create” to add surface textures.
Add dynamic visuals with effects such as “feedback” and “blur”.
Adjust parameters and transparency for increased visual comfort. Post-processing:
Combine “composite” for image synthesis.
Enhance image brightness and adjust overall transparency and effect through “bloom”.
Adjust the edges and background frames, as well as the overall size and visual effects.
Effect adjustment: such as modifying color saturation, transparency adjustment, image composition Settings, etc.
The first video serves as a trailer for our entire performance, succinctly showcasing the content of the show within two minutes to heighten the audience’s anticipation. This video will be displayed on a TV screen at the front end of the corridor passage and will also be projected on the left side of the passage using a shower curtain as the screen. The shower curtain is thin, lightweight, and translucent, making it ideal for depicting underwater scenes.
I created the first scene of the video using AI. The overall process involved initially generating images through ChatGPT and then using these images to create videos in Runway. When generating images with ChatGPT, it is crucial to precisely describe the needed content and style to produce effective images. Based on our theme, I needed several images related to fish and dragon gates in different styles. For instance, describing the color and texture of carp, such as golden scales shimmering in the sunlight; depicting their vigorous leaping motion with splashing water; key environmental descriptions like turbulent rivers and distant dragon gates; and capturing an atmosphere filled with challenge and hope. These detailed descriptions help AI more accurately understand and achieve the visual style I want.
After confirming the images, I imported them into Runway and used the Gen-2 model to generate videos. A single image can develop into a video clip of about four seconds, which can then be extended. Tests show that to maintain the original content and style, a video generated from one image can be extended up to 16 seconds at most. Subsequently, I used Runway’s animation tools in the video editing interface, such as panning, zooming, and rotating, along with camera-related settings, to manually adjust the video montage based on the generated video.
In modern society, due to lack of knowledge, generation gap, busy pace of life and other reasons, many parents may have difficulty understanding or recognizing depression in their children. This story is a real situation in my friend’s real life. She suffered from depression for a long time, but was not understood by her family, which led to her becoming more and more serious and even attempted suicide. In the process of accompanying her, I learned about the physical and psychological status of real depression patients, and their needs. Through this project, I want to let more people know the inner world of depressed patients, know more about them, and give them appropriate help when they need it.
Storyline
The main character: a young girl suffering from depression, feeling lonely and not understood.
Background: Family members did not understand her condition and saw no reason for her to feel unhappy, which created additional stress on her.
Dream and reality: The little girl experiences terrifying dreams in her sleep, dreaming that she is being hunted, reflecting her inner fear and struggle.
Double ending:
(1) Dark self: The dream self represents the dark side of her heart, and her jumping off the building may symbolize being consumed by the darkness of her heart.
(2) The victory of the light: On the other hand, this can also be interpreted as her victory over the inner darkness, thus achieving inner liberation.
Stage design
Visual design: The combination of dynamic images, words, symbols to express the change of mental state.
Interactive elements: The audience can participate in the story through touch screens or mobile devices and influence the development of the story.
Technology application
（1）Tactile and visual interactive experience
Scene design: Through Touch Designer, create a series of interactive visual effects that symbolize different aspects of mental illness. For example, a scene can be designed to represent psychological stress and anxiety, using visual elements of instability and chaos to simulate the feelings brought about by mental illness.
Audience interaction: The use of camera capture and gesture recognition technology allows the audience to interact with visual elements on the stage through gestures, such as changing visual patterns or colors by waving, symbolizing the struggle and interaction with mental illness.
(2) Emotional trigger sound effects
Sound Design: Use Max/MSP to design dynamic sound effects that reflect the emotional changes in the performance. For example, in scenes depicting depression or anxiety, the sound effects can gradually change from calm to tension and unease. Combined with audience interaction, touch or movement can trigger different sound effects, enhancing the audience’s sense of immersion. For example, the audience’s movements can affect volume, rhythm, or tone, making them part of the performance.
Risk assessment
Emotional trigger: The performance may touch on sensitive topics and have an impact on the audience with mental health issues. Provide emotional trigger warnings, prepare mental health support resources at the show.
Misunderstanding and misdirection: Performances can be misunderstood and send the wrong message. Design content carefully to avoid misleading information and work with mental health professionals to ensure accuracy.
