System Diagram-Hongpei

Current rent list:

1. Genelec speakers.        * 7
2. Genelec sub.                   * 1
3. Projector.                         *3
4. 6.5 to XLR cables.         *8
5. HDMI long cables.       *3
6. Black Curtain.                *1
7. GoPro                                *2
8. RME UCX.                        *1

System Test – March 23rd

During this week, we conducted our first system connectivity test in the Atrium. Owing to time constraints, we separately tested the interactive installations and the surround sound system components. The surround sound setup consists of five Genelec speakers and a subwoofer, connected to an RME UCXII sound card to receive audio from the computer.

Given the complexity of routing surround sound audio files directly from the computer, during actual performances, the main stage’s surround sound audio and projection videos will be played through a Protools output sound card and on the first expansion screen. Meanwhile, the television visuals will be displayed using QuickTime on a second expansion screen, ensuring there is no interference between the two systems.

Initially, we added spatial sound variations to a segment of pink noise. Through testers listening from the audience’s position, we achieved a relatively balanced effect by real-time adjustments of each speaker’s input level on the virtual mixing console. Subsequently, we played a completed surround sound segment, which resulted in a satisfactory performance outcome.

Following that, we tested the projection system and the Max interactive installation. Our initial idea was to create rock props fitting the scene, inviting audience members to jump around in the setting. We planned to capture the vibrations with a geophone and transform the visuals accordingly. However, in practice, we found the geophone to be extremely sensitive; it could detect strong sound pressure levels even from a distance or as soon as an audience member entered the door, making it challenging to set an activation threshold for the device.

Therefore, we intend to connect the system to percussion instruments, transforming the corridor scene into an interactive performance. When performers strike the drums, it will cause changes in the visuals. Additionally, we plan to optimize the interactive system to make it appear more diverse.

Video Design of Interactive Installation – Fangyi

1.Use ChatGPT and Runway, two tools, to create unique images and videos. Start with ChatGPT to generate an image.

This process begins by defining the type of image you want. Whether it’s a landscape, object, or abstract concept, you need to provide a detailed description, including the scene, objects, colors, and style. Once you have this concept, you can tell ChatGPT, and it will generate a unique image based on your description.
After the image is generated, the next step is to use this image as the basis for video production. This is where the Runway tool comes in. Runway is a powerful machine learning platform, specifically for image and video editing. First, you need to register an account and log in on Runway. Then, select an AI model that suits your project. Runway offers a variety of choices, each with different functions, and use various tools and settings to transform and enhance the image, such as adjusting colors, adding dynamic effects, or converting the image into a series of animated videos.

2.Add transitions in Final Cut

3.Creating font effects in Touch Designer:

• Create a text node, modify the font content. Use a merge node to combine different texts. Adjust the text’s position on the Y-axis. Increase the font thickness.
• Create and modify a grid:
Use a grid node to create a grid. Adjust the grid’s length, width, and height to make it rectangular. Add a noise node, connect it to the grid to create dynamic effects.
• Attach fonts to the grid:
Use attribute create to add normals and surface textures. Set up geometry, light, camera, and render nodes for rendering.
• Colors and materials:
Add materials and colors (such as pink) to the fonts. Optimize effects, including adjusting background color and resolution.
• Adding special effects:
Use nodes like feedback, blur, level, etc., to add visual effects. Combine different effects with a composite node.
• Creating a box background:
Create a box node and use geometry and render nodes to make a box background. Add dynamic effects with feedback, transfer, label, and composite nodes. Refine effects by adjusting parameters like scale and opacity.
• Final adjustments:
Adjust the position and angle of the lights. Adjust colors and transparency to ensure a cohesive overall effect. Add a bloom effect to increase brightness and vividness. Finally, add a breathing effect to make the box dynamically change.

Arrangement Table

Here’s a detailed arrangement of the visual part of the total show that I was responsible for. It’s worth noting that we’ll have a variety of forms incorporated into the performance (e.g. projections, puppet show, dance, etc.) but I don’t want to make the show overly complex and confusing, and I’d like to try to highlight only one of these forms per scene, e.g. in Scene 2 it will be mainly performed by the fish lantern, and in Scene 4 it will be more of a dance performance. I have highlighted the focus of each part of the performance in red in the table and tried my best to make it the centre of the performance.

Visual communication 2

Content details

Based on last week’s meeting, Jules’ suggestion was for us to cut back on unnecessary content so that the show tries not to be too complicated, and Andrew’s suggestion was for us to prioritise first and see if any elements can be cut back by evaluating them. Therefore, I only made some tables for evaluation.

Development – Integration attempts

We explored multimedia techniques to enrich the audience’s experience. Working closely with composer Xianni, we carefully synchronized visual elements with the music. Drawing upon a diverse array of artistic materials(first/second-hand videos) and technological tools like AI Generation and Cutcap software, I aimed to create a dynamic backdrop that enhances the music’s emotional impact. Through trial and error, the video production showcases the harmony of creative vision and technical skill. We worked together and joined forces to push forward to form a merger of music and video to make it look more harmonious.

After that, we tried a projection experiment with the borrowed projectors… the colours are excellent while there’s only one projector perfect fit.

To sum up, we are all confident about the performance so far, and further details will be confirmed at the rehearsal.

This substantial project requires significant time, materials, and patience. Given its pivotal role in a typical performance, I desired to grant it greater expressive freedom. Hence, I opted to create a fish-shaped lantern capable of mimicking the fluid motion of swimming fish.

Research

I studied the traditional fish lanterns from southern Chinese lantern festivals, tweaking them slightly. Opting for a classic red and green palette, I aimed for better visibility. Though crafting fish lanterns is intricate, I honed my skills through traditional sources and online tutorials. Having successfully made three in high school, my prior experience assured me of success this time around. Unlike in high school, though, this time I made mine movable.

【Video tutorial on how to make fish lanterns for the web】 https://www.bilibili.com/video/BV1hV4y1P75T/?share_source=copy_web&vd_source=aa7293d854a97fa85bdb47314a5b6fbe

Production

I started by sketching designs and structural blueprints, figuring out the ideal cane size and estimating the fish lantern’s dimensions. Choosing the fish’s colour and shape was next. For assembly, I divided the lantern into four sections, connecting them with wire hooks to create a swinging motion. This approach is rooted in the principle of energy conservation, akin to how a whip works.

Here are all the materials required for the production.

Production Video:

This part of the music comes from the second half of a previously uploaded music demo.This part, being the climax and the most important part of the performance was considered a priority to produce and finalize a relatively complete version of the show, so that rehearsals for this part of the show could be brought forward.

This part of the music is still composed in a combination of frame and character instruments, with the string section’s staccato as the frame, and considering the cultural background of the mythological story of the Carp Leaping Dragon Gate, the character instrument chosen is the traditional Chinese Suona, which possesses a special, extremely loud and clear tone, and has a special auditory impact, which is able to push the whole story to the climax.

It is suggested that, Suona’s affordance that “thy songs were small pieces, but thy tunes sounded louder” makes it adept at conveying explicitly strong and straightforward emotions.(Wang, 2022). As a traditional Chinese instrument, the Suona often takes on the role of expressing emotions of great joy and great sorrow in music, and is often used in ceremonies, such as weddings or funerals, so the emotions and cultural colors conveyed by the Suona are extremely intense and prominent.

Reference:

Touch Designer Text effect making

For the subtitles in the video of the first scene, I used touch designer to make dynamic art fonts. Compared with ordinary traditional subtitles display, such a production method makes the subtitles more flexible, interesting and artistic. I will explain a little about my production process and engineering.

Key steps

Font making: First create the text, adjust the font and position.
Effect production:
Use the “creep” tool to connect to a grid or sphere to achieve dynamic font effects.
Use “noise” to add noise, modify its parameters, and confirm the effect through “reset”.
Materials and colors: Add materials and colors (such as pink) and adjust the background materials. Set the font and box background to black, adjust the resolution and camera position.
Dynamic adjustment: Use “attribute create” to add surface textures.
Add dynamic visuals with effects such as “feedback” and “blur”.
Adjust parameters and transparency for increased visual comfort.
Post-processing:
Combine “composite” for image synthesis.
Enhance image brightness and adjust overall transparency and effect through “bloom”.
Adjust the edges and background frames, as well as the overall size and visual effects.
Effect adjustment: such as modifying color saturation, transparency adjustment, image composition Settings, etc.

This part is the first part of the official show that begins after the performance in the corridor. Because there are no lines in this section, the design of the music should include not only the depiction of the background, but also the content related to the protagonist’s moving line, so I chose to use an instrument to represent the protagonist – bamboo flute.

The soft and soothing piano playing column chords and breakdown chords serve as background depictions, easing the audience’s thoughts into the overall storytelling environment, while a bamboo flute solo appears as the main character’s moving line. At the same time, the bamboo flute solo has been given some acoustic treatment to match the movement of the actors and the live performance of the bamboo flute.

The possibility of including a live bamboo flute in the formal performance is also being considered, with two main influencing factors being taken into account: the integration of the live instrument into the overall acoustics of the ambient sound and the possibility of borrowing and setting up a microphone for instruments.

We have tried the possibility of using the microphones in the classroom for live performance amplification, which seems feasible initially, but we need to take into account the setup of the live sound system, and will need to make further attempts in the subsequent official rehearsals.

Interactive Stage Installation

On the afternoon of this Monday, the team members conducted their first rehearsal in Alison House, which did not include an actual scene. During the rehearsal, our primary focus was on refining the length information and presentation format of each scene, to set a rough timeline for sound production. Below is the preliminary stage equipment layout diagram discussed.

Participants will first pass through a meticulously arranged corridor, which will feature an interactive system that blends with the scene. This includes one to two passively triggered devices and one actively triggered device, with the specific triggering methods still under experimentation. One of the triggering devices implemented so far is a contact microphone. By affixing it to side walls, floors, or surfaces like whiteboards, we can effectively prevent the device from being triggered by people talking. For instance, it could be placed beside the floor, which is arranged to resemble a stream that the audience needs to cross. When the audience steps on it, the device will trigger, causing changes in the Max video. The reference for the Max project is as follows.

The basic principle involves using Video Blender to merge two videos in different ways, while utilizing the volume levels of the audio input in Max to trigger operations in Blender. When the audio input reaches a certain level, it outputs a “bang,” which triggers a switch. This allows for the automatic switching of the videos used in the blending process.

After passing through the corridor, the audience will enter the main stage. Firstly, there will be a pseudo-dynamic moving image on the TV behind the main screen, narrated in reverse along with a voiceover. The provisional script is as follows: “I remember crossing the Dragon Gate. I remember seeing the light. I remember hearing their voices. I remember, I’m almost unable to recall those times.”

Then, the audience will turn their attention to the performance on the main stage. The main stage is equipped with a 5.1 surround sound system and two screens. The front transparent screen is used to display some foreground elements, extending the stage’s visual field. After the performance, a pre-made closing video will be played on TV3 to guide the audience to leave.

Music Demo for Scene 1 & 5

We will prepare 7 parts of music for thie digital media project, six of them correspond to our story lines and scripts, as well as there will be a independent song after the sixth act.The use of Musical Instruments and the arrangement of harmonies will be associated with this Chinese mythological story and traditional Chinese culture.
It will illustrate the theme of our whole story——There are reflections on the busy, stressful lives of the modern day，as well as the thinking about the general blind competition.The song will be performed live after the storyline.
So far I’ve finished the music for the first act and the sicth act. The first part consists of piano, bamboo flute and chimes,something worth mentioning is，we have a special performance-related design for the dramatic expression of the main character——We want to use bamboo flutes to express the actions of the protagonist，and play bamboo flute live.
At the sixth part, emotions will be pushed to a climax, in this section we have prepared a live performance of the drum.

Scene one video production

The first video serves as a trailer for our entire performance, succinctly showcasing the content of the show within two minutes to heighten the audience’s anticipation. This video will be displayed on a TV screen at the front end of the corridor passage and will also be projected on the left side of the passage using a shower curtain as the screen. The shower curtain is thin, lightweight, and translucent, making it ideal for depicting underwater scenes.

I created the first scene of the video using AI. The overall process involved initially generating images through ChatGPT and then using these images to create videos in Runway. When generating images with ChatGPT, it is crucial to precisely describe the needed content and style to produce effective images. Based on our theme, I needed several images related to fish and dragon gates in different styles. For instance, describing the color and texture of carp, such as golden scales shimmering in the sunlight; depicting their vigorous leaping motion with splashing water; key environmental descriptions like turbulent rivers and distant dragon gates; and capturing an atmosphere filled with challenge and hope. These detailed descriptions help AI more accurately understand and achieve the visual style I want.

After confirming the images, I imported them into Runway and used the Gen-2 model to generate videos. A single image can develop into a video clip of about four seconds, which can then be extended. Tests show that to maintain the original content and style, a video generated from one image can be extended up to 16 seconds at most. Subsequently, I used Runway’s animation tools in the video editing interface, such as panning, zooming, and rotating, along with camera-related settings, to manually adjust the video montage based on the generated video.

This piece is an original theme music composition for our project, it is related to the background, the development of the storyline, and the overall thematic significance of the story. It is suggested that, the Chinese have long used images to convey auspicious wishes, such as plants, flowers, animals, birds and objects. (Laing, 2017). In this situation, fish express hopes for wealth because the pronunciation of the character for “fish” is identical to “abundance.”This is based on the tonal nature of the Chinese spoken language, one sound could have different meanings. In this context, carp were especially favored, because of their jumping ability, they were always associated with the idea of leaping through the Dragon Gate (an ancient Chinese myth) which symbolizes the attainment of success.

To furthermore, this original demo maintains plenty of Chinese traditional elements, for example Chinese pentatonic scales, as well as a part of Chinese traditional music instruments. The entire composition is structured into three distinct parts:

To begin with, the first section delineates the environment, spotlighting the protagonist (the carp) navigating its existence within the polluted waters, some Chinese traditional instruments like Instruments‘XUN’, evoked a somber and murky ambiance.

Additionally, the second segment delves into the character’s actions, emphasizing its leap over the dragon gate and the ensuing series of endeavors.

Next, the third part crescendos into the climax of the piece.Instruments such as the Suona, Pipa, and Bamboo Flute build upon the atmosphere layer by layer, culminating in a climactic finale. The unique technique of the Suona unveils the resolution of the story.

Reference list:

Laing, Ellen Johnston. “Carp and Goldfish as Auspicious Symbols and Their Representation in Chinese Popular Prints.” Arts Asiatiques 72 (2017): 97–109. http://www.jstor.org/stable/44656691.