• • Donger LIU:

1. 1. Visual director: overall aesthetic
   2. Illustration/animation design
   3. Handcrafts: set dressing, props making
   4. Adaption of scripts
   5. Portfolio producer, with study annotations
   6. Managing resources allocated to visual design

• • Honepei Cao

1. 1. Sound effects design
   2. Max interactive audio installation design
   3. Max particle design
   4. Lighting control
   5. Surround sound environment design
   6. Sound recording and editing

• • Qiyan Zhang:

• • Xianni Sui:

1. 1. Music Composition
   2. Sound Recording
   3. Music Instruments Performance
   4. Background Research
   5. Surround Music Effect design

• • Fangyi Qu:

1. 1. Sound effects design
   2. Touch designer design
   3. Surround sound environment design
   4. Sound recording and editing
   5. Max and Touch Designer interactive
   6. Risk assessment

Rough Script (15-20 minutes)

Set Dressing Mood References

Making screens out of fabric

Ballons imitate crouding fish / Fish lantern exhibition at Shanghai

Theatre Study

It is a theatre with a screen made of transparent strip curtains, named Hamlet. Directed by Thomas Ostermeier.

We would like to apply this idea in our performance to introduce audiences to the more significant stage site behind, letting them find the joy of exploration in this.

The installation Christian Boltanski by Marcos Rabello, shows the flickering light cast onto the wall and the shadows of little figures. We intend to cast audiences’ shadows in this way, transform them from spectatorship to being part of the performance and enhance participation. However, we need to address and experiment with the issue of audience members misunderstanding that they are blocking the projection and hiding from the lights. Our goal is to find effective ways to communicate to the audience that they are invited to participate.

Christian Boltanski

Experiment

A test of the projector inspired by Christian’s work

Interactive Installation Research

Explore the possibility of creating a soft screen by projector and curtain for audience members to walk through.

Use different coloured lights to highlight the main characters. Or consider, shadow show can also be a form.

Site Inspection

We aimed for an immersive stage experience, divided into three sections: the entry tunnel (orange), performance area (blue), and viewing area (green).

The entrance allows audiences to pass through a narrow, decorated passageway resembling the ocean environment of carp. The narrow space is decorated with balloons and coral to simulate the environment of carp in the ocean. We aim to let the audience understand the survival of the carp in the water through this crowded environment.

Upon entering the viewing area, projections in the performance area and on the sides will enhance immersion. Yet, details regarding projection equipment usage and room setup are still pending.

We propose to make an additional foreground curtain on the left side of the central performance area. This versatile curtain serves as both a projection surface and an entry/exit point for actors, enhancing stage hierarchy and facilitating actor movement.

Next, we will visit the designated rooms for field measurements to make necessary adjustments and refine the design for accuracy based on the room’s dimensions.

The Modified Design

We made modifications to the setup based on the data from the field measurements (right image).

As you can see, the room will be divided into a small square film theatre in the centre, with a ring of encircling passages around it. Red arrows show the viewing route, which will then be guided by props.

The image below is the previous set dressing idea, which references and illustrations are provided to illustrate the prototype.

Concept Illustration

To illustrate the conceptual design of our set, I have drawn a schematic storyboard to follow an audience member through the experience of our performance from a third-person perspective.

1.1) There will also be an interactive screen in the passway, adding engagement.

1) The entrance passway is decorated with balloons to simulate the fish crowding.

2) Fish lamps hanging overhead, introduce the context of their living conditions.

3) Follow the projection with crowding moving forward, walking through the curtain to see the next scene behind.

4) Follow the projection with crowding moving forward, walking through the curtain to see the next scene behind.

5) Follow the projection with crowding moving forward, walking through the curtain to see the next scene behind.

6) All lights go out, leaving only bottom lighting behind the audience. Their shadows merge with the projected aquarium.

Projected Video Demo

The projecton background mainly explains the environment in which the story develops. The projecton is mainly divided into three scenes: the river environment that creates the life of the carp, the rough river surface when the carp leaps the Dragon Gate, and the carp leaping through the empty door to enter the Heavenly Court.

As a traditonal Chinese story, the design of the background also incorporates Chinese style, hoping to reflect the cultural background of the story.

Projection Scene Demo 1, Rough river, painted with colour blocks, generated with AI

Projection Scene Demo 2, Reach the Heavenly Palace, painted with colour blocks, generated with AI

Projection Effect Test

1. 1. Chromatic Aberration Problem

During our testing of the place andprojector, we discovered that the colors did not match as anticipated, so that the mage did not appear as expected.

1. 2. Size adaptability between Projection andImage.

After many attempts, we found that the range and distance of projection equipment we currently use do not match the images in our works.

Response:

We would investigate the next steps, practice the application of the projector, adjust the scale of the image and be flexible in the choice of projection equipment according to practice.

Technical Details

Touch Designer for Interactve Dialogue and Background

Touch Designer Image Partcle

Scene One: crowding living conditions.

Particle effect: using particle technology to simulate the crowding in the river.

With Touch Designer, we could create streams of particles that look like fishes, which allows the audience to control the fish swimming directions by waving their arms.

Scene Three: toward the illusory dragon gate.

Particle effect: the particle gantry may change suddenly, releasing light particles that appear to attract but push the carp toward the trap.

Visual cue: as the carp get captured, the particles on stage shift from an orderly flow to an eruption of chaos, symbolising betrayal and pain.

Scene Five: fall into the aquarium

Particle effect: in the end, the particles in the scene can gradually disappear until the stage is completely darkened, leaving the audience to ponder.

Touch Designer Camera Hand Recogniton Partcle Interacton

Scene Five: fall into the aquarium

Partcle farewell: with the carp’s “last breath”, the audience can gently wave hands to make the partcle rise slowly, symbolising the departure of the carp’s soul and the end of the story.

Touch Designer Image Partcle

Scene 2: A large touch screen that acts as an interactve “canvas” where viewers can touch and “paint” on the screen, producing music and visuals with each of their touches.

1. Sound trigger: Max/MSP captures the sound produced by touch and processes and synthesizes sound effects according to different touch positons and intensifies. Each touch corresponds to a specific sound effect

2. Visual feedback: TouchDesigner receives audio signals generated by Max/MSP, and then generates real-tme visual effects based on audio propertes (such as pitch, volume, rhythm). For example, a bass might produce a blue ripple, and a treble might produce a red spark.

3. Audience partcipaton: The installaton will allow multple audience members to interact simultaneously to create a dynamic musical and visual artwork.

Technical implementaton

1. Touch input: Use the touch screen to capture the movement and touch of the audience.

2. Max/MSP: Analyze touch data and convert it into musical elements. Use audio synthesis and processing to create music that responds to touch.

3. Send Max/MSP music elements to TouchDesigner via OSC or MIDI.

4.TouchDesigner: Receive music data and create corresponding visual effects. Visual effects can be dynamic geometric shapes that map directly to musical features, or they can be more abstract visual representatons such as colour gradients or light and shadow flows.

5. Visual output: The visual effect is projected back onto the interactve canvas, synchronizing with the audience’s touch to form a complete audio-visual experience.

Max8 for Sound and Live Particles

General Description

In our project, the application of Max8 is primarily focused on audio interaction in Scene 1 and in creating the atmosphere for subsequent horror or mysterious scenes.

For the audio interaction, we may use Touch Designer in conjunction with ProTools for sound design to accomplish audio triggering. If it’s technically infeasible to achieve stable interaction between Max MIDI output and Touch Designer, we will resort to using only Touch Designer for image output. This output will be linked to Max8 via the same touch input to simultaneously trigger audio playback in Max.

This approach carries the risk of audio-video desynchronization, so our priority is to use a single software for production

Interective Sound Installation

For this part , we’ve created a demo of interactive sound effects using ProTools’ 5.1 surround sound format. This demo ranges from normal whispers with a slight reverb to distorted, heavily reverberated sounds resembling a dragon’s rumble, with added surround sound effects.

Given the space constraints in the Scene 1 corridor where the actual setup will take place, it might not be feasible to use surround sound speakers. To address this, we can render the surround sound content in binaural format, which is then outputted to headphones.

This approach allows for an immersive audio experience using a headphone amplifier, supporting multiple audience members simultaneously. This method ensures that the spatial quality of the sound is maintained, providing a realistic and engaging auditory experience that complements the visual and interactive elements of the scene.

https://drive.google.com/file/d/1eaQXOGu59cZgbWMIUCUkM2u7VTatm0as/view?usp=share_link (5.1 dialogue sound design

Possible lines:

“Beyond the gate, freedom calls.”

“Leap into your destiny, find your strength.”

“Embrace the unknown, for it holds your liberation.”

“In the leap, lies the path to transformation.”

“Dare the impossible, become the unimaginable.”

“Seek the gate, embrace your true form.”

“The gate beckons, promising a new dawn.”

“Cross the threshold, claim your power.”

“In the leap, your chains break.”

“The gate is the key, unlock your potential.”

Particle Animation

Another aspect is the video particle feature in Max8. Due to the complexity of implementing particle effects in Max8, with fewer plugins and preset options available, it requires inputting advanced functions for real-time calculation of particle positions.

After preliminary attempts, we managed to achieve the demonstrated particle movement effects and tested them using a portable projector. The actual effects look very impressive in a completely dark environment. We can use the reverse motion of particles, coupled with sound effects, to create a dispersing effect, simulating the inner turmoil of the protagonist Yue or creating an atmosphere. We are also exploring the use of OpenProcessing and other software, aiming to leverage a vast array of presets and public cases to create more visually appealing scenes.

ADJ MyDMX Controller for Lighting Control

The technology for controlling lighting is somewhat limited. After reviewing the lighting equipment available for rental on Bookit, it appears that the options for small-scale performances, especially in terms of lighting controllers, are quite restricted.

There is only one portable ADJ control box and a single console available. However, the console is frequently occupied for extended periods by various events. This situation limits the practical use to just two large LED lights that can be operated simultaneously. These two lights are mainly used for creating special effects, such as flashing lights to enhance the atmosphere during dramatic moments like the ‘Crossing the Dragon Gate’, or in Scene 5’s aquarium setting, where they project the audience’s shadows onto the screen.

Protools & Live for Immersive Sound Design

In the current design, Protools is used to create all sound effects and simulate the desired interactive effects. If some of the audio interactions mentioned earlier cannot be achieved, we can substitute them with live control through Protools and the mixing console as a backup plan. Considering that each act of the performance might have a fixed duration, pre-produced surround sound tracks can significantly reduce the risk of performance issues and allow for multiple tests to achieve the best effect. If we opt for a format with variable durations, we could consider using Ableton Live for looping the ambient sound tracks.

Protools Session Dialogue Design

Sound Environment Demo Audio Link: https://drive.google.com/file/d/1fp9mix5ti7y1oirtB_dtwCjSIWte-D0t/view?usp=share_link

Original music composition demo & Cultural Background

Link of demo:

This piece is an original theme music composition for our project, it is related to the background, the development of the storyline, and the overall thematic significance of the story. It is suggested that, the Chinese have long used images to convey auspicious wishes, such as plants, flowers, animals, birds and objects. (Laing, 2017). In this situation, fish express hopes for wealth because the pronunciation of the character for “fish” is identical to “abundance.”This is based on the tonal nature of the Chinese spoken language, one sound could have different meanings. In this context, carp were especially favored, because of their jumping ability, they were always associated with the idea of leaping through the Dragon Gate (an ancient Chinese myth) which symbolizes the attainment of success.

To furthermore, this original demo maintains plenty of Chinese traditional elements, for example Chinese pentatonic scales, as well as a part of Chinese traditional music instruments. The entire composition is structured into three distinct parts:

To begin with, the first section delineates the environment, spotlighting the protagonist (the carp) navigating its existence within the polluted waters, some Chinese traditional instruments like Instruments‘XUN’, evoked a somber and murky ambiance.

Additionally, the second segment delves into the character’s actions, emphasizing its leap over the dragon gate and the ensuing series of endeavors.

Next, the third part crescendos into the climax of the piece.Instruments such as the Suona, Pipa, and Bamboo Flute build upon the atmosphere layer by layer, culminating in a climactic finale. The unique technique of the Suona unveils the resolution of the story.

Reference list:

Laing, Ellen Johnston. “Carp and Goldfish as Auspicious Symbols and Their Representation in Chinese Popular Prints.” Arts Asiatiques 72 (2017): 97–109. http://www.jstor.org/stable/44656691.

Risk Assessment

1. 1. Insufficient budget: materials for the set dressing, such as iron frame and curtain to be purchased, which leads to a high cost. The soluton is to adjust the scope of the project without compromising the core objectves of the project, and examine the project budget to evaluate the unnecessary expenses that can be cut or replaced with something else at a lower price. This may include reducing certain features, simplifying the design, or changing the way the project is delivered.

1. 2. Shortage of equipment: we have to book the required equipment two weeks in advance before our performance, (Bookit could be booked up to two weeks in advance). Moreover, we should consider some alternatve equipment as plan B.

1. 3. Technical challenges: seek advice and help from relevant professionals when computer system problems lead to program errors. the connecton between devices is a big difficulty because of the need to use sensors and touchable screens to interact. For wirelessly connected devices such as ZIG SIM and touch designer, check and test the network environment in advance. For wired devices such as lights and computers, prepare spare cables in advance to prevent problems with the wiring.

1. 4. Insufficient project human resources: due to the shortage of site resources, we have to arrange more devices, arriving at the site in advance is necessary. Prioritse tasks (ensure that the most important tasks are assigned to a single person to complete, and one person can take charge of two tasks for simple or minor tasks). Find friends who are willing to help and invite them to help set up the scene once the tme is not enough.

Further Expectaton

To begin with, try out and solve problems with the practcality of all technologies, then adjust and improve them, including equipment, sound effects, music, lightng control and image display.

Secondly,to enhance project completon, team members created props, sound effects, lightng, and music, all while staying in touch, sharing ideas, and coordinatng the tmelines to ensure each component is finished on schedule.

Finally,stay vigilant for any project issues as they arise, and address them promptly to enhance workflow efficiency.