We had a looping workshop today to play with the sound and music. We learnt how to use co-recorded voiceovers and multiple tracks to create looped noise music, coordinated with video clips on the day. Following Andrew’s guidance, we created various visual scores and gathered textures from nature.

During the workshop, we tested various materials and challenging to coordinate each other’s parts. Here is the documentation of the production:

By integrating looping techniques, we generate intricate textures and rhythms that harmonize with the visual elements and the overall ambience of the performance. Moreover, it aids in establishing and sustaining a unified musical framework during the entirety of the performance, facilitating smooth transitions between different segments or movements. This consistency enhances the overall fluidity and cohesion of the performance, resulting in a more immersive and captivating experience for the audience. In essence, the looping music workshop acts as a collaborative and interactive component of the performance, enriching the creation of a compelling artistic experience that surpasses conventional boundaries between music and visual art. Therefore, we might consider using the looping technique for our live performance.

It’s good to see the feedback on our initial submission, and I have compiled a summary of the project’s progress so far. This blog describes our ideas, experiments, solutions and decisions in response to the tutor’s comment.

Initially, I’d like to emphasise the vital role that interactive media plays in our performance, as we strive to swiftly immerse the audience. Given the brevity of the actual performance, engaging the audience beforehand becomes invaluable as a warm-up tactic.

Reflection 1:

The primary format still requires clarification—what is the format we intend to present to the audience? Essentially, we plan to convey the content primarily through live performances by puppet shows and video projection at the main stage, complemented by captivating music and set decoration. Ideally, we aim to incorporate a blend of video and other formats displayed on the screen, such as shadow play or puppet shows. Our goal is to enhance the visual presentation and achieve an extravagant effect, thereby providing a relatively varied show to deliver an immersive experience. Moreover, we are currently investigating the potential of generating animations with the aid of Artificial Intelligence. There exist various methods and tools for production, and below are my experiments:

• Use TensorFlow’s pretrained style transfer model to transfer the style from an image to the target content image. The model inference was performed on Google Colab, with minor modification on the code to accept personalized inputs(my drawings). After several tests, I find that the style image is suppose to be photo with white spaces, which is a drawback. The modified code was generated by ChatGPT. I assume that the generated images could be put into our projection video. https://www.tensorflow.org/hub/tutorials/tf2_arbitrary_image_stylization

• Producing numerous similar images within Midjourney and assembling them into moving images or frame-by-frame animations. （AnimateDiff）

• Inputting the images to Runway Gen-2 to generate animations of up to 4 seconds. Nevertheless, challenges include the high cost and the complexity of ensuring stability and dynamism in AI-generated videos. As evident, the carps exhibit very gradual and subtle movements in the video, making it challenging to accentuate the visual impact.
  
• Overlaying the drawings, photographs, animations and videos captured on-site to construct a blended video by CapCut.

Reflection 2:

More importantly, it’s valuable to ponder the moral stance we aim to convey, urging resistance against our modern urges for competition, and emphasizing that fulfilment originates internally. It’s essential that the performance doesn’t simply conclude abruptly but instead offers a narrative resolution for the audience. Consider integrating elements reflecting the pervasive theme of today’s competitive lifestyle into every aspect of the performance narrative. Firstly, how to visualise the concept to the audience: how can they perceive and comprehend the theme of peer pressure within the story? One approach to tackle this is via illustrating the intense competition, highlighting the scene where carp scramble to jump. We might apply abstract ways and intense drum beats we can take to express this tension, for instance, viewing through the fish’s perspective, the splashes of water from their leaps into the world evoke a sense of rapid transformation of everything, accelerating heartbeats, and tension with a psychedelic effect. Secondly, this could involve incorporating specific sounds reminiscent of social media or email notifications into the soundscapes, complementing the narrative. We’ve now got some voice-over sound composition, infusing it with a sense of dark urgency that aligns with the overarching concept.

Reflection 3:

We’re all working hard on our own, but now it’s key to blend these elements and see how they fit in a performance, along with our physical presence. Thereby, the next step is to determine how the visuals and music of each scene work together, dedicated to a complete narrative performance –what will the audience be able to understand the narrative as being performed live? I’ll write in detail listing how the visuals and music work together for each scene.

Reflection 4:

Things to consider, listed by Andrew:

• Consider what can be performed live with real input when performing this can be particularly engaging for an audience – e.g. the voiceover for part 1 or later voiceovers.
• Lighting control – it will be important for you to still use lights but it may be necessary to use simpler lamps – you may consider building simple control dimmers/devices.
• Linking live instruments to the control of the visual story – local microphones having video implication or effect – link the live performance to the video performance/visual textures.
• The theatre set studies you highlight with soft screen projectors, backlighting and shadow puppetry are all very intriguing – it might be interesting to combine front projection with a scene or some narrative device along with backlit shadow puppetry – maybe using a localised focused light such as a torch to be able to layer different elements to your performance.
• Think very carefully about audience choreography/orchestration – how will they know to move through a space – when they need to linger, when they need to move on – this can easily be done through lighting or where audio is coming from but does need consideration to avoid needing to usher people.
• May a clear plan for each scene with what is happening – what effects, set dressing, audience location, audio is occurring and where is it coming from – this will be a working document to enable you to keep track of where you are at developing the performance.

Here are a few additional suggestions: conduct more experiments!  Don’t need to invest heavily in purchases; instead, utilize waste materials and plastics to craft props.

DMSP 2024 Performance – Submission 1

Links to PDF: Jump_Carp_Jump_DMSP_Performance_Submission1

What

Why

We aim to reflect on the rat race phenomenon in this involuted generation, so as to rebel against this culture of involution.
Nowadays, we face escalating peer pressures in crowded environments. However, sometimes, individuals get ahead routinely in a pointless pursuit and engage in aimless competition with the masses.
We adapted the story of the carp, revealing our sadness of being trapped in the rat race.

How

We would set an intimate theatre for the audience, mainly using projectors, lighting and sounds to show the performance (set dressing/ sound design/interactive props).
Research and experiments will be the central practice surrounding the project:

– script writing

– storyboard and visual design

– props making and animation output

– projector setting and lighting tests, location decisions

– music demo

– sound environment

– interactive sound installation

Member’s Responsibilities

Illustration

Set Dressing Mood References

Making screens out of fabric

Ballons imitate crouding fish / Fish lantern exhibition at Shanghai

Theatre Study

It is a theatre with a screen made of transparent strip curtains, named Hamlet. Directed by Thomas Ostermeier.

We would like to apply this idea in our performance to introduce audiences to the more significant stage site behind, letting them find the joy of exploration in this.

The installation Christian Boltanski by Marcos Rabello, shows the flickering light cast onto the wall and the shadows of little figures. We intend to cast audiences’ shadows in this way, transform them from spectatorship to being part of the performance and enhance participation. However, we need to address and experiment with the issue of audience members misunderstanding that they are blocking the projection and hiding from the lights. Our goal is to find effective ways to communicate to the audience that they are invited to participate.

Christian Boltanski

Experiment

A test of the projector inspired by Christian’s work

Interactive Installation Research

Explore the possibility of creating a soft screen by projector and curtain for audience members to walk through.

Use different coloured lights to highlight the main characters. Or consider, shadow show can also be a form.

Site Inspection

We aimed for an immersive stage experience, divided into three sections: the entry tunnel (orange), performance area (blue), and viewing area (green).

The entrance allows audiences to pass through a narrow, decorated passageway resembling the ocean environment of carp. The narrow space is decorated with balloons and coral to simulate the environment of carp in the ocean. We aim to let the audience understand the survival of the carp in the water through this crowded environment.

Upon entering the viewing area, projections in the performance area and on the sides will enhance immersion. Yet, details regarding projection equipment usage and room setup are still pending.

We propose to make an additional foreground curtain on the left side of the central performance area. This versatile curtain serves as both a projection surface and an entry/exit point for actors, enhancing stage hierarchy and facilitating actor movement.

Next, we will visit the designated rooms for field measurements to make necessary adjustments and refine the design for accuracy based on the room’s dimensions.

The Modified Design

We made modifications to the setup based on the data from the field measurements (right image).

As you can see, the room will be divided into a small square film theatre in the centre, with a ring of encircling passages around it. Red arrows show the viewing route, which will then be guided by props.

The image below is the previous set dressing idea, which references and illustrations are provided to illustrate the prototype.

Concept Illustration

To illustrate the conceptual design of our set, I have drawn a schematic storyboard to follow an audience member through the experience of our performance from a third-person perspective.

1.1) There will also be an interactive screen in the passway, adding engagement.

1) The entrance passway is decorated with balloons to simulate the fish crowding.

2) Fish lamps hanging overhead, introduce the context of their living conditions.

3) Follow the projection with crowding moving forward, walking through the curtain to see the next scene behind.

4) Follow the projection with crowding moving forward, walking through the curtain to see the next scene behind.

5) Follow the projection with crowding moving forward, walking through the curtain to see the next scene behind.

6) All lights go out, leaving only bottom lighting behind the audience. Their shadows merge with the projected aquarium.

Projected Video Demo

The projecton background mainly explains the environment in which the story develops. The projecton is mainly divided into three scenes: the river environment that creates the life of the carp, the rough river surface when the carp leaps the Dragon Gate, and the carp leaping through the empty door to enter the Heavenly Court.

As a traditonal Chinese story, the design of the background also incorporates Chinese style, hoping to reflect the cultural background of the story.

Projection Scene Demo 1, Rough river, painted with colour blocks, generated with AI

Projection Scene Demo 2, Reach the Heavenly Palace, painted with colour blocks, generated with AI

Projection Effect Test

1. 1. Chromatic Aberration Problem

During our testing of the place andprojector, we discovered that the colors did not match as anticipated, so that the mage did not appear as expected.

1. 2. Size adaptability between Projection andImage.

After many attempts, we found that the range and distance of projection equipment we currently use do not match the images in our works.

Response:

We would investigate the next steps, practice the application of the projector, adjust the scale of the image and be flexible in the choice of projection equipment according to practice.

Technical Details

Touch Designer for Interactve Dialogue and Background

Touch Designer Image Partcle

Scene One: crowding living conditions.

Particle effect: using particle technology to simulate the crowding in the river.

With Touch Designer, we could create streams of particles that look like fishes, which allows the audience to control the fish swimming directions by waving their arms.

Scene Three: toward the illusory dragon gate.

Particle effect: the particle gantry may change suddenly, releasing light particles that appear to attract but push the carp toward the trap.

Visual cue: as the carp get captured, the particles on stage shift from an orderly flow to an eruption of chaos, symbolising betrayal and pain.

Scene Five: fall into the aquarium

Particle effect: in the end, the particles in the scene can gradually disappear until the stage is completely darkened, leaving the audience to ponder.

Touch Designer Camera Hand Recogniton Partcle Interacton

Scene Five: fall into the aquarium

Partcle farewell: with the carp’s “last breath”, the audience can gently wave hands to make the partcle rise slowly, symbolising the departure of the carp’s soul and the end of the story.

Touch Designer Image Partcle

Scene 2: A large touch screen that acts as an interactve “canvas” where viewers can touch and “paint” on the screen, producing music and visuals with each of their touches.

1. Sound trigger: Max/MSP captures the sound produced by touch and processes and synthesizes sound effects according to different touch positons and intensifies. Each touch corresponds to a specific sound effect

2. Visual feedback: TouchDesigner receives audio signals generated by Max/MSP, and then generates real-tme visual effects based on audio propertes (such as pitch, volume, rhythm). For example, a bass might produce a blue ripple, and a treble might produce a red spark.

3. Audience partcipaton: The installaton will allow multple audience members to interact simultaneously to create a dynamic musical and visual artwork.

Technical implementaton

1. Touch input: Use the touch screen to capture the movement and touch of the audience.

2. Max/MSP: Analyze touch data and convert it into musical elements. Use audio synthesis and processing to create music that responds to touch.

3. Send Max/MSP music elements to TouchDesigner via OSC or MIDI.

4.TouchDesigner: Receive music data and create corresponding visual effects. Visual effects can be dynamic geometric shapes that map directly to musical features, or they can be more abstract visual representatons such as colour gradients or light and shadow flows.

5. Visual output: The visual effect is projected back onto the interactve canvas, synchronizing with the audience’s touch to form a complete audio-visual experience.

Max8 for Sound and Live Particles

General Description

In our project, the application of Max8 is primarily focused on audio interaction in Scene 1 and in creating the atmosphere for subsequent horror or mysterious scenes.

For the audio interaction, we may use Touch Designer in conjunction with ProTools for sound design to accomplish audio triggering. If it’s technically infeasible to achieve stable interaction between Max MIDI output and Touch Designer, we will resort to using only Touch Designer for image output. This output will be linked to Max8 via the same touch input to simultaneously trigger audio playback in Max.

This approach carries the risk of audio-video desynchronization, so our priority is to use a single software for production

Interective Sound Installation

For this part , we’ve created a demo of interactive sound effects using ProTools’ 5.1 surround sound format. This demo ranges from normal whispers with a slight reverb to distorted, heavily reverberated sounds resembling a dragon’s rumble, with added surround sound effects.

Given the space constraints in the Scene 1 corridor where the actual setup will take place, it might not be feasible to use surround sound speakers. To address this, we can render the surround sound content in binaural format, which is then outputted to headphones.

This approach allows for an immersive audio experience using a headphone amplifier, supporting multiple audience members simultaneously. This method ensures that the spatial quality of the sound is maintained, providing a realistic and engaging auditory experience that complements the visual and interactive elements of the scene.

https://drive.google.com/file/d/1eaQXOGu59cZgbWMIUCUkM2u7VTatm0as/view?usp=share_link (5.1 dialogue sound design

Possible lines:

“Beyond the gate, freedom calls.”

“Leap into your destiny, find your strength.”

“Embrace the unknown, for it holds your liberation.”

“In the leap, lies the path to transformation.”

“Dare the impossible, become the unimaginable.”

“Seek the gate, embrace your true form.”

“The gate beckons, promising a new dawn.”

“Cross the threshold, claim your power.”

“In the leap, your chains break.”

“The gate is the key, unlock your potential.”

Particle Animation

Another aspect is the video particle feature in Max8. Due to the complexity of implementing particle effects in Max8, with fewer plugins and preset options available, it requires inputting advanced functions for real-time calculation of particle positions.

After preliminary attempts, we managed to achieve the demonstrated particle movement effects and tested them using a portable projector. The actual effects look very impressive in a completely dark environment. We can use the reverse motion of particles, coupled with sound effects, to create a dispersing effect, simulating the inner turmoil of the protagonist Yue or creating an atmosphere. We are also exploring the use of OpenProcessing and other software, aiming to leverage a vast array of presets and public cases to create more visually appealing scenes.

ADJ MyDMX Controller for Lighting Control

The technology for controlling lighting is somewhat limited. After reviewing the lighting equipment available for rental on Bookit, it appears that the options for small-scale performances, especially in terms of lighting controllers, are quite restricted.

There is only one portable ADJ control box and a single console available. However, the console is frequently occupied for extended periods by various events. This situation limits the practical use to just two large LED lights that can be operated simultaneously. These two lights are mainly used for creating special effects, such as flashing lights to enhance the atmosphere during dramatic moments like the ‘Crossing the Dragon Gate’, or in Scene 5’s aquarium setting, where they project the audience’s shadows onto the screen.

Protools & Live for Immersive Sound Design

In the current design, Protools is used to create all sound effects and simulate the desired interactive effects. If some of the audio interactions mentioned earlier cannot be achieved, we can substitute them with live control through Protools and the mixing console as a backup plan. Considering that each act of the performance might have a fixed duration, pre-produced surround sound tracks can significantly reduce the risk of performance issues and allow for multiple tests to achieve the best effect. If we opt for a format with variable durations, we could consider using Ableton Live for looping the ambient sound tracks.

Protools Session Dialogue Design

Sound Environment Demo Audio Link: https://drive.google.com/file/d/1fp9mix5ti7y1oirtB_dtwCjSIWte-D0t/view?usp=share_link

Original music composition demo & Cultural Background

Link of demo:

This piece is an original theme music composition for our project, it is related to the background, the development of the storyline, and the overall thematic significance of the story. It is suggested that, the Chinese have long used images to convey auspicious wishes, such as plants, flowers, animals, birds and objects. (Laing, 2017). In this situation, fish express hopes for wealth because the pronunciation of the character for “fish” is identical to “abundance.”This is based on the tonal nature of the Chinese spoken language, one sound could have different meanings. In this context, carp were especially favored, because of their jumping ability, they were always associated with the idea of leaping through the Dragon Gate (an ancient Chinese myth) which symbolizes the attainment of success.

To furthermore, this original demo maintains plenty of Chinese traditional elements, for example Chinese pentatonic scales, as well as a part of Chinese traditional music instruments. The entire composition is structured into three distinct parts:

To begin with, the first section delineates the environment, spotlighting the protagonist (the carp) navigating its existence within the polluted waters, some Chinese traditional instruments like Instruments‘XUN’, evoked a somber and murky ambiance.

Additionally, the second segment delves into the character’s actions, emphasizing its leap over the dragon gate and the ensuing series of endeavors.

Next, the third part crescendos into the climax of the piece.Instruments such as the Suona, Pipa, and Bamboo Flute build upon the atmosphere layer by layer, culminating in a climactic finale. The unique technique of the Suona unveils the resolution of the story.

Reference list:

Laing, Ellen Johnston. “Carp and Goldfish as Auspicious Symbols and Their Representation in Chinese Popular Prints.” Arts Asiatiques 72 (2017): 97–109. http://www.jstor.org/stable/44656691.

Risk Assessment

1. 1. Insufficient budget: materials for the set dressing, such as iron frame and curtain to be purchased, which leads to a high cost. The soluton is to adjust the scope of the project without compromising the core objectves of the project, and examine the project budget to evaluate the unnecessary expenses that can be cut or replaced with something else at a lower price. This may include reducing certain features, simplifying the design, or changing the way the project is delivered.

1. 2. Shortage of equipment: we have to book the required equipment two weeks in advance before our performance, (Bookit could be booked up to two weeks in advance). Moreover, we should consider some alternatve equipment as plan B.

1. 3. Technical challenges: seek advice and help from relevant professionals when computer system problems lead to program errors. the connecton between devices is a big difficulty because of the need to use sensors and touchable screens to interact. For wirelessly connected devices such as ZIG SIM and touch designer, check and test the network environment in advance. For wired devices such as lights and computers, prepare spare cables in advance to prevent problems with the wiring.

1. 4. Insufficient project human resources: due to the shortage of site resources, we have to arrange more devices, arriving at the site in advance is necessary. Prioritse tasks (ensure that the most important tasks are assigned to a single person to complete, and one person can take charge of two tasks for simple or minor tasks). Find friends who are willing to help and invite them to help set up the scene once the tme is not enough.

Further Expectaton

To begin with, try out and solve problems with the practcality of all technologies, then adjust and improve them, including equipment, sound effects, music, lightng control and image display.

Secondly,to enhance project completon, team members created props, sound effects, lightng, and music, all while staying in touch, sharing ideas, and coordinatng the tmelines to ensure each component is finished on schedule.

Finally,stay vigilant for any project issues as they arise, and address them promptly to enhance workflow efficiency.

In this meeting, Andrew and our team first established the overall storyline and the themes involved. The theme is negative, featuring horror elements, and it involves reflections on pollution and personal growth. After discussion, we decided to focus the narrative design around the protagonist Yue’s psychology. In stage and interactive design, we will incorporate considerations of ocean pollution, allowing the audience to empathize while experiencing the story.

Regarding how to attract the audience, we are considering the use of interactive and immersive sound designs to create music, ambient sounds, and special effects. These special effects include sounds for stage narration and interactive experiences. For example, in the first scene, we plan to use Touch Designer or Max to create an interactive installation. When the audience interacts with the TV, real-time generated images and sounds will appear on it. Additionally, we are thinking of creating props that can be interacted with or remotely controlled, such as rotating fish lamps or hanging balloons, to enhance the audience’s experience.

This document represents a formal record of a meeting jointly submitted by our team.

During this meeting, team members primarily engaged in discussions about various topics and the feasibility of their implementation. The discussions were reoriented based on the remarks made during the lecture. The most critical task identified was to settle on a singular narrative and to progress according to our conceptualized ideas, allowing for creativity within a reasonable scope and enhancing the use of visual and auditory technology.

In our previous discussions, we identified three distinct themes: marine environmental protection, dark fairy tales, and mental health issues. Among these, a complete performance story had already been developed for the dark fairy tale theme. However, considering the difficulty in stage design and prop production, as well as the performance skills of our group members, we decided against choosing a story with human characters as protagonists. Instead, we opted to integrate story elements into animal characters. By combining this approach with the previously selected theme of marine environment, we ultimately decided on a narrative featuring fish as the main characters.

The backdrop of our story is the Chinese fairy tale “The Carp Leaps over the Dragon Gate,” in which the carp, after enduring numerous hardships, leaps over the Dragon Gate and transforms into a real dragon. However, we aimed to adapt this into a dark fairy tale, deepening the thematic essence of the story to enhance audience empathy. By integrating elements of marine environment and mental health, we have developed the following script: