Any views expressed within media held on this service are those of the contributors, should not be taken as approved or endorsed by the University, and do not necessarily reflect the views of the University in respect of any particular issue.
Although it has been discussed in the 1st submission, and has been referenced throughout, I thought I should make a blog post about how the music was constructed in the first layer, as to be very clear. The chords are based on spectral analysis on both samples of whalesong, and the sea creature sound effects made for the section. The spectral analysis was done by using the programme SPEAR. Figure 1, analysis of a sea creature sound effect.
This is then converted into notation by using an SDIF (SPEAR file) to score converter, as notating out manually would give a much less precise result. Figure 2: Image of a spectrogram-chord generator on Openmusic.
This technique of creating harmony from different sounds/timbres is a common technique in spectralism (Fineberg, 2000) a musical style emerging in the 1970s where the musical compositions are based on the natural spectra and/or acoustics of certain sounds (Harvey, 2000). Gérard Grisey work Partiels specifically uses this technique, instead with the harmony is based on the sound spectra of a low E on a trombone (Hasegawa, 2009). This is a more ‘literal’ interpretation of the sea than the other approaches, which I think is appropriate for establishing this layer, the fear of the sea creatures is represented by harmony based on the acoustic qualities of said creatures. Resulting sounds and notation can be found in the appendix.
Reference list.
Fineberg, J. (2000) Guide to the basic concepts and techniques of spectral music. Contemporary music review. 19 (2), p. 81–113.
Harvey, J. (2000) Spectralism. Contemporary music review. [Online] 19 (3), p. 11–14.
Hasegawa, R. (2009) Gérard Grisey and The “Nature” of Harmony. Music analysis. 28 (2/3), p. 349–371.
This is a showcasing of the sounds I created, the sources part of my research, and the technology I engaged with. A gathering of many of the tools used (this is my no means exhaustive).
Sea creature layer.
Spectral analysis of sound, using the programme SPEAR.
Tool used to create notation from spectral analysis (SDIF files), using the open source music-based visual programming environment OpenMusic.
Chords produced via this process.
Blue whale chord 1.
Blue whale chord 2
Whale chorus chord 1
Whale chorus chord 2
Chaos layer.
Image of room and technologies used during the improvisation.
the improvisation recorded.
the specific sounds sampled from this.
Unknown layer.
the chords used, each having subtle differences between them.
sampled tubular bells.
re-pitched voice.
Consonant chords
Short videos from the instillation, demonstrating the sounds being used.
One in the higher point of the atrium
Bibliography/sources used.
Canonne, C. (2018). Rehearsing Free Improvisation? An Ethnographic Study of Free Improvisers at Work. Music Theory Online, 24(4).
Féron, F.-X. (2011) The Emergence of Spectra in Gérard Grisey’s Compositional Process: From Dérives (1973-74) to Les espaces acoustiques (1974-85). Contemporary music review. 30 (5), pp. 343–375.
Fineberg, J. (2000) Guide to the basic concepts and techniques of spectral music. Contemporary music review. 19 (2), p. 81–113.
Harvey, J. (2000) Spectralism. Contemporary music review. [Online] 19 (3), p. 11–14.
Hasegawa, R. (2009) Gérard Grisey and The “Nature” of Harmony. Music analysis. 28 (2/3), p. 349–371.
Hill, S. C. (2018) A “Sound” Approach: John Cage and Music Education. Philosophy of music education review. 26 (1), p. 46–62.
Kennedy, K. Kennedy, M. and Rutherford-Johnson, T. (2013). The Oxford dictionary of music. 6th Ed. Oxford: Oxford University Press.
Latham, A. (2011). The Oxford Companion to Music. Revised 1st Edition ed. Oxford University Press.
Persichetti, V. (1961). Twentieth Century Harmony: Creative aspects and practice. New York: W.W Norton
Smith, B. (2016). Repeater Orchestra. Available at: https://codepen.io/barefootfunk/pen/ZWoLm.
vimeo.com. (n.d.). Bryant Smith: Repeater Orchestra Performance at CodePen Chicago, June 29 2016. [online] Available at: https://vimeo.com/172845834
Wood, J. (1986). Microtonality: Aesthetics and Practicality. The Musical Times, 127 (1719), pp. 328–330.
In the final work, I felt there were aspects that came across very strongly, whilst other points could have done with more refinement and improvement. One of the main points that I felt worked well was, in spite of the very clear distinctions between the sections, there was still continuity within the whole sound world. This is something that was designed into my sounds, with all of the chords being rooted with the note D. Even the improvisation was extremely rooted within the note D. This ‘implied drone’ throughout meant that even when there was a dramatic shift in sound, there was still some kind of ‘rooting’ to the sound world.
Figure 1: highlighting the rooting of D.
This focus on a single fundamental note across a long time is heavily influenced by the French composer Gérard Griseys 90 minute cycle les espaces acoustiques, which is entirely based on the same low E (Féron, 2011).
However, even if there was strong musical connections, there were some issues with panning between the sections, which lent to the changes being less smooth than planned. Although a lot of this was down to issues within MaxMSP, which wasn’t my part of the project, the individual sounds I created could have been faded in/out in a smoother manner, to make this process easier.
Another strength of the project was the sensors being used, which worked really well, and centered interaction in the instillation. It was commented on that the sensors, which controlled the visuals and the sounds, functioned like a musical instrument. This was an interesting observation, and curiously recontextualized my own perception of the sensors interaction. Instead of being seen as a series of sensors, it felt more like an experimental, electronic, sound and visual autoharp, an instrument which produces chords by suppressing keys (Kennedy et al, 2013, p. 28). The autoharp is purely harmonic, with limited control of what is possible. Similarly, the sensors allow for a degree of control, however, it is limited to the set of sounds we created for it, many of which are chords, though distinctly non-triadic and non-tertian, the typical chords used in functional tonal harmonic practice (Persichetti, 1961, p. 66). The one small issue with the sensors, was with the light sensor. The sensor itself worked very well, and was sensitive to changes in light. However, because of the darkness of the room, it would only respond strongly to torchlight, typically that of a phone torch. Although this work effectively, it did feel slightly awkward to work the other sensors with the phone in hand. Likewise, this made the instillation inaccessible to those who don’t own a smartphone, or a phone with a torch in general. In hindsight, I should have planned to source a light that would mean the light sensors didn’t require the use of a phone torch, whilst also not messing with the rest of the instillation.
I felt the sea-creature layer and the chaos layer worked very well and was happy with how the sound turned out. The sounds/chords created fit very well within the sound world, as they had strong thematic ties to their respective concepts. However, I wasn’t as certain about the harmonic structure in the unknown layer. This was the most difficult to represent from my perspective as a composer; how does one represent the unknown through sound? One of the ideas as to use the harmonic series, a series of ascending pitches from a fundamental which regularly occur in most acoustic sounds (Kennedy et al, 2013, p. 207), as the basis of the harmonic language. This would have been justified as ‘bringing out the unknown sounds around us’, but the aesthetic quality of this kind of harmony didn’t evoke fear in my mind. The denser, microtonal harmonies I felt got closer to this idea, exploring sounds unknown to the 12 tone system used predominantly in western music (Wood, 1986). However, some of the chords felt more curious, and peculiar, rather than evoking fear. In hindsight, I should have spent more time focusing on the harmonic structure of this section, as I was more comfortable with the rest of the material.
Another part I felt need further improvement and additions was in the calmness/peaceful layer. Although I was happy with my chords and harmonic structure in this section broadly, I felt more could have been added in terms of sound material. Most of the sound design at that point was stripped down as a deliberate creative choice, which did make sense as a representation of the calm. I should have responded more so on my end with more additions of ‘musical’ sounds to fill in, as everything dropped away, and felt slightly too sparse. Here are some of the sounds that could have been used in hindsight, using different timbres of the note D on the piano.
Lastly, a final critique mentioned was the lack of high fidelity sounds. All sound was heavily weighted towards the lower and middle frequencies, and therefore, didn’t fill out the whole frequency spectra. In our defense however, it is arguably thematically appropriate to focus more on lower frequencies, as there is more of a sense of being ‘under the water’. Likewise, the spectral analysis that was done to the sea creature sounds and the whalesong showed more prominent partials in the lower and mid frequencies.
Figure 2, spectral analysis of one of the sea creature sound effects in layer 1 (spectral analysis done using the program SPEAR).
That being said, this lack of high fidelity sound does create a less ‘full’ sound quality , and overall feels slightly less immersive than desired. If this instillation was to be done in the future, I would likely add only one other track; a droning improvisation using the high D notes on the piano. There was a last minute idea to add this on during the work live as there was a piano in the room. However, this might have meant the instillation would be perceived as a live performance. For context, this is a rough mock up of what the track could potentially be.
I would have this track play cyclically throughout the work, which would possibly have the second benefit of smoothing the transitions between the sections.
In conclusion, there was a lot of space for improvement and refinement to my sounds, both how some of them where formed, and with more additions to create a fuller experience. Even in terms of the broader ideas of the instillation there was space to improve, for example, clarity of the theme of thalassophobia could have been improved when presenting the instillation. However, the instillation felt strong, and received mostly positive feedback. In spite of the places to improve on for future practice, the project was overall effective, and I believe showcased each of our individual strengths of multimedia programming, sound design, and composition, contributing to a single, consistent, well formed multimedia instillation.
Reference list.
Féron, F.-X. (2011) The Emergence of Spectra in Gérard Grisey’s Compositional Process: From Dérives (1973-74) to Les espaces acoustiques (1974-85). Contemporary music review. 30 (5), pp. 343–375.
Kennedy, K. Kennedy, M. and Rutherford-Johnson, T. (2013). The Oxford dictionary of music. 6th Ed. Oxford: Oxford University Press.
Persichetti, V. (1961). Twentieth Century Harmony: Creative aspects and practice. New York: W.W Norton.
Wood, J. (1986). Microtonality: Aesthetics and Practicality. The Musical Times, 127 (1719), pp. 328–330.
The presentation method chosen by our Presence group is an immersive interactive installation.
The following is the setup structure diagram (see Figure 1) and flow chart of our installation, including content flow chart (see Figure 2) and interaction flow chart (see Figure 3).
Figure 1: Setup Structure Diagram.
Figure 2: Content Flow Chart.
Figure 3: Interaction Flow Chart.
Then there is the overall interactive operation on site (see Video 1).
Our project was held in Atrium. Before audiences entered the room, we affixed a poster to the front door, introducing our project and effectively conveying its theme.
Figure 1: Poster
Atrium provides enough space for a certain number of spectators to walk around. We placed chairs around the front area to prevent individuals from obstructing the projection or inadvertently stepping on cables. This layout clearly delineated designated pathways for audience movement and restricted areas to maintain an immersive viewing experience.
Figure 2: Venue
Ideally, we aimed for a completely darkened space. However, a large window above Atrium meant that natural light would influence the lightness in Atrium, preventing the space from achieving complete darkness throughout the presentation. Thus, to improve our project, we will need to use a large black cloth to block out the light from the top to create an all-black, immersive, deep-sea-like environment.
Figure 3: Venue
Setup
Because our project is an audio-visual installation with surround sound techniques, the used equipment includes:
4 Genelec 1030a speakers
1 Faderfox DS3
1 HD NEC V302H projector
1 FireFace UCX
1 projection screen
5 sensors
Figure 4: Equipment setup
We wrote instructions on the designed box next to each sensor to guide the participant during the presentation.
Although the final presentation is much better than the previous production, there is still room for improvement through on-site user feedback and personal experience.
The filter effect not as obvious enough. It may be related to the fact that the sound samples are played for too short a time and users cannot quickly feel the filter changes.
The range of ultrasonic sensors could be set more clearly. Although it has almost no impact in actual operation, it can be improved from the numerical perspective.
The interactive connection between sound and vision could be stronger. For example, adding other forms of interaction between the sound part and the visual part in Max to improve this. But this may require a significant amount of time to resolve. I will continue to explore this point towards the end of the course. This will also be a topic worth discussing after the semester is finished.
Personal Reflection
During the production of this project throughout the semester, I found that I was not just doing a group assignment, but also absorbing a large amount of knowledge from various aspects through various channels. I have greatly improved both in terms of technical operations and teamwork skills. This was such a meaningful and worthwhile experience for me.
Technically, I had no experience with sensors before doing this project. Although I know that it is very challenging to choose a brand new thing to study and implement it into actual operations, I still want to try hard. As our project is an interactive installation, interactivity is something I always think about the most. At the beginning, my thinking was limited to optimizing this by modifying some codes in Arduino. This did have some effect, but after many discussions with Jules, Leo and Joe, I gradually discovered that there are many ways and angles to better solve the problem. For example, I can modify the sensor connection method, add some transitional objects in Max, and so on. It gave me more ideas in learning technical skills.
In terms of teamwork, our team has a clear division of labour from the beginning, which makes our subsequent work efficient. But we don’t just focus on the part we are responsible for. We will also help each other when other people’s parts encounter difficulties. My interaction part received strong support from Ruojing and Yuan in the early testing and rehearsals. This has provided me with great help, and I am very grateful to them.
I learned a lot during the making of this project. These 12 weeks of struggle have been a very beneficial experience on my learning path. In the future, I think I will continue to study interaction-related skills in depth and strive to create more creative interactive works.
Thanks to the teachers and classmates who have given us strong support. Special thanks to Jules, Leo and Joe who have been patiently guiding us in technology. And finally, thanks to my hardworking group members.
In the Week 10 lecture, we received feedback from Jules and tutors on the effectiveness of our rehearsals: Since the sensor has many cables, it is not particularly beautiful when viewed on site, so it is best to make several boxes or covers for them, leaving only the sensor itself. We also agreed that this was a very meaningful suggestion and therefore designed the decoration for the hardware installation.
Inspiration
Thalassophobia is the chosen theme for our group. Thalassophobia, as the name suggests, is about the phobia of the deep sea. Therefore, the decoration should be related to the deep sea atmosphere.
In terms of color, we chose the dark blue of the sea. For objects that need to be decorated, in addition to the sensors with many cables there is also the MIDI controller that only need to have one button among many. And they are only allowed to expose the small part that can receive external information for interaction, which is very similar to the bright light seen by a person after falling to the bottom of the sea (see Figure 1, Figure 2 and Figure 3). So we prepared blue and white cardboard, and planned to make two boxes for the two Arduino Unos for vision and sound, and a cover for the MIDI controller that only displays one of the buttons.
Figure 1: Deep Sea Environment.
Figure 2: Deep Sea Environment.
Figure 3: Deep Sea Environment.
Handcrafting Process
Ruojing, Yuan and I concentrated on handcrafting during the weekend of Week 10 (see Video 1 and Video 2). At the same time, I also produced instructions for using the sensor and MIDI controller, which were displayed to users on two iPad screens on site (see Figure 4).
Video 1: A Video Showing the Sensor Decoration Production.
Video 2: A Video Showing the MIDI controller Decoration Production.
Figure 4: Instructions for Using Sensors and MIDI Controller.
On-site effect
All hardware equipment on site is placed on the table. During the formal presentation, we also received feedback from a tutor who suggested that we could attach the sensor usage instructions directly to the sensor boxes, so that users do not have to glance at the iPad next to them and then come back to interact with the sensor, which reduces some troublesome steps. Therefore, we wrote small notes with instructions for each sensor and posted it on the sensor boxes for the convenience of users (see Figure 5 and Figure 6).
Figure 5: Decoration of Sensors and MIDI Controller.
Figure 6: The Overall Effect of Decoration in the Installation.
Sound dominates this project. We emphasise the expressive force of sound to achieve the effect of creating an atmosphere, and compared with vision, using max in sound will make me more confident.The following table is my idea about sound effects and music.
We decided to use the expression of four emotional levels. The first thing that came to my mind was the square object that can be used in the patch, which can be pictslider or nodes. So at first, I put both of them in my test patch and made some attempts to decide which one is more suitable for our project. This is also one of the questions that we asked Jules in the first presentation.
pictslider & nodes
Deconstruction of sound part patch
With the progress of the project, the concept gradually became clear, and my patch became more perfect.
As shown in the figure, you can see that I set the four corners of pictslider as A、B、C、D points, and the circles in them can be controlled. The four playlists below correspond to A、B、C、D points respectively. On the first level, we show the scene of sea monsters, so these sound effects are all sea monsters and underwater creatures.In this way, the other three layers of scenes are also evenly distributed in these four corners.
Because the variables to be regulated by the sensor were not determined at that time, I set a number of manually controllable elements, such as speed, pitchshift and timestretch, as dynamic quantities to make the audio change and control it by the sensor.Detailed comments have been marked in the picture.
In terms of artistic presentation, I have made a lot of cuts in the sound effects.We have produced many sound effects on each layer, but considering that many people will not have the patience to listen to so many similar sound effects in actual performances, which will make people feel tired aesthetically. In the end, in every emotional layer of the whole project, we used 12-18 sound effects evenly distributed, while the last layer used only four sound effects as a finishing touch to express the sense of calm.
here is the sound effect part on patch:
For the music part, I do the same thing. Corresponding to each level of emotion, the chords of music are more concise, and each level is exactly eight chords, so it is two to be assigned to four corners.
I met some new problems in the process of patch improvement, which also made me learn new knowledge about max. The problems encountered are as follows:
Based on the audience’s listening comfort, how to make these four layers of emotions cross-fade and naturally present when switching?
Based on the need of immersion, how to output it to multiple speakers to make it have audio-visual changes?
Based on the need of diversified regulation of sound, how to make the sound itself change more?
For these three questions, the following three parts are used in max to give corresponding answers.
autopattr&pattrstorage
For the first question, Leo gave me this solution when we had a meeting with Leo.I use pattrstorage object as a preset to increase and decrease the level in order to make the transition between layers smooth.I use it to connect a midi controller to control the switching. Leo refers to the instructional video of this link:https://youtu.be/5-4YZwbKB-k?si=1NeUJhZddplc0xT_
Video shows specific functions:
As shown in the figure, the file read in the operation is a code file with. json suffix:
panner
For the second question, we asked Jules for a lot of help. I want to create the fear of sea monsters surrounded from all directions, so I need to use sound image processing. At the same time, I also need A、B、C、D points to correspond to different speakers in speaker settings.
filter setup
In the final presentation, Leo gave us the suggestion that we need more changes to make our project sound less monotonous. So we use the cascade~to filter an input signal using a series of biquad filters and use filtergragh~to generate filter coefficients.
As shown in the figure, our control range is cut off 500-5000 Hz, and(30 ,100) is the range given by the humidity sensor.
Improvement suggestions
The sound part received valuable opinions from many teachers.
Set more speaker controls to seek a better sound experience.
Add frequency levels of music or sound effects to make it sound richer, such as high frequency.
If there are more channels coming from above, it will highlight the immersion of the characters on the seabed.
Personal reflection
The construction of the sound part in max patch takes the longest time among all tasks, because new problems are constantly discovered and new tasks appear due to the requirements of project presentation, and at the same time, we are constantly asking the teacher for help to solve the problems.
In the third rehearsal, I was still trying to make sure that the signal sent by panner could not be transmitted to four speakers. Although the sound signal was successfully transmitted in the end, more speakers were not used to reflect the immersion in the final presentation.
We each have our own responsibilities throughout the birth of the project, but I have participated in every part of our group except for the music part. Because of the company of team members and the support of teachers, I was not very anxious from beginning to end. This embryonic project gave me the courage to continue exploring audio-visual interactive software.
Thank you again for Leo’s responsible company and all support, Jules and Joe’s technical support, all the teachers, classmates and friends who came to watch, and all the members of DMSP_presence group.
The inspiration for the first layer comes from my research towards thalassophobia. Jarrett (2022) argues that there is a significant relationship between thalassophobia and mythology. Ancient and modern myths of deep-sea monsters have impressions on the collective consciousness, influencing the perceptions and fears associated with the deep sea and shaping individuals’ experiences of thalassophobia.
Based on my findings on sea monsters across various cultural myths and legends, it’s evident that they are often described as exceptionally large creatures.
Figure 1. Kraken
Figure 2: Loch Ness Monster
Figure 3: Leviathan
Figure 4: Hydra
To create the sounds for these sea creatures, I applied Mconvolution on self-recorded material to make them sound mysterious and distorted. I focused more on generating low-frequency, creature-like sounds to correspond with the giant size of these sea monsters. The siganificant low frequencies serve to show their enormity and provide a sense of dread. Additionally, I incorporated reverb into the monsters’ sounds to give a sense of the deep, dark expanse of the sea, enhancing the eerie and ominous atmosphere surrounding them.
Figure 5: Using Mconvolution
The sounds of Siren are the high-frequency sounds that were created in this layer. I pitched up a synth violin and applied a reverb to make these sounds more ethereal and mysterious.
Figure 6: Siren
Layer 2: Choas
The sounds in this layer draw inspiration from research on the physical and mental symptoms associated with thalassophobia. Studies, like that by Bence (2023), highlight common experiences such as dizziness and faintness among individuals with thalassophobia. To deliver this sensation, I layered whoosh sounds to evoke a sense of disarray and chaos. Additionally, I reversed some of the whoosh sounds to further enhance the feeling of dizziness. Introducing high-frequency sounds resembling tinnitus adds an additional layer of disorientation, conveying the sense of dizziness experienced by people with thalassophobia.
I also manipulated some noises through Mconvolution to further highlight the chaos.
This layer is crafted based on Craig’s improvisation.
Figure 7: Collaboration with Music
Layer 3: Unkown
In this layer, my inspiration came from the definition of thalassophobia. Thalassophobia is defined as the persistent fear of vast, deep, and often dark bodies of water that feel dangerous (Anwar, 2021). Specifically, thalassophobia describes a person’s fear of the great unknown in the water (Anwar, 2021). I created some hollow sound effects using Mconvolution, and I applied large reverbs to convey the fear of the unknown.
Furthermore, I incorporated recordings of my voice, which I pitched the recording down and layered together to produce a whisper-like, hollow effect.
Figure 8: Recording my voice
Layer 4: Calmness
The last layer is a peaceful state. Therefore, I used Mconvolution to produce some organic sounds, such as wind, chimes and cave with water sounds. These organic soft sounds contribute to conveying a sense of chill and relaxation with the soft high frequency.
Certain sounds were directly recorded in the studio. These include the crisp sounds of ice cubes gently swirling in water, the delicate touch of a necklace, the pouring of rice, and the subtle rustle of pages turning.
Each sound across the various layers has been carefully trimmed to ensure immediate responsiveness to audience interactions, avioding any delays. This editing enhances the immersive quality of the experience, allowing participants to seamlessly engage with the soundscape and maintain a fluid connection between their actions and the auditory feedback they receive.
Find all the sound effects through the link below:
Anwar, B. (2021) ‘Thalasopphobia: Fear of the Ocean‘, TalkSpace, 20 August. Available at: https://www.talkspace.com/mental-health/conditions/thalassophobia/#:~:text=The%20thalassophobia%20definition%20is%20pretty,water%20right%20below%20their%20feet (Accessed: 15 April 2024).
Bence, S. (2023) How Do I Know if I Have Thalassophobia?. Available at: https://www.verywellhealth.com/thalassophobia-5093770 (Accessed: 15 April 2024).
Jarrett, C. (2022) ‘Thalassophobia: Everything you need to know’, BBC Science Focus, 16 July. Available at: https://www.sciencefocus.com/the-human-body/thalassophobia (Accessed: 15 April 2024).
Mythicalcreatures.info (2024) Leviathan. Available at: https://mythicalcreatures.info/characters/leviathan/ (Accessed: 15 April 2024).
Mythicalcreatures.info (2024) The Hydra. Available at: https://mythicalcreatures.info/characters/hydra/ (Accessed: 15 April 2024).
Potts, K. (2014) The hunt for the Loch Ness Monster – valid scientific research or a fool’s errand?. Available at: https://www.abdn.ac.uk/news/6504/(Accessed: 15 April 2024).
Salvador, R. B. & Tomotani B. M. (2014) ‘The Kraken: when myth encounters science’. Medievalists.net. Available at: https://www.medievalists.net/2015/01/kraken-myth-encounters-science/ (Accessed: 15 April 2024).
Villamirella (2024) The legend of Siren Molpé. Available at: https://www.villamirella.it/en-blog/myth-sirena-molpe-palinuro (Accessed: 15 April 2024).
