After several practice sessions with Arduino, our team is ready to progress to the next stage of our project, as previously outlined in our blog posts. The goal is to detect the movement of audience members in order to trigger the playforward or playback of an interactive animation. Ideally, we would directly connect all the sensors with the software used for creating visual elements, such as TouchDesigner or Unity. Although we initially devised a backup plan to trigger recorded video through Processing, the results were not optimal, warranting further exploration of our primary option.
Experimenting with Arduino and Unity
During a weekly meeting, Molly and I attempted to connect an Arduino joystick with Unity. However, we discovered that compatibility was limited to specific versions of Unity that allowed for Arduino integration. As a result, we decided to shift our focus to TouchDesigner, which offered more online tutorial videos and implementation support.
Learning from Online Tutorials
I followed two instructional videos on YouTube: one demonstrating how to link Arduino with TouchDesigner by P.G. Lucio (2021), and another explaining how to control video playback with audio input by Bileam Tschepe (2022). These tutorials guided me through obtaining the correct data from the Arduino board, displaying messages with a serial port object in TouchDesigner, and processing the average data as input (see Figure 1). Additionally, I experimented with connecting distance data to video length to approximate the final project outcome (Figure 2).
Figure.1. Screenshot of serial port information in TouchDesigner.
Figure.2. The distance sensor is operating the video play.
Preliminary Results and Next Steps
Our initial tests have yielded promising results. We can now jump to different points in the video based on input from distance sensors, although the exact ratio between time (minutes) and distance (centimeters) requires further refinement. We have also successfully utilized audio beats to trigger video playback at this stage.
As our Arduino testing phase nears completion, I am eager to collaborate with Yijun and Yuxuan on the TouchDesigner aspect of the project. Acquiring TouchDesigner knowledge will be crucial in case we encounter issues with the software or if ultrasonic sensors prove unsuitable for influencing point cloud properties. Additionally, the integration of Kinect technology should also be considered in our ongoing development process.
If you’re reading this in order, please proceed to the next post: ‘From Raw Data to Unity’.
Allison Mu.
Reference
Bileam Tschepe. (2020, February 16). Controlling Video With Audio – TouchDesigner Tutorial 14 (see description) [Video file]. Retrieved from https://www.youtube.com/watch?v=oviwpILXo5A
P. G. Lucio. (2021, September 7). How To Use TOUCHDESIGNER & ARDUINO Together – Beginner Tutorial [Video file]. Retrieved from https://www.youtube.com/watch?v=V_Q_fDukTI0
