Real-time AudioVisual Composition with RayTone

Overview

Are you interested in experimenting with sound and visuals to create an original multimedia performance? This five-day workshop offers an introduction to real-time audiovisual composition using RayTone - a node-based sequencing environment designed to promote a playful workflow for transforming creative ideas into artistic content.

The workshop is hosted by authors of RayTone, Eito Murakami and John Burnett, both of whom are artists/engineers in the fields of computer music and graphics. We will also welcome Professor Ge Wang as a guest lecturer to introduce ChucK - a strongly-timed music programming language. With ChucK embedded in RayTone, users can customize behaviors of audio units that perform digital signal processing at run time. Additionally, RayTone allows users to load OpenGL Shading Language (GLSL) scripts for graphics programming such that audio and visual elements can be controlled simultaneously.

The workshop is structured into three categories: an introduction to RayTone sequencing, lectures on ChucK and shader programming, and activities to design original RayTone patches. In addition to learning from instructors, attendees will exchange knowledge and collaborate with each other to develop creative ideas. Upon completion of the workshop, attendees will learn to combine digital signal processing with graphics programming to perform real-time audiovisual composition. There will be an opportunity to present an original piece using RayTone on the last day of the workshop. Prior musical and programming experience may be helpful but not required.

More information can be found at https://www.raytone.app.
If you have any questions, please feel free to contact Eito at eitom@ccrma.stanford.edu

Requirements

Please bring a laptop (Windows / MacOS) and a pair of headphones to the workshop. CCRMA has a few extra pairs of headphones for those who cannot bring their own.

Schedule

Day 1: Introduction to RayTone & Digital Signal Processing

• Self introduction

• RayTone example projects

• Digital Signal Processing (DSP) with ChucK

- discrete time sampling

- additive / subtractive synthesis

- filter design

Day 2: More DSP + Shader Programming

• More DSP with ChucK

- granular synthesis

- physical modeling

• Shader programming

- GLSL examples

Day 3: More Shaders + RayTone development

• More shader programming

• Integrating webcam, images, and videos with shaders

• (Optional: Anatomy of RayTone Engine)

Day 4: Final project preparation

• Musical composition with MIDI controllers

• Communication with other software via OSC

• Expressive audiovisual integration

Day 5: Presentations

• More preparation time

• Presentations by participants!

About the instructors

Eito Murakami is a master's student at Center for Computer Research in Music and Acoustics (CCRMA) at Stanford University. He graduated from University of California San Diego with bachelor's degrees in Interdisciplinary Computing and the Arts Music (ICAM) and Political Science/International Relations. Eito is an electronic composer, performer, sound designer, and virtual reality developer. By combining his classical music training with proficiency in audio and graphics software, he creates digital interfaces and instruments that promote intuitive musical performance. Specifically, he specializes in Unreal Engine to develop audiovisual infrastructure that allows multiplayer interactions in virtual 3D environments via network. At CCRMA, his research involves designing audio playback systems in virtual reality to achieve dynamic spatial reverb.

John Burnett (b. 1993) is a multimedia artist based in San Diego, California. Drawing from a background in music composition, sound design, and technology, they create technologically-augmented and reactive multimedia installation works, sound and projection design for dance and theater productions, as well as concert works and film scores. John is a former member of the Sonic Arts research team, based in the Qualcomm Institute at UC San Diego, where they research audio spatialization and audiovisual technology.

As former members of the Sonic Arts Research and Development group at UC San Diego's Qualcomm Institute, Eito and John presented a virtual reality composition titled "Becoming" at ACM SIGGRAPH 2022 - Immersive Pavilion.