BINAURAL VERSION
HEADPHONES RECOMMENDED
The binaural mix of Essential Voices simulates the sound of the room as heard by walking through the space. It is intended to be enjoyed with headphones. Binaural audio considers how sound interacts with the human head and ears to create an immersive listening experience that accurately represents the performance venue soundscape.
Background
Daniel Ketter, born and raised in Kansas City, specializes in performance of contemporary and classical chamber music as cellist of American Wild Ensemble, Opus 76 String Quartet, Ensemble Alla Balena, and Cellax Duo. He is Associate Professor of Music at Missouri State University where he teaches courses in cello performance, pedagogy, chamber music, and music theory.
Phosphor Studio specializes in creating audio visual journeys that transcend traditional storytelling boundaries using immersive 3D/spatial sound and visual media. Phosphor produces transformative experiences through innovative projection mapping technology and interactive multimedia integration.
Project objectives
- Demonstrate multi-disciplinary collaboration to communicate a musical idea
- Create a in-person immersive experience using projection mapping and spatial sound
- Integrate a live classical performance with the immersive recorded experience
Collaboration
Daniel Ketter and Phosphor Studio came together to produce an immersive event that aimed to transport the audience into the imagination of a musician while playing their instrument. Daniel created “voices” to accompany J. S. Bach’s Suite No. 1 in G major for Solo Cello. The voices, four in all, accompany the lead performance for a total of 5 individual parts. Audio and video for each part were recorded at Phosphor Studio then assembled in post and synced to playback together. When played back the parts harmonize to create a new and unique experience to this ubiquitous piece of music.
Another aim for the collaboration is to produce a pre-recorded and live immersive experience. Utilizing projection mapping, each part is projected on 5 separate screens, 1 main screen for the lead and 4 “voice” screens on the left and right walls of a large room. The live version swaps the prerecorded lead with a live cello performance and a live video feed for the main screen. The sound location of each part is placed in the room relative to it’s video screen. The combined audio visual experience immerses the audience in the mind of the musician as the sound harmonizes all around.
Event
For this event, Phosphor Studio simplified their workflow to use a single workstation that played back all of the audio and visual elements that made up the supplementary harmonies to Daniel’s live performance. These supplementary harmonies were placed sonically and visually in different parts of the room to entice audiences to travel around the space and observe the sound field from different positions.
Workflow
This event had a tight deadline which required a fast and efficient workflow between Daniel and Phosphor Studio’s team, built upon Phosphor Studio’s collaboration with Fleshy Temples. This involved a half day of recording, time in post production for edits and time alignment and eventually rehearsals with projections and 3D spatial sound.
Planning and Pre Production: Preproduction consisted of defining the physical space and the material to be produced. Once we knew how the room would be set up, projection screens and audio format, we could define the video assets and sound recordings we needed to capture.
Production: Recording consisted of a half day of shooting each harmony of the piece as well as the lead. This footage was shot on Canon c200bs in 4K Canon Cinema Raw Light. This gave the biggest flexibility in post production to color grade as the creative intent became clear. Traditional cello recording was done for the spatial mix as well as ambisonic recordings.
Editing and Post Production: Time aligning the different voices with the lead proved the most difficult task. A click track played for a consistent four bars when recording each voice; this gave an audible cue with a consistent amount of time before the proceeding movement, and made it simple to line up all the harmonies in time with the lead. Two cameras were used for a close up and a wide shot allowing us to switch for visual interest.
Spatial Mixing: Once the edits were complete, the video and audio recordings were imported into Protools for a 7.1.4 Dolby ATMOS Mix. Each voice was placed in space next to its corresponding visual in the room. Reviews were done with a binaural version and .mp4 video representation for convienience. Any notes were taken into account and the audio mix was finalized. Using the Dolby ATMOS system allowed for the flexibility to easily move sounds around the room without excessive routing to various outputs. It also allowed for easy calibration of the system, whereas a custom or ambisonics setup would require all of the measurements and calibration to be done manually, which would be tedious and time consuming.
Projection Mapping and Playback Programming: After the audio mix was finalized, and the visuals edited and color graded, Phosphor could move on to programming the show. The 7.1.4 mix was loaded into the show control software for audio playback and the video clips were mapped to their designated locations in the room. Various network commands played back in time to sync the audio and video as well as trigger effects to highlight different parts of the harmonies.
Rehearsal: The live show: Once all the assets were approved, rehearsals were scheduled so Daniel could get a feel of the transitions from movement to movement and get used to seeing himself on the projection screens. Phosphor and Daniel also took this time to identify any musically important sections and highlight them visually by brightening the screen of the important harmony and dimming the screens of the others. A live camera feed of Daniel’s performance of the lead was also projected on the main screen.
Performance
“There are many things in music which, not being heard, must be imagined.” Carl Philipp Emanuel Bach, Versuch über die wahre Art das Clavier zu spielen, 1762
I have been playing J. S. Bach’s Suite No. 1 in G major for Solo Cello since I was 12 years old. Alan Harris, my cello mentor at Eastman School of Music, was known to say to students, “What you play on the cello is a history of what goes on in your mind.” This project brings listeners into a history of over 25 years of cello performance and study of Bach’s music, offering an immersive realization of my interior, imagined experience I attempt to share with the audience in each solo performance.
Cellos normally play single musical lines meant to contribute to a fuller musical texture of a quartet or orchestra, whether melodies or bass lines. With the cello alone, and almost entirely only one note at a time, it is a remarkable feat that Bach is able to create such full musical textures, not requiring any accompaniment. By taking advantage of our psychological capacity for auditory streaming, notes that may be disconnected in time or register are strung together into continuous voices creating chorale-like effects that outline full harmonies.
This chorale-like texture is not heard, but the performer’s imagination of this full harmony shapes their interpretation which seeks to recreate the same effect for the listener. The arrangement created here immerses the audience in one person’s perspective on what Bach has asked me to imagine and share with the audience in each performance, and an intimate window into what goes on in my mind while playing.
Production Technologies
Original Video Footage: As part of the preparation process, Phosphor Studio shot, edited and synchronized the video footage of Daniel’s complementary performance designed to play back in time with his live performance.
Projection Mapping: The visuals for the performance were mapped onto various surfaces using projection mapping techniques. By precisely aligning the projected visuals with the physical structures, Phosphor Studio brought life to the space and created a dynamic visual experience.
Object Based Sound Field Mixing: Phosphor Studio employed object-based mixing techniques to create a cohesive audio experience. By treating sound sources as individual objects, they were able to manipulate sounds in a virtual environment, allowing for precise control, placement and movement of audio elements within the space related to the visual elements.
Networked Audio: Phosphor Studio leveraged network-based audio systems to achieve a spatialized sound experience. By distributing audio channels across multiple speakers and using techniques like ambisonics and Dolby Atmos, they created a sense of 3D sound that drew attention to different visuals throughout the performance.
Show Control: Each song was broken down into parts which were assigned a cue. From each cue, triggers and commands are distributed to multiple machines and software to do their part in concert.
Conclusion
The collaboration between Daniel Ketter and Phosphor Studio proved to be a rewarding challenge that leveraged Phosphor’s advanced studio space with technologies like networked audio, projection mapping, video editing and of course 3D sound field mixing. Ideas and demonstrations that Daniel developed for his students in 2020 were able to come to life for a fully immersive experience on Phosphor Studio’s soundstage.