Conducting with Kinect: Seattle Symphony to Use Microsoft's Sensor in World Premiere Performance

In a dramatic departure from typical “symphonic” fare, maestro Morlot’s movements will both guide the human musicians and — using a Microsoft Kinect sensor — control a 24-reedhorn sculpture, a set of concert chimes, and a “prepared” grand piano equipped with small robotic devices.

Morlot is conducting Above, Below, and In Between by composer-in-residence — and kinetic sculptor — Trimpin as part of its “[untitled]” series of concerts. The Symphony-commissioned work not only highlights the talents of a soprano and a small orchestra, but also the kinetic instruments.

“Kinect is one of the few off-the-shelf depth sensing solutions available to the broader creative technologist community,”

explains Dimitri Diakopoulos, a former student of Trimpin’s who is now with the Perceptual Computing division of Intel.

“Their proprietary body tracking is the secret sauce that allows installations like this to be possible: instead of having to worry about low-level computer vision algorithms, developers can focus on their idea.”

The symphony is using the Kinect camera that ships with the Xbox One. The sensor tracks the gestures, Diakopoulos says, and software was created to make it as straightforward as possible for the conductor to use natural movements to get the desired sound.

So what will the audience see when the approximately 22-minute piece is premiered?

First, they’ll be seeing it in the lobby, not the large auditorium in Benaroya Hall. The kinetic sculptures are, well, sculptures that have to be installed between the nine columns in the lobby’s promenade. Nine Symphony musicians and the singer will enter from various directions just before the start of their parts. And Morlot will have dynamic, gestural command of the sculptures.

“I will be able to control the volume level of the kinetic instruments, and also start and stop the instruments,” Morlot says. “For example, there is a gesture to grab the machine when I want to control the instruments with my left hand, while conducting with my right hand. When I want it to stop, I wave my hand in the air as if I’m hailing a taxi.”

“The component that will require a lot of work is the task of synchronizing live musicians with kinetic instruments … in such a big space,” he adds.

Even though the Kinect itself uses wireless sensing, the kinetic instruments are wired to prevent unwanted lag time or interference. Diakopoulos, who collaborated on the concert, says that’s just good practice: “Several collaborators of mine have horror stories where an installation would work fine during development, but when performance time came, the interference caused by 500 cellphones in the audience destroyed the signal.”

And, uniquely, there is the matter of avoiding an unexpected guest conductor. “Another factor we have to consider is the movement of others who will be around me,” Morlot says. In concerts held in the lobby area, “often there are members of the audience seated on the ground within just a couple of feet of the musicians. For this project, we have to design the staging so that no one can accidentally distract the motion sensor.”

The tech-tinged performance is part of an increasing anti-tradition of symphonies reaching out to new audiences, or giving existing audiences new ways to think about what local orchestras do.

From a perception standpoint it can be an uphill climb, starting from the popular misconception that symphonic music is mostly “classical” music. To music nerds, “classical” refers to a period of primarily European music from roughly 1750 to 1820 (or between Bach and Beethoven) when the symphony, concerto and sonata were pretty much standardized. A lot of tunes touted as classical now really are baroque (earlier) or romantic (later), or even modern and contemporary (much later, such as last or this century).

That’s one way this new work from the Seattle-based sound artist Trimpin breaks the moldy mold. The Seattle Symphony also has toyed with tech in other ways, from synchronized live soundtrack performances for films (the upcoming Disney Fantasia is one example), to behind-the-scenes functionality such as an updated mobile ticketing app for iPhone and Android that’s about to be released.

As Morlot puts it, “My biggest hope is that this project opens up new possibilities of what we can do as musicians and with our instruments.”

The concert will be held on Friday, May 1, at 10 p.m. It also features pianist Michael Brown playing works by American composer George Perle. And for those who are curious but can’t make it? The Symphony says the kinetic instruments will remain installed in Benaroya Hall for both viewing and educational tours after the premiere, with another concert — featuring student compositions — planned for June 1.

Microsoft Kinect included.