Stretching their boundaries, artificial intelligence researchers have teamed up with musicians on an unlikely project: a digital conductor of improvised avant-garde performances.

A conductor that could guide such performances must be capable of “high-level reasoning,” said Professor Selmer Bringsjord, co-principal investigator, director of the Rensselaer Artificial Intelligence and Reasoning Laboratory, and head of the Department of Cognitive Science.

The problem is an excellent candidate for artificial intelligence, Bringsjord says, because a conductor of the unpredictable musical style would need to employ interconnecting elements of cognition  perception/action, reasoning, decision-making, planning, memory  to understand and respond appropriately to the music.

“Most people understand music in terms of pitch, rhythm, and volume. We’re concerned with texture and density and timbre, as well. These parameters are more complicated for the system recognizer and more exciting for us.”
 Pauline Oliveros

“Is there a way to render in formal logic and reasoning what Leonard Bernstein does?” said Bringsjord. “We will need to capture what the musicians are doing in a musical calculus. Then the system reasons over the calculus.”

The “Creative Artificially-Intuitive and Reasoning Agent” (CAIRA) project is supported by a three-year $650,000 NSF grant, and joins Bringsjord with musicians and researchers Jonas Braasch, an acoustician, assistant professor of architecture, and principal investigator; Pauline Oliveros, a virtual accordionist and clinical professor of music; and Doug Van Nort, an electronic musician and music technology researcher. The latter three form the musical trio Triple Point, which acts as a “performance laboratory” for the project.

The challenge of creating a digital conductor is greater given the trio’s musical style than it would be with music that fits a set genre or convention, said Oliveros, co-principal investigator and founder of the Deep Listening movement.

“Most people understand music in terms of pitch, rhythm, and volume. We’re concerned with texture and density and timbre, as well,” Oliveros said. “These parameters are more complicated for the system recognizer and more exciting for us.”

The CAIRA project builds on a two-year pilot project in which the trio built a software accompanist to their music. Their pioneering work led to software that analyzes and classifies qualities related to density, texture, and timbre, said Van Nort.

Oliveros said the pilot project was “mostly about getting the software to respond to what we’re playing.”

“The software listens, extracts, and parses what we’re playing and may feed it back to us in a different form or a replica,” Oliveros said. “It makes decisions about what it thinks is working in improvisation as it’s happening.”