Computer algorithms are at the cutting edge of modern classical music composition and can even write ‘new’ works by Bach and Mozart. Will such music ever be considered authentically human?

One day, you bump into your favorite author walking down the street. His works have made an indelible imprint on you, expressing sentiments and feelings about the world that you’ve always had but never quite known how to express. So you grab him by the crook of his elbow and make all the usual gushing and gurgling noises that fans make when they meet their favorite authors. You tell him how deeply his works speak to you, how amazing his plots are. How does he write such beautiful, haunting prose?

advertisement

advertisement

But your favorite author doesn’t brush you aside by claiming some inner muse. Instead, he confides in you. “The truth is, I wrote a computer program that allows me to algorithmically generate entire novels in any style I want!” he excitedly explains. He tells you that your favorite novel was, in fact, written by this algorithm: He just told it to analyze his favorite authors, told his computer about the characters, setting, and themes he wanted to use, and then asked it to output him a novel he could call his own.

Not only that, your favorite author says any novel could be written this way. “I could tell my program to analyze the works of Vladimir Nabokov for style, Dan Brown for plot, use the complete cast of Scooby-Doo for characters, and the themes of James Joyce’s Ulysses, and my algorithm could generate a thousand different unique novels in just a few days!” he explains. “All you need to do is know how to tell my algorithm what all those things mean.”

Novels, of course, are not written this way, at least not yet. If they were, you’d likely feel betrayed. But music is, and more so every day. The future of music, in fact, may largely be written by algorithm and could even be used to engineer the next big hits. But how does that impact our understanding of music as an authentic form of art?

“Ninety-nine percent of the time, when I’m composing, I’m actually naming things.”

Slightly lupine and with the quiet nervous energy of the programmer and musical composer alike, 29-year-old Josiah Oberholtzer lives in front of a computer. Looking over his slight shoulder and past the laboratory of Chemexes that sustain him, Oberholtzer’s computer screen looks like any hacker’s. But to Oberholtzer, the code he writes in Vim is analogous to any other composer’s score sheet, largely thanks to Abjad, a Python library Oberholtzer co-created that allows him to generate human-readable scores through the music engraving program LilyPond.

A grad student pursuing his doctorate in composition at Harvard University, Oberholtzer applies the techniques of electronic music to compose works meant to be played by human orchestras. Instead of just stringing note after note, Oberholtzer uses a series of custom tools to translate a nebulous musical intention into a human-readable score. He does this by trying to define in words what the finished piece will sound like.

advertisement

“When I set out to compose, I start out by describing what I’m trying to do in prose.” Oberholtzer tells Co.Design. “It starts out abstract and poetic, but it gets more concrete over time. Some things are easy to describe, like harmonies, but others are harder, and I’ll have to come up with my own methods to describe them. So that sound that gets made when 20 violins all sort of slap down together? I’ll try to describe it: How long it is, how thick it is, and so on.”

My computer isn’t writing my music for me. It’s just handling the version control.

Over time, Oberholtzer establishes a taxonomy of musical meaning that a computer can understand. It’s the DNA of his own musical spirit, written in code: What he likes, what he wants his music to sound like, what different melodies and harmonies mean to him, what he thinks people should feel when they hear a piece. And having teased out, coil by coil, the strand of his intentions, Oberholtzer can use a series of custom tools to help him actualize what he wants to do.

“When I want to capture some new music concept or idea, I’ll usually write a tool first, then think about it a lot and work it into a piece,” says Oberholtzer. “These tools are kind of like meta-instruments, and I can even write tools on top of tools, giving me a wider palette.”

For Oberholtzer, this seems like a perfectly natural way to write music. “All art is a kind of curatorship. You work through all these possibilities mentally, and then in the end, you try to reproduce the one you’ve decided upon. There’s no difference for me. My computer isn’t writing my music for me. It’s just handling the version control.”

To many, there’s something affronting about composers using algorithms to write their music, but the technique is as old as the hills, a fact that David Cope is quick to remind you of.

“Mozart and Bach wrote music using algorithms,” says Cope, a 73-year-old pioneer in the field of generative music composition. “They didn’t have computers, but they used a system to randomly generate music from a number of pre-composed sections, just by rolling dice.”

advertisement

Rolling dice is an algorithm, albeit a simple one. And algorithms, to Cope, are as natural as breathing. “For most of my life, I’ve felt that we, as humans, are walking algorithms,” explains Cope. “The way we blink, think, move, and tie our shoes… all of that’s decided by algorithms in our DNA.”

There was a time when Cope composed music more traditionally. But then writer’s block–or in this case, composer’s block–struck. “Around 1980, I had this commission for around $5,000 to write a new piece. I hadn’t written a single note of music in five years, but I took it anyway, because I had four children to support. I had already spent the money. Now it was time to actually deliver the composition and I was stuck.”

With deadline looming, Cope’s panicked eyes fell upon an old terminal in his garage, linked to his university’s mainframe. “It wasn’t even a computer,” Cope laughs. “It was just a TV set and a modem without any guts in it. There was no computer in it at all.” But the terminal gave Cope an idea. He was already used to using algorithms to help him compose, albeit simpler, analog ones. What if he wrote a computer program to help him get over the hump?

“I’d taught Bach in music theory courses for decades, so what I decided to do was input everything I knew about Bach into a database,” remembers Cope. “For every note of Bach I put in, I had to record five different parameters. Once I was done, though, I was able to program an analysis tool that could examine the collected work of Bach for its salient features, and then produce entirely new music in that style.”

But Cope’s system didn’t just work for Bach. It worked for Cope. With his tools, Cope could actually input his existing body of work into a database and come up with entirely new “Cope originals.” “It ended up being quite the windfall,” laughs Cope. Not only that, but he was able to generate an entire series of compositions in which a computer modeled classic composers, like Bach, Mozart and Rachmaninov. Often times, even trained ears have a hard time distinguishing these compositions from real “lost” works.

“Early on, I did a lot of blind tests of my synthesized works versus real ones,” Cope recalls. “Eventually, I stopped, because people got so mad I feared for my safety.” Once, Cope’s latest composition was given a scaling review, in which the reviewer said that the composition was obviously computer generated. Later, that same reviewer came to Cope after another performance. “He said that the two compositions were night and day, because the latter had ‘soul.'”

What using computer algorithms to compose music requires is a way of explaining how music relates to human beings in a language a computer can understand. That isn’t the future, or even the outlier of the present. We already live in an age when even the computers in our pants pockets know how to do this. Load up Spotify or Rdio on your iPhone and, behind the music, you’re loading up an engine that knows how to examine the music you listen to and recommend new songs to you based upon what you like. That engine is built by the Echo Nest, a music intelligence company that uses computers to ‘listen’ to music, analyze it, and then contextualize what its hearing by crawling the web for human reviews.

Every single day, the Echo Nest recommends tens of millions of songs to people based upon algorithms. In all likelihood, you have listened to a song that a computer in the server room of the company’s Somerville, Massachusetts, headquarters has told you that you will like. What you might not know is that the same techniques used by composers like Oberholtzer and Cope to generate entirely new musical works are also being used by your favorite streaming music app to recommend your next jam.

It’s a multimillion-dollar business, and growing more lucrative every day. But back in 2005, the Echo Nest’s core business was built on the back of a Ph.D. thesis by co-founder Tristan Jehan, not in how to recommend music to customers in the digital age, but in how to generate entirely new musical works just by listening.

The difference between Oberholtzer and Cope’s techniques and those of the Echo Nest come down to approach. “The way we did it was very different from the way some composers do it,” Jehan tells Co.Design. “We came up with a system where a computer could just listen to existing audio and figure out all the rhythms, harmonies, tempos, and beats. We didn’t examine existing works and try to tell our computer what it all meant. We taught the machine to listen to music and derive the meaning for itself.”

Once Jehan’s algorithm had listened to enough music by an artist, it could then synthesize new works in that artist’s style. “The way it worked was by finding the closest sounds in a huge library of audio examples of an artist’s works,” recalls Jehan. “Let’s take James Brown. We could use James Brown’s existing sound library and recombine them intelligently in such a way that it would produce an entirely new work, not based on Brown’s intention or what he wrote down, but based just on what James Brown’s music actually sounds like.”

advertisement

Computers and algorithms don’t usurp my agency as an artist. They empower it.

Teaming up with fellow MIT alum Brian Whitman, who was working on the other half of the problem–teaching computers what music meant to us–the Echo Nest was eventually born. But it almost became something more than a recommendation engine.

“Early on, when we started the Echo Nest, some people wanted to use our technology to help them come up with algorithms that could come up with the next big hit,” says Jehan. “We could do it, but we find that to be evil in some way. It would help publishers, but what we really want to do is help people find the music they like, not strip music down to a single formula of success.”

Music. Composed by algorithms. Programmed for big hits. Even music publishers are interested. Could this be the future of music? And would anything be lost if it were?

All of this returns us to the question of authenticity. Is music that a computer or algorithm has helped to write as “authentic” as music that has been written in a more traditional way?

“If you write music with a computer, there’s this perception from people that somehow authenticity has been removed,” says Oberholtzer. “What people are really saying here is that computers are magic. That’s not true. I know how computers and algorithms work. They are my tools. When people talk about how computers remove authenticity, or soul, or the human touch, what they are talking about is agency: an artist’s ability as a human to make creative decisions based upon what they want to express. But computers and algorithms don’t usurp my agency as an artist. They empower it.”

At least in part, music has always been written by algorithm.

Echo Nest co-founder Tristan Jehan agrees. “Electronic, rap, hip-hop. There’s all kind of music that couldn’t exist if the technology to do it wasn’t there. Machine listening and automatic composition are just tools for humans to use, and a human always needs to program them. The machine will never replace the artist. There’s always a creative process.”

advertisement

None of this is new, as David Cope is eager to stress. “At least in part, music has always been written by algorithm,” he says. “All computers do is allow the algorithms we write to be more exquisite and more sophisticated. I can’t think of anything we can’t do in the future with them.”

Throughout history, every artist has ultimately faced the same task: To examine the creative murk inside themselves and come up with a process to explain the unique way they see the world to another human being. These processes have always been algorithms, and every brain on Earth is a computer of a kind. The computer algorithm is as old as art is. If that’s not authentic, what is?