Generating MIDI with physics

October 5, 2016

Tinguely sculpture

This is an idea for an application that generates MIDI notes from moving machine parts. Not real machines however, but software simulations. Cogs, wheels and gears that rhythmically rotate and drive each other and while doing so transmit their rotation, torque and most importantly their collisions as MIDI notes and CC messages.

I had the idea when I was at the Stedelijk Museum and saw an artwork by Jean Tinguely (see below). It’s a machine-like construction of fragile looking metalwire cogwheels. Although it could clearly function and even has a small electric motor at the lower left corner, a short electric wire ends with an unattached connector.

The machine probably wouldn’t last long if was turned on. The wires are so thin. I imagined how the parts would move if it was set in motion. I realised that I would probably never see it turned on in reality, but that something like this could be simulated in software with a physics engine.

Machine sounds

I’ve seen other machine sculptures by Jean Tinguely over the years. Many of the larger ones are still in operation. It’s fascinating to see the large and complex but useless machines work and move endlessly without ever reaching a goal. But equally impressive is the sound. The screetching noise of metal sliding over metal and the percussive sounds of rusty iron parts hitting each other.

A simulation of a Tinguely machine should always include sound as well as motion.

Box2D and the Web MIDI API

This post shows a web application that uses Box2D to create the machine, and the Web MIDI API to send MIDI out of the application to connected hardware or other software that can receive MIDI messages to generate sound. A sampler, synthesizer or drum machine for example.

Box2D is a ‘2D rigid body physics engine’. It simulates how physical objects would behave and interact as if they had mass, elasticity and friction, and were subject to forces like gravity. If you see a wall of blocks tumbling down realistically in a game, that’s probably calculated by a physics engine.

The Web MIDI API is a new browser standard that provides access to MIDI devices within web browsers. That means a web application can detect MIDI ports and send or receive MIDI messages from Javascript to connected MIDI devices. It’s not yet available in all browsers. Check CanIUse.com to see which ones currently support it.

This is only a first proof of concept, so there are no complex systems of gears and cogs. Only four rotating rectangles, two of which are motorized. To keep the animation going.

Select an output if you have any MIDI equipment to try the app out yourself.

The Reset button resets the four rectangles to their start position. This is useful because they usually get stuck after some time. It also randomly changes the length of each rectangle somewhat, so the animations will be different each time.

The Animate checkbox pauses the animation.

To have a quick look at the application in action I made a short screengrab video as a demonstration:

What the app does

The app sends MIDI note and pitch bend data to the selected output.

A note is sent each time two objects collide, and its pitch is always 60 (C3).The note’s velocity depends on the force with which the two objects collide.The note’s channel depends on which two rectangles touch:

Channel 1 if the left and center objects collide

Channel 2 if it’s the center and right top

Channel 3 for center and right bottom

Channel 4 for right top and right bottom

Pitch bend is sent continuously. It’s directly translated from each of the rectangles’ rotation. Each rectangle sends on one of the first four MIDI channels.

Because the rectangles don’t rotate freely but bounce off each other in unpredictable ways, the pitch bend value goes up and down following an interesting and random path.

This is the the pitch bend data of a demo song recorded in Ableton Live:

MIDI pitch bend data from the Box2D app recorded in Ableton Live.

The results

When I had a working prototype the first thing of course was to try and make some music.

I connected the app to Ableton Live through the IAC Bus (on Mac OS X) and made a demo song.

It’s obvious this music is completely random. There are absolutely no patterns to be found. It’s more of an ambient soundscape than a typical song. But I like it. The sounds that pitch up and down and the white noise fading in and out remind me of tuning an old radio in between stations where you hear beacons, test signals and bits of morse code through the noise.