Death to CAPTCHA! Google wants to make them invisible using AI

How to make music with gestures

This article was taken from the October 2011 issue of Wired magazine. Be the first to read Wired's articles in print before they're posted online, and get your hands on loads of additional content bysubscribing online. "I wanted to create a gestural language that could talk with my gear," says Grammy Award-winning musician Imogen Heap. So she replaced her Moogs with custom-made gloves that allow her to create and loop music on the fly. Heap entranced at a recent Wired Future of Music event. Here's how she plucked a masterpiece from thin air.

I. DESIGN YOUR HARDWARE

ADVERTISEMENT

Heap created her gloves with Tom Mitchell, a senior lecturer in music systems at the University of West England. They began with off-the-shelf fibre-optic data gloves made by South African company 5DT. The gloves can sense the position of Heap's fingers. Mitchell combined them with two X-IMU boxes, each containing a gyroscrope, accelerometer and magnetometer. "This was to add on extra layers," says Heap. Mitchell took all the outputs and created a program in C++ to integrate the data.

2. PLAN YOUR CONTROLS

Heap's gloves use four basic movements: fist, open hand, pointing and the "rock" sign (fingers in horns). The gloves operate in six modes: voice record, multi-effects, wrist record, drums, synths and rock mode. Heap switches between them with a gesture on one hand to indicate the start of a mode switch, then a second gesture with her other hand to select a particular mode. LED lights on each wrist tell her which mode she's in -- "otherwise it would be like flying blind," says Mitchell.

3. GET SAMPLING

Heap has Shure lapel-mics attached to each glove. By flicking her wrist, these start recording whatever they're near. Opening her hand starts a recording loop, and clenching it into a fist closes it. Heap uses different hands for different loops -- "like catching the sound". By switching to synth mode, Heap plays notes based upon the position of her hand in the air. In drum mode, for example, a karate-chop right sounds the kick drum, and left hits the snare.

ADVERTISEMENT

4. WARP THE SOUNDS "If your arms are open, the sound should be louder," says Heap. "If your arms are cradling something, it would sound small." By pointing, she can "pan" the sound; raising one hand increases reverb, the other volume. Throwing horns enables rock mode, and distortion is controlled with her two middle fingers.

HOW IT WORKS

The gloves contain 14 fibre-optic elements. An analogue sensor picks up the light, then converts it to a digital signal.

The X-IMU box contains the accelerometer, magnetometer and gyroscope, determining the pitch, roll and yaw of Heap's hands.

The LED light display helps Heap track which mode she's in - flashing white is neutral, green is ready to record, red is on-air.

On her back, Heap has three wireless mics, a wireless return for monitoring and the hub where all sensors are connected by USB.

An earpiece supplies Heap with a clicktrack of 140 beats per minute, in a key of C sharp