An Artist Uses an iPhone to Visualize Sounds in AR

Zach Lieberman is making sounds. “Click.” “Psh.” “Ah.” “Oorh.” “Eee.” With every noise, an amorphous white blob bursts on a screen, leaving a trail of shapes lingering in the air. As Lieberman moves his phone backwards through the cloud of blobs, the noises replay in reverse as if he were rewinding a vinyl record.

It looks and sounds like his phone is traveling through a time portal, but Lieberman has a different explanation: “I’m basically recording audio in space,” he says.

The reality, of course, is slightly more complicated. Lieberman, who runs the School For Poetic Computation in Brooklyn, built his real time sound map using Apple’s ARKit and the coding toolkit OpenFrameworks. Like all apps built with ARKit, Lieberman’s uses SLAM Technology—that’s simultaneous localization and mapping—which leverages the phone’s sensors and camera to build a low resolution map of a room’s boundaries and contours.

With this environmental information in hand, Lieberman can capture sound through the phone’s mic, process and visualize it with the app he built, and then map the illustrations to an exact location in the room. You can leave a snap in one corner, a word in another, and watch a trail of sound waves appear exactly where the phone first captured them. Move the phone along the 3-D path, and it’ll work like a scrubber, replaying the audio forwards and backwards in real time. “It plays that chunk of audio that’s closest to where the devices is,” Lieberman says.

Compared to the glossy apps that game developers and companies like Ikea are creating with ARKit, Lieberman’s living room experiment feels almost like a rough draft. And in many ways, it’s just that. In its current form, the app doesn’t remember the position of the sound waves indefinitely. Close the app and reopen it, and it won’t recognize the exact spot where you sang happy birthday. “It knows where you paint the sound in space, relative to where you open the app,” he says. “But it doesn’t know that space is in Brooklyn or at this intersection or in this room.”

Still, the app provides one of the most obvious and transparent illustrations of how augmented reality works and what it can potentially do. Lieberman, like many of the other artists using augmented reality to produce new experiences, views his app as more of an experiment than anything. It’s a way to move past the commercialized experiences that inevitably follow hyped technology and think more critically about how it can be used. “It’s like, OK, we have a camera or microphone in space, so what does that mean?” he says. “It’s important to think about how we can push the interaction with that in different directions.”