Supersensory Substitution Technology

I’m biased, but my wife, Nicola Twilley, had a great feature in The New Yorker’s “Innovation” issue earlier this month, about an emerging type of device known as “sensory-substitution technology.”

For the piece, Nicky met a man named Erik Weihenmayer, a congenitally blind mountain climber—in fact, he is “the only blind person to have climbed Mt. Everest.” Weihenmayer climbs using a device called the BrainPort, held in his mouth; it converts one sense (sight) to another (touch).

A decade ago, Weihenmayer began using the BrainPort, a device that enables him to “see” the rock face using his tongue. The BrainPort consists of two parts: the band on his brow supports a tiny video camera; connected to this by a cable is a postage-stamp-size white plastic lollipop, which he holds in his mouth. The camera feed is reduced in resolution to a grid of four hundred gray-scale pixels, transmitted to his tongue via a corresponding grid of four hundred tiny electrodes on the lollipop. Dark pixels provide a strong shock; lighter pixels merely tingle. The resulting vision is a sensation that Weihenmayer describes as “pictures being painted with tiny bubbles.”

What’s particularly interesting, however, is that these are still just the earliest days of investment and research into what sensory-substitution devices might someday be able to achieve.

They could lead, for example, to the creation of artificial “superabilities,” or synthetic senses that act as a mix between our existing bodily inputs. Through the use of these sorts of devices, Nicky writes, humans “may, depending on the data transmitted through their skin, be able to ‘feel’ electromagnetic fields, stock-market data, or even space weather,” or “enable us to ‘see’ bodies through walls using the infrared spectrum or to ‘hear’ the location of family members using G.P.S. tracking technology.”

I suppose the next question would be to imagine a world in which this is possible—humans feeling space weather or seeing bodies through walls—and then to design the landscape accordingly. Stage sets in which people moving behind walls is part of the action, or outdoor gardens and parks tingling with the pinprick stimulation of otherwise invisible solar flares. Financial analysts high on the fumes of laser printers sit pensively in a dark room feeling stock market data wash over their arms and faces.

Recall, of course, the “Animal Superpowers” project by Chris Woebken and Kenichi Okada, that allowed human users to “see” the world through the senses of animals, one example of which is pictured above.