Damn Interesting

Can You Hear the View?

Cybernetic senses have been the subject of science fiction for decades. The idea of using sophisticated technology to repair damaged bodies, or even to enhance normal ones, has a tremendous appeal – but how far have we progressed towards that goal?

In some ways, we’ve gotten amazingly far. Cochlear implants are now a normal – if controversial – treatment for deafness. They substitute for damaged or missing portions of the inner ear, gathering and processing sound. The first generation of cochlear implants provided only a distant approximation of sound, making them of limited usefulness, particularly for understanding speech sounds. Even the more sophisticated models of today have yet to approach the functionality of a normal ear, though they are far more useful than their predecessors.

While those dealing with cybernetic hearing seem to have decided upon their basic approach, dealing with lost vision is a different ball game. Several different research groups, using various methods, are attempting to produce cybernetic vision. Some, like cochlear implants, seek to replace a malfunctioning part. Others are attempting to produce something entirely new that will nonetheless function as vision. One of the projects that is furthest along is using exactly that kind of substitution. Rather than attempting to somehow re-engineer the eye, the vOICe system (Oh, I see) is using sound to bypass the eyes altogether and substituting the ears in their stead.

The vOICe system was developed by Dr. Peter Meijer – a senior researcher with Philip Research Laboratories (a Netherlands based company). It uses a computer program, a pair of video-sunglasses (sunglasses with a small video camera on the bridge), and a pair of stereo earphones to provide an auditory image of the world. Scanning left-to-right, once a second, the program translates the images seen by the camera into coded sound for the user to interpret. The format is fairly simple – louder sounds mean brighter, higher pitch means that something is higher up in the visual field, and so forth.

Learning to translate the sounds into visual meaning is another task altogether. Users of the device liken it to learning a foreign language. As with a foreign language, the more the vOICe system is used, the more quickly the user gains facility. Most users appear to start the learning process with a set of images on a computer screen (available as a free download off the Internet), and then progress to using the mobile system. The amount of information that can be distinguished even by a novice user is fairly impressive. Within a week of starting use, at least one congenitally blind woman was reporting being able to distinguish walls, stairs, and windows in her house, as well as whether the lights were on or not. The vOICe website reports that a trained user can distinguish approximately 1000 to 4000 pixels per 1 second scan. Comparatively speaking, an average sighted person can identify visual objects in an image of 32×32 pixels – or about 1024 pixels.

In what may be the most interesting part of the vOICe system, its constant use seems to cause a sort of induced synesthesia in the user – a cross-wiring of the senses, where input from one sense is perceived in another. Brain plasticity – the ability of the brain to rearrange itself in response to demand – seems to come into play, as the brain sorts auditory input into visual data. Some previously sighted vOICe users report consistent visual responses comparable to blurry or foggy vision, while their awareness of the sounds themselves recedes into the background. Users who have been blind from birth obviously cannot compare their experiences to a previous experience of vision, but they too seem to rapidly stop processing the vOICe input as auditory data.

32 x 32 pixel image

While the results so far are exciting, there are a few downsides to the vOICe system. Since distinguishing the auditory landscapes requires good hearing, the system is not going to work well, if at all, for someone with any sort of hearing impairment. Additionally there is some concern that the headphones and the sounds produced by the system could interfere with normal hearing function while the system is in use. However, the difference in how the brain processes the vOICe data seems to enable less interference between the competing sounds than one might imagine, with some users reporting being able to use the system while sewing, listening to TV, or even listening to music.

Slow scanning speed is likely to be the most difficult problem to improve. While retinal or brain implants (neither yet available for general use, but under investigation) allow scanning the visual field between four and eight times a second, the vOICe scan is only once a second – a fairly slow pace for interacting with the world while moving, but dictated by the need for the user to process the sounds. The brain seems to be able to compensate, though, and the vOICe users have few complaints about slow updating speed.

The last major limitation is that of current technology. While the system is portable, the need to carry a laptop, and wear a video camera and earphones does use up some carrying capacity. The limitations of the laptop battery also make venturing forth for long periods of time problematic. These problems will ease with time though. Even in the short time the vOICe has been available, new and smaller devices have become available for each of the system components. A major plus for the system is the ease of upgrading as newer technology comes on-line. Since the system is physically separate from the user, upgrading is as simple as buying the new equipment and integrating it. Upgrading a retinal implant, let alone a chip in the brain, is a much trickier proposition.

As promising as it is, the vOICe system is far from the last word in cybernetically enhanced vision. Brain implants, retinal implants, and devices that translate visual imagery to touch are all under active investigation. Thus far vOICe seems to have a clear lead by utility, by its lack of invasiveness, and even by cost, but who’s to say which of the other contenders may surpass it tomorrow?

Related Content

It reminds me of the line from the first Ghostbusters, “Listen!…Do you smell something?”

scarr

Posted 16 June 2006 at 06:37 am

hmmm, if that pic on the upper right side is any indication, it makes me think Mr. Laforge from “Star Trek: The Next Generation”

I guess it wasn’t that far off at all.

except for the whole warp speed thingy.

scarr

Posted 16 June 2006 at 06:40 am

wait!

think of the possibilities!
You could make Tecno music that while a normal person would say is nonsense, someone with this apparatus could “see” as a movie!

you could implant systems in a car for driving, and have it go through the stereo!

this is damn interesting.

schuylercat

Posted 16 June 2006 at 07:03 am

I spent several months working in the IT group at the House Ear Clinic in LA – Dr. House invented (I think he did, anyway) cochlear implants. I was stunned at the whole process and especially the comparative functionality – the description in the article: “The first generation of cochlear implants provided only a distant approximation of sound, making them of limited usefulness, particularly for understanding speech sounds….” was an understatement – the “sounds” perceived by the recipient of an implant were more like crackles and static than sound, and made absolutely no sense to me whatsoever.

While there I was also introduced to a man who performed an experiment back in the 1970’s: he made a pair of prismatic glasses that inverted vision from top and bottom for the wearer. He wore them for several weeks and lived his daily life. For several days he doddered around, bumping into things, seeing everything upside down. He vomited, the experience was so disorienting.

The breakthrough came, he said, after only a few days – his brain “compensated” for the upside down world – he still saw things upside-down, but that became “normal,” and he stopped “…trying to pour my coffee onto the ceiling instead of my cup on the counter.”

After the test was over he went through the whole thing again when he took the glasses off – dizziness, disorientation, nausea, while his brain rewired itself. Coupla days – good as new.

The whole experience was a fascinating view into the abilities of differently-abled people and their bodies’ ability to sometimes make much out of little or nothing – sound, vision, movement, whatever they have…or don’t have It made me feel guilty for every snide, snotty comment I ever made as a kid about deaf, blind, and disabled people. “Lame-O” was a favorite of mine when I was ten or twelve, little bastard that I was.

When I had a small stroke and became paralyzed on my right side years later the point was driven even closer to home: I got to learn how to do everything – going to the bathroom was particularly frustrating – all over again, like a child. I regained movement after a year and suffer few ongoing effects (I trip over the nap in the carpet – totally clumsy), but I look at people – from a guy with a white cane to a child with an amplifier on their waist for the hearing implant to Stephen Hawking – a little differently now.

Damned interesting, indeed – Thanks, Cynthia, for that piece.

Buckelew

Posted 16 June 2006 at 07:34 am

My son received a cochlear implant just before the age of 2. He’s now 8 and does have a problem understanding many of the lazy lipless dialects here in Alabama, but he is a great communicator.

He loves to interact with puppets as if they can talk. He loves to play house with “action figures” and other dolls, to pretend he’s a storekeeper with great window displays, drawing the whole family into his act as shoppers.

He loves music and dance and song. And talks and talks and talks. He insists on wearing his device from the first waking minute to well after he has talked himself to sleep.

From the first day his device was turn on, we have been blessed to see our beautiful son blossom. We have many hurdles to clear and difficult times ahead, but our (his) experience is nothing short of miraculous.

We’re forever thankful to the visionaries and deaf recipients who helped bring cochlear bionics to fruition. The world waits with anticipation of the next great wonder.

Anthony Kendall

Posted 16 June 2006 at 07:58 am

One of the best parts of Damn Interseting for me have been the comments. These past two comments added so much to this article. Thank you both for your experiences!

kc0dxh

Posted 16 June 2006 at 08:20 am

Buckelew, sounds like you have a DI story yourself, as does your son. I’d love to hear more! How well does he play with other kids his age? How has that been different from others? How has that paralleled others? Is he able to participate in a classroom setting without additional help? Is his speach understandable to those outside his circle of friends? Does he have some of that southern drawl?

klone

Posted 16 June 2006 at 10:26 am

i have heard about that guy who “made a pair of prismatic glasses that inverted vision from top and bottom for the wearer” (schuylercat’s comment). we always wanted to do that as teenagers, out of curiosity and boredom, but the whole nausea thing kind of kept us back.

Buckelew

Posted 16 June 2006 at 10:32 am

Austin, my son, has many school friends and many close cousins his age. His best friend is his sister, who is almost exactly his “hearing age”. She’s very articulate and intelligent and works hard to include him in all conversation.

When Amber was 3, she asked me, “Daddy, why did God allow me to be born?” I replied, “You are meant to be — you have a purpose.” And she said, “I think He let me be born to help Austin learn to hear.” She deeply loves her brother and is the greatest help.

Austin gets discouraged when people (children) don’t take the time to speak clearly and keep him involved in activities and conversations. He reacts by doing his own thing or gravitating towards the very young – whom no one can understand, anyway.

He’s in an inclusion classroom and has some assistance, but he is passing and is well-liked by his peers. He always has a “girlfriend”.

There are other implant children who have absolutely excelled in all ways. Austin has struggled a little. And we’re working on it. I believe there will be a time when he realizes what’s at stake and focuses on becoming a better listener and student. Right now, he’s a typical boy : ) Most folks who meet him are blown amazed by his articulation and understanding. And there’s no drawl LOL.

I’d give everything I’d every have or do to make sure he could hear without assistance. But our experience has been great.

ballaerina

Posted 16 June 2006 at 11:14 am

I completely agree with Anthony Kendall – these comments add so much to the story because they bring it down to a personal level.

On another note, has anyone heard about the blind man who taught himself to echolocate? Apparently he walked around clicking to himself and could report images. I have no idea if this is just a myth I heard somewhere or if it’s actually possible.

Josh Harding

Posted 16 June 2006 at 11:44 am

I use echolocation with my kids…when they start screaming, I can tell where they are, if they are hurt or just offended, and which one of them it is. I can do the same with my siblings’ kids when we are together.

But joking aside, the miracles of modern medicine are tremendous. A couple months ago, I saw some show were they gave a boy back his sight in one eye. I take my sight for granted too often.

lp

Posted 16 June 2006 at 12:01 pm

Great comments. Good stuff. Buckelew, that’s awesome about Austin.

Stephen Gordon

Posted 16 June 2006 at 01:05 pm

I read recently about another project that allowed the blind to see via electrically stimulating the tongue. The user would put in a mouth piece and mild electric shocks would be sent to the tongue. Why the tongue? It took less power for some reason – perhaps the moisture – and its very sensitive.

Reportedly users could learn to navigate around rooms pretty quickly with this. Well, this wasn’t the original article I read, but this is definitely about the same testing:

scarr said: ”
You could make Tecno music that while a normal person would say is nonsense, someone with this apparatus could “see” as a movie!

you could implant systems in a car for driving, and have it go through the stereo!”

Ooooh! What a genuinely wonderful idea, scarr! (Quickly grabs it and starts running…)

Who needs video anyway? How about ultrasonic (or radar or infrared) imaging presented in the same way in the car? So much for fog-bound!

Perhaps your other idea has more merit, techno music. It works both ways: music that blind people “see” as movies, yes; also “scenes” recorded by blind “cameramen” replayed as symphonies for the rest of us plebs.
Think of the instruments involved; the video, the ultrasonic, the radar, the infrared, … the possibilities are endless. Oh yes, a symphony is definitely on the cards.

I’m off to the patent office now!

AzianBlaster69

Posted 16 June 2006 at 06:28 pm

My father had a cochlear implant (one of the earlier versions that required a magnet to hold a transceiver to a part of his scalp) and it worked very well. Over time, of course, my dad was able to hear much better because he learned to interpret sounds better. I have no doubt that Austin will be able to nearly replicate the same vocal abilities and interpret nearly the same number of sounds as kids born with normal hearing will over time. He has his whole life ahead of him and, boy, is it going to be fun! I have never been so amazed by technology then when my dad heard me say “I love you” without me having to re-write it on a piece of paper.

cornerpocket

Posted 16 June 2006 at 07:41 pm

The missing ingredient in most science articles is the human and individual aspect, part of the reason they are often boring. Seeing personal testimony gives the science a ‘realness’ that adds dimension and flavor. I applaud all of those who added the personal touches that makes this on MegaDamnInteresting.

OmniNegro

Posted 16 June 2006 at 08:40 pm

Stephen Gordon said: “I read recently about another project that allowed the blind to see via electrically stimulating the tongue. The user would put in a mouth piece and mild electric shocks would be sent to the tongue. Why the tongue? It took less power for some reason – perhaps the moisture – and its very sensitive.

Reportedly users could learn to navigate around rooms pretty quickly with this. Well, this wasn’t the original article I read, but this is definitely about the same testing:

I could write an article about how I got lost in a golf coarse right off a lake and managed to wander into a swamp where I got several bad cuts and wandered around bleeding for hours before I found my way out.

If I could see in some way then I wouldn’t have done that.

I think when they get such a device ready for the market I would pay up to a thousand for a unit that’s comfortable to wear and has several hours of battery life, provided it has enough detail to make it worth while.

Many others would pay more.

Before anyone asks, I like walking around in the night time because I like in Texas and in the summer the temperature is in the upper 90s in the night time and I don’t want to think of the temperature in the infernal daytime.

It’s a good thing that they didn’t make the double-used sense the olfactor sense.

You could get yourself some night vision goggles. Much less relearning required.

donlaudanny

Posted 16 June 2006 at 09:01 pm

I just read an article about the development of artificial limbs. One of the biggest stumbling blocks that researchers are trying to overcome is to replicate the senses (heat, cold, location, deep pressure, light pressure, deep pain, light pain) collectively known as touch.

At the rate technology is progressing, I bet it will be within most of our lifetimes when we’ll start to see the first artificial replacements that are superior to their human organic counterparts. Indeed, I can see how jobs like the military where performance is life or death will require the prosthetic replacements.

JustAnotherName

Posted 17 June 2006 at 04:18 am

scarr said: “hmmm, if that pic on the upper right side is any indication, it makes me think Mr. Laforge from “Star Trek: The Next Generation”

I guess it wasn’t that far off at all.

except for the whole warp speed thingy.”

Jordy, right? I HATED episodes where his visor caused a whole series of errors. My friends and I would refer to them as “The Jordy Visor Episodes.”

Orestes

Posted 19 June 2006 at 07:14 pm

ballaerina said: “I completely agree with Anthony Kendall – these comments add so much to the story because they bring it down to a personal level.

On another note, has anyone heard about the blind man who taught himself to echolocate? Apparently he walked around clicking to himself and could report images. I have no idea if this is just a myth I heard somewhere or if it’s actually possible.”

yes this is true, he teaches classes to people and after about an hour they can manage their way around. he also takes rides on his mountain bike for fun, even though hes blind lol

needles

Posted 23 June 2006 at 09:15 am

A girl in my classroom had cochlear implants. Without them she could not hear at all, but with them she could hear, except not backround noise. My teacher got mad when she turned them off though. :)

ForShame

Posted 25 June 2006 at 01:22 pm

“At the rate technology is progressing, I bet it will be within most of our lifetimes when we’ll start to see the first artificial replacements that are superior to their human organic counterparts. Indeed, I can see how jobs like the military where performance is life or death will require the prosthetic replacements.”

There’s an anime (japanimation) tv series and movie about this called Ghost in the Shell. A lot of it deals with the possible problem of someone outside the body being able to control the function of electronic devices within other people. Still, if I was blind or deaf I’d be the first in line to get artificial eyes or a cochlear implant. I take my senses for granted far more than I should, I don’t think I could operate without them.

Matt Faus

Posted 26 June 2006 at 10:49 pm

The singularity is near.

Emmy

Posted 28 June 2006 at 07:24 pm

Hey, cool!

Buckelew

Posted 19 September 2008 at 06:41 am

I feel compelled to share Austin’s progress with those interested in cochlear bionics.

He’s in the fourth grade now and attends therapy twice a month, mostly for motor skill issues, not an uncommon problem for some deaf people. He’s making A’s and B’s in a regular classroom and has very little abbreviation in his workload. And he’s the teacher’s pet. We were told he would probably behave badly because of frustration with his environment (being deaf in a class full of hearing children), but the inverse has been true. He has plenty of friends and is already girl-crazy. Nearly everyone who learns of his disability is stunned and bewildered by the talking, singing, wonderfully imaginative deaf child.

braver13

Posted 22 December 2008 at 01:40 pm

this site is seriously operated by a supernatural force
been a visitor for a year now and have had too many days when i scrouge the site for unread articles, without finding a single one
but today, the random feature was on my side,
first read for this article and really DI

Share This Page

Comments

Scroll to Top
⇧

HALP

Sorry to interrupt.It seems you've really digging into our curated links.We put a lot of effort into finding these gems.Please consider donating so we can keep up the effort.Would you like to know more?

Sorry to interrupt.It seems you've read at least whole articles now.Yay!This is a reader-supported project, and you can get stuff for donating.Would you like to know more?

Hello! This site is an independent project.We despise advertising so we ask for direct support from readers.If you donate you can get stuff.Would you like to know more?