Using iPads to bridge communication gap with dolphins

The iPad may indeed be "magical" if, as research scientist Jack Kassewitz …

Research scientist Jack Kassewitz has found that the iPad's touch-based interface is so intuitive that even some nonhuman species can use it. In this case, that species happens to be dolphins. Kassewitz is using iPads with custom-developed software to help facilitate two-way communication between humans and dolphins.

Kassewitz has worked for years studying the behavior and communication patterns of dolphins. Numerous studies on dolphin language show signs of advanced intelligence, and it is believed that the high-frequency sounds dolphins make underwater are capable of communicating information that is holographic in nature. Since humans don't communicate natively with holograms, Kassewitz is currently working on a project to build a symbolic language that dolphins and humans can use to communicate with one another.

Kassewitz searched for nearly two years to find a touchscreen device that dolphins could reliably activate with their rostrum (or beak), while still being powerful enough to record or play back the high frequency sounds associated with dolphin language and durable enough to work in underwater environments. He had originally settled on the Panasonic Toughbook, but recently began evaluating the iPad as an alternative.

The iPad is suited to Kassewitz's research in a number of ways. "It's small and lightweight," Kassewitz told Ars. "It's very forgiving. For example, if I turn it the 'wrong' way, it turns itself back the 'right' way. And the iPhone OS system is fast—more than fast enough for my use."

Kassewitz is currently using a sealable bag that protects the iPad underwater to depths of a few feet, though he is also working with Otterbox to make something more robust and with better anti-glare capabilities to make it easier for the dolphins to see the screen. Bluetooth allows him to connect to speakers to "hear" the underwater dolphin speech, and he can view a spectrograph of the sounds on the iPad's screen.

Kassewitz is also taking advantage of the undocumented USB audio capabilities of the iPad Camera Connection Kit to interface with some specialized audio recording equipment. He uses a series of underwater microphones (or hydrophones) to record the unique sound patterns of dolphin speech made while interacting with the iPad, to try and determine what patterns are associated with symbols displayed on the screen. "We think that once the dolphins get the hang of the touchscreen, we can let them choose from a wide assortment of symbols to represent objects, actions, and even emotions," Kassewitz said. He believes that his team will then be able to develop a rudimentary symbolic language.

"I've been doing this for a long time, just trying to understand dolphins as a species," Kassewitz told Ars. "One of the things I am convinced of is that dolphins are as frustrated with us as we are with them in terms of attempting to have some kind of cross-species communication."

The first step in building that system of communication is a very simple game wherein a dolphin named Merlin is shown an object, such as a ball or a rubber duck. (Kassewitz told us that dolphins respond well to the color yellow.) Then Merlin has to point to an image of the object on the iPad's screen, selecting it with his rostrum.

"Games are a relatively simple way to build an understanding between two animals—humans included," Kassewitz told Ars. "Games require agreements to work, and agreements require some high-level thinking." Ultimately, Kassewitz will build a library of symbols that dolphins can recognize that form the basis of "a complete language interface between humans and dolphins."

Kassewitz's research team will conduct more tests this July, pitting the Toughbook directly against the iPad to determine which platform will be used going forward. However, he believes that the iPad's size and weight advantage may prove to be the deciding factor. "We could use two or three iPads showing different sets of images, and the dolphin would be able to choose among them," he said.

Well said, I think any tablet/slate type computer with a sensitive enough touch-screen display, a webcam, a mic and an accelerometer would easily have accomplished what the iPad has done in the article, perhaps even more, since other devices are not so locked down like this amazing iPad device we all love so much.

To be fair, I was planning on posting the same thing, but there is this sentence from the Article: "Kassewitz searched for nearly two years to find a touchscreen device that dolphins could reliably activate with their rostrum (or beak), while still being powerful enough to record or play back the high frequency sounds associated with dolphin language and durable enough to work in underwater environments."

Much of the tablet PC stuff has been resistive touch screens that take more pressure or are less sensitive, or even only work with a stylus. I can buy that he had a little bit of trouble finding a capacitive touch screen tablet pc that could be easily triggered by a dolphin's nose.

I'm not saying something like that hasn't existed for a long time, I'm not really sure, but at least I don't think any old tablet PC would work.

This has to be the funniest 'first post' I've ever seen on an Ars article:

nemo20000 wrote:

How long before the dolphins complain about the lack of Flash?

Secondly, if you're ranting about how this guy must be paid by Apple or how the iPad interface isn't intuitive enough or how any touch-capable device could have done this or whatever, you might just want to get your meds adjusted. This whole thing is kind of interesting, but it just isn't that big a deal.

I'm not saying something like that hasn't existed for a long time, I'm not really sure, but at least I don't think any old tablet PC would work.

The Joojoo tablet.

Quote:

Now you've done it. Don't say I didn't warn you. At this point, I guess I'll spare you some time and point out this awesome video of Iggy the cat showing his artistic side:http://www.youtube.com/watch?v=6Tqy56tHFeQ

Cool! I like how at the end the cat seemed to figure out if he stretched his paw out he'd get different colors with each digit. What else you got in your animal - iPad inventory?

So, when a dolphin talks to another one about a fish. We believe they are actually sending an image of the fish and not just a symbolic word like we do? Interesting thought right there. I mean I know that humans think in images also. When I think of a fish, I see it in my mind...I guess Dolphins are just able to use the better medium they are in to streamline the transfer of data between entities.

I do find this fascinating as Dolphins can understand us in ways we can not really understand each other. Dolphin Therapy being one example. Now who knows...may be we can finally have a Dolphin Universal translator...heh.

When I think of Fish... usually the first thing that pops into my head is a visual of the actual word in english (times new roman font), then it is quickly replaced by sounds, smell and a visual representation.

I'm not saying something like that hasn't existed for a long time, I'm not really sure, but at least I don't think any old tablet PC would work.

The Joojoo tablet.

The Joojoo tablet has active cooling. I don't think it would work very well for extended periods underwater or sealed in a watertight container. At this moment, there are really no options for tablets big enough, fast enough, and able to be used in sealed containers. The iPad seems the best choice right now when you take the environment it's used in into account.

Next question--anyone figure out a way to start talking to whales? The iPad seems out because even if they had similar beaks they could use on a touch screen, whales are just so much bigger...

This is really intriguing. I will be keeping an ear out for further developments!

Yeah, hopefully the symbolic language can be adapted to whales, or at least I hope we figure out how to understand their language before that giant space cylinder shows up asking questions we can't understand and wrecking havoc on the planet.

One thing that I always wonder about when it comes to highly niche iPad/iPhone apps like this is do these people have to publish their apps in the App Store when they only want to use it themselves? Do they have random people downloading and using their software?

There is certainly evidence that similar sounds reoccur in similar situations (http://upcommons.upc.edu/e-prints/handle/2117/6150). There is also recent evidence that dolphins have encoded "names" that refer to individuals (http://www.cell.com/current-biology/ret ... 2206018203). But as of yet, any speculation about full-fledged language is still speculation. On the other hand, dolphins are clearly interested in us, including mocking humans they interact with, which provides excellent opportunities to conduct research.

One more point. Humans have at least one form of three-dimensional language: sign languages. ASL, for example, uses modifiers and positional indications as three-dimensional grammatical constructs. If it does turn out to be holographic, sonar-based language wouldn't be unique. It would make sense if sensory organs adapted for sonar were used for abstract communication that it might more closely resemble the language that evolved for our visual system than that which evolved for our auditory system.

Research scientist Jack Kassewitz has found that the iPad's touch-based interface is so intuitive that even some nonhuman species can use it. In this case, that species happens to be dolphins. Kassewitz is using iPads with custom-developed software to help facilitate two-way communication between humans and dolphins.

What in the heck kind of reporting is this?

You say the the iPad's interface is intuitive and then say he used CUSTOM software to communicate with dolphins?

I'll just forego the assumption that humor is universal, but let's remember that even "CUSTOM" software uses the touch-based interface that is common among all iPhone OS apps. So, the iPad is not running a "CUSTOM" OS, but it is running an app designed to present a dolphin with one more more images to choose from, while also capturing any vocalizations that the dolphin makes when making a choice. The hope is that similar patterns will emerge that can be associated with the image of a particular object. Taking it further, the app may then be able to playback identified patterns with those images as they are displayed.

Quote:

He could have just as easily used ANY touch based computer, smartphone, or tablet and had the same response because (shocker alert!) its CUSTOM software.

And saying its small, lightweight, has a working accelerometer, and is "fast" does not mean that the iPad is the guys only choice. Sheesh.

The article quite explicitly states that Kassewitz is also considering a Panasonic Toughbook. Although I think other commenters have pointed out that there aren't other products on the market that combine the iPad's sensitive capacitive touchscreen, light weight, size, passive cooling, and more. I'm sure Kassewitz would have considered other possibilities that fit his criteria if they existed.

There is certainly evidence that similar sounds reoccur in similar situations (http://upcommons.upc.edu/e-prints/handle/2117/6150). There is also recent evidence that dolphins have encoded "names" that refer to individuals (http://www.cell.com/current-biology/ret ... 2206018203). But as of yet, any speculation about full-fledged language is still speculation. On the other hand, dolphins are clearly interested in us, including mocking humans they interact with, which provides excellent opportunities to conduct research.

I think we might be delving into the semantics of what constitutes a language or whether our concept of a language is robust enough to include the communication systems of being other than humanoids. But the overwhelming body of research clearly shows communication is happening, so it certainly seems possible that we might be able to develop some way of communicating with each other, whether that is with a "language," or mental projections, or some cool computer translation à la SeaQuest DSV.

Quote:

One more point. Humans have at least one form of three-dimensional language: sign languages. ASL, for example, uses modifiers and positional indications as three-dimensional grammatical constructs. If it does turn out to be holographic, sonar-based language wouldn't be unique. It would make sense if sensory organs adapted for sonar were used for abstract communication that it might more closely resemble the language that evolved for our visual system than that which evolved for our auditory system.

One thing that I always wonder about when it comes to highly niche iPad/iPhone apps like this is do these people have to publish their apps in the App Store when they only want to use it themselves? Do they have random people downloading and using their software?

Yeah, hopefully the symbolic language can be adapted to whales, or at least I hope we figure out how to understand their language before that giant space cylinder shows up asking questions we can't understand and wrecking havoc on the planet.

There is certainly evidence that similar sounds reoccur in similar situations (http://upcommons.upc.edu/e-prints/handle/2117/6150). There is also recent evidence that dolphins have encoded "names" that refer to individuals (http://www.cell.com/current-biology/ret ... 2206018203). But as of yet, any speculation about full-fledged language is still speculation. On the other hand, dolphins are clearly interested in us, including mocking humans they interact with, which provides excellent opportunities to conduct research.

One more point. Humans have at least one form of three-dimensional language: sign languages. ASL, for example, uses modifiers and positional indications as three-dimensional grammatical constructs. If it does turn out to be holographic, sonar-based language wouldn't be unique. It would make sense if sensory organs adapted for sonar were used for abstract communication that it might more closely resemble the language that evolved for our visual system than that which evolved for our auditory system.

Actually the guy is an idiot since he didnt think of it sooner using tablets before the slate... (vertial slice idiot, not entirely an idiot) Any touch based interface large enough could ahve software written for it to do this.

BTW - Wild dolphins, for the most part, dont give two shits about humans in any real way, but if he wants better communication with NON wild dolphins, then more power to him... MAybe one day we can understand the "language" of more animals. But lets not act like this is only possible because of the iPad.

Its a story like this that makes the tin foil wearer's scream "bias" or something like that... I mean, why isnt this in the science section to begin with?

...communicating information that is holographic in nature. Since humans don't communicate natively with holograms..

I don't understand this, anyone care to explain?

The gist is that scientists are convinced that when one dolphin communicates with another, it is transmitting three-dimensional imagery, as opposed to speaking a "verbal" language as we think of it.

Ouch, my brain just popped out of my head. You know it really does make so much sense, since dolphins "see" with sonar and have amazing acoustical range why bother teaching your kids the word Fish when you can send an image of a fish (i.e., what a sonar echo would show!)

Truly amazing beyond belief! I wonder what it is like to be a dolphin with dyslexia…

Kassewitz seems like he is paid by Apple.For the purpose of product endorsement not to his fellow scientists, but to regular people.

Yes, of course...anyone that would do this must be a paid shill or a fanboy. "Normal" people certainly wouldn't use Apple products, am I right? This has to be totally made up crap...just like I don't believe for a moment they've sold over 2 million of these things. Yeah, right....they tried to fool me about the moon landings too but we all know the truth about that too, am I right jdietz?

People like you and me know the truth about Apple. We have to make sure everyone knows our displeasure at them and the people that use their products. They certainly aren't like WE are, am I right brother? They're not gonna fool us. We're not "sheeple"...and I bet the dolphins wouldn't use any Apple products either. They're obviously OSS fans. I saw one wearing a Cory Doctorow tee-shirt once and they all smell similar Stallman, so you KNOW the truth!

High five my brother! WOOOOOOO!

Actually, the problem is that there have been tablets and slates on the market for about.. mmm.. 8 years now... and some of them have been compact enough to do what this guy has done with the iPad.

It seems really weird that all these people are suddenly doing things with the iPad right now that they could have been doing before with other touchscreen devices - as if somehow the idea of talking to dolphins using a touchscreen computer simply couldn't have happened until Apple built the iPad.

Scientists and other innovators are also after exposure, so they pick Apple products knowing that the media will cover it, especially new hot Apple products

I think the perception is that you run a story like "Scientists use tablet pc to communicate with Dolphins" and no-ones really interested they'll assume that the tablet PC is some sort of obscure technical science device and switch off, however say "Scientists us iPad to talk to Doplhins" and people will think hey I have/know someone with/could get easily a iPad, missing the point that the innovation here is actually the idea and the software.

The iPad doesn't matter. Having an API that supports accelerometers and multitouch without requiring you to roll your own or to hook up a separate device, that's the interesting part.

If you could use something else but hacking around the OS's lacks or deficiencies costed you more time - and more money - you'd go for the fastest, cheapest solution, too. It's why all these people on Youtube doing cool motion tracking stuff are hacking Wiimotes. It's easy, cheap and ubiquitous.

"One of the things I am convinced of is that dolphins are as frustrated with us as we are with them in terms of attempting to have some kind of cross-species communication."

I hate that we as humans are that arrogant to think for some reason a dolphin is "frustrated" that it cant "talk" to us

Actually, Kassewitz had a very nice anecdote to explain why he thought this, though I ended up leaving it out because it seemed quite tangental to the focus of the article. However, and I'm paraphrasing here, the gist is that Kassewitz was recently working with one of his dolphins named Jupiter. Kassewitz was using an underwater HD camera rig to film the dolphin, while the dolphin had its own compact video camera rig that it can hold in its mouth and film with as well—it even has a display that the dolphin can monitor. At one point, Jupiter dropped his camera. He then went to Kassewitz, and for 25 minutes got right in his face and was "vocalizing." Kassewitz believes that Jupiter was trying to tell him something about the dropped camera.

It's not definitive scientific proof, but I'm willing to trust the perceptions of someone that has dedicated his life to learning to communicate with dolphins.

This is a cool article and concept, but while the article itself mentions the use of other devices, the search for the best device, and at least one issue with the iPad (besides not being waterproof), it is slightly annoying how cheerleader-ish the overall article and title come across as. I agree with the dolphins, I don't like glossy screens either. But I do agree with the poster who commented that since the iPad is hot, people piggyback on it to get media exposure, just feeding into the entire mystique of Apple products, which of course tends to be self-fulfilling, since that gets people interested in developing for it. Still, cool what people are trying to do with the device. Since he was talking about using several of them, maybe he can ask Microsoft for a Surface. Or make a wall of iPads, I suppose. Actually, that does sound fun...

Scientists and other innovators are also after exposure, so they pick Apple products knowing that the media will cover it, especially new hot Apple products

Do you have some evidence for the two claims that you're advancing here?1. Scientists are after exposure (IME, most don't care for publicity other than journal publications)2. Scientists choose Apple products because they're 'hip' (again, IME, scientists choose equipment that's good for the job, or that they're familiar with; 'hip' or 'hot' is generally meaningless to them)

Or is the above just 'common sense'?

Edit for clarifiation: I'm not trying to talk smack here, nor to pick a fight. It's just that comments such as this one seem very much to be 'just so stories' which are plausible-sounding, but which are generally offered up without any actual evidence to back them up, or perhaps with evidence cherry-picked from a much larger and less-conclusive set. Maybe scientists really are out for media exposure, and the ones I've met are oddballs; I don't know. But those offering these stories seem quite certain of them and I'd like to know more about the evidence which founds that certainty.