Getting in touch with our haptic sense

Do you prefer ‘vibrate on’ or ‘vibrate off’? Well, either way—heads up, as we explore the world of haptics. To get the best information from whatever you choose to touch, haptic sensing involves a lot of neural effort. We'll hear about how this sensing has been examined in the past, as well as some speculation on where haptics might go in the future.

Transcript

Lynette Jones: Some people would make a distinction between touch, which are things that sort of come in contact with the skin and can be under passive conditions, you know, you're resting your hand on the table and you feel the surface of the table just as you're resting it, and haptics, which really in terms of its pure definition involves both the sense of touch and kinaesthesia because haptics involves active touch, so it's not the process now of resting my hand on the table, it's if I want to find that there is a mark in the table or it's been scratched, I will move my finger across the surface, and that is called active touch or haptic sensing.

Lynne Malcolm: Today, producer Diane Dean guides us through the sensory world of haptics.

That was Dr Lynette Jones, Department of Mechanical Engineering, MIT, in Cambridge, describing how our passive sense of touch turns into an active sense, sending information to our brain about an object's texture, temperature, and movement.

David Parisi: So how do you allow someone to touch over a distance? And if you're talking about someone working on an object in space or working on an object under the sea where there is a robot operator and the human operator is operating the robot from a distance, how do they feed back those sensations that the machine is encountering to the human operator? They found that without touch, the machines were really imperfect.

Lynne Malcolm: David Parisi, Associate Professor of Emerging Media at the College of Charleston, whose interest in how signals travel around the body was motivated by a combination of life events after a family member had a bad accident.

David Parisi: When I was 14 my sister was involved in a car accident that left her paralysed from the waist down, and before that I hadn't really thought much about how information travels around the body, about how contingent regular operations of our bodies are on the successful movement of signals around the body, from one point in the body to the next point, and how easy those movements can be interrupted. Routing those signals around the damaged portion of the spine seemed like it could present a solution to these sorts of injuries, except that one of the things that I learned was that the data of tactile sensations, as it turns out, is just wildly complicated, and it doesn't lend itself easily to being translated by machines.

And then fast-forward a couple of years to the first time that I held a vibrating video game controller somewhere around the late '90s, it struck me that these feedback mechanisms in the video controller were attempting to accomplish exactly the sort of thing, the type of digital encoding and translation that I was thinking of a few years earlier this, albeit in diminished forms. So they couldn't translate data that precisely but they were attempting to transmit some sort of tactile information through the controller.

And as I started looking into this technology, I realised that vibrating video game controllers were just only the tip of the iceberg that was at the time a very mature and maturing field of haptics, a rapidly maturing body of engineering research attempting to solve this problem of how to capture, store and transmit touch sensations.

Lynne Malcolm: And indeed it's no simple matter to collect a wealth of sensory data about objects and then funnel them all to our mind in order to tell us what the object might be. Not only that, our senses also can be misled by the data, and present us with a haptic illusion. Lynette Jones:

Lynette Jones: The information that comes from the skin and comes from sensory receptors in the muscles tends to travel in the same pathways. There's some segregation obviously in the somatosensory cortex and then those inputs transfer to other areas, the superior parietal cortex where it comes together obviously so that when you feel an object it's not that you feel these discrete properties, you sense everything all together. The one sensory system from the skin that is separate from this that is involved also in haptic sensing is temperature sensing. So we are able to extract information about some of the properties of objects by the fact that our skin may cool or warm on contact and that tells us something else about what we are making contact with and its properties, if it's metal versus plastic versus something that's cold from the fridge versus something that has just come out of the oven, all that helps us identify some of the properties of what we are working with.

Diane Dean: But sometimes we are deceived by them, aren't we, for example with weight. So how does that work?

Lynette Jones: So there are a lot of weight illusions…well, not a lot of weight illusions, but in comparison to other haptic illusions we can be deceived by volume, things that are larger with the same weight as a smaller object perceived to be lighter than the smaller object with the same weight. Shape affects our perception of weight, the density of the object, even the surface texture and temperature. So, surprisingly, cold objects feel heavier than warm objects or neutral temperature objects with the same weight. So most often illusions represent, if you like, a disturbance from the normal patterns. We go on these assumptions that make sense to us most of the time, things that are larger are typically heavier, so it's only when there is a really strong discrepancy between our expectation and our experience that these illusions manifest themselves. So they represent the normal processing of information that gives us some certainty about how we interact with the world, but in this case is anomalous.

Diane Dean: We are kind of seeing difference, it's alerting us to the difference in objects.

Lynette Jones: Correct, yes.

Lynne Malcolm: Diane Dean speaking with Lynette Jones.

When David Parisi was just getting interested in haptics he went to a show by the confrontational performance artist Stelarc, who passed electricity through the bodies of the audience as a means of assuming remote control of them.

David Parisi: As I started to look into this issue, I was also just getting initiated into media studies, and media studies is really concerned with the transmission of images and sounds, and it seemed like haptics was taking up that question of how do we transmit touch, and doing so in a way that was very much aware of the potential of this technology to upend the way that we communicate. So if you look at the literature on haptics, all the hapticians in the 1980s and 1990s are talking about this as a way to transform the way that we communicate over distance. So it was a really interesting technical problem but they were trying to solve a technical problem that had significant social, political and cultural ramifications if they were able to solve that problem in the way that they can thought they could.

Lynne Malcolm: And an added complication is that haptic data is relayed in a hierarchy, depending on the type of thing we want to know and which of the senses can tell us the most about it at that particular instance.

Lynette Jones: We do know that in terms of the metrics that we use to understand sensory systems, and some of those are related to how fast things are processed, temporal processing, and then the other part of that is spatial processing, how we understand the extent of objects or events in the world, that touch and haptics is an intermediary sense. So in terms of temporal processing, audition is the best, if you think of speech and everything else that our ears are acutely attuned to process, it is better than both touch and vision, though touch is better than vision in those metrics.

But then with respect to special processing, vision is obviously the superior sense, and then it's haptics and then it's audition. So it is intermediary in terms of its processing, and so we look at comparing how efficiently they can access information and make decisions about it, depending on the sort of particular properties of the events being presented we see these very marked differences in performance.

Diane Dean: So there's a hierarchy dependent on the situation at hand.

Lynette Jones: Yes.

Lynne Malcolm: Diane Dean with Lynette Jones. Historically, haptics have been explored from the perspective of our other senses, through our ears and voice, judging how the hierarchy of accumulated data could be channelled to our sense of touch. It gave rise to a machine called a Teletactor, and a language called Vibratese.

David Parisi: The development of the field of haptics begins in earnest in the late 1800s, right around the 1890s, and what they are trying to do is they are trying to subdivide this sense of touch into its different component sensations. So different systems responsible for sensing temperature, different nerves responsible for registering weight, different nerves that allow for the perception of vibration and so on. So they are trying to subdivide touch, and then they're also trying to figure out just how good those nerves are at discerning the difference between different stimuli. So, how good is the skin at gauging very slight increases in temperature or weight and judging the difference between different vibrations.

But this knowledge just kind of sat there for a few decades. No one really knew how to apply it. And it was really with the Teletactor that they started to use this very technical knowledge about how touch functioned in a practical way. So the Teletactor basically was an attempt to transmit speech sounds through the fingers. And what researcher Robert Gault did working in the 1920s was he took…very much inspired by Bell's work in the telephone, he took speech sounds and divided them into five different bands, one band for each finger. And he designed this device which he called the Teletactor that would allow speech to be routed through the hand, with the idea being that once it passed through the skin and reached the brain, the brain could be trained to reconstruct the sensations of vibration as speech sounds, kind of a neural plasticity of the brain that Gault posited.

Gault was really taken up by this device, he spent about 20 years of his life working on it, and he thought that it could be used as a means of restoring hearing to the deaf. He described it at one point (and I love this phrase) as grafting a mechanical ear upon the skin. So this is one of the very early phases in research on touch; how can we use touch to transmit speech?

A couple of decades later, Paul Bach-y-Rita, using a kind of similar principle, thought that he could use touch to transmit images through the skin. He designed a device that he called Tactile Television, it was basically an array of vibrators that would sit against the back, be projected onto the back, and, connecting that array of vibrators to a camera, he could effectively create black-and-white images on the skin. So the vibrators that were active, the ones that were vibrating were black, the ones that were inactive were white, and he could teach people to see basic shapes using this Tactile Television system, which was initially in its first iteration really cumbersome. So the user had to be sitting in basically a modified dentist chair in order to be able to use the device, to see these really rudimentary shapes and images. So it wasn't very practical.

The other technology that you mentioned, Vibratese, kind of a similar principle there too, effectively working as a type of bodily Morse code. Five vibrators, each positioned at a different point on the torso, each one capable of producing three steps of intensity (low, medium and high), three steps of duration (short medium and long). So basically each vibrator had nine characters, five vibrators, nine characters, and this gave the researchers working on this project 45 characters that they could sign letters and numbers to, so the idea being that you could pass language, you could route language through the skin with this very efficient mechanism that would actually be faster at sending messages to the receiver than Morse code would be itself. So it's a sort of modified form of Morse, active in the 1950s, mostly funded by the US Office of Naval Research as a way to facilitate faster military communication for soldiers in the field. There's sort of two histories here really, history of disability on the one hand, and then history of military technology, military communication on the other hand.

Lynne Malcolm: Associate Professor of Emerging Media, David Parisi. And enticing as the Teletactor sounds, there's still huge potential in haptics research in the future.

Lynette Jones: So there are various ways that the development of this technology impacts other areas. So for neuroscience I would say mainly put more on the rehabilitation, brain machine interface area, trying to understand, for example, someone who has a prosthetic limb, who has no tactile feedback, how do we present feedback to them? Do we put nerve cuffs around the remaining muscles in the limb, do we have an artificial sensor that detects contact and then presents vibration further up the limb, and is that effective, do people come to feel that as being a realistic experience of moving their fingers, their prosthetic fingers across the surface? How does the brain rewire itself if we start presenting information up the forearm for what's going on in an extension of the forearm that isn't real? So I think that those sorts of questions in terms of the spatial organisation of the brain, its plasticity, the best way of presenting information so that people learn to process these signals in a somewhat intuitive way that they don't require a lot of cognitive load or concentration in order to understand what's going on in their peripheral device.

So I think I would see the impact much more in things like that and also as aids for the visually impaired. You know, we try and develop systems that provide them with much more information about the real world that they can't extract visually, and that's why things like touchscreen surfaces and other surfaces that potentially may present information or give them a better feel for something spatially that is very difficult to extract in any sort of medium now, I would see that as much more an area of neuroscience and haptics interaction.

Lynne Malcolm: Dr Lynette Jones.

You're with All in the Mind on RN, I'm Lynne Malcolm. Today Diane Dean takes us by the haptic glove to explore the latest in touch technology.

Aside from virtual worlds such as that shown in the film Ready Player One, there's the practical, remedial application of haptic technology. But with the advent of online shopping, perhaps there's the possibility to use touch in marketing. Is one side likely to win out?

David Parisi: I don't see the remedial side as necessarily taking over, and I think that we will continue to see attempts made in both areas, and I think the success of efforts in both of these areas is really contingent on not just building good technology, in other words technology that works, but building technology that works outside of the lab, that people are going to find has a practical value. And I think that this is one of the things that people who work in the area of haptics sometimes struggle with, is that question of translation. So we have this thing that we are really excited about, we have this thing that maybe it works exceptionally well at doing what we want it to do, but does it necessarily have a use value in someone's day to day life. So you mentioned an application in online shopping, what can touch add to that process? Is it transformative or is it merely just layering like an extra little sensation onto something that you are already engaging? And I would say that in terms of the remedial prosthetics question, there there has been an extraordinary amount of progress in the last few years that I think we are still coming to terms with.

Lynne Malcolm: David Parisi.

And the field attracts a wide range of students. Let's imagine a Haptics 101 course; what do they study? Dr Lynette Jones:

Lynette Jones: Haptics is a really interesting field to work in because it's people from all sorts of disciplines. So you'll get people in neuroscience, cognitive science, mechanical or electrical engineering, computer science, because there are all sorts of different pieces that all fit together, so it's not that it's a typical stream from one academic discipline. Obviously I am in a department of mechanical engineering, and the people that I work with tend to be primarily interested in building technology displays, evaluating them. But whenever you build displays or look at new motor actuated technology, you always need to understand if it's relevant to how people would perform using the display or whether it has any advantage in performing certain sorts of tasks. So you always need to understand all those aspects, it's not sufficient just to build a display and assume it has performance metrics based on the motors and all the control systems that you have in it. You always need to have the human in the loop in terms of your evaluation.

So students who do haptics courses will learn across the disciplines. They will learn about display technologies, actuators, sensors. They need to know some neuroscience and understanding the properties of the senses and the skin and muscles and how that information is relayed to the brain. If they are a little more interested in what we call haptic rendering, which is how you present virtual haptic cues, then you have a much heavier load on computer science and learning to develop the algorithms that you need in order to create realistic experiences in virtual worlds. So it really spans quite a gamut of different areas of expertise.

Diane Dean: What about gamers? Are they interested?

Lynette Jones: They are, and there are certainly some companies out there that have put their technology into game controllers. I think a lot of the early technology was really just pretty crude in terms of what it presented to people. You know, it was rumble vibration motors in a game pad as you went over a rough surface in a road or as you fired a shot to an alien or an opponent. And so I think there has been a drive more recently to increase the level of sophistication of the feedback. One of the challenges there is just cost. In order to get high fidelity you obviously need more expensive systems in there and that increases the cost and then it's a consumer market and you've got to decide what the payoffs and benefits are in terms of the cost of the technology you're adding into the device.

Lynne Malcolm: Just taking the human momentarily out of the loop, at the same time as learning the myriad of ways our sense of touch operates, we're also attempting to teach machines what it is to touch something, and how to tell humans what robotic touch is. David Parisi:

David Parisi: So here I would say what I think is interesting and significant is the way that these two research tracks kind of developed in parallel. So back in the 1990s Mandayam Srinivasan founded the Touch Lab, the Laboratory for Human and Machine Haptics at MIT. He divided the study of haptics up into these three research areas. So there's human haptics on the one hand, how do humans feel, how do humans touch. There's machine haptics on the other hand, so how do machines touch, how do machines feel, how can we build machines that can sense and feel in similar ways to how humans do, what sorts of mechanisms do we need to embed in a machine in order to give it something that approximates the sense of touch?

And then there's this sort of meeting moment, this meeting point of those two fields, human and machine haptics, and it sort of depends on research in both human and machine haptics developing and people doing research in those areas, and then trying to allow the human and machine to meet at this point of contact. That's where the haptic interface happens. You know, is the machine feeling and sensing? Can it feed those sensations that it's feeling and sensing back to a human in such a way that it's intelligible and sensible to the human?

But it raises this very interesting question that I don't necessarily have a great answer for, but it's a question of what counts as a sensation, what does it mean for a robot to feel in the very material sense? So when a robot senses temperature, when a robot senses weight, it does so in a way that is far more precise than a human can, but it also doesn't sense that data as tactile data. In other words, it just registers it as a sort of non-differentiated form of data.

So the question that we might ask there is do the human senses work in a similar way? This is kind of what motivated a lot of that earlier research into sensory substitution that we talked about earlier, so the Teletactor, Tactile Television, assumes that touch functions as a type of universal translator among different sense modalities. Gault even said that all of the senses are basically just modified forms of touch. And I think with robot sensations we might ask a similar question; what does it mean to say that a robot touches? What does it mean to say that a robot sees if the robot is ultimately just taking in data from those sensors and rendering it as data?

Lynne Malcolm: And the challenges don't stop there. Dr Lynette Jones:

Lynette Jones: So one big challenge in both robotics and prosthetics is creating an electronic skin that can sense like our skin can but that is robust. So we can put our hands in water, they can be subjected to quite a considerable range of temperatures, we can scrape them across rough surfaces, smooth surfaces, and we can really abrade the surface and those sensors are never affected unless we do something really quite violent to the skin. And creating that level of robustness and tactile sensors that we put either on prosthetic or robotic hands, it's been a real challenge. We can develop the technology, but to make it so it's not fragile, so it can withstand the sort of insults that we normally affect our hands with is a very difficult task.

Diane Dean: Because the surfaces are not self-renewing as our skin is.

Lynette Jones: Right, and our sensors are beneath…you know, though we have a range in the hairless skin on our hand, our glabrous skin, we have four different types of sensors at different depths under the skin, and they all pick up specific cues about contact and relay that to the central nervous system, and then we have thermal sensors and then we have other sensors in our hairy skin. So we just have this vast number of sensors in there, that even when a few fail, there is estimated to be around 17,000 mechanoreceptors just in the skin of the hand, so to have that array of sensors all transmitting information being processed in real time in an artificial device is quite a formidable engineering challenge.

Diane Dean: Yes, enormous.

Lynne Malcolm: Dr Lynette Jones.

An interesting fact about science generally is that study of one particular facet is sometimes transferred to study in another area and can be taken up by painters and artists, for example the development of linear perspective alongside optics experiments, or early studies on heat which were later applied to the study of sound. Are we likely to see the creation of new sensors or new art forms with the development of haptic technology? David Parisi:

David Parisi: You know, that process of discovery that you're talking about, where new knowledge in one area leads us to the discovery of new aesthetic forms or a greater appreciation of that sense, I think with haptics what is really fascinating to me is that in some ways this has kind of already happened within the field of haptics itself. So if you talk to people who work in the field, and this goes back to…you talk to engineers who are active in the field now, psychologists who are active in the field now, but you can also go look back at the archival material, and one of the things that they constantly say is they are really motivated by this belief that touch is an underappreciated or a neglected sense modality, that we don't really understand fully how it works.

So part of what is motivating them, a lot of researchers in this field describe feeling isolated. They are like the only people studying touch, everyone else is working on vision or sound, and they are off in their little corner working on touch. And part of the thing that I think keeps them coming back to it is this notion that they are in on a secret that other people haven't yet gotten clued into, that they understand exactly how rich and complex this sense of touch is because they've studied it in this really careful way. And for the engineering community they are constantly trying to do new things to machines that allow these machines to transmit and to capture and to store one very particular, one very small slice of touch information.

So the victories in the field of haptics, oftentimes if you are an outside observer they look kind of small, but they are won at the cost of decades of research, just to figure out one small component of this puzzle that is touch. So I think that's the state of where we are at now. And in terms of where it's going to go in the future, and this is where things get, from my perspective, very interesting because we don't really know. We have an imaginary of where it's going to go. So if you watch science fiction films over the past few decades, like most recently Ready Player One, it always ends in a bodysuit, it always ends in this haptic bodysuit that allows people to touch and feel in a computer-generated world, a photorealistic world just like they would touch and feel in real life. And that's a really seductive endpoint for the technology, but it's also quite at odds with the on-the-ground research that hapticians are doing in their day-to-day work. But if you want a nice sampling of where this research is headed, I think one of the things that is a lot of fun for me to do is just to look at the types of devices that are getting patented and see the techno-scientific imaginary that people are thinking about touch through that are good enough for us to want to touch through technology.

When we talk about haptics, we talk about haptics a lot through computer haptics, you know, what it's like to reach in and feel an object that's not really there. But the first step, before we had computer graphics, the military and the space industry were very interested in remote manipulation. So how do you allow someone to touch over a distance where there is a robot operator and the human operator is operating the robot from a distance, how do they feed back those sensations that the machine is encountering to the human operator. When they tried to design these remote manipulation machines, they found that without touch the machines were really imperfect because it's really hard to manipulate something you can't feel.

Lynne Malcolm: David Parisi, Associate Professor of Emerging Media at the College of Charleston, South Carolina, and author of Archaeologies of Touch, published by the University of Minnesota Press.

Lynette Jones: MIT's motto is mens et manus, and I fundamentally believe in that, mind is fine, but you need the hands too, you need to have the physical interaction with objects to understand their properties. And I think as we move much more and more into a computer-based world, we do lose some of that familiarity that people would have had with physical systems decades ago.

All in the Mind is produced by Diane Dean who led us through this fascinating world of haptics. Definitely a field worth watching in the future. Our sound engineer today was Andrei Shabunov. I'm Lynne Malcolm, thanks for joining us, catch you next time.