Psychophysics & the bizarre science behind robots that know what we want

If you have a physical disability that makes it difficult to feed yourself, a teleoperated robotic arm might help. Small industrial robotic arms have been adapted to wheelchairs for exactly that purpose. With advances in actuator technology, the precision and capabilities of those arms has improved dramatically.

But those improvements, which are supposed to make life even easier for the user, paradoxically make the robots much harder to control. With multiple degrees of freedom and complex grippers, the training and dexterity required to teleoperate the robots makes them less viable as aids to those with certain kinds of disabilities. But what if a robot could predict what its operator wanted?

More robotics

“Think about last time you did a cooperative task with a partner,” Henny Admoni, head of the Human and Robot Partners (HARP) Lab at Carnegie Mellon University, told me on a recent call. “Take cooking in kitchen.

You’re able to give off nonverbal cues to a partner as you work across from each other.” Someone glances at a spatula, the intuitive and attentive partner passes it over without a word. “The way we do this is by picking up cues,” explains Admoni. “What I’d like to do is make robots as good at doing that as humans are.”

It’s a fascinating challenge, part of the still-emerging research field of Human Robot Interaction (HRI). Drawing from disciplines like cognitive psychology, anthropology, and ethnology, robotics researchers are studying how people interact in the real world and express their intentions.

“The beautiful thing about HRI is it’s so interdisciplinary,” says Admoni. “These are very open research questions.” Cognitive psychology, including a branch called psychophysics, has made headway in understanding how, when, and why humans give off non-verbal cues.

But people are complex and display astonishing variability. There’s no generalizable model for cue giving and receiving, no dictionary of cues that might be used to program a robot to respond appropriately. For Admoni, a big part of answer lies in machine learning.

Using a baseline understanding of cues, robots an learn matching up perceived cues with resulting human behavior. “We have some sense of what cues certain people pick up on because psychology has given us some insight. Robots can automatically detect those cues and then learn from observing people what effect those are correlated to.

Is a grumpy face correlated to a violation of personal space? Is looking around correlated to not having the right tool in front of you? The robots learn over time in a data driven approach.”

The science is still young, but what it predicts is the grandaddy of all UX: machines that feel truly responsive to our needs. Whereas the computer or mobile device you’re reading this on can only respond to a direct input, such as a keystroke, a computer of the future might anticipate that you’re about to share this article with a friend and open a draft email for you. And all that based on your goofy, bright-eyed grin.

The same kind of intention-based controls can be used to make complex robots, such as assistive robotic arms, much easier and more intuitive to control. A human can manipulate a joystick and the robot, in turn, can predict what the human is about to do and help achieve that goal. The arrangement is called shared autonomy, and will play a major role in how social robots react to humans in the near future.

And the work cuts both ways. Just as robots will need to pick up on subtle intentional cues to integrate into the human world, they’ll also need to transmit those cues to humans if they are to pass as realistic agents. The research is still in its early days.

In addition to her work on assistive devices, Admoni is collaborating with other CMU researchers on a project investigating how robots might be used from stovetop to tabletop in a restaurant. The project is a partnership between CMU and Sony Corporation. Part of Admoni’s job is to figure out how robot servers can safely navigate crowds while carrying piping hot food.

Like a good waiter, a robot server will have to read the intentions of the crowd and plan an appropriate path in order to avert a clatter of dropped dishes and scalded patrons. “It’s very cross-cutting in terms of discipline. We’re looking at robotics through the lenses of manipulation, vision, deep learning, AI.