MIT Technology Review: A Robot with Insane Fork-Handling Skills

The robots are coming—to do the dishes, if we’re lucky.

Speaking at EmTech MIT 2016 today, Stefanie Tellex, an assistant professor at Brown University, showed how her research group taught a robot to perceive and grasp difficult objects, including a fork in a sink of running water, using a clever camera trick and a powerful machine-learning process.

The simple task of grabbing a fork from a sink filled with running water is an important step forward for robot-kind. Robots still struggle to perform many tasks reliably, such as grasping unfamiliar objects, especially when lighting conditions are challenging. “Most robots can’t handle most objects most of the time,” says Tellex. “That’s the hard problem we’re solving.”

The project also points to ways that robots can learn how to take on new jobs in industry and around the home. There is a huge opportunity for robots to help with elder care, for example, if they can be programmed to perform reliably in messy and ever-changing domestic situations.

Tellex’s team used an off-the-shelf industrial robot from Rethink Robotics, which has a camera in its arm, to perform the fork-grasping trick. By moving the camera and combining different images, they were able to build a virtual light-field camera, meaning it captures not just the intensity of light but also the direction of individual rays. This made it possible to build a 3-D model of the scene, and to cope with problems like reflectivity.

In separate experiments, the group is also using a machine-learning approach known as reinforcement learning to train robots to pick up unfamiliar objects. This involves letting a robot that is controlled by a large neural network experiment with different grasps and reinforcing behavior that seems to produce positive results. The results can be impressive, making it possible for a machine to devise a strategy for grasping previously unseen objects that would be extremely difficult to program manually. Tellex is also exploring ways for robots to share what they have learned, something that promises to accelerate the training process dramatically (see “10 Breakthrough Technologies: Robots That Teach Each Other”).

“Our approach is to make the robot learn to adapt itself to the environment it finds itself in,” says Tellex. “Through that learning it can reach a reliability that wasn’t possible previously. It can also use this information to generalize to other situations.”

New robot learning approaches are rapidly moving into industrial settings. Existing robotics companies are developing products that will use reinforcement learning to accelerate robot programming (see “A Japanese Robot Giant Gives Its Arms Some Brains”). Companies specializing in AI and machine learning also see the technology as a way to break into an industry that looks set to evolve rapidly (see “Google Builds a Robotic Hive-Mind Kindergarten”).