Perception and Reasoning about Liquids

To robustly handle liquids, such as pouring a certain amount of water into a bowl, a robot must first be able to perceive and reason about liquids in a way that allows for closed-loop control. Liquids present many challenges compared to solid objects. For example, liquids can not be interacted with directly by a robot, instead the robot must use a tool or container; often containers containing some amount of liquid are opaque, obstructing the robot’s view of the liquid and forcing it to remember the liquid in the container, rather than re-perceiving it at each timestep; and finally liquids are frequently transparent, making simply distinguishing them from the background a difficult task. Taken together, these challenges make perceiving and manipulating liquids highly non-trivial.

Recent advances in deep learning have enabled a leap in performance not only on visual recognition tasks, but also in areas ranging from playing Atari games to end-to-end policy training in robotics. In this work, we investigate how deep learning techniques can be used for perceiving and reasoning about liquids. We developed a method for generating large amounts of labeled pouring data for training and testing using a realistic liquid simulation and rendering engine, which we used to generate a data set with over 4.5 million labeled images. Using this dataset, we showed that deep learning can be successfully applied to the task of perceiving and reasoning about liquids.