Underwater robot understands sign language

Breakthrough can facilitate underwater exploration

Divers use sign language to communicate with one another underwater. Now, researchers have developed a robot that can do the same – understand gestures, as well as receive and implement commands.

Normally, underwater robots are controlled using long cables from boats or the shore. They are used for archaeological investigations or geological research in the often-challenging underwater terrain.

Under an EU-funded CADDY (Cognitive Autonomous Diving budDY) project, researchers at Jacobs University have developed underwater robots that can detect gestures from human divers and then implement the required command.

Supported by the National Research Council (Consiglio Nazionale delle Ricerche), the research team is headed by Professor Andreas Birk, Professor of Electrical Engineering and Computer Science at Jacobs University. His team had equipped the robot (called 'Artu') with a 3D stereo camera with a processing and evaluation unit, enabling the robot to detect the 'arrangement' of the diver's arms, hands and fingers and interpret it as specific commands. Complex instructions such as creating a map of a specific area are also possible.

Of course, a big challenge is the visibility underwater. The algorithms must be robust enough to deal with unprecise and sometimes incorrect data collection. At the same time, the processing speed has to be fast enough to facilitate the interpretion of human gesture in real time.

“The field tests in Biograd were very successful. We are very proud that our system to detect divers' gestures has passed the first tests under real-world conditions. This opens entirely new avenues for underwater robotics,” said Professor Birk.