Advertisement

Advertisement

New Scientist Live

Automatic sign language translators turn signing into text

No longer lost in translation

Peter DaSilva/Eyevine

By Matt Reynolds

LANGUAGE barriers continue to fall. Machine translation systems that convert sign language into text and back again are helping people who are deaf or have difficulty hearing to speak to those who cannot sign.

KinTrans, a start-up based in Dallas, Texas, is trialling its technology in a bank and government offices in the United Arab Emirates. SignAll, a company based in Budapest, Hungary, will begin its own trials next year.

KinTrans uses a 3D camera to track the movement of a person’s hands and body as they sign words. A sign language user can approach a bank cashier and sign to the KinTrans camera that they’d like assistance, for example. The device then translates the signs into English or Arabic text for the cashier to read.

Advertisement

Translation works both ways. A person who can’t use sign language can type a reply and have it converted into signs recreated by an animated avatar on a screen. It is often more natural for people who use sign language to interact this way than via text, says Sudeep Sarkar at the University of South Florida.

Around 70 million people sign as a first language and there are more than 100 different dialects used around the world. Word order in sentences can differ between these languages as well as from written text. So KinTrans’s machine learning algorithm translates each sign as it is made and then a separate algorithm turns those signs into a sentence that makes grammatical sense.

Founder Mohamed Elwazer says his system can already recognise thousands of signs in both American and Arabic Sign Language with 98 per cent accuracy. Future versions will include support for Portuguese Sign Language and Indo-Pakistani Sign Language, he says.

“It’s great to see innovative technology that could transform the lives of sign language users”

Meanwhile, SignAll’s system uses four cameras that capture data from a signer’s face as well as their hands and body. “In sign language, half of the information is in the face,” says CEO Zsolt Robotka. Raising your eyebrows turns a sentence into a question, for example.

“It’s great to see innovative technology being developed that could really transform the lives of sign language users,” says Jesal Vishnuram at Action on Hearing Loss, the UK’s largest charity for people who are deaf.

This article appeared in print under the headline “Automatic translator works with sign language”