Blog

How Helpicto’s developers has used Artificial Intelligence to help communicate better

Some children with Autism Spectrum Disorder can experience barriers that can make it difficult to communicate and verbalize their thoughts in order to successfully navigate their world. Equadex worked to provide a tool to alleviate communication difficulty with an easy-to-use mobile app that provides a visual representation of language. With the Microsoft Cognitive Services REST APIs and Microsoft Azure tools, Equadex was easily able to incorporate powerful machine learning and artificial intelligence into its Helpicto app. Equadex hopes that Helpicto will eventually help all people with language difficulties communicate more easily.

“Please get dressed and brush your teeth” may seem like a straightforward request from a parent to a child, but if the child has language difficulties or is nonverbal, this can be an issue. For a person with Autism, if they don’t understand a question or suggestion, they can feel anxious or have a negative reaction.

Karine is the mother of 15-year-old Arthur, who is on the Autism Spectrum and nonverbal. In the past, she often struggled to communicate with Arthur. But now, Karine takes out her tablet and speaks her request to Helpicto, an app on the tablet. Helpicto parses the meaning of her sentence and presents a series of pictograms that show Arthur images of what she wants to communicate. With one look at the screen, he heads off to start taking care of his daily tasks.

Building new tools for communication

Based in Toulouse, France, Equadex has several employees who have experience with Autism Spectrum Disorder in their own lives. This opened up discussions with InPACTS, an association that helps adults and children with disabilities. With the help of Dr. Carine Mantoulan, Director at InPACTS, Equadex created Helpicto to facilitate communication between children on the Autism Spectrum and their parents, teachers, and medical providers, for example.

“We wanted to deliver to the market an innovative technology that could translate natural language into a universal form that someone who is nonverbal could use and understand,” explains Anthony Allebée, Chief Technology Officer at Equadex. “In the past, families and an attendant used a physical binder containing pictogram cards to communicate. Family members lined up the appropriate cards to construct a sentence and showed that to the child. But this required carrying the binder from room to room, and pictogram cards could get lost or misplaced.”

Equadex engineers envisioned a digital app for multiple devices, such as tablets and smartphones. Containing a database of pictograms that could easily travel with families, the app would convert spoken text into series of images. To create this app, the engineers needed to access powerful artificial intelligence (AI) technology without having to build the AI tools from scratch—a process that would take a long time and require help from expert data scientists. Equadex found its solution in the Microsoft Cognitive Services REST APIs and the tools available in the Microsoft Azure cloud platform.

Incorporating powerful AI

Helpicto uses several Cognitive Services REST APIs as part of its standard operation. When someone speaks a command into the app, the Bing Speech API converts it into a sentence in real time, then the Language Understanding Intelligent Service (LUIS) analyzes the command to determine its intent. Helpicto uses this information to retrieve the appropriate images from its database. For example, the question, “Do you want to eat an apple?” produces pictures of the child, the action of eating, and an apple, along with on-screen buttons for “yes” and “no.”

The app uses Azure Functions and Azure SQL Database to manage data processing and storage. Equadex found it straightforward to incorporate both Azure and Cognitive Services. “It was essentially plug and play,” says Allebée. “We develop in Xamarin so that we can deploy the app on multiple platforms, and we found it easy to use the Microsoft tools in that environment. And one great advantage of the Cognitive Services REST APIs, like LUIS, is that they are learning all the time, so the app just keeps getting better and better.”

As a member of the Microsoft Partner Network, Equadex has a longstanding technology partnership with Microsoft. So when Equadex decided to expand Helpicto with additional functionality, it turned to Microsoft for expert support. “One critical feature we hoped to give people was the ability to add their own photographs to the app to help their children be even more comfortable with the results,” says Allebée. “We wanted children to see themselves, their school, and other features of their own environment right there on the screen.”

To develop this enhanced functionality, Equadex organized an intensive three-day programming marathon known as a “hackfest,” gathering its programmers together along with a Microsoft technical evangelist. The team incorporated the Computer Vision API into the app to recognize images, combined with the Translator Text API for multilingual purposes; they also included the ability to add keywords to help properly classify each image. Because the programmers had an easy time using the technology, they realized their ideas and completed the enhanced version of Helpicto by the end of the three days. “With features like LUIS and the Computer Vision API, Cognitive Services helped us quickly turn our dream of an enhanced communication tool into a reality,” says Allebée.

Helping people around the world

For now, Helpicto is a tool that parents and caregivers for children with Autism, for example, can use to communicate nonverbally. Equadex hopes that in the future it can also help improve communication for nonverbal children and adults with other conditions that make communicating verbally difficult, such as Alzheimer’s disease. The company currently offers the app on the French market but intends to release it globally, and the partnership with Microsoft is an important part of that goal.

“We can’t imagine working without our Microsoft colleagues,” says Franck Depaillat, Program Director at Equadex. “We’ve worked with them in the past, we’re working with them now, and we’ll work with them in the future. In addition to providing excellent technology, Microsoft also offers communication and marketing support that will help us as we expand our product and make it available to the rest of the world.”

Equadex also relies on the flexibility and scalability of Azure to help support that expansion. “With Azure, we have an architecture that is as efficient and inexpensive as possible,” says Allebée. “In the future when we’re reaching thousands of users, all we have to do is click a button and increase our available computing resources in the blink of an eye.”

The business goals are important, but it is the human impact that truly drives everyone involved in the project. “With Helpicto, we have one interface that can be applied to any context in a person’s life,” says Doctor Mantoulan. “It encourages the development of personal autonomy and more spontaneous communication. It can help rebuild relationships.”

Adds Karine, “It’s much more fun and far easier to use than the old binders, and Arthur understands better. It might not seem like much, but Helpicto makes a huge difference.”