Microsoft’s AI-powered app can help you learn Chinese

Subscribe

If you really want to learn to speak a foreign language, everyone knows that practice is essential. The challenge is to find someone with the time, patience and skill to help you practice pronunciation, vocabulary and grammar.

For people learning Chinese, Microsoft is aiming to fill that void with a new smartphone app that can act as an always available, artificially intelligent language learning assistant.

The free Microsoft Learn Chinese app is now available on Apple’s iOS platform.

The app aims to solve a problem that is familiar to any language learner who has spent countless hours in crowded classrooms listening to teachers, watching language-learning videos at home or flipping through stacks of flashcards to master vocabulary and grammar — only to feel woefully underprepared for real-world conversations with native speakers.

“You think you know Chinese, but if you meet a Chinese person and you want to speak Chinese, there is no way you can do it if you have not practiced,” explained Yan Xia, senior develop lead at Microsoft Research Asia in Beijing. “Our application addresses this issue by leveraging our speech and natural language processing technology.”

(left to right) Yan Xia, Jonathan Tien, Frank Soong

Teaching assistant

The application is akin to a teacher’s assistant, noted Frank Soong, principal researcher and research manager of the Beijing lab’s speech group, which developed the machine-learning models that power Learn Chinese as well as Xiaoying, a chatbot for learning English that the lab deployed in 2016 on the WeChat platform in China.

“Our application isn’t a replacement for good human teachers,” said Soong. “But it can assist by being available any time an individual has the desire or the time to practice.”

Human teachers, he noted, are limited in the time they can devote to coaching students on their speaking skills even in the best of circumstances such as private, one-on-one sessions. The application’s speech technology is robust enough to help students make progress toward their learning goals in the absence of a human partner.

The language learning technology relies a suite of AI tools such as deep neural networks that have been tuned by Soong’s group to recognize what the language learners are trying to say and evaluate the speakers’ pronunciation. They are based on a comparison with models trained on data from native speakers as well as the lab’s state-of-the art text-to-speech synthesis technology.

When individuals use the app, they get feedback in the form of scores, along with highlighted words that need improvement and links to sample audio to hear the proper pronunciation.

The app includes lessons for both beginners and intermediates. The lessons for beginners mainly help users to learn short words and sentences, and then provide lessons with read-after-me and dialogue to practice Chinese that users have just learned. The lessons for intermediates are the interactive lessons that help users learn how to navigate real-world scenarios such as renting an apartment and ordering food in a restaurant.

“The app will work with you as a language learning partner,” said Xia. “It will chat with you and give you feedback based on what you are saying.”

Language independent, culturally dependent

The Learn Chinese app was developed as part of an ongoing innovation incubation project at Microsoft’s Asia research lab in Beijing. The incubation project is one instance of a class of decades-long research within the lab on speech and language technologies, including recent breakthroughs in speech recognition, natural language understanding and text-to-speech synthesis.

The machine-learning models and neural networks that power Learn Chinese and Xiaoying are language independent – they work equally as well when helping people learning to speak Chinese or English. “If we are successful on one side, one direction, then we should be able to do that on the other side, the other direction,” said Soong.

The differences between the Chinese and English language-learning applications are cultural, noted Jonathan Tien, director of engineering in the Engineering Innovation Group at Microsoft Research Asia.

Tien’s team performs what’s known as deployment driven research – learning how the applications are used in the real world and tweaking models, interfaces and software architecture based on data collected from users. The process, he noted, is eye opening and rewarding.

“Chinese is our native language,” he said, “but learning Chinese is not native. It is difficult. During this process, we are learning a lot ourselves.”

The researchers are collaborating with human foreign language teachers to understand best practices and figure out how to deploy the language learning technology in a way that augments the teachers’ approach by encouraging frequent use and real-world language improvement.

“In today’s technology environment, you can deploy your application and improve it based on users’ feedback all the time,” said Tien. “That is the beauty of deployment driven research.”

Related Items

Natural Language Processing (NLP) is a technology to help computer process or understand human languages. It includes: Syntactic-semantic Analysis: Conduct word segmentation, the tagging of parts of speech, named entity recognition and linking, syntactic analysis, semantic role recognition, and polysemy disambiguation for a given sentence. Information extraction: Extract important information from a given text, such as the time, the place, the people, the event, the cause, the result, numbers, dates, currency, proper nouns, etc. Put…

The amount of today’s rich media content is unprecedented. Every second, people are creating and sharing enormous amounts of information, and particularly content that includes complex image and text information. Among these contents, the combination of text and images has become mainstream. In the process of content creation, a great challenge that people face is how to design an eye-catching layout when dealing with a variety of textual and graphic information (for example, magazine covers,…

Microsoft Research Asia (MSRA) Collaborative Research Program is a vital platform for faculty members in Asia working together with MSRA researchers on mutually interested topics. We aspire to foster talents and establish Microsoft as a valuable research and technology partner for higher education in Asia. Since the announcement of Call for Proposals in mid-July, there has been a large number of submissions. MSRA Collaborative Research Program Committee carefully reviewed them based on the state-of-the-art and…

First established in the 1950s, China’s Friendship Awards honor the achievements of foreign experts in the country’s economic and social progress. Beijing’s municipal government established the Great Wall Friendship Award in 1999, and it remains the highest award that foreign experts working in Beijing can receive. On September 26th, the Beijing Municipal Government presented a 2014-2016 Great Wall Friendship Award to Dr. Hsiao-Wuen Hon, Corporate Vice President of Microsoft, Chairman of Microsoft Asia-Pacific R&D Group,…