A floating panda, tai chi in China: Learning Mandarin in a VR classroom

In the classroom, a restaurant setting with a floating panda head as a waiter. Credit: Mandarin Project

For nervous new learners of Mandarin, a project that combines AI and virtual reality (VR) technology can help do away with the horrors of speaking to a real human to practise the language. In fact, a floating panda which brings you a plate of jiaozi (dumplings) can help them fix the four tones that most new learners struggle with.

This summer, 12 students at a university in New York were teleported to the busy streets of Beijing, ordering dumplings at a Chinese restaurant, haggling with street vendors and practising tai chi — all through a 360-degree virtual environment.

The Mandarin Project, by the Cognitive and Immersive Systems Laboratory (CISL), a collaboration between the Rensselaer Polytechnic Institute (RPI) in Troy, New York and IBM Research, is an immersive classroom rolled out for the first time in a six-week course at RPI.

Rahul R Divekar, who is pursuing a PhD in Computer Science at RPI and is closely involved in this project, studied Chinese for two semesters there. “As I look back at my classroom experience with my research knowledge now, I realise what I had been missing out on is closely related to what the scientific community calls ‘immersion in language learning’,” he told The Indian Express in an email interview.

“The most idealistic version might mean catching a plane to China and spending a few months there living and breathing the language and culture. But that’s hard for most people,” he said.

With Mandarin being among the fastest-growing languages spoken by the largest number of people, Divekar said it was a no-brainer that the team chose to focus on it. The project, which continues to be a work in progress, is “continuously iterative”, he said. As for the classroom itself, the space students and teachers occupy is surrounded by a panoramic scene 3.8 m in height and 14 m in diameter. The AI agents can both listen and see occupants in order to appropriately respond.

“We try to have markerless technology — users don’t need to wear any headsets or sensors other than a lapel mic, as such devices cause an intrusion to the collaborative aspect of the classroom. We use state-of-the-art AI technology including speech recognition, gesture recognition, headpose orientation estimation, dialogue technologies… spatially aware systems and natural language understanding systems,” Diverkar said.

The massive screens are perforated to allow audio to pass through, which helps reach speakers behind the screens. “Development on the visual aspects of the scene is also quite important… In short, many moving parts and a lot of them are state-of-the-art AI,” he added.

Can this replace classroom learning? Maybe, says Diverkar. “In a few years, we may have a new definition of what a language learning classroom looks like. If you’re asking if it can replace the teacher — probably not so soon. It is geared more towards augmenting what teachers do rather than replacing them.”