If you want to learn Mandarin, living in Beijing would be ideal. 

The next best thing? A virtual reality classroom that teleports you to the city. IBM Research’s Cognitive and Immersive Systems Lab, known as the Mandarin Project, enables students to wander around Beijing and chat with street vendors, using AI-assisted 360-degree immersive technology projected on classroom walls.

While the technology is still at the experimentation stage, the Mandarin Project has teamed up with Rensselaer Polytechnic Institute that will use it in a for-credit course this summer. Students can walk into a VR Chinese restaurant and complete tasks by ordering food and chatting with the virtual waiter in Mandarin, all while standing in their classroom in upstate New York. 

The Mandarin project’s “spatial context technology” differs from other immersive edutech companies in that they focus on “group learning activities” and “spatial context technology,” Hui Su, director of the Cognitive and Immersive Systems Lab, told Karma. They are also open to additional partners to test new technology, Su shared.

“We focus on the group learning activities when more than one students are involved in the language learning process,” Su explained. “We believe it’s important for students to learn together in a collaborative way even for immersive language learning.”

Through AI-assisted tools, the Mandarin projects aims to address both the specific challenges of learning Chinese for non-native speakers, such as visual feedback on pronunciation, as well as technology that goes further and does not only pick up speech dialogue, but also gestures and conversations involving several speakers. 

Investors are confident about the future of immersive learning technology, and noted that its applications reach far beyond straightforward language acquisition. 

Los Angeles-based Embodied Labs provides immersive healthcare curriculum where caregivers can practice end-of-life conversations with virtual patients or experience the world from an Alzheimer patient’s perspective. San Francisco-based startup Mursion, which raised $8 million in March, provides VR environment for corporate professionals to practice interpersonal skills.

  • As a part of the Mandarin Project,  cameras, microphones and sensors capture human speech and gestures. Cognitive computing technologies are used to analyze and interpret the information, and then present the response in narratives.
  • A Cambridge study finds that students gained “contextualized communicative competence” when learning Japanese in a VR world of Tokyo.
  • Tech giants Google and Microsoft also have immersive learning products. Google’s VR education tool Expeditions Kits enables students to explore historic sites such as Buckingham Palace.
  • Karma Takeaway: IBM’s collaboration with Rensselaer Polytechnic Institute is the latest development in the burgeoning field of immersive learning, as there is plenty of investor interest and potential for commercialization across industries beyond education.