Microsoft is Imagining a NUI future

You don’t have to look very far to realize that technology is becoming more natural and intuitive. In a typical day, many people use touch or speech to interact with technology—on their phones, at the ATM, at the grocery store and in their cars. The learning curve for working with computers is becoming less and less of a barrier thanks to more natural ways to interact.

As Craig Mundie has stated on many occasions, technology is beginning to behave like we do. For Microsoft, natural user interface (NUI) technology is an area we’ve invested in for many years. The fruits of those investments are now being seen across many of our products, including Windows Phone 7, Microsoft Surface 2.0, Bing for Mobile and Office 2010 Mini Translator. One product that has gotten a lot of attention recently is our Kinect for Xbox 360, which incorporates facial recognition along with gesture-based and voice control. The device knows who you are, understands your voice or the wave of your hand and is changing the face of gaming as we know it. We’ve also witnessed how Kinect has inspired others to explore the potential of NUI, and we’re excited about the potential that others see in this technology.

Even with all the excitement, the reality is that our industry – Microsoft included – has only scratched the surface of what is possible. Most people’s view of natural technology is sensory related – touch or speech – but Microsoft’s exploration into NUI is multi-dimensional. By combining sensory inputs with the knowledge of what you’re trying to do (contextual awareness), where you are and what is around you (environmental awareness), 3D simulation and anticipatory learning, we can foresee a future where technology becomes almost invisible. Imagine a world where interacting with technology becomes as easy as having a conversation with a friend.

Natural and intuitive technology like Kinect can be more than just a great platform for gaming and entertainment – it opens up enormous opportunities to address societal problems such as healthcare, education, and providing greater access for persons with disabilities.

A recent poll we conducted of about 6,000 people across six countries showed how nascent NUI is: Only about half of the respondents were familiar with the various emerging dimensions of NUI, such as 3D simulation technology. Yet nearly 90 percent of all audiences view natural and intuitive technologies as more than a fad. They believe these technologies are the way of the future.

Even more telling, 70 percent of those polled believe natural and intuitive technology can help improve health care, education, workplace productivity and other societal issues. In the medical arena, more than 87 percent of people polled say they believe natural and intuitive technologies could be used in more ways to improve health care. Similarly, 91 percent of professional educators in both developed and emerging economies said that natural technologies would help improve educational outcomes in their countries. The infographic below illustrates peoples’ awareness, perceptions and usage of natural user interface applications.

These benefits are not some far-off pipedream. Even in this early phase, we see amazing ways in which “natural” can be applied to solve real-world problems.

• Researchers from the Spaulding Rehabilitation Hospital and the Harvard School of Engineering and Applied Sciences have adapted our first-generation Microsoft Surface table technology to help children with cerebral palsy in getting some valuable, fun therapy. The table was programmed with custom-developed games and activities that encourage the principal elements of cerebral palsy therapy. Since cerebral palsy comprises a variety of neurologic disorders with different degrees of symptoms and mobility difficulties, the adaptation of touchscreen technology is key to the end result, since games can be designed around specific goals, such as stretching, movement, range-of-motion, use of both or arms together and increased visual skills.

• Advances in natural user interface technology also have the potential to bring medical care to some of the most remote and underserved areas of the globe. Early this month, I wrote about a talk that Craig Mundie gave at The Cleveland Clinic. In his speech, he showed a prototype of a Robotics Triage Nurse. This was inspired by the reality that many parts of the developing world just don’t have access to the millions of additional doctors needed to provide adequate medical care. Working with partners, Microsoft Research built a system that combines machine vision technology, machine learning, Bayesian inference and avatar generation to create an experimental, computer-based nurse with the ability to triage basic medical issues. Although the Robotics Triage Nurse is a prototype, such a “nurse” could one day give entire village communities easy access to world-class medical resources.

• Kinect for Xbox 360 is inspiring invention in nearly every part of the globe. More than 8 million consoles have been sold in just the first 60 days, and the enthusiasm we are seeing in the scientific community – specifically the research and academic communities – around potential applications of Kinect, is exciting to see. Students at the University of Washington’s Biorobotics Lab, for example, used Kinect with a commercially available PHANTOM Omni Haptic Device to explore how robotic surgery could be enhanced by incorporating the sense of feel. The idea is that the system would provide a surgeon with information on “no-cut zones,” and those barriers could be “felt” by the surgeon performing robotic surgery. The UW refers to the project as a “first proof of principle,” but the possibilities are inspiring given the many scenarios where this could come into play. In the same Cleveland Clinic talk, Craig Mundie shared his thoughts on this project.

While these ideas and projects are at various stages, it’s inspiring to see and dream up what could be next. Imagine a world where doctors on opposite sides of the planet communicate using technology that translates their conversation in real-time and are able to quickly diagnose a patient. Imagine a world where students in rural France can go on a “field trip” to the Great Wall of China using virtual 3D immersion technology, while their counterparts in rural China take a “field trip” to the Louvre in Paris. When presented with these potential scenarios, three-quarters of consumers surveyed said they are excited about these examples of how natural and intuitive technologies might help improve society.

As any technology moves from plausible to possible, a number of important questions arise such as: “Is there a need, is the market ready for it, will it be embraced?” Our research shows that the vast majority of people polled in both developed and emerging markets see great potential for NUI applications beyond entertainment. This is especially true in China and India, where 9 out of 10 respondents indicate they are likely to use NUI technology across a range of lifestyle areas – from work, education and healthcare, to social connections, entertainment and the environment. We believe that taking technology to the next billion can be aided by NUI – making technology more accessible and more intuitive to a wider audience.

It’s an exciting time for Microsoft, our customers and partners as we explore the possibilities NUI has to offer and how we can make them a reality – Kinect for Xbox 360 is just a first step. For our part, we’ll look to apply our learning and technology to fields such as healthcare, education and beyond. We’ll continue to push the boundaries, lower the barriers of use for technology, and I think we’ll surprise people with the true potential of NUI.

This week we’ll be inviting business and government leaders from across the globe to share our enthusiasm by experiencing the potential of NUI at the World Economic Forum in Davos, Switzerland. For more information, read this post on the Microsoft on the Issues Blog.