There is more money and talent invested in virtual and augmented reality than ever before. Indeed, more than $3.5 billion has been invested into virtual and augmented reality startups in the past two years. The industry is growing fast; Goldman Sachs suggests the combined hardware and software market for VR and AR will reach, on a base case, $80 billion by 2025, with a potential to reach more than $180 billion.

Many experts believe that in the long run, augmented reality will represent the larger opportunity, as the ability to introduce all manner of information and experience into the natural environment transforms markets, and indeed the nature of our existence.

Companies such as Magic Leap are attracting enormous investment on such promise. But from a technical perspective, augmented reality is considered more difficult, as creating responsive media in the real environment is full of technical challenges.

Recently, seven scientists from universities with leading research programs in virtual and augmented reality published the proceedings of discussions related to the technical challenges in realizing the AR opportunity.

Co-authored by Christian Sandor, Martin Fuchs, Alvaro Cassinelli, Hao Li, Richard Newcombe, Goshiro Yamamoto and Steven Feiner, Breaking the Barriers to True Augmented Reality considers the main approaches to realizing augmented reality, as well as the technical and ethical challenges, in order to draw conclusions on what direction future development of AR might take.

First, the researchers consider the four main approaches to achieving true augmented reality, which can be represented on a scale of “decreasing order of physicality” from “manipulating atoms” to “manipulating perception.” These four approaches include:

Controlled matter: Arguably the most technically challenging approach would be to manipulate or reconfigure atoms in order to change the physical environment. Think Star Trek Holodeck. While this may seem outlandish today, there is research in this direction. The researchers point to “displays that use magnetic fields to rapidly create shapes out of ferromagnetic fluid,” and another class of displays that “levitate solid objects in a field of overlaid ultrasonic or magnetic waves.” The challenges to realizing this approach include safety and energy requirements.

Surround AR: The “next best thing to manipulating atoms is manipulating photons,” in order to make objects in the environment visually indistinguishable from physical reality. The researchers imagine environments replete with light-field displays that create very realistic visual effects. Haptics might be achieved by “stimulating the user’s skin through ultrasound waves.” The challenges to this approach include the immensity of the data processing required and the ability to achieve high resolution. Thus, technology for plenoptic displays remains “in its infancy,” even while light-field sensors have advanced.

Personalized AR: This approach revolves around displaying information only in the subset of the environment that a particular user is experiencing. Examples of this approach include some the most commonly known devices today, such as Google Glass and Microsoft’s HoloLens. Challenges to this approach include tracking at “sufficiently high update rates and low latency.”

Implanted AR: The researchers admit this may be the most “extreme” approach to achieving true augmented reality, but another option is not to manipulate the information sent to a user’s perceptual system, as in the prior three approaches, but rather to manipulate the perceptual system itself.

This approach has a long history of being depicted in science fiction, including movies such as The Matrix and Total Recall. This approach may first become widespread as new technologies augment the experience of those with conditions such as blindness. Later, it may be applied to augment the reality of the healthy.

Related Articles

Beyond the technical challenges to achieving true augmented reality, the researchers pause to consider ethics. Imagining such powerful technology, which may permit us to entirely manipulate the human experience of reality, raises big questions: Who will control its deployment? Who will ultimately be in control of its augmented content and for what purposes will it be used? Will individuals be freed or locked into purely commerce-driven experiences? Will augmented reality enhance our quality of life and promote better communication and deeper understanding, or “isolate and project us into a world of delusion?” Open questions.

But as the desire to sate the human imagination has driven the development of other media throughout history, augmented reality will continue to emerge, as advances in disciplines beyond just “optics, computer graphics, and computer vision” converge to make the above approaches possible.

While our experience of this world has in past included physical, biological, ethical and other practical limitations, the researchers note, augmented reality will free us from such boundaries. Market estimates, even in their billions, seem insignificant next to the scale of such ambition.