Where VR, AR and mixed reality is going next

Mixed reality (MR) expert Mark Knowles-Lee may be labelled an idealist, or self-described "very optimistic" dreamer, but he’s got exciting visions for the future of MR, with progress moving much faster than you might imagine.

Mark is the chief executive of Fracture Reality, a studio solely dedicated to creating and designing MR and augmented reality (AR) applications within the Microsoft HoloLens. Mark lives and breathes the partnering of a virtual and real world, selling ideas to companies with access to lots of data that can be visualised using MR, in areas of architecture, transportation, aviation, energy systems and heritage sites.

There has always been confusion over whether mixed reality is any different to augmented reality, or if it’s just a term brought into the spotlight by Microsoft to make it seem like a different technology. It's important to clear this up first, so I posed the question to Mark. He defines MR as technology that is "embedded into the real world", rather than AR, which he describes as "putting a layer of digital content over the world" (essentially saying the MR digital illusion is harder to break).

As an example of Fracture Reality's work, here is a MR visualisation showing proposed changes to London’s city skyline using the HoloLens, and shot from an external camera with live HoloLens content superimposed.

Although Fracture Reality doesn’t strictly design apps for entertainment, the team develops visual concepts specific to the HoloLens, making them experts in MR and capable of making predictions of its future potential.

There are numerous developments happening at the moment with mixed reality and the Microsoft HoloLens specifically, with the most recent announcement being Ford's designing of cars in MR.

And with the recent democratisation of AR available on smartphones in the form of Pokemon Go, Google Tangle and the new Zenfone (which can facilitate both AR and VR) there has been huge development in immersive technology in general. But what’s the next step for MR?

Lightweight hardware

Mark is a huge fan of "all the realities" – MR, AR and virtual reality (VR), and he believes all of these technologies will merge into one lightweight pair of glasses for everyone within the next few years.

"There won’t be a real distinction between real, mixed or augmented reality. A small number of hardware companies will have cracked it – a very lightweight headset – probably not too far off a pair of glasses, that can do mixed reality, probably VR too, and some combination in between," he says.

Most industry experts won’t dispute this idea, but will estimate different timeframes. Mark estimates five years, but he says this doesn't necessarily mean the technology will be accessible and affordable to the public by then – the Microsoft HoloLens currently sits at £2,719 for the development edition.

The current Microsoft HoloLens aesthetic

Mark isn’t concerned with the price of the HoloLens though. His business clients will always be able to afford it if they choose, and he’s not even sure if mass market adoption will ever happen. He says everyone might not want to walk around with a headset – even if it's as subtle as frame glasses – for vanity reasons. He says accessibility is more of a question of human economics.

Holoportation

I asked Mark what he would love to design in MR if there were no constraints whatsoever.

"I think the holy grail in the immediate future, is for us to have a hologram sitting in Paris at the table with us [in the UK], talking in real-time. We click our fingers and there’s something we’re all interested in floating in space, and there’s no friction to that – it’s just a natural experience," he says.

"I think that will be a reality sooner than people think."

And he’s not speaking out on a limb – Microsoft is already making advancements in this area. If you take a look at Microsoft’s progress videos of Holoportation, you can see the beginnings of this concept already happening in the back of a moving car, proving holoportation is already mobile, however still with the use of a clunky VR headset and some fuzzy edges around the human.

Microsoft researchers disclosed in July that the next generation of its Holographic Processing Unit (HPU) will support Deep Neural Network processing with an emphasis on Artificial Intelligence. This means the new HPU chip, one of the signature features of the HoloLens, will be totally programmable and self-contained, but what it will actually recognise is not quite clear yet. Microsoft’s current HPU – and by extension, HoloLens – does a good job of recognising surfaces and edges and projecting virtual objects on top of them. Whether Microsoft can begin to interpret what those real-world objects are remains to be seen.

This would be a huge step up from the MR experiences Fracture Reality provided me with, such as the heritage site demonstrated in the video below, but these were still impressive, if a little simplistic in concept.

Merging MR and VR for design

Currently Fracture Reality is working to blend both MR and VR to improve design processes. Mark demonstrates this concept with the architectural design of a football stadium. For example, two architects wear the Microsoft HoloLens to look down on the stadium design like ‘Gods’, and one person wears a VR headset to experience what it would feel like to be standing on the ground of the football pitch.

It would facilitate a much more detailed design discussion from within the office, while still getting insight into the ground experience of the football pitch in realtime. Mark says this wouldn’t just change one industry, it would change everything, including entertainment-based sectors like gaming.

Here’s an example of how architectural design can take place between two architects viewing the design as 'Gods'.

Although Fracture Reality doesn’t strictly develop apps for the entertainment industry just yet, Mark sees it as a possibility if there was a "decent hardware base".

"Fracture is not in the gaming space now. There’s not a large enough install base in the devices just yet. Entertainment is dependent on access to hardware. You can’t make money making a video game if there isn’t a decent hardware base."

So if low-power hardware is preventing the HoloLens from working in entertainment markets, what other design constraints does Fracture Reality deal with?

What are the constraints of MR right now?

Mark and his team of designers come from a background of creating video games, but they’ve learnt to work around the constraints of MR design. Differing to VR, mixed reality can only be experienced if your eyes remain in level with the headset.

In the early days of the company, Mark and his team imported HoloLens devices from the US (before they were available from Europe), deconstructed what they thought the hardware did, then in VR built an emulator.

"So we built a room, and then we got field view, we put a VR set on and started building MR apps in VR. It let us discover a lot of interaction problems early on."

He says designing apps for the HoloLens means to find "elegant ways around these constraints", making sure the user has a refined experience, especially if they’re experiencing MR – or "the brave new world" – for the first time.

"The last thing you want is to put a device on your head and have some software that is clunky to use.

"The experience should be leading the way, so we obsess a little about the quality of the interaction, and guiding people through this new space very carefully," says Mark. As an example of this visual flow, here's a video of Fracture Reality's airport command and control visualisation created for SITA Lab (in collaboration with Helsinki airport).

Fracture Reality relies on user testing and simple analytics to understand what works and what doesn’t. This helps the team to analyse people’s behaviour to finalise app design.

Although in VR you can render pretty much anything without worrying about brightness, there are still teething problems in terms of using the HoloLens outside, such as glare, contrast and brightness. But Mark says within a few hardware cycles these problems will vanish.

"We found there are certain types of contrast levels we like to use to make complex menus nice to read, that don’t give you eye strain. We want everything to be a gentle, released experience.

"Brightness, stability, size of device – all of that will vanish very quickly because people can reiterate hardware very fast now."

Should I be concerned about the future of MR?

Discussion about mixed reality, AR and VR becoming accessible and subtly integrated into our daily lives is probably Mark’s, and many others, dream – but what about those who are deeply concerned for this projected hybrid future of humanity?

Mark says with every technological change there will be backlash, but that’s not a bad thing.

"It’s right and healthy that people call out the problems, and there are undoubtedly problems with all technology, I think we have to be aware of them," he says.

"Standards and legislation are a good thing. We’re having these discussions right now about data and privacy anyway. But if I’m in the near future, able to 'see' the ubiquitous internet data-sphere, that needs some careful thought and consideration."

Mark puts the onus on governments and companies to figure out a set of ethical standards, but says this will cause difficulty with globalised sales.

Fracture Reality's data visualisation for SITA

How do I get into design for MR?

There are lots of opportunity to design for MR, VR and AR, but a good designer needs to understand people first and foremost, along with understanding spatial awareness. Technical skills would require a background in either UI design, 3D design, coding or developing.

"What people are looking for are also designers who’ve thought about how this technology can be used," says Mark.

"A development team can make content to any platform, but what we’re trying to do is lead the way, so we have to solve problems that people possibly didn’t know existed.

"If designers can come to the table with some thought through ideas would make them stand out from the crowd."