For the second time in as many months, I feel like I've taken a step into the world of science fiction—and for the second time in as many months, it's Microsoft who put me there.

After locking away all my recording instruments and switching to the almost prehistoric pen and paper, I had a tantalizingly brief experience of Microsoft's HoloLens system, a headset that creates a fusion of virtual images and the real world. While production HoloLens systems will be self-contained and cord-free, the developer units we used had a large compute unit worn on a neck strap and an umbilical cord for power. Production hardware will automatically measure the interpupillary distance and calibrate itself accordingly; the dev kits need this to be measured manually and punched in. The dev kits were also heavy, unwieldy, fragile, and didn't really fit on or around my glasses, making them uncomfortable to boot.

But even with this clumsy hardware, the experience was nothing short of magical.

Microsoft calls it holography. I'm not sure if it really is (Wireddescribes HoloLens' "light engine" as having a "grating," so perhaps it really is using interference patterns to reconstruct light fields rather than providing the same simple stereoscopic 3D found in VR systems), but this is a detail that only pedants will care about. (Though if it is true holography, it should solve the focus issue that many people find with existing 3D systems.)

However it works, HoloLens is an engaging and effective augmented reality system. With HoloLens I saw virtual objects—Minecraft castles, Skype windows, even the surface of Mars—presented over, and spatially integrated with, the real world.

It looked for every bit like the holographic projection we saw depicted in Star Wars and Total Recall. Except that's shortchanging Microsoft's work, because these virtual objects were in fact far more convincing than the washed out, translucent message R2D2 projected, and much better than Sharon Stone's virtual tennis coach. The images were bright, saturated, and reasonably opaque, giving the virtual objects a real feeling of solidity.

Minecraft comes alive

We ran through three interactive demos and watched a fourth. My favorite was Minecraft. I was in a room with a couple of tables and a picture on the wall. After putting on the headset and looking around the room so that the HoloLens could figure out where everything was, the world around me suddenly transformed. The table was no longer just a table. It had a big castle on it, with a river flowing beneath. The middle of the table was no longer there; I could peer through the hole to see the river below. As I moved around the room, I could examine the castle from all angles.

My attention was then turned to a second table which had something of a zombie infestation. Fortunately, the foolish zombies were clustered around a block of TNT. Detonating the TNT blasted through the table, revealing a lava pit below. The zombies toppled through the hole and fell to their deaths.

The picture frame on the wall now housed more TNT. Triggering it revealed a large cavern beyond—and out flew a bunch of blocky bats.

Through it all, the 3D effect was thoroughly convincing. The system felt very low latency; as I moved my head and walked around, the objects retained their positioning in the real world, with the castle, for example, never becoming detached from or wobbling around on the table. While Minecraft of course falls some way short of having photorealistic graphics, the melding of real and physical nonetheless felt convincing.

If Microsoft can get the price of HoloLens right, it could become the must-have Minecraft accessory at Christmastime. Microsoft's decision to buy Minecraft's developer all of a sudden makes sense.

Using Skype

A Skype demo was intriguing. I, with the headset, was talking to a person using regular Skype on a Surface Pro 3. The person was helping me wire up a light switch (and it seems that, yes, it was a real light switch with real electricity running through it). My assistant saw the world through my eyes; I saw him on a floating Skype pane that I could pin in place, where his head would remain. Using his Surface Pro 3 pen, the assistant drew diagrams showing me how to wire the switch up, pointing out which tool I should use for each task.

Enlarge/ We weren't allowed to take any pictures of the HoloLens experience, and helpfully, Microsoft's image demonstrating the Skype experience shows someone changing a plumbing fixture rather than wiring up a light switch. The effect is much the same, mind you.

While I'm not so sure about the specific situation of using Skype to perform simple home repairs, the ability to overlay directions and guidance onto the real world feels like something that could be valuable in all manner of technical fields. The ability to see things through someone else's eyes is also intriguing. It almost brings to mind the film Strange Days, and I'm sure we can all remember what the S.Q.U.I.D.s were used for there; I imagine that HoloLens will find similar use cases.

Mission to Mars

Our third and final demo took me to the surface of Mars. I walked around a 3D world constructed from data captured by the Curiosity rover. (NASA intends to use HoloLens to explore data from Curiosity and collaboratively make decisions on how the rover should spend its time.) The experience reinforced just how immersive this kind of augmented reality can be; the Martian imagery obliterated most of the room I was in, except for a computer workstation. I joked that I was surprised to see a computer desk on the Martian surface, because that's what I was seeing.

Further Reading

The Mars demo extended the HoloLens experience in a few ways. In Minecraft and Skype, the interactivity was provided by a mix of voice command and hand gesture—a sort of finger wag serving as a mouse click—with the cursor driven by the direction I was looking. On Mars, I could use a mouse cursor to perform finer manipulations.

On Mars I was also joined by a second person, who appeared before me as a sort of golden apparition. This other person was using HoloLens, too, and so I could see a gaze line emanating from the face, showing me exactly what was being looked at. The apparition talked to me about some of the rocks and how they indicated that we were likely standing in what was once a lake bed.

Finally, the one non-interactive demo showed off HoloStudio, Microsoft's 3D modeling application for HoloLens. We saw a koala in a space suit and a monster truck assembled in front of us, again using a mix of voice and gesture control. Microsoft described this as "print preview for 3D printing;" by projecting the objects in 3D space and integrating them with the real world, artists and designers can get a sense of just how their objects will look before sending them to the 3D printer.

If this comes out as described, it will show what a disaster Ballmer was. Microsoft must have been doing this kind of stuff for a long time, but it must have been kept hidden by the Windows/Office warlords for fear it might put their fiefdoms in peril.

From a gaming perspective, it will be interesting to see how it pans out. When the 3DS launched, AR was meant to be one of its big selling points... but nobody managed to make much of it.

Gaming surely is going to be the thing that makes or breaks the product.

From a business perspective, this could be a great thing for telepresence. In a generation, it may be common to see company directors with AR goggles having long-distance board meetings around a virtual round-table.

It's just a shame some of the images they've shown are a bit uninspired. The idea of annotating in virtual space is interesting, but given the annotation on a tablet is being done in a 2D plane, I can't see how it would be a huge technical advantage over just annotating with two tablets with front-facing cameras. That said, the hands-free aspect is interesting.

This plus photosynth=virtual tourism in a very cool way. MS labs has been doing solid research for over a decade, glad to see something finally done with it. I really do wonder if this is Nadella letting all the pent up creativity at MS have a go, if so it could be a Steve Jobs moment for MS. Now if only HP could get a CEO with a clue.

If this comes out as described, it will show what a disaster Ballmer was. Microsoft must have been doing this kind of stuff for a long time, but it must have been kept hidden by the Windows/Office warlords for fear it might put their fiefdoms in peril.

I didn't personally like Steve Ballmer very much. When he got overly emotional about things, I cringed. I may never fully erase the vision of those sweaty armpits. Still, I try to be fair in what I say about all human beings, regardless of whether I think I would enjoy an hour over beers with them.

So I have to say, this remark is just plain wrong. Much of the work on this product had to have occurred during Mr. Ballmer's time, and therefor is attributable to him. The idea that he somehow encouraged keeping it hidden comes from no evidence at all.

I wonder if any of these were revealed in patent applications but were somehow missed by the peeling eyes of journalists. Regardless, this is super cool. I am secretly glad that I didn't get a Kinect, and I can't wait to get my hands on one of these.

I'd love to play with the development tools, too. I wonder how robust the production model will be. Could I run with one of these? Augmented reality in a fitness setting could be fun and maybe add some cool stuff to the experience.

I wonder if any of these were revealed in patent applications but were somehow missed by the peeling eyes of journalists. Regardless, this is super cool. I am secretly glad that I didn't get a Kinect, and I can't wait to get my hands on one of these.

There's been all sorts of MS patents showing up about headsets and augmented reality type stuff, but I don't think this specific concept ever really came through in a coherent way. E.g. "Fortaleza" or whatever. But often people associated that stuff with Kinect/Xbox accessories anyway. I don't think anyone really saw this coming, which is remarkable given their usual track record on secrecy.

If this comes out as described, it will show what a disaster Ballmer was. Microsoft must have been doing this kind of stuff for a long time, but it must have been kept hidden by the Windows/Office warlords for fear it might put their fiefdoms in peril.

Now here's a person that doesn't realize this tech was in development for years under Ballmer and while he wasn't exactly the most mind-blowing presenter, he did keep Microsoft in profit and did bring a lot of really great changes to Windows, Xbox, Office, servers, the cloud, etc...

Ol' Steve might have seemed like a boring old ape, but he was the one laying/ approving the groundwork for the current version of Microsoft. I'm sure Nadella is going to do great and he is already proving to be a better presenter (what little we've seen of him), but much of what we see now is all legacy Ballmer.

As for the Hololens... It's going to be awesome (probably). And I bet there are a lot of people in the porn industry salivating at this. I can see it now... A blank play doll in the home of lonely neckbeards everywhere, with the Hololens on a hook, just above.

Honestly though, I really want to play Doom with this thing. Or think about how amazing a racing game would be.

If this comes out as described, it will show what a disaster Ballmer was. Microsoft must have been doing this kind of stuff for a long time, but it must have been kept hidden by the Windows/Office warlords for fear it might put their fiefdoms in peril.

I didn't personally like Steve Ballmer very much. When he got overly emotional about things, I cringed. I may never fully erase the vision of those sweaty armpits. Still, I try to be fair in what I say about all human beings, regardless of whether I think I would enjoy an hour over beers with them.

So I have to say, this remark is just plain wrong. Much of the work on this product had to have occurred during Mr. Ballmer's time, and therefor is attributable to him. The idea that he somehow encouraged keeping it hidden comes from no evidence at all.

I left Microsoft about 2 years ago after working there for 18 years. Over that time I had 3 meaningful interactions with Steve Ballmer. You would certainly enjoy having a beer with him. He is a genuine human and a generally good guy. Working with, or for him I should say, can be as trying as with any other boss who is extremely passionate about the work.

I don't have any direct knowledge about this product but if I had to guess I'd say that it likely truly wasn't ready to see the light of day while Steve was there. OTOH the one thing that I found hard with Ballmer is that his view of the consumer market always struck me as merely tech that an enterprise employee might use at home. I don't think he valued serving the greater consumer market much. He may have held off funding something like this for that reason. I'm certain he wouldn't have arbitrarily held the project back though. Microsoft Research has huge leeway in what they choose to do. They need executive approval to productize something like this but not to work on it.

As someone who still sees things like the Oculus as nothing more than a fad that will go the same way as its predecessors, I find this take on VR to be far more promising. It also seems to have a far lower barrier to entry (adjustment etc.) and much more practical application that will hopefully lead to much wider adoption than Oculus or Samsung's VR tech.