Walking on Mars With Microsoft HoloLens

The HoloLens Mars demo allowed the press to try a test version of the headset and the holographic extensions to Windows 10.

Last week, it looked to me like I was walking on Mars. I was using Microsoft's new HoloLens (pictured), which it announced as part of the Windows 10 introduction, and the Mars demo was one of a few that the company allowed the press to try using a test version of the headset and the holographic extensions to the operating system.

I've used a number of virtual reality and augmented reality headsets over the past few years and spent a lot of time with Google Glass. In some ways, HoloLens feels like an extension of what some of those products are trying to do; in others, it feels very different.

"Welcome to Windows Holographic," project head Alex Kipman (above) said when the product was unveiled. The concept will lead to more immersive ways to play, new ways to learn, collaborate, and create, he said.

Microsoft went to great lengths not to call the project virtual reality or augmented reality, and instead used the concept of this being holograms constantly. It's a little different from holograms you might see in the movies—in fiction, you don't usually need a special headset—but the images it displays are three-dimensional, so I guess the name makes sense.

Kipman talked about the many possible uses for holograms in the real world, such as an engineer seeing instructions in an overlay on top of an object he or she was trying to build or repair, an architect walking through a design, a surgeon learning a procedure, or just turning the living room into a surreal gaming environment.

Microsoft actually announced two products—one is a series of APIs focused on human and holographic understanding called Windows Holographic that will be a part of Windows 10, and the second is a piece of hardware called HoloLens. Microsoft clearly wants developers and potentially other hardware device makers to create the applications and devices that will make this into a real market, and specifically called to developers working on Facebook's Oculus Rift, Magic Leap, and Google Glass to "create holograms with us." Of course, one issue is there are lots of platforms, and the purpose of the announcement is to try to start the process of getting a critical mass of developers behind Windows Holographic. For VR at least, Oculus seems to have the most buzz right now.

HoloLens is a set of see-through holographic lenses with sound and sensors. It has the normal CPU and GPU components, as well as what the company described as a third, holographic processing unit (HPU) that understands gestures and spatially maps the world around us. The device runs without wires while processing terabytes of information, Kipman said, but the company has yet to give real hardware details.

The company did a number of interesting demos on stage, but I was more interested in trying it out myself, and Microsoft did set up four demo "experiences" for the press in the laboratory where HoloLens was created, located in the basement of the company's visitor center.

One of these was really more of a demonstration by Microsoft of HoloStudio, an environment for creating holograms and optionally 3D printing them (which is usually done via a 3D service, as the holograms typically involve multiple colors and are more elaborate than most personal 3D printers can handle). The on-stage demo involved creating a customized quad-copter, and later a smaller group demo showed us how to make a personalized USB key with the image of a "space koala."

The environment looked like an interesting design studio, one where you can get started fairly easily but that actually has a fair amount of depth. It would probably take me some time to get the hang of using it, but my guess is that people who use products like Autodesk's 3D Studio Max or similar software would find it pretty straightforward.

For the other three demos, we put on prototype versions of the HoloLens headset. Rather than the relatively compact red units that were shown on stage, these really did feel like engineering prototypes—the hardware that ran the headset was in a separate box you wore, and both were tethered to a computer. (The finished hardware looks to be considerably larger than Google Glass but a bit smaller that the Oculus Rift headset, and it works wirelessly. It's not as sleek as you might like, but it looks like a consumer product.) The headset appears to have a number of cameras and sensors on the front and sides. You place it on top of your head (over your glasses, if you wear them), tighten it into place, and then the unit seems to project an image in front of the lenses in the unit, while still letting you see the outside world.

The first of these was of HoloBuilder, a program that lets you create, move, and destroy what appear to be objects in the room around you. It felt a lot like Minecraft in the way you manipulate small objects but with some key differences. You could put virtual items on a table in the physical world, or hang them on a wall, and it really did look like the holographic images were in fact real objects sitting in front of you. In parts of the demo, it looked like you could see things on top of real-world objects; in other cases, it looked like you were cutting through the table and seeing things underneath. It seemed quite cool. I can easily imagine a future version appealing to Minecraft fans.

Another demo showed how you can use it with a version of Skype for HoloLens. In this demo, the task was to install a light switch, while being talked through the process by someone on the other end of the conversation who was using the normal version of Skype on a laptop. Within the HoloLens, I could see the person talking in a window, which could float in front of me as I looked around or be pinned into a specific location. In turn, he could see what I was seeing through the HoloLens. So he saw the tools, the switch, and the opening for the light switch. The visor has microphones and speakers in it, so the conversation seemed pretty standard. The person on the other side of the conversation was able to draw pointers to the tool he wanted me to use and diagrams showing me how to orient the switch and which wires went where. When I was done, they turned the power back on, and the light switch worked.

Now, I'm not sure we need this for such simple operations, but you can easily imagine it being used in a variety of industrial applications, from assembly to repair, where you want hands-free instructions. The idea isn't new: I've seen similar ideas in a number of "smart glasses" designed for industrial users, such as Epson's Moverio Glasses and Vuzix's M-100 glasses. But augmented reality in industrial applications makes a lot of sense, and the ability to have drawings appear directly over what you see seems like a good step forward.

But the most interesting demo was of the Mars surface, using an application developed with NASA's Jet Propulsion Laboratory (JPL) via data and images collected by the Curiosity Rover on the planet's surface. The goal is to use HoloLens to help the scientists control the rover and better understand the Martian landscape. (JPL previously showed exploring the world in a demo using Oculus Rift, though I haven't tried that.)

As a first step after I put on the glasses, I was able to experience walking around the Dingo Gap area of Mars, and what impressed me was that as I walked about the room, it really looked to me as if I was walking on the planet's surface. It was here that the three-dimensional aspect of holograms really came to life.

In the second scenario, you could explore the area known as John Klein where you could look around and even underneath objects to get a sense of the dimensionality of the area. At one point, you could look through the visor at a screen that showed the original black-and-white images, but those looked very static; seeing what appeared to be the actual landscape in color and in 3D was a completely different experience. It was as if the real world had broken through into the artificial one in which I was walking.

The final part of the Mars demo involved exploring a waypoint known as "The Kimberley" near Mt. Sharp, where I was able to collaborate with someone from JPL, who was also using the HoloLens technology. He appeared on screen as a gold-suited avatar, looking a bit like a 1950s science-fiction alien, and was able to talk to me about the landscape and what we were seeing. I was able to place flags on a section of the landscape for the rover to use its MastCam, which can take more detailed pictures, and choose some rocks for the rover's ChemCam, which sends a laser to burn a section of rocks to determine its chemical composition.

This was just a demo, but you can easily imagine how a team of NASA scientists could use the technology to better understand the Mars surface and to control the rover to make the best use of it. And of course, you can imagine some very immersive experiences.

Overall, HoloLens was one of the most impressive demonstrations I've seen. Other augmented-reality headsets have been quite interesting, and I certainly see their potential in industrial applications, and I've seen a couple of very immersive games in Oculus Rift. But the holographic effect—the idea that everything appears in 3D and you can actually move around in the world—embedded in HoloLens did feel like something very new. We will have to wait to try the final visor and real applications to see if this concept is really ready for a mainstream market, but it's certainly impressive.

Michael J. Miller's Forward Thinking Blog: forwardthinking.pcmag.com
Michael J. Miller is chief information officer at Ziff Brothers Investments, a private investment firm. From 1991 to 2005, Miller was editor-in-chief of PC Magazine, responsible for the editorial direction, quality and presentation of the world's largest computer publication.
Until late 2006, Miller was the Chief Content Officer for Ziff Davis Media, responsible for overseeing the editorial positions of Ziff Davis's magazines, websites, and events. As Editorial Director for Ziff Davis Publishing since 1997, Miller took an active role in...
More »