Hands-on: Microsoft’s “Mixed Reality” VR Motion Controllers

The Entire VR Industry in One Little Email

The Daily Roundup is our comprehensive coverage of the VR industry wrapped up into one daily email, delivered directly to your inbox.

Email Address

Microsoft debuted the lengthily-named “Windows Mixed Reality motion controllers” back in May, but until now we haven’t had a chance to actually try them out. During a recent meeting with Microsoft in San Francisco, I got to try the VR controllers for the first time paired with the Acer Windows VR headset.

Microsoft’s VR controllers are designed to let you reach into VR and interact naturally with the virtual world. With both a trackpad and a thumbstick, they look like a crossbreed of the Oculus Touch controllers and the Vive controllers.

In addition to the trackpad and thumbstick, there’s also a menu button and a Start button, as well as a grip button along the handle. The big circular parts on the front contain an array of LEDs which provide bright markers for the headset’s on-board cameras to detect and track. Microsoft tells us that the shipping version of the controllers will indeed use visible-light, just like we’ve seen in renderings and promo videos. (Microsoft didn’t allow any pictures of the controllers during my hands-on time).

Buttons and Inputs

Image courtesy Microsoft

Grabbing the controllers for the first time, they didn’t feel quite as elegant as either Touch or the Vive controllers. The odd side-by-side trackpad & thumbstick arrangement is useable, but seems to effectively put neither of the two in an ideal position for your thumb. The grip button is indeed a binary button (rather than being pressure sensitive), and doesn’t feel so much like a “grab” as it does a clicky button press with your palm.

Ergonomics

Though they resemble Touch with their ring-shaped tracking appendages, the Windows motion controllers are actually noticeably larger and clunkier thanks to the placement of the tracking rings, which don’t encompass your hand like Touch, making the controllers easier to bump together, especially when their physical outline is hidden in VR.

The shape of the rings is necessary though, as they need to present a substantial surface area from which the headset’s on-board cameras can track their movement. Though I was using the Acer dev kit headset, our understanding is that these controllers will work with any of the soon to be released Windows VR headsets (all of which feature on-board cameras).

Pros and Cons of Inside-out Controller Tracking

Image courtesy Microsoft

This method of controller tracking differs from both the Rift and the Vive in that it’s the cameras on the headset which are watching the controllers to track their movement (which is called ‘Inside Out’), whereas the Rift and Vive both use external sensors to track their controllers (called ‘Outside In’).

The upside to this approach is that you don’t need to set up any external trackers, but the downside is that the controllers must always be in view of the headset’s front-facing cameras to be properly tracked. Thankfully, the size of the tracking volume felt reasonable; for basic use (like reaching out in front of me to grab virtual objects), I didn’t feel like my reach was artificially limited by the camera’s field of view.

Outside of the Box Tracking

And for times when your hands will go out of the camera’s field of view, Microsoft is doing its best to compensate. When that happens, the system relies purely on the controller’s on-board IMU to estimate positional movement until it reappears in the camera’s view. This works well enough for quick jumps in and out of the camera’s view, but after a second or two, the IMU-only tracking estimation is too unreliable, and it appears that the system will eventually freeze the location of the controllers in the air and only feed them the rotation data from the IMU, though they snap quickly back into their proper place as soon as they’re brought back into view. It remains to be seen how much this limitation (the need to be seen by the front-facing cameras) will impact different VR games and apps, and how effectively it can be designed around.

Tracking Accuracy

As for the tracking accuracy when they are in sight of the camera, I did see some jumpiness here and there—especially if I was rotating my body while moving the controllers—but on the whole they seem entirely usable, and (in my short time with them thus far) to be more accurate than the PlayStation Move controllers.

As part of my testing, I played Arizona Sunshine (2016), and found that guns were steady when I held them out in front of me and aimed down their sites; I didn’t have any trouble landing zombie headshots. Granted, for inside-out controller tracking, holding a gun up in front of me to aim is pretty much the best case scenario—I’m curious to see how other common input modalities hold up (like shooting a bow and arrow or swinging a sword).

The occasional jumps didn’t present much issue in a shooting scenarios, but for more precise uses, like VR drawing, painting, and animating, it remains to be seen if those jumps will cause any usability issues.

– – — – –

Microsoft says that the Mixed Reality motion controllers will be bundled with Windows VR headsets starting at $400 this holiday.

I had a feeling that it was going to run into tracking issues when the controllers were not visible to the cameras… I was hoping to get rid of the need for my rift sensors by moving to the Windows VR platform so I can quickly move from room to room, but looks like the game experience may suffer. I will wait and see how things progress over the next year before jumping into a different VR system…

Foreign Devil

You’ll still be tethered to your PC if you go for the higher powered experience similar to Rift.

Andrew Jakobs

Or you could strap a higend laptop to your back..

Kev

Room to room could mean you just move it between different pc’s or to your laptop etc. when you need access to it there. I.e. I want to watch a movie on my laptop in my couch you could theoretically do that with this. Moving lighthouses around and tracing the room and all of that is probably a much larger hassle. My lighthouses are on semi-permanent spots on my walls.

Graham J ⭐️

This is what I’m thinking. I’ve got a Vive room but would love to be able to throw it in a bag and use it at lunch time at work with my laptop. This would be feasible with WMR devices.

S B

” whereas the Rift and Vive both use external sensors to track their controllers (called ‘Outside In’).”
Wrong! Vive’s basestations is not sensors. Sensors in HMD and controllers.

benz145

You’re right, however, since the distinction matters little (to the scope of this article), we tend not to bog down the article with a re-description of the inner workings of SteamVR Tracking. If the difference mattered to the article, you can be sure that we would mention it : ).

An article like this, for instance, approaches it with much greater detail:

Then don’t use the word outside in as its clearly wrong. Just say it uses external trackers or something else

benz145

“Inside-out” and “outside-in”, as they’re used in the VR/AR industry, are used correctly in this article. “External trackers” also suggests the same as the word “Sensors.”

Sam

But the Vive does not use external trackers or senors, but markers.
The difference between the windows hmds and the Vive is, that the windows hmds use “markerless” inside out tracking and the Vive inside out tracking with markers (the stuff the lighthouse cubes emit).

Andrew Jakobs

Yeah, but the controllers are tracker outside-in, not inside-out. it would be inside-out if the controllers had their own camera’s (which ofcourse could be possible).

Graham J ⭐️

Anyone who calls Vive tracking “outside-in” is making a mistake.

Graham J ⭐️

He’s right, instead of “sensors” something like “markers” would have been more accurate. The distinction not being relevant to the article is not a valid justification for inaccuracy.

Andrew Jakobs

But then again, it’s still wrong, as the controllers are still tracked outside in, with the headset being the outside tracker.. the controllers would be called inside out if the both had their own camera’s..

Sam Illingworth

Yeah, but the sensors are in the controllers, thus external to the headset :P

Adrian Meredith

Its very interesting to see that they use visible light. Why? because that means it can’t interfere with the technology used by the kinect. Image this combined with a (next gen?) kinect and you could have a killer combo,

Andrew Jakobs

who says it’s visible light? it might also be some IR leds which the camera’s can pick up, but then again, it could also be just regular leds..

Dave

I wonder if they could allow for an extra sensor to be bought like Oculus. If you want to portability of the Inside Out Tracking but once in your own home having extra sensors to allow for more robust tracking.

benz145

It’s possible, given that there are active markers on the controllers, however the way the rings are shaped is specifically designed to face the user’s head, so an outside perspective may not see them as well.

PrymeFactor

Is there really any game or app where your hands would be that far back for multiple seconds?

benz145

Yes, aiming a bow and arrow (The Lab), throwing a frisbee perhaps (Rec Room), and some apps with certain types of locomotion (Lone Echo or Windlands).

I’m interested to see how well the system handles throwing things (like frisbee, grenades, baseball), because most systems rely on historical information about where the hand is and how it has moved in order to calculate the trajectory; if there’s only a frame or two of camera-tracked data as the throwing hand enters back into the camera’s bounds, it could cause problems for those sorts of actions.

It’s possible to design around these limitations, but it remains to be seen how much that might (or might not) detract from the experience.

K E

If they had placed the tracking cameras on the bottom corners of the headset instead of facing straight ahead, they could have increased the fov in the direction where the hands usually are without any extra cost. If you ever meet any of the designers behind these headsets, feel free to ask them why they didn’t do that.

Graham J ⭐️

I don’t think the cameras do face straight ahead, they look like they’re pointed outward and down, probably for that exact reason.

Armando Tavares

The answer to how well the system handles throwing things like frisbees, grenades, baseballs is simple: If it can’t, it wont.

The experience wont suffer because stuff that the system can’t handle wont be presented to the users. It’s simple enough.

I want to play soccer using OCULUS. Why can’t I?? Because it’s something the system wont handle properly thus, OCULUS wont set themselves up for failure providing a soccer game.

If these devices can’t handle throwing stuff you wont find games you’ll have to.

Graham J ⭐️

I think bow and arrow should be fine as the closer hand is right next to the face which I think the cameras will see (I read that the cameras have roughly the same FOV as human vision)

Throwing things should be ok because accelerometers work best when accelerating and gyros will provide accurate orientation.

I think the worst errors will occur when the controller is both out of the camera FOV and almost stationary.

Andrew Jakobs

How about shooting offscreen, with that I mean looking left while shooting right.

Armando Tavares

How about not shooting at all? :)

Content will be made and released with the success of the platform in mind.

Will they release stuff the platform can’t achieve? Or struggle with? No.

This doesn’t make or break anything. This has been done with every gaming/content platform throughout the years. Even now…. why can’t you move around freely in VR games?

Andrew Jakobs

Problem is, now you have 2 platforms that can have ‘free controller movement’ and one that doesn’t. And personally when I’m using a ‘roomscale’ then it should support being able to have my controller pointed to something behind me while I look forward, and it not being limited due to the controller not being detected. (but then again, these MS controllers will be able to be tracked using optional cams, if MS supports it).

Graham J ⭐️

I read that the camera setup has roughly the same FOV as human vision so try turning your head all the way one direction, looking the other direction, and see where you can begin to see your opposite outstretched hand. It’s pretty close to straight to the side so I think this should work pretty well.

Andrew Jakobs

should work pretty well, but it won’t have me shooting behind me or shooting left when I watch right.. I’m just saying, it doesn’t have full tracking which makes some things just impossible which one would expect when using ‘roomscale’.. So until the controllers themselves also have inside out tracking, it’s useful for only ‘in front’ of you..

Graham J ⭐️

I wouldn’t say that it’s impossible, it’s just that the accuracy will vary. Hopefully it will be good enough in most circumstances.

Sam Illingworth

What about games like Space Pirate Simulator where you switch weapons by reaching over your shoulder?

Duane Locsin

The inside tracking with the narrow scope sound more like hindrances, and big limits at best.

If it were implemented for mobile VR instead, we could understand but it seems something MS are half ass*ng and rushing.

Jean-Sebastien Perron

Microsoft and it’s friends (Acer, Dell, HP), the underachiever lame ducks, why can’t they make a good product like other companies : Apple, Samsung, Oculus, HTC. They are too late to the game and bring nothing new, even worst they bring old abandoned technology that are way behind what the others were 2 years ago. And the price is not even remotely competitive at all. So this is google cardboard for PC. Half of the time I play in VR, my hands are not in front of me. Can you crawl on the floor with that thing? No.

Insulting people is not an argument. Next time try to provide some logical argument and facts.

Armando Tavares

Awaiting your logical arguments and facts when you insulted Acer, HP and DELL as the «underachiever lame ducks».

Your post reeks of fanboyish butthurting.

Facts (you seem to love them):
ACER, HP and DELL make a FRIGIN TON of good products (as do Apple, Samsung, Oculus and HTC).

. You don’t get to decide if they are late too late to the game or not.
. You don’t get to decide if the technology they bring in will succeed in the current market or not
. You don’t get to decide if the price is competitive or not and even if it ISN’T, you don’t get to decide if people will still buy it or not (Oculus/Vive are overpriced in my opinion and you seem to have one)
. You’re irrelevant in the big picture and you don’t get to decide pretty much anything…

… the MARKET does.

So, take a chillpill, relax, sit back and watch carefully as the future unfolds.

Konchu

I do think overall this is a good thing with dell Acer Dell and HP joining in. I’m not sold that this is a way worse experience yet. I think inside out has potential for mass penetration as it will be easier to use. Vive and Oculus need dedicated spaces for good immersion. This is actually fairly portable though cabled at the moment you could theoretically have this calibrated in a new space at a fraction of the time no mounting sensors/Cameras and unwieldy usb/powercables, head set takes 2 ports HDMI and USB. Users may not be able to dedicate a large room all the time with this I could move to the garage or back patio with a laptop and be setup in a minutes.

Quality wise I think there will be a wide spectrum here and that is a good thing as it will drive cost down and let more people play. I’m excited for more VR market and for some standards for competition I don’t think OSVR is going to cut it from my personal experience maybe this can though.

Sam Illingworth

Are those sticks as awful as they look? Concave or convex is an argument we’ve been having for many years, but I’ve never heard anyone stick up for flat!

Konchu

I hope they offer these as a standalone purchase soon since I got one of the dev headsets. I do have concerns a bit for inside out on the controllers but to be fair it will probably be pretty alright. I do like blind fire in VR but this could be addressed a bit with IMU and smart use of inverse kinematics and some aim assist like functions. AKA the arm has limited range so if it knows the length of the arm and where it was last seen and current hand orientation you could make some logical generalizations of location possibly enough to suspend disbelief.

RFC_VR

Arm kinematics and Imu/good algorithm can track pretty well once camera loses sight.

Luke

Will be possible to see your body rendered in 3D in VR thanks to the front facing cameras? it should work like kinect right? could it rebuilt the shape in 3D?
would be nice to see my foots and interact with them, I ask just this, not necessary all the body.

Fredrik Bengtsson

Maybe they could add an extra camera or two to the back of the headset to fix occlusion issues?

Roger K

two more lenses towards the sides angled slightly up could allow the the two lenses in front to be be positioned angled lower or just have all four vary the angle for full side upper and lower coverage sans double jointed posterior coverage! :)
Actually I imagine posterior controller “blind spots” might still be an issue if multiplayer controller positioning relies on each player’s contribution? And or puck/baLL controller tracking if such a paradigm is introduced?
Eitherway… at this point I doubt as much finds it’s way into 1rst gen MR production? ( unless these are actually developer kits still?

Camm

Smaller setup footprint is tantalising, but think I might wait to see some more development on the controllers (and maybe even wireless headsets over wigig) before I change headsets.

Between the cheapest price and the easiest setup, I do think however that Microsoft’s headsets are the most compelling. And because its just an SDK, I’d expect pretty rapid hardware improvement compared to Occulus or Rift

Paul Schuyler

I agree, portability/ease of setup and a modest price advantage are a big deal when considered together. But people also miss the 3rd and 4th major advantages this platform has. They call it Mixed Reality because they see VR or AR as subsets of a singular generation of devices that do similar things at different price points. There’s been a lot of hollering about these being VR-only. But consider the 3rd advantage, unlike the Vive or Rift, this tech has AR functionality built in. Because of the inside out tracking, these VR headsets can identify and track fixed objects and lighting in real-world foreign environments. And those are exactly the same developer features which Google and Apple’s new AR kits unleash. The 4th feature advantage is that Windows natively supports the hardware in its OS, and has been re-conceived around them. Recent Windows 10 updates have re-architected the Windows vision to be a spatial metaphor and/or to utilize MR apps.

Rerry1000

is possible o play non vr games in non vr headset?

ej2020

Any ideas on how to purchase the controllers? I purchased the HP Mixed Reality Headset Developer’s Edition which did not ship with controllers at the time.