Hands-on (and on camera) with Xbox One’s new controller and Kinect

REDMOND, WA—The schedule for today's Xbox event on Microsoft's Redmond campus included time for what was billed as a half-hour "interactive experience." That experience unfortunately did not include a chance to sample any actual gameplay running on Xbox One hardware, but it did include an opportunity to get some time with the new Xbox One handheld controller and revamped Kinect camera.

A group of journalists and I were first taken to one of Microsoft's testing labs to check out the new controller. The main point of this demo was to show off what Microsoft is calling impulse triggers, a term that refers to magnetic motors in both the left and right triggers that add a highly variable rumble directly to your fingertips (which are much more sensitive than the palms of your hands). This is in addition to two more traditional and more powerful rumble motors in both sides of the base of the controller.

These new triggers were shown off in six simple demos that coordinated force feedback with animation on the screen. The demos encompassed an impressive range of rumbling power for the triggers, from the subtle lub-dub of a heartbeat and a gentle laser gun shot to the rat-a-tat of helicopter blades and a powerful car ignition.

It's an odd sensation at first, feeling more like a gentle tickle than the real, powerful force feedback most gamers are used to. The light touch works though, adding a directionality and a level of fine-tuned impact that is lacking from standard controller rumbles. The best way I can describe it is as the touch equivalent of one of those surround sound stereo system demos that adds a sense of motion to sound that once seemed flat and unidirectional.

The most intriguing impulse trigger demo was probably one that combined the more traditional force feedback motors to simulate the rumble of a car engine, then used the impulse triggers to highlight the squealing of brief taps on the brakes. I could see this acting as an important new gameplay cue, outside of sight and sound, that could help add a new level of intuitive response to racing games that's not possible on other systems.

The actual controllers were early prototypes and not final designs, so it was hard to gauge other new controller features like more textured grips on the analog sticks (which we're told are smaller and require 25 percent less force to move than the Xbox 360 controller). The controller did feel more comfortable in my hands, thanks to the lack of a jutting battery pack in the back that can bump against the fingers on the Xbox 360 controller.

The new, plus-shaped d-pad felt perfectly fine on my fingers, although it's hard to gauge for sure how well it works until they let us loose on some fireball motions on an old-school 2D fighter. Microsoft pointed out later that the new controller also has repositioned the seams and screw holes that annoy the fingertips of some Xbox 360 users, but I can't say I really noticed at the time.

A better Kinect

From there it was on to a demonstration of the new Xbox One Kinect sensor. From the moment we entered the demo room, it was clear how much the new sensor improved on the depth sensing resolution of the old Kinect, just from viewing a live, greyscale 3D model of a couch sitting about seven feet from the sensor (a representative said it was about three times the fidelity of the old Kinect). When a volunteer got up in front of the couch, I could easily make out details, from the ripple of his shirt as it fell on his chest to the individual fingers as he rotated his wrists.

The new Kinect doesn't have the motor that the old unit used to make sure it was pointed perfectly at the room in front of it. Instead, the unmoving slab simply has a much wider field of view that captures substantially more of the room when placed on a level surface. A six-foot tall volunteer was completely visible to the sensor while standing only three or so feet away, based on my rough eyeballing of distances.

The demo then switched to a 1080p live color feed of the room, which was unimpressive on its own to anyone with a decent webcam. What was more impressive was the Active Infrared feed, which showed a night-vision style greyscale view of the room at high resolution, even when the room was pitch black (or only lit by a flashlight). This should help for things like computer vision algorithms, which have previously needed extremely bright and consistent lighting conditions to detect certain details in the flat images provided by the Kinect's RGB camera (and yes, the depth-sensing works in the dark as well).

After a quick demo of how the new Kinect's four-microphone array can filter out distracting sounds from a TV speaker, I jumped in front of the camera to test how the new Kinect's improved skeletal tracking worked. The first thing I noticed was a complete absence of the kind of lag and jumpiness that so often characterized avatars on the original Kinect. The second thing I noticed was the way the on-screen skeleton could detect even small motions like a rotating wrist or a shrugging shoulder.

When the avatar in the demo changed from a skeleton to a thicker person made of blocks, it showed how the new Kinect could easily detect which direction I was looking as I tilted my head at the neck. It can also measure the amount of force being put on either side of the body, Wii Balance Board style, and project the force of a punch or a kick based on the speed of the fist behind it.

The creepiest part of the Kinect demo was when the system used a combination of the RGB and IR cameras to detect my pulse rate just by looking at my face. Not only that, but the system could tell when I was smiling and/or looking away from the screen and tell which of two controllers I was holding, even as I handed one off to the demonstrator (a process the PR rep said worked by "magic").

It's still hard to say whether the new Kinect is truly responsive and high-resolution enough to fulfill the promise of controller-free, motion-based games. Still, the improvements to the raw specs, combined with the fact that literally every Xbox One owner will have a Kinect right out of the box, have us tentatively excited about the potential that developers will be able to squeeze from it.

Otherwise, pretty impressive stuff. I'm still not sold on it resulting in games I'd want to play, but the extra capabilities can't hurt.

Suppose we'll get an awesome upgrade to the whole Kinect Sports thing but beyond that I can't think of any substantial hardcore games making use of it. ME3's integration was pointless at best, Steel Battalion was a waste in every sense of the word. On the other hand maybe this Kinect would have helped a little.

It does sound interesting. Being able to accurately take a persons pulse would be a funny feature to play with and even use in a game. Though if this thing has to connect to the internet just to take a pulse I'll laugh. Still interesting if the hardware is up to this though.

Being able to accurately take a persons pulse would be a funny feature to play with and even use in a game.

I was just thinking the same thing. Something like this would be cool to use to dynamically increase difficulty or adjust gameplay based on feedback from the player (pulse and maybe a special controller that can detect how tightly the player is gripping). I'm thinking something like F.E.A.R where Alma adjusts behavior based on how the player responds.

The one comment that a lot of people had when the kinect came out was that support for it was sort of up to the developer, and without knowing how many people had it, it wouldn't be a primary focus. Putting one in every box fixes that, and sets it up so that more developers can play with it. My kinect is mainly used for video, in place of a remote control (and I'd be loathe to give that up).

I like the kinect because it lets me run the OS without a remote. The next generation is going to improve on that, apparently, by a large sum. This is a good thing. The game adaptation will be scattershot, I suspect, from people tacking it on (as they did early on with achievements) to ones that makes use of all the cool features. What I like about it is that it's another cool tool for devs to play with, and now they know that everyone will have one, so it's worth it.

And honestly, even without a "killer app" that proves why it's so worthwhile...I'll still use the kinect to get me to my content while I'm doing something else. The Kinect as a controller to the OS still makes it worth being part of the system.

An improved always on audio/video recorder with dramatically improved capabilities that's directly connected to the internet. Oh, and last I read the Xbox One won't work unless the Kinect is attached. Can't wait for some "technical flaw" to cause 1080p video streams of me scratching my ... to be uploaded to the cloud.

An improved always on audio/video recorder with dramatically improved capabilities that's directly connected to the internet. Oh, and last I read the Xbox One won't work unless the Kinect is attached. Can't wait for some "technical flaw" to cause 1080p video streams of me scratching my ... to be uploaded to the cloud.

Being able to accurately take a persons pulse would be a funny feature to play with and even use in a game.

I was just thinking the same thing. Something like this would be cool to use to dynamically increase difficulty or adjust gameplay based on feedback from the player (pulse and maybe a special controller that can detect how tightly the player is gripping). I'm thinking something like F.E.A.R where Alma adjusts behavior based on how the player responds.

...just thinking aloud...

Or when you're playing D.O.A. Beach Volleyball and you start to get a boner...

Neato. Did you literally jump in front of the Kinect? That may have made a difference.

I like the directional rumble concept! You brought up cars, but my mind went immediately to MGS-styled sneak games. Imagine not being able to see much, either because your avatar's head is down or because of the darkness (not MGS there), but you can feel the footsteps or the presence of the NPC moving toward you and react in their direction.. Forget sneaking - let's see a new Fatal Frame!

Because of the "always-on" nature of the Kinect microphone and that the system won't turn on if the Kinect isn't plugged in, will the system still function if the camera is intentionally covered/blocked? You probably didn't get a chance to find out during the demo, but I'm curious if anyone has come across that yet.

Because of the "always-on" nature of the Kinect microphone and that the system won't turn on if the Kinect isn't plugged in, will the system still function if the camera is intentionally covered/blocked?

I'm wondering that myself. If the answer is "no", the outrage will be magnificent.

Because of the "always-on" nature of the Kinect microphone and that the system won't turn on if the Kinect isn't plugged in, will the system still function if the camera is intentionally covered/blocked?

I'm wondering that myself. If the answer is "no", the outrage will be magnificent.

I'm not trying to make it an anti-Xbox thing out of the question, but I agree that it will be very important for some.

Personally, I'm not scared of some government conspiracy or something, but I am just paranoid enough not to want such a thing always active if it were to be in my bedroom (as opposed to the livingroom) since that is typically where my consoles reside. In that case I'd rather know beforehand what I am getting instead of finding out after-the-fact.

I think biofeedback in the Xbox One is huge. Gabe Newell just mentioned this lat month that the Steam box is going to incorporate the same biofeedback type systems into their controller.

I would assume most developers would love to play around with this new tool in their bag, considering the majority of console and PC users (Steam Box, but hey, I should be able to just buy a biofeedback controller for my PC I want to ) will have access to the new hardware pretty soon.

I think there are a TON of new possibilities in game play with biofeedback, either implemented with the facial heartbeat sensor of the Kinetic or a hand sensor in a Steam box controller . I am primarily a PC Gamer, but imagine the heartbeat sensor in the Kinect, coupled with the ability to determine your mood based on expression, could provide us with a larger variety in game play and more immersion into those games. If a game knows I'm happy/sad/mad/pissed what experience will it give me? Will it increase the difficulty on me, or ease up? Will it play themed music to match my moods or against my moods? Could it force me into a particular mood? Well games already do that to me I suppose. but to take it a bit further, it can then take advantage of the mood it just put me in. Ray Kurzweil are you paying attention?

Now, If I could only get a new generation Kinect for biofeedback, an Oculus Rift for visual immersion, and a set of bluetooth force feedback gloves with a wii mote sensor built in, I could play some seriously some Tetris! Seriously, I am pretty excited for the future of gaming, I mean, what would a game like Hard Rain be like with biofeedback?

Because of the "always-on" nature of the Kinect microphone and that the system won't turn on if the Kinect isn't plugged in, will the system still function if the camera is intentionally covered/blocked?

I'm wondering that myself. If the answer is "no", the outrage will be magnificent.

I'm not trying to make it an anti-Xbox thing out of the question, but I agree that it will be very important for some.

Personally, I'm not scared of some government conspiracy or something, but I am just paranoid enough not to want such a thing always active if it were to be in my bedroom (as opposed to the livingroom) since that is typically where my consoles reside. In that case I'd rather know beforehand what I am getting instead of finding out after-the-fact.

Worry about the feds second, and the advertisers first...

*Microsoft* "Hey, Nielsen, are you enjoying living in 1955? We are this close to rolling out several million boxes, that consumers will pay us for, that allow us to track what parts of the screen the audience is focused on, their body language, and their pulse, in real time! Did we mention our deals with multiple content providers and the integration with our own accounts and payment systems?"

I'm wondering when custom options will be available, as tuning the controller is important to me.This one is the wife's.Of course mine has silicon pads on the sticks because I am a giant pussy, but anyway I am wondering how the force feedback on the triggers would be on highly adjustable speed triggers like these.....might need more silicon.

The controller should feature onboard audio. I should be able to plug a set of Earbuds (Stereo + Mic) into the controller and hear all in game volume, chat, and be able to use the earbud's mic (as well as adjust the volume for each thing independently).

Because of the "always-on" nature of the Kinect microphone and that the system won't turn on if the Kinect isn't plugged in, will the system still function if the camera is intentionally covered/blocked? You probably didn't get a chance to find out during the demo, but I'm curious if anyone has come across that yet.

"No people detected in the room, shutting down system."

Of course it will not be allowed for camera to be blocked, what do you expect. I guess this also ties in with that patent that Microsoft has to disallow the use of the system if too many people are detected watching a movie/gameplay and so on.

*Microsoft* "Hey, Nielsen, are you enjoying living in 1955? We are this close to rolling out several million boxes, that consumers will pay us for, that allow us to track what parts of the screen the audience is focused on, their body language, and their pulse, in real time! Did we mention our deals with multiple content providers and the integration with our own accounts and payment systems?"

It's useful to know that Microsoft created a Privacy and Online Safety Policy for the Kinect that lets us know the kind of data that is being collected. There are four kinds of data that the Kinect tracks:

-Kinect performance data. Used for improving the sensor. Does not identify you. Sent to Microsoft. -Gameplay data. Used to control games. Does not identify you. Sent to Microsoft. -Sign-in data. Optional, used to sign in. Identifies you. Not sent to Microsoft. -Voice and picture data. Optional. Stores commands and game pictures. Does not identify you. Stored locally, or on game company's server, if you allow it.

Identifiable information is kept on the box, and is opt-in. The only data sent to Microsoft is anonymized...which means they can't target you with ads based on your actions. Google scanning your email for keywords, connecting that information to your account and selling ads to target you based on the content of your emails seems infinitely more dastardly than Microsoft anonymizing your data so that they can't target ads at you. And honestly...people don't mind Google doing it because they get the services for free.

If Microsoft offered a program where basic data, such as the number of people in the room, the attentiveness of those people, the expressions of the people and stuff were collected (without identifying you by name to advertisers) and content was suggested to you, and ads presented to you that were based on that, but you got free TV services out of the deal, you'd probably get a lot of people interested. If the NFL content deal let you watch all the NFL games for cheap, and the cost was giving up basic session data so that ads could be sold on it...well, it would save people money and make ads more relevant to their interests.

Right now though...Microsoft doesn't get any information about who you are, so it can't target you or your XBox with ads based on kinect data. If the new kinect has a different policy, then maybe I'll join the group worrying about this.

Because of the "always-on" nature of the Kinect microphone and that the system won't turn on if the Kinect isn't plugged in, will the system still function if the camera is intentionally covered/blocked? You probably didn't get a chance to find out during the demo, but I'm curious if anyone has come across that yet.

"No people detected in the room, shutting down system."

Of course it will not be allowed for camera to be blocked, what do you expect. I guess this also ties in with that patent that Microsoft has to disallow the use of the system if too many people are detected watching a movie/gameplay and so on.

This is going to be "fun". But without me, I'm not buying this ting.

Because we all know that every patent that is ever filed for is put into place, right? The likelihood of Microsoft shutting down services because too many people are in the room is very, very, very remote. They did, however, patent that idea so that if anybody else uses a system like that, Microsoft makes money off of it. Putting that system into widespread use...yeah, not so much.

I mean, I've yet to see shape-shifting screens or decoder rings sold with iMacs, yet Apple has patents on those things. The "omg, they're going to turn off my xbox if another person comes into the room" rumor is just patently ridiculous at this point.

Wow, I won't be getting this, because MS, but sounds impressive nonetheless. There's a tipping point when this kind of tracking gets good enough, and it will be interesting to see if this is good enough to pass it. Imagine being able to use small motions like rotating a wrist or your expression- the bandwidth of the UI could take a big jump....

Wow, I won't be getting this, because MS, but sounds impressive nonetheless. There's a tipping point when this kind of tracking gets good enough, and it will be interesting to see if this is good enough to pass it. Imagine being able to use small motions like rotating a wrist or your expression- the bandwidth of the UI could take a big jump....

Why imagine? It's right there.

There is not here. When you read about a device, you aren't using it, you are imagining using it. Chill out, you guys. Note that I was expressing APPROVAL of this technology.

I'm also interested in this. A while ago there was an article on Cracked (#2) about the best possible games. One of the suggestions was a wizard combat game, where two wizards would be duelling.

The Spells would be cast through different gestures and voice commands, with weak spells being some simple handwaving and maybe a word, and more powerful spells being intricate gestures and incantations.Graphical effects would of course go along with that.

If the Kinect can track all of this reliably and such a game will be made, I will give them ALL OF THE MONIES!

I'm also interested in this. A while ago there was an article on Cracked (#2) about the best possible games. One of the suggestions was a wizard combat game, where two wizards would be duelling.

The Spells would be cast through different gestures and voice commands, with weak spells being some simple handwaving and maybe a word, and more powerful spells being intricate gestures and incantations.Graphical effects would of course go along with that.

If the Kinect can track all of this reliably and such a game will be made, I will give them ALL OF THE MONIES!

It does not track individual fingers. It tracks your thumb and your "hand tip", meaning the four fingers all at once. So it can tell if you're opening and closing your hand (hence the grab gestures in the presentation) but not, out of the box, wether you're making the peace sign or giving it the finger.

However, people were able to do some pretty impressive stuff regarding finger tracking using the original Kinect, so I can only imagine that this will be even better with triple the depth fidelity.

I'm not particularly interested in the console or in the use of a Kinect for motion gaming (any sort of motion peripheral without some sort of haptic feedback is missing the entire point), but once I can hook this new Kinect up to a computer as a machine vision camera, I'll be very exited indeed. Especially the low latency skeletal tracking: outside-in positional head-tracking will be very nice for VR, and adding the rest of skeleton for full avatar embodyment is just icing.

So kinect must always be on so it can hear you say "Xbox on"? The spooks in the government will love this. They now have a window into everyones home. No warrant needed (they say). Not a chance in ~!@#$ in my home!

Looking at this from a game or a movie dev's perspective, I think it's a great opportunity to measure engagement etc - finally they will realise I am eating pizza while being bored by the endless CoD SP cutscenes

*Microsoft* "Hey, Nielsen, are you enjoying living in 1955? We are this close to rolling out several million boxes, that consumers will pay us for, that allow us to track what parts of the screen the audience is focused on, their body language, and their pulse, in real time! Did we mention our deals with multiple content providers and the integration with our own accounts and payment systems?"

It's useful to know that Microsoft created a Privacy and Online Safety Policy for the Kinect that lets us know the kind of data that is being collected. There are four kinds of data that the Kinect tracks:

-Kinect performance data. Used for improving the sensor. Does not identify you. Sent to Microsoft. -Gameplay data. Used to control games. Does not identify you. Sent to Microsoft. -Sign-in data. Optional, used to sign in. Identifies you. Not sent to Microsoft. -Voice and picture data. Optional. Stores commands and game pictures. Does not identify you. Stored locally, or on game company's server, if you allow it.

Identifiable information is kept on the box, and is opt-in. The only data sent to Microsoft is anonymized...which means they can't target you with ads based on your actions. Google scanning your email for keywords, connecting that information to your account and selling ads to target you based on the content of your emails seems infinitely more dastardly than Microsoft anonymizing your data so that they can't target ads at you. And honestly...people don't mind Google doing it because they get the services for free.

If Microsoft offered a program where basic data, such as the number of people in the room, the attentiveness of those people, the expressions of the people and stuff were collected (without identifying you by name to advertisers) and content was suggested to you, and ads presented to you that were based on that, but you got free TV services out of the deal, you'd probably get a lot of people interested. If the NFL content deal let you watch all the NFL games for cheap, and the cost was giving up basic session data so that ads could be sold on it...well, it would save people money and make ads more relevant to their interests.

Right now though...Microsoft doesn't get any information about who you are, so it can't target you or your XBox with ads based on kinect data. If the new kinect has a different policy, then maybe I'll join the group worrying about this.

So kinect must always be on so it can hear you say "Xbox on"? The spooks in the government will love this. They now have a window into everyones home. No warrant needed (they say). Not a chance in ~!@#$ in my home!

Kyle Orland / Kyle is the Senior Gaming Editor at Ars Technica, specializing in video game hardware and software. He has journalism and computer science degrees from University of Maryland. He is based in Pittsburgh, PA.