It's now more than a year since Microsoft's Kinect motion controller hit the stores, bringing a new era of motion controls to the games console. While the first 12 months has been dominated by family titles and the usual multi-event sports sims, we're now seeing developers experiment a little more broadly with the technology.

Forza Motorsport 4 introduced headtracking, while the forthcoming Xbox version of Minecraft is set to allow gamers to physically construct buildings. Meanwhile, the release of the device APIs into the wild has led to an explosion of homebrew "Kinect hacks" using the tech in many fascinating ways.

At the forefront of work on the Kinect technology you'll find Rare, the veteran Twycross-based developer, responsible for creating several of the earliest Kinect titles – including Kinect Sports.

Nick Burton has been at the company for 13 years, originally working on titles such as Starfox and Kameo, before becoming incubation director and working with emerging tech. He's been involved with the Kinect project since its inception, and recently we met up with Burton at the excellent Launch Conference in Birmingham.

"I'm passionate about the Midlands as a centre for gaming," he said. "You always hear about the North East and the South, and I'm like, 'Hang on a minute – the biggest hub in the UK is here, guys!'"

Rare has its own studio in the city's Fazeley Studios ("This is where Embarrassing Bodies is made," says Burton proudly as we tour the building), and clearly fits in neatly nearby Aston University and the rapidly evolving Science Park, where numerous digital start-ups are flourishing.

But we wanted to find out more about Kinect and where this fledgling technology wants to take us.

Via the Xbox Live avatars and Kinect, Rare has transitioned from a straightforward developer into something of a hardware and interface innovator. How has that been for the company?Well, we've always been about new and exciting things – I don't necessarily mean technology I mean we think about what can we do that's different to what's been done before. Ever since I first walked through the door 13 years ago it's always been about innovation; it was with the Stamper brothers way back in the eighties, it was with Nintendo, it certainly has been with Microsoft.

With the latter, it has become, "okay, you're about innovation, we have this new technology that you guys can probably do something with…" I mean, we developed avatars when Xbox was all about core gamers – avatars have softened the face of that. We were thinking about them years before they were introduced. Now we have something that's a huge IP for Xbox.

And of course, Rare has been instrumental in the launch and game development for Kinect. How did that come about?I remember three years ago, going over to see Alex Kipman and Kudo Tsunoda when they'd just finished incubating what was then known as Project Natal – although, actually, it didn't have a name at the time. It was the depth sensing 3D thing. Seriously. I remember I was there with our creative director George Andreas, Mark Stevenson and Gregg Mayles our design lead, and the four of us looked at this 3D feed and this skeletal tracking in its embryonic form, thinking... wow. I had this Fast Show moment, looking at my representation on the screen saying, "does my bum look big on this?"

I thought, for the first time it's like entering the matrix, I truly had a digital representation of myself in real-time, which we'd never seen before. I've always been about technology, I come from a computer visualisation background, and sitting on the flight home, I was like, "ooooh, you could do this and this!" I was thinking, we can properly track the player, background subtraction works now, we don't have to worry about the environment, we can work out what's in the room! Oh and we need to talk to Microsoft Research about the speech recognition. The three designers were just asking, "how do we make games for it?!"

What did you tell them?I said, guys, years ago you were wondering how to make a first-person shooter on a console – this is no different. It comes from sitting down in a room and thinking, okay, what are the important facets that this technology has got?

And we came up with an initial set of ideas – it was all marker pens and a whiteboard. Then we went off and started experimenting. We incubate lots of different ideas within a very small agile tech group. We literally started with just the tracked skeletal feed, mapping that onto a wireframe stick man. We put the camera on the eyes so you could look down and see your own wireframe feet, which was weird.

Then, Matt South, the guy who was developing that, said, "I want something to interact with." So he put a sphere in there and it had physics on it so you could pat it around. And he thought, "hmmm, that's interesting, I'll put some gravity on it." So it falls to the ground and, ooh, we've got feet let's try kicking it. A very simple football prototype developed from that. Then the design team thought, okay, we can actually walk around the ball and address it like a real player – this was all done with physics at the time. And then they'd go off in a design huddle and think, what other sports can we do? That's the process we've always gone through – and with Kinect, we really got under the skin of what it was, very early on. So far Kinect games have only really touched on this whole idea of telepresence of having a digital version of yourself on the screen. Do you think we'll see this developing, perhaps with multiplayer spaces, and head mounted cameras? So we feel like we're in a virtual room with our friends?Yes, there's no reason why not. Even now, one of the thing that I love when I play Kinect Sports over Live is that I can recognise the body language of the person I'm playing with, albeit in a fairly rudimentary form right now. But I can see that kind of thing developing – it will develop to a point where you and I absolutely will have telepresence, whether it's head-mounted, whether it's Kinect-like sensors placed around a room or in a mobile device … there's a heap of ways it could go.

I think the tracking and depth sensing technology that we have today is kind of where joypads were with the NES back in the early eighties – we've only really started down this path. I don't want this to be a sales pitch for Kinect Sports 2, but we've done a ton of work behind the scenes, the tracking technology is a lot more advanced. If you look at darts for example, the fidelity of control there is way ahead of what we had in the first title. It's all about software and the machine learning behind it. And that was just 12 months of iteration. If you extrapolate out from that, you can see how far we have to go.

How about the camera and audio functionality of Kinect. Will we get to a stage where it can pick up on emotional queues from gamers – watching body language and listening to the tone of voice?Yeah, I mean it's that process of continual development – and it depends on what any given developer wants to pick, and to push. With Kinect Sports 2 we really pushed voice – we had that virtual golf caddy, where you could just say, "change club, nine iron" and it does. The fact that it just does work is no mean feat in itself!

But right now, we're kind of commanding the box, rather than it necessarily interpreting and understanding us. Looking at where and how technology is developing, and the speed it has developed, we absolutely could have an emotional controller. Think about something simple like measuring excitement: without doing any speech recognition, just looking at the pattern of amplitude, pitch, the period between sounds, you can tell if someone is getting excited – the frequency of words increases, the pitch increases and the volume increases.

If someone is getting angry, typically, the frequency of words and the pitch doesn't increase but the volume does – and this can all be tracked by Kinect without it knowing anything about what's actually being spoken. That's just one idea. You could look at the colour feed from the camera, the depth feed, facial characteristics – there's no reason why you can't detect everything about the user, as long as the detectors are good enough – and the detectors are all software-based. That's what's really cool about this.

From an entertainment interface point of view, I love the idea of eventually being able to say to your console, 'Xbox, I want to watch a film' and it selecting a relevant movie, based on your emotional state…Yeah, I mean, if it's already learned the kinds of things you like because you've been using it as your media consumption tool, it can learn about you in the same way that Tivo learns. But if you then merge that information with data coming from the camera and mic about the mood that you're in, that could definitely work!

How about augmentation of regular control interfaces with motion elements? I'd love to have a version of Modern Warfare, where I can tilt my head to look round corners; but I still want to use the joypad to move and shoot. Is this on the way?We've always thought about that. When we launched Kinect we didn't want to confuse the issue by saying, "well this is a joypad-controlled Kinect game." You have to wear your heart on your sleeve and show everyone what motion control is all about. But from there, you can definitely start to say, well, we can augment different experiences with different bits of technologies.

Look at what Turn 10 has done with Forza Motorsport 4 – that's the first use of head-tracking control in an Xbox game, and you can absolutely see that going in to Modern Warfare or Battlefield. I could stick my head out of a foxhole, command my AI squad with voice… you have to make sure you fit the control mechanisms to the experience, and not the other way around. That's important.

I truly see a world where everything is Kinect-enabled in some way, but it shouldn't be, "oh, here's the Kinect moment." I don't want to stand up for three hours to clear a first-person shooter – I like using the joypad. But there are interactions where it just makes sense for the console to know what I'm doing. I was talking to someone about this yesterday – if I'm playing a shooter and I look down for a second at the joypad because I've forgotten the controls, why doesn't the game pause? Something as simple as that could be a really cool feature.So now the APIs for Kinect are out there, they're freely available, and people are experimenting with the technology at home, at schools and at universities… what are the most interesting projects you've seen so far?I'll be non-committal and say all of it! Seriously though, I've been impressed with everything that people are doing with the incorrectly titled Kinect Hacks – because there is no hack, there was nothing to hack. The Kinect drivers whether they're official or unofficial – there's just such a wealth of ideas. A lot of them were things we'd thought of when we first started brainstorming a couple of years ago, but we couldn't do because we're game developers. At the time, I thought, oh, wouldn't it be cool to control an RC helicopter with Kinect, and then someone goes and does it and sticks a clip on YouTube!

My personal favourite as an old Star Wars fan is something I saw at a hackspace maker fair a while back, where some guys had built some huge Tesla coils, and there was another guy there who had a Kinect. They got together and they added Kinect controls to these coils, so you could throw massive sparks into the air with your hands, like the Emperor. [There's a video of Tom Wyatt and Brightarc's Tesla/Kinect invention here.]

But in all seriousness, some of the things that have come out of Microsoft Research and other universities, especially for natural user input in different ways, they're really interesting. I mean, using a projector and Kinect to turn your hand into a keypad. Someone's even made a pop video with it. As the technology filters out there, we will see much more of this.