WiSee can detect arm and body motions anywhere in house, use them as commands.

Flipping off your television may gain a whole new meaning thanks to a technology being developed by a team of researchers at the University of Washington. The team, led by Assistant Professor of Computer Science and Engineering Shyam Gollakota, developed a system dubbed WiSee, which uses radio waves from Wi-Fi to sense human body movements and detect command gestures from anywhere within a home or office. The results of the WiSee team's research have been submitted to the ACM's 19th International Conference on Mobile Computing and Networking (Mobicom '13).

Unlike other "machine vision" sensors such as Microsoft's Kinect, the system can sense gestures anywhere within a house or office environment using the Wi-Fi signals created by devices already in the environment. The user doesn't need to be within line of sight of the WiSee receiver—or even in the same room.

"The nice thing about Kinect is that it does motion detection without you having to carry anything around," said Shwetak Patel, an assistant professor of computer science and electrical engineering at UW and one of the lead researchers behind WiSee, in a phone interview with Ars. "We started out looking to see if there is a way to do what Kinect does in a larger area, and we started looking at RF." The ubiquity of Wi-Fi and the multiple antennas of newer MIMO Wi-Fi routers were a natural fit for the research.

A demonstration of WiSee in action.

WiSee "sees" gestures by detecting subtle changes in the radio signals bouncing off of and passing through human bodies as they move. Changing the body's position or moving a hand or foot causes a small Doppler frequency shift in Wi-Fi signals that can be detected by a receiver. When Wi-Fi signals hit a human body, "some is absorbed, and some is reflected," said Sidhant Gupta, another UW researcher contributing to WiSee. "The reflections cause a very subtle shift in frequency, in the tens of Hertz." Wi-Fi protocols generally are robust enough to handle those variations, but WiSee uses them to detect motion.

Using algorithms to screen out normal variations in the broadband signals of devices and correct for normal gaps in broadcasts, WiSee can separate out the signatures of a series of movements from the rest of the broadband signals. "The Wi-Fi channel by itself is 20 MHz wide," Gupta said. "You can't just look at that whole spectrum and find subtle changes of 2 Hz." WiSee's detection algorithm breaks the broadband Wi-Fi spectrum into smaller chunks, and processes them to discover the shifts hidden within them.

By using multiple antennas and a Wi-Fi receiver with multiple input multiple output (MIMO) capability, WiSee can "lock on" to a specific user with an antenna from among a group of other people in a space.

"You can determine where a gesture is coming from," said Gupta, "whether it's in the kitchen or the dining room. That's one way of compartmentalizing who the gesture is coming from." And gestures in different locations of the house can mean different things, or be localized to where the user is—for example, by changing the appropriate lighting based on what room the user is in.

The UW team applied machine learning to the RF signatures of a series of body movements to help the system identify them as gestures associated with a command. WiSee was able to identify a set of nine gestures with a 94 percent accuracy rate during the experiment. Patel said that the team stuck to nine gestures because of the limits of the simplified machine learning process they used for the first effort—the next version of WiSee will use a Hidden Markov Model to look for sequences of gestures and allow a much larger vocabulary of commands. "The intent is to make this into an API where other researchers can build their own gesture vocabularies," Patel said.

Two of the research team members working on WiSee brought experience from two similar projects sponsored by Microsoft Research, which also used Doppler shifts to detect body movements. One of them, Humantenna, used changes in signals from electrical "noise" and other background radio frequency radiation picked up by the human body as it moved to detect gestures and motions. Another, SoundWave, uses speakers and a microphone to detect a Doppler shift in reflected sound waves.

Humantenna, a Microsoft Research project, turned the user's body into an antenna, picking up shifts in radio frequency from background electrical "noise".

SoundWave uses an "inaudible" tone as sonar, picking up Doppler shifts in the signal from movement in front of a PC's microphone.

But both of these previous projects required the user to be in a specific room or directly in front of the device they are interacting with. WiSee can "see" through walls, making it more practical for applications like home automation as well as the usual Minority Report-like interactions with media and computing devices. "We haven't tested the upper limits of gesture recognition," Gupta said. Patel added that the practical limits of gestures "depend on the entropy of the signal. It won't ever be as smooth and crisp as Kinect."

One of the next concerns to be addressed will be "how do you make it secure," said Gupta. "Someone walking by your house should not be able to turn your kettle on by waving his arms." Currently, users can be distinguished by a "startup" gesture that identifies them before allowing a command gesture. But the team is looking at how to "geofence" commands within a specific room or the limits of the house.

This story was updated with additional data from the University of Washington research team.

It could also allow for the use of a series of physical gestures as a "password" before allowing a command gesture

You could lock someone out of their account by breaking their arm!

A 'password' of physical gestures is just represented by some bits in a computer. It would be simple to have a backup keyboard workaround. But there wouldn't be much point in getting access for command gestures, if you couldn't make the command gestures anyways.

You don't think the ability to detect human body positions through walls in a residential setting, like an apartment, is going to cause any issues?

That technology already exists. If you are that worried about it, buy a shit load of aluminum foil.

Gold foil would probably work better. Gold is a better reflector of infrared, so they can't scan your house with infrared thermal cameras, and an excellent conductor to form a Faraday cage against sensors like this. Plus, you'll be TEMPEST certified, guaranteed, and the envy of rappers everywhere when you coat your house with it.

As a gesture recognition system, I can imagine some limited use for this. If the resolution and positioning are accurate enough to recognize when someone's looking directly at their television, and making a very specific gesture, towards their television while doing so, then hey maybe you can... change the volume up or down or something.

But imagine this being used to see through walls as spying tool. Now THAT seems incredibly useful in the right scenarios for the right people. Wifi beaming drones that can see through walls anyone?

Forget the dog, I don't want to turn my TV on and off every time I stretch. Especially if I'm in another room.

Any consumer application for this would have to be very short-range IMO, or remotely turned on/off somehow (maybe that's where the minority report glove could come in). Say I have some friends over and there's 6 of us crowded around my TV. I imagine many run-of-the-mill gestures would occur that could drive the thing crazy were it programmed to accept similar gestures.

My two-year-old brother once bought a PPV movie without this technology. I fear that this new technology will make it easier for him to buy movies and for me to cry because of my TV bill (yes, I pay my parents' TV bill).

Reminds me of a Continuum episode from the first season where the protagonist used the cell-phone signal from a nearby tower to reconstruct what was happening inside a building. Maybe not as sci-fi as it appeared back then after all.

Reminds me of a Continuum episode from the first season where the protagonist used the cell-phone signal from a nearby tower to reconstruct what was happening inside a building. Maybe not as sci-fi as it appeared back then after all.

In a non-imaging configuration like this, you're basically looking at changes in the resonance of the wifi signal through a building. By having a subject move around in predefined ways, the change in the spatial distribution of the reflected signals can be mapped out with great sensitivity. I actually worked on something similar for a very different application ages ago. It really does work because interferometric effects like this are incredibly sensitive to small changes.

But the thing is, it depends very precisely on the state of the surrounding not changing between calibration and measurement. Its not robust like an imaging system because EVERYTHING in the surrounding area feeds into the signal you measure, not just target. You have to be able to measure the background and remove it otherwise you can't make sense of what you observe. So better hope there aren't more people moving around, no one moves a couch or opens a door, and the weather doesn't change too much or you're going to have to redo the whole calibration.

Think about alternative uses for this. Something as simple as using detectors in a nightclub to fashion some sort of living light show (now THAT could be entertaining and cool), or for more practical uses, mobility for the handicapped, or military use in activating remote sensors.

Or even for traffic lights, to detect the flow of traffic in various directions, so you dont get those annoying red lights at 3am when there's no traffic in sight. The WiFi detection doesnt have to be just physical gestures, it can just be for detection in general.

While I'm sure there are lots of ways in which this could be abused as with all technology, I think it's a pretty cool and novel idea and a pretty amazing application of machine learning. Seems like it could also be pretty useful as a security mechanism as well. Maybe as some sort of motion detection or intrusion alert. Set it to vacation mode and have it e-mail you if it detects some sort of unexpected movement.

Or even for traffic lights, to detect the flow of traffic in various directions, so you dont get those annoying red lights at 3am when there's no traffic in sight. The WiFi detection doesnt have to be just physical gestures, it can just be for detection in general.

Using radio for proximity detection! We should call it something catchy. How about "Radar?"

Or even for traffic lights, to detect the flow of traffic in various directions, so you dont get those annoying red lights at 3am when there's no traffic in sight. The WiFi detection doesnt have to be just physical gestures, it can just be for detection in general.

Using radio for proximity detection! We should call it something catchy. How about "Radar?"

We've had the technology since the 1940's. It is a huge disappointment that Ars ran the story (I made a joke about /. running it earlier today in the iPhone ban thread).

Seriously, nearly every time you walk into a grocery store, you're using exactly this "gesture recognition" technology. At least the door opener is in a more optimal frequency range than your microwave.

When I want to wire up my house with defence turrets I'll be sure to use a future version of this. I assume they'll have worked in IFF by then.Reminds me of movies where invisible people break into places. This is an alternative to the infrared camera they usually use. What if the infiltrators are using a temperature shielding suit? What then!?WiSee, that's what!