KinectoPhone – Kinect and Windows Phone working together

Description

A while ago we saw a demo from Microsoft that showed how it is possible to use Windows Phone to interact with a Kinect-powered game. During the Kinect Code Camp, right before the official release of the Kinect for Windows SDK beta, Adam Kinney, Rick Barazza, and I decided to work on a proof-of-concept project that would provide the same platform integration in a non-commercial development environment. Because it involved the main character hopping around collecting bonus items, we called the project Bunnyhop (codename KinectoPhone).

Initial Mashup

The project was divided into three major components:

The desktop client – a Kinect-controlled 3D environment. It is only partially controlled by the motion sensor, since other responsibilities are delegate to the phone. Because it is a proof-of-concept and not a full-sized game, WPF is the primary sub-framework.

The mobile client – a Windows Phone application that also is in charge of partially controlling the 3D environment on the desktop client. We decided to use Silverlight in favor of XNA because we didn't want complex graphics and all we needed was the bird's eye view of the playable area.

The server – one of the most interesting (and issue-generating) components. This was the link between the desktop and the mobile client. Initially we planned on using a WCF service (and we actually did for a while), but then we decided to use sockets for better performance..

Developing the communication layer

Since we decided to remove support for WCF and focus on sockets, we needed to figure out a way to send appropriate data to both clients. Given the fact that the amounts of data that are being sent across clients are not really big, it seemed a good idea to make the server broadcast a dual data set—a collection of indicators that are used by both the phone and desktop clients. It will be the client's responsibility to select the required value.

Here is the general structure of a data package broadcasted by the server:

We tried to keep it as simple as possible. The desktop client is going to process the camera direction sent by the phone, and the phone will adjust the character position according to the movement generated by the Kinect-driven desktop client.

When the clients (both desktop and mobile) send data to the server, they identify themselves through a prefix. For example, the desktop client sends the following string to the server:

The c prefix identifies the sender as the desktop client (computer). The phone sends the camera angle, led by the p prefix (phone):

Before going any further, I must mention that socket communications on Windows Phone are fairly generic, so the code we decided to use is close to the Sockets Sample on MSDN. We added a custom string parser that makes sure that the proper data is read between devices.

The server

Our server is entirely based on socket interactions through the TCP layer. It is a .NET-based console application that is listening to incoming connections (limited to two) and is broadcasting the received content, synchronizing both the desktop and the mobile clients. It is a "dummy" server and so doesn't perform any data processing—it simply receives the data from one client and then passes it to another.

The received byte array is later transformed to a standard string that is parsed according to the pattern described above. The parser detects the prefix and assigns the correct values depending on the sender.

The difference in received data is present because each client is designed to affect only a part of the gameplay, and character movement cannot be directed by a desktop client or by the mobile application on its own. The mobile client reports the angle of character rotation, and the desktop client controls general movement progress and bonus pickups.

This is pretty much it when it comes to defining the server that handles cross-platform communications. It can be installed on a machine other than the one where the desktop client is running, as long as all of them work on either the same subnet or a network in which the IP addresses are static and have the invoked port open.

The desktop client

The desktop client was a bit of a graphical “fortress.” Rick was building the entire 3D environment while Adam and I were building the mobile application and the server interoperability layer.

We used the newly launched Kinect SDK to access the connected Kinect and it was initialized the moment the main window is loaded:

Runtime is the standard device class that identifies the Kinect currently connected to the PC. As long as we were using just one sensor, we did not need to provide a numeric index when instantiating the Runtime class itself.

Since we are only using skeletal tracking—the desktop client is responsible for moving the character forward on a detected jump—the Runtime instance is initialized with a RuntimeOptions.UseSkeletalTracking argument. The event handler is invoked once we detect a tracked skeleton:

The Kinect sensor monitors six skeletons at once, but not all of them are tracked for movement. That's the reason to check the SkeletonTrackingState before displaying the joint indicators on the screen.

The joint indicators themselves are declared in the XAML layout in the main page:

In the skeleton tracking code above, we are not only tracking the joint position, we're also adjusting the character position based on the center hip location in space as well as detecting the jump by coordinating joint positions by the Z-axis.

The data is sent via an instance of the same AsynchronousClient I used on the server side, but with small modifications. I implemented a short verification mechanism that ensures the fact that I am using an IPv4 address as the endpoint will be functional with only IPv4 and not IPv6 addresses for the time being:

Phone client

The phone client has only one responsibility—adjust the camera angle as the user tilts the device. We used standard accelerometer-based capabilities to track the movement of the device in space.

The app uses a different implementation of the AsynchronousClient called SocketHandler. It works well to link the phone to the server. For example, take a look at the handler that processes the received input:

Due to threading limitations, we had to use the dispatcher class to notify the UI about the fact that all data from the server was received and that it should be transformed to the position of the character on the map.

In the main page constructor, we are initializing the SocketHandler with the same values for the server as we did in the desktop client. Additionally, we're wiring an event handler that will handle the received string. Unlike ProcessReceive above, the wired event handler actually parses the string for required values and accordingly takes the appropriate action:

The turn time is checked so that the character on the device screen doesn’t constantly rotate. Since the accelerometer is very sensitive and without a time limit, a slight tilt will have unexpected consequences on the rotation behavior.

How to test

Virtual Router is a great way to test the set-up. Virtual Router allows the server computer to become a WiFi hotspot so that any connected device will be automatically registered as a part of the local subnet.

You will need:

A computer with a WiFi card (most laptops have them anyway)

Make sure that you have the WiFi card enabled and launch Virtual Router Manager. Set a new network name, as well as specify a password. It is worth mentioning that this network will act like a standard wireless network, so make sure you secure it with a strong password if you don't want it accessed from the outside.

Once you enable the hotspot, you can run the desktop client on the same machine. Connect the phone to the newly created wireless network and make sure that the proper IP is set in the code-behind for both the desktop client and the Windows Phone app.

The IP for the server machine can be found by running the ipconfig command in the console.

The IP address for the Windows Phone device that is connected to the hotspot is shown in the main Virtual Router Manager window.

In case the router cannot be started, make sure that connection sharing is enabled for your main connection and that it points to the Miniport interface.

Once the network is correctly set up, I would highly recommend launching the project components in the following order:

Server

Desktop client

Phone client

Conclusion

With the latest 7.1 SDK update for Windows Phone as well as the 7.5 (Mango) OS, it is fairly easy to set up a Kinect-to-WP communication layer through sockets. With small modifications and performance optimizations we can also send Kinect imagery (captured through the depth and video sensors) directly to the phone without major delays. The field of third-party games that interoperate with Kinect and run on mobile devices is yet to be fully developed, but we already made the first steps in that direction.

About the author

Dennis Delimarsky is a Windows Phone Development MVP. He enjoys technical writing as well as building cool stuff around the .NET platform. Besides Coding4Fun, he also blogs at Den by default. He can be contacted at ddelimarsky@live.com.