Parallel

Gestures, Multi-touch, and Microsoft Surface

Surface is an innovative computing platform that responds to touch, natural hand gestures, and real-world objects

With us today is Robert Levy who leads the design development of the APIs and controls in the Microsoft Surface SDK.

Q: Robert, what is "Microsoft Surface"?

A: Microsoft Surface is an innovative computing platform that responds to touch, natural hand gestures, and real-world objects placed on the display. It provides easy interaction with information and digital content in a simple and intuitive way, and has a large, 360-degree, horizontal user interface. By offering a unique gathering place where multiple users can collaboratively and simultaneously interact with content and each other, Microsoft Surface provides businesses with unique value in delivering information and services, and a way to better engage with customers.

Q: What's different from, say, a standard desktop device when it comes to developing software for it?

A: Microsoft Surface is architected as an extension to Windows, WPF, and XNA. The team focused on leveraging existing development tools, technologies, and patterns as much as possible. Because of this, the learning curve for Microsoft Surface development is actually quite minimal.

A popular demo we'll do for developers is to pull a sample app out of the Vista SDK and run "find and replace" command against it (to turn Button into SurfaceButton, ListBox into SurfaceListBox, etc) and show that this is all it takes to port an application over to the multi-touch world of Microsoft Surface.

That said, there is a significant learning curve and innovation opportunity when it comes to designing Microsoft Surface applications. Simply porting apps over with find-and-replace won’t result in an app that leverages the unique attributes of Microsoft Surface: multi-user, massive multi-touch, direct interaction and object recognition. The Microsoft Surface SDK includes a number of innovative new controls, tools, samples and documentation to help developers take advantage of these unique attributes and the new Service Pack 1 (SP1) release represents a significantly increased investment in this area.

Microsoft Surface is built around a standard Windows Vista PC with a layer of code running on top of the OS to provide the UI. You can run a standard Windows application on Microsoft Surface, but why would you want to limit yourself when you can do much more? Direct interaction allows users to "grab" digital information with their hands -- interacting with content by touch and gesture, without the use of a mouse or keyboard. Multi-touch functionality also creates a new area of challenges, as Microsoft Surface recognizes many points of contact simultaneously, not just from one finger like with a typical touch-screen, but up to dozens of items at once.

The horizontal form factor makes it easy for several people to gather around Microsoft Surface together, providing a collaborative, face-to-face computing experience. Another facet for developers to consider is the object recognition feature, which allows users to place physical objects on the Microsoft Surface to trigger different types of digital responses, including in the future being able to transfer digital content.

Q: I recently saw a demo of a Nokia N97 smartphone running a gesture-based application that used the phone's camera to capture gestures. So my question is, how does multi-touch software scale between platforms ranging from cell phones to Surface?

A: Within Microsoft, there is close collaboration across the company to establish a consistent developer platform with gestures and other multi-touch interaction concepts. While hardware capabilities may vary between Microsoft Surface, mobile and desktop, it is important to end users that they be able to interact with all devices in the same basic ways. Likewise, it is important to developers that they be able to reuse tools, design patterns and even code across these platforms so that they can get the broadest possible reach.

A good example of this cross-company collaboration is the Manipulation & Inertia Processor APIs. These were originally created for Microsoft Surface using .NET, but Windows 7 includes a COM version and a specialized version is included in the recently released WPF 4.0 Beta 1. Because of this collaboration, all developers -- whether they are targeting Microsoft Surface or Win7, writing native or managed code -- will be able to work with manipulations and inertia (how objects are moved/rotated/resized/flicked with gestures) in the same way.

Q: What's the most difficult part of developing multi-touch software?

A: The real challenge of developing for multi-touch platforms like Microsoft Surface is that developers are forced to rethink a lot of the basics of UI design in order to account for the strengths and weaknesses of touch interaction. With multiple users simultaneously interacting with an application for all sides of Microsoft Surface and while using physical objects, there are great opportunities to build innovative experiences that couldn’t exist before and really redefine the way people interact with technology -- but these opportunities require a lot creativity.

We're very pleased with the success our 180+ partners have had in designing innovative experiences for Microsoft Surface, but we’ve also gotten a lot of feedback from them on how we can make it even easier for them to create and deploy these applications. We’ve taken this feedback and used it to define Service Pack 1 of the Surface v1 platform which is now available.

SP1 allows for faster and easier application development with an improved SDK and developer tools, including API enhancements, WPF library and menu controls to help speed up the development process. A new stress testing tool simplifies testing of massive multi-input scenarios and a new 128-bit identity tag includes tools and APIs to help developers quickly generate identity tags that Microsoft Surface can recognize and track.

SP1 allows for new platform capabilities in the designing of new applications. Tagged object routing enables users to quickly switch between applications by using tagged objects and gives developers innovative new interaction options. Additionally, service applications run in the background and automatically launch new applications or notifications to create an improved user experience.

Microsoft SurfaceSP1 also includes improvements for end users, including enhanced access points with new audio and visual cues which allows for easier discoverability, as well as single-application mode, which provides easy access to one key application. Contact visualization was also added, displaying visual feedback when users touch Microsoft Surface, providing a faster learning curve with the touch experience.

With a growing partner ecosystem around the world, it was vital that SP1 provide increased reach through broader international support. Localized end-user and administrative interface in available countries (U.S., Canada, Austria, Belgium, France, Germany, Ireland, Italy, Netherlands, Qatar, Spain, Sweden, UAE and the UK) enables multi-language rollouts in top world markets.

SP1 also allows for easier installation, configuration and maintenance of Microsoft Surface. A simplified system configuration uses new calibration guides and doesn't require an external monitor. Integration with Microsoft Update provides automatic updates to keep units on the latest software and the Microsoft Management Pack support enables proactive and remote management of units.

Q: If readers want to learn more about developing multi-touch software for Microsoft Surface, is there a website they can go to?

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task.
However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Video

This month's Dr. Dobb's Journal

This month,
Dr. Dobb's Journal is devoted to mobile programming. We introduce you to Apple's new Swift programming language, discuss the perils of being the third-most-popular mobile platform, revisit SQLite on Android
, and much more!