Introduction

Multi-touch has officially become a buzzword. Inspired by Minority Report, Microsoft Surface, the iPhone, and the new Windows 7 Touch APIs, developers have many reasons to invest in this trend. In this article, I'll start with an introduction to multi-touch development with WPF, with special focus in setting up the development PC and inputting or simulating multi-touch. Then, I'll show a touch RSS reader, along with some quirks and interesting points when developing for touch.

Multi-touch development by itself is not hard; the hardest part is correctly choosing your API and setting up your device. So, we'll start by...

Choosing the multi-touch API

To develop multi-touch apps with WPF, we have currently three main options:

The first solution is the one that aligns with Microsoft's current strategy. However, the WPF 4.0 beta 2 APIs are still in a very raw state; even though they are functional, they don't provide easy to use controls - you'll have to program most of the interaction logic by hand. Besides, you are limited to Windows 7 as deployment platform. This solution will certainly evolve, and in the future will probably become the best, but for now, we'll pass.

Surface SDK is much more interesting. It's a very complete API, with interesting controls and a growing community. Its main limitation is the hardware - it only works with MS Surface or its own simulator app.

Therefore, we'll go with the third solution. It's not only the most flexible solution (works with almost any hardware and OS), but it's also easy-to-use and Open Source. In this article, I'll use Multi-Touch Vista [^], a very complete package by Daniel D, a.k.a. "Nesher". It includes a flexible input system and a WPF framework with many controls, licensed under the GPL. Because of that, you'll have to publish your source code along with your apps, under the same license. In this article, I'll refer to Multi-Touch Vista as MTVista.

Multi-Touch Vista overview and basic set-up

The next step is to understand MTVista's input layer in order to correctly configure it. This is MTVista's basic architecture:

As you can see, if you don't want to write a custom input provider for MTVista, you'll need either a TUIO-compatible device or multiple mice for simulation. TUIO is a widely-adopted open protocol to broadcast touch messages. Most multi-touch hardware isn't TUIO-compatible out of the box, so here we have several cases:

You have a Windows 7-compatible device, such as a Dell XT or HP TouchSmart:

After you have a TUIO device or multiple mice correctly set up, you can download and extract the MTVista binaries or build them from source. I recommend building it from source so that you have the most recent code.

Both the source code and the demo project included with this article contain a copy of the Multi-Touch Vista binaries built from change set 29484.

All set? So, let's continue...

Starting up the Multi-Touch Vista input service

To start the MTVista input service, you can either:

Run it directly each time, by running MultiTouch.Service.Console.exe, or

Install MultiTouch.Service.exe as a Windows Service using installutil.exe (i.e., run %windir%\Microsoft.NET\Framework\v2.0.50727\installutil.exe /i Multitouch.Service.exe) and then set up automatic startup using services.msc.

After it's up and running, you should see one or more red dots in your screen (one for each connected mouse). That happens because MTVista uses the Multiple Mice provider as its default. If you want to use TUIO, you must run MultiTouch.Configuration.WPF.exe, select the TUIO provider, and click the blue arrow to apply your selection. You may need to unblock the service in Windows Firewall, as it broadcasts the TUIO messages locally using UDP packets.

That's it! The hardest part is done :-).

Developing a multi-touch app

To create a new application with MTVista, the path is almost always the same:

Create a normal WPF application.

Add a reference to MTVista's Multitouch.Framework.WPF library.

On the XAML files, add reference to the MTVista namespace:

xmlns:mt="http://schemas.multitouch.com/Multitouch/2008/04"

Change the <Window> tag to <mt:Window> and change the code-behind class from System.Windows.Window to Multitouch.Framework.WPF.Controls.Window.

That's it! Now you have an empty multi-touch-enabled window:

The rest of the development flows the same way as a normal WPF app. In the next section, I'll show the demo app, detailing some interesting points that are specific to multi-touch development and some patterns that might be useful when working with multi-touch.

The demo application: a multi-touch RSS reader

RSS readers are everywhere today; they're simple, useful, and familiar - almost every reader looks the same: a list of feeds with or without categories, a list of news items for each feed, and a reading area for the selected article. But how can we make it more interesting and easy-to-use in a multi-touch environment?

The demo application here tries to change the common RSS reader layout a little bit. Its main features are:

Integration with the Windows RSS platform (Internet Explorer Feeds) for getting its data.

A panel-based system for folders, feeds, and items. The panels can be dragged and zoomed with multi-touch gestures, and can bounce off the "walls" of the application.

A "trash" icon to close panels by dragging into it.

Feeds can be read inside the application, without the need for an external browser.

Now, let's dive into each of these features do discover what's behind them.

Integration with the Windows RSS platform

To enable integration with the RSS platform, I've used Cristian Civera, a.k.a "Ricciolo"'s excellent PaperBoy library, part of his PaperBoy application [^]. This library wraps the RSS platform in a simple model, and exposes the data through a singleton called FeedManager.

In this app, we'll use the Model-View-ViewModel (MVVM) pattern to connect this model with our multi-touch front-end. MVVM is a pattern first described by John Gossman, and it's widely used to develop WPF applications. If you want to know more about it, take a look at Josh Smith's article on MSDN Magazine [^] or several others here in CodeProject [^]. In this project, I've used a simple set of ViewModels:

These ViewModels inherit from ObservableObject, from the MvvmFoundation Framework [^], developed by Josh Smith. This framework simplifies repetitive tasks with MVVM, such as inheriting from INotifyPropertyChanged and managing the PropertyChanged event.

Each of these ViewModels correspond to one data type we want to represent (Folders, Feeds, and Feed Items). They expose all of the properties that we want to display, like items, titles, and unread items, which come from the Model (Windows RSS Platform).

To initialize the application, a FolderViewModel pointing to FeedManager.Current.Root is created and bound to the corresponding view (more about this in the next section).

Multi-touch panel-based views

After the Model and ViewModels are set, we must think about the user interface. Users expect panels to have support for dragging, zooming, and rotating. Also, it's necessary to enable menus to work like an iPhone menu (with finger-scrolling and inertia).

To create a scrollable view using MTVista, all we have to do is wrap an ItemsControl in a MTVista ScrollViewer. Here, we'll use MTVista's DraggableScrollViewer because it also enables dragging support (it fires an event when you drag elements out of it). A scrollable view in this app without styles and behaviors would look like this:

In the TouchablePanel, the damping properties are used to customize inertia and friction. EnableWalls will make elements bounce off, and RandomizePositions is set to false to make sure we can control the initial positions of this panel's children.

The Trash element will be explained in detail in the next section.

Now we have a TouchablePanel, a set of Views and ViewModels. The next step is to enable an item to be dragged out of a view and create another view, depending on what's being dragged.

To do that, the main idea is to create a Factory-like class that would enable the TouchablePanel to create its children views. One solution would be to subclass the TouchablePanel and add methods to create the views, making it work as a Factory.

However, for this article, I've tried to leave this solution as general as possible: I wanted to create a reusable Factory interface that could be implemented in each specific case, allowing for more reusability. In this app, the Factory architecture looks like this:

The implementation of the Factory must decide how to create a view based on the DataContext of the new item and its position. It also needs to know the target Panel to add this new view to its children accordingly.

In this app, the implemented Factory method works like this:

First, it checks if the DataContext is a valid RssViewModelBase.

If it is, it finds the correct view from its type.

Then, it creates the view and associates it to the DataContext.

The view is positioned using Canvas.SetTop and Canvas.SetLeft. This works because the TouchablePanel is derived from Canvas.

Next, it adds an event handler to this new view, enabling the element to capture contacts when touched. This enables views to be dragged without having to start the movement directly over them.

Finally, it adds the view to the panel's children, thus displaying the view.

Finally, to connect the Factory to the views, we'll use Attached Behaviors (see this article from Josh Smith [^] for more). In this app, two behaviors are used:

A behavior that allows us to link an IPanelViewFactory to a Panel, and

A behavior that handles the DraggableScrollViewer's ElementDragged event and calls a Factory to create a View from the dragged element's DataContext.

The usages for these behaviors would look like this:

<Panelb:PanelBehavior.ViewFactory="{Binding [reference to the factory]}".../><mt:DraggableScrollViewerb:ElementDraggedCreateViewBehavior.PanelViewFactory=
"{Binding [reference to the factory]}".../>

You can see the usages of these behaviors by taking a look at the source code in MainWindow.xaml and in the scrollable views (FolderAndFeedView.xaml).

This approach is very flexible: whenever you want to build another app that needs to control how views are created in a panel, all you have to do is implement an IPanelViewFactory and attach it in a similar way.

After everything is connected, the last thing is to initialize the application by calling the main panel's Factory to create the root folder view (from FeedsManager.Current.RootFolder):

[MainWindow.xaml.cs]

public MainWindow()
{
InitializeComponent();
// Initialize the form by creating the root folder view,// using the mainPanel's ViewFactory
PanelBehavior.GetViewFactory(mainPanel).CreateViewFromDataContext(
new FolderViewModel(FeedsManager.Current.RootFolder),
new Point(100, 100)
);
}

Trash icon to close views

The next feature we want to implement is a Trash icon. The idea here is to enable views to be dragged to the trash when we want to close them, and then hide them with an animation. Here, you can see a view being "swallowed" by the trash:

Now, let's break down the requirements for the Trash to find out what we need to make it work:

The panels that will be dragged must be inside a Panel that supports dragging (usually a TouchablePanel).

In MTVista, the event that says that a contact has been "released" from an element is the ContactRemoved event. In that case, the PreviewContactRemoved event will bubble up to the parent Panel.

The Trash doesn't have to belong to the Panel - if it's contained in the panel, it will be "draggable", and that's not what we want here.

A single Panel could have multiple trashes (e.g., a multi-user table where each user has his/her own trash), but a trash can only be associated to one Panel.

The root folder can't be closed, or we will end up with an empty panel with nothing to add new items.

When an element is dragged to the Trash, we want to use an animation to shrink it to zero pixels, giving the impression that the trash "swallows" the element.

As we can see from the first and third requirements, it would be useful if the Trash knew its source panel, so we can handle the panel's PreviewContactRemoved event and react accordingly.

In this app, I've used another Attached Behavior to add a SourcePanel property to the Trash control. The usage of this behavior looks like this:

<local:Trashb:TrashBehavior.SourcePanel="{Binding [reference to the panel]}".../>

This behavior, when attached to a Trash control, will add it to a list of trashes associated with the panel. It also handles the panel's PreviewContactRemoved and attaches it to a handler that works like this:

First, we check the dragged element to see if it's not the root folder or another trash.

Then, we calculate the intersection between the element and each of this panel's trashes to see if there's overlapping.

If an item overlaps a trash, an animation will be started. This animation acts on the ScaleX, ScaleY, and Opacity properties, reducing them to zero. A dictionary is used to associate the animation to the dragged element.

Finally, when the animation is over (the CurrentStateInvalidated event fires), we use that dictionary to recover the element and remove it from the Panel.

For more details about this behavior, see the PreviewContactRemoved handler method in TrashBehavior.cs.

Reading feed items inside the application, without an external browser

The last feature in this demo app is the ability to read the feed contents in the multi-touch environment, without the need for an external browser. To do this, I've used another set of classes from Ricciolo's PaperBoy [^] application to convert the HTML code to a XAML FlowDocument. This set of classes is an evolution of a Microsoft sample that shipped with the WPF SDK (see this blog post [^] for more details).

To use this converter, all you need to do is use code similar to this:

xamlString = HtmlToXamlConverter.ConvertHtmlToXaml(htmlString, true);

In this method, call:

The first parameter is the HTML content as a string

The second parameter is a boolean that indicates if the content should be wrapped in a FlowDocument

The return value is the XAML string converted from the HTML source

After the conversion, all we need to do is to parse the XAML code using a XamlReader:

FlowDocument document = XamlReader.Parse(xamlString) as FlowDocument;

In this demo app, the FlowDocument is displayed in a FlowDocumentScrollViewer inside a MTVista ScrollViewer, to enable touch panning:

For more details about the HTML to XAML conversion and display, see FeedItemViewModel.cs and FeedItemView.xaml. I'd also recommend a visit to the WPF SDK sample [^] and to the HTMLConverter classes included with the source code.

Wrapping up

In this article, we touched the fundamental aspects of multi-touch development with WPF, like set up and choice of the API. We've seen a demo app that contains many common patterns that you may find interesting in your multi-touch apps, such as the "Trash" and creating views from dragged elements.

I hope that this article could provide you with some tools and ideas to start your own multi-touch app development. I'm looking forward to your next-generation multi-touch experiences!

For the future

Some interesting next steps for this app could be:

Developing a version with the WPF 4.0 Beta 2 APIs or with the Surface SDK.

Share

About the Author

Hi! I'm Roberto. I'm a Brazilian Engineering student at the University of São Paulo and the Ecole Centrale de Lille (France).

I've participated in the Imagine Cup competition and went to the world finals every year from 2005 to 2009. I also won the 1st place award in 2006, in India, for the Interface Design invitational, in 2007 in Korea, for the Embedded Development invitational, and in 2009 in Egypt for the Windows Mobile Award.

I was trying a MTV Hello World tut here and was going to try this one next. Both tuts tell me to add this namespace: http://schemas.multitouch.com/Multitouch/2008/04 with the line xmlns:mt="http://schemas.multitouch.com/Multitouch/2008/04" but the website is down. I also get
The type 'mt:Window' was not found. Verify that you are not missing an assembly reference and that all referenced assemblies have been built.
and the same for mt:TouchablePanel even though I included the Multitouch.Framework.WPF.dll reference and Multitouch.Framework.dll just in case

The schema URL ("website") represents the CLR namespaces that are being represented by the XML namespace. The website doesn't have to be online for it to work. In that way, all you need to do is add the Multitouch.Framework.WPF.dll and the namespace should work. You can try downloading the source code from this article to see how it was built.

I am impressed by your work , it is awesome.
I need your help in my multi touch application.
I have a question about 3d model in wpf application :
I made 3d model of liver (reconstructed from CT medical images to generate stl file) ,
I imported this file into the wpf form , i want to make an application that allows the user to cut and
make cross section in the model (note : this model isn’t solid , it should view details inside the liver)
what could i do? from where could i start?
any help is appreciated.
thanks

First of all, I have to thank you for your advices for multi-itouch develop. And I have been using Surface Toolkit to develop my multi-touch applications. But there is a question that when I defined a element, such as a image, in ScatterView, you know we can resize the image. But the resize operation changes the width and height of the image at the same time. My question is if I can resize the width of the image whose height was not changed. In the other words,can we stretch the image that is changed its width only.
Thanks for your helping!

I want to develop multitouch programs with Multitouch Vista. And do you know where I should get some articles just like "SDK documentation", that can give me some advises about how to use Multitouch Vista'elements like "TouchablePanel".

And the other qusetion is that I defined a image (used ) that the size can not be stretched unlimitedly. Can you tell me why and what shold I do?

Hi. I like to thank you for such a wonderful article and I actually manage to use the same application for one of my customer.

I am developing another application where an images are floating in a panel and the panel contains a trash box, as displayed in your application.

I wanted to do the same action, when image touches the trash box it will go inside, but I am not able to find the right event for the same action. Unfortunatly drag and drop is not an option since image are floating everywhere.

Is there any event which can tell me that a FrameworkElement is now in the area/region of trash box so that I can trash the image like you are doing?

Again, you can use the same TrashBehavior as explained in the article. The event that this behavior captures is the PreviewContactRemoved, that bubbles up to the parent (TouchablePanel) and allows us to find the intersection between the elements (trash / item to remove), and then animate and remove the item.

I'm trying to use Multi-Touch Vista for an app i'm creating. I'm stuck on figuring out a way to fix the position of a panel so that it cannot be dragged. However, I need the children objects of the panel to be draggable.

Hi,
I am running window 7. I try to run the application but RSS doesnt work for me.
I am able to run the multimouse but when I open RSS, there is only 1 window (without any RSS) and a trashcan icon. Could you show me how to fix it?

Hello wizi83,
To add RSS feeds, just add some feeds and folders to your Internet Explorer feeds. To do that, open IE and navigate to an RSS feed (ex. http://feeds.feedburner.com/virtualdreams) and subscribe to it.
You can also organize your feeds and create folders by opening the Feeds tab in the Favorites pane in IE (Ctrl-J).

I got it. Thanks for the quick reply. Its awesome. Have you ever try it with a Wiimote and 2 IR pens? Its pretty cool .
I have one more question: what if I want to do this with images instead of feeds? Is it difficult to change the code?

Yes I have tried with a Wiimote and pens - it's really cool, but a touch table/tablet is even cooler

If you want to do this with images it's not hard at all. The steps would be something like this:

1. Change the Model and ViewModel to represent your picture collection. For example, you could create classes to represent categories and pictures; get a list from a folder or a XML file, etc. The ViewModels would expose that in a bindable way to the Views.
2. Create Views that display these pictures in the screen. To do that, change the DataTemplates to something that displays an image (ex. Image control with the Source property bound to the Image URL)

And that's it. If you need more specific info, just ask.
Thanks!
Roberto

I have the above code, and it runs, but how do I trap mouseclick events to each image? Basically, I want to know what image is being clicked and be able to get/set the corrdinates and other properties of any selected image. I've built a surface computing table, i've built a shell in WPF to run it on, but I just need to get events when an image is clicked or manipulated.

What about drag and drop events for images created by datasource binding... Basically I want to be able to drag and drop an image on top of another image which is a picture of a recycle bin and have the event be triggered so I can remove the source object from the screen.

To simulate drag and drop, you can use a strategy similar to the one I've used in the Trash, in this article.
The idea is to handle PreviewContactRemoved in your canvas, and check for intersections between elements each time the event is fired.

In the article I've created an Attached Behavior to do that. You'll simply need to attach it to a Trash UserControl, binding it to the source panel; feel free to change the code (TrashBehavior.cs) to adapt to your requirements.