Post Categories

Post Archives

Blog Stats

Touched ( Part 3 )–Manipulation and Gesture Support

quick note – this post has a lot of sketchy code in it that I’ve not really tidied up – apply a pinch of salt and look for better examples elsewhere as I’m just trying to skate across the surface of the different API options.

Following up on that previous post, I thought I’d carry on a little and think about gestures.

What’s a gesture? There’s a list over here on MSDN about the types of gestures that Windows recognises;

and those include;

tap/double tap

panning with inertia

selection/drag

press and tap

zoom

rotate

two finger tap

press and hold

flicks

although this is for Windows and so that doesn’t mean that every one of those gestures shows up (e.g.) down on the Windows Phone 7 or in particular frameworks like WPF and Silverlight.

Then there’s the touch based manipulations such as rotate, pan and zoom which are often paired with inertia in order to give a more real-world feel to the interactions.

There’s at least 3 sides to this looking across WPF, Silverlight and Silverlight for Windows Phone 7.

Does the platform have native support? If not, can you work around it?

Is that support something that you can hook into?

Do the built-in controls automatically make use of that support to (e.g.) scroll through a ListBox when you flick downwards or upwards?

In this post I’ll look a little at (1) and (2) and I’ll leave (3) for another post.

WPF and Manipulations/Gestures

How does WPF cope when it comes to gestures? Pretty well – WPF is really the Swiss Army Knife of UI frameworks and it’s rare to find something that it doesn’t cope with although as we’ll see in the follow on post that doesn’t mean that we can’t add to it with things like the Surface Toolkit.

WPF supports gestures in 3 ways.

It supports raw touch input as we saw previously so you could use that to build your own gestures but I doubt that you really want to do that given that…

It supports manipulation events on UIElements and those events report translation, rotation and scaling of the element in question. It also supports inertia for those events.

It has direct support for some of those other system gestures that I mentioned previously – “two finger tap”, “press and tap” and so on. These were present in versions of WPF prior to 4.0 because they mostly come over from the stylus world of a Tablet PC and WPF has long-term support there.

Note – I’m not sure that you’d combine (2) and (3) here without caution as I suspect that’s likely to lead to a UI that’s going to at the least confuse the user.

System Gestures

To hook into the system gestures, I can make use of the [Preview]StylusSystemGesture event on UIElement which gives me data about the type of system gesture detected which would be enough for a gesture like “two finger tap” but for a gesture like “drag” it’s likely you’d need more information and that is left to the developer to do either by monitoring stylus events more generally as they lead up to the event or by interrogating the system gesture event data after the event. I made a tiny UI;

and as I run that UI, I can get a picture of the gestures that I’m using;

Manipulations

WPF 4.0 also has support for the touch manipulation gestures that drive rotation, translation and scaling on a particular element. The UIElement class has a IsManipulationEnabled property and if you switch that on then ( as long as you don’t swallow the raw touch events for that element ) you’ll get manipulation events such as;

ManipulationStarting, ManipulationStarted

ManipulationDelta

ManipulationCompleted

ManipulationBoundaryFeedback

ManipulationInertiaStarting

So, if we build what feels like it has become the “Hello World” of the touch user interface like this one defined in XAML;

and you’ll notice that both my Rectangle and my Viewbox here have set IsManipulationEnabled and are handling ManipulationStarting, ManipulationDelta and ManipulationInertiaStarting – I wrote a little bit of code to try and handle those events;

there’s undoubtedly a lot more that could be done to make this “correct” and those deceleration numbers are picked out of the air but seem to give me a reasonable experience and I not only get manipulation but also inertia thrown in by the framework and I can drag things around on my UI;

Silverlight & Manipulations/Gestures

Silverlight does not have support for manipulations/gestures. As discussed in the previous post, you can get access to the raw touch events but you’d have to write all the code around gestures yourself.

The Surface Sample for Silverlight

This sample is licensed under the Surface SDK 1.0 Sp1 license agreement and provides an implementation of System.Windows.Input.Manipulationswhich is present in .NET 4.0. You get an assembly which you can reference but no source for the implementation.

There’s really 2 things in here. There’s the ManipulationProcessor2D and there’s the InertiaProcessor2D. You feed a set of touch events into the former and it combines them into manipulations for you. The latter can continue those manipulations to provide a sense of inertia to user actions.

This is largely giving you what WPF gives you in terms of manipulation and inertia but it hasn’t been pre-wired into the Silverlight UI Framework for you as it has been in WPF by tying it onto UIElement. So, you have to do some work yourself.

Again, I put together a very simple bit of UI just like the previous one except this is now Silverlight;

and that “more or less” gives me a usable UI. I’m sure there’s a bunch of refinement that needs adding to it;

Within that sample library, there is also an InertiaProcessor2D class which you can use to add inertia. When a manipulation completes, you spin up the inertia processor and hand it the initial velocities and then you run a timer until the decelerations complete.

However, I didn’t set about this because other people have already walked this path and done a better job of it than me so…

Libraries Based on the Surface Sample for Silverlight

There’s a couple of libraries that are based on top of this sample so I thought it’d be worth exploring those. Here’s 2;

Touch – a CodePlex project that gives you Silverlight gesture support and, not only that, supports versions of WPF prior to version 4 in order that you can be consistent across WPF3,4 and Silverlight 4 in the way in which you get touch support.

There’s another library on CodePlex as well which (AFAIK) isn’t based on this sample. This is the MIRIA project which I haven’t taken a look at at this point.

Multi-Touch Behavior for Blend

This comes from the mighty trio of Davide, Laurent and David and it’s based on the Surface sample so you need that assembly as well.

I got a bit confused in that I downloaded from here but that only seemed to give me bits for the Windows Phone 7 so I took the source code from here and built it myself instead. That seemed to work out fine.

I found this pretty easy to use. Making the simplest of UI’s like this one;

then, without code, that pretty much gives me an object that I can drag around on screen. A couple of things I found here though that I’m not 100% sure of;

I seemed to have a better experience here using a Canvas than a Grid – not sure if there’s something Canvas specific in the behavior.

Widths and Heights seemed to get changed by the behavior – I found that my UI elements seemed to always end up being square rather than the widths/heights I’d set them to.

I daresay I was doing something wrong but even with those caveats it works a whole lot better than my sketchy behavior and there’s no code to write;

Touch

This project also makes use of the Surface Sample implementation of System.Windows.Input.Manipulations and offers up some helper methods, extension methods and 3 behaviors – AutoClipBehavior, TouchScrollBehavior and TranslateRotateScaleBehavior.

I went with the latter and dropped the behavior onto another one of these simple UIs containing just an image as in;

and that pretty much got me rotate, scale, translate gestures fairly easily. It’s worth pointing out that if I don’t use a Canvas as the parent here then I get an error from the behavior itself;

and so it’s only going to work for me with a Canvas parent.

Silverlight for Windows Phone 7 & Manipulations/Gestures

Unlike on the desktop, Silverlight for Windows Phone 7 does have built-in support for manipulations. The semi-familiar trioof ManipulationStarted, ManipulationDelta and ManipulationCompleted are present on the UIElement class and so that means that the WPF code that I had previously should work fine on the phone except;

I noticed that the phone version has ManipulationStarted, not ManipulationStarting.

I noticed that the phone version does not need the IsManipulationEnabled property that WPF uses.

I noticed that the ManipulationDelta type on the phone lacks the Rotation property so you can only pick up scale, translate deltas.

I noticed that I seemed to get a lot of zeroes through as values for my scale values at times and so had to filter them out.

but with those minor changes it seems to work “ok” although I think I’d need to do something about the phone’s orientation changing but with another one of those trivial UI’s;

Note that the XNA framework also has support for touch on the Windows Phone 7 and, from what I can see, the GestureListener here is using that in order to build its support (otherwise, I’d be asking why this GestureListener isn’t present in the full version of Silverlight).

Summarising

What if I wanted to do something dead simple? What if I wanted to have a UI that displayed 2 photos and when I did a simple drag gesture across the screen it switched to the other photo?

And what if I wanted to do that in WPF, Silverlight on the desktop and Silverlight on the Windows Phone 7?

What’s the path of least resistance? I’ll put that in another post because this one has got too long…