Over the years, traditional menu design practices have developed along the lines made possible by hardware. Unfortunately, as a result, many of these practices don’t apply to apps built for natural user interfaces (NUIs) like the Leap Motion Controller – so that creating great menus is an ongoing challenge for Leap Motion developers. Recently, by experimenting with some alternative approaches, we’ve managed to push past these growing pains and overcome some of these hurdles in menu design.

Why menu design in particular? It’s rarely the most interesting thing about an application, but accessible and usable interface design is absolutely critical to creating an app that people will love. Developing for an entirely new type of interface means that we’re faced with a veritable wild west – as full of possibilities for growth as it is with enticing pitfalls and hidden vipers.

We recently addressed a number of best practices for menu design in our documentation. These guidelines will continue to evolve in the hope that they’ll help jumpstart your own menu designs, help you avoid some of the common pitfalls we’ve seen (and occasionally fallen into), and suggest some ways you can help navigate the natural user interface wilderness – which is pretty exciting.

While designing a new application, we burned through a bunch of different designs for some very simple 2D menus and structures. We only had a short time for this particular round of prototyping, so we focused on single-finger interactions. Ultimately, we ended up focusing on one menu, which we call a marching menu. Afterwards, we wanted to share some of what we learned during the process. Keep in mind that these are super-rapid prototypes – there’s no aesthetic treatment here, just enough to test the usability of the concepts.

Want to try it out yourself? You can download our Unity3D project (much cleaned up from our prototyping) with the marching menu prototype on BitBucket.

The final marching menu wireframe prototype.

When testing our menu prototypes, we’ve found it useful to couch our analysis in terms of three stages of menu interaction:

Activation: how do I bring up the menu?

Navigation: how do I find the choice I want?

Selection: how do I confirm the choice I want?

To throw a wrench into things right off, we’re going to look at these in reverse order.

Selection: how to confirm the choice I want

To tap or not to tap? That is the question. Now completely intuitive thanks to touchscreens, the screen tap gesture is an obvious first choice for Leap Motion apps. The user is using their fingers just like on a touchscreen!

In reality, screen tap actions for Leap Motion are often passable, but not great. Users don’t get any tactile feedback. It’s hard to provide sense of visual depth, so screen tapping tends to require a lot of uncomfortable extended pointing. Because of these reasons, users are not often acutely aware of where their hand lies in Z-space.

Users aren’t generally accustomed to paying attention to the Z-axis.

Plus, since human joints move in arcs rather than straight lines, it’s often hard for someone to perform a Z-space translation while maintaining their X,Y location. Needless to say, this is problematic when they need to move forward and stay centered on a button. You can do a good job following our menu design guidelines (which include some great live JavaScript code examples with LeapJS), but the screen tap gesture isn’t always the best way for users make a selection.

So how can we do better?

X,Y translation with boundary crossing can be better

An X,Y boundary crossing.

While developing these prototypes, we worked on the hypothesis that crossing a boundary in the X,Y plane would be more comfortable and useable than tapping in the air. Users are much more attuned to where their hands are relative to the plane of the screen, and it’s quite easy to give clear X,Y feedback on a 2D display. This approach also allows for menu designs that reflect what users are used to in traditional applications.

Set selection actions apart

We discovered that users are easily confused between when a selection leads to a submenu vs. an actual decision. To resolve this, we created very different actions to distinguish navigational selection from item selection. In our marching menu prototype, we used hover (with a tiny delay) to bring up submenus, and an X,Y boundary crossing for item selection.

Note the different feedback mechanisms in play between selecting a sub-menu (hover, left side) and selecting an option (translation, right side).

Navigation: how to find the choice I want

Navigation design depends a lot on your content and its structure. Naturally, many traditional usability best practices are important, including sorting your content, laying it out well, and making it easy to access commonly used items.

Good menu navigation principles

users can assess their options quickly

provide good breadcrumbs so that users know where they are, and how they got there

make it quick to navigate

make it easy to back out or undo a mistaken action (while crucial, this principle is often overlooked)

This menu layout makes it clear where you are in the structure, and how to reach other options.

Choose the right format for your content

Basic radial menus are great for small data sets and icons, but no good for quickly scanning text. In the Western world at least, users read top-to-bottom and left-to-right. If your users will need to scan through their options easily, your layout should account for this. As you can see, even with a small number of items, the left menu below is much easier to scan than the right:

Definitely consider readability when laying out your content.

3D Geometry uses a menu very much like our marching menu prototype to good effect, allowing users to quickly and intuitively access a wide variety of polyhedra.

However, for the proper set of content, radial menus can be perfect:

Brush Size lends itself almost perfectly to a radial menu in Sculpting, because it represents a spectrum rather than a set of discrete options. You can twirl your finger around the menu and push out the option you want to select.

Give dynamic feedback about the actions your interface is tracking

In an early prototype of the marching menu, we used static icons to inform users what sorts of actions they could perform. However, users often didn’t associate these static icons with actions. Some people tried tapping, others pinched, and others were just plain confused. When users did make a selection, sometimes they didn’t know how they’d done it.

Original static feedback. Users didn’t associate the icons with actions.

For the next iteration, we added some simple feedback, emphasizing the X-axis location of the user’s finger relative to the currently selected button. In testing, users picked up on what they were supposed to do quite quickly, but were often uncomfortable flinging their finger off a menu item to select it. There was no indication of what would happen if they did.

With a little added polish though, the new users we tested with understood and started using the menus with ease. We saw a real marked improvement from some very small changes – design elements that reveal the menu’s behavioral structure at a glance. This sort of basic feedback is great for all manner of menus.

Adding immediate hover feedback and X-axis feedback added to the usability of the menus. Instead of being forced to remember how each level works, the menu actively informs users as they navigate through it.

Allthecooks Recipes uses this sort of dynamic feedback well – making it clear which element is being activated and which actions you can take. Read more about Lucas Goraieb’s work on AlltheCooks Recpies’ menu system in his post Pushing Boundaries.

Be aware of momentum

One of the challenges with using X,Y translation for selection and navigation is the user moving too far and accidentally jumping to the wrong item. If the user has to change their direction of movement – or has a large, obvious space to pause – this will greatly reduce the chances of accidental selections, cancellations, and the like. Giving users insights into what to expect next can help them adjust how quickly they’re moving, while forcing a directional change leaves room for natural motions rather than more stressful, constrained ones.

In the marching menu, changes in momentum flow naturally from the menu structure and obvious stopping points.

Activation: how to bring up the menu

Deciding how your menu will activate depends a lot on your application. The biggest thing to pay attention to is making sure that users understand the effects of their actions within your app.

In our testing, we tried to activate a contextual menu by having the user pull their hand back towards them. We often saw people accidentally activate the menu, but rather than discover the action, they didn’t realize how they’d turned the menu on! Even worse, they’d often think that something else had caused the menu to activate, teaching them the wrong thing.

We also had some menus which, when activated, took up a large portion of the screen. Without proper transitions, users would often become disoriented. With larger popups, you need to strike a balance between keeping a large interaction space and maintaining a sense of place in your application.

We’d have loved to have spent even more time playing with our concepts – and we will – but naturally we only had so much time for this round of testing. Here are some ideas we’d love to test more, or see from design-minded developers:

A multi-level radial menu. Just imagine a radial menu that can handle many levels of data while still having good bread-crumbing, good ways of backing out to select other information, and an obvious difference between selecting sub-menus and making a final selection. We got very excited about how quickly power users could navigate with tiny gestures, but a lack of usability stymied our prototype. A random thought on this to get you thinking – rotary phones?

A gesture for menu activation. The one “gesture” we played with to activate our menu was pulling a hand back towards the user in z-space. This didn’t work so well. We’d love to see developers explore this one more. The biggest challenges here are a gesture that is unique enough not to accidentally occur during normal use, while also easy enough to be learnable and reliably activated. We’ve recently seen a few really cool ones around the office (especially two handed gestures) but there’s a lot of cool stuff to explore on this topic.

A depth-based solution to multi-level menus. The idea that deeper levels sit behind top-level content makes a lot of sense, and reaching in to pull submenus out is an enticing possibility. We’re only just getting started.

We’d love to hear your ideas around interaction designs for the Leap Motion Controller. What are some other ways to think differently about accessing options, browsing content, or creating something new? Post your thoughts in our thread on the community forums.