File size

File size

File size

File size

366.0 MB

For many, Natural User Interface (NUI) is a new term, but over the past few years it's become an incredibly important one to Microsoft. If you want to know what we're thinking when it comes to computing in the future, you have to know NUI. So we go right
to the source speaking to Principle Researcher
Bill Buxton, author of Sketching User Experiences.

Occasionally I read a comment in articles about Windows 7 Multitouch where a hater says "I'm not going to hold my arms up to my screen all day, I'd be sore", we know that. But I can't tell you how nice it is to be in a meeting, see an email come up and being
able to touch it to open and flick to scroll through the pages. The idea is that we want to give users the best modality of UI for any given interaction.

So watch as I quiz Bill on where NUI fits and what it means going forward.

A couple of points first, I’m not a “hater” of Win7 multitouch (must have hit a nerve there), but as Bill himself says, until you can understand what could be (and can be) wrong (invalid scenario usages, etc.) with a product, how can you get it right? I
want touch to be a simpler, two way extension of communication with the computer. Multitouch is a great start, but it needs context. Like taking the step closer to someone changes the context of the interaction. (Side point: my hands were made to work towards
each other, not down to a keyboard/mouse or out towards a screen) Second, I said that you would need “painters shoulders” if you are required to have your arms up and out in front of you with touch screens. (http://channel9.msdn.com/posts/LarryLarsen/A-Look-Behind-Mouse-20/)

Bill makes an excellent point about reusing known or acquired skills. I agree that we shouldn’t limit touch to finger usage only as we’ve developed extensive skills with using the pen/pencil (stylus type) tools. These should be taken advantage of, rather
than throw them out the new product door. I’ve often wished I could sketch out UI design, case scenarios, UML design, mind maps, etc. straight to the computer rather than having to manually (and awkwardly/un-naturally) transfer to the computer via mouse and
keyboard. That’s why I like the concept of blend, but even the designers say the ability to draw would be handy. Right click for the digital pen.

One thing I’d disagree with Bill is voice usage in motor vehicles. From the studies I’ve read and the first hand observations seen, once you get the brain to start the process of speech, the ability to focus on driving drops like you are under the influence
of alcohol.

I would love to see the design of the software that can recognise a device to provide help. Angles, light conditions, shapes, sizes, distance… That fuzzy logic could be very useful to a lot of future projects.

Great interview! "Sketching User Experiences" (guys it's "Experiences" not "Interfaces") was one of my favorite books from last year. It had a similar wake-up / call-to-action effect on me that I got the first time I read the .NET Framework Design
Guidelines. I'd recommend Bill's book to anyone.

I purchased it awhile ago after seeing Bill's
keynote at MIX. The keynote was a nicely polished and exciting product, while the
book goes all over the place. Although I enjoyed the obsession circling around the
orange juice machines and Bill's ability to trash Adobe while trying to make it seem something of a side note, I cannot say that I have taken anything tangible away from the
material. If I were to produce such a book, I would have printed one with many white pages intermixed with instructions to draw something on them.

The beauty of design cannot be fully expressed by the use of unbounded creativity. It requires hard constraints that need to be addressed with due diligence. If you want to see this in action, look no further than our own fossil record. Nature provides
great solutions to difficult problems and occasionally some awesome
mistakes.

Yes, "painter's shoulders" is an issue. If we are working on a kiosk or a whiteboard, things may be fine. In the former, the interaction will likely be brief, and in the latter, we can get close enough that the arms need not be extended out so far as
to cause excess fatigue. On the other hand, when sitting, and reaching over a keyboard, desk, etc. to a vertical monitor, we can generally expect less use and more fatigue, due to extent of reach. There is a reason that drafting tables were in a sloping
desk rather than vertical wall format!

As for human skill, the interesting thing is that the essence of skill acquisition (and hence skill utilization and transfer) is the notion of "chunking". (See the following if you want more background:
http://billbuxton.com/chunking.html). This prompts me to go beyond wondering if/when I should use touch or stylus, and rather - recognizing that I have two hands, and can use them together - consider when it
is best to use them together: for example, touch in my non-dominant hand, and stylus in my dominant one. This reflects what we do in the everyday physical world, and provides a mechanism to "chunk" together low level tasks into a higher level one, using
an existing skill. We have to stop looking at technologies as being in competition.

As for operating devices while driving, I don't think that we have a disagreement - however, your comment suggests to me that I should make the point more strongly than I did. You will note that in the interview I did say minimizing cognitive interferance
as well as visual and manual. As well, I qualified things by saying something, "to the extent that it can be done safely." But you are right, while having the perfect voice only phone (let's pretend that there could be such a thing) sitting on the passenger
seat, and speaking to a remote person using it, it is definately NOT the same as having the same conversation with a live person sitting in the same seat under the same conditions. The real person has cose to the same ability to perceive the world and context
in which you are driving as you, so knows why you go silent, or when they need to shut up for a moment. Hence, the driver's ability to flow in and out of the conversation with minimal distraction/overhead, is much better than it would be with the person on
the phone, who as no idea that something was about to happen that really needed your full attention. The reality is, that there are a large number of things that already distract us from our primary task of driving (tuning the radio, adjusting the heat, sipping
on a soda, listening to an interview on the radio, thinking about your spouse ...). So the real question is, where do we draw the line? The related question is this: with an understanding of the issues above, by worthy design, how low
can we get the interference of a remote conversation? Can we get it below the threshold of acceptable risk? For example, would augmenting the audio with video that gives the remote person a better sence of context help? I'm not advocating a design,
or even suggesting that there is an acceptable solution. The fact is, unless I ask such questions, I can't answer the question. But you are absolutely right: to even begin, we have to be aware of the issue, or we won't begin asking them. What I do know
is that it is certainly unacceptable to text, read email, search for an address, etc. by any combination of voice and touch while driving. We both know that. Thanks for the reminder.

I have no issues with your comments about my book - except for one: in what way did I "trash Adobe"? Why would I? They are a great company - one very much worthy of respect. That is why I used them as an example. That is the same reason that I used
Apple. The message in the Adobe example was simple: even outstanding and successful software companies have a poor record of developing new products in-house, and rely upon mergers and acquisitions for their growth. Another reason that I used Adobe was that
their data was available, and in a form that readers could easilly check. Finally, the argument that the example was intended to support was that
if companies invested in a design process, then in-house development of new (as opposed to N+1) products would become another viable option for growth. If there was criticism implied in the example, it applied to the software industry as a whole,
not to any one company. I thought that I made that clear. If not, I apologize.

The great thing about stories is that everyone takes away something different. From the Adobe story, the one line that stood out was on the bottom of page 70. "But can you rely upon mergers and acquisitions as your sole strategy for growth? Not in any
company that I want to manage, work for, or invest in." It seems to me that you nailed Adobe's corporate strategy right on the head. In nowhere else in the book did you write about investment. For Apple you did write about their terrible puck mouse, and
with Dell's unfortunate foray into the MP3 player market. When you ventured out of the design world and into the world of finance, specifically dealing with the future valuation of a partner or competitor, I found it worth noting.

Perhaps "trash" was poor word choice. However, I would not have considered an antonym as an alternative. There isn’t a designer alive that doesn’t have a love-hate relationship with Adobe.

-Josh

PS. On the write-up on "chunking", has the use pervasive use of the copy and paste metaphor, over the last few decades, altered your perception of the burden imposed by un-chunking the move operation? It seems to me that many of these micro-ops are in
some way compiled by our brain, in such a way, as to hide the details involved. For example typing, I do not think of the letters my fingers are pressing. I just think of the words and my brain takes care of the rest. The mind also does this when you read.

"Aoccdrnig to rsceearh at an elingsh uinervtisy, it deosn't mttaer in waht oredr the ltteers in a wrod are, the olny iprmoatnt tihng is taht the frist and lsat ltteer is in the rghit pclae. The rset can be a toatl mses and you can sitll raed it
wouthit graet porbelm. Tihs is bcuseae we do not raed ervey lteter by itslef but the wrod as a wlohe."

Bill Buxton is an inspiration, and a very wise man. I think it is important to ensure that applications and interfaces allow for flexibility and multi-modal interaction. This is something that Bill Buxton has worked towards over his career. It is not
surprising that he has a background in music!

Good question on chunking w.r.t. cut/copy & paste. It helps tease out some interesting issues. The interesting thing about human skill is that we can - through practice - transform from attentive problem solving behaviour to automatic skilled behaviour.
Once acquired, we can do remarkable things. The question then reduces to, "At what cost?" So let's look at the basic action, MOVE as an example. As we all know, there are at least 2 ways to do this with a GUI. (1)
cut-and-paste, and drag-and-drop. From the perspective of skill acquisition, a key challenge with cut-and-paste lies in the concept of the clip-board. It s this magic place which is (for most) invisible. In contrast, with drag-and-drop,
the whole transaction is visible and there is a real-world metaphor. I think that it is pretty clear that the former is harder to learn and more prone to error for beginners. However, with practice, cut-and-paste and the clipboard will become a highly learned
skill, and therefore be automatic, not require concious attention during execution. That is, when a highly learned skill, it will impose no higher demand on the end user than drag-and-drop. While both enable one to effect a MOVE, the two techniques have important
differences. The prevue capability of drag-and-drop makes it especially approrpriate for some graphical tasks, for example. But, there is also confusion whether drag-and-drop means move (yes, if on same disk) or copy (if from one disk to another) - a conceptual "bump"
that will hit users, frequently after they think that they "get it". Cut-and-paste is less visible, but having grokked the concept of the clipboard, one can exploit the skill to effectively "move" something from one location, to multiple locations. That
is, the added complexity of the clipboard means that the move operation can be extended to multiple pastes.

A quick summary up to here: (1) even otherwise disconnected actions can chunk together to form a skill; (2) the UI design needs to consider impact on, and find a balance betwen user's acquisition of skill as well as performance once skill is acquired; (3)
different ways to do the same thing, such as move, differ in the range of their semantics, and not just in how they are articulated.

A key point of my chunking & phrasing paper is to say that through what I called "kinesthetic connectivity" we can accelerate the aggregation of sub-tasks into the desired skill. However, that does not neccessarily rule out being able to still take advantage
of the individual atomic skills that chunk to constitute the higher order aggregate skill.
Paste as a component of the higher level skill Move in the cut-and-paste-with-clipboard method, is an example of this.

So far so good. But up to now, I may seem to be avoiding your question, and stating the obvious. So let's behave like designers and see how this approach to thinking about things might shed light on other ways to do things. Can we augment current techniques,
and/or find new and better techniques?

In our work in my old Input Research Group at the University of Toronto, we (students Garry Hardock and Gord Kurtenbach and myself) looked at this very example. So the one thing that leaps out from my description above, is the initial problem of the invisibility
of the clipboard. Not only are its contents visibile, rarely is there any effective explicit visible representation of the clipboard itself - something rare in the GUI. On the other hand, there is the problem disadvantage of the drag-and-drop metaphor, in
that it does not extend to multiple pastes. Finally, as anyone who uses a stylus or touch for input (as with a Tablet-PC), cut-and-paste becomes a real pain without CTL-x and CTL-v, for example.

So, can we use some of our thinking about chunking, visibility, and richness (extensibility of Move to multiple paste, for example) to come up with a new design? One that might do a reasonable job of getting the best of both worlds, and be compatible with
emerging pen and touch interfaces at the same time?

So, the approach that we came up with - significantly, by folowong this train of thought - was as follows:

- begin by using the traditional proof-reader's approach of circle and extend line to destination.

- create a temporary buffer (i.e., clipboard) by drawing a symbol in the margin of the page

- let that buffer be a legal destination to which something can be moved.

- to paste the buffer contents, draw it in the margin draw a line desired destination

- this enables moving things across page boundaries

- this enables multiple pastes

- by drawing different symbols, one has clear way of havng multiple self-defined buffers/clipboards

- by the symbols, one has a visual representation of clipboard(s)

- one could, for example, then double click on symbol to examine contents (which is consistent with GUI)

The key thing to note in the example is that it is builds upon skills that most users already had acquired, from a life-time of living in the everyday world, prior to ever using a computer. Do these techniques render the existing GUI ones obsolete? Not
at all. But what the example does do, I hope, is help illustrate the complexity of even seemingly simple transactions, and how some of these theories can aid us in thinking about them, and generating designs that may address some of our emerging UX challenges.

Sorry if I have run on a bit, and especially sorry if I missed your point. But I tries

Remove this comment

Remove this thread

Comments Closed

Comments have been closed since this content was published more than 30 days ago, but if you'd like to continue the conversation,
please create a new thread in our Forums, or
Contact Us and let us know.