GNOME Shell Usability Test Plan

The need to test GNOME Shell

For the past couple of weeks I’ve been working with Jon McCann, Jeremy Perry, and Owen Taylor on developing a usability testing plan for GNOME Shell. It’s a work-in-progress, and I wanted to make a quick posting about the effort and where it’s going.

Now as you may have learned from earlier postson variousblog planets, or in Marina Zhurakhinskaya’s GNOME Journal article last November, GNOME Shell defines the user experience for GNOME 3, to address the more fully-networked computing environment of today as well as to reach a wider audience. A great change has occurred in the way technology affects people’s lives since the original GNOME 2 was designed, including the introduction of new and disruptive technologies such as micro-blogging, large-scale social networking, and the increasingly predominant availability and usage of web-based mobile phones. With so many streams of content and information we need to manage, and it’s hard to keep focused and on task on the desktop. GNOME Shell is designed to account for these changes, to make desktop computing delightful and comfortable in spite of the amount of information we need to process in the course of our work on the computer.

The GNOME Shell design is based upon a few assumptions about users’ desktop actions. While many of these assumptions are informed by seasoned, long-time GNOME hackers and designers working on it such as Owen and Jon, it’s important to define those assumptions and test them to make sure that they are valid. We wouldn’t want GNOME 3 and GNOME Shell to be built upon a shaky foundation. GNOME Shell is finally at the point where it’s stable and we’ve got enough functionality in place to run some tests on it.

Figuring out a usability testing methodology

‘So, Mo,’ you might say, ‘why don’t you do some usability tests of GNOME Shell with your swanky usability lab kit, like you did for Fedora Community?’ Well, Fedora Community is an application that, at this point in time, focuses quite heavily on supporting a particular domain: make Fedora package maintainership easier. To test how successfully Fedora Community does this, sure, you do usability testing to see how it goes when folks using the application try to complete tasks related to package maintainership. GNOME Shell, however, isn’t an application. It’s an environment in which a wide spectrum of users are meant to complete tasks within any number of different domains – within a variety of different applications. If I can’t successfully write a book report on To Kill A Mockingbird while using GNOME Shell, for example, how much of that is related to GNOME Shell, how much is related to my word processor, how much of that is related to my internet browser? Or, to analogize, if I built a highway, how helpful would it be to make sure I can make a trip to Grandma’s house, the supermarket, and to the local national park using it?

What are the considerations you should make when building a highway?

What speeds is the highway meant to support? Make sure turns and ramps are banked appropriately to support these speeds, make sure the angles in the turns are safe to make at those speeds, and that the materials used to build the road can withstand that level of usage.

Under what weather conditions might the highway be under? (If it’s in New England, think snow and ice! In Oahu, not so much.) Can the chosen material withstand the weather conditions?

How is the terrain along the planned route? Can the path of the highway be routed such that it avoids dangerous or difficult-to-navigate paths such as along a cliff face?

Can testing the trip to Grandma’s house on the highway expose issues in the design decisions made in building the highway? Maybe if it happens to snow during the trip, you might uncover some issues, but it would really be based on chance. Would you feel safe on a highway tested based on the success of arbitrary trips along it, or would you rather it be tested in a fashion targeted to expose any issues that might be present in the design decisions made in its construction?

Usability testing GNOME Shell to see how well folks can download and listen to music and send emails to their Grandpa I think is like testing a highway’s safety based a trip to Grandma’s house: you might get lucky and incidentally happen to uncover a flaw in the design, but more likely you’re going to cover issues that aren’t directly related to the design (the car I’m driving isn’t comfortable to ride in!) My colleague Ben, who is a seasoned usability professional with over 15 years’ experience, suggested a methodology where we make a catalogue of various design assumptions followed in GNOME Shell’s design thus far and treat them as research hypotheses we could construct targeted tests for. For example, if we assume users find it easier to search for documents than browse for them for a file tree, we could construct the following test:

Assign a user the following task: “You’re trying to create a print-out calendar for next month to plan your schedule. Download the OpenOffice template for a calendar available at some URL. Now, open up the calendar file.”

Observe whether or not the user browses the file hierarchy to open the document up after saving it. How long does it take them to find it when asked to find it? How many clicks? Do they seem annoyed?

If the user browsed for the file, ask them to search for it. If they searched for it, ask them to browse for it. Again, note how long it took them to find the file – how many clicks – and their general mood.

Present the user with a quick, 3-4 Likert scale-based questionnaire to assess which method they preferred and how they felt about each method.

This way, the tester does not need to rely on chance that the user is going to open a file on the file system and can be prepared for when they do so to observe in a manner focused on a particular design hypotheses. In this example, the tester can be focused on whether or not the user finds searching or browsing more comfortable, so they can filter out other design issues that might just happen to crop out in the process in order to focus on running the test.

This, of course, is not to say more scenario-based and task-oriented usability studies are not useful. I just want a test method that will target specific design assumptions so I can run a series of tests on that assumption to help decision-making, rather than test various tasks and hope some issues related to the particular design assumptions I’m interested at the time to come up. It will be important to do scenario-based and task-oriented usability studies to complement this work however, especially to identify potentially problematic design assumptions we didn’t think to test.

Ideally, we’d do a longitudinal study so we could better get at how users interact with GNOME Shell over time and after they’ve become familiar with it. These types of studies necessarily take a long time, however, and a lot of investment, so we want to make sure GNOME Shell is ready before we get into that. I think we probably need more confidence in the hypotheses we’ve set out to test first, so we know we won’t be wasting our test subjects’ and our own time with a longitudinal study at this point.

The current test plan status – and how you can help

Right now, with the helpof otherfolks, I’ve written up 36 design assumptions / hypotheses to test. The next steps are to figure out some test plans for them. Since testing 36 hypotheses is no small feat, I think first I’ll pick the top ten in priority and develop hypotheses for them first. I’ll be working on that next.

The first cut of the test plan is on the live.gnome.org wiki with those 36 hypotheses. I would love to hear what you think about them and if you have ideas for other GNOME Shell design assumptions you’d like to see tested. I would also love to hear your ideas on how we might devise some tests for each hypotheses. I’ve set up a page for commentary on the wiki so please feel free to add your comments and suggestions! (Although of course you can feel free to leave your feedback in the comments area of this blog post instead.)

Rate this:

Share this:

Like this:

LikeLoading...

Related

About Máirín Duffy

Máirín is a principal interaction designer at Red Hat. She is passionate about software freedom and free & open source tools, particularly in the creative domain: her favorite application is Inkscape. You can read more from Máirín on her blog at blog.linuxgrrl.com.

Discussion

112 thoughts on “GNOME Shell Usability Test Plan”

The problem is that I have tried to use the version from the Ubuntu 10.04(testing) repo and it will not run on my box.

This is a Dell XPS420 with the Q6600 Core2 Quad, 3Gb ram, Radeon HP2400 Pro. The last seems to be the problem. The bios were installed on this box in March 2008. This is not, to me, and old box.

I have one install of 10.04 with the 2.6.33-999 kernel and xorg.edgers ppa. This helps in that, unlike my other, more standard, installs of 10.04, it does not freeze so hard that the only option is to unplug the box. I can actually get a tty prompt and shut down and Alt+ SysRq+b will work too.

That install also supports compiz to the point that there is some functionality there. Not complete but there. Any other install just freezes. This does not bother me because I don’t like the effects and do not want them. I would like to be able to use gnome an gnome shell is to be gnome so this concerns me.

I asked one of the driver developers about your system. We don’t know about Ubuntu but this system should be capable of running GNOME Shell on Fedora 12 if you install the mesa-dri-drivers-experimental package. And will work out of the box in Fedora 13. Hope that helps.

Having 3 panel entries like: “Applications”, “Places” and “Workspaces” instead of “Activities” would radically simplify the UI, allowing for removal of the “popup bubbles” like the one in the screenshot. Just show one thing at the time. Trying to put everything in one screen is a bad idea. It distracts users by showing unneeded things: a user launching an app doesnot want to see workspaces; and a user swithing workspace does not want to see apps. Also it won’t scale with other concepts that may emerge in the future.

My assumption is that users need simple and elegant layouts in order to stay focused and enjoy a UI. As it is, the Activities screen tries to do too many things at once. For example, when choosing an application to launch they want to scan a simple list (or grid) of apps. Making them work on a small section of the screen while displaying unnecessary elements that steal screen estate like “Locations” is just confusing and less than optimal.

On the other hand, putting everything in one screen allows you to defer the
decision to resume an activity or start a new one until you can see everything that is already open and see what all the options are. So you don’t have to remember if you have the application running or the document open and make a decision about which panel option to pick up-front.

So it seems there’s another task in there too. The statement we came up with focuses on users trying to launch an application, but what about users who just came to a break in their work – or maybe they just came back from lunch, and they are trying to decide what to do next. To show their work across the buckets of ‘application’ ‘document’ ‘place’ would serve those folks well.

* Zooming in and out of Activities is not disorienting (i.e. does not cause a noticeably longer delay before the user can act on what he sees, compared to having no animation)
* Switching windows doesn’t take noticeably longer using Activities instead of a taskbar-style window-button-list.
* People will not feel uneasy about having no representation of minimized windows or windows on other workspaces on screen.

So if I use micro-blogging only marginally, don’t really care about social networks and do not have a data plan on my mobile phone, then GNOME 3 is not for me? This is why I hated so badly GNOME Shell in F12?

About the usability, and I talk again about my experience with it from F12, when for a basic task I do *all the time* like switching from one running application to another I have to click 3 times and watch 2 animation, then it is a total failure.

And how do you think you can minimize the need of switching between running apps? Right now I use this browser window, a video editor, a terminal, chat, IM and mail client. Doing stuff in all of those windows, some are minimized and some overlapping.

If a new message arrives you get notified and can then decide if you want to switch to the app via the icon at the bottom or if you want to reply to an instant message directly via a popup textfield, for example.

I agree with Nicu. As it is now, to me the Gnome-Shell is only usable as a fancy workspace switcher. I find that zooming out the whole desktop just to open the applications menu is way too obtrusive and very distracting. Another problem: minimized windows are invisible!! Again, just to see them you have to go to the same crazy animation. What’s wrong with a task bar? I am a bit worried about the direction Gnome seems to be taking… all this integration with social networking stuff, shouldn’t that be left to applications?

I think because of some of the negative opinions expressed about the animation, that the mechanics / physics of it could be tweaked to be a little less noticeable.

I will tell you the first time I used GNOME Shell – the animation has been tweaked since then – I felt like I was being punched in the face when I went to the Activities Overview.
But last night, I noticed when I switch between applications in my Playstation 3, a very similar although gentler / less noticeable zooming animation occurs.

So I am not sure that there is an animation is the problem – it may be that we need to test different permutations of the animation to discover what is most comfortable and less-distracting for users.

I will tell you what I think is wrong with the task bar – you’ll find this design assumption in the test plan if you read through it:

“A persistent window list is distracting to the user. A user will be less distracted if the window list is not always visible on the screen. ”

I actually found the fact that the task list was not there, after I got over the anxiety of it not being there when I was so used to it – a real breath of fresh air. I feel less stressed and less ‘busy’ when I don’t have 30 little rectangles, some of which are blinking, down there constantly in my face. I don’t know how widely this is going to be felt by users though, which is why I constructed a design assumption to test it and added it to the usability test plan.

Can you come up with a reasonable design assumption that forms the basis of your opinion on the matter? Maybe we can add it to the test plan?

yeah, I noticed in the wiki the assumption about “A persistent window list is distracting to the user. A user will be less distracted if the window list is not always visible on the screen” and I disagree with it, I like my window list in the bottom panel and find it really useful for fast switching apps. Years ago I experimented with auto-hiding the panel but didn’t liked that, I have a 20″ display and the panel is only 24px, not something distracting.

Mairin, I read your assumptions, and I think I don’t agree with most of them. Unfortunately, I haven’t got the time to comment everything I don’t agree with, just a couple of things:

About, the animation. What is distracting is that it takes you to another screen, another “place”. So, every time you go through it you have to take stock of where you are. As I said, this is alright for workspace switching, because in this case you are actually going to another “place”, but for opening a simple menu it’s unnecessary, unexpected and tiresome, in my opinion.

About the task bar. To me, a persistent task bar is hardly distracting, and I don’t think I could work without one. For example, if I want to open a web page, the first thing I do is look for a minimised Firefox window. It’s easy and fast, all you have to do is visually scan a small area of the screen. It needs no interaction. If I can’t see one, then I click on the Firefox icon on the top panel in order to open a new Firefox window.

Without a task bar, it takes more work to find minimised windows, so I tend to forget them, which is very annoying. It’s easy to end up with two different windows of the same application, both editing the same document. It’s also a waste of resources.

Granted, the task bar can get cluttered, but you can always switch to another workspace if it gets too cluttered.

I think it’s a good sign that you disagree with many of them, because that means they are worth testing :) The one thing I definitely want to avoid in this process in testing things that are obvious and not under contention.

“About, the animation. What is distracting is that it takes you to another screen, another “place”. So, every time you go through it you have to take stock of where you are. As I said, this is alright for workspace switching, because in this case you are actually going to another “place”, but for opening a simple menu it’s unnecessary, unexpected and tiresome, in my opinion.”

I see what you mean. What do you think could be a good design hypothesis to test as the basis of this, then? Earlier, Flavio and I came up with a related hypothesis: ‘Displaying document launching and location launching tools alongside application launching tools will not cause users trying to launch an application to lose focus.’ Can you think of one that hits more on your concern so we can test it and see if it’s valid in practice?

“About the task bar. To me, a persistent task bar is hardly distracting, and I don’t think I could work without one. ”

But for me, it is very distracting. It is like staring at an inbox stacked high of papers of tasks I must do. It took me a while using GNOME Shell before I appreciated not having it around. We both have very logical arguments for opposing sides of whether or not it’s worth having – which is why I think it’s really critical for us to see how it holds up in practice.

“Without a task bar, it takes more work to find minimised windows, so I tend to forget them, which is very annoying. ”

This sounds like another good potential design assumption. What windows do you typically have minimized? I actually usually only minimize windows I’m not actively working in and don’t want to pay attention to, and keep all the windows I’m working on maximized and stacked on top of each other. So for me, it’s a positive that I can forget about the minimize windows until I want to go retrieve them. There are a lot of different styles of windowing like this, though.

What do you think about testing the following assumption, then?: ‘In an environment no running applications have a persistent task list entry in order to open them or bring them to the top of the window stack, users will not forget or lose access to the windows they need to interact with.’ OR ‘In an environment no running applications have a persistent task list entry in order to open them or bring them to the top of the window stack, users will feel less distracted by windows they are not actively working with.’

“Granted, the task bar can get cluttered, but you can always switch to another workspace if it gets too cluttered.”

This sounds like a method of housekeeping that comes straight from the ‘Mo manual of housekeeping’, hehe… shove it all into a closet or under the bed and start ‘fresh’. :)

I would like the help, unfortunately I’m don’t know what a design hypothesis is :)

I will try to explain how I manage windows. In my experience, there are two types of users, the ones that like to work with a single window at a time, and the ones that work with multiple windows at the same time. I usually work with multiple unmaximised windows, but no more than 3 or 4. The others I keep them minimised in the task bar. I also use the “focus-follows-mouse” mode of Metacity, so that I can type things in a window that is partially covered by another one without bringing it up to the front. Left-clicking on the title bar brings the window to the foreground, and middle-clicking on the title bar puts the window to the back. The point is that if you work this way, it’s crucial to see what windows are minimised, because you are constantly minimising and unminimising windows. Also, in this situation it’s not unusual that you unminimise the wrong window every now and then. If this happens, clicking again on the task bar -without moving the pointer- just minimises the window back. So it’s very handy. Applications that run in the background, such as music players and the like, I don’t like them in the task bar, instead I have them in the notification area.

Now, if you tell me what does “testing an assumption” mean, maybe I can answer your questions… Does it mean whether I agree or disagree with that particular assumption??

“We both have very logical arguments for opposing sides of whether or not it’s worth having – which is why I think it’s really critical for us to see how it holds up in practice.”

That would be why I suspect it would be helpful to have whatever behaviour is decided upon optional in some way. Apparently one size does not fit all, and what if usability study finds that 60% of users really like working with way A and really find B causes them problems, while for 40% it is the other way around?
Sane defaults are a good idea, but unless one can find some admirable synthesis there are things you will want to let people decide–ideally making it fairly easy to do and obvious that they can decide.

I’d suggest also that one design assumption (not a usability assumption as such, perhaps one step before that) being made is roughly
“The majority of users make extensive use of ‘new and disruptive technologies such as micro-blogging, large-scale social networking, and the increasingly predominant availability and usage of web-based mobile phones'”.

If that assumption is not correct, then while making provisions for integrating all that is still important (because certainly *some* users do that stuff, and likely more will in the future), it will also be of critical importance to ensure that the new approach doesn’t get in the way of more traditional desktop users who, while perhaps more boring, still have an equal need to get things done as more innovative users.

“That would be why I suspect it would be helpful to have whatever behaviour is decided upon optional in some way.”

Forcing choice on the users is bad. I think we should only ever consider forcing an option on them if our test results show the users are truly split on the matter. But we should not be diving into solutions yet, we don’t have the data.

““The majority of users make extensive use of ‘new and disruptive technologies such as micro-blogging, large-scale social networking, and the increasingly predominant availability and usage of web-based mobile phones’”.”

Nice try but this is not a design-assumption. In fact, it’s not even a proper sentence. Users don’t use ‘predominant availability’. I know what you’re trying to say though, a better assumption would be,

“A majority of desktop users communicate or would like to communicate with other people via their desktop.”

I really have a hard time believing that is not the case though, and as you point out, out of of those who don’t, likely more will in the future. Even if someone doesn’t use identi.ca or twitter, email use is so pervasive these days it would be extremely difficult to prove the assumption that majority of people don’t use it or don’t want to use it.

Not even a proper sentence? Excuse me? If the remainder of your reply were as false and defensive as that bit, it would be heinous indeed.

The majority of users make extensive use of (X).

That is most certainly a proper sentence. It’s not even a compound sentence. Subject, verb, object. A chunk of prose within quotation marks operates effectively as a blob–the precise grammatical content within isn’t particularly relevant as long as the whole can be considered workable as an object to the sentence. The (X) in this case was quoted from your article–it wouldn’t have been relevant if it were not. It’s a phrase listing “new and disruptive technologies such as A, B, C.” Technologies seem like something users could use, so yes, it’s a perfectly good sentence.
I suppose the final phrase within the quote could be considered to cause difficulty as it mentions “usage of” a technology rather than being a technology itself, but that’s just due to your faulty parallelism.
My degree was in English literature. I don’t think you’re apt to have a positive experience trying to tell me what is and is not a proper sentence. Sheesh (Note: This poster is aware that “Sheesh” is not a sentence).

As to “forcing” people to use options–I’ve heard that sort of thing before, but I’ve never seen what’s so heinous about the existence of options within reason. Surely in any given case one option will be operational by default. For those not annoyed by the default behaviour, the option effectively does not exist–they never need to go looking for it so probably will not find out it even exists. If it’s clear that a use situation exists in which all available defaults will seriously annoy a sizable subset of users, the existence of an option is surely the only sensible response.
So for instance, if one set of people find that *not being able* to tell at a glance what windows they have open is bad for their workflow, while another set of people find that *being able to tell* is bad for theirs, not having an option means somebody’s workflow will be impaired–whichever group is in the minority. What if, for the sake of argument, your group is in the minority as it turns out? Your experience will still be valid. Will you be happy if there is no way for you to get the behaviour you want?

Your rephrasing of my sentence to make an appropriate assumption to test (not really a relevant exercise, as I had said from the outset that’s not really what it was) substantially changes its meaning, in a way that makes it less relevant to your original statements about what drives the GNOME Shell. For instance, email is relevant to your changed version–but email is hardly a new and disruptive technology. To the contrary, its roots are in the original Arpanet. If your objectives encompass making email use more convenient, that’s great, but that was not so much as suggested in the article; if anything quite the contrary.

““The majority of users make extensive use of ‘new and disruptive technologies such as micro-blogging, large-scale social networking, and the increasingly predominant availability and usage of web-based mobile phones’”.”

If you break the sentence down and omit some of the clauses, it contains the following:

“The majority of users make extense use of… the increasingly predominant availability and usage of web-based mobile phones.”

People make use of availability? People make use of usage? I don’t care what you have a degree in, I really don’t – you accuse me of being defensive, yet you feel the need to inform me of your credentials as if they somehow make your poorly-communicated message more understandable.

“What if, for the sake of argument, your group is in the minority as it turns out? Your experience will still be valid. Will you be happy if there is no way for you to get the behaviour you want?”

Why is your behavior in the minority? What is so different about the way you perform tasks that necessitates you do things differently? I would rather spend my time collecting solid, real data rather than postulating on what-if what-if scenarios. We can sit here and worry about all these different problems that might happen, but that would get us nowhere. I think it’s a better plan to actually go out and get the data and discuss THAT.

“Your rephrasing of my sentence to make an appropriate assumption to test (not really a relevant exercise, as I had said from the outset that’s not really what it was) substantially changes its meaning, in a way that makes it less relevant to your original statements about what drives the GNOME Shell.”

Your sentence as originally posed is not testable. Since I am apparently incapable of rendering it to be testable and maintain your meaning, perhaps you’d like to give a go at providing a test plan for it? How would you define and measure each of the points in your hypotheses? I’m guessing you would need come up with a survey to hand users, and create some kind of formula as to how disruptive-technology-linked-in the user was. How do you measure something like that?

“For instance, email is relevant to your changed version–but email is hardly a new and disruptive technology. To the contrary, its roots are in the original Arpanet. If your objectives encompass making email use more convenient, that’s great, but that was not so much as suggested in the article; if anything quite the contrary.”

Please define ‘new and disruptive technology’ for me and why you are so curious about studying it. I personally am concerned about GNOME being able to not only to account for usage of today’s ‘new and disruptive technology’ but also tomorrow’s. The fact of the matter is that GNOME 2 was developed at a time when internet usage was not as widespread as it is today, many Americans did NOT have broadband access, and sites like hulu, facebook, myspace, and twitter did not exist. Streaming video was just not practical back then. The world is very different today. You seem to want to argue that. That argument has no place in this usability test. I remember when email was a new and novel thing when I was in high school. Now it’s everywhere. I remember when it was a new and novel thing for someone to have a website. Now you can pick up a can of coke and see http://www.coke.com on it. Many US state license plates have the URL of the state printed right on the license plate. I’m not even 30 years old, and the world is vastly changed from when I was in high school and college. If you want to argue that, this is not the forum.

I completely disagree. When you offer choices, you are in no way “forcing” choice on anyone. If they like what they see, no choice is to be made, and they are happy with the defaults. If they don’t like what they see, then NOT having choices becomes a huge problem, while having choices becomes a huge relief.

The whole reason I chose Gnome 2 over any other desktop environment was it immense flexibility. I like being able to place any item I like on the panel in order to configure it to my personal workflow. I like being able to completely change functionality on a whim to meet my needs.

As Gnome-shell currently stands, it completely eliminates all the reasons why I currently use Gnome. In Gnome 2’s menu system, I click the section I want, holding down the button while gliding smoothly through the submenus, and release the mouse button when I get to the item I want. Effectively, it’s one mouse click. In Gnome-shell, I have to click Activities, then click the tiny “more” option next to “Applications”, then click through each submenu. This is not an improvement for my workflow, but an undo increase in work needed for a task.

Some might like the clock in the center of the Gnome-shell bar. For me, however, I like having the clock applet far off to the side where it is less obtrusive. Having it as a movable applet means it can be anywhere the user wants it, or not there at all (if, for example, they use a Screenlet clock or other type of clock). Also, I use the calendar function extensively, and having on the side keeps it from interfering with any documents I’m using the calendar to help construct. In the center, the calendar covers parts of the program I’m using.

You mentioned the task list being too distracting. Unfortunately, I use the task list extensively in my workflow as well, since I need to swap quickly between windows and programs many times every minute in some of my work. Not having the task list available means that my workflow and productivity slows considerably. Again, keeping it as an applet means it can be there, or not be there, depending on the user’s needs.

I also like having many custom items available on the top bar so that I can get to them quickly with a single click. Gnome-shell currently eliminates this function. Again, applets and links directly on the bar work wonderfully, but aren’t necessary if the user doesn’t want them.

As for appearance, I personally don’t like Metacity/GTK themes, and use Compiz with Emerald, as it looks much nicer and adds some great functionality (and choice). Gnome-shell also removes this possibility by tightly integrating the window manager, thereby removing another choice from me.

Basically, it all comes down to this: Choice is never a bad thing. I have never seen a situation where having choices available *when needed* is a bad thing, while they are nicely hidden away when not needed. By the same token, having fewer choices IS a bad thing when the defaults reduce productivity, which is inevitably going to happen when you don’t offer choices. It is the reason that I steer clear of Windows.

I think this whole thing could be solved by turning this Gnome-shell idea around a bit. My thinking is this: What if you make Gnome-shell an invisible layer that runs Nautilus and Gnome Panel on top of it. You then make the “Activities” button an applet that resides on the panel, along with everything else currently in Gnome 2. When this Activities button is pressed, Gnome-shell springs into action with its animation (or just a drop-down slab like the Linux Mint menu, depending on the way the user sets the preferences). It takes control of the system until the user decides to return to the normal desktop environment, when Gnome-shell becomes invisible again. This way, you can have your cool new functions, while keeping the ones Gnome users cherish. No sacrifices necessary. And, if the user decides not to use it, then all the old Gnome 2 pieces are still in place in order to return to current functionality. Linux distros can chose which defaults are best for their target users, and the users can change them to fit their needs, if necessary. Everyone’s happy. I really don’t see the need to remake Gnome into some monolithic, option-bare monstrosity when it’s just about perfect as it is, in my opinion.

I do not like the way Gnome-shell is currently going. It is too tightly integrated, too unfriendly to my workflow, and eliminates the customization possibilities that has made Gnome such a Godsend in the past. You know, it reminds me a lot of the way Microsoft does business, and I won’t use that OS, either. I think we’ve come to this point where we’ve refined things down almost to their most efficient point they can be with a mouse/pointer/menu/icon environment, and any new innovations, while maybe neat to look at, are now becoming hindrances more than progress. Just my opinion, obviously.

“As Gnome-shell currently stands, it completely eliminates all the reasons why I currently use Gnome. In Gnome 2’s menu system, I click the section I want, holding down the button while gliding smoothly through the submenus, and release the mouse button when I get to the item I want. Effectively, it’s one mouse click. ”

No it’s not. In HCI-language this is referred to as the ‘steering task’ – and it’s well-studied and documented as being a physically difficult interaction to have with the computer.

“The whole reason I chose Gnome 2 over any other desktop environment was it immense flexibility”

In spirit it sounds like KDE is more for you.

“Some might like the clock in the center of the Gnome-shell bar. For me, however, I like having the clock applet far off to the side where it is less obtrusive.”

When the clock is off to the side, you then spoil the opportunity of using the upper right corner for a more common mouse target. I assume folks adjust their clock far less than they log out of the system or switch users or access their system preferences.

“In Gnome-shell, I have to click Activities, then click the tiny “more” option next to “Applications”, then click through each submenu.”

No, you do not. In fact, you do not have to make a single mouse click. You can hit the windows key on your keyboard and type ‘inkscape’ and hit enter, and inkscape will launch. You can also thrust your mouse to the upper left corner, without clicking, and start typing ‘inkscape’ and hit enter and it launches. Additionally, if you are as much in love with inkscape as i am, you can simply click the inkscape icon in the application well without having to click ‘more’ because if you love inkscape it’ll be there always.

“Having it as a movable applet means it can be anywhere the user wants it, or not there at all (if, for example, they use a Screenlet clock or other type of clock).”

Having it be a moveable applet also means that users can quite easily accidentally move it to a very undesirable place or lose it completely. (ever managed a linux lab for elementary and middle school kids at a community center? I have. just because you can doesn’t mean you should.)

“Also, I use the calendar function extensively, and having on the side keeps it from interfering with any documents I’m using the calendar to help construct. In the center, the calendar covers parts of the program I’m using.”

it blocks 80 pixels at best at the very top of the content in my firefox window – hardly a substantial chunk or information. you must have quite poor resolution or a large-font-size theme or both.

“I also like having many custom items available on the top bar so that I can get to them quickly with a single click. ”

the more target areas on the screen though, the greater opportunity to mistakenly target one. you don’t need everything at a single click, but i believe the tendency is to create more launchers than you really need, so the ‘one click’ comes at the expense of you having to sift through all the other ‘one clicks’ EG in principle ‘one click’ sounds quite sexy, but ‘one click amongst 100 one clicks’ is really not a compelling story and I believe is actually a bad experience.

“You mentioned the task list being too distracting. Unfortunately, I use the task list extensively in my workflow as well, since I need to swap quickly between windows and programs many times every minute in some of my work”

what kind of work are you doing that this is necessary? are you an alt+tab user?

“As for appearance, I personally don’t like Metacity/GTK themes, and use Compiz with Emerald, as it looks much nicer and adds some great functionality (and choice). Gnome-shell also removes this possibility by tightly integrating the window manager, thereby removing another choice from me.”

i don’t use compiz, i can’t really tolerate it. there’s only so much window hula dancing i can stand, and last time i used it, it basically required me to sit through screens of checkboxes and dropdowns to configure it to behave sanely. maybe it’s different today though.

“Choice is never a bad thing.”

This is absolutely a point for debate, as I am quite sure I am not alone in believing that choice can become a very bad thing. I’m certain that some rather smart folks agree with me. (although I believe GNOME historically has been on the pro-simplicity side of the debate, so again I am curious why GNOME is your desktop of ‘choice’ (bwahaha) rather than KDE):

Please read the first link as it explains the position better than I ever could.

“Just my opinion, obviously.”

I see a lot of that in all these comments. It’s quite dismaying to me as I was hoping to have a more productive conversation than just folks throwing their opinions around. Opinions on gnome shell are really worthless to me right now, as they put me in the place of defending it and that really is not my position to take. Rather, I would like opinions on the USABILITY TEST PLAN that this blog post is about. Remember that?

Mairin, while I think it’s great that you are taking your time to address people’s concerns on this blog, I feel like sometimes you are ignoring the fact that there are 3 main types of “hand modes” when a person is on a computer:

mode 1: both hands on keyboard
mode 2: one hand on keyboard, the other on the mouse
mode 3: only mouse*

(* I know some will immediately think “pr0n” here, but I find mode #3 very useful when reclining on my chair browsing the news in the morning, for example.)

In my experience, users do not like to switch between hand modes. It slows them down (or gives them the impression that it does). Therefore, it is desirable for the most common actions to be easily doable from all 3 hand modes.

Going back to your Feburary 23rd post: some of the solutions you offer in your reply are too cumbersom for mode #3.

Solutions like “press Super then type ‘Inkscape'” are great for mode #1, while ones like “hover over the ‘Activities’ corner and type ‘Inkscape'” are great for mode #2. Also “use Alt+Tab” is a great solution for either mode #1 or #2. But what about #3?

I love that Gnome Shell is paying attention to keyboard users in making it *as fast as possible* to use the keyboard to launch apps and switch between open apps. This is a great step forward for Gnome.

But the same level of attention is not being given to the mouse. Launching apps with the mouse takes either a large number of clicks or a large number of motions. Even worse, the list of recently used applications, with its dynamically changing ordering, will leave users second-guessing *where* they should be looking, which will slow them down.

And these are things that are *very* simple with the current Panel. In fact, they are close to *as fast as possible*: to launch an app, click on the launcher on the panel (one motion, one click). To switch to an open app, click on its button on the taskbar (one motion, one click).

I do not claim to know the best solution for any of these problems. I would just like to hear that the Shell developers are aware of the issue and will be working to improve support for all 3 “hand modes”.

I agree with Nicu also. You go through as much zooming as switch apps and launching apps. Lets say you switch through 100 app a day back and forth then you zoom 100 times! You dont think thats unpractical?

Gnome Shell should focus on these design goals to be any good:
Simple; no complicated concept. Any users does not need to spend time at all to know it. Doesnt take much action to do common tasks.
straight forward; Users dont need to figure out how to do common tasks.
Smart; desktop does all the thinking not users. At the moment, users have to spend much time thinking about how to organize apps, launchers, virtual desktops, activities.
fast; need to operate fast. why depend on javascript? If you really need to why not chrome V8 instead of libmozjs?
good looking; nice easy to use buttons and scrollbars

Hey Nicu, I don’t think it’s not for you because some of the social networking stuff hasn’t even really been implemented yet. but I would really like to see GNOME Shell work for you.

I was wondering if it would be possible for you to take a video showing you switching between tasks so I can gauge how much of a delay is involved? Can you record yourself doing something you would normally do, let’s say checking in on the Fedora design team list and bringing up design mockup links in Firefox and switching back to the mailing list in your mail app?

I went through your assumptions, but I haven’t seen anything about the level of customisation the Gnome Shell will allow.
I understand what you are doing, trying to cover the maximum of possible use cases.
However, as it is always impossible to suit perfectly to all use cases or preferences, I think it is very important to allow (power) users to perform some customisations.
Is it planed ?
If yes, in which way ? (I can give some ideas/points).

* There is an extension system, meant to work very much like the Firefox add-ons model where you can build plugins/extensions into the GNOME Shell core. More information is available here: http://live.gnome.org/GnomeShell/Extensions

There are also definitely plans to have an area within the Shell where users can customize and express their personality – adding little apps and gadgets and things like that. Plans, but no designs yet. :)

Can you think of any design assumptions we could test for this usability test plan to help figure out the design of such an area? For example, are there design assumptions you see in the customizability of social networking apps like Facebook that you think GNOME Shell could learn from?

How do you typically personalize and customize your desktop? We’d love to hear your ideas!

I can tell you how I customize *my* desktop, since I am quite boring, stay close to the default and change only a few things:
– add a couple of icons on the top panel (thunderbird, window killer) and a couple of applets (system monitor, disk mounter);
– make sure some apps are in the session startup, so they are in the notification area (pidgin, xchat, gwibber, deluge);
– but back the icons in menus and buttons;
– make sure i use metacity, not compiz, since compiz is broken when taking screenshots of a window;
– change the background *only* if it is solar or lion;
…and I think this is about all, there may be some minor tweaks (like the old, not the new increased space between icons) which I don’t remember as I keep my /home when migrating to a new Fedora.

I was indeed thinking about my current set-up (using Gnome + Compiz-fusion).

One thing that I like a lot with Compiz, is that it allows you to configure thinly the composer effects (via CCSM). Especially transition and animation times. Thus if I want a quicker transition effect, it’s always possible. The granularity of the customization is also perfect, as I can have a different set-up for each animation.
It would be great to have a mutter, or a gnome-shell “control center” to set-up those kinds of things (a centralised place).
Being able to extend Gnome Shell core through extensions is also a very good thing. Thus, the functionalities won’t be frozen.

What I would like on the current Gnome Shell is also :

– The possibility to use launchers in the “old fashion” way. I mean clicking several times to launch several windows (useful to launch several terminal windows). I don’t like the idea to use a contextual menu for that (“New window”), or to use a middle-click.

– The possibility to move and place windows exactly where I want on each desktop, in the Shell’s “global view”, like I actually do it with the Compiz Expo plugin.

I’ve other ideas / remarks… but this post is already too long :p

Just a last question : Is it still relevant to use the Gnome Shell RPM available in Fedora 12 repository in order to test it (may be there is an updated version already) ?

I think maybe one good extension to GNOME Shell could be an app that lets you launch apps the old way. If it proves to be a good design, similar to how add-ons sometimes get pulled into the Firefox core, it might be pulled into the GNOME Shell core if it works out.

I’ll have to read up more on the Compiz Expo plugin because I’m not familiar with it.

Please feel free to add more ideas and comments :) Don’t worry about length, as you can see in other comments I’ve made I’m quite longwinded myself. :)

The GNOME Shell RPM available in Fedora is very outdated at this point. What I would suggest is try the jhbuild – I’m not a developer, I’m a designer / mere mortal and I was able to set it up without too much difficulty on F12. The instruction are here:

If you want accurate feedback from a large number of people, consider talking one of the devs into providing a repo from his fedorapeople account (pre-alpha is too distruptive to get one single package from rawhide)

This is a good suggestion, I’ll look at doing this. We could potentially push F12 updates but some parts of the stack (gobject-introspection, mutter) are used by other things and I’m worried about disruption there.

One good extension? So, does that mean as it stands, GNOME Shell will *not* allow me to just put a launcher on a taskbar and click once to launch an application?

Um. I have a relatively small number of apps I use frequently. I have them all as launchers on the taskbar. This lets me start every app I typically use with one click each, and do so while I’m in another app if I wish. Except when I start dealing with the file manager, this represents the majority of my interaction with GNOME. The replacement for this functionality will have to be pretty deuced awesome to beat “one click and it’s done” for most of my core needs. If it requires more than one click, it will quite simply make my computing experience worse on my most basic day-to-day usage. At which point, the GNOME shell may have scads of advanced features and be gorgeous and comfortable, but that is unlikely to outweigh bollixing my basic experience.
All the hypotheses about how it would be easiest to search for an application are fine and worth testing, but they only apply to situations where the user *is searching for an application*. The applications I use every day I don’t want to search for, I want to be able to put them where I want them so I can just invoke them without a search; better search is not an answer to losing such functionality.
It’s like having a set of files on my desk. It’s a very good idea to have the stuff in the files arranged so it’s easy to find. But I don’t want my pen to be in an easy-to-find location in the files, I want it on my keyboard, or in a pen-holder, or in my favourite prominent spot on the desk–in short, I want to just reach and get it. If you improve my filing system but decide that it is such a cool filing system that you insist it contain my pen and my stapler and my phone, that will not be an overall improvement.

I find myself *searching* for applications only very rarely. Most of the time I use only a few apps: a couple of them have launchers on the panel, other (terminal and file manager) are launched right from the desktop, other are opened starting from files (open with…) and for the rest I remember their position on the menu (for example the video editor is in the Sound & Video menu, about at the middle) and the launch is mechanical.

GNOME Shell is IMO colossal failure. They are trying to be so desperately different, and result is horrible sh*t. Why they not listen to their users? Do they even care? GNOME user want Mac like interface. GNOME2 Global menu feature should have been implemented long time ago across entire GNOME, but noo… We know better what you want, you want this hibrid, unusable something.

But who cares, by the time GNOME 3 will be released will anyway majority of linux users switch to KDE 4.5.x or Windows 7

Did you miss the fact that this post is about testing GNOME Shell with users to get feedback on it? Or did you just see that the blog post was about GNOME Shell and feel it was a good platform for you to spew your thoughts without actually reading it?

I think having a shortcut to log out is a reasonable request, so I don’t think you are trolling :) I don’t think it supports ctrl-alt-backspace but I don’t think that would be GNOME Shell anyway – I believe that’s X, and X changed upstream to not allow that IIRC.

Honestly, when I tried GNOME Shell I had also trouble finding a fast way to “get me the hell out of it” but wasn’t able to so I opened a terminal ans started the desktop-effects dialog.
Didn’t figure I should click on my name to go to a control panel (and I *hate* having my full name shown all the time on the desktop, like I don’t know it and being forced to use a different account for taking screenshots for web use).

That’s definitely one of those familiarity issues though that I think you get used to after using it for a while. I don’t think there’s anything fundamentally wrong with having log out connected to your user name and online status info – many, many web applications these days including facebook and Yahoo! Mail have the logout button in the upper right corner by the user name. It’s just different, I don’t think necessarily worse.

Maybe… but it should definitely NOT spell my *full name* there, that make my experience feel formal and unfriendly, having there a nick or an avatar would be a lot better. That’s the main reason I don’t use the fast user switch applet, it does put your full name on the desktop (the first version used to give a choice: full name, user name or avatar, but the useful feature was removed in the next Fedora release).

I would guess that 5% of the facebook users do not know how to log out. That is not a problem, because everyone can leave facebook by closing the browser. On the other hand it would be a problem for gnome-shell to give 5% of the users a bad initial experience of not being able to turn off the computer.

One idea is to rename the upper right menu into “account”. That would make it more consistent with major websites, but I would personally prefer “log out” to be placed in the big menu.

Maybe you could add a log-out session to the usability test. Here a hypothesis.

Unfortunately that hypothesis I believe is not going to avoid the very effect we are trying to avoid with this whole methodology –

I drove a Volkswagon Jetta with a stick shift for many years. You put me in an automatic BMW, I am not going to know where the hood release is, the stupid switch to open the gas tank the cars have today, how to move the sideview mirrors – Does this mean they are in a bad place on the BMW? Or does it mean that these things are different sometimes in different cars?

We know the logout button it’s in a different place. We don’t need to test for that because we know. Users are not familiar with it there. We know that too.

What are you trying to get at that the log out menu is problematic for? Think about how you would craft a hypothesis were you to test a user who was familiar with GNOME 3 and had been using Shell for a few months? How would you change it?

The car analogy I made is more accurate than yours at least in the United States, because cars made for the American market always shut their engines off based on the turn of the key. My analogy is more accurate because across different desktops, the methods for logging out and shutting down are different so already there isn’t a consistency.

That’s not the say that your analogy couldn’t work. The first time I rented a car in Ireland, I got a French-made car (Opal I think was the brand name?) that operated based on a card rather than a key. And yes, I had trouble figuring out how to turn the engine on and off.

On another topic, I’ve followed the different posts related to the Gnome Usability Hackfest in London and I’m looking forward to test the implementation of your ideas.
Thanks for your work and for sharing that !

I hate to be picky, but what about GNOME Shell’s design is meant to “deal with the constant stream of digital information” in this age of social networking and stuff?

I don’t have a grudge against the Shell, but nothing I’ve seen in the Product As Is even suggests this is a priority — I’ve seen more work from Zeitgesit, Gwibber and Ubuntu’s Me Menu on this front, and the Shell can’t take credit for those.

And how far can GNOME afford to support Shell development without actually testing it? If you can’t test it “until it’s finished”, how do you know the Shell metaphors are even worthwhile?

Hypothetically it could be the case where the Shell is insufficient for your usability tests, and then all this time has been sunk into a development dead end. Since most of the user experience hope for GNOME 3 revolves around the Shell, what then?

I’m not saying the Shell will fail, but I’m saying that this development workflow – sink a ton of time and effort into something without validating it as you go, and then trying to do it in less than six months at the end – is not a wise way to prototype a new UI.

There has literally been no concerted feedback effort around the Shell – until now. Does GNOME really think it will be able to fully iterate this revolutionary change in less than a year for the GNOME3 release? Or have I misunderstood their expectations, and this is but the beginning of a long feedback and maturation process that will ultimately create an excellent environment?

I am very excited to see you talking about testing the Shell. From your blog you seem to really care about this stuff. But also, it’s about time someone did finally put the Shell to some open feedback-driven workflow, because if they have, they certainly haven’t been talking about it.

“I hate to be picky, but what about GNOME Shell’s design is meant to “deal with the constant stream of digital information” in this age of social networking and stuff?”

I will give you a solid example that I already brought up in the comments. Because the taskbar is not always present along the bottom of the screen, I am not constantly stressed out by the gwibber, empathy, gmail, and other task bar items that blink and sit there demanding my attention as I try to get other work done.

Here is another example – the design of the notification area, which is still a work-in-progress, is to allow the user to set policies as to whether or not they want to be bothered. E.g., if your little sister keeps IMing you to ask for a ride to a friends house, and your coworkers keep messaging you to heap more items on your plate, while you are at work trying to get some work done and focus on a particular to-do item, you could tell the notifications to ‘hush’ for a particular period of time. I do this physically at work all the time if I need to focus – when I sit in my cube people know where I am, and they come up to me and ask me for stuff. :) So when I need to focus on a tight deadline I grab a conference room and ‘hide out’ so I can get it done.

“Ubuntu’s Me Menu ”

What is this?

“And how far can GNOME afford to support Shell development without actually testing it? If you can’t test it “until it’s finished”, how do you know the Shell metaphors are even worthwhile?”

Did someone say you can’t test it ‘until it’s finished’? I certainly hope I did not give that impression. What I said was, ‘GNOME Shell is finally at the point where it’s stable and we’ve got enough functionality in place to run some tests on it.’ It’s nowhere near finished. But it’s at the point now where enough of the design assumptions are put into functional practice, and I don’t have to worry about the thing crashing in the middle of the test. Does that make sense? I am aiming to test the assumptions behind the metaphors themselves in practice.

“Hypothetically it could be the case where the Shell is insufficient for your usability tests, and then all this time has been sunk into a development dead end. Since most of the user experience hope for GNOME 3 revolves around the Shell, what then?”

Do you think it is preferable to not test and forge ahead? (I’ve seen many software companies so afraid of the negative feedback they might get that they avoid usability testing all together. It doesn’t seem like a smart approach to me.) Do you think it is preferable to stop development and just give up? I do not think so. There’s a lot of great technology in GNOME Shell, some real solid stuff, and this first round of testing is just another means to help refine it. I don’t think things are as doom and gloom as I’m getting from your questions? Have you been involved in software development before? GNOME Shell is in a lot better shape than many software development projects I’ve worked on. :)

“There has literally been no concerted feedback effort around the Shell – until now. ”

Hmm. Are you being serious when you say this? Have you read through the GNOME Shell wiki pages and the many blog posts that have made it to Planet GNOME and other Planets asking for people’s feedback? Have you followed the mailing list at all? Folks have been giving a ton of feedback and even contributing core functionality to address feedback & suggestions: http://blogs.gnome.org/mccann/2010/02/06/action-pack/

So, I think while that is a very provocative and interesting statement to make, it unsupportable – how can you back that up after looking at the amount of effort that’s been expended to collect and address user feedback? So let’s respect each other and not make statements like that :)

This is a usability test plan so yes it will be a more formalized way to collect feedback. That doesn’t make the informal feedback collected over the past several months any less valid though. Design processes are not procedural, oftentimes they are not predictable, design is an organic process and you have to start somewhere if you’re to get anywhere. Know what I mean?

“Does GNOME really think it will be able to fully iterate this revolutionary change in less than a year for the GNOME3 release? Or have I misunderstood their expectations, and this is but the beginning of a long feedback and maturation process that will ultimately create an excellent environment?”

I think you’ve misunderstood the expectations – it’s more the latter.

“I am very excited to see you talking about testing the Shell. From your blog you seem to really care about this stuff. But also, it’s about time someone did finally put the Shell to some open feedback-driven workflow, because if they have, they certainly haven’t been talking about it.”

Well thank you for the nice words, I am very excited to do a usability test of this magnitude and to do it out in the community! :) That being said, I don’t think it’s correct to say folks haven’t been talking about GNOME Shell and asking for feedback, because they have. http://live.gnome.org/GnomeShell/ <= look at how many communication vehicles and blog posts are listed there!

I do want to try to drive even more communication about GNOME Shell out though, as part of this usability testing project! :) So look forward to hearing an earful :)

I did not mean to disrespect. You are quite right that the team has been very open with their development — there have been many posts, requests for contributors, and they keep it all in one public place.

But saying “try it and tell us what you think!” – is not the most optimal feedback method. What that actually means, to date, is that the primary method of giving feedback for GNOME Shell is either 1) post a message on their shell-devel-list (which definitely has a not-great noise/signal ratio, with many good discussions but a bunch of complaints and whining as well) or 2) actually join the development team.

Your effort is, to my knowledge, the first goal- and user-focused testing initiative for this project. Everything else has been fly-by-the-seat-of-development ad-hoc drive-by “this is great!” or “this is lame!” feedback. My awareness of their feedback methods could be wrong, but I think my impression (from keeping a keen eye on Planet GNOME and the shell-mailing list) seems accurate. You are correct I am no developer, so perhaps that clouds my judgement.

This will be the first time, in my eyes, that the project must be subjected to true systematic criticism in a public study, to either validate or disprove its ideas. People have been free to show up in a mailing list and say whatever they want, but the developers haven’t had to take any heed to, for example, a message they don’t like (“I disagree with the Shell’s absence of a persistent list of open windows”). The give-and-take of criticism has been ad-hoc and at the developer’s convenience. This is fine for a smaller hobby project (which was what Shell started as), but not for one that will eventually require large-scale testing, QA, and deployment.

So I do think that this new testing initiative is something that Shell hasn’t done before.

A bit of my own personal dislike is the way that the PR was handled for Shell. It may become great, but from the start, it was clear that the Shell had no previous planning and was in fact a brand new, from-scratch endeavor.

The GNOME project, nearly an entire release ago, put its momentum behind Shell before a product even materialized. The Shell may be an excellent environment when it’s released, but it will be In Spite Of its haphazard beginning.

[PS. I would argue with you about how Shell treats notifications and information overload. Your main point seems to be that Shell helps me manage this data…by hiding it. While that may be a valid approach, I think changing notification standards is a far cry from helping me actually manage my networked life. What you mention seems to be a feature-by-absence, which may be useful in itself, but hardly a proactive initiative toward managing internet data simpler and faster. In addition I highly doubt that Simpler Notifications was the impetus for Shell’s development, so I believe I am right in saying that managing internet-age information overload is not one of the Shell’s current priorities.]

Fair enough. I would say maybe, in light of your further explanation, it would have maybe been more accurate for you to instead of saying:

“There has literally been no concerted feedback effort around the Shell”

to say something more like this:

“There has been no rigorous / formal user testing effort around the Shell.”

And *that* I think is a fair statement. I do feel a little responsible for that situation as while my intentions were always to help GNOME 3 by doing exactly this project for the Shell, I always had bigger fires to put out and GNOME 3 was more R&D stuff at the time. Now that the GNOME 3 schedule is a little more serious I’ve carved out the time and resources to be able to do this right. It is quite unfortunate that I couldn’t have done it earlier – although to be fair, I was also kind of hoping that more design / usability-minded folks would have been willing to step up and help out now. I mean,this is GNOME 3 – serious stuff, great potential.

I do think your comments on the ad-hoc form of the feedback thus far are also fair – although also consider that the folks who have been working on the Shell, while established GNOME rockstars, are *developers*, not designers (except for Jeremy), so they weren’t even quite sure how to go about getting this kind of formalized / rigorous feedback, nevermind on top of actually developing it. The developers asked me and many other designers in the community repeatedly over the past few months for assistance (and I always had to say, ‘yes, cool project! But… the flames of doom are burning around me, give me more time to get back to you!’) – as a result my colleague Jeremy has been working with them closely on design decisions in recent months.

Unquestionably we do not have enough designers in the FLOSS community to meet the general need. So yes, your critique of the feedback process thus far is indeed quite reasonable, but also I think must be understood in terms of the climate we have for design work in FLOSS in general these days.

“So I do think that this new testing initiative is something that Shell hasn’t done before. ”

s/Shell/GNOME. Where else do you see open usability testing happening in upstream GNOME, shared and discussed within the community? If it is happening, I’m truly missing it. I believe it was a survey of our foundation membership’s that uncovered the complaint we’re not working on usability enough in GNOME in general – and that was a major part of why the usability hackfest in 2008 and the one a couple of weeks from now were planned.

“A bit of my own personal dislike is the way that the PR was handled for Shell. It may become great, but from the start, it was clear that the Shell had no previous planning and was in fact a brand new, from-scratch endeavor.”

That’s true, it was a brand new endeavor, and it was started at that 2008 usability hackfest. Can you explain a little bit more what is problematic about that?

Many of the ideas behind GNOME Shell were brainstormed together as part of the first GNOME Usability Hackfest in October 2008 (http://live.gnome.org/Boston2008/GUIHackfest). As a community we had decided it was time to start planning GNOME 3.0. You have to start with a plan before you have something materialized right? I mean, finished software products don’t appear from thin air? (I think I must be missing your point?)

“[PS. I would argue with you about how Shell treats notifications and information overload. Your main point seems to be that Shell helps me manage this data…by hiding it.”

It’s not just hiding it. Are you familiar with blogs like zenhabits.net, or books like Getting Things Done, or even more strongly-affiliated-with-Buddhist/Zen-principles-stuff like Awake at Work? It’s not about ‘hiding it’ – it’s about clearing your space to focus and be mindful on what you are working on in the present. Mindfulness I think is the key goal – I’m really into meditation and mindfulness. I first got into it because (this is very funny given this discussion we’re having right now) I was so overloaded and stressed out over the huge disparity between demand for design work and what I could personally put out. I’m very much a ‘get shit done’ kind of person and I have a very hard time telling people ‘no.’ Over the past year, though, I’ve been stretched beyond my limits and have finally, finally learned I do not scale. I picked up ‘The Present Moment’ CD set by Thich Nhat Hanh because a friend recommended it to me as a good way to learn how to cope with the stress I was facing. That opened up the whole area of mindfulness and meditation to me, this world I had not known much of anything about before.

GNOME Shell really clicked for me in light of the mindfulness practice I had started for myself, it suddenly clicked at some point a while back, and I saw quite a few of the Buddhist mindfulness principles I’d read about as part of my practice right in the GNOME Shell… that is what excites me the most about it. It seems to be a general moment lately as well – for a while now I’ve been seeing posts every now and then about Getting Things Done on Planet GNOME (there is even a Getting Things GNOME app :) ), and it seems more and more blogs like unclutterer.com and zenhabits.net and lifehacker.com are getting popular – I think in response to the overload, in response to the clutter. Mindfulness is a great solution and a lot of these sites draw directly or indirectly from mindfulness principles.

Sorry to go all incense-y Buddhist on you, but hopefully that helps you understand my perspective a little bit more?

There’s nothing horribly problematic about planning for GNOME 3.0, and indeed I didn’t know the Shell was first envisioned at the Usability Hackfest 2008.

I just meant that GNOME should have waited for it to develop first, and then endorse the actual product, rather than endorse the idea of it from the start, as a future GNOME release UI. But this is not a development issue, and I have no experience in planning big releases of desktop environments, so I only have my own gut feelings which are probably inaccurate.

And it does seem that the Shell’s lack of formal usability testing isn’t because of apathy, and I appreciate how busy every developer and designer is.

Also, you’re right, I don’t see general usability testing elsewhere in GNOME, except for yours.

I do understand your opinion about hiding unnecessary information in the middle of work. I just think that in addition to scaling back the notification overload, there is still plenty of room for experimentation in UI for actually managing internet data – contacts, communication, and online documents and downloads – and I seen Shell doing nothing for -those-, which are more interesting problems to me than merely fixing the over-talkative environment.

I want to see contacts made first-class objects. (I hear Zeitgeist might help with that). I’d love to see all of my online information sources integrated, like on the Palm Pre – a commingling of information from Google, Facebook, Twitter, etc. I just don’t see any great workflow innovation in Shell like I do on new mobile UIs. I just see better window management.

That’s my wishlist, and so that’s what I think of when dealing with communication-overload – I not only want to do less of it, but I want to do it smarter when needed. I think this is a potential still untapped in the GNOME desktop space. But this is all secondary to your central point – I do understand that it is hard to organize usability testing at the FOSS level, and I do appreciate that you enjoy the Shell’s more austere interface.

I admit that I like the idea of austerity myself. But personally, I question the usefulness of being rebombarded with everything when I go to the Activity view – and I bring to mind here your “moving mess to another room” post above — isn’t that possibly what Shell does with its Activity view? But I digress. I love the idea of a simpler main interface and I do look forward to trying it. I just want it to be straight-forward, and powerful (and I believe you can do both).

But I didn’t mean to comment on the Shell’s design, merely its development process, and you’ve already addressed everything I asked. Thanks.

Quick comments. What is currently missing is some customization about the color because black does not really suit for users.
Now the current gnome-shell on F12 fully works with stylus input from tablet, I intend to regularly use it. It is very similar to Sugar Interface minus virtual desktop.

What is it about the black color that you don’t think suits users? Maybe we can come up with a hypothesis to test on that? What kind of design belief do you have that leads you to conclude black won’t work?

Mairin, it’s partly a matter of personal preference – some users like dark color schemes, others hate them. But you might also consider factors around lighting conditions – does the color scheme work as well indoors under artificial light, vs outside in sunlight?

Well, I can sense some hypotheses to test brewing here, then. :) What do you think about this one:

A dark color scheme for the desktop shell will not cause readability problems for users in both indoor and outdoor lighting conditions.

Regarding preference, I think we need to be careful. We don’t want to literally be fighting over the color to paint the bikeshed with our users, you know? So rather than measure user ‘preference’ for particular color schemes, can you think of the kind of feeling we’d like them to get from it, or we’d like to avoid from it, in order to test something a little more measurable and useful than simple preference?

Preference is perhaps the wrong word, but I refer to any factors that vary between individuals.

For example, I personally find light text on dark background harder to read – not as a matter of legibility, but because the contrast causes greater strain on the eyes than dark-on-light. I find that white-on-black like the current GnomeShell screenshots looks great, but is hard to look at for long periods. Yet I also know that there are people around for whom the opposite is the case.

I guess it comes down to identifying a default that looks good and works best for the majority, but allowing alternatives through theming (just as existing Gnome 2 users can pick a different theme if Clearlooks doesn’t suit them). I guess to frame it as a hypothesis, something like “A dark color scheme for the desktop shell will not cause discomfort over extended periods of use”? Relative, of course, to lighter color schemes such as current versions of Gnome.

marin wrote:
> So rather than measure user ‘preference’ for particular color schemes, can you think
> of the kind of feeling we’d like them to get from it, or we’d like to avoid from it, in
> order to test something a little more measurable and useful than simple preference?

From me you may want to avoid the “awww… that black is too much like Vista”

Rather than wasting time on a color hypothesis, why not just keep it set to the GTK theme? Or, even let the user decide between the GTK theme, a color, and a background image, just like the panel does now? Keep choices a part of this, PLEASE, rather than deciding that “one-UI-fits-all”. Let the user be the deciding factor rather than a silly hypothesis test.

“something a little more measurable and useful than simple preference?”

Wait a minute, to me it’s ALL about simple preference. Let me decide. I can always go to Microsoft if I want my decisions made for me. I use Linux because I want to make up my own mind, thank you.

It looks like the gnome-shell does not support ctrl-alt-shift-left and ctrl-alt-shift-right. I know that most people don’t use these two short cuts, but I do and i would like to see them added to gnome-shell (if they aren’t already there).

In gnome 2.28 these two shortcuts allow me to use the keyboard to rearrange windows among desktops without ever touching the mouse. If I open a new program then I can use the two shortcuts to move the program to a good desktop, or if epiphany suddenly opens a link in a new window, then I can use the two shortcuts to get the window out of the way quickly without having to close it,

Perhaps your uasbilty test could include a small session where the user is not allowed to touch the mouse. That would force him to use some of the features that are also useful for power users.

I just tried, and Ctrl+Alt+Shift+Left Arrow/Right Arrow work perfectly fine under gnome-shell. these are shortcuts of Metacity/Mutter and are configurable. you should check in the keyboard preferences.

as for keyboard accessibility: yes, it should be tested, but along with the overall a11y of the environment, not in terms of the Shell usability. just because power users want to have a the key combo of the Vulcan Death Grip it doesn’t mean it’s a useful usability concept for the general user population.

as for keyboard accessibility: yes, it should be tested, but along with the overall a11y of the environment

Please don’t think of keyboard use purely in terms of accessibility, as if it’s a niche requirement. Gnome currently rates pretty well in terms of support for keyboard shortcuts – particularly for manipulating virtual desktops. I’d be very disappointed if that were lost under GnomeShell.

just because power users want to have a the key combo of the Vulcan Death Grip it doesn’t mean it’s a useful usability concept for the general user population.

And yet, that doesn’t mean it’s not a desirable ability for the power user. The general user population might not use keyboard shortcuts or virtual desktops, but they’re absolutely fundamental for the power users.

I’d like to propose two assumptions. They are quite negative I’m afraid: they address two of the biggest fears I have about this.

Really impressed with the testing concept. Seems very well thought.

• Notifications shown in a short messaging tray aligned along the length of the bottom of the screen will not be missed too often.
• A zoom animation of the whole screen to shift the user from full-size windows to an overview of open windows will not cause eye fatigue to an excessive number of users.

• When switching frequently between windows, users will find that switching through the overlay is fast and convenient enough.

Plus, I’d like to comment on two of yours:

# Category-based application lists are more distracting than usage-based application lists: What is the meaning of “usage”? I think it is too vague a concept. Do you mean, applications related to the task you are doing? Most frequently used applications? I think all approaches have different sets of pros and cons. For example, categories would be superior to frequency, if the person searching the application is not the usual user.

# The placement of the Activities hot corner in the upper left will cause users to be less prone to accidentally triggering it than if it were placed in the upper right corner. Wouldn’t it be better “if it were placed in any other corner”? Or maybe there should be a different, previous assumption about the top placement of the taskbar?

I just installed Gnome Shell and I was pleasantly surprised at how usable it is. My only comment is switching between windows within the same application requires me to first Alt-Tab then either use the mouse or use the arrow keys to select the correct window. Aside from one-handed users not being able to Alt-Tab and use arrow keys at the same time, I think two-handed users will it difficult to select the right window. I don’t have a good solution for you.

A design assumption might be:

“Requiring the use of two hands to switch between windows of the same application will not be difficult for the user.”

One of the design principles of GNOME Shell is that it will provide more advanced users the tools they need without getting in the way of beginner users. There are certainly quite enough Alt+Tab users that I don’t think it’s a waste to make sure it works just as well for them as for less sophisticated users.

what is supposed to happen by hitting ` while in alt+tab mode? I guess this is the key above Tab, but I have no result by hitting it. It would be great if this key would switch between windows from the same application (i am not using us keyboard)

I don’t know if anyone mentioned it yet, but i really think, that GNOME Shell should allow to change colors of overview mode. For example, if user has light wallpaper on his desktop, constant switching to overview mode with black background could be very unpleasing to eyes. Well, it would be annoying to me personally. Maybe GNOME Shell could automagically adapt its colors to user’s desktop background?

I have been seeing a few GNOME Shell Video on the web. I personally think, it push the Activity button too “hard”. What I mean by that is that the feed back of that button is simple overwhelming. Every time the button being pass, The whole screen Shift. all the “workspace” shift, and all the “app” on that workspace “shift. And it seems like all this does is to allow user to “add and move” app and workspace. I think that is super annoying. I impress the first time but after like 3 times, it gets really annoying. The question is why things has to “move” like all the time.

Dont’ make me wrong, is not like I don’t like that “Activity” button, but I think I will like it better if it is just a little icon on the concern, that I can address once for while to organize something. But I will have to have my “Menu” button that only give my the menu of App. I usually use Alt + F2 to launch anything. but as you know might already know Linux do have some weird / long named App, that I can’t remember how to spell it, that is where the menu got very usually. I also love the hidden icon option that come with GNOME menu editor, because sometime great App doesn’t come with sexy icon or icon at all. so I can use that option to hidden some icon that doesn’t mean anything or ugly in order to kind up the clean look of Desktop, but it can simple turn it on like anytime I want too. Just too lovely.

One other thing that brothers me so much is that Clock in the cent of the panel. I mean just why it has to be on the center, like die center!!! I might be miss out sometime, but I can find no reason to put that clock on the center!!! Make it visible but don’t block my the usable space I have. To me Panel is good for menus like the one in mac. although not everyone will agree with me. or the traditional which use for active windows that got open, so you can hide it and open it quickly. Panel is also a great place to hold all the little monitor app like system monitor and network monitor. Seriously, these things are GREAT!!!! I might be old school. but once I configure good environment for myself, I usually don’t want to change it. It simple work. This always lead to my next point is that

What GNOME Shell should be doing is give ability for the user to design what they want. No designer in the world can design something that fix everyone. Just accept this. That is why a design while allow the user to configure what they want and explore will always winning people’s heart. We Open Source user know what are we doing. (At least we think we do, haha) However, what the designer can do is set up a general Default, a good default, for people who just want thing out of box, and give people an good impression. I think this is always important.

I love GNOME, and hope the GNOME team realize that sometime when you trying to design for everyone, They are not really design for everyone, sometime feature that makes easy for everyone might be feature that brother and slow down the user who have been using GNOME, and learned GNOME. The act that try to Gain one new user might ending up losting millions of old users.

Ok, writing a design assumption is pretty tough, but here’s my first attempt at it:

* although the notification area is on the bottom-right corner of the screen, users will not be confused by the fact that they must click on “activities” (top-left corner) to review expired notifications

A bit convoluted, I know, but that’s the best I can do in 5 minutes time :)

“A persistent window list is distracting to the user. A user will be less distracted if the window list is not always visible on the screen.”

This seems like a very sound hypothesis, but I’m not sure that’s the best approach. I’m just guessing here, but I think that Windows and OSX keep some sort of windows list in their offerings for a reason. For instance, because it’s faster to switch between applications (or instances of an application) if you have an on screen window list, and that is an important feature for professionals that need to manipulate multiple documents in various applications at the same time. “A persistent windows list allows the user to work faster. A user will complete complex tasks sooner if a window list is always visible on the screen.”

Since these hypotheses don’t seem to be totally contradictory in nature (you can be less distracted and still be slower to finish your task) it seems to me that the real usability issue here lies elsewhere.

“A persistent windows list allows the user to work faster. A user will complete complex tasks sooner if a window list is always visible on the screen.”

Agreed. This is the main argument for having a window list always visible. I’m having trouble imagining myself working with multiple apps and multiple documents without going crazy without a window list.

If Gnome-Shell ran smoothly enough in any of my computers I’d be tempted to give it a try for a week, but it fails in that respect even though compiz could do it 3 years ago. :-/

I hope the devs work on the performance of the shell as soon as possible so more users can give it a longer-term test drive. Fingers crossed.

I have a few concerns (or perhaps misunderstandings) about gnome-shell as well as some ideas for the usability test plan. I’m eager to hear your feedback.

Anyway, I use workspaces to “hide” details of other things I’m working on. This seems to integrate well into the gnome-shell concept of tasks, but what concerns me is that in order to show all windows in my current workspace/task, I have to show all windows in all workspaces as well as the launcher . I am a Mac user as well as a GNOME user, and I am very accustomed to using Exposé on my Macs and Compiz’s Exposé clone with GNOME. This allows me to show all windows in my current workspace without introducing the visual noise of all of my other workspaces and windows (or launcher). Particularly on a small screen, it is much more difficult for me to manage many windows and workspaces on gnome-shell than when I use Compiz or OS X’s Spaces/Exposé.

(Usability test plan idea: Can a user manage windows in a single workspace as easily as with Exposé?)

Additionally, is there an easy way to show the desktop in gnome-shell? The best workaround I have is to create a fresh workspace. I often use the desktop as temporary storage or as a scratch pad for files and Exposé and Compiz both allow me to quickly show the desktop with a hot corner. This allows me to quickly drag-and-drop files from the desktop into open windows using only hot corners, no clicks. However, in gnome-shell, this workflow does not seem to be possible.

(Usability test plan idea: Can a user easily retrieve/access files and folders on the desktop?)

There also seems to be a lot of wasted space in the panel/taskbar. On my netbook’s small screen, over 50% of it is empty and there would be a lot more if the font were not so large. Does anything else populate the panel? On a very large screen, it seems like the amount of wasted space could exceed 90%. By moving the clock into the system tray (not necessarily all the way into the corner), a global menu or window list could be comfortably fitted into the free space left behind (at least as an option). Am I misunderstanding how this bar is used?

(Usability test plan idea: Not really sure of a straightforward test on this one)

Lastly, I just want to voice my opinion about how important it is for me to be able to theme gnome-shell. I’m personally not a fan of dark themes, but more than that, gnome-shell just doesn’t ‘fit’ with my font, gtk theme, or window decoration. It just ends up looking obtrusive and interferes with the overall design of my desktop. I understand that gnome-shell is nowhere near finished, I just hope that theming will not be left out when it is. I think details like this affect the emotional attachment a user has to his/her desktop environment.

I’d appreciate any and all feedback you have regarding these concerns. Please let me know if I am misunderstanding how gnome-shell is intended to be used or if I am simply trying too hard to make gnome-shell work like my Macs.

the add in the menu bar of a simple menu that lists preferred apps, places, and recent documents would be nice

I think that if I just need to open a preferred application, a place or a recently opened document the menu way is simpler: no need to truncate text, no need to open a full screen panel made also for other things (a sort of shortcut for most used activies items)

It can be just right the activities menu.

maybe the replace of “Activities” in the menu bar with an icon can be better looking

I’ve read the whole page without a halt, and I feel like most of the people here are as stagnated as well-water. I feel GNOME shell is a great improvement over the previous panel (though I love the panels for the degree of customisation). I’m using the GNOME shell for 1 month now and I’m really impressed by the usability and subtle eye-candy that it offers (Compiz is too bloated ). The main thing is, that GNOME is showing great progress. I’ve been a great fan of GNOME from the day I started using linux 2 yrs ago(I know that’s quite less but considering my age, which is just 18, it’s quite good). I intend no offence, but to me, KDE is pure copy of Windows user interface, with no or negligible innovation. Yet, GNOME is always accused of being stagnated. Over the last two years, I’ve seen a lot of improvements in the GNOME UI, most of which take into account the usability, rather than eyecandy. While KDE has improved on the visual side.

With the release of KDE 4.3, GNOME was more and more accused of being *ugly*(people still think so, thanks to he brown colors of ubuntu, which can only appeal to Mr. Shuttle). But GNOME shell will be a major setback for KDE 4.3.x.

There are quite a few things I(and many others) would like to have in the GNOME shell
:- ability to customize the color, transparency of the top bar and the overlay.
:- ability to change the background of the overlay.
:- the alt-tab swithcer to be persistent, means you dont have to keep holding the alt key to switch

BTW, GNOME shell is great. I was using the fedora version, which may be outdated, so i’m currently downloading the latest packages to build it myself. I also think I should do some development, as I’m currently into javascript and would love to code for one of my favourite softwares.

kudos to the GNOME-shell team, for bringing out such a brilliant thing..

Although the comments section seems to be more or less closed, I would like to add one. First of all, I have to say that I haven’t tested GNOME Shell personally due to lack of a machine with working 3D acceleration (I didn’t feel like installing the non-free NVidia drivers), but I think that the comment will stay relevant. I am going to offer some assumptions almost diametrically opposed to Flavio’s one. The first is that the user will be more productive if the recently/often used menu does not distinguish between different types of object (e.g. recently/commonly used files, non-document applications, recently/often visited disk locations or URLs). Some intelligence is assumed regarding what is displayed in the list and what not (in particular, not every URL which is clicked through while the user was searching for something else), to ensure that things that interest the user are not swamped by irrelevant items.

A second, related assumption is that it will be more useful to the user if the menu holds both open and unopened objects of all sorts, as is the case now for applications (with the little light to indicate whether or not it is currently in use). I think that this would make both a GNOME 2-style taskbar and in many applications also tabbed document interfaces unnecessary.

A third and final assumption is that the search box could usefully be made to cover all sorts of objects, as in the first assumption, with some intelligence to predict what is being searched for when the user starts typing (e.g. if it looks like a web page address, try opening it as a web page first), and that this would boost the user’s productivity. It could also make the location bar in web browsers unnecessary and save browsers from having to implement search and store functionality for locations visited themselves.

To be honest, I can understand from a developer scope the need of having EXPLICIT last documents manipulated. Even Microsoft has disabled on 7 the “Last Opened Documents” by default concerning users privacy.

It’s a sad feature of gnome shell and Nicu pretty much filled in the rest. I understand the need to improve and specially to innovate, but I’m just way too much skeptical about this gnome-shell.

So far it’s unusable due to massive lag on the interface despite of my system theoretically being capable of chewing far more than gnome shell demands.

A cool thing you can test is to do CPU and GPU stress tests while running gnome shell, those could be enlightening to devel I suppose.