Sunday, November 4, 2012

Usability testing

(This post has been updated with a few improvements, based on suggestions. Edits are in blue.)

Next week, I plan to start doing usability testing on the three programs: Gedit, Nautilus, Firefox.

Since I work on a university campus, I'll start by inviting students, facutly, and staff to participate in the usability study. In most usability tests, you give your testers a small gratuity for their time. Since I don't have a budget ($$) to give out cash rewards, I'll instead offer free cans of pop to take with them, and as much candy as they can carry away with them. That's pretty inexpensive for me - and on a university campus, that should still garner a fair turnout.

I'll also invite folks from the surrounding town (we're small, it's easy to get word around) to participate in the test too. And I'l try to get friends and family to submit to a usabilty test, as well.

My goal is about a dozen testers from different age groups. From the discussions I've had with "thought leaders" in usability testing, I may not need as many as a dozen folks to get usability good results - so I'll be satisfied if I only attract half that (five or six).

Each tester will start with a fresh instance to test against. This needn't be a separate install; it can be the same workstation, but different test accounts so that subsequent testers always get the same "vanilla" starting point as the first tester. My thought is to have the tester work from a bootable USB fob drive running Fedora 17, with separate test accounts for each tester: "test1", "test2", "test3", ... Since each test account can only modify it's own settings, everyone starts from the same "default" settings. This is important, especially since I'll likely ask testers to create or modify bookmarks (Firefox) or change a default font (Gedit).

The goal will be to have the tester engaged in a usability test for no
longer than 45 minutes - that's 15 minutes per program. I'm thinking
about the following steps:

Welcome (less than 5 minutes)

Brief context of the study, what is my study about (inform tester about the purpose of the study, and it's output)

Short explanation for what a "usability test" is - we are not testing you, but you are helping us examine software; there are no "wrong" answers

Quick rundown of what the test will cover: Gedit, Nautilus, Firefox. (This is a good opportunity to share comparisons. For example, Nautilus is a file manager, like Windows Explorer, or Finder on Mac. Having that shared reference can help set the stage for the tester, but I need to be careful about drawing too direct a comparison.)

Ask if they have participated in a usability test before. This is probably a new experience for many of the potential testers.

Introduction to how to do a usability test - please "speak aloud" when you are looking for something, etc

Testing (30-45 minutes, depending on the tester)

Gedit (10-15 minutes)

Firefox (10-15 minutes)

Nautilus (10-15 minutes)

For each program, start by giving the tester a set of written scenarios - one at a time, one scenario per page - and ask him/her to complete that scenario. Also, ask the tester if they have used this program before, or how they would expect to use the program.

When the tester has completed a scenario, move on to the next scenario: each scenario should be printed on a separate piece of paper. At the end of all scenarios for a program, take a short break before moving to the next program. This is where I do some basic Q & A by asking followup questions. Typical usability followup questions include: "What were you thinking when you tried to do X?" "You seemed lost when you were working on Y - what would have made it easier?" "What were you expecting to see when you were doing Z?" Also, a quick "plus/delta": what worked well, and what needs to be improved?

Wrap-up (less than 5 minutes)

"Thank you for your time."

Provide them with a 1-page information sheet about the three programs, as a giveaway. (Testers may like to refer back to this afterwards, as a way to better connect with the test.)