5
5 Differences •Qualifying participants: –Moderated—ask them; have them fill in a questionnaire at start. –Unmoderated—ask them in a screener section and knock out anyone who doesn’t fit (age, geography, disease, etc.).

6
6 Differences •Test scripts: –Moderated—the moderator has tasks he or she wants the participant to do, and the moderator and the notetakers track the questions and difficulties themselves. –Unmoderated—the script contains both the tasks and the questions that the moderator wants to address.

7
7 Differences •What you can test: –Moderated—anything that you can bring into the lab. –Unmoderated—only web-based software or web sites.

9
9 Creating an unmoderated test script •Screener: “Do you meet the criteria for this test?” • For each task: “Were you able to…?” –Ask “scorecard” questions--satisfaction, ease of use, organized –Ask “what did you like?” and “what did you not like?” –Provide a list of frustrations with an open-ended “other” option at end. •Wrap-up: –Overall scorecard, “would you return,” “would you recommend,” address for gift

10
10 What a test looks like: Screen The first few slides ask demographic questions. They can be used to eliminate participants from the test.

11
11 What a test looks like: Task

12
12 For your first task, we would like your feedback on the tugpegasus.org home page. Without clicking anywhere, please spend as much time as you would in real life learning about what tugpegasus.org offers from the content on the home page. When you have a good understanding of what tugpegasus.org offers, please press 'Answer.'

13
13 What a test looks like: Task You can create single-select questions as well as Likert scales.

14
14 What a test looks like: Task You can tie probing questions to earlier answers. The questions can be set up to respond to the earlier answer, negative or positive.

15
15 What a test looks like: Task You can have multi-select questions that turn off multiple selection if the participant picks a “none of the above” choice.

16
16 What a test looks like: Task You can make participants pick three (or any number) of characteristics. You can also randomize the choices, as well as the order of the tasks and the questions.

17
17 What a test looks like: Wrap-up The last set of questions in a study are score-card type questions: Did the participant think the site was easy, was she satisfied by the site, was it well-organized? Usability = credibility

18
18 What a test looks like: Wrap-up A participant might be forced to return to the site for business reasons, but if he’s willing to recommend it, then he’s probably happy with the site.

19
19 What a test looks like: Wrap-up Answers to these exit questions often contain gems. Don’t overlook the opportunity to ask for last-minute thoughts.

22
22 Comparisons: Statistical validity •What’s the real difference between samples of 10 (moderated) and 100 (unmoderated)? –“The smaller number is good to pick up the main issues, but you need the larger sample to really validate whether the smaller sample is representative. –“I’ve noticed the numbers swinging around as we picked up more participants, at the level between 50 and 100 participants. At 100 or 200 participants, the data were completely different.” Ania Rodriguez, ex-IBM, now Keynote director

23
23 Comparisons: Statistical validity •It’s just math…

24
24 Key Customer Experience Metrics Q85 – 88. Overall, how would you rate your experience on the Club Med site. Ease of use Overall Organization Level of Frustration Perception of Page Load Times Significantly higher or lower than Club Med at 90% CI n=50 per site “Site was slow site kept losing my information and had to be retyped.” – Club Med “I could not get an ocean view room because the pop up window took too long to wait for.” – Club Med Club Med trailed Beaches on nearly all key metrics (especially page load times).

25
25 Comparisons: Statistical validity •What’s the real difference between samples of 10 (moderated) and 100 (unmoderated)? –“In general, quantitative shows you where issues are happening. For why, you need qualitative.” –But “to convince the executive staff, you need quantitative data. –“We also needed the quantitative scale to see how people were interacting with eBay Express. It was a new interaction paradigm [faceted search]—we needed click-through information, how deep did people go, how many facets did people use?” Michael Morgan, eBay usability group manager; uses UserZoom & Keynote

26
26 Comparisons: Statistical validity •“How many users are enough?” –There is no magical number. –Katz & Rohrer in UX (vol. 4, issue 4, 2005): •Is the goal to assess quality? For benchmarking and comparisons, high numbers are good. •Or is to address problems and reduce risk before the product is released? To improve the product, small, ongoing tests are better.

27
27 Comparisons: Shock value •Are typed comments as useful as audio or video in proving that there’s a problem? •Ania: –“Observing during the session is better than audio or video. While the test is happening, the CEOs can ask questions. They’re more engaged.” –That being said, “You can create a powerful stop- action video using Camtasia and the clickstreams.”

28
28 Comparisons: Shock value •Are typed comments as useful as audio or video in proving that there’s a problem? •Michael: –“The typed comments are very useful—top of mind. However, they’re not as engaging as video.” So, in his reports, he combines qualitative Morae clips with the quantitative UserZoom data. –“We also had click mapping—heat maps and first clicks,” and that was very useful. “On the first task, looking for laptops, we found that people were going to two different places.”

29
29 Comments are backed by heatmaps

30
30 Comparisons: Quality of the data •Online and in the lab, what are the temptations to be less than honest? –In the lab, some participants want to please the moderator. –Online, some participants want to steal your money.

31
31 Comparisons: Quality of the data •How do you prompt participants to explain why they’re stuck if you can’t see them getting stuck? –In the task debriefing, include a general set of explanations from which people can choose. For example, “The site was slow,” “Too few search results,” “Page too cluttered.”

32
32 Comparisons: Quality of the data •How do you prompt participants to explain why they’re stuck if you can’t see them get stuck? –Let people stop doing a task, but ask them why they quit.

33
33 Comparisons: Quantity of data •What is too much data? What are the trade-offs between depth and breadth? –“I’ve never found that there was too much data. I might not put everything in the report, but I can drill in 2 or 3 months later if the client or CEO asks for more information about something.” –With more data, “I can also do better segments” (for example, check a subset like “all women 50 and older vs. all men 50 and older). —Ania Rodriguez

34
34 Comparisons: Quantity of data •What is too much data? What are the trade-offs between depth and breadth? –“You have to figure out upfront how much you want to know. Make sure you get all the data you need for your stakeholders. –“You won’t necessarily present all the data to all the audiences. Not all audiences get the same presentation.” The nitty-gritty goes into an appendix. –“You also don’t want to exhaust the users by asking for too much information.” —Michael Morgan

35
35 Comparisons: Missing data •What do you lose if you can’t watch someone interacting with the site? –Some of the language they use to describe what they see. “eBay talk is ‘Sell your item’ and ‘Buy it now.’ People don’t talk that way. They say, ‘purchase an item immediately.’” — Michael Morgan –Reality check. “The only way to get good data is to test with 6 live users first. We find the main issues and frustrations, and then we validate them by running the test with 100 to 200 people.” — Ania Rodriguez –Body language, tone of voice, and differences because of demographics…

39
39 Comparisons: Time •Which type of test takes longer to set up and analyze: moderated or unmoderated? –What’s your experience?

40
40 Comparisons: Subjects •Is it easier or harder to get qualified subjects for unmoderated testing? –Keynote and UserZoom offer pre-qualified panels. –If you want to pick up people who use your site, an invitation on the site is perfect. –If you do permission marketing and have an list of customers or prospects already, you can use that. •How do you know if the subjects are actually qualified? –Ask them to answer screening questions. Hope they don’t lie. Don’t let them retry (by setting a cookie).

41
41 Comparisons: Environment •In unmoderated testing, participants use their own computers in their own environments. However, firewalls and job rules may make it difficult to get business users as subjects. •Also, is taking people out of their home or office environments ever helpful—for example, by eliminating interruptions and distractions?

42
42 Comparisons: Geography •Remote unmoderated testing makes it relatively easy to test in many different locations, countries, and time zones. •However, moderated testing in different locations may help the design team understand the local situation better.

46
46 Questions? Contact us anytime! Susan Fowler has been an analyst for Keynote Systems, Inc., which offers remote unmoderated user-experience testing. She is currently a consultant at FAST Consulting and an editorial board member of User Experience magazine. With Victor Stanwick, she is an author of the Web Application Design Handbook (Morgan Kaufmann Publishers) ; cell