Mistake#1: Revealing everything upfront

It’s better to explain to them everything in detail right from the start so that they can start testing with a realistic expectation.

While it’s important to put your testers in the picture to give them a realistic scenario to work in, you shouldn’t miss the opportunity to have THEM tell YOU what they believe the purpose of your site, product and/or service is.

Mistake#3: Using words from interface items

All you can do is to minimize this effect by trying to lead your testers as little as possible.

One way to do this is to ensure that you aren’t priming your testers by using words that can be found in the interface of your site to describe the task you want them to do.

If you’re using the same words in your task as on your site your testers will unintentionally switch into tunnel vision and hunt for exactly this wording.

All you’ll learn is how your site performs on a scavenger hunt for the wording used.

“The way you design tasks could have a dramatic outcome on the results, without your even realizing it. In a testing situation, the participants really want to please you by following your directions. If the tasks direct participants to take a certain path, that’s the way they’ll go. If it’s not what real users do in the true context of the design’s use, then you may get distorted results.”

Hearing the words your participants use to name things on your site can often be very rewarding as well. Thus you get valuable feedback if the wording used by your users is similar to yours’.

✘ WrongYou want to order this product now (Button reads: “Order now”); please proceed by setting up your account (The following step is labeled: Set up your account).

✓ RightYou want to get a gift for your girlfriend’s upcoming birthday. Try to find something according to her taste and buy this product.

Mistake#4: Ask about a hypothetical future

As a marketer it’s very interesting to understand the marketing funnel from people moving from visitors to leads.

So how about asking them in your usability test what makes them click?

During the last months we came over quite many tasks like this:

✘ Bad

Would you pay for our App? Would you use our service? Would you subscribe to this newsletter in the future?

Don’t ask people what they would do in the future. They simply don’t know.

– People overestimate how much money they will have in the future.
– People overestimate how much time they will have to use your product in the future.
– And they genuinely want to be nice and tell you yes.Teresa Torres – Ask about the past rather than the future

You’ll learn far more understanding what triggered past behaviors of your testers than letting them imagine a fictive scenario in a nebulous future.

✓ BetterHave you ever paid for a service like this? When? Do you still use that service?

Mistake#5: Ask for preferred usage

Usability tests let you find out what’s clear and what’s not clear to people as they use your website.

Usability testing is in no way about opinions.

Even worse then asking about opinions is asking for a preferred usage.

✘ BadWould you either search using Version A or Version B? Is the feature of starring interesting? Would you use this feature?

If you want to learn about preferred usage, A/B testing is the way to go.

Setup a study with a specific, quantitative goal and compare the results of both versions.

Of course you can add usability testing and test both versions but observe people and try to spot usability errors and don’t expect usability testing to deliver data about preferred usage.

“When people express an opinion, in the course of a usability test, pay attention to it, act like you’re listening to it and taking it in. But, sort of, immediately let go of it and don’t fixate on it, don’t worry about it, just let it go. And make sure that you don’t give them the impression that you’re looking for more opinions.“– Steve Krug

I can’t tell you how much I hate it when a website asks me vague questions and then want a “on a scale of 1-10” answer. I am always thinking what does this even mean? How are you going to process my number if you don’t allow me to give you any other data beside just a “5”.

Markus Pirker

I completely agree. After all we’re humans, not numbers. Despite the ambitions of top level management to measure every action and have quantitative data for the complete customer journey.