What you get from unmoderated and moderated #testing of #prototypes

Posted on October 22nd, 2015

Steven Cohen @Validately spoke about the goals and advantages of prototype testing.

He started by reviewing his history in startups: one successful start and one failure. After these experiences he asked why does innovation fail?

False positives – estimated that 50% of a typical product’s features are not used (see http://versionone.com)

He then asked what you can learn from users

UI usability – can user complete a task by clicking on the appropriate links? – the inventor is too close to the product to know this

Real life usability – get a lot of false positives here. “It’s not a big enough problem.”

Are there blockers – usable in real life, such as privacy/security issues.

Better alternatives – something I’m using now. What is it like?

Does it solve a need? Needs to be big to change inertia. Need to get to “no” to show you know the issues. For instance introduce a price and ask if they would buy it (time, reputation, money)

Steven then presented two ways to test prototypes

Unmoderated – asynchronous testing as the user explores the functionality – good for UI usability

Less work, subjects pace themselves and give verbal/written feedback

Can’t learn real life issues

Moderated – live test interacting with the user and asking for opinions – better for real life usability

Deep user learning – usability

More time commitment

Steven did a simulated interview to show how the moderator would probe the user his/her actual usage of the product. One of the most important aspects is to get the user to make decisions on feature tradeoffs. The main tradeoff is whether a feature is work a specific amount of money and if not, why not and would there be a lower price point.

He wants to put up road blocks and see how people react – e.g. put up a page and ask for a credit card (even if you don’t keep the number) – see how much commitment. This is an attempt to get the feedback that users might not way to give you since they do not want to hurt your feelings or be confrontational.

Other observations were

If you need to pay your own existing customers to participate in a test, then the problem is not that important. Power users will probably always test for free.

Validately will start video recording the user, but does not believe eye tracking adds much since mouse movements correlate highly with eye movements

For unmoderated tests, if 5 of 6 users can do it. it’s probably adequate

For moderated test, start with 6 to 10 people. If there is a consensus, you can get by with fewer. If there is no consensus, then your persona (a straw man demo) is probably not well defined.

Once you have made a change, go back to the initial set of users and see if it fixes the problem – did we understand the comments?