Prototyping Methodology. Step 3 – Test

Attention: This is a super important step.

You’ve got your prototype ready (see Step 2 – Prototype), now what? Time to test it with your users. It sounds obvious, but you’d be surprised to find out how many people who do prototyping do not go outside and test it with their target audience. This is the whole point of doing this – so you can show it to real users (or prospects) and find out if you are going in the right direction or not. Otherwise, you miss out on the main value of prototyping and risking the success of your product.

In the first step (see Step 1 – Define), you’ve already decided on the high-level logistics – where you are going to test it and if you need any tools to do this. Assuming you know exactly who your users are and how to connect with them, here are the main points:

Recruitment

You can do it yourself or through an agency (quite expensive). DIY is definitely cheaper, but you have to invest your time in this, and this can get very time-consuming – searching, reaching out, scheduling, rescheduling, etc. This is my least enjoyable part, feels like a job I’d prefer delegating to someone else, but if you are bootstrapping, you can do it yourself. Hopefully, you have been investing in building your relationships beforehand, so you can ask your network to help with a few introductions. This will speed up the process. I am very grateful to have such a great network that is willing to help me with introductions or insights.

Number of participants

Many people think that you need at least 15-20 tests to get at least some decent statistical validity, but the guru of usability testing – Jakob Nielsen – discovered that the optimal number of test participants is 5. You should be able to discover about 75% of usability issues, which is more than enough to make adjustments and go for another round. If you could get 20 people who agreed to help you with that, you’d be better off splitting them into 4 rounds. Such approach will reveal more valuable results, than wasting all 20 right away. From my experience, you should recruit 7 people to have a 2-person buffer, in case they don’t show up or if the feedback they provide does not bring a lot of value.

Incentives

It varies and depends on how much does their time cost and how close your connection is. For bootstrapping, considering that the person doesn’t have to travel to your location, and you have some common connections, $10-20 Starbucks gift card for 15-30 minutes of their time would work. If you ask for more of their time, increase the amount appropriately. Ideally, try to understand the person’s preferences and choose the gift card that’s more relevant for this particular participant. For example, I chose David’s Tea instead of Starbucks for a person who was really into loose leaf tea. And when there wasn’t a Starbucks close by, I bought a Waves card because it was right across the street from their place.

Test Plan

The agenda completely depends on your current phase and what assumptions you are trying to validate. I will write a more detailed post on what questions to ask, how, and when.

For understanding the problem and concept testing, you want to ask more open-ended and broad questions. E.g. if you want to validate your idea, do not start with your solution. Rather ask them to describe their “flow” related to the problem you are trying to solve, how often they face this problem, in what scenarios and environments, how they deal with it, are they using existing products to help with this, how painful this is for them, and how satisfied they are with the current solution. For this phase, I just prefer asking several very open questions, e.g. “Describe me the last time you did {this}, walk me through the steps you took”, and let the conversation unfold itself. You want to uncover the underlying cause why they do what they do. And only then, show the prototype and ask what they think about this. One of my favourite methods to understand the core of the problem is 5-Whys.

For later phases, you want to discover usability problems, e.g. if they can find what they are looking for if they can understand the gestures, buttons, and the flow of your app. This is more of a traditional usability testing and I will be covering this topic in future posts, as there is a lot to learn there.

Before going out to users, you want to run through your test with somebody you work with (or a friend). This “rehearsal” has been valuable for me every single time. It’s so easy to miss small things here and there, and testing your test plan may help you find the gaps, and when you hear yourself you can feel if it flows smoothly. If I am working on a project alone, I just ask my wife to be the pretend user and simulate the testing session with her.

Moderating

Even if you prepared a great test plan and took care of the rest, what can ruin your testing is the way you conduct the test: how you present yourself, how you treat the person, how good you can control your ego and biases, when do you give incentive, how you respond to participant’s questions, etc. A few most important things to keep in mind: be polite and respectful, ask open-ended questions, let the user do the talking, don’t take it personally, and be grateful for their time and desire to help you.

Oh, and don’t forget to capture all their feedback (notes, audio, or video). Human’s memory is not a reliable data storage =) What was your name again? =)) What I find challenging is doing the testing and taking notes at the same time. You want to be following what the participant does with the prototype and maintain eye contact with them while they talk to you, not scribble stuff in your notebook. If possible, ask your partner or a friend to take notes, would help a lot, and you will less likely to miss something important if you can focus on the person, and your buddy can focus on the notes.

Logistics

What equipment do you need? Smartphone? Laptop? Don’t forget to charge them.

Do you need Internet? Will the test location have Internet?

Do you need to book a meeting room or just meet at a coffee shop location?

Don’t forget to remind the participant in advance, and give them all details about the timing, location, driving directions, and contact details.

PS this article is just a high-level overview of the testing process. I will be writing the whole series because there is a lot to learn.