Doing the Dirty Work of User Testing

We’ve all heard the old adage: test early and test often. Yet user testing can be a daunting task for some. After all, there’s gathering the participants, doing the actual test, documenting and analyzing the results. Not to mention, making sense of the feedback, figuring out what to prioritize before taking action. And that can feel like dirty work.

In this post, we’ll break down how to make sure that what you’ve discovered during testing is properly documented and turned into solutions that create a better user experience.

Now this article isn’t a “how to” guide on user testing. If you want to know more about performing a user test then check out UXPin’s free e-book, The Guide to Usability Testing​.

Formal vs. Lean Methodology

When I started my career, everything was documented in a very formal way with big reports, which were presented to clients in hour-long presentations. To me, formal documentation and multiple-paged deliverables encourages discussion rather than action — but I’ll get to that later. That was how I did things back then. Nowadays I’m all about action rather than discussion rather than action.

After my time at Trade Me, I moved to Optimal Experience, NZ’s leading UX design consultancy, which was recently acquired by Price Waterhouse Coopers. We have recently moved away from formal approaches towards our adaptation of Google Venture’s design sprints. No one in New Zealand seems to be doing this type of work, which has allowed our team to pioneer the way these sprints run. We’ve discovered new ways to do things and the best way to create action over discussion. A lot of the time it’s “load, fire, aim” instead of “load, aim, fire.”But doing things this way allows you to learn quickly.

With that, let’s look at how we “do the dirty work” in both formal and lean environments.

Great Testing Documentation Takes Preparation

Creating great documentation and solutions begins when preparing for user testing. It’s like a great meal. You can’t have something that tastes amazing if you don’t prepare the ingredients, trim the meat and cut the veggies ahead of time. Same with user testing documentation. If you don’t have everything lined up beforehand, you won’t have anything spectacular to document.

First things first,, you need to organize the roles of everyone involved. The types of people involved in a user testing session include:

The participant: This is the person who represents one of your user personas that you want to test. If you want to learn more about creating user personas, you can check out UXPin’s previous blog post on the subject.

The facilitator: T​he person who is in charge of facilitating the process with the participant. The facilitator asks the questions, probes to discover the true problem, and makes the test feel as realistic as possible.

The scribe: This is the person who is in charge of noting everything that happens during the testing session, such as w​hat participants say, what participants do and the questions the facilitator asks. This is really important because it’s the safety net. These notes are the first place anyone looks if someone wants to know what happened and when.

The clients: It’s really important to have your clients observe the user test. This is so they can see first hand how participants struggle with the product. This will motivate them to improve their product.

The designers and developers: You definitely want the designers and developers who have created the product to observe the test. Seeing their creations fail will bring them back down to earth and motivate them to design and develop from their users’ perspective.

The wrangler: Have someone to wrangle the clients. The wrangler’s job is to ensure the clients leave their discussions until after the session, keeping everyone concentrated on noting down as many findings as possible.

Obviously you can’t fit all of these people into the same room, so it’s a good idea to arrange screen sharing software to stream what’s happening on the participant’s device to an observation room where the clients, the wrangler and the scribe sit.

What You Need to Document During the Testing Sessions

When the session is happening, the clients and the wrangler will be taking down as many findings as they can on Post-it notes. Each note should be color coded, based on the type of finding you’re taking down. Let’s take a look at how this works.

Positive = green: For example, participant registered for our service without an issues.

Number the Post-its based on the finding: This will help you track this finding back to the notes to see what exactly was happening. It will also help you identify what finding relates to what persona. For example, all the participants who don’t shop online couldn’t find the shopping cart.

Only put one finding on each note: Once you write down your finding, stick it up on a surface dedicated to findings. Big foam boards are good for this.

Here’s a few other things in mind:

Record the session: You’ll want to record the session using screen-capturing software, such as Silverback. This is another safety net and can be shared with clients and stakeholders who can’t make it to the session. If you have a good dedicated note taker., you won’t need to go back to the video.

Cluster notes: In between sessions, start clustering the notes based on findings. s. For example, there may be a cluster of findings relating positives, negatives and observations in the onboarding process. Discussion of the findings can also occur between the sessions.

Don’t rush into changes: Clients will probably want to make changes to the designs based on one or two sessions. It’s your responsibility to remind them that more evidence is needed before making changes. Four participants is the minimum you’d want to test with before making changes. If you were to make changes after the fourth participant then the finding will have to be really clear. For example, 100% of our participants found this issue so far. To find out more on confidence levels of findings and when to make changes read ​this article​.

At the end, you’ll have a bunch of formal notes that the dedicated note taker has taken down, video recordings of your participant’s sessions, and a surface full of Post-it notes that are clustered into similar findings.

As we said, there’s two approaches you can take after the sessions are complete, which we’ll dive deep into next:

The formal approach: a document handed over to the client with a summary of findings.

The lean approach:an iterative process that delivers working prototypes as documentation.

The Formal Approach

In a formal approach, all the findings are summarized into a document with the impact of the findings and the consultant’s recommendation. One of the main pain points with formal reporting is ranking findings. It’s easy to prioritize findings into two groups; those that prevent participants from achieving their task and those that just trip them up, but it’s difficult to rank them beyond that.

We usually categorize findings based on three different criteria.

Minor: Issues that cause irritation/confusion or rarely occur.

Major: Issues that make it difficult to use the site, but users might recover.

Severe: Issues that prevent participants from achieving a task.

The way that findings are categorized completely depends on the perspective you’re categorizing them from: do you categorize them by what’s important to users? What’s important to clients? What’s easiest or hardest to fix? This is a major pain point because you can’t take all perspectives into account in a formal report.

Another pain point is categorizing the findings into the three categories. You can still get one category with 20+ findings. Surely some of those 20+ findings are more important than others, but how do you know?

The Downside of the Formal Approach

In a formal document, you have to prioritize the findings for the developers and designers. This will give them a starting place before implementing a solution. And because time and budget are often short, you’ll have to also prioritize the solutions. Why? Because you can’t simply hand a client a document that has just findings — they won’t know what to do next unless you offer a game plan.

This is where your UX and interaction design knowledge comes in handy. At Optimal Experience, we get two members of our team to recommended solutions.

However, the big downside is that these are solutions coming from the consultant’s perspective, not the client’s or user’s. Sure, you can attempt to put these hats on but there’s no way you can design from the same perspective as these people.

The final issue with formally documentation is that quite often the clients read the document once, file it away for future reference, and then forget about it.

The Lean Approach

The lean approach is far more effective than formally documenting your user testing findings because it favors action over documentation. The end result is a working prototype instead of a formal report. The difference between the lean and formal approach happens after the testing sessions. Let’s walk through this.

Go over the notes: Once the sessions have been completed, everyone except the participants, will go over the Post-it notes and ensure all the findings are accurately clustered and that none missing. This gives the team a chance to have a good discussion, that will help build their knowledge of what’s happening with their current design and why.

Come up with solutions: The next step is to come up with solutions to your findings. The process used to do this is called co-design, which is a collaborative way to design solutions. The key reason to use the co-design approach is to generate solutions from different perspectives, by involving different types of people. For example, you’ll get much better solutions if you involve UX and IXD consultants, clients, customers, customer support staff for your product, and other industry experts.

Diverging and Converging on a Problem

Let’s take a closer look at co-design.This collaborative method works by diverging and converging on a problem. In other words, creating as many solutions as possible and narrowing down on the best ones.

You begin this process by diverging on a cluster of Post-it notes from the user test. Each member creates as many solutions as possible and then presents them back to the group. After hearing everyone’s presentation, the group writes feedback on Post-it notes and sticks them on the idea the feedback relates to. Next, everyone votes on what they think are the best ideas by sticking a red dot on them.

The group then converges on the solutions with the most votes. The team starts by refining their designs based on their feedback, what was voted on, and stealing other people’s ideas. Once this is done, the group votes again to settle on the top one or two solutions for that cluster of findings. This process is repeated until all the clusters of findings have gone through a divergence and convergence.

This type of session takes between one and two days depending on how many clusters of findings you need to design solutions for.If you want to know more about this process then read this article by the ​Google Ventures design team. Although, we’ve also just started playing with the new qualitative research tool called ​Reframer by Optimal Workshop​, which is quite helpful for ferreting out trends in user research.

Creating a prototype gives a working example of how your solutions should work. These wireframes are annotated with notes to give an explanation of the interactions and key changes. It also gives your clients something to hand over to their designers and developers for them to create.

The Value and Downside of the Lean Approach

Processing findings using a lean approach is far more effective than documenting findings in a formal way. Firstly, because all the findings are already documented on Post-it notes, meaning putting them into a report will basically be repetitive data entry. Secondly, because you can have a cross functional team of people who have different perspectives creating solutions.

Post-its can become lost: Sticky notes,can easily be lost. They don’t stick forever and often go missing after a session. Take photos so reference them later, just in case.. Be warned, however, that photos aren’t a great means of documenting sticky notes. The photos can often turn out blurry.

Hard to prioritize notes: Also, it’s hard to prioritize your findings with sticky notes. This isn’t so bad because you are able to tackle a cluster of findings during co-design sessions. . For example, if there is a cluster relating to on-boarding you can co-design the entire on-boarding process, covering off all findings in that particular cluster, compared to prioritizing individual findings in the formal method.

Another thing to remember: the process doesn’t stop at creating the working prototype. If you were to run through the full lean process properly, you would user test the prototype to see if your proposed solutions solve the findings. The first test won’t be 100% effective in solving all of the findings, so you will need to repeat the process of user testing, co-designing and prototyping again. Ideally, this process will be repeated until the best solution for the product is found.

One last caveat. With many iterations, it may be hard to keep track of what solution relates to what finding. So if a stakeholder asks why a change has been made, be prepared to clearly explain your design decisions.

Takeaway

Neither approach is perfect. And hopefully with tools like ​Reframer​, this process will become a lot less painful.

In the meantime, it’d be best to go with the lean method because it favors action over documentation. At the end of the process, you’ll have a working prototype that teams can implement instead of a document that gets filed away.

Note from the editors: If you’d like to learn more about usability testing, check out UXPin’s free 100+ Guide to Usability Testing. Over 30 types of usability tests are described with practical tips.

Found this useful? Share with

Alex Gamble is an experience designer at Price Waterhouse Coopers New Zealand. He has helped a variety of businesses, from small start-ups to big corporates, develop user-centered products. Alex's goal is to bring forward a lean product revolution. Follow him on twitter

Still hungry for the design?

UXPin is a product design platform used by the best designers on the planet. Let your team easily
design, collaborate, and present from low-fidelity wireframes to fully-interactive prototypes.