How do we gain consensus within a team of highly diverse stakeholders?

These are some of the questions we typically ask when testing design concepts with users. In this article, we’ll tell you how we used a method called Rapid Iterative Testing and Evaluation, or RITE, to help us answer them, and we’ll offer some tips that you can try within your own team. We’ve found RITE to be a powerful yet low-risk way to test, validate, and improve on our designs, while engaging and building consensus among stakeholders.

Project Overview

As designers and researchers within the Citrix Customer Experience group, we often test prototypes in order to gather feedback and iron out usability issues before an engineer writes a single line of code. Last year, we decided to consider a new design direction for one of our virtualization software tools. The product was conceptually challenging for end users who were unfamiliar with the concept of virtualization, and our UX designers had created a new design that we believed might help those end users better understand the product. We also wanted to fix existing usability issues and update the product’s look and feel.

The big questions were, how could we accomplish all of these goals, and what would be the most effective and efficient way to do so?

The RITE Method

Our answer was to test our prototype using the RITE method. This method is similar to typical usability testing in that participants are asked to complete tasks using think-aloud protocol. The major difference is that, instead of waiting until the end of the study to gather the findings and suggest improvements, the team iterates on the design as soon as issues are discovered by one or two participants. In this way, designers can quickly test and get feedback on new solutions and ideas.

The Tests

Rather than testing wireframes, we chose to create high-fidelity mockups in order to create an experience that would seem “real” to users. In addition, we created a simple clickable prototype so that users could interact directly with the screen. While this made the process more labor intensive, we felt that it would also result in rich, insightful feedback.

We recruited six participants online who came into our usability lab at Citrix for one-hour sessions over the course of three days. Half of our participants were new to virtualization and only one had used our product before. The observers of the sessions included the designers, the UI writer, two engineers, and the product manager, most attending the sessions in-person (we used our GoToMeeting web conferencing tool to include those who could not travel to the testing site).

The Iterations

Because we were using the RITE method, we allocated some time after each session for the product team to discuss any changes that needed to be made before the next participant began. Overall, we went through two rounds of iterations (one round per day). Our changes to the design were not complete overhauls. Instead they included alterations like language rewrites, adding notifications, and reorganizing settings.

The Results

Using the RITE method was productive for us and we experienced positive results. Being able to iterate quickly on the design reduced our fear of failure because we could try something out and, if it didn’t work, try again. In addition, our team members were highly motivated by the opportunity to express opinions on what we had observed, which made for lively and engaging discussions. While the lead designer made the ultimate decision, being able to listen to and build on other observers’ insights helped make the end result better.

At the end of the study, we found that we were unable to address a few larger design issues within our testing timeframe. For example, some usability problems could not be solved with UI design because they stemmed from technical issues with the product. Our researchers compiled these findings, as well as recommendations for next steps, to share with the core team as well as other stakeholders who did not attend the sessions.

Tips for Using RITE

If you are considering using the RITE method for your product, here are some tips to help get you started:

Schedule 30 minutes after each session to talk about what people observed, make hypotheses about why users were confused or stuck on certain parts, and come up with ideas for design improvements.

Designate one person as the ultimate decision-maker on how the design should be changed before the next participant.

Have a designer and/or prototyper on-site who is willing and able to make quick iterations (note: if you don’t have the time to invest in high-fidelity prototypes, this technique would also work well with paper prototypes).

Be willing to fail early and often! The need to make decisions quickly before the next participant will help you to overcome any lingering fears.

In Summary

While we had productive results using RITE in this project, it’s unlikely that we’ll use the method for every study we do. RITE requires a large time commitment and deep involvement from the core product team. We believe that this type of testing is most successful when all the team members are able to observe the sessions together in person, so that discussions can happen face-to-face and the team can implement changes quickly. For us, the cost of flying out team members to the design site was worthwhile, as it helped us to effectively engage and collaborate with engineering and product management.

Have you had successes (or failures) using iterative testing methods? Let us know what worked for you and what other methods you think we should try.

ABOUT THE AUTHOR(S)

Jenny Shirey is a product designer at Citrix, where she helps to improve people’s experience with the tools they use every day at work. She is an advocate for simple, holistic design, and has spoken at several international conferences on information design and design for behavior change.

Ann holds a Masters in Learning, Design & Technology from Stanford University and a Bachelors in Cognitive Science (Human-Computer Interaction specialization) from UC San Diego. She is passionate about improving the user experience for enterprise and consumer software products and has worked at companies including Intuit, Qualcomm, Apple, and Citrix.

Comments

We're using a version of RITE that is working well. We're testing 4 people for an hour each with 1 hour in between sessions, for just one day. The hour in between is sufficient to discuss and implement most of the small tweaks we need to make to get users through all the tasks. Then we analyse everything the next day and determine what more substantial changes we need. For this current project we're doing this each week for three weeks in a row (we would have preferred cycling every two weeks over a period of six weeks, but you work with the time you're given). So during the testing day we have little tweaks and then the next 4 days are bigger design changes. It's fast and furious but overall it seems the best way to get something meaningful out of a compressed timeline.

Great article. Thank you for sharing your experiences. I am a big fan of the RITE method. I was part of a study that tested paper prototypes using RITE method. We debriefed with our client after every session and at the end of the day we had better prototypes to test the next day. The design team was able to make quick changes to the designs based on participants' feedback and test the changes at the same time. By the end of the third day of testing our client was very happy as the team had a few designs to go ahead with along with some good findings to work on later.

I think this method is very useful in situations when you do not have the time to wait till the end of the study, and in an agile environment where we seek quick turnarounds.

Thanks for this article! In my interaction design class we are all designing entrepreneurial apps. I brought this method up to my teacher and we went through it in class. It was VERY helpful. Love to see more techniques for UX. I need to keep building my toolkit.

This is where we used our good judgment/research and design experience. Some of the problems that surfaced were very obvious design issues whereas others that surfaced were more "opportunistic " - we weren't sure how big of a problem they would really be. We iterated on the ones we felt were obvious and held back from making changes on the other ones until we gathered more data that it's a problem. Since this is qualitative testing, we would not be able to wait til we had enough statistical data, but with every ocurrance of a problem, we had more confidence about acting on it.

Our user research project was geared towards the users of one of our IT products, XenClient. These are IT admins. We specifically recruited for admins who may use our feature/interface that we were testing based on their workflows.

Our user research project was geared towards the users of one of our IT products, XenClient. These are IT admins. We specifically recruited for admins who may use our feature/interface that we were testing based on their workflows.

Jenny, Ann, Quynh – This is great, and I've worked on projects in the past that used similar rapid test/iterate methodologies. I'm curious how your team mitigates the problem of iterating on a design too soon—that is, after only one or two users—versus waiting to iterate until a more "statistically significant" number of users have tested the design.

It's been over a year since Mike posted but I thought I'd offer an answer to his question about attaining statistical significance. This kind of rapid testing cycle isn't about being statistically significant. It's about seeing what obvious (to the user) problems exist and getting rid of them so you can move forward quickly. You could build everything first and then test with a huge number of participants (thereby getting statistically significant results) but by that point you've invested a lot in that particular design. Many times it's better to test small numbers as you go so you can avoid becoming deeply entrenched in bad early decisions. Do one test cycle with a small number of participants, make changes, and repeat. Over the life of the project you'll have a large number of users backing up your work.