Sustainable User Research

The title of my new column, Practical Usability, reflects my intention
to explore practical and useful aspects of usability that are
relevant to UX designers and user researchers, as opposed to focusing on
an academic or narrowly defined vision of usability. In this column, I’ll
use a broad definition of usability, encompassing all user research activities
that contribute to a better user experience.

“In-person research is much better than remote
research. But … now, more than ever,
it’s important to determine when it’s feasible to save money and
the environment by conducting more user research remotely.”

Traditionally, user research involves directly observing and
talking with people in the context of their work or play. Either researchers
travel to observe participants in their natural environments or participants
travel to a usability lab or focus-group facility. How better to understand
how people use a product or technology than to observe them using it firsthand?

Although lack of time and money for travel have always been
barriers to conducting in-person user research, the current recession and
concerns about global warming and wasted resources have pressured businesses
to
cut back on business travel and conduct more business remotely. Should user
research be any different?

There’s no denying that, in most cases, in-person research
is much better than remote research. But who are we to say that creating good
user experiences is more important than the environment? Now, more than ever,
it’s important to determine when it’s feasible to save money and the environment
by conducting more user research remotely.

“Remote user research can be either moderated
or unmoderated.”

Remote user research can be either moderated or unmoderated.
In both cases, the participants and researcher are in separate locations.
However, in moderated, remote user research, the researcher and the
participants go through the research activity together virtually, while in
unmoderated,
remote user research, the researcher is not involved during the study.

In this column, I’ll discuss the advantages and
disadvantages of moderated and unmoderated, remote user research. Then, I’ll
reflect on some deciding factors for conducting either in-person or remote
user research—or
both in combination. Understanding all of these considerations can help you
to decide when it’s most appropriate to use in-person or remote methods
of user research—and
if the latter, whether to do moderated or unmoderated research—or to combine
both approaches and get
the best of both worlds.

Moderated, Remote User Research

“During a moderated, remote user research session,
the researcher and a participant typically communicate by phone and use software
that lets the researcher observe the participant’s computer screen.”

During a moderated, remote user research session, the
researcher and a participant typically communicate by phone and use software that
lets the researcher observe the participant’s computer screen. In fact, Web-conferencing
applications such as WebEx or software like TechSmith’s UserVue allow the
researcher and participant to both see and control each other’s computers. It’s possible to record a user research session,
using screen-recording software such as TechSmith’s Morae and a microphone. Using a Webcam to view the participant is also an option, as I’ll explore later.

Advantages of Moderated, Remote User Research

Eliminating Travel and Saving Time, Money, and the Environment

Remote user research eliminates travel, thereby saving time
and money. You can avoid wasted time in airports, there’s no need to rent or
reserve a usability lab or other research facility, there’s no wear and tear
on your equipment, and usually, incentives can be lower, because participants
don’t have to travel to your lab’s location. Obviously, reducing travel offers
important environmental benefits, too.

Including Participants from Anywhere in the World Is Easy

“Remote research lets you include people who
would otherwise be left out, so you can have a more representative sample.”

If you conduct only in-person user research, budget and time
considerations often limit you to using only local participants or traveling to just a few locations. Remote research
lets you
include people who would otherwise be left out, so you can have a more
representative sample. Moreover, it’s often easier to recruit participants
when they don’t have to travel to a usability lab to participate.

Conducting Usability Testing Sessions That Are More Representative
of Real Situations

In remote usability testing, participants use their own
computers in their home or workplace, where they typically feel more
comfortable than in a usability lab. In such environments, you can witness
participants’ Internet network speed, browser choice, bookmarked sites, and
so forth, as well as the natural interruptions and interactions that occur in
real-world environments.

Thinking Aloud Is More Natural for Participants

“Participants are more likely to communicate
with you verbally and explicitly narrate their actions as they move through
a task, which gives you a better understanding of what they are thinking and
doing.”

Because you can’t see the participants, it’s even more
important than usual for them to think aloud as they work, so you can
understand what they are doing. When a participant stops thinking aloud during
an in-person testing session, you can usually see why—he or she may be reading,
confused, in deep concentration, or frustrated and about to give up. But
in a
remote session, you miss these visual cues.

Fortunately, because participants can’t see you either, the
need to think aloud is obvious, and doing so comes more naturally to
participants who are working remotely. Participants are more likely to
communicate with you verbally and explicitly narrate their actions as they
move
through a task, which gives you a better understanding of what they are thinking
and doing.

Focusing the Researcher on Observing Rather Than Interacting

“Greater focus can lead to richer insights
and more fruitful discussions with participants.”

Engaging with participants in remote locations removes the
dynamics of body language and eye contact, allowing you to focus on your computer
screen and your notes. Greater focus can lead to richer insights and more
fruitful discussions with participants. Plus, during a remote session, you
can log events on your computer, saving time, making data analysis easier, and—as
an extra environmental benefit—not using up paper.

Disadvantages of Moderated, Remote User Research

Seeing Participants and Their Context Isn’t Possible

“During a remote user research session, the
only context you can see is a participant’s computer screen. You miss
seeing the participant’s facial expressions and body language….”

Since participants’ computer interactions are the focal
point of user research, it’s easy to imagine how you could successfully
conduct remote usability testing. But remote contextual
inquiry is an oxymoron. How can a user interview be contextual when
you can’t see the user’s context firsthand?

During a remote user research session, the only context you
can see is a participant’s
computer
screen. You miss seeing the participant’s facial expressions and body
language, as well as any interactions with other people, the surroundings,
or supplemental materials—for
example, Post-it notes, printouts, calendars, or documents.

You can use Webcams to capture participants’ facial
expressions and body language. But sending a Webcam to each participant is
expensive and coordinating the installation of Webcams and software for
multiple
participants can be difficult, and seeing participants’ facial
expressions through the typically delayed video a Webcam provides is rarely
worth the effort. Sending
disposable cameras to participants so they can photograph their
surroundings and materials is a much less expensive way of filling in some
of the details you miss when doing remote user research.

Establishing Rapport with Participants Can Be More Difficult

“You’ll have to work harder to establish
a sense of rapport with participants. ”

Distance and the technology for conducting remote research
can add a level of impersonality to the interactions between you and participants
during test sessions. You’ll have to work harder to establish a sense
of rapport with participants.

Maintaining Control Over the Situation Is More Difficult

Because you’re not physically present, you have much less
control over a research session. Frequently, coworkers, family members, phone
calls, instant messages, and even pets interrupt participants. Although these
interruptions provide a sense of participants’ real lives, which is part
of what
you want in a contextual inquiry, they can completely disrupt tasks during
usability testing and ruin your studies’ results.

Participants May Invite Others to Attend

For some reason, participants in remote user research occasionally
invite other people to attend a test session—often a coworker they believe
might have useful information. Perhaps the use of a conference call and Web-conferencing
software causes participants to think of a user research session as a meeting.
If you’re lucky, the participant will inform you that the extra person
is present. However, your inability to see who else might be lurking in the
background makes it difficult to eliminate the presence of these extra attendees.

Web-Conferencing Software Can Have Technical Problems

It happens less often than it used to, but some participants
still have problems using Web-conferencing software. Those with a slow Internet
connection may not be able to connect and share their screen. Slow connections
can also delay your view of a participant’s actions.

Unmoderated, Remote User Research

“While moderated, remote user research separates
the researcher and participants by distance, unmoderated, remote user research
separates the researcher and participants by both time and distance.”

While moderated, remote user research separates the
researcher and participants by distance, unmoderated, remote user research
separates the researcher and participants by both time and distance.
Although unmoderated, remote user research offers the advantage of letting
you reach
more participants, expending less time and money, it also brings the
major disadvantage of further separating the researcher from participants.

In unmoderated, remote user research, the researcher sets up
a study, then participants complete the study on their own, without any involvement
of the researcher. Finally, the researcher views the results collected
by
the software the participant used during the study.

Surveys are the oldest and most familiar type of unmoderated
user research, while tools for online card sorting and self-guided usability
testing are more recent inventions. Today, a wide variety of online
tools
combine questionnaires, usability test tasks, and Web analytics. These
tools can track the paths participants took, how long they spent on each
page, and where they clicked during a test task. Unmoderated, remote user
research is a useful addition to the researcher’s toolkit, but it, too,
has advantages and disadvantages.

Tools for Unmoderated, Remote User Research Do a Lot of
the Work for You

After setting up a study and contacting the participants via email, you
are free to work on other things, while the participants complete the study
on
their own time. Unmoderated user research tools automatically collect, analyze,
and present data in useful visualizations, eliminating time-consuming data
collection, organization, and analysis tasks for you.

Including a Large Number of Participants Is Easy

Because participants complete a study on their own time and
unmoderated user research tools automatically collect and present data
for you, it’s easy to include
hundreds or even thousands of participants at a fraction of the cost and
effort
of in-person research. This is especially helpful for activities that require
a
large sample size—such as card sorting, for which you need 30 or more
participants to generalize the results to a larger user population. [1]

Including Participants from Anywhere in the World Is Easy

“With no geographical limits and higher numbers
of participants, you can aim to get a truly representative sample of your users.”

As with moderated, remote user research, you can easily
reach participants anywhere in the world. Again, this enables you to include
many participants who would otherwise be left out because of time constraints
and travel costs. With no geographical limits and higher numbers of participants,
you can aim to get a truly representative sample of your users.

Eliminating Travel and Saving Time, Money, and the Environment

Unmoderated, remote user research also eliminates the need
for travel and provides the same savings of time and money as moderated testing
does. Again, this eliminates the negative environmental impacts of travel.

Disadvantages
of Unmoderated, Remote User Research

“Unmoderated, remote user research is best
suited to gathering quantitative information.”

While moderated, remote user research prevents you from
observing a participant’s context, unmoderated, remote research additionally
makes any dialogue with the participants impossible. Although most
unmoderated, remote user research tools support survey questions or allow
participants to make comments, such responses tend to be very shallow and
limited in comparison to in-person discussions. Because of this, unmoderated,
remote user research is best suited to gathering quantitative information.

Unmoderated, Remote User Research Tools Have Limited Scope

The best unmoderated, remote user research tools tend to do
only one thing well. For example, online card sorting tools communicate how
users would organize and label a set of information, but they don’t do
much beyond that. Similarly, most online usability testing tools are good
at things like testing findability tasks and posing survey questions, but
they can’t capture
the complex interactions and user behavior you could observe either in-person
or through remote, moderated usability testing.

Unmoderated, Remote User Research Offers the Least Control
Over Participants

“With no researcher present during a test session,
participants are more likely to get distracted by interruptions, take breaks,
or put less serious thought and effort into completing a study.”

With no researcher present during a test session, participants are more likely
to get distracted by interruptions, take breaks, or put less serious thought
and effort into completing tasks. With many online tools, you have no way
of knowing whether a participant suddenly got distracted by a phone call,
went out for a snack, or just took a long time to complete a task. Obviously,
this can seriously throw off the results of your study. Without a moderator,
test sessions need to be much shorter to ensure participants can complete
them before losing interest.

There’s No Moderator to Prevent Errors and Misunderstandings

Because there’s no researcher to explain the study or help
participants complete tasks when they get stuck, unmoderated, remote user
research tools must provide extremely clear instructions and have very usable
user interfaces to prevent bad data from corrupting a study’s results.

Deciding Between In-Person or Remote User Research

“Choosing between in-person or remote user
research depends on what you’re trying to accomplish. For many studies,
it’s
very important to directly observe and have in-depth discussions with participants.”

Choosing between in-person or remote user research depends on
what you’re trying to accomplish. For many studies, it’s very important
to directly observe and have in-depth discussions with participants. For
example, technology workflow studies require seeing the participants’ environment,
materials, and people with whom they interact. However, for other activities
such as card sorting, using automated tools offers time-saving benefits,
because they can collect, analyze, and present the data from a large number
of participants, in many different locations.

To determine whether remote user research might be
acceptable, first define your research goals, then ask yourself the following
questions:

How many participants do you need?

Where are the participants?

Is it necessary to include participants in many
locations?

How much time do you have to conduct the user research?

How much money do you have for user research?

Are you trying to gather qualitative or
quantitative data?

How important is it to observe the context of
use around what you’re studying?

How important is in-depth discussion with
participants?

What will you miss by not being with participants in person?

Consider Doing Both Remote and In-Person User Research

“Remote and in-person user research can complement
each other, providing the best of both worlds.”

Of course, you don’t have to choose one method of user research
over the other. Remote and in-person
user research can complement each other, providing the best of both worlds.
For example, you can conduct remote usability testing or contextual
inquiries to include participants in distant locations and in-person
sessions with local participants.

To make up for the lack of qualitative data from unmoderated
card sorting or usability testing studies, conduct some in-person card sorts
or
test sessions to gather some qualitative feedback or further investigate
issues you’ve discovered during the unmoderated testing.

Both in-person and remote user research have their
appropriate place in the usability professional’s toolkit. By using these
techniques wisely, you can take advantage of their strengths, minimize their
weaknesses, and save time, money, and the environment by eliminating unnecessary
travel.

Reference

1 Comment

Allan Roger wrote:

Yeah, I use the RHUB remote user research appliance for conducting remote sessions with my participants. And you are right. It allows host and participant to communicate by phone typically and allows the host to observe the participant’s computer screen.