Santa Barbara User Experience Testing

12:32 PM

Local Loans User Experience Testing

--

When designing an onboarding flow for Local Loan’s borrower app, our goal was to increase the number of users who input their
invite code and connected with a loan officer.Our hypothesis was that once the borrower has input their invite code both the borrower & loan officer
would receive more value, and thus have a higher likelihood of converting the loan officer into
a paid customer.Here’s what the borrower home screen looked like at the time:

The borrower would have to tap on "Invite Code" & then input their invite code they received from the loan officer. The concept was sound, but the user experience still needed a bit of polish.

The user onboarding flow was simple with no tutorial, but required the user to know to tap on "Invite Code" even though the green "Get Pre-Qualified" button was attracting all of their attention.

Recruiting the right
people

--

In order to get the right results from the user test we
needed to recruit the right users.For our Local Loans user onboarding designs, we
wanted to get feedback from 10 people.To recruit them for the study, we sent emails like this one to unengaged
trial users.

--

Hi Steve,

My name is Derren and I am on the research team at Local
Loans.I see you had signed up for a
trial of Local Loans recently.

How’s everything going?

My team is interested in improving the Local Loans
experience.Would you be open to
touching base on a quick call to discuss why you signed up for Local Loans and
your experience with invite codes thus far?If you’re available
this Thursday (4/21), Friday (4/22), Monday (4/25), or Tuesday (4/26), I’d love
to schedule some time to talk.

I just want to clarify that this is not a sales call.You will be speaking with myself and a Local
Loans product manager.

The goal of the call is to learn more about your experience
so far with Local Loans, why you started a trial, and to show you a new feature
we’re working on to hear your feedback on it.This is not a sales call – it is purely for research, but we’re happy to
answer any questions you have about Local Loans along the way.For taking my call, I will be happy to offer
you a $25 Amazon gift card.

The call will take around 30 minutes and will be at a time
of your choosing within the next week.I
look forward to hearing from you.

Thanks,

Derren Ohanian

Usability Testing & Research

Local Loans

--

I focus on a few key points during these e-mails:

I make it very clear that it is not a sales call.This will improve the odds of people
committing to the call.

Include an incentive to improve the odds of
people committing to the call.

Select users that have signed up, but did not
complete the task you wanted them to.This will improve quality of results from potential users. In this case we were looking for loan officers that have not sent out invite codes to borrowers or realtors yet.

Generally response rates to these types of emails are pretty
high (around 50%), so I will send out no more than 10 at a time until I have
the desired number of tests scheduled.If people do not respond to the first email I make sure to follow up a
second time before dropping them.

At Local Loans, we schedule these calls virtually and use Appear.in
to run them.When possible we will
schedule meetings in person & videotape sessions.

Appear.in is awesome & free. Little is better in the world than being awesome & free!

Creating the User Testing Process

--

I follow the same process for each user test I perform. This helps ensure that I
can compare my observations apples to apples, and draw logical conclusions
without bias from my findings.There are
three primary parts to my process.

Background questions

Task-based walkthrough

Wrap up

Background Questions

First I ask the basic questions to get to know the user, put
them at ease, and discover relevant information about their background
with the product thus far.Here are some
background questions we used in the Local Loans user test:

How did you find out about Local Loans?

What do you know about us?

How would you explain Local Loans to a friend?

How do you keep track of referrals & leads?

How do you find potential leads or clients?

Why did you decide to start a trial?

What expectations did you have about what Local Loans is?
Did the trial match that?

What have you done with Local Loans so far?

What value have you found in Local Loans?

Why haven’t you used the invite codes to lock in potential leads?

I tend to stray away from yes or no questions to get the
person to open up about their experience.

Task-based
Walkthrough

Next, we move to the task-based portion of the
interview.Before the user takes any
action, we set the context.In our Local
Loans test, we told one user to “Imagine that you’ve been doing research on the
Local Loans app and decided to start a free trial.You know that you’ve been looking for a way
to lock in new business.Start the trial and walk me through what you are
doing.”

We knew this user was looking for a way to lock in leads through background questions and our recruiting processes,
and that’s what we use to make the tasks contextual.We could have framed the task for another
user by saying, “Your boss said you need to sign up for Local Loans and decided
whether it’s the best lender solution for your company.Start the trial and explain what you are
looking for.How do you decide that
this is the best solution?”

During the task based portion of the interview, I ensure
that we:

Record the session so I can share the observation with my
colleagues later on.

Encourage the user to think out loud.I want to hear everything that goes through
the user’s head when completing the tasks.

Tell the user that I am testing my designs & not them.This helps put them at ease by letting them know they cannot do anything
wrong.

Get every user to go through the same tasks without my
help.This allows me to aggregate my
findings and agree on design changes with my team.

At Local
Loans for this test we gave users a beta product to test with. However, I have used staging versions of
software, hand drawn mockups, or clickable prototypes built in Invision as well.

With each new screen or step in the task, I allow the user
to think out loud and make a note of his or her initial impressions.If I feel that there is more to learn then I
will ask the user follow up questions to gain further insight.

Wrap Up

Once the user has completed all the tasks I’ll wrap up the
interview. Letting them know that they have reached the end of the tasks, their feedback was helpful and insightful, before finally thanking them for their time and saying goodbye. Shortly thereafter, I follow up with a thank you email and the promised compensation.

After each interview, I’ll take 10 minutes to go over
everything I just heard with my team.This ensures that everyone who was watching the interviews is on the
same page and leaves a little time for discussion and
synthesis.This 10 minutes potentially saves hours of
debating and going over video later. It is vitally important to digest the most important takeaways, document them, and decide on the next step together to keep the process moving forward smoothly.

Turning Observations into Action

After all the interviews have been completed its time to aggregate and see how many users completed the tasks. This could be as simple as:

Sharing questions or comments that came up during testing tend to be helpful as well. For example, I took note of things like:

“What
if I want to service multiple areas?” - This user wanted the ability to
represent multiple localities to be able to receive more leads.This indicated an additional potential
revenue

"The underwriting
guidelines allows me to completely do my job on the go. This is great!” – This indicated that the feature
set was correctly chosen for the product.

Conducting these customer interviews, there were three
themes that emerged:

Loan Officers were sending invite codes out, but Borrowers were confused about how to input the invite codes properly.

Loan Officers wanted to share the invite
code with realtors as that is their primary means of referrals.

Add
tooltips to help explain what each field is during the sign up process to
reduce confusion.

From these observations, we made the following changes to the product:

We
added 5 marketing slides to the user onboarding flow at the start of the
app.

We added the ability for users to share
their invite code with realtors & track the amount of referrals they receive
from realtors.

We
extended the onboarding flow to include tooltips and additional step by
step walkthrough once the user was inside of the dashboard to walk the
user through the most valued features of the app.

After making these changes, we shipped a variation of the
below screens and saw a 400% lift in the number of invite code use after one
week. All because we solicited feedback on a design before building it out.

Instead of relying on the borrower to find out where the invite code was we prompted each user at the beginning of the app to input their invite code. If the user did not have a code they would fall into the top flow which included new user onboarding marketing materials. If they did have a code they would link directly with the loan officer & begin their pre qualification process.

You Might Also Like

0
comments

About

DNA specializes in web and mobile app design and development in Santa Barbara. We have been in business for 10+ years while working with enterprise level companies & startups to help build custom digital products.