The Cons

Without moderation, there are no follow up questions — it’s harder to get to the ‘WHY’.

You need to spend more time filtering through the results.

Sensitive data/designs can leak easily. Make sure you get people’s consent when filming them with tools such as lookback.io.

Not the preferred testing method for participants who need assistive technology.

How to set it up – The process pipeline

I’ve tested a few setups in the last few months and you need to assess what works and integrates best with your own workflow, existing tools and the design/prototype you need to test. Here are my suggestions and learnings:

Step 1: Get participants

Access your own pool of users. On my own product website, I’ve automated a few key steps. For example; as soon as a new website visitor purchased a product, they will get an automated email from Customer.io inviting them to test new features or product ideas. I can filter my audience with a few criteria such as the type of purchased product, visitor’s location, site entry point, etc. and turn all of this on and off with one switch.

Access a new pool of users. There are several ways to recruit friends, family and everybody else via accessing your social media outlets, customer service tools and Amazon Mechanical Turk (M-Turk). Amazon M-Turk is extremely convenient to setup and you will get people to complete your task(s) almost instantly, as long as you recruit them from the US and avoid filtering down or using any of their participant selection tools. Facebook and Twitter ADs provide another great opportunity to narrow down your audience and launch a campaign for little money. You can get a simple campaign started on Facebook for around $20, although, it isn’t that predictable; you never know how many participants you’ll get and you need to incentivize their participation. And finally – consider using Intercoms chatbot (or any other) to trigger an email or notification on certain events. Depending on where you place your chatbot – either on your logged-in (authenticated) or on your logged-out pages, you can address existing customers or new prospects.

Step 2: Filter participants

In most cases, it’s legally necessary that you let people opt into your user testing sessions and communicate that you are about to film their behaviour and collect other personal data. So after the initial email or social media invite, I funnel prospects through a simple Google form, that adds data to a Google sheet. I’ve used Zapier to notify me, as soon as a new participant signed up for a study.

Step 3: Self-guided task completion

I assume you want to test a rough design concept and get early feedback about a solution you are uncertain about how well it performs in terms understanding and usability. The best way to go about this is to use your favourite prototyping tool — such as Axure RP or SketchApp/InvisionApp — to create a clickable prototype. The goal is to let people DO SOMETHING you can later observe.

Axure & Fullstory.com

I often use Axure, since it generates actual HTML/CSS/JS websites and you can add code snippets to the head tag for real-time user behaviour tracking (Amplitude, Google Analytics, Segment). Each of my prototypes starts with an intro screen that explains the main task and allows people to kick-off their session when they feel ready for it. Remember, short-term memory is limited and you want to provide a clue on each screen, so participants have a way to refresh their memory by clicking on a dynamic layer at the bottom or top of the page. This way, you can also see later in the video screen recordings how many people needed to check back on the task. If this number is substantial, consider tweaking your task to make it easier for participants to comprehend and understand your task. Remember, they are completely left alone and cannot reach out to someone with questions.

SketchApp/InvisionApp and lookback.io

Axure has a steep learning curve and you’ll probably find more designers familiar with SketchApp/InvisionApp. It’s really intuitive and easy to setup in combination with lookback.io as the final participant tracking and remote observation tool. The disadvantages are that you won’t be able to add a lot of fidelity and depth to your prototype interactions and tracking customization. More about that in the next step.

Step 4: Recording test sessions

In case you went with Axure as your preferred prototyping tool, you now have the ability to add code snippets to your header and body tags via the plugin feature on axshare.com (the place you host Axure prototypes by default). This step might take you a while, but; practice makes perfect and there aren’t many complicated hoops you have to go through after all. Swiftly head over to the Google Tag Manager (free account) and create a new account, with a link to your prototype (exclude the Axure sidebar). The new account will provide you with a JS code snippet which you add to your prototype head tag. Next, sprint over to Fullstory.com, create a free account and take the generated JS code snippet and add it to your prototype, again. Boom – you are done.

Visit your prototype, click around a bit and give it a few seconds before you check the recording on Fullstory.com. This setup is already quite satisfying and will save you a lot of time in the future. Fullstory.com comes with a lot of awesome features that allow you to segment users, track events, generate heatmaps and watch mouse behaviour on great quality video recordings. If you want to take it a step further, add segment.com to the game to further partition visitor types and combine all multiple tracking tools into one place and create a mega dashboard.

In case you went with SketchApp/InvisionApp – then the steps you have to go through to get basic video/audio recordings of your sessions are fewer. Just generate your prototype in InvisionApp, grab the link and post it to lookback.io. The testing and recording tool will then help you setup everything else. Keep in mind that lookback.io asks participants to install a Google Chrome extension (desktop) or an app (mobile), which might cause a barrier and you have to help participants to overcome this step. Provide simple instructions and make sure Chrome is the preferred browser (make it a participant selection criteria). TIP: You need advanced users when going with InvisionApp/lookback.io, since internet newbies or older people can feel quickly overwhelmed by the installation steps they have to go through. A big plus for lookback.io is, that it works extremely well on mobile and you get a recording of your participants’ facial expressions and what they do on the screen in parallel. It is just magical. Of course – lookback.io also works with Axure. Mix and match as you please.

Step 5: User segmentation and sharing

Fullstory.com already gives you a lot of insights and customization tools, to analyze user behaviour. If you work in larger product teams, you want to add the findings to a ‘team dashboard’, so all the data streams from marketing, design and engineering collate in one place. Like many other product teams, I use segment.com to connect the data stream from Fullstory.com and feedback results to the teams. If you do that, delete the JS code snippets from your Axure head tag and only work with the JS code snippet from segment.com – otherwise you will screw up the results.

ABOUT US

REWORK is a resource for tracking global innovation and the transformation of work. As a global discovery engine, REWORK discusses, explains and highlights the best, most disruptive and most useful ideas of the 21st Century related to changes in the ways we create, play and work – showing which themes and which ideas can best be used to drive creativity and change in your workplace, office or industry. Our network of knowledge-bots delivers you the latest discoveries from every part in the world….in real time. Our purpose is to keep you and your teams up to date, to inspire you as you create your future. From Berlin to London, from Beijing to San Francisco, learn from the world’s most forward-thinking projects how to disrupt, innovate, collaborate, learn new skills and do work better.