The New Way to Conduct Real-Time User Research

Overview

Conducting research using your live product sounds pretty crazy right? It seems almost backwards and counter-intuitive. Why would you go through all this trouble researching, designing, and building a new feature to just repeat step 1 (research) again?

This reason is contextual observation. Contextual observation is the process of observing how your product meshes with the day-to-day lives of your users. There is absolutely no substitute for observing people using your product in their own environment. Think of this as an out-of-the-lab usability experiment using a production-ready feature.

Benefits of Contextual Observation

Contextual observation is not meant to replace your initial research, nor should it be a ‘bug-catcher’ or a test of functional efficacy. Rather, it is used to assess user engagement with a new feature. Additional benefits of contextual observation include:

Quantitative insight into your feature’s performance. How well is it scaling? Are users using it? How is it impacting your system? How are your levels of engagement?

Contextual insight into your feature’s efficacy. Do users see the new feature? How is it meshing within the context on your existing feature set? Are people using it as intended? Are people using it once and then not using it again?

Qualitative feedback. Are users complaining? Are they happy? Are they neutral?

There are two primary ways to approach contextual observation:

Launch the feature and observe

Use embedded analytics to asses engagement and performance.

Contextual inquiry

Use your real users as research participants. Select a few ‘lucky ones’ and ask for their feedback. It’s important to ask users who are using the feature and users who are NOT using the feature. You can do this through Intercom-style messages, emails, or subtle feedback call-to-actions on your website.

Caveats

As a developer or product manager, you may see live contextual observation as risky or just a semantic way to frame a normal launch. “Well, what if the feature breaks?” “How is this different than normally launching a feature?”

The answer is that normal feature release cycles do not let you:

Gradually rollout a new feature to your users (1% -> 5% … 20% and so forth)

Target segments of your users or customize who gets to see the features

Turn off the feature if it breaks without harming your product’s performance

Turn on the feature with one click

Using Feature Flags

This is where feature flag driven development (FFDD) comes into play. FFDD lets you wrap your feature in a flag so you can control its life cycle from development to QA and production. This allow you to control rollouts, target users, and control your feature’s visibility throughout development. More importantly, it mitigates risk by allowing you to continuously deliver features without worrying about user rejection and performance degradation. Everything you release gets placed in a flag, so you can toggle it on or off to whomever you want, when you want.

Imagine the power of flagging for contextual observation. For example, you can get real user feedback by gradually rolling out a new ‘one click checkout screen’ to 5% of your users. Let’s say your users love the feature and you see sales increase by 20%. Cool! Now you can increase the rollout to 20%… 50%… 100% and measure impact along the way. However, let’s say your users hate the feature. In fact, the new ‘one click checkout’ is drastically increasing accidental purchases which is hurting your bottom line and frustrating your users. No problem. Since you only had it rolled out to 5%, you can simply roll it back to 0% while you reassess the feature and try to improve the user experience.

Conclusion

Contextual observation using feature flags is a practice that leading product teams use to launch features like Google, Facebook, Amazon, Quora, Asana…and many more. For modern UX, where users want the latest and greatest features instantly, contextual observation cultivates more usable software that can be delivered quickly and safely.

As a designer who can code, Justin can empathize with a developer's workflow and design intuitive interfaces to address extremely complex functionality. He has built dozens of user interfaces for high-traffic applications — winning the Best of California IT Design award in 2012. He frequently contributes feature flag management and design theory articles to DZone, Tech.co, and DesignerHub. He holds degrees from UC Davis and USC, and is finishing an MS in Information Design at Northwestern. When he's not making developer's lives easier, he enjoys tennis, computer games, and writing.