Using Multiple Data Sources and Insights to Aid Design

Often when we think of using data to influence our designs, we think of “data” in a very narrow way. When trying to identify opportunities or support our design decisions with evidence, some gravitate solely towards web analytics or conversion data, others only to research findings in lab studies. Some shy away from the thought of “data-driven design” (usually with the exception of qualitative research) in fear that our UX expertise can be overwritten by automated experimentation and optimization. However, data that you gather shouldn’t be treated in isolation and it doesn’t have to be seen as an opposing force to design. When used properly, multiple data sources can be joined together to generate powerful insights that will help us make our products and services better.

Data sources that can be used to influence design

There are many different data sources that can be used to help you better understand your users and the context in which they are using your product(s):

Market research – Your marketing partners will often have market segmentation and trend data that can help bring insight into the lives of your users. Market research helps shed light on how your products can help address needs in a target market.

Industry reports – Organizations such as Forrester and Gartner help provide comprehensive industry analysis whether it be from independent assessment, wide scale industry research, or survey data. These reports can help you better understand the higher-level context in which your products reside.

Competitive analysis – Looking at what your competitors are doing can help identify gaps and opportunities for differentiation through design, capabilities, marketing positioning, and overall experience.

Social monitoring analysis – Use social monitoring tools to find out what your customers are saying about you. If someone’s motivated enough to share what they like or don’t like about your product, you should listen. Identify trends that reveal points of pain and opportunity areas.

Customer service reports – If your company has a call center or customer service line, check their issue logs to find out what issues customers have that generate phone calls. Often times this will reveal opportunities to reduce troubleshooting or issue calls by addressing or fixing the issues on your site. You can also use this as an opportunity to improve the phone channel user experience.

Surveys – Surveys can help you identify trends in user needs, preferences, and activities. Surveys can be either serve as a large-scale closed response study, or can be focused more towards qualitative open-ended responses. Surveys can help identify further points to research or to validate assumptions about your user base.

Qualitative research – Research though observation, usability tests, interviews, or other ethnographic studies can give you insight into people’s behaviors and why they do the things that they do.

Web analytics – Analytics can tell you what is happening on your website through clickstream and conversion data. Analytics can be a powerful tool when segmented by source, user groups, and navigation paths. It’s important to approach analytics with insights in mind as opposed to simply reporting on meaningless high-level data points.

User session data – Products such as Tealeaf allow you to capture real-time page-by-page session data from your users. This, in aggregate, can help you identify points of pain that may not be immediately evident in web analytics or from smaller scale usability testing.

Search log analysis – Search logs can help you identify important topics your users are looking for that may need to be elevated in the design, as well as phrases that can help influence labeling and content organization.

Experimentation – Techniques such as multivariate and A/B tests can help you optimize a design by seeing what performs better against certain goals.

If there are any other specific techniques I missed, please feel free to mention them in the comments!

The importance of a multi-faceted data approach

It’s tempting to rely on one or two sources of data that may be “easier” to get or analyze. It may seem sufficient to interview ten users or look at high-level web analytics and generate conclusions that will be the source of your design decisions. And in some cases, such as when you are trying to optimize a more micro-level element of a design, that may actually be sufficient. But for most larger design/UX initiatives, especially large redesigns, relying on a small selection of data may do more harm than good.

Think of each data source as a puzzle piece. You might be able to see a pattern and general shape of a puzzle’s picture with only some of the pieces put together, but you can’t see the entire picture unless you have every piece of the puzzle. Without every piece, there are gaps that you can only fill in by imagining or guessing what the full picture looks like, which carries with it a lot of risk. You may make incorrect assumptions about what may best solve a user need or that would optimize your site’s effectiveness from a business perspective. The weaker your insights, the weaker your overall strategy becomes. When put together, multiple data sources can give you powerful insights to make strategic design decisions.

Identifying patterns between multiple data sources helps to achieve insight as opposed to disconnected data points. Using data to influence your design doesn’t mean the end of creativity or that design can be whittled down to what is written in spreadsheets and displayed in graphs. When used correctly, data from multiple sources can allow us to better identify the context in which our designs live. It can help us validate our assumptions and approach design with confidence and not subjective opinion. This not only helps to create better design, but also helps us achieve that all-important buy-in from stakeholders. It’s easier to defend a design when you have deep, rich insights to back it up.

How to identify which data sources to use

While a multi-faceted data approach is important, it’s unrealistic to gather data and generate deeply researched insights using every data gathering method for every design problem. Time and resources make that simply impractical. Instead, you should focus on one or two techniques that fall in the three primary areas:

Quantitative data – Who are your users, what are they doing with your product today, and where are they failing to complete tasks or accomplish business objectives?

Qualitative data – Why do users act the way that they do? What are their thoughts and motivations?

Competitive/industry data – How does your product fit among your competitors? What will differentiate your product from the rest? How does your product fit into the context of your users’ lives?

How you go about picking a technique to answer the primary areas above depends on several factors:

What question are you trying to answer? If you’re trying to answer “what” is happening on your site, data pulled from analytics, social monitoring, experimentation, surveys, and search logs can help you cultivate a base knowledge of what activity is happening both on and in the greater context around your site. If you’re trying to answer “why” things are happening, qualitative techniques can help identify reasons for specific behavior. If you’re trying to answer “how does this fit within the industry?” you can look at industry reports, market reports, and conduct a competitive analysis. Make sure the technique you choose is going to provide you with insights to answer specific research questions you have or will prove/disprove a hypothesis.

What tools do you have at your disposal? Not everyone has access to expensive industry reports or highly technical monitoring tools. However, that shouldn’t serve as an excuse for not taking a multi-faceted data approach. Use Google Analytics instead of Omniture, business magazine survey results instead of Forrester reports, or informal interviews instead of formal usability lab studies. In most cases, the free/cheap techniques for gathering data can be almost if not equally effective as more expensive methods. Use techniques that are most suitable for your organization.

How does your project align with previous initiatives? If your project builds off of previous projects, you can likely leverage previous research (assuming it was documented properly) as opposed to trying to reinvent the wheel each time. Once you’ve identified your product’s audience, build solid research-based personas that can be used in the future. You can then focus your research and data gathering efforts on other facets as opposed to questioning who your users are over and over again. Be sure to leverage this past research effectively and use it in conjunction with newly researched data that’s specific to your current initiative.

How much time/money do you have? The techniques you choose will be highly dependent on what will best answer the questions you have in the time you have to answer those questions. Relying on pre-existing research can save you the most time, followed by relatively quick techniques such as interviews, competitive analyses, and some web usage analysis. Other techniques, such as detailed lab studies, complex surveys, and social monitoring can take more time and skill to do properly. While you don’t want to sacrifice the quality of your research, it is important to be realistic in terms of what can be done in the time you have allotted.

Tying it all together

The goal with taking a multi-faceted data gathering approach is to find patterns among multiple data sources. You should be able to tell a story that connects qualitative, quantitative, and competitive research together. Big data reports that simply throw together dissociate data points into a big deck are meaningless other than to show off that you know how to gather data. Looking at a variety of data sources can also help you achieve a holistic design approach that can help you identify opportunities across channels, user segments, and business units within an organization. The user experience is not an isolated activity that can be summed up in a single data point or through a single user research study. The best way to approach a holistic design problem is to take a holistic research approach.

User Experience Design is not data-driven, it’s insight-driven. Data is just raw material for insight… We have to be able to do both: use data to inform the fullest possible understanding of the behavior and context of potential users, as well as bring our own experience and talent to the challenge. – Andrew Hinton

I am a User Experience Designer with a passion for making people’s lives better through design. I have helped over a dozen organizations obtain a competitive advantage by delivering great user experiences across desktop, mobile, tablet and other channels.

Great article. I am glad that you hit on this topic and that it dives deeper than the argument of qualitative data married with quantitative data… we've been beating that drum for a while. What makes this article great is that it talks about market and industry data… and of course, that it hits on social monitoring and A/B testing. A well rounded approach. Keep writing these articles Catriona, you're spot on. 🙂

inspireUX

Thanks so much, Nick! I really appreciate all of your feedback on these articles 🙂

Nice article, Catriona. It's always good to remind us of the info sources at our disposal.

The challenge I see often is that when we go into Research mode, whether Primary or Secondary, we look first and – in many cases, last – at the data relating strictly to our clients' customers interactions with the business itself – what are people doing on the site, what are people telling customer service, what gaps are they communicating through surveys, etc. You've covered those bases pretty well above.

Hallie Wilfert and Lorelei Brown put on a great workshop at the IA Summit in 2009 titled Information Archeaology. While I won't get into the whole of the session, I will share with you and your readers some additional sources for gaining insight into your clients' customers. These sources, and others related, have really helped my clients see greater returns on their UX/CX investments:

– Pew Internet/American Life– Government Agencies and services like Data.gov– Non-Government Organizations like SCORE– Related charitable Foundations and Non-profits– World Bank– US Census– Bureau of Labor Statistics– Center for Disease Control– Google News and other news services like the AP and Thomson Reuters

You've alluded to some of the above when you mention Industry Reports, but Google News and trade reports really influenced the outcomes of two projects just by aligning what we saw from analytics and customer service to events in the world affecting consumer behavior. On the other hand, the US Census or World Bank data aren't necessarily germane to every account or project which crosses our plates. It's all about context.