Metrics: The Key to Product Optimization

Great product people understand their users in a qualitative way — what their motivations are, what they use your product for, and what issues they have with your product. It’s a core part of being a product owner, particularly when you are looking to build something new. But when you are looking to optimize your site experience, you also need product metrics.

Qualitative research tells you “why,” but product metrics tell you “what.” It’s metrics that help product owners make decisions and, even more important, understand if those decisions were ultimately right or wrong.

Let’s say that you are looking to optimize your search experience, and one of the items in question is your advanced search: should you hide it or or do you make it more prominent?

Sitting with a handful of folks to talk about how they use your advanced search is great, but you need to keep two things in mind:

1. If all the customers that you speak with say they love your advanced search, chances are that you are on to something, but you must remember that they are a small sample of your users. Metrics let you know if these customers are a true representation of your total user base.
2. People tend to say one thing and do something else. Users may tell you that they love your advanced search options, but in reality they never actually use it. Metrics will let you know how they really use your site.

Ultimately you are using customer research and product metrics to make a decision. Let’s say that after you talk with a few folks, you learn that they don’t really use your advanced search, and then you look at the overall usage metrics, and it turns out that only 50% use it on a daily basis. Perhaps you decide to hide advanced search and only have it appear when users click on a button.

You may have assumed that this was the case before you started talking to folks and looked at the metrics, but by making a decision based on research, you have mitigated the risk of just going with your gut or just talking to one or two people.

After you make a decision, it’s also important to monitor the site and see if you made the right decision. For instance, if you hide advanced search, does search success go up or down, does repeat usage go up or down, do new user conversions go up or down, does the time on site go up or down?

Just as much as you need metrics to make a decision about hiding a feature, or changing how something works, you need to have tracking in place to understand what the outcome is, to understand if you are successful or not — in other words, whether or not you made the right call.

In order to know what metrics you should be tracking, answer the following questions before you make a change:

What problem are we trying to solve?

How do people use this page, feature, etc., right now?

What are the primary metrics we are trying to improve? (For instance, we want to increase search success by 2%.)

What are the primary metrics today? (E.g., search success today is 25%.)

What are the secondary metrics that we want to make sure that we do not adversely affect? (We want to increase search success, but we don’t want to decrease new customer acquisition while we do it.)

This sounds like a lot for a product person to keep in mind, but there are ways to manage this:

Dashboards. Have your key metrics on a big screen so that everyone on your team can see them. When you make a change to your site, you can all see if something changes.It’s not just about you. Your team, developers, designers, etc. should feel just as much ownership over these metrics as you do. The more eyeballs you have looking at these metrics, the better off you and your team will be. In the end, your customers will benefit.

In a previous post, I wrote about making sure you focus on priorities over deadlines, and one of the main ways you do this is with metrics. Make sure that everyone in your business is aware of them, and make sure they know that you pay attention to them. Metrics help you mitigate the risk of your decisions and to track their success or failure.

About this Gun

Matt Smith

is currently the Director of New Products at Shutterstock, where he leads the company's video, mobile, and new product strategy. Previously he was a Senior Product Manager at Gerson Lehrman Group (GLG), the world’s leading expert network, where he built and ran the company's survey business. He is a business and strategy expert with a focus on product management, with over ten years of experience building media and technology companies. He approaches all challenges from the same angle: understand the long-term strategy, understand the value, and break things down to simple ideas. Follow @mjordonsmith.

About The Hired Guns

We’ve built The Hired Guns to help the digital creative class expand their potential far beyond the 9-to-5, bringing their expertise to companies in unexpected and exciting ways and guiding their careers through transitions from freelance gigs to fulltime jobs and back again.

We have but one over-arching rule for comments: Do not add to the chaos of the universe.

This blog is devoted to developing a point of view around the Future of Work through the lens of the digital creative class. It offers some of the best career writing out there to help you get ahead as well as some brand new bloggers livin' the dream and tellin' it like it is. We encourage you to use the comments to drive conversations to the next level, bounce ideas off our bloggers, challenge them, and engage in dialogue with your fellow readers.