Metric Component Analysis: Mean Metrics

Today we’ll look at how to break down a more complex calculated metric, the mean, in The Case of the Lower Conversions! Our case starts with a drop in our conversion rate from visit to purchase, which is the percentage of visits that result in a purchase. The drop is obvious in the following evidence:

As you can see, the conversion rate drops on February 4th from around 50% to almost 45%. That’s a 5% drop in just one day – and a drop that persisted for the rest of the week! We need to solve this case, but where should we start?

When we examined sum metrics, we were able to find the answer by charting all the components for a given dimension, so let’s try that again here [1]. Let’s plot the conversion rate for each product we sell as a line chart:

Hmmm… that is not as useful as the chart yesterday. I see that Product A’s conversion rate increased on the 4th, not dropped, while all of the other products dropped. What is going on?

Mean metrics are calculated in a different way than the sum metrics we discussed yesterday. In this particular case we are using a weighted mean. While the sum metric adds each individual component, a weighted mean adds a proportionate, or weighted, amount of each component – for example, the proportion of the total population:

Hence, charting the conversion rate for each product isn’t enough information – we need to know how many purchases each product involves. So to investigate this case, we also need to see how many transactions each product was involved with:

Product

Percent of Transactions

Product A

2%

Product B

3%

Product C

3%

Product D

16%

Product E

31%

Product F

46%

A massive change in a component that is only a very small part of the total population won’t move the weighted mean, nor will a very small change in a massive part of the population. For example, a big change in Product A likely won’t change the overall metric since it is such a small part of the population. We need to look for significant changes in significant populations.

To make this clear, let’s only consider the products with the largest (Product F) and smallest (Product A) populations in our chart above:

As you can see, Product A’s conversion rates actually increased when the overall metric decreased! However, since it’s population was so small it did not move the overall metric. Product F is a large part of the population so its dip in conversion rate on February 4th pulled down the overall metric. Product F is the culprit!

Case closed! This was a little harder, but we are just warming up. Tomorrow we’ll cover how to break down another type of metric in The Case of the Disappearing Users!

Do you do look for metric drivers in your business? Outlier is a product designed to help! Outlier looks deep into the dimensions of your data to identify the drivers and emerging trends that result in changes to your main KPIs. If you’re interested in seeing a demo, schedule a time to talk to us.

[1] A stacked area chart does not make much sense in this case because adding conversion rates does not have the same meaning as adding revenue.

Quote of the Day: “Just because everything is different doesn’t mean anything has changed.” ― Irene Peter