Simpson’s Paradox: When the Data Just Doesn’t Seem to Add Up

Tom and Sally throw a ball through a hoop over two days. In total, they each throw the ball the same number of times.

Over Day 1 and Day 2 respectively, Sally’s success rate is greater than Tom’s. When the two days’ totals are combined, Tom walks away a clear winner.

This is an example of Simpson’s paradox (also known as the Yule-Simpson effect). It’s confusing, and usually met with disbelief; part of us instinctively rejects the mathematical truth that this can occur - and that’s because it jolts against our personal systems for interpreting data.

But, it is worth noting, that ‘Simpson’s paradox’ is not a paradox at all; it is simply the result of misconceptions about causality, and poor interpretation of data.

University College Los Angeles has produced research on the paradox, and it is worthwhile highlighting a distinction made by their computer science professor Judea Pearl.

He assesses that ‘Simpson’s Paradox’ is a psychological matter, and a reflection of the surprise associated with the data - that feeling of ‘what?’ when you read the riddle at the start.

‘Simpson’s reversal’ is a mathematical phenomenon relating to calculus - the mathematical truth that testifies to Tom’s victory.

Below is the data of Tom and Sally’s competition:

When you see the raw data, the problem unravels. The ‘reversal’ in data comes about because of the unequal distribution of balls thrown by each person on each day. As already noted, Sally’s success rate - or the percentage of balls she put through the hoop - is greater than Tom’s over Day 1 and Day 2, but, in total, Tom wins.

In other words, comparing Tom and Sally on either given day is not an equal comparison, and the ‘success’ needs to be understood in that context.

The alternative is that you see Sally’s victories as signifying more than they should. The ‘paradox’ comes by way of my presentation of the data, and the narrative I created around the competition; a narrative, nonetheless, that was empirically upheld.

This points us to the importance of Simpson’s paradox: there is data, and then there is data.

Why Simpon’s Paradox matters: data and decision-making

The obvious antidote to all this is to look at your data more carefully. However, in the business world, the conditions for such a paradox become more complicated, and the paradoxes themselves more opaque.

Here’s a hypothetical example. A women’s shoe retailer is undergoing market research for product validation: they have two shoes at the beta phase, and only have the budget to release one. They need to make sure they release the shoe that will sell best.

To reach their target market, the company draws up four demographic qualifiers. Shoe A performs better than Shoe B in all four categories.

The percentages and parentheses represent the people who responded that they would purchase the shoe.

Yet in total across the four demographics, 1,068 (35.6%) people would buy Shoe A, and 1,790 (59.6%) would buy Shoe B. That is significant swing in favour of the shoe that performed poorly on each category at an individual level.It looks like Shoe A is the clear winner here.

Across the market in aggregate, Shoe B dominates.

The solution

The solution resides in distinction. Do not conflate each subsection or each demographic’s performance with the overall outcome; equally, do not dismiss the subsection results as unrepresentative or skewed.

Shoe A’s performance is relative and valuable, and the comparison with Shoe B is mutually illuminating - it serves to reveal where Shoe A performs well, and its competitor less so.

It is about understanding the data in front of you. What is helpful is to work out the story behind the data, and the reasons for why you have these results in front of you.

For example, it is important to recognise that if the demographic sample sizes were the same for each demographic category, Shoe A would walk out a clear winner (for more on how sample sizes and data representation can skew data interpretation, see our article on denominator neglect).

It is vital, too, to recognise that, relatively speaking, Shoe A is more popular with Over 25s, but Shoe B is popular with more Over 25s.

If you can make that kind of distinction, then you will avoid the pitfalls and costly business decisions that Simpson’s paradox causes.

Related posts

Who's going to stand out this year? Will brands that took a battering in 2017 make a statement in 2018? Will it be the year of challenger brands or incumbents?

To bring you answers to these questions (and more), we reached out to 5 experts with very different backgrounds across startups, content, social media, experiential marketing and audio to share their unique perspectives on who are the brands to watch in 2018.

Monzo: The digital mobile-only challenger bank saw nearly half a million new users sign up for its services and claim their bright orange bank cards last year. Monzo is a fantastic way to manage your budget thanks to their instant updates in the app showing you how much you've just spent, and provide added value when used abroad thanks to their free withdrawls up to £200.

Having just received their full UK banking license from the FCA and PRA in 2017, Monzo is rolling out "the best current account in the world". With their slick app and excellent communication, they are playing to millennials by offering a unique customer experience and we're set to see even more new banking features in 2018.

Sanctus: The mental health startup based in London has the vision to create the world's first mental health gym, where people can go and work out their mental health fitness as they would their physical fitness. Right now, the company is working with businesses to create space within a company for people to take time off and talk to a Sanctus coach. In 2018, the company aims to work with 50 business partners and continue to spread awareness of mental health. FounderJames Routledgewrites an excellent weekly newsletter on mental health and growing the startup, which is honestly written and is well worth a read.

Neom Organics:Hot off the heels of significant new investment, this Harrogate-based beauty and wellbeing brand is set to launch a new range of products in 2018, as well as new retail stores both in the UK and abroad. Neom was found by two friends, one of which was an ex Glamour magazine editor who realised her own wellbeing, and that of her close friends, was affected by the stress and demands of modern life. She quit journalism to train as an aromatherapist and nutritionist before founding Neom. The brand's products focus on improving people’s wellbeing through home fragrances and skincare.

My first pick is Pepsi. Lets be honest, Pepsi had an awful 2017 from a brand perspective, they created what they thought was going to be a work of advertising art, an ad that would change the world, but instead it turned them into a global laughing stock.

This is also on a backdrop of huge backlash and increased legislation against sugary drinks. The days when all they had to worry about was competing against Coca-Cola are probably looked on with nostalgia by the marketing team. However Pepsi are a brand with true marketing pedigree, iconic campaigns, partnerships and experiences.

I’m really interested to see how they come back. The test of a great brand is how they react when they are at their lowest. I will be watching Pepsi closely in 2018 to see what they have planned.

My second one to watch for 2018, is the darling of the Aim, BooHoo. The online based fashion retailer has gone through exceptional growth over the last few years, along with some very smart acquisitions.

However they are now at the point where brand building is becoming as important as performance marketing. I expect an innovative business such as BooHoo to evolve its marketing activity to ensure it not only continues its business growth but becomes a brand leader in its own right.

This will be a year to watch brands take the design aspect of their branding in new and exciting directions.