Simpson’s Paradox: When the Data Just Doesn’t Seem to Add Up

August 07, 2017

Tom and Sally throw a ball through a hoop over two days. In total, they each throw the ball the same number of times.

Over Day 1 and Day 2 respectively, Sally’s success rate is greater than Tom’s. When the two days’ totals are combined, Tom walks away a clear winner.

This is an example of Simpson’s paradox (also known as the Yule-Simpson effect). It’s confusing, and usually met with disbelief; part of us instinctively rejects the mathematical truth that this can occur - and that’s because it jolts against our personal systems for interpreting data.

But, it is worth noting, that ‘Simpson’s paradox’ is not a paradox at all; it is simply the result of misconceptions about causality, and poor interpretation of data.

University College Los Angeles has produced research on the paradox, and it is worthwhile highlighting a distinction made by their computer science professor Judea Pearl.

He assesses that ‘Simpson’s Paradox’ is a psychological matter, and a reflection of the surprise associated with the data - that feeling of ‘what?’ when you read the riddle at the start.

‘Simpson’s reversal’ is a mathematical phenomenon relating to calculus - the mathematical truth that testifies to Tom’s victory.

Below is the data of Tom and Sally’s competition:

Screen Shot 2017-08-08 at 12.35.33.png
When you see the raw data, the problem unravels. The ‘reversal’ in data comes about because of the unequal distribution of balls thrown by each person on each day. 
As already noted, Sally’s success rate - or the percentage of balls she put through the hoop - is greater than Tom’s over Day 1 and Day 2, but, in total, Tom wins.

In other words, comparing Tom and Sally on either given day is not an equal comparison, and the ‘success’ needs to be understood in that context.

The alternative is that you see Sally’s victories as signifying more than they should. The ‘paradox’ comes by way of my presentation of the data, and the narrative I created around the competition; a narrative, nonetheless, that was empirically upheld.

This points us to the importance of Simpson’s paradox: there is data, and then there is data.

Why Simpon’s Paradox matters: data and decision-making

The obvious antidote to all this is to look at your data more carefully. However, in the business world, the conditions for such a paradox become more complicated, and the paradoxes themselves more opaque.

Here’s a hypothetical example. A women’s shoe retailer is undergoing market research for product validation: they have two shoes at the beta phase, and only have the budget to release one. They need to make sure they release the shoe that will sell best.

To reach their target market, the company draws up four demographic qualifiers. Shoe A performs better than Shoe B in all four categories.

The percentages and parentheses represent the people who responded that they would purchase the shoe.

Screen Shot 2017-08-08 at 12.36.11.png
Yet in total across the four demographics, 1,068 (35.6%) people would buy Shoe A, and 1,790 (59.6%) would buy Shoe B.  That is significant swing in favour of the shoe that performed poorly on each category at an individual level.
It looks like Shoe A is the clear winner here.

Across the market in aggregate, Shoe B dominates.

The solution

The solution resides in distinction. Do not conflate each subsection or each demographic’s performance with the overall outcome; equally, do not dismiss the subsection results as unrepresentative or skewed.

Shoe A’s performance is relative and valuable, and the comparison with Shoe B is mutually illuminating - it serves to reveal where Shoe A performs well, and its competitor less so.

It is about understanding the data in front of you. What is helpful is to work out the story behind the data, and the reasons for why you have these results in front of you.

For example, it is important to recognise that if the demographic sample sizes were the same for each demographic category, Shoe A would walk out a clear winner (for more on how sample sizes and data representation can skew data interpretation, see our article on denominator neglect).

It is vital, too, to recognise that, relatively speaking, Shoe A is more popular with Over 25s, but Shoe B is popular with more Over 25s.

If you can make that kind of distinction, then you will avoid the pitfalls and costly business decisions that Simpson’s paradox causes.


Related posts

5 Perspectives on The Brands to Watch in 2018

Who's going to stand out this year? Will brands that took a battering in 2017 make a statement in 2018? Will it be the year of challenger brands or incumbents?

To bring you answers to these questions (and more), we reached out to 5 experts with very different backgrounds across startups, content, social media, experiential marketing and audio to share their unique perspectives on who are the brands to watch in 2018.

Alison Battisby, Founder, Avocado Social

Monzo: The digital mobile-only challenger bank saw nearly half a million new users sign up for its services and claim their bright orange bank cards last year. Monzo is a fantastic way to manage your budget thanks to their instant updates in the app showing you how much you've just spent, and provide added value when used abroad thanks to their free withdrawls up to £200. 

Having just received their full UK banking license from the FCA and PRA in 2017, Monzo is rolling out "the best current account in the world". With their slick app and excellent communication, they are playing to millennials by offering a unique customer experience and we're set to see even more new banking features in 2018. 
Sanctus:  The mental health startup based in London has the vision to create the world's first mental health gym, where people can go and work out their mental health fitness as they would their physical fitness. Right now, the company is working with businesses to create space within a company for people to take time off and talk to a Sanctus coach. In 2018, the company aims to work with 50 business partners and continue to spread awareness of mental health. Founder   James Routledge   writes an excellent weekly newsletter on mental health and growing the startup, which is honestly written and is well worth a read .
Neom Organics:  Hot off the heels of significant new investment, this Harrogate-based beauty and wellbeing brand is set to launch a new range of products in 2018, as well as new retail stores both in the UK and abroad. Neom was found by two friends, one of which was an ex Glamour magazine editor who realised her own wellbeing, and that of her close friends, was affected by the stress and demands of modern life. She quit journalism to train as an aromatherapist and nutritionist before founding Neom. The brand's products focus on improving people’s wellbeing through home fragrances and skincare. 
Adam Azor, Managing Director, Curb
My first pick is Pepsi. Lets be honest, Pepsi had an awful 2017 from a brand perspective, they created what they thought was going to be a work of advertising art, an ad that would change the world, but instead it turned them into a global laughing stock.
This is also on a backdrop of huge backlash and increased legislation against sugary drinks. The days when all they had to worry about was competing against Coca-Cola are probably looked on with nostalgia by the marketing team. However Pepsi are a brand with true marketing pedigree, iconic campaigns, partnerships and experiences.
I’m really interested to see how they come back. The test of a great brand is how they react when they are at their lowest. I will be watching Pepsi closely in 2018 to see what they have planned.

My second one to watch for 2018, is the darling of the Aim, BooHoo. The online based fashion retailer has gone through exceptional growth over the last few years, along with some very smart acquisitions.

However they are now at the point where brand building is becoming as important as performance marketing. I expect an innovative business such as BooHoo to evolve its marketing activity to ensure it not only continues its business growth but becomes a brand leader in its own right.

This will be a year to watch brands take the design aspect of their branding in new and exciting directions.

Posted by Mark Walker on January 16, 2018

10 Ways To Understand and Shift Your Brand Perception in 2018

Think you know what your brand represents? I’ll let you in on a little secret; it’s not what you say in your slogan, brand values or advertisements. It’s whatever consumers say you are.

Posted by Bel Booker on January 09, 2018

5 Things That the Strongest Brands Focus on

Nothing lasts forever.

Posted by Mark Walker on January 08, 2018

5 New Year Resolutions to Help Your Brand in 2018

New Year resolutions don't just have to be for individuals. The idea of improving, quitting bad habits and enjoying a more productive, successful year can be just as valid for business and teams. 

Here are some resolutions you may want to consider for your brand to avoid the pitfalls others experienced in 2017.

Posted by Alex Rees on January 05, 2018