These strange lights were seen above my house last night. (Well, sort of…)

It’s actually a photo of a lightning bolt stretching across the sky from left to right, as seen through the eyes of someone who only understands binary comparisons.

You see, I’ve blogged about binary comparisons before. I grumble that comparing just two isolated numbers (e.g. ‘this year compared to last year’) is misleading. I moan that it’s impossible to say whether the numbers are going up, going down, or staying the same. I gripe that when used as a method for assessing performance, crime statistics, or anything else, the use of binary comparisons is a big fat waste of time.

Look at the unedited version of the lightning bolt to see why…

Beautiful, isn’t it?

When I saw this, I thought of a control chart. The top and bottom of the frame would be the control limits. The lightning bolt would be the zig zags of a data series over time. Sad, aren’t I?

But there’s a point. Viewing data in this way gives it history; context. You can see what’s ‘normal’. It’s possible to identify whether the overall picture suggests there are any trends or unusual spikes, or, as in this case, the numbers are consistent over time.

Simply taking any two points from this lightning bolt and comparing them against each other will give a different result. Look –

See? Depending on which two points you pick, you get different directions of travel. (For more on control charts, have a look at the ‘measures’ section of this blog post). Therefore, if you’re still using numeric tables stuffed full of binary comparisons, such as the one below, then effectively what you’re doing is trying to see the whole lightning bolt by looking at those two isolated dots in the picture at the top of this post.

As you can see, this table is constructed of lots of little snapshots of lightning. It’s impossible to ascertain anything at all about crime trends. What makes it worse is the use of percentages to emphasise the binary comparisons – even more so when really small numbers are compared. That’s why you get a 100% reduction in one crime type and a 400% increase in another.

It astounds me that these things are still used by anybody. They lend themselves to misguided assumptions and conclusions about stuff which isn’t actually there. This means that any decision based on a binary comparison is pure guesswork. Because of normal variation in any data set, the chances of picking two isolated points and them actually being part of a trend is miniscule.

So, here’s the deal. If you stop comparing the two tiny pieces of lightning, you will see the total picture, thereby leading to informed decision making, evidence-based responses, and better results all round. It will also prevent knee-jerk reactions, reduce costs and waste, plus help you understand what is actually happening out there.

Binary comparisons vs control charts – one is blunt, misleading and completely useless; the other is simple, practical, and the most reliable method of understanding data known about since the 1920s. What’s not to like?

8 Responses to Message From The Skies

……Binary comparisons vs control charts – one is blunt, misleading and completely useless; the other is simple, practical, and the most reliable method of understanding data known about since the 1920s.

What’s not to like: lots and lots. Frequentist stats does not work when applied to a service organisation.

Thanks for your comment Geoff. Here’s a suggestion – Rather than popping up on my comments pages (and emailing me) just to tell me I’m wrong, why not share your perspective so we can all learn from you? I’d be really keen to understand why you believe what I’ve written about wouldn’t work in service organisations. Surely you’re not endorsing binary comparisons? Cheers, Simon.

Aren’t Control charts only useful if the data is normally distributed or ratehr if you assume data is normally distributed? If this is the case, then since most real life data dosn’t appear to be normally distributed, a control chart could be misleading too.

Not that I am a fan of Binary Comparisons, though gold love them if they say the right thing!

Control charts are only useful if the process in question is stable. Whether or not the data are normally distributed will be visible when they are placed into a control chart. Chicken / egg thing I guess, although I’ve found that lots of real life data do tend to be normally distributed, meaning control charts can be really useful.