Over the years, I’ve gravitated toward a few favorite questions. “Compared to what?” is definitely one of my favorites.

Anytime you are analyzing something you need context. What should this result be compared to? Basically everyone knows this. In theory. This is why we run A/B tests. The whole point of a control is to give you a baseline to compare against your fancy new idea. If you simply ran your fancy new idea, and you had nothing to compare it to, how would you know whether it was actually better? Having a point of comparison gives that new data context and meaning.

I expect this all makes sense to you. Everyone typically gets why you need a good point of comparison. HOWEVER, in reality, making bad comparisons is sadly very common. Why? I’m actually not sure. But, it seems that the further the analysis is from a pure math exercise the higher the chance of this happening. One theory I have is that it’s because people aren’t aware they are making such comparisons.

Over the past week I saw a great example of this bad comparison making with the news coverage around Powerball.

In the days leading up to the $1.5B drawing many of these news reports covered tips and tricks on how to win Powerball. I’m sure you heard the same report. Some of these tips and tricks included choosing numbers that were the most commonly picked. So, just based on that nugget, you know that these tips and tricks are a bunch of crap. To be honest much of what they were saying made me crazy. But, for this post, I’ll focus on one part.

The “tip” I heard repeated often, which gave me a headache each time, was around whether to select your own numbers or to have the computer do it for you. Here’s a great example:

Rather than picking numbers based on birthdays, anniversaries or other meaningful dates, lottery organizers note that the majority of past winners have left the big decisions up to the computer.

About 70 percent of past winners used Quick Picks, the computer system that spits out numbers, according to the official Powerball website.

"Does this mean that you are more likely to win with a computer pick ticket? Maybe," the site states.

This made me crazy. For a couple reasons. And here’s why.

Ok, so the fact we are working with is that 70% of winning tickets used quick picks. The reporter seems to have then figured that was better than the 30% of winners who picked numbers themselves. So, therefore you should use quick picks to get your numbers. But, this type of comparison is all wrong.

The core of the problem is that the article doesn’t tell you the split of tickets purchased between quick pick and manual pick numbers. If 1% of the tickets purchased were quick picks, and 70% of the winning tickets were quick picks, then quick picks certainly have the advantage! But if 99% of tickets purchased were quick picks and only 70% of the winning tickets were quick picks, there’s a statistical advantage to manual picked number tickets.

The percentage of purchased tickets has to be compared against the percentage of winning tickets in order to understand statistical advantage!

The article doesn’t make the comparison. That’s pretty frustrating. But it wasn’t the worst part. The article ended with this:

"Does this mean that you are more likely to win with a computer pick ticket? Maybe," the site states.

That’s when I yelled ‘COMPARED TO WHAT?’

I couldn’t believe that was on the Powerball website. So I went to their FAQ’s page. I found the question, but not that line. Here’s what is there (at least now):

WHICH HAS THE BETTER CHANCE OF WINNING: COMPUTER PICKS OR PLAYER PICKS?

About 70% to 80% of purchases are computer picks. About 70% to 80% of winners are computer picks. Perhaps just one of those weird coincidences?

So this means that your odds of winning are the same regardless of how you get your numbers: quick pick or manual. How you get your numbers and which numbers you choose won’t matter. Which is exactly how a lottery of chance should work.

Was this changed in the last week or so? I don’t know. But, I was very happy to see that they got this right. Especially with a little bit of enjoyable snarkiness at the end.