How to Create Misleading Statistics in 6 Easy Steps

1. Do a survey and use a biased sample population
People focus on the survey result and seldom pay attention to information about your sample population. So feel free to bias your result by surveying people you know will answer the way you want. Want a survey result that shows a majority of people favoring smoking in public? Just ask smokers. Want a strong statistic showing approval for your new system? Just ask people who you know are on your side.

2. Use biased questions in your survey
Your can further strengthen your survey results by asking biased questions which make the survey respondent more likely to answer the question the way you want. Political organizations are masters of this. Want to show approval for guns? Just ask a question like, “In these days of increasingly frequent home invasions, do you feel that having a gun in the house would help your personal defense?” (and don’t forget step 1 and use gun owners as your survey population). You’ll get an amazingly strong show of support for guns.

3. Ask the wrong question
You can often support your cause by asking the wrong question (the one that’s not quite as relevant to your desired result), and then answering it in a way that implies something you want. For example, if you want to show public support for a war, then ask a survey question about supporting the troops (which is, after all, an entirely different thing). If you want to hide the fact that you’ve got occasional critical system downtime, then show overall system uptime — those system failures will be lost in the statistical noise. If you want to show high consumer interest in product A, then ask consumers about A versus C, where C is a not-so-popular alternative. But don’t ask consumers about A versus B, where B is A’s closest competitor.

4. Use misleading graphs
If you want to show high variability in a statistic, then chop off the bottom of the graph and just show the top (for example, using a y axis of 5,000 to 5,001 instead of 1 to 5,000). This magnifies the area you’re interested in, and makes small variations look like huge peaks and valleys. On the other hand, if you want to show stability in a statistic, then make sure you show the full x axis (e.g., from 1 to 5,000), and make the graph as small as possible to minimize the appearance of variation.

5. Imply cause and effect when you only have correlation
Two factors can often be correlated without any cause and effect relationship. Both factors can be caused by a totally different factor, or the cause and effect can even be backwards from what’s being implied or stated (i.e., B causes A instead of A causing B). For example, I once saw a claim that older people who walk are more likely to live longer. That may be true, but it’s also just as likely that older people who are sick and can’t walk are more likely to die sooner. So maybe the walking isn’t the cause — maybe the ability to walk is just an indication of continued good health.

6. Make your results more precise
A number with higher precision gives the impression of being more accurate. If I say that 91.3% of system users are happy, it sounds more accurate than 90% of users, even if both numbers are made up. In the old advertising campaign, Ivory soap was touted as “99 44/100% pure” (pure what?). People trust precise numbers because they sound more scientific, but that doesn’t mean the numbers are correct.

Conclusion

In reality, I’m not advocating your use of any of these six techniques, but I wanted to show you how easy it is to mislead people with statistics. So to avoid being fooled:

1. When you see survey results, look at the survey methodology. How was the survey sample determined? Is the sample large enough to be meaningful? Is the sample biased?

2. Where possible, look at the original questions in the survey to see if the questions themselves are biased. Don’t trust a survey that uses biased questions.

3. Make sure that the statistic being cited is one that matters. Are the right questions being asked? What is the analysis leaving out? What is being deliberately omitted?

4. Where graphs are used, look carefully at the x axis to see if the graph has been enlarged. Has the graph been manipulated to bias your impression?

5. Look carefully at cause and effect. Does the evidence support the cause and effect that’s stated? Is the opposite cause and effect possible? Is it possible that both factors are being caused by something else entirely?

6. Don’t be deceived by precision in numbers — extra decimal places for the sake of looking good. There’s often little justification for a more precise number, and more precision doesn’t imply more accuracy.

In general, look at the source of the statistic and think about what that source wants you to believe. Look carefully at the case the source has made. Be skeptical of anything that isn’t consistent with your common sense. Trust your gut on these things — radically new views on things happen only rarely, so if the statistic seems surprising, it has a good chance of being wrong.

Share this:

10 Ways to Find the Truth In my previous post I talked about the problem of determining the truth in current events (and in other areas) when we’re faced with conflicting views from thousands of media and Internet sources. In this...

Why ROI Isn’t Working ROI (Return on Investment) is the most common and popular method for project ranking, both in IT and elsewhere. But ROI isn’t working in most companies, and as a result, businesses are making bad project...

The Politics of Information Technology I’m writing this on November 7, 2006. That’s election day in the United States. During the last few months we’ve been besieged with television and radio advertising for candidates, and even recorded messages sent to...

Fixing Broken Windows (not the Microsoft kind) A few weeks ago there was an article in an Atlanta newspaper about George Kelling, the author of the book, Fixing Broken Windows: Restoring Order and Reducing Crime in Our Communities. Dr. Kelling has been...

4 Reasons We Disagree, and What to Do About It You’re trying to get a new project approved, and you’re having trouble. Or you’re trying to get an employee to do things your way, and the employee keeps fighting you. Both these situations are disagreements,...

Taking Shadow IT Out of the Shadows, Part 1 Shadow IT is one of the names for the Information Technology work and expense that’s done outside of the control of the formal IT organization and outside the formal IT budget. It’s more prevalent in...

How to Deal with Complaints — 7 Steps How do you deal with someone who complains — whether it’s a customer of your company, a customer of your department, or even an employee or family member? How do you turn the complainer into...

3 Keys to Service Success What do capability, motivation, and expectations have in common? All three are essential for a successful service organization, whether that organization is in Information Technology or in any other field. Capability Let’s use a help...

IT is Moving toward Property Management Last month I had the unique opportunity to help a large university plan its future curricula for its undergraduate and graduate degrees in computer information systems. The university recognizes that Information Technology is changing, and...

12 Problems in IT Caused by the E-Myth In 1986 a book called “The E-Myth” attracted a lot of attention. The “E” in “E-Myth” refers to entrepreneur, and according to the myth in Michael Gerber‘s book, if you’re good at a particular skill then...

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this. For more information on the use of cookies on this web site, see http://blog.makingitclear.com/cookies/