Why the Plural of Anecdote is not Data

A common theme when I see bad science on the TV or internet is the personal testimony pitch - for me, this holds about as much water as a bucket with no bottom, but not everyone knows why anecdotes do not count as evidence. When talking to people about alternative medicine or psychics, I always get told “but it worked for me, how can you explain that?” It can be explained very easily actually - if I went around to a large enough number of people and gave them a vague Barnum statement, or asked them to pick a card and showed them all the same card, a few would guess correctly, and for those people magic has just happened, and they will go and tell their friends about a miraculous handsome man who did the impossible. That is exactly what alternative medicine quacks and psychics do; take credit for an event that is bound to happen eventually if enough chances are given.

No one is perfect, and there are flaws in how we think. We are very subjective in our thinking, like how we assume that if one thing happens after another, then the first thing caused the second thing to happen (post hoc ergo propter hoc). We take a pill with no active ingredients and our headache goes away, so we assume it must be the pill. Though actually, the headache probably just subsided after reaching it’s worst point, and we were going to return to our normal state (regress to the mean). To regress to the mean is just our cycle of health and sickness, for most of us we spend most of our time well, occasionally becoming unwell, then regressing to the mean.

Another example of post hoc ergo propter hoc is when a child gets an injection and shortly afterwards the child is diagnosed with a developmental problem – we assume it must be the injection that has caused this. We also have confirmation bias in this example because such large groups of people expect something to happen after the injection that they see each case that arises as proof of their expectation, whilst ignoring when it does not happen or when it happens without the injection.

We also suffer badly from conformation bias, we remember the times that something has worked, like if the psychic says “I see a journey and a mysterious man” and we respond “yeah, I just came back from holiday” – ignoring the part about the ‘mysterious man’. When we expect something to work, we see it working; our belief that something is working can actually cause us to have a physical response, which is the placebo effect. The placebo effect, conformation bias, regress to the mean and post hoc ergo propter hoc are the foundations of an anecdote and the tools of the quack and the psychic.

We are all subject to these kinds of bias; it does not make someone stupid or crazy, but we must learn to combat these flaws in how we think, and how we do that is through the scientific method, the placebo control, double blind and randomized study. Some people I have spoken to do not fully understand how a study differs from anecdotes, to compare a study with an anecdote is like comparing apples and horse crap. In a study, a group is set up and split into several individual groups. A good study will randomly separate the people into groups to avoid a subconscious effort to achieve a positive result (if a researcher wants a pain killer to work, and chooses the group members themselves, the researcher may subconsciously pick younger and fitter people who are more likely to heal well and with higher pain thresholds). The study should also be double blind, which means that neither the group members nor the researchers know who is in each group. One group will be given a real treatment, another group will be given a placebo, and there may be other groups with competitive treatments to see which works the best. Each group is monitored and their progress is chartered, at the end of the study the groups are revealed and all of the data is compiled and compared. If the treatment they were testing works better than the placebo then the treatment works, but it is also important to compare the results with the competitive treatments to see which is most effective.

A lot of alternative medicine has been tested and failed to provide anything more than a placebo response, therefore it can be determined that it does not work. “It provides a placebo” some people may say, but so does the real treatment – your expectation that it will work gives you the benefits of a placebo and those of the real treatment. If a car company released a car that did not work, we would not call it an ‘alternative’ car and push it around like in ‘The Flintstones’.

We can see the flaws in how we think, and we have an answer to these flaws. Is our answer perfect? No, but it is the best method we have for determining what is true.

Related

My name is Myles Power, and I run the educational YouTube channel, powerm1985. I spend what little free time I have sharing my love of SCIENCE! through home experiments, visiting sites of scientific interest, and angrily ranting at pseudoscience proponents. I am also one of the founding members of the podcast 'The League of Nerds' - which I co-host with James from 'The History of Infection'.

Support Myles

Follow Myles

Skeptical Talks

I have recently made the jump from the virtual world into the real world by giving Skeptics in the Pub talks. Skeptics in the Pub are groups which meet and converse in the UK and beyond, usually gathering once every month to talk about science, history, religion, etc. - all over a nice pint!