The Plural of Anecdote is not Scientific Evidence

Note: I am in no way saying that racism do not exist in the skeptical community. It does. It is probably as prevalent in the overall population. My general point is that we need scientific evidence, not anecdotes, to accurately quantify the size of the problem and that such a quantification helps us find appropriate solutions to the problem in ways that anecdotes do not (clarification added 20120802 22:36 GMT +1 DST).

Note: Yes, anecdotes and testimonials could be extremely useful in convincing people in practice, but that is not the same as being useful for rational refutation (clarification added 20120803 01:29 GTM + 1 DST).

Let’s say that a person, which we can call X, comes up to you and tells you that X has a successful treatment for a given chronic disease. Or maybe X tells you that the scientific community is oppressing or discriminating proponents of this treatment.

A successful treatment for a chronic disease is great news of course, if it actually works. Also, discriminating is clearly morally unacceptable. Or is this another quack “treatment” that is exploiting chronic disease patients? Is X just playing the martyr card?

As a scientific skeptic, however, the first thought that goes through your mind probably is something along the lines of “That’s interesting! I wonder if there is any scientific evidence exists for that?”. So you ask X about the evidence, and X responds “There is a lot of evidence!” and shows you a couple of testimonials where people say that the treatment has cured them of their chronic condition. X uses other testimonials as examples of discrimination from the scientific community.

“But wait a minute…”, you interject. “That is not scientific evidence, those are merely testimonials!” Something is obviously wrong here. “Why are you dismissing the experience of these people? They have gone through a great ordeal of having a chronic disease and on top of that, been discriminated against by the scientific community!”, replies X in a high-pitched voice, signaling agitation and emotional arousal.

Clearly, you have to tread slowly here or this might get ugly. You take a deep breath before responding that “I am not trying to dismiss their experience, I just want to know about the evidence. When you present testimonials, there are two problems. First, there is no independent support for what is being said. How do we know they are accurate? Should we just trust people’s beliefs on issues as serious as discrimination and medical treatment? Second, there is no way of telling, just from the testimonials themselves, if they are representative. Do these successes represent 99% success of treatment, 50% or 0.00001%? How do we know that it is not just spontaneous remission, or the placebo effect? The plural of anecdote is not scientific evidence. It is just a bunch of anecdotes with questionable scientific relevance.” You exhale, thinking that you nailed the situation and that X will understand your reasoning. You could not be more wrong.

X responds predictably that “So you support discrimination and would refuse giving successful treatments to these chronic disease suffers!?”. You sigh. “No, that is not what I am saying at all because….” You do not get a chance to finish, because X interrupts you with “They can’t win, can they!? If they don’t speak out about our treatment and the discrimination, they have to suffer in silence, but if they do speak out, they get your repugnant skepticism and hypertechnical demands for evidence!” You have finally realized that conversation is impossible. You give it one last try. “I just want to know what evidence…” you explain slowly, but are yet again interrupted: “Evidence!? What kind of evidence do you need then!? What exactly do we need to do to proceed and convince you die-hard skeptics!?” Bingo! Now we are becoming productive.

You take another breath and launch the winning response: “I want well-controlled scientific studies published in reputable, peer-review journals. Only then can we find out if the treatment actually works and the extent of discrimination of proponents of your treatment.” That ought to do it. Yet you are wrong again. X replies that “but the treatment and presence of discrimination cannot be tested because of special feature A, B and C with the treatment.” Predictable response by X, but now the tables have turned. A, B and C are arbitrary, irrelevant or can be included in the study. Your final response becomes “if scientific studies cannot show that the treatment work or that discrimination is extensive, then you have no justification for claiming that it does. End of story. Less anecdotes, more science. If your treatment actually works, then present the evidence or stop screwing chronically ill people out of their money.” before you walk away. Not all discussions that can be had are worth having.

9 responses to “The Plural of Anecdote is not Scientific Evidence”

Your analogy is deeply flawed. First, with regards to racism, the equivalent of bunk cancer treatment would be the person saying that racism hasn’t been “proven”. Just shut your eyes and it ceases to exist. DeepakChopra-esque! Anyway, there’s a reason social sciences stopped trying to be physics about a hundred years ago: it was very unscientific (i.e. it didn’t work, results were in fact catastrophic). The scientific method in social sciences is different, social sciences must use more than one method to gather information. Collecting data alone does not work. In this instance, the only way we could really quantify racism perfectly is if we had a way to distinguish racism from other motivations in the brain of the person dismissing another person of a different ethnic background. This is why peer reviewed papers from social scientists often consists of a “plurality of anecdotes”.
Here’s an example. As with everybody else, I’ve been in many situations were I was dismissed. If you take the aggregate of the times I was dismissed, it appears to be higher than the male counterpart of similar background. You will find individual males who have been dismissed more often than I. So far, we don’t have much information. However, if we get a large random sample of women, take account of their experiences with as much objectivity as possible (i.e. some women do choose to work part-time to take care of their family, therefore cannot reasonably become CEO, so other factors need to be accounted for), we note a trend. The average white middle class man will have less obstacles than the average white middle class woman. Now that’s when you can turn this into relevant information, not before. Very rarely can we know for a fact that a particular action was directly due to racism. However, we can see an aggregate of anecdotal evidence that underline a specific dynamic favoring a certain group.
I used the example of sexism here because I am a white woman, but I believe it applies to race as well. I’ve had frustrating experiences with people calling me a racist. For example, I worked as a server and always emptied the tip jar when I started my shift. One time, I did just that and a customer got angry because he thought I was hiding the money because he is a Native American. If that information got into a study, that particular instance would have been wrong. This is where the researchers have to be careful, there is a confirmation bias on the part of the oppressed group. The group has identified systematic racism therefore attribute to racism actions that due to other factors.
I’m not sure if you are trying to imply that people raising the problem of racism in the atheist community are like the Burzynski clinic or something… I don’t want to put words in your mouth; are you saying that racism could be non-existent in our community? Because that would be baffling. (and would require you to learn beyond an overview of an “introduction to methodology in social sciences” class)

1. You are making a straw man argument. The analogy was not between the treatment for chronic disease and racism, but between discrimination and racism.

2. Actually, social sciences have increasingly been using similar methodologies to natural sciences, such as statistical analysis of quantitative data. It works and it is very common.

3. To quantify racism we do not need to separate it neurologically from other motivations, just analyze behavior. Some behaviors are considered racist, even if the person intended something else. In other words, we have a different view on how to operationalize racism.

4. Furthermore, the quantification does not need to be perfect. In fact, no quantitative study, in natural or social sciences, needs to be or claim to be perfect.

5. You are confusing scientific evidence with anecdotal evidence. Filling out questionnaires may seem like they are providing anecdotal evidence, but this becomes scientific evidence after careful and appropriate statistical analysis by a scientist. That is because it provides, if the study is good, a defensible quantitative estimation on prevalence, rather than an over-reliance on anecdotes.

6. I am not saying that racism is non-existent. It exists, and it is probably prevalent. The general point is that we need to use scientific studies, rather than anecdotes, to get a good idea about the size of the problem and that it is such considerations that help us find solutions in ways that anecdotes do not.

“Anyway, there’s a reason social sciences stopped trying to be physics about a hundred years ago”

Do you mean, say, analytical (in much the same sense that Anglo-American philosophy is called “analytical”), facts-not-values-oriented, mathematical / computational, etc.? Ah no, that’s an unwarranted generalization: agent-based computational economics and sociology are two examples of research programs in that vein that are undoubtedly alive and kicking. There is even a modern effort to bridge physics and social sciences in a less naïve way than was attempted in the 19th and early 20th centuries, not surprisingly called econo- and sociophysics.

You should know: there’s a great wide world outside of “[name of protected group] Studies 101″.

I just gave you my personal support on Greta’s blog post. Not taking sides on the argument, but saying you’re a good guy and to give your blog a chance. I did a bit of analysis on you there, too. I apologize if it offends, let me know if it does. Just trying to help.

Thanks for your input. I spent about five hours commenting on the two blog posts and I feel that I have gotten through to Greta with the substantive parts of my position and I interpret we are in broad agreement, although we may differ on some specific issues.

It looks like your arguments over at Greta’s blog are being dealt with by the commentators by their usual “rational” method, namely, piling-on. As with many of the “freethought” blogs, things like nuance and disagreement over specifics quickly go out the window, and things get reduced to either towing the line or being The Enemy. Hang in there – your points are stronger than the dismissive shoutdowns they’re receiving would indicate.

Debunking Denialism

Modern life presents us with an apparent paradox: science has a strong cultural authority, yet primitive darkness is coming back in the shape of creationism, quack medicine, opposition to vaccination, HIV/AIDS denialism, anti-psychiatry and so on.

Debunking Denialism takes on the enemies of reason.

Article Library

If you want to read more content from Debunking Denialism, check out the article library, or the main content below.

"I realize that 'complementary and alternative medicine' (CAM) or, what quackademics like to call it now, 'integrative medicine' (IM) is meant to refer to 'integrating' alternative therapies into SBM or 'complementing' SBM with a touch of the ol’ woo, but I could never manage to understand how 'integrating' quackery with SBM would do anything but weaken the scientific foundation of medicine."

- David Gorski, cancer surgeon and debunker of pseudoscience (source).

"Postmodernism, the school of 'thought' that proclaimed 'There are no truths, only interpretations' has largely played itself out in absurdity, but it has left behind a generation of academics in the humanities disabled by their distrust of the very idea of truth and their disrespect for evidence, settling for 'conversations' in which nobody is wrong and nothing can be confirmed, only asserted with whatever style you can muster."

"If I am ignorant about a phenomenon, that is a fact about my state of mind, not a fact about the phenomenon; to worship a phenomenon because it seems so wonderfully mysterious, is to worship your own ignorance; a blank map does not correspond to a blank territory, it is just somewhere we haven’t visited yet"

"As an aside, it is ironic that CAM proponents often simultaneously tout how individualized their treatment approach is, but then claim that one product or treatment can cure all cancer. Meanwhile they criticize the alleged cookie-cutter approach of mainstream medicine, which is actually producing a more and more individualized (and evidence-based) approach to such things as cancer."

- Steven Novella, neurologist and founder of the New England Skeptical Society. (source).

"Twenty epidemiologic studies have shown that neither thimerosal nor MMR vaccine causes autism. These studies have been performed in several countries by many different investigators who have employed a multitude of epidemiologic and statistical methods. The large size of the studied populations has afforded a level of statistical power sufﬁcient to detect even rare associations. These studies, in concert with the biological implausibility that vaccines overwhelm a child’s immune system, have effectively dismissed the notion that vaccines cause autism. Further studies on the cause or causes of autism should focus on more-promising leads."

"To me, skepticism is not believing what someone tells you, investigating all the information before coming to a conclusion. Skepticism is a good thing. Global warming skepticism is not that. It’s the complete opposite of that. It’s coming to a preconceived conclusion and cherry-picking the information that backs up your opinion. Global warming skepticism isn’t skepticism at all."

- John Cook, Climate Communication Fellow for the Global Change Institute at the University of Queensland (source).

“Rather than persisting in the view that they have been ‘rejected by science’, advocates in the cryptozoology community have more work to do in order to produce convincing evidence for anomalous primates and now have the means to do so. The techniques described here put an end to decades of ambiguity about species identification of anomalous primate samples and set a rigorous standard against which to judge any future claims."

“In our reasonings concerning matter of fact, there are all imaginable degrees of assurance, from the highest certainty to the lowest species of moral evidence. A wise man, therefore, proportions his belief to the evidence.”