No Right to Believe

Insignificant December 5, 2015

“First you state your null hypothesis, which is your default position in the absence of any evidence, and your significance level, which is the maximum probability you’re willing to accept for rejecting the null hypothesis when it’s actually true. Then you perform your observations, calculate the p-value (the probability of obtaining a result at least as extreme as what was observed if the null hypothesis were true), and reject the null hypothesis if and only if the p-value is below the significance level.”

“Wait a minute: a significance level of zero means there’s no evidence that could ever convince you to abandon the null hypothesis.”

“Oh, is that bad? All right, then: My significance level is five percent…”

“That’s better.”

“…and my null hypothesis is that I will not change my significance level retroactively based on the outcome of the observations.”

“Hmm, let me test that… OK, the results are in, and they are statistically significant: p-value is two percent. You should reject the null hypothesis.”

“No problem — but I’m afraid that means I’ll be changing my significance level to one percent, making your observations insignificant. So my null hypothesis has been proved true after all!”

“The null hypothesis is never proved, it can merely fail to be rejected. And anyway, if your null hypothesis were true, wouldn’t that mean you should not have changed your significance level? Actually — never mind; this is a waste of time. Do you even care whether your belief is based on evidence?”

“Absence of evidence is not evidence of absence. Just because you can’t measure something doesn’t mean it’s not there.”

“Excuse me, but I must be going now: evidence has just come in forcing me to reject my null hypothesis.”