Thursday, August 11, 2005

Suppose that a priori X has probability p of being true. We now look for evidence for X of a certain type. Suppose that there is a probablity q that we find this evidence if X is true and probability q' that we find this evidence if X is false. We will assume p<1 (otherwise we wouldn't bother looking for evidence) and that q>q' (otherwise it couldn't be said that the evidence we're looking for is evidence for X).

So we have four possibilities:

X is true and we find evidence for X: probability pq

X is true and we don't find evidence for X: probability p(1-q)

X is false and we find evidence for X: probability (1-p)q'

X is false and we don't find evidence for X: probability (1-p)(1-q')

TheoremUnder the hypotheses above, the conditional probability that X is true given that we failed to find the evidence is p(1-q)/(p(q'-q)+1-q').

ProofUse Bayes' Theorem.

Some elementary rearrangement shows this is always less than p given the above hypotheses. It doesn't matter if we are unable to assign an a priori probability, this holds whatever value p has as long as it's less than 1. And if we don't know that q>q' then we shouldn't be in the business of looking for evidence. If the experiment we're doing is any good then q'=0 but as I have shown, the result holds even if we relax this condition.

So clearly failing to find evidence for X should lower our estimate of the probability that X is true.

I wonder what made Sagan say this. I think that maybe he meant to say "absence of evidence is not proof of absence". The theorem shows that under the original hypotheses the conditional probability is never 1, and so while we have evidence of absence, we don't have a proof. But if we can look for enough independent types of evidence it's quite possible for the conditional probability to get close to 1.

Update: The first paragraph was badly written and so I've edited it slightly. The main argument is unchanged. The argument stands or falls regardless of what Sagan actually said, but read the comments below on some context for this quotation. (02 Jan 09)

19 comments:

I suppose what you're saying is fairly intuitive, but it's good to see mathematical proof of it. I assume (without looking at the original) that Sagan's point might have been that absence of evidence often stems from something other than having looked for evidence once and not found any. If in any significant number of cases, evidence of absence stems in fact from not having looked for it, then the prior probability of presence will overshadow the probability rendered by evidence or its lack.

If you do a google news search on absence of evidence you can see it's been used quite a bit by politicians lately - especially with reference to WMD. Again I think we have to interpret what they say as "absence of evidence isn't proof of absence" otherwise it makes no sense.

I think Sagan was definitely thinking of it from a deductive logic framework; one wouldn't be surprised if Sagan cautioned one against using ad hominem arguments, but couldn't we also show that a speaker's honesty and other personal characteristics should affect our priors about that speaker's statements?

You define q' as the probability of not finding evidence, given that X is false. So, q' = P( not E | not X). So, the probability of X being false and not finding evidence for X (4) is:

P( not X and not E) = p( not X ) * P( not E | not X ) = (1-p) * q'.

You seem to mix up numbers 3 and 4. That definitely changes your denominator, though I'd guess your conclusion still works out to be correct.

On the other hand, I always assumed that "evidence" referred more closely to absolute "proof", rather than probabilities as you suggest. Otherwise, as you say, the concept of "evidence" as a means to work toward a fact, is meaningless.

However, if q' is the probability that you don't find evidence for false X, isn't it always 1?

It's not clear what kind of things we're talking about here. So, I assume X is a logical formula and in this system logical axioms are randomly generated initially. Then, we can say that X has a probability p of being true.

The evidence we're looking for is a deduction. We don't know the axioms, and we're going to carry out the same steps with some assumptions. However, we have a machine that can halt our deduction sequence if at a given step our assumption is not satisfied by the axiom. Then we can speak of a probability q that we complete the proof. q could be 0 for some true X due to Godel's incompleteness, but q' has to be 1 because our deduction just can't terminate with X as the result, or else our logic is inconsistent.

Furthermore, I don't think P(X is true and we find evidence for it) is just pq because they're not independent: an axiom that invalidates X will not let you find evidence for a false X.

Sorry but the truth of a statement doesn't have a probability. From that critique, everything else falls down.

And no, Carl Sagan meant exactly, "absence of evidence is not evidence of absence" and he was not wrong. He is a scientist, and so wouldn't talk about proof anyway. No, he meant evidence, which is what he said, and what he said makes sense. It's called "Argument from ignorance". Look it up.

Anonymous writes, “And no, Carl Sagan meant exactly, "absence of evidence is not evidence of absence" and he was not wrong. He is a scientist, and so wouldn't talk about proof anyway. No, he meant evidence, which is what he said, and what he said makes sense. It's called "Argument from ignorance". Look it up.”

• appeal to ignorance — the claim that whatever has not been proved false must be true, and vice versa (e.g. There is no compelling evidence that UFOs are not visiting the Earth; therefore UFOs exist — and there is intelligent life elsewhere in the universe. Or: There may be seventy kazillion other worlds, but not one is known to have the moral advancement of the Earth, so we're still central to the Universe.) This impatience with ambiguity can be criticized in the phrase: absence of evidence is not evidence of absence.

Carl Sagan criticizes the impatience with ambiguity in the phrase “Absence of evidence is not evidence of absence” as an example of an appeal from ignorance.

Wikipedia uses the example of cancerous cells in a body. Finding no cancerous cells in a biopsy makes it less likely you have cancer. Some people might call the lack of cancer cells a lack of evidence (for cancer), some might call it evidence (that cancer is not present).

As for scientists not talking about proof, have you ever talked to a scientist? I guess they tone the language down when it comes to actual publications.