Pseudoscience and Evidence

Pseudoscience is, as the name implies, something that claims to be scientific
but is not. The modern world is LOADED with pseudoscience. You can learn
to detect it because it has properties that you can compare to the real
scientific method.

Page 162 of Schick and Vaughn shows one form of the scientific method. We'll
summarize it here.

Observe some phenomenon

Develop a hypothesis (possible explanation) for it.

Use the hypothesis to predict another phenomenon.

Check out the prediction by observation or experiment.

There is something that many people don't realize: science
involves creative thinking. The scientist uses all prior scientific training
as the basis for inventing a hypothesis. This is inductive reasoning.
The next step (3 above) uses the new hypothesis to predict some other
phenomenon; this is deductive reasoning. The scientist must use
both types of reasoning in the work.

It doesn't always work exactly like this; some things (like the Sun) cannot be
experimented with. However, according to S & V, "any procedure that serves
systematically to eliminate reasonable grounds for doubt can be considered
scientific." Along the same line, in Shermer we find "Scientific progress
is the cumulative growth of a system of knowledge over time, in which useful
features are retained and nonuseful features are abandoned, based on the
rejection or confirmation of testable knowledge." This is how we have
learned everything we know about the universe.

Pseudoscience, on the other hand, ignores one or more of the steps. The true
scientific method is NOT followed, although the pseudoscience is littered
with scientific-sounding terminology. It's the method that counts, and that
is what fails. True objective evidence analysis is usually lacking.

Makes no progress and generates no new knowledge, even over many decades.

Claims of effects or forces never measured, or even observed.

Claims of impossible precision of measurement.

Makes no useful predictions.

Tests that have VERY small samples.

...

These are some things to look for, but it is NOT an exhaustive list.
Be sure to keep a copy of the Baloney Detection Collection and Fuzzy Thinking
pages. Even those are not exhaustive but they will be helpful.

Evidence

This is important. Remember that Carl Sagan said
"Extraordinary claims require extraordinary evidence." A skeptic wants
evidence in support of a claim. A skeptic will accept a claim IF
it is accompanied by impeccable evidence.
So what is evidence?

Let's begin by discussing some things that are NOT evidence.
These things are, unfortunately, what pseudoscience uses.

Anecdotes or testimonials:
You encounter these all the time,
mostly in advertisements. "I had no musical sense whatever until the day
I was frightened by a tuba. Now I can belch Mary Had a Little Lamb."
Try "I was slowly going bald until I started using HairBlast;
now my hair is growing back, my teeth have turned green, and I smell funny."
A personal story (anecdote) describes only one event (assuming it is truthful).
It tells you nothing useful about any larger population. Everybody knows
someone who _________ (fill in the blank). A testimonial is even more suspect.
They are frequently used in ads for a good reason - they are not regulated.
Advertisers are supposedly barred from making unsupported claims in their ads,
but they can use all the testimonials they want. What you don't know is
whether they are genuine or written by an ad copy writer. You also don't
know of any other pertinent circumstances that apply to the situation.

For example, on January 4 2007 the FTC went after some weight-loss pill vendors.
They had been making claims for their product that were not supportable;
the manufacturers will be required to remove the unsupportable claims from
their advertising. Some of the testimonials these vendors used in their
ads were from people who had lost a substantial amount of weight while taking
the pills. What you didn't get from the ad was the fact (found by the FTC)
that those people had also been following a good diet and exercise program,
which was almost certainly the real reason for the weight loss.

FTC Chairman Deborah Platt Majoras said "Testimonials from individuals are
not a substitute for science, and that's what Americans need to understand."
One of the marketers actually had a study that showed that people who got
the placebo lost more weight than those who got the actual weight-loss pill!
You can bet that study was never publicized.

Patents:
Believe it or not, a patent does not give you any assurance
that the patented device or process actually works!
You wouldn't believe some of the stuff that gets patented.

Photographs:
Given modern image processing software and hardware,
almost any photograph can be faked. It is practically impossible to rule
out hoaxes with photographs. An out-of-focus, fuzzy image is worthless also.

Attacks on existing explanations:
Someone can fire all the arrows
they want at existing theories or explanations, but that in no way constitutes
evidence for the new claim. It's not evidence unless it supports the new claim.

"It can't be, so it isn't":
This is a form of wishful thinking. You might also translate this as meaning
"I don't like it, so it can't be true."
If you are convinced that something can't be true, then it isn't.
This may or may not have any connection to reality.

"I never heard of that - it can't be true"
Maybe, maybe not. It is not reasonable to think that you have heard of everything.
If someone says "Well, I never heard of that" you can take it as a statement of fact:
they have not heard of it. What you are not allowed to do is to extrapolate
that to "it can't be true." It is perfectly possible that there is some important
result, mechanism, theory, or phenomenon that the person has not heard of.

"Post hoc, ergo propter hoc" arguments:
This thinking involves
making unjustified connections between events. Someone who lives near a
power line get cancer, so obviously the power line caused the cancer.
Wrong. Extensive research has showed no connection.

Coincidence:
Event A and event B happened together, so they must be related.
We may forget that events A and B happen frequently and not together.
When the coincidence occurs, we forget about all the other times.

Correlation:
Two phenomena may appear to be related. For example, the
stock market and some sports team's performance may seem to track each other.
Would you invest based on a ballgame score? Very important: Correlation
does not mean causation. If a correlation appears, one must investigate
thoroughly to see if there is an understandable linkage between the two. IT is also
possible that there is some yet unidentified factor working that is responsible for
both the observed phenomena.

Ok - so what does constitute evidence?
This depends somewhat on the claim.
Take the example of "HairBlast" that we made up earlier. What would it
take to convince a skeptic that the stuff really works? The anecdote
certainly won't do it. It's going to require a properly done experiment
with a large enough sample and proper protocols so that the result is
unambiguous - either the stuff works or it doesn't. You might do it like this.

Gather a large sample of balding men.

Randomly divide the sample into two groups.

Give one group HairBlast to use along with instructions.

Give the other group something which looks exactly like HairBlast but
is completely inert.

No participant is told what they are using.

After the test period, someone who does not know which participant
got what measures the hair growth for all.

Compare hair growth of the two groups.

If the group that used the real HairBlast showed significant growth of hair
while the group using the inert material did not, you have some evidence that
the stuff actually works. If hair growth in the two groups is the same,
then the stuff probably does not work.
It's even better if the experiment can be replicated.

By the way, that example is what is known as a
randomized, double blind,
placebo-controlled trial.
This is considered to be the gold standard in
experiments. We'll see more of this later.

There are two experimental criteria that sometimes need to be applied.
In the case of "psychic" phenomena, such as ESP, the experiment must be
designed so that

If the claimed effect exists, the phenomenon will occur, and

If the claimed effect does not exist, the phenomenon
will not occur.

That second criterion is the hard one. Lots of psychic experiments have
run into grief because they did not achieve that. Magician's tricks
can fool a lot of people.

Direct Presentation:
If a friend of yours insists that they are
communicating with alien visitors, have them introduce you to one.
That should be interesting. See if you can wangle a spaceship ride.
That would be convincing. We are looking for something concrete which,
by existing, backs up the claim.

You are looking for evidence that is clear and unambiguous, not requiring
tortured interpretation or assumptions.

Cautions

There are, of course, cautions to be observed. It is very important to
rule out errors, self-deception and hoaxes. These phenomena are major
problems in most of the pseudo-scientific areas we will visit in 3333.
Suppose that your friend does introduce you to an "alien." You need
to check very carefully to be sure it's not some short guy in a funny suit.
Any experiments must be checked very carefully for flaws and errors.
Poorly done experiments have been responsible for a lot of BS.
We'll explore some of these a bit later.