experimenter effect

The experimenter effect is a term used to describe any of a
number of subtle cues or signals from an experimenter that affect the
performance or response of subjects in the experiment. The cues may be
unconscious nonverbal cues, such as muscular tension or gestures. They may
be vocal cues, such as tone of voice. Research has demonstrated that the
expectations and biases of an experimenter can be communicated to
experimental subjects in subtle, unintentional ways, and that these cues can
significantly affect the outcome of the experiment (Rosenthal
1998).

Robert Rosenthal has found that even slight differences in
instructions given to control and experimental groups
can affect the outcome of an experiment. Different vocal intonations, subtle
gestures, even slight changes in posture, might influence the subjects.

Double-blind experiments can reduce experimenter effects. For example, an
experimenter who works for a drug company that is trying to produce a new
drug for depression would be wise to have another experimenter randomize the
participants into the group getting the new drug and the group getting a
placebo. The subjects should not be told what
group they are in; their expectations might affect the outcome. The experimenter who does the
randomization should keep to himself any information regarding who is in
which group and should not be the one to distribute the pills to the
subjects. The one who hands out the pills and keeps records of the effects
of the participants should not know what group any of the participants is
in. In this way, any bias on the part of the experimenter is minimized. Only
after the experiment is concluded should the members of the groups be unblinded. (There should be an exception, of course, if something bizarre
began happening, such as several patients dying of heart attacks or going on
manic shopping binges. In such cases, the experiment might be halted and the
participants unblinded to see if the new drug might be killing people or
triggering manic episodes.)

The experimenter effect may explain why many experiments can be conducted
successfully only by one person or one group of persons, while others
repeatedly fail in their attempts to replicate the results. Of course, there
are other reasons why studies cannot be replicated. The original
experimenter may have committed errors in design, controls, or calculations.
Or he may have committed fraud.

One area of research that has failed to be able to consistently produce
replicable results, though its advocates have been trying to do so for more
than one hundred years, is parapsychology. Fraud,
incompetence, and error are common charges made by skeptics against
parapsychologists. However, fraud, incompetence, and error do not seem to be
restricted to parapsychologists and examples of such things can be found in
all of the sciences (Smith 2003: 72; Broad and Wade). Why, then, the
consistent failure at replication in psi studies? Some researchers have
tried to explain the inability to replicate psi
experiments by claiming that the results of experiments depend on the
beliefs of the experimenters. They divide psi experimenters into two groups:
the psi-conducive and the psi-inhibitory. The former are those
who tend to get favorable results for psi. The latter are those who
consistently do not find evidence for psi. Studies on these two types of
experimenters have found that psi-conducive experimenters "come across as
more enthusiastic, warmer and less egoistic than do their less successful
counterparts" (Smith 2003: 77). This difference in experimenter beliefs and
traits, even if true, doesn't do much to justify belief in psi. What if we
found out that all of the cheery researchers believed in Zeus and none of
the skeptics did? Would that justify belief in Zeus? Or, as psychologist
James Alcock (2003) puts it

I could posit that Zeus exists and likes to torment parapsychologists,
and thereby gives them significant outcomes from time to time, but does
not allow replication outside parapsychology. The significant outcomes
would provide as much support for my hypothesis that Zeus exists as it
does for the Psi hypothesis....(p. 43)

It may seem to some skeptics that the notion that only true believers can
get positive results in psi research is an insurmountable barrier to ever
establishing psi research on par with any other type of scientific endeavor.
There is, however, an even more problematic proposal made by many
parapsychologists: The psychic abilities of the experimenter may directly
affect the psychic abilities of the subjects in the study. In 1976, Kennedy
and Taddonio introduced the expression "experimenter psi effect" to refer to
"unintentional psi which affects an experimental outcome in ways that are
directly related to the experimenter's needs, wishes, expectancies, moods,
etc." (Smith: 79).

Alcock (2003: 35) notes that the appeal to an experimenter psi effect to
explain irregularities in attempts at replicating psi experiments is simply
begging the question.

When there has been a failure to replicate, it is not appropriate to
engage in the circularity of assigning to this failure a label (psi-experimenter
effect), and then implicitly suggesting the label as its explanation.
Since there is no other way of defining or identifying the psi-experimenter
effect, it has no explanatory value. Using it as a possible
explanation only leads to a tautology: By substituting the definition of
the psi-experimenter effect, one gets: 'The failure to replicate may be a
manifestation of "one researcher failing to replicate a finding that
another researcher had made".' This circular reasoning excludes from the
debate a possibly fruitful aspect of research, in terms of coming to
understand the reasons, other than psi, that might account for the fact
that different experimenters have obtained different results.

Alcock does not think it bodes well for psi as a scientific subject if
only psi-conducive researchers can replicate studies.

...what a risky adventure it would be to yield to special pleading and
relax the very rules of scientific methodology that help to weed out
error, self-delusion and fraud in order to admit claims that violate the
basic tenets of science as we know it (2003, p. 35).

Or, as Matthew D. Smith (2003) puts it: "if sceptical researchers wishing
to attempt replication cannot be expected to be successful due to their a
priori beliefs about psi....then parapsychology cannot be treated as a
truly scientific discipline (p.82).