selection bias

Selection bias comes in two flavors: (1) self-selection of individuals to
participate in an activity or survey, or as a subject in an experimental
study; (2) selection of samples or studies by researchers to support a
particular hypothesis.

Edzard Ernst, M.D., who
was trained in various non-conventional medical therapies, provides an
example of selection bias that occurred while he was studying the
therapeutic effect of mistletoe injections on cancer patients. He was told
that the effect would be a lessening of suffering.

Whenever I gave mistletoe injections, the results seemed encouraging. But
young doctors are easily impressed, and I was no exception. What I didn't
appreciate then was a relatively simple phenomenon: the hospital where I
worked was well known for its approach across Germany; patients went there
because they wanted this type of treatment. They were desperate and had very
high expectations - and expectations can often move mountains, particularly
in relation to subjective experience and symptoms. We call this "selection
bias". It can give the impression that a therapy causes a positive health
outcome even when it has no positive action of its own.*

Selection bias partly explains why there are so many satisfied customers who go to
psychics, tarot card readers, palmists, and faith healers. The subjects are
highly motivated to be helped and to have the reader or healer succeed.
Subjects are often extremely generous in their efforts to personally
validate the words, images, or advice of the reader/healer. Some will even
assent to claims they know are false, as one of
Gary Schwartz's subjects did with a medium who got the subject to agree
that her husband was dead when in fact he was still alive. (Schwartz did not
see this event as an example of the power of selection bias to affect
subjective validation of psychic claims. Instead, he saw it as a evidence
for clairvoyance on the part of the psychic and
the sitter whose husband died in a car crash shortly after the session.) In
fact, Schwartz engaged in selection bias when he published several papers in
the Journal of the Society for Psychical Research that support the
hypothesis of survival of consciousness after death. In his book, The
Afterlife Experiments, he describes numerous subjects in his experiments
who are conspicuously not mentioned in the published papers on those
experiments.

Skeptics and parapsychologists have accused each other of selection bias in
determining which studies to include in the
ganzfeld meta-analysis (Bem
and Honorton 1994; C.E.M. Hansel 1989; Hyman 1989; Marks 2000; Radin 1997).
Ray Hyman did the first meta-analysis of 42 experiments and found no
evidence of ESP. Honorton did his own meta-analysis and found evidence of
"anomalous information transfer." In 1994, Bem and Honorton published the
results of a meta-analysis of 28 ganzfeld studies and once again found
evidence for anomalous information transfer. Julie
Milton and Richard Wiseman (1999) published their own meta-analysis of
ganzfeld studies and concluded that "the ganzfeld technique does not at
present offer a replicable method for producing ESP in the laboratory."

selection bias in prayer studies

Elisabeth
Targ's study on distance healing using prayer
is "widely acknowledged as the most scientifically rigorous attempt ever to
discover if prayer can heal" (Bronson 2002), even though it only had 40
participants. The study has since been discredited for improprieties in
mining the data. The study was originally designed to test whether prayer
could have an effect on the death rate from AIDS, but was changed after the
study was completed to see if there were any significant correlations
between prayer and about two dozen illnesses associated with AIDS such as
diarrhea and oral thrush. The statistician was able to find about half a
dozen significant correlations. These were selected as if they were the main
purpose of the study, after it was apparent that prayer had no effect on
survival.

selection bias in polls and surveys

Norman Bradburn,
director of the National Opinion Research Center at the University of
Chicago, coined the acronym SLOP to describe polls that use selection bias
to get their samples. SLOP is an acronym for self-selected listener
opinion polls. He compares them to radio talk shows: they attract a
slice of America that is not representative of the country as a whole. “As a
result, SLOP surveys litter misinformation and confusion across serious
policy and political debates, virtually wherever and whenever they are used”
(Morin).

The inaccuracy of such polls should be obvious. Those who call in to give
their opinion are self-selected rather than randomly selected. It appears
that people who are willing to call in their opinion once will sometimes
call in their opinion more than once. For example, in a USA Today
call-in poll 81 percent of the more than 6,000 respondents said that “Donald
Trump symbolizes what made the U.S.A. a great country.” However, 72 percent
of the favorable calls came from two telephones in one insurance company
office.

CBS tried the gimmick of call-in polling in “America on the Line,” which
featured two surveys conducted immediately after President [George Herbert
Walker] Bush’s
State of the Union speech. There were 314,786 self-selected callers in one
survey and 1,241 adults previously selected by a more scientific method in
the other survey. The latter was to act as a check on the call-in survey.
CBS’s Dan Rather commented on the similarity of results in the surveys, a
sentiment that was echoed the next day in the Washington Post, which
wrote, “by and large, the two polls produced the same or similar results.”
The facts, however, do not support this judgment. “On two of the nine
questions asked in both polls, the results differed by more than 20
percentage points. On another five, the differences were 10 percentage
points or more” (Morin, cited in Carroll 2005: 147).

Some very famous surveys have been based upon selection bias. The work of
Alfred C. Kinsey, for example, is the source for a homosexuality statistic
widely reported in both the mass media and in scientific publications,
though the statistic is based on biased samples. Kinsey’s famous studies on
sexual behavior in the 1950s have been repeatedly cited as the basis for the
claim that 10% of the population is gay. In fact, numerous studies have been
done that put the percentage of adults who describe themselves as
exclusively gay as much lower than the 10% figure. Some found the rate to be between one and two percent.*
However, it should be noted that "survey research methodologies often result
in underreporting of stigmatized behaviors."*

Kinsey gathered his data, in part, by distributing questionnaires to
prisoners and to people who attended his lectures on sexuality, neither of
which was likely to be a good cross section of Americans (Carroll 2005:
140). For his studies on male sexuality, "he interviewed only white men, and
these respondents were disproportionately from lower socioeconomic classes."*

A study that considered attraction to the same sex, in
measuring homosexuality, found "8.7, 7.9, and 8.5% of males and 11.1, 8.6,
and 11.7% of females in the United States, the United Kingdom, and France,
respectively, report some homosexual attraction but no homosexual behavior
since age 15."*

In 1994, sociologist Edward Laumann headed a team of
sociologists that studied U.S. sexual behavior. They interviewed a
representative sample of the U.S. population between the ages of 18 and 59.
Laumann found that over a five-year period, 4.1 percent of U.S. men and 2.2
percent of U.S. women had sex with someone of their own sex. If the time
period is extended to include their entire lives, these totals increase to
7.1 percent of the men and 3.8 percent of the women.*

Paul and Kirk Cameron reported in 1998: "The 1994 University of Chicago
"definitive" survey of adults estimated prevalence of homosexuality among
males at 2.8% and among females at 1.4%. Corrected for the exclusion of
those over the age of 59 years, the estimates should be 2.3% and 1.2%."*
A study in Britain in 2000 found that about 2.6% of men and women reported having had a
same-sex partner within the previous five years and 8.4% of the men and 9.7%
of the women reported having had at least one sexual experience with a
member of the same sex.*

In 2013, a nationwide Gallup poll reported that 3.5% of U.S. adults identify themseves as lesbian, gay, bisexual, or transgender.*

One wonders, however, if anything approaching unbiased
data is possible for determining what percentage of any human population is
homosexual. Given the long history of religious prohibition of homosexuality
and the widespread revulsion of homosexual behavior that has often led to
torment and persecution, it is likely that researchers in this area will be
motivated by something other than a generic search for the truth. Results will differ
depending on how one defines 'gay,' 'lesbian,' and 'homosexual'. Methods of
gathering data samples will vary widely and the participants in such studies
may not be highly motivated to reveal much about their sex lives.

There is some irony in the fact that the Kinsey studies are cited as the
source of the statistic that 10% of the population is gay. As Michael
Shermer notes, Kinsey made it clear that he did not believe human males
"represent two discrete populations, heterosexual and homosexual." Kinsey
maintained that "it is a fundamental of taxonomy that nature rarely deals
with discrete categories. Only the human mind invents categories and tries
to force facts into separate pigeon-holes" (Shermer 2005: 246). Nature
has a bias toward variation. The idea that people should fall into neat categories as
'gay' and 'straight', or even 'male' and 'female', is not consistent with
the lessons of evolution. Any study that creates such false dichotomies will
be misleading.