Bayesians and Frequentists

Date: 08/21/2001 at 21:49:56
From: Ellen
Subject: Bayesian Statistics?
Dr. Math -
I am going to be taking a statistics course this semester and noticed
a chapter called "Bayesian Statistics." What is the difference between
this and "regular" statistics?
Thanks,
Ellen

Date: 08/22/2001 at 15:27:17
From: Doctor Jordi
Subject: Re: Bayesian Statistics?
Hi, Ellen - thanks for writing to Ask Dr. Math.
The difference between Bayesian statistics and regular (Frequentist)
statistics is essentially a different interpretation of what
probability signifies, and thus a different way to make an inference
about a population given that we have a sample of that population.
When I tell you, "The probability that this coin lands heads is 1/2,"
what do you make of it? There are a couple of ways to think about it.
A frequentist, and I imagine that you are more familiar with this
interpretation, reasons as follows:
If the probability of landing heads is 1/2, this means that
if we were to repeat the experiment of tossing the coin very many
times, we would expect to see approximately the same number of
heads as tails. That is, the ratio of heads to tails will approach
1:1 as we toss the coin more and more times.
A Bayesian, however, would interpret that statement in a different
way:
For me, probability is a very personal opinion. What a probability
of 1/2 means to me is different from what it might mean to someone
else. However, if pressed to place a bet on the outcome of tossing
a single coin, I would just as well guess heads or tails. More
generally, if I were to bet on the roll of a die and was told that
the probability of any face coming up is 1/6, and the rewards for
guessing correctly on any outcome are equal, then it would make no
difference to me what face of the die I bet on.
That is why the Bayesian point of view is sometimes called the
Subjectivist point of view. In other words, Bayesians consider
probability statements to be a measure of one's (personal) degree of
belief in a certain hypothesis in the face of uncertainty - a
subjective measure.
The two points of view are widely differing and affect the way in
which we conduct statistical inference. Allow me to elaborate.
In statistics, we make an inference, a guess about a population based
on a sample we draw from it. We may, for example, want to know what
the speed of light in vacuum "really" is.
[As reader Steve Dodge points out: "Since 1983, the speed of light has
been a _defined quantity_, set at the integer value of 299 792 458 m/s.
The meter is then defined as the distance light travels in vacuum after
1/299 792 458 s, and the second is defined in terms of an actual
measurement of an atomic system, in an atomic clock." So let's assume
that the following imaginary discussion takes place before 1983.]
We have a problem, however: our experiment is imperfect and random errors
will always crop up in our measurements, no matter how carefully we make
them. So say we repeat our experiment five times and observe the following
measurements on each experiment, in meters per second.
299,792,459.2
299,792,460.0
299,792,456.3
299,792,458.1
299,792,459.5
In this example, our population is the abstract infinity of all
possible measurements we could make. Our sample is the five
measurements we have made. Now we wish to estimate a parameter of this
population, namely the population mean, or the "true" speed of light
in a vacuum. How do we deal with the random errors?
For a Frequentist, there exists a fixed, true, but unknown speed of
light in vacuum. The Frequentist would assume that random errors have
a certain probability distribution (probably normal distribution, also
known as Gaussian, which looks like a bell curve) and would proceed to
take the arithmetic average of the above five measurements. The
resulting statistic (a statistic is a function of your sample) would
be used as an estimator for the population mean. The estimator itself
is a random variable, so we can say, as Frequentists, that
If we were to repeat this sequence of 5 measurements a repeated
number of times, approximately this many realizations of my
estimator will be this close to the true speed of light. However,
on this particular occasion where I have already calculated my
statistic, I have no clue how close I actually am to the true
value, but I feel comfortable that I am doing okay because of
certain properties that my estimator has on repeated uses.
For a Bayesian, the above paragraph is nonsense. The Bayesian DOES
have a clue how close this particular realization of his estimator to
the speed of light, because, unlike the Frequentist, she can make a
probability statement about this realization. The random errors have
no probability distribution. They are fixed realizations; they are
reality. Instead, a Bayesian claims that the speed of light is a
random variable with its own probability distribution. For a Bayesian
there is no "true" speed of light; there is only a certain probability
distribution associated with it.
In Bayesian statistical inference, we first make a guess on what the
probability distribution of the parameter in question is. This is
called a prior distribution. Then, we observe our sample. Based on our
observations, we use a theorem called Bayes' theorem (hence the name
for a Bayesian) and modify our guess about what the distribution of
the parameter in question is. This modified guess is called a
posterior distribution.
Summing it up, Bayesians and Frequentists give opposite answers to the
question: "Does there exist a true fixed and nonrandom population
parameter, even if we cannot know its value because all we can see is
the realizations of SOME random variable?" Frequentists say yes;
Bayesians say no.
Does this answer your question? Please write back if you have other
questions or if you feel that I did not explain myself well enough.
- Doctor Jordi, The Math Forum
http://mathforum.org/dr.math/