Secondary menu

Search form

What are sigma levels?

When physicists at CERN announce their results, there is
always a lot of talk about sigma levels. But what exactly are
they?

You can listen to an extended version of this interview in our podcast.

Let's take the Higgs boson as an example. When it was discovered
back in 2012, physicists didn't actually catch sight of the particle
itself. They inferred it was there from an irregularity in the data
they collected from the Large Hadron Collider (see here to find out
more). But of course, such an irregularity could be down to mere
chance: it could occur even if there wasn't any Higgs boson at
all. Sigma levels reflect people's confidence that the result is not
just down to chance. The higher the sigma level, the greater that
confidence. Particle physicists require a sigma level of at least five to
announce a new discovery, such as the Higgs boson.

"Five sigmas means that there is
only a tiny chance of getting the result they did by just a fluke — that
the Higgs boon wasn't there, and that because of a bit of bad luck or
whatever, they got this blip in the [data]," says David Spiegelhalter, Winton Professor for the Public Understanding of Risk at the University of Cambridge. "[Five sigmas] means that there's a less
than 1 in 3 million chance that they would have observed this result
just as a matter of luck."

David Spiegelhalter

Physicists aren't the only scientists who quantify the confidence in
their results in this way. People who test medical drugs in clinical
trials, for example, also need to be confident that any effect they
observe wasn't just a fluke and is really down to the drug's
efficacy. But interestingly, medical scientists require a lower sigma
level than particle physicists do. "To get a new drug passed and approved by regulatory authorities,
you need to do two studies, but they only have to be two sigma
discoveries," says Spiegelhalter. "People generally in science would
consider three sigmas, that's a 1 in 1000 chance of getting
the result they did just by chance, as being very strong evidence
indeed."

Why the difference? Partly because people don't want to
make fools of themselves. "People don't want to claim things and then
later discover that they are wrong. It's happened before in particle
physics and it's happened in genetics endless times — it's what they
call false discoveries. Nobody likes to make false discoveries and
then have to retract them. This is embarrassing. So you can say that
the number of sigmas is how much you are prepared to try to avoid
embarrassment. But it means you have to do a huge amount of more work to
find something of five sigmas." Generally, it's more embarrassing to make a false claim about the fundamental nature of physical reality than, say, to be slightly wrong about the amount by which a drug reduces blood pressure, so the extra work particle physicists need to put in is worth it.

But there is a practical reason too. "If drugs required five sigmas,
there would never be any drugs on the market. People would require
huge numbers of patients to prove the drug is effective,"
says Spiegelhalter. "Whereas as it stands they can get by with a few
hundreds or thousands of patients."

A third reason for the difference in sigma levels is a little more subtle. It relates to the fact
that, the more you look for a sign of something, the more likely you are to find
something that looks like the sign, but isn't. "In CERN they searched for the Higgs boson over an entire
range of frequencies and they found one little blip," says
Spiegelhalter. "The more places you look, the more your sigma level
gets reduced, and therefore you have to go for a really high number to
be really confident you haven't made a false discovery." This is
called the look elsewhere effect. It plays an important role
in genetics, where scientists look for signals in a huge number of
different places. "This is why geneticists also demand very
high sigma levels."

If you know a little bit about statistics, then all of this may
sound familiar. In other areas of science people often use p-values to quantify their confidence in a result. So what's the
relationship between p-values and sigma levels? "The sigma level is
only a short hand for the p-value, which is the probability of
observing something as extreme as you did if the thing that you are
trying to find isn't actually there, or if the drug isn't effective,
or the gene doesn't have a relationship with what you are interested
in, and so on," explains Spiegelhalter.

To be confident of your result, the p-value therefore has to be
very small, and becomes awkward to express: 1 in 1000, 1 in 3 million
etc. "It's much easier to use the short-hand of 1, 2, 3, 4, or 5
sigma," says Spiegelhalter. "At CERN they actually calculate the
p-value from a very complex statistical procedure and then translate
it back to an effective number of sigmas."

So, next time you hear an announcement from CERN, remember: five sigmas are needed to claim a discovery. Signs for the famous Higgs boson existed at lower sigma levels long before 5 sigmas were reached. It would have been embarrassing, however, to claim a discovery earlier, only to find that the blip in the data was a fluke and the particle not there.

Comments

During my Engineering days we would occasionally run into physical phenomena that didn't have a Gaussian basis. Some of these were lengths of lightening strokes, she speed of sound in water, etc., etc.

Specifying a few sigma as a measure of reliability, can occasionally represent unreliability when the underlying physical parameters are not symmetrical. This can lead to misleading results. I have read that the scientists at CERN generally use a 5-sigma threshold for high reliability determination of physical parameters.

There are problems that are easy to solve in theory, but impossible to solve in practice. Intrigued? Then join us on a journey through the world of complexity, all the way to the famous P versus NP conjecture.