How to Plus or Minus: Understand and Calculate the Margin of Error

Sometimes in the day-to-day work of conducting and interpreting market research, it’s easy to forget that many people who work with surveys on a daily basis have not had formal training in statistics. Even for those who have been trained, it can be useful to have a refresher from time to time.

UNDERSTANDING MARGIN OF ERROR

One of the most basic concepts in market research is the confidence interval, commonly referred to as the “margin of error.” The confidence interval is a range of values within which a survey result can be assumed to accurately represent the underlying construct being measured.

The general public has a basic if vague understanding of this concept. Indeed, media reports of election surveys often report a result “plus or minus” a certain number of percentage points.

The confidence interval is important because it helps us as marketers and researchers understand the limitations of our survey results. The confidence interval estimates the inaccuracy of our results due to “sampling error,” that is, error stemming from the limitation of conducting our survey among a single sample of the population of interest (rather than the impractical or impossible alternative of conducting a census of the entire population).

Sampling error is distinct from other types of survey error – including measurement error, coverage error, and non-response error – but those are topics for another time.

Here are the factors that affect the margin of error:

confidence level

proportion in the sample

sample size

Confidence level. You must choose how statistically certain you want to be. The most common confidence level is 95%. The conceptual meaning of a 95% confidence level is as follows. If you were to conduct your survey one hundred times with randomly drawn samples and everything else were equal, the result of your survey question would be expected to fall within the confidence interval ninety-five of those times and outside it five times.

Proportion in the sample. Proportional estimates closer to 50% are subject to more variability than estimates near the ends of the spectrum, e.g. 10% or 90%.

Sample size. The greater the sample size, the lower the margin of error because variability due to sampling anomaly is reduced.

CALCULATING MARGIN OF ERROR

There are three ways to calculate the margin of error: use a formula, use a look-up table, or use an online calculator.

Use a formula. There are a number of formulae you can use with slightly varying assumptions. If you want to go through the calculations yourself using a formula, I refer you to this web page: “Guide to Computing Margins of Error for Percentages and Means” from Professor Ted Goertzel’s at Rutgers University, who explains the calculations better than I can hope to do.

Use a look-up table. Here’s a table that will be appropriate in most circumstances. This table is based on a 95% confidence level. In order to find the confidence interval (the “plus or minus” amount) for a particular proportion, go the the row closest to the proportion of interest and the column closest to the sample size of interest. For example, if an N=500 election poll showed a race tied at 50% to 50%, you would go to the 50% row and the N=500 column, yielding a margin of error of plus or minus five percentage points.

N

N

N

N

N

Proportion

1,000

750

500

250

100

10%

2%

2%

3%

4%

6%

20%

3%

3%

4%

5%

9%

30%

3%

4%

4%

6%

10%

40%

3%

4%

5%

7%

10%

50%

3%

4%

5%

7%

11%

60%

3%

4%

5%

7%

10%

70%

3%

4%

4%

6%

10%

80%

3%

3%

4%

5%

9%

90%

2%

3%

3%

4%

6%

Use an online calculator. The above exercises are great, but guess what, you’re in luck! There are many online calculators out there. Here are two examples:

I think this is useful and helpful. But what I’m sure will be far more useful and helpful would be to make clear that what you have just described is one aspect of QUANTITATIVE Research, properly conducted. Because what you are discussing is well designed QUANTITATIVE Research, you are able to use such refinements as confidence intervals to communicate a sense of HOW accurate your research is. MY problem is with ‘research’ like that constantly on the television (QUALITATIVE Research, like Focus Groups) which can never be used to support any ‘quantitative conclusions whatsoever’. Qualititative research almost always fails on grounds of sample size but even if it didn’t, it also fails on many other – even MORE important grounds – to sustain quantitatively valid inference. Translated, Frank Luntz’s Groups (and other Groups) CANNOT BE USED TO DRAW INFERENCE ABOUT ANY GROUP, SAMPLE OR POPULATION OTHER THAN THE ONE SITTING THERE RIGHT NOW AROUND THE TABLE. Nothing (I mean NOTHING) can be inferred from one focus group to any other group or population.

Since this precept is indeed violated over and over and over again, I wish you’d join me in fighting it.

Lew Pringle

http://danastanley.com Dana Stanley

Thanks for your comments, Lew. Indeed, qualitative research is an entirely different animal from quantitative. While qualitative certainly has its place, many incorrectly treat it as if it were quantitative.

Emreerdo

All these calculations are based on the assumption that your sampling framework is simple random sample, and nobody uses it.

majority of sampling schemes are quota based, meaning that you cannot calculate your margin of error.

Those preferring probabilistic sampling methods prefer multistage sampling, where standard errors are calculated in a very different way.

Harris Interactive already accepted that you cannot calculate margin of error, and they don’t declare such a figure.

My idea is using as an “heuristic”, underlining that these scores are calculated “with the assumption of SRS”.

anonymous

thank you, this was very helpful. i am in a statistics class right now, and you did a much better job explaining margin of error than my book did!

Annie Pettit

Even when you’re not using a probability sample, margin of error is a very useful concept. It’s a reminder that there is no such thing as a guaranteed perfect number, something we often forget as evidenced when we show 2 and 3 decimal places. Unless you’ve measured every single item/person, every number based on a sample is an estimate or a really good guess based on available data.