Sampling Error: What it Means

Oct. 8, 2008

By Gary Langer
Director of Polling, ABC News

Surveys based on a random sample of respondents are subject to sampling error – a calculation of how closely the results reflect the attitudes or characteristics of the full population that's been sampled. Since sampling error can be quantified, it's frequently reported along with survey results to underscore that those results are an estimate only.

Sampling error, however, is oversimplified when presented as a single number in reports that may include subgroups, poll-to-poll changes, lopsided margins and results measured on the difference. Sampling error in such cases cannot be described accurately in a brief television or radio story or on-screen graphic.

Sampling error assumes a probability sample – a random, representative sample of a full population in which all respondents have a known (and not zero) probability of selection. Given that prerequisite, sampling error is based largely on sample size, but also on the division of opinions or characteristics measured and on the level of confidence the surveyor seeks. A larger sample has a lower error margin. A result of 90-10 percent has a smaller error margin than a 50-50 result; when more people agree, there's less chance of error in the estimate. And a result computed at the 90 percent confidence level has a smaller error margin than a result computed at 95 percent confidence.

Assuming a 50-50 division in opinion calculated at a 95 percent confidence level, a sample of 1,000 adults – common in ABC News polls – has a margin of sampling error of plus or minus 3 percentage points. The error margin is higher for subgroups, since their sample size is smaller. Given customary subgroup sizes, for 800 whites the error margin would be plus or minus 3.5 points; for 560 women, +/- 4 points; for 280 Republicans, +/- 6 points. Click here for a list of examples using averages from recent ABC News polls.

At a 90-10 division of opinion, rather than 50-50, still at 95 percent confidence, sampling error for 1,000 interviews is +/- 2 points, not 3. For a sample of 100 cases – roughly the minimum sample size ABC News will report – the error margin is +/- 10 points at a 50-50 percent division, +/- 8.5 points at 75-25 percent and +/- 6 points at 90-10 percent. (ABC News polls at times will oversample small populations to increase their sample size to a level we consider reliably reportable.)

As noted, the confidence level is the third chief variable in sampling error. (There are other factors in some surveys, such as design effects – see the addition to the end of this piece - and finite-population adjustments, which we'll leave aside here.) The 3-point error margin at 95 percent confidence for a sample of 1,000 declines to a +/- 2.5 points at 90 percent confidence and +/- 2 points at 80 percent confidence.

(Some organizations round sampling error to whole numbers; others report them to the decimal. ABC's practice is to round them to the half. That acknowledges the differences caused by sample size – 800 and 1,500 both round to +/-3; better to show the former as 3.5 and the latter as 2.5 – without suggesting the level of precision in the data implied by entirely unrounded decimals, e.g. +/-3.3.)

The calculations above are based on a single sample, using a standard formula – multiply the division in opinion (e.g., .50 times .50 for a 50/50 split), divide the result by the sample size, take the square root and multiply by the so-called "critical value"– for 95 percent confidence, 1.96.