FIVETHIRTYEIGHT; A Justice’s Use Of Statistics, Viewed Skeptically

By NATE SILVER

Published: March 8, 2013

oral arguments before the Supreme Court last week, Chief Justice John G. Roberts Jr. introduced a statistical claim that he took to imply that an important provision of the Voting Rights Act has become outmoded.

Section 5 of the Voting Rights Act, which is being challenged by Shelby County, Ala., in the case before the court, requires that certain states, counties and townships with a history of racial discrimination get approval (or “pre-clearance”) from the Department of Justice before making changes to their voting laws. But Chief Justice Roberts said that Mississippi, which is covered by Section 5, has the best ratio of African-American to white turnout, while Massachusetts, which is not covered, has the worst, he said.">

Section 5 of the Voting Rights Act, which is being challenged by Shelby County, Ala., in the case before the court, requires that certain states, counties and townships with a history of racial discrimination get approval (or “pre-clearance”) from the Department of Justice before making changes to their voting laws. But Chief Justice Roberts said that Mississippi, which is covered by Section 5, has the best ratio of African-American to white turnout, while Massachusetts, which is not covered, has the worst, he said.

Chief Justice Roberts’s statistics appear to come from data compiled in 2004 by the Census Bureau, which polls Americans about their voting behavior as part of its Current Population Survey. In 2004, according to the Census Bureau’s survey, the turnout rate among white voting-aged citizens was 60.2 percent in Mississippi, while the turnout rate among African-Americans was higher, 66.8 percent. In Massachusetts, conversely, the Census Bureau reported the white turnout rate at 72.0 percent but the black turnout rate at just 46.5 percent.

As much as it pleases me to see statistical data introduced in the Supreme Court, the act of citing statistical factoids is not the same thing as drawing sound inferences from them. If I were the lawyer defending the Voting Rights Act, I would have responded with two queries to Chief Justice Roberts. First, are Mississippi and Massachusetts representative of a broader trend: do states covered by Section 5 in fact have higher rates of black turnout on a consistent basis? And second, what if anything does this demonstrate about the efficacy of the Voting Rights Act?

One reason to be suspicious of the representativeness of Mississippi and Massachusetts is the high margin of error associated with these calculations, as noted by Nina Totenberg of NPR.

Like other polls, the Current Population Survey is subject to sampling error, a result of collecting data among a random subsample of the population rather than everyone in the state. In states like Massachusetts that have low African-American populations, the margin of error can be especially high: it was plus-or-minus 9.6 percentage points in estimating the black turnout rate in 2004, according to the Census Bureau. Even in Mississippi, which has a larger black population, the margin of error was 5.2 percentage points.

As a general matter, I would prefer that everyone be more careful when citing statistical data, and be more explicit about describing the potential sources of error and uncertainty associated with the calculations. But the headline associated with Ms. Totenberg’s article at NPR makes a strong claim: it asserts that Chief Justice Roberts has “misconstrued” the data by ignoring the margin of error.

In fact, several things can be said in Chief Justice Roberts’s defense. Ms. Totenberg cites 2010 voting rates in her article, when the difference in black turnout between Mississippi and Massachusetts was within the margin of error. But Chief Justice Roberts appears to be referring to a lower-court brief that cited 2004 data instead, when the difference was larger and outside the margin of error.

Furthermore, estimating the degree of uncertainty associated with a statistical estimate is not quite so straightforward as it might seem. There is no bright line at which a particular piece of statistical evidence goes from being meaningful to meaningless.

For example, in a poll of 1,000 people, a candidate who is ahead 52-48 has a 90 percent chance of holding the lead (assuming that there are no other sources of uncertainty apart from sampling error). A candidate who is up 53-47 has a 97 percent chance of holding the lead.

If one applies the conventional definition of the margin of error, which usually refers to a 95 percent confidence interval, then the second candidate’s lead would be described as being outside the margin of error while the first candidate’s would be within it. Nonetheless, the first candidate is still nine times more likely to lead his opponent than to trail him. Conversely, while we can be somewhat more confident about the second candidate’s lead, there is still some chance (3 percent) that he actually trails in the race and that the poll was an outlier. Statistical certainty exists along a continuum of probabilities and not in absolutes; I am therefore reluctant to endorse arguments that rely on semantic distinctions about how terms like “margin of error” or “statistical significance” are applied.

Another problem is that sampling error refers to only one potential source of uncertainty in a poll. In surveys of voting behavior, for example, some voters give erroneous responses: lying about whether they voted, misremembering whether they did so or being uncertain about whether their ballot was ultimately counted. This measurement error is in addition to sampling error and will not be accounted for by the margin of error. Further errors can be introduced by the polling method: since some people are more likely to respond to surveys than others, the sample may be biased in some way rather than being truly random. Thus, the true degree of uncertainty in a polling result is usually larger than implied by the margin of error alone.

oral arguments before the Supreme Court last week, Chief Justice John G. Roberts Jr. introduced a statistical claim that he took to imply that an important provision of the Voting Rights Act has become outmoded.

Section 5 of the Voting Rights Act, which is being challenged by Shelby County, Ala., in the case before the court, requires that certain states, counties and townships with a history of racial discrimination get approval (or “pre-clearance”) from the Department of Justice before making changes to their voting laws. But Chief Justice Roberts said that Mississippi, which is covered by Section 5, has the best ratio of African-American to white turnout, while Massachusetts, which is not covered, has the worst, he said.">