Executive Summary

The measure of voting system accuracy that ultimately matters to the
success of our democracy is the following:

How close does the outcome of the election, as reflected in the
official canvass, come to the actual intent of the voters who
participated in the election?

This is an end-to-end measure of performance, in the sense used by
Saltzer, Reed and Clark in their 1984 paper,
End to End Arguments in System Design.
It is extremely difficult to connect the kinds of accuracy standards
found in the current (1990) or draft FEC standards with this ultimate
standard.

Both recounts and exit poll data give a useful approximation of this measure,
and we should use them regularly! I have not looked at exit poll data, but
by the recount measure, current paper-based election systems routinely
achieve overall accuracies bettern than 1 in 5,000. This is a far cry from
the 1 in 500,000 or 1 in 10,000,000 figures cited in standards, but those
are not end-to-end measures, and in fact, they verge on being meaningless
because they ignore many issues of both law and voting technology.

In
www.cs.uiowa.edu/~jones/voting/optical/,
my tutorial on optical-mark sense voting, I have documented in fair detail
the interaction of law, human factors and technology that we must address
if we are to correctly deal with these issues.

The relationship between law and questions of vote-counting accuracy
is stunningly illustrated by the machine model of what constitutes
a valid vote; this model is central to the arguments and appeals in
Touchston and Shepperd vs. McDermott (U.S. Court of Appeals,
11th District, Dec 6, 2000). The machine model defines, as valid, those votes
that the machine counts. Interpreted rather foolishly, this standard makes
the entire question of voting machine accuracy moot!

We must formulate state laws that distinguish between
the prescribed mark on a ballot, the mark voters are instructed to
make to cast a vote, and the class of acceptable marks, those marks
that the law counts as votes. For each ballot tabulating system, vendors
must document the class of reliably counted and
reliably ignored marks. For all real systems, there will also be
marginal marks, marks that may be counted under some circumstances
and ignored under others.

Only when all of these issues are pinned down can we reasonably begin to
ask the key questions that allow us to grapple with the end-to-end accuracy
of the voting system: How likely are voters to express their intent by
making acceptable marks, a question of both human factors and law, and
how likely is the machine to detect these acceptable marks and count them as
votes, a question of technology.