This is the accessible text file for GAO report number GAO-08-97T
entitled 'Elections: Further Testing Could Provide Increased but Not
Absolute Assurance That Voting Systems Did Not Cause Undervotes in
Florida's 13th Congressional District' which was released on October 3,
2007.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
United States Government Accountability Office: GAO:
Statement before the Task Force on Florida-13, Committee on House
Administration, House of Representatives:
For Release on Delivery:
Expected at 4:00 p.m. EDT:
Tuesday, October 2, 2007:
Elections:
Further Testing Could Provide Increased but Not Absolute Assurance That
Voting Systems Did Not Cause Undervotes in Florida's 13th Congressional
District:
Statement of Dr. Nabajyoti Barkakati: Senior-Level Technologist:
Applied Research and Methods:
GAO-08-97T:
GAO Highlights:
Highlights of GAO-08-97T, a statement before the Task Force on Florida-
13, Committee on House Administration, House of Representatives.
Why GAO Did This Study:
In November 2006, about 18,000 undervotes were reported in Sarasota
County in the race for Florida’s 13th Congressional District (FL-13).
After the contesting of the election results in the House of
Representatives, the task force unanimously voted to seek GAO’s
assistance in determining whether the voting systems contributed to the
large undervote in Sarasota County. GAO agreed with the task force on
an engagement plan, including the following review objectives: (1) What
voting systems were used in Sarasota County and what processes governed
their use? (2) What was the scope of the undervote in Sarasota County
in the general election? (3) What tests were conducted on the voting
systems in Sarasota County prior to the general election and what were
the results of those tests? (4) Considering the voting systems tests
conducted after the general election, are additional tests needed to
determine whether the voting systems contributed to the undervote? To
conduct its work, GAO met with officials from the State of Florida,
Sarasota County, and Election Systems and Software (ES&S)—the voting
systems manufacturer—and reviewed voting systems test documentation.
GAO analyzed election data to characterize the undervote. On the basis
of its assessments of prior testing and other activities, GAO
identified potential additional tests for the Sarasota County voting
systems.
What GAO Found:
In the 2006 general election, Sarasota County used voting systems
manufactured by ES&S, specifically iVotronic direct recording
electronic (DRE) voting systems during early and election day voting
and the Unity election management system, which handles the election
administration functions, such as ballot design and election reporting.
GAO’s analysis of the 2006 general election data from Sarasota County
did not identify any particular voting machines or machine
characteristics that could have caused the large undervote in the FL-13
race. The undervotes in Sarasota County were generally distributed
across all machines and precincts.
GAO’s analysis found that some of the prior tests and reviews conducted
by the State of Florida and Sarasota County provide assurance that
certain components of the voting systems in Sarasota County functioned
correctly, but they are not enough to provide reasonable assurance that
the iVotronic DREs did not contribute to the undervote. Specifically,
GAO found that assurance is lacking in three areas, and proposes that
tests be conducted to address those areas. First, because there is
insufficient assurance that the firmware in all the iVotronic DREs used
in the election matched the certified version held by the Florida
Division of Elections, GAO proposes that a firmware verification test
be conducted on a representative sample of 115 (of the 1,499) machines
that were used in the general election. Second, because an insufficient
number of ways to select a candidate in the FL-13 race were tested, GAO
proposes that a test be conducted to verify all 112 ways that GAO
identified to select a candidate. Third, because no prior tests were
identified that address the effect of a miscalibrated iVotronic DRE on
the undervote, GAO proposes that an iVotronic DRE be deliberately
miscalibrated to verify the accurate recording of ballots under these
conditions. GAO expects these three tests would take 2 weeks, once the
necessary arrangements are made.
Should the task force ask GAO to conduct the proposed tests, several
matters would need to be addressed before testing could begin,
including obtaining access to the iVotronic DREs that have been subject
to a sequestration order, arranging for a test site, obtaining some
commercially available test tools, developing test protocols and
detailed test procedures, and arranging for the video recording of the
tests. Sarasota County election officials have indicated that they can
help GAO access the machines and provide a test site between November
26 and December 7, 2007.
Although the proposed tests could help provide increased assurance,
they would not provide absolute assurance that the iVotronic DREs did
not cause the large undervote in Sarasota County. The successful
conduct of the proposed tests could reduce the possibility that the
voting systems caused the undervote and shift attention to the
possibilities that the undervote was the result of intentional actions
by voters or voters that did not properly cast their votes on the
voting system.
To view the full product, including the scope and methodology, click on
[hyperlink, http://GAO-08-97T]. For more information, contact Keith
Rhodes at (202) 512-412 or rhodesk@gao.gov, or Naba Barkakati at (202)
512-4499 or arkakatin@gao.gov.
[End of section]
Mr. Chairman and Members of the Task Force:
I am pleased to appear before the task force today to present the
findings on our review of voting equipment used in Florida's 13th
Congressional District (Florida-13), which we are conducting in
response to your request of May 25, 2007.
In November 2006, about 18,000 undervotes were reported in Sarasota
County in the race for Florida's 13th Congressional District.[Footnote
1] Following the contesting of the election results in the House of
Representatives, the task force met and unanimously voted to seek GAO's
assistance in determining whether the voting systems contributed to the
large undervote in Sarasota County. On June 14, 2007, we met with the
task force and agreed upon an engagement plan, which included the
following review objectives: (1) What voting systems and equipment were
used in Sarasota County and what processes governed their use? (2) What
was the scope of the undervote in Sarasota County in the general
election? (3) To what extent were tests conducted on the voting systems
in Sarasota County prior to the general election and what were the
results of those tests? (4) Considering the tests that were conducted
on the voting systems from Sarasota County after the general election,
are additional tests needed to determine whether the voting systems
contributed to the undervote?
To conduct our work, we met with officials from the Sarasota County
Supervisor of Elections, the Florida Department of State and Division
of Elections, and Election Systems and Software (ES&S), the
manufacturer of the voting systems used in Sarasota County. We reviewed
voting system documentation, including standards documents, audit and
testing documentation, submissions from the contestant and contestee,
and selected Florida election laws and rules. In Sarasota County,
election officials demonstrated how the ES&S voting system was used to
support the 2006 general election. To determine the scope of the
undervote in Sarasota County, we collected election data from the
Supervisor of Elections and analyzed it to determine whether the
undervote could be attributed to particular voting machines or machine
characteristics. Specifically, we examined ballot image logs and event
logs from the voting systems and technician and incident reports
generated by elections staff from Sarasota County on election day. We
also conducted various statistical analyses to characterize the
undervote and to identify whether a subset of machines or precincts may
have caused the large undervote.
We reviewed test documentation and interviewed officials involved with
testing from ES&S, the Florida Division of Elections, and the Sarasota
County Supervisor of Elections. To determine the need for additional
tests, we also reviewed the tests conducted following the election,
including those conducted or sponsored by the Florida Division of
Elections, including the parallel testing, the examination of Sarasota
County's election procedures and practices, and the source code review
conducted at Florida State University's Security and Assurance in
Information Technology (SAIT) laboratory. We reviewed the final reports
of these tests and also met with the leader of the source code review
team. Following the agreement to and execution of a non-disclosure
agreement with the Florida Department of State and ES&S, we obtained
access to the iVotronic source code and reviewed it to further our
understanding of the system and to verify some of the source code
review's findings. We analyzed the available information and identified
a key set of voting system objectives that, if implemented properly,
would provide reasonable assurance that the voting systems did not
malfunction and cause the large undervote in Sarasota County. Using
these objectives, we used the results of testing previously conducted
and assessed the extent to which these key voting system objectives
could be met. For those objectives that could not be adequately
assured, we assessed the significance of those objectives and
identified tests that could be conducted to help try to assure those
key voting system objectives were met. For each test, we identified
resources that would be required, including time and manpower.
We provided a draft of this report to the Florida Department of State,
ES&S, and the Sarasota County Supervisor of Elections for their review
and comments. The Florida Department of State and ES&S also conducted a
sensitivity review to ensure that business proprietary information is
not disclosed in this statement.
We conducted our work from June to September 2007 in Washington, D.C.;
Tallahassee and Sarasota, Florida; and Omaha, Nebraska.
Results in Brief:
In the 2006 general election, Sarasota County used voting systems
manufactured by ES&S, specifically iVotronic direct recording
electronic (DRE) voting systems during early and election day voting
and the Unity election management system, which handles the election
administration functions, such as ballot design and election reporting.
Our independent analysis of the 2006 general election data from
Sarasota County confirmed the large undervote in the race for Florida's
13th Congressional District, but did not identify any particular voting
machines or machine characteristics that could have caused the large
undervote in the election. The undervotes in Sarasota County for the
congressional race were generally distributed across all machines and
precincts.
We found that some of the prior tests and reviews provide assurance
that the voting systems in Sarasota County functioned correctly, but
they are not enough to provide reasonable assurance that the iVotronic
DRE voting systems did not contribute to the undervote. For example,
prior reviews provide reasonable assurance that the Unity election
management system did not contribute to the undervote, and the votes
captured by iVotronic DREs at the precincts match the voter count from
precinct records within acceptable margins of error.
Portions of the Florida state audit, such as the firmware comparison
and parallel tests, provided useful information, but the results could
not be applied to all the iVotronic DREs used in the election because
the number of machines tested was too small. Additionally, the machines
were not tested for all different ways a voter can select a candidate
in the congressional race. We also did not find any prior testing that
would help us understand the effects of a miscalibrated touch screen.
To address these issues, we propose that (1) a firmware verification
test, (2) a ballot test, and (3) a calibration test be conducted to try
to obtain further assurance that the iVotronic DREs used in Sarasota
County during the 2006 general election did not cause the undervote.
The firmware verification test would compare the firmware in a
representative sample of iVotronic DREs with the certified version of
firmware. The ballot test would exercise 112 ways to select a candidate
on 10 iVotronic DREs. The calibration test would deliberately
miscalibrate an iVotronic DRE that uses the certified software and
verify the functioning of the machine. We expect the testing would take
2 weeks using a staff of about 6 to 8 people, once the necessary
arrangements have been made. Although the proposed tests would provide
increased assurance, they would not conclusively eliminate the machines
as a cause of the undervote.
Before commencing the testing, we would need to obtain access to the
iVotronic DREs that have been subject to a sequestration order in the
state court system of Florida, arrange for a test site, obtain some
commercially available software and hardware for the firmware
comparison test, develop test protocols and detailed test procedures,
and arrange for video recording of the test. Sarasota County election
officials have indicated that working around the county's election
schedules, they could help us access the machines and provide a test
site between November 26 and December 7, 2007.
Our proposed tests could help reduce the possibility that the undervote
was caused by the iVotronic DREs. However, even after completing the
tests, we would not have absolute assurance that the iVotronic DREs did
not play any role in the large undervote. Absolute assurance is
impossible to achieve because we are unable to recreate the conditions
of the election in which the undervote occurred. By successfully
conducting the proposed tests, we could reduce the possibility that the
iVotronic DREs were the cause of the undervote and shift attention to
the possibilities that the undervote was the result of intentional
actions by the voter or voters that did not properly cast their votes
on the voting system.
Draft copies of this statement were provided to the Secretary of State
of Florida, the Supervisor of Elections of Sarasota County, and ES&S
for their review and comment. The Florida Department of State provided
technical comments, which we incorporated. The Sarasota County
Supervisor of Elections did not provide us comments.
In its comments, ES&S stated that it believes that the collective
results of prior testing have demonstrated that the voting systems
worked properly in Florida's 13th Congressional District race, and that
the focus should be on testing the effect of the ballot display on the
undervote. We disagree that the prior test results adequately
demonstrate that the voting systems could not have contributed to the
undervote. Our analysis identified three areas where further testing
could provide increased assurance that the undervote was not caused by
the voting systems. We agree with ES&S that the large undervote in
Florida's 13th Congressional District race could have been caused by
voters who intentionally undervoted or voters who did not properly cast
their ballots, potentially because of issues related to the human
interaction with the ballot. However, our review focused on whether the
voting systems could have contributed to the large undervote. ES&S also
provided technical comments, which we incorporated as appropriate.
Background:
The 13th Congressional District of Florida comprises DeSoto, Hardee,
Sarasota, and parts of Charlotte and Manatee Counties. In the November
2006 general election, there were two candidates in the race to
represent the 13th Congressional District: Vern Buchanan, the
Republican candidate, and Christine Jennings, the Democratic candidate.
The State of Florida certified Vern Buchanan the winner of the
election. The margin of victory was 369 votes out of a total of 238,249
votes counted. Table 1 summarizes the results of the election and shows
that the results from Sarasota County exhibited a significantly higher
undervote rate than in the other counties in the congressional
district.
Table 1: Results from 2006 General Election for Florida Congressional
District 13:
County: Charlotte;
Buchanan: 4,460;
Jennings: 4,277;
Undervotes: 225;
Total ballots cast: 8,962;
Percentage undervote: 2.51.
County: DeSoto;
Buchanan: 3,471;
Jennings: 3,058;
Undervotes: 142;
Total ballots cast: 6,672;
Percentage undervote: 2.13.
County: Hardee;
Buchanan: 2,629;
Jennings: 1,686;
Undervotes: 269;
Total ballots cast: 4,584;
Percentage undervote: 5.87.
County: Manatee;
Buchanan: 50,117;
Jennings: 44,432;
Undervotes: 2,274;
Total ballots cast: 96,828;
Percentage undervote: 2.35.
County: Sarasota;
Buchanan: 58,632;
Jennings: 65,487;
Undervotes: 18,412;
Total ballots cast: 142,532;
Percentage undervote: 12.92.
County: Total;
Buchanan: 119,309;
Jennings: 118,940;
Undervotes: 21,322;
Total ballots cast: 259,578;
Percentage undervote: [Empty].
Source: GAO analysis of Florida Division of Elections, Charlotte
County, DeSoto County, Hardee County, Manatee County, and Sarasota
County data.
Note: Numbers do not add up because of overvotes - where voters select
more than the maximum number of candidates allowed in a race; in this
case, a ballot that had votes for both Buchanan and Jennings.
[End of table]
In Florida, the Division of Elections in the Secretary of State's
office helps the Secretary carry out his or her responsibilities as the
chief election officer. The Division of Elections is responsible for
establishing rules governing the use of voting systems in Florida.
Voting systems cannot be used in any county in Florida until the
Florida Division of Elections has issued a certification of the voting
system's compliance with the Florida Voting System Standards.[Footnote
2] The Florida Voting Systems Certification program is administered by
the Bureau of Voting Systems Certification in the Division of
Elections.
An elected supervisor of elections is responsible for implementing
elections in each county in Florida in accordance with Florida election
laws and rules. The supervisor of elections is responsible for the
purchase and maintenance of the voting systems as well the preparation
and use of the voting systems to conduct each election.
Sarasota County Used ES&S Voting Systems in 2006 General Elections:
In the 2006 general election, Sarasota County used voting systems
manufactured by ES&S. The State of Florida has certified different
versions of ES&S voting systems. The version used in Sarasota County
was designated ES&S Voting System Release 4.5, Version 2, Revision 2,
and consisted of iVotronic DREs, a Model 650 central count optical scan
tabulator for absentee ballots, and the Unity election management
system. It was certified by the State of Florida on July 17, 2006. The
certified system includes different configurations and optional
elements, several of which were not used in Sarasota County.
The election management part of the voting system is called Unity; the
version that was used was 2.4.4.2. Figure 1 shows the overall election
operation using the Unity election management system and the iVotronic
DRE.
Figure 1: Overview of Election Operation Using the Unity Election
Management System and iVotronic DRE:
[See PDF for image]
This figure is an illustration of flow of election operations, as
follows:
Election Data Manager: Program, store and format election data for a
jurisdiction; flows to: Ballot Image Manager: Create paper ballot
layouts and create information for personalized electronic ballot to be
used in iVotronic DREs; flows to: Hardware Programming Manager: Copy
election definitions for iVotronic DREs into PEBs; flows to: Precinct
Count (iVotronic DRE): iVotronic DREs are used to capture votes and
tabulate votes for a precinct; flows to: Data Access Manager, then to
Election Reporting Manager, then to election reports.
Precinct Count (iVotronic DRE) also: Copy precinct-level votes from
PEBs into Election Reporting Manager and ballot images and event logs
from compact flash cards.
Source: GAO.
[End of figure]
Sarasota County used iVotronic DREs for early and election day voting.
Specifically, Sarasota County used the 12-inch iVotronic DRE, hardware
version 1.1 with firmware version 8.0.1.2.[Footnote 3] Some of the
iVotronic DREs are configured with Americans with Disabilities Act
(ADA) functionality, which includes the use of audio ballots. The
iVotronic DRE uses a touch screen--a pressure-sensitive graphics
display panel--to display and record votes (see fig. 2).
Figure 2: The iVotronic DRE Voting System and Its Components.
[See PDF for image]
This is a photograph of the iVotronic DRE Voting System, highlighting
the following components: connection to communications pack and printer
(removed during voting); Vote button; Personalized electronic ballot;
and Touch screen.
Source: GAO.
[End of figure]
The machine has a storage case that also serves as the voting booth.
The operation of the iVotronic DRE requires using a personalized
electronic ballot (PEB), which is a storage device with an infrared
window used for transmission of ballot data to and from the iVotronic
DRE. The iVotronic DRE has four independent flash memory modules, one
of which contains the program code--firmware--that runs the machine and
the remaining three flash memory modules store redundant copies of
ballot definitions, machine configuration information, ballots cast by
voters, and event logs. The iVotronic DRE includes a VOTE button that
the voter has to press to cast a ballot and record the information in
the flash memory. The iVotronic DRE also includes a compact flash card
that can be used to load sound files onto iVotronic DREs with ADA
functionality. The iVotronic DRE's firmware can be updated through the
compact flash card. Additionally, at the end of polling, the ballots
and audit information are to be copied from the internal flash memory
module to the compact flash card.
To use the iVotronic DRE for voting, a poll worker activates the
iVotronic DRE by inserting a PEB into the PEB slot after the voter has
signed in at the polling place. After the poll worker makes selections
so that the appropriate ballot will appear, the PEB is removed and the
voter is ready to begin using the system. The ballot is presented to
the voter in a series of display screens, with candidate information on
the left side of the screen and selection boxes on the right side (see
fig. 3).
Figure 3: Second Ballot Page Showing the Congressional and
Gubernatorial Races in Sarasota County's 2006 General Election:
[See PDF for image]
This figure is an illustration of the ballot used in the general
election.
Source: Sarasota County Supervisor of Elections.
[End of figure]
The voter can make a selection by touching anywhere on the line, and
the iVotronic DRE responds by highlighting the entire line and
displaying an X in the box next to the candidate's name. The voter can
also change his or her selection by touching the line corresponding to
another candidate or by deselecting his or her choice. "Previous Page"
and "Next Page" buttons are used to navigate the multipage ballot.
After completing all selections, the voter is presented with a summary
screen with all of his or her selections (see fig. 4). From the summary
screen, the voter can change any selection by selecting the race. The
race will be displayed to the voter on its own ballot page. When the
voter is satisfied with the selections and has reached the final
summary screen, the red VOTE button is illuminated, indicating the
voter can now cast his or her ballot. When the VOTE button is pressed,
the voting session is complete and the ballot is recorded on the
iVotronic DRE. In Sarasota County's 2006 general election, there were
nine different ballot styles with between 28 and 40 races, which
required between 15 and 21 electronic ballot pages to display, and 3 to
4 summary pages for review purposes.
Figure 4: First Summary Page in Sarasota County's 2006 General
Election:
[See PDF for image]
This figure is an illustration of the summary ballot used in the
general election.
Source: Sarasota County Supervisor of Elections.
[End of figure]
Analysis of Election Data Shows that Undervote Was Distributed across
All Machines and Precincts:
Our analysis of the 2006 general election data from Sarasota County
does not identify any particular voting machines or machine
characteristics that could have caused the large undervote in Florida's
13th Congressional District race. The undervotes in Sarasota County for
the congressional race were generally distributed across all machines
and precincts. Using voting system data that we obtained from Sarasota
County, we found that 1,499 iVotronic DREs recorded votes in the 2006
general election; 84 iVotronic DREs recorded votes during early voting,
and 1,415 iVotronic DREs recorded votes on election day.[Footnote 4]
Using these data, we verified that the vote counts for the contestant,
contestee, and undervotes match the reported vote totals for Sarasota
County in Florida's 13th Congressional District race. As can be seen in
table 2, the undervote rate in early voting was significantly higher
than in election day voting.[Footnote 5]
Table 2: Undervotes in Florida's 13th Congressional District Race
during Early and Election Day Voting:
Machines:
All voters: 1,499;
Early voters: 84;
Election day voters:
1,415.
Ballots cast:
All voters: 119,919;
Early voters: 30,877;
Election day
voters: 89,042.
Undervotes:
All voters: 17,846;
Early voters: 5,445;
Election day
voters: 12,401.
Undervote rate:
All voters: 14.88%;
Early voters: 17.63%;
Election day
voters: 13.93%.
Source: GAO analysis of Sarasota County data.
[End of table]
The range of the undervote rate for all machines was between 0 and 49
percent, with an average undervote rate of 14.3 percent. When just the
early voting machines are considered, the undervote rate ranged between
5 and 28 percent. The largest number of undervotes cast on any one
machine on election day was 39. While the range of ballots cast on any
one machine on election day was between 1 and 121, the median number of
ballots cast on any one machine was 66. The range of undervote rate by
precinct was between 0 and 41 percent, and the average undervote by
precinct was about 14.8 percent.
Prior Tests and Reviews Provide Some Assurance, but Do Not Provide
Reasonable Assurance That the iVotronic DREs Did Not Contribute to the
Undervote:
Prior to the elections, Sarasota County's voting systems were subjected
to several different tests that included testing by the manufacturer,
certification testing by the Florida Division of Elections, testing by
independent testing authorities, and logic and accuracy testing by
Sarasota County's Supervisor of Elections. After the 2006 general
election, an audit of Sarasota County's election was conducted by the
State of Florida that included a review of the iVotronic source code,
parallel tests, and an examination of Sarasota County's election
procedures. Although these tests and reviews provide some assurance, as
do certain controls that were in place during the election, that the
voting systems in Sarasota County functioned correctly, they do not
provide reasonable assurance that the iVotronic DREs did not contribute
to the undervote.
Prior Tests and Reviews of Sarasota County's Voting Systems Provide
Useful Information, but Have Some Shortcomings:
According to ES&S officials, ES&S tested the version of the iVotronic
DRE that was used in Sarasota County in 2001-2002, but they could not
provide us documentation for those tests because the documentation had
not been retained.
The Florida Division of Elections conducted certification testing of
the iVotronic DRE and the Unity election management system before
Sarasota County acquired the system from the manufacturer. The
certification process included tests of the election management system
and the conduct of mock primary and general elections on the entire
voting system. ES&S Voting System, Release 4.5, Version 2, Revision 2,
was certified by the Florida Division of Elections on July 17, 2006.
According to Florida Division of Elections officials, testing of each
version focuses on the new components, and components that were
included in prior versions are not as vigorously tested. The 8.0.1.2
version of the iVotronic firmware was first tested as a part of ES&S
Release 4.5, Version 1, which was certified in 2005. Version 2
introduced version 2.4.4.2 of the Unity Election Management System,
which was certified in August 2005. Certification testing was conducted
on software that was received from an independent test authority, who
witnessed the building of the firmware from the source code. An
independent test authority also conducted environmental testing of the
iVotronic DRE in 2001 that was relied upon by the Florida Division of
Elections for certification.
A logic and accuracy test was conducted by Sarasota County on October
20, 2006, on 32 iVotronic DREs, and it successfully verified that all
ballot positions on all nine ballot styles could be properly recorded.
In addition, the use of a provisional ballot and audio ballot were
tested, as well as machines configured for early voting with all nine
ballot styles.
After the 2006 general election, the Florida Division of Elections
conducted an audit of Sarasota County's 2006 general election that
included two parallel tests, an examination of the certified voting
system and conduct of election by Sarasota County's elections office,
and an independent review of the iVotronic DRE firmware's source code.
After the conduct of this audit, the audit team concluded that there
was no evidence that suggested the official election results were in
error or that the voting systems contributed to the undervote in
Sarasota County.[Footnote 6] The parallel tests were performed using 10
iVotronic DREs--5 used in the 2006 general election and 5 that were not
used. Four of the machines in each test replicated the votes cast on
four election day iVotronic DREs. The fifth machine in each test used
an ad hoc test script that involved picking a random vote pattern along
with a specific vote selection pattern picked from 10 predetermined
vote patterns for the 13th Congressional District for each ballot cast.
The audit report asserts that testing a total of 10 machines is more
than adequate to identify any machine problems or irregularities that
could have contributed to undervotes in the Florida-13 race. However,
we concluded that the results from the testing of 10 machines cannot be
applied to all 1,499 iVotronic DREs used during the 2006 general
election because the sample was not random and the sample size was too
small.
In examining whether voting systems that were used in Sarasota County
matched the systems that were certified by the Florida Division of
Elections, the Florida audit team examined the Unity election
management system and the firmware installed on six iVotronic DREs. The
audit team confirmed that the software running on the Unity election
management system and the firmware in the six iVotronic DREs matched
the certified versions held in escrow by the Florida Division of
Elections. On the basis of its review, the audit team concluded that
there is no evidence to indicate that the iVotronic DREs had been
compromised or changed. We agree that the test verifies that those six
machines were not changed, but any extrapolation beyond this cannot be
statistically justified because the size of the sample is too small.
Therefore, these tests cannot be used to obtain reasonable assurance
that the 1,499 machines used in the general election used the certified
firmware.
A software review and security analysis of the iVotronic firmware
version 8.0.1.2 was conducted by a team led by Florida State
University's SAIT Laboratory. The eight experts in the software review
team attempted to confirm or refute many different hypotheses that, if
true, might explain the undervote in the race for the 13th
Congressional District. In doing so, they made several observations
about the code, which we were able to independently verify. The
software review and our verification of the observations were helpful,
but a key shortcoming was the lack of assurance whether the source code
reviewed by the SAIT team or by us, if compiled, would correspond to
the iVotronic firmware that was used in Sarasota County for the 2006
election. According to ES&S and Florida Division of Elections
officials, in May 2005 an independent testing authority witnessed the
process of compiling the source code and building the version of
firmware that was eventually certified by the Florida Division of
Elections. According to ES&S officials, if necessary, ES&S can recreate
the firmware from the source code, but the firmware would not be
exactly identical to the firmware certified by the Florida Division of
Elections because the embedded date and time stamp in the firmware
would be different.
The software review team also looked for security vulnerabilities in
software that could have been exploited to cause the undervote.
Although the team found several software vulnerabilities, the team
concluded that none of them were exploited in Sarasota in a way that
would have contributed to the undervote. We did not independently
verify the team's conclusion.
Reasonable Assurance of Some Voting System Objectives Has Been
Achieved:
The Unity election management system and the iVotronic DREs are the
major voting system components that may require testing to determine
whether they contributed to the large undervote in Sarasota County. Our
review of tests already conducted and documentation from the election
provide us reasonable assurance that the key functions of the Unity
election management system--election definition and vote tabulation--
did not contribute to the undervote. The election definitions created
using the Unity election management system are tested during logic and
accuracy testing to demonstrate that they include all races,
candidates, and issues and that each of the items can be selected by a
voter. The votes tabulated on the iVotronic DRE at each precinct
matched the data uploaded to the Unity election management system, and
the totals from the precinct results tapes agree with that obtained by
Unity. Further, the state audit confirmed that the Unity election
management system software running in Sarasota County matched the
escrowed version certified by the Florida Division of Elections.
We have reasonable assurance that the number of ballots recorded by the
iVotronic DREs is correct because this number is very close to the
number of people recorded on the precinct registers as showing up at
the polling places to vote either during early voting or on election
day. This assurance also allows us to conclude that issues, such as
votes cast by "fleeing voters"--votes that are cast by poll workers for
voters who leave the polling place before pressing the button to cast
the vote--and the potential loss of votes during a system shutdown, did
not affect the undervote in this election. If these issues had
occurred, they would have caused a discrepancy between the number of
voters who sign in at the polling place to vote and the public counts
recorded on the iVotronic DREs.
We have reasonable assurance that provisional ballots were
appropriately handled by the iVotronic DREs and the Unity election
management system. We also verified that during the Florida
certification test process, the Division of Elections relied on
successful environmental and shock testing conducted by an independent
test authority.
Reasonable Assurance That All iVotronic DREs Used in the 2006 General
Election Used Software Certified by the Florida Division of Elections
Is Lacking:
We found that prior testing and activities do not provide reasonable
assurance that all iVotronic DREs used in Sarasota County on election
day were using the hardware and firmware certified for use by the
Florida Division of Elections. Sarasota County has records indicating
that only certified versions were procured from ES&S, and the firmware
version is checked in an election on the zero and results tapes.
However, because there was no independent validation of the system
versions, we cannot conclude that no modifications were made to the
systems that would have likely made them inconsistent with the
certified version. As we previously mentioned, the firmware comparison
of only 6 iVotronic DREs in the state audit is insufficient to support
generalization to all 1,499 iVotronic DREs that recorded votes during
the election. Without reasonable assurance that all iVotronic DREs are
running the same certified firmware, it is difficult for us to rely on
the results of other testing that has been conducted, such as the
parallel tests or the logic and accuracy tests.
The Ability of Voters to Make Selections in Different Ways and Have
Their Votes Properly Recorded Has Not Been Fully Tested:
Prior testing of the iVotronic DREs only verified 13 of the 112 ways
that we identified that a voter may use to select a candidate in
Florida's 13th Congressional District race. Specifically, on an
iVotronic DRE, a voter could (1) initially select either candidate or
neither candidate (i.e. undervote), (2) change the vote on the initial
screen, and (3) use a combination of page back and review screen
options to change or verify his or her selection before casting the
ballot. By taking into account these variations, our analysis has found
at least 112 different ways a voter could make his or her selection in
Florida's 13th Congressional District race, assuming that it was the
only race on the ballot. Out of 112 different ways to select a
candidate in the congressional race, Florida certification tests and
the Sarasota County logic and accuracy tests verified 3 ways to select
a candidate; and the Florida parallel tests verified 10 ways to select
a candidate--meaning that of the 112 ways, 13 have been tested. By not
verifying these different ways to select a candidate, we do not have
reasonable assurance that the system will properly handle expected
forms of voter behavior.
The Effect of Miscalibrated iVotronic DREs Is Unclear:
During the setup of the iVotronic DRE, sometimes referred to as the
clear and test process, the touch screens are calibrated by using a
stylus to touch the screen at 20 different locations. The calibration
process is designed to align the display screen with the touch screen
input. It has been reported that a miscalibrated machine could affect
the selection process by highlighting a candidate that is not aligned
with what the voter selected. We identified two reported cases on
election day where the miscalibration of the iVotronic DRE led to its
closure and discontinued use for the rest of the day. While a
miscalibrated machine could certainly make an iVotronic DRE harder to
use, it is not clear it would have helped to contribute to the
undervote. We did not identify any prior testing or activities that
would help us understand the effect of a miscalibrated iVotronic DRE on
the undervote.
Further Tests Could Provide Increased but Not Absolute Assurance That
the iVotronic DREs Used in the Election Did Not Cause the Undervote:
On the basis of our analysis of all prior test and audit activities, we
propose that a firmware verification test, a ballot test, and a
calibration test be conducted to try to obtain increased assurance that
the iVotronic DREs used in Sarasota County during the 2006 general
election did not cause the undervote.
We propose that the firmware verification testing be started first,
once the necessary arrangements have been made, such as access to the
needed machines and the development of test protocols and detailed test
procedures. Once we have reasonable assurance that the iVotronic DREs
are running the same certified firmware, we could conduct the ballot
test and calibration test on a small number of machines to determine
whether it is likely the machines accurately recorded and counted the
ballots. If the firmware verification tests are successfully conducted,
we would have much more confidence that the iVotronic DREs will behave
similarly when tested. If there are differences in the firmware running
on the iVotronic DREs, we would need to reassess the number of machines
that need to be tested for ballot testing and calibration testing in
order for us to have confidence that the test results would be true for
all 1,499 iVotronic DREs used during the election. In other words, if
we are reasonably confident that the same software is used in all 1,499
machines, then we are more confident that the results of the other
tests on a small number of machines can be used to obtain increased
assurance that the iVotronic DREs did not cause the undervote. Although
the proposed tests would provide increased assurance, they would not
conclusively eliminate the machines as a cause of the undervote.
Conduct Firmware Testing to Verify That the Firmware in the iVotronic
DREs Used in Sarasota County Matches the Certified Version:
We propose to conduct a firmware verification test using a statistical
sampling approach that can provide reasonable assurance that all 1,499
iVotronic DREs are running the certified version of firmware. The exact
number of machines that would be tested depends on the confidence level
desired and how much error can be tolerated. We propose drawing a
representative sample from all the iVotronic DREs that recorded votes
in the general election. With a sample size of 115 iVotronic DREs,
which would be divided between sequestered and nonsequestered machines,
and assuming that there are no test failures, we would be able to
conclude with a 99 percent confidence level that no more than 4 percent
of the 1,499 iVotronic DREs used in the election were using uncertified
firmware.
We suggest a test approach similar to what was used by the Florida
Division of Elections when it verified the firmware for 6 iVotronic
DREs. We estimate that the firmware testing for 115 machines could be
conducted in about 5 to 7 days and would require about 5 or 6 people,
once the necessary arrangements have been made. The machines would be
transported to a test facility specified by Sarasota County election
officials where we could perform the test. The activities involved in
conducting a firmware validation test would include locating and
retrieving the selected iVotronic DRE from the storage facility,
transporting it to the test facility, opening the DRE, extracting the
chip with the firmware, reading the contents of the chip using a
specialized chip reader, and conducting a comparison between the
contents and the certified firmware to determine if any differences
exist. To conduct this test, we would need commercially available
specialized hardware and software similar to that used by the Florida
Division of Elections in its firmware comparison test.
Conduct Ballot Testing of iVotronic DREs to Confirm Correct Operation:
We propose conducting ballot testing on 10 iVotronic DREs, each
configured with one of the nine different ballot styles, with the 10th
machine configured as an early voting machine with all nine ballot
styles. We would test 112 ways to select a candidate on the early
voting machine. On the election day machines, we would test the 112
different ways distributed across the 9 machines in a random manner,
meaning each machine would on average record 12-13 ballots. Assuming
that (1) reasonable assurance is obtained that all iVotronic DREs used
during the election were using the same certified firmware, and (2) we
found no failures during the ballot testing, this testing would provide
increased assurance that the iVotronic DREs used during the election,
both in early voting and in election day voting, were able to
accurately record and count ballots when using any of the 112 ways to
select a candidate in the Florida-13 race.
We would plan to code each ballot by including an identifier in the
write-in candidate field for either the U.S. senator or governor's
race. Using this write-in coding, we could examine the ballot image and
confirm that each ballot was accurately recorded and counted by the
iVotronic DRE. Any encountered failures would also be more rapidly
attributed to a specific test case, and we would be able to more
readily repeat the test case to determine if we have a repeatable
condition. Testing 112 ways to select a candidate on a single machine
would also provide us some additional assurance that the volume of
ballots cast on election day did not cause a problem. We note that
casting 112 ballots on a single machine is more than that cast on over
99 percent of the 1,415 machines used on election day. We estimate the
ballot testing would take about 2 to 3 days and require the equivalent
of 2 people, once the necessary arrangements have been made.
Deliberately Miscalibrate an iVotronic DRE to Understand the Effect on
the Undervote:
Because little is known about the effect of a miscalibrated machine on
the behavior of an iVotronic DRE, we propose to deliberately
miscalibrate an iVotronic DREs and verify the functioning of the
machine. We propose to identify different ways to miscalibrate a ballot
and to test ballots on the miscalibrated iVotronic DRE to verify that
it still properly records votes. With this test we would confirm
whether (1) the review screen displays the same selection in the
Florida-13 race as was highlighted in the selection screen, and (2)
that the vote is recorded as it was displayed on the review screen.
Again, we would plan to use the write-in candidate option to verify the
proper recording of the ballot. This test would demonstrate whether the
system correctly records a vote for the race and hence whether it
contributed to the undervote. We estimate that the calibration test
could be completed in about 1 day by 2 people, once the necessary
arrangements have been made.
Several Matters Remain to Be Addressed to Conduct Further Testing:
Should the task force ask us to conduct the proposed testing, we want
to make the task force aware of several other matters that would need
to be addressed before we could begin testing. These activities would
require some time and resources to complete before testing could
commence.
First, we would need to gain access to iVotronic DREs that have been
subject to a sequestration order in the state court system of Florida.
If we do not have access to the needed machines, we would be unable to
obtain reasonable assurance that the machines used on election day were
using certified software, and without this assurance, the results from
prior tests and any results of our ballot and calibration tests would
be less meaningful because we would be unable to apply the results to
all 1,499 iVotronic DREs used during the election. Second, we would
need to agree upon an appropriate facility for the tests. Sarasota
County Supervisor of Elections has indicated that we can use its
warehouse space, but because of upcoming elections in November and
January, the only time the election officials would be able to provide
us this space and the necessary support is between November 26 and
December 7, 2007. If testing cannot be completed during this time
period, Sarasota County officials stated that they would not be able to
assist us until February 2008. Third, some tests may require
commercially available specialized software, hardware, or other tools
to conduct the tests. We would need to make arrangements to either
borrow or to purchase such testing tools before commencing testing.
Fourth, in order to conduct any tests, we would need to develop test
protocols and detailed test procedures and steps. We also anticipate
that we would need to conduct a dry run, or dress rehearsal, of our
test procedures to ensure that our test tools function properly and
that our time estimates are reasonable. Finally, we would need to make
arrangements for video recording of our testing. It would be our
preference to have a visual record of the tests to document the actual
test conduct and to facilitate certain types of test analysis.
Other Observations on Touch Screen Voting Systems:
We recognize that human interaction with the ballot layout could be a
potential cause of the undervote. Although we have not explored this
issue in our review, we note that there is an ongoing academic study
that is exploring this issue using voting machines obtained from ES&S.
We believe that such experiments could be useful and could provide
insight into the ballot layout issue.
During our review, we noted that several suggestions have been offered
as possible ways to establish that voters are intentionally undervoting
and to provide some assurance that the voting systems did not cause the
undervote. First, a voter-verified paper trail could provide an
independent confirmation that the touch screen voting systems did not
malfunction in recording and counting the votes from the election. The
paper trail would reflect the voter's selections and, if necessary,
could be used in the counting or recounting of votes. This issue is
recognized in the Florida State University SAIT source code review as
well as the 2005 and draft 2007 Voluntary Voting Systems Guidelines
prepared for the Election Assistance Commission. We have previously
reported on the need to implement such a function properly.[Footnote 7]
Second, explicit feedback to voters that a race has been undervoted and
a prompt for voters to affirm their intent to undervote might help
prevent many voters from unintentionally undervoting a race. On the
iVotronic DREs, such feedback and prompts are provided only when the
voter attempts to cast a completely blank ballot, but not when a voter
undervotes in individual races. Third, offering a "none of the above"
option in a race would provide voters with the opportunity to indicate
that they are intentionally undervoting. The State of Nevada provides
this option in certain races in its elections. Decisions about these or
other suggestions about ballot layout or voting system functions should
be informed by human factors studies that assess their effectiveness in
accurately recording voters' preferences, making voting systems easier
to use, and preventing unintentional undervotes.
Conclusions:
The high undervote encountered in Sarasota County in the 2006 election
for Florida's 13th Congressional District has raised questions about
whether the voting systems accurately recorded and counted the votes
cast by eligible voters. Other possible reasons for the undervote could
be that voters intentionally undervoted or voters did not properly cast
their ballots on the voting systems, potentially because of issues
relating to the interaction between voters and the ballot. The focus of
our review has been to determine whether the voting systems--the
iVotronic DREs, in particular--contributed to the undervote. We found
that the prior reviews of Sarasota County's 2006 general election have
provided valuable information about the voting systems. Our review
found that in some cases we were able to rely on this information to
eliminate areas of concern. This allowed us to identify the areas where
increased assurances were needed to answer the questions being raised.
Accordingly, the primary focus of the tests we are proposing is to
obtain increased assurance that the results of the prior reviews and
our proposed testing can be applied to all the iVotronic DREs used in
the election. Our proposed tests involving the firmware comparison,
ballot testing, and calibration testing could help reduce the
possibility that the undervote was caused by the iVotronic DREs.
However, even after completing the tests, we would not have absolute
assurance that the iVotronic DREs did not play any role in the large
undervote. Absolute assurance is impossible to achieve because we are
unable to recreate the conditions of the election in which the
undervote occurred. By successfully conducting the proposed tests, we
could reduce the possibility that the iVotronic DREs were the cause of
the undervote and shift attention to the possibilities that the
undervote was the result of intentional actions by the voter or voters
that did not properly cast their votes on the voting system.
Comments and Our Evaluation:
We provided draft copies of this statement to the Secretary of State of
Florida, the Supervisor of Elections of Sarasota County, and ES&S for
review and comment. The Florida Department of State provided technical
comments, which we incorporated. The Sarasota County Supervisor of
Elections appreciated the opportunity to review the draft, but provided
us no comments.
In its comments, ES&S stated that it believes that the collective
results of testing already conducted on the Sarasota County voting
systems have demonstrated that they performed properly and as they were
designed to function and that all votes were accurately captured and
counted as cast in Florida's 13th Congressional District race. Further,
ES&S asserts that tests and analyses should be conducted to examine the
effect of the ballot display on the undervote, which it believes is the
most probable cause of the undervote.
We disagree that the collective results of testing already conducted on
the Sarasota County voting systems adequately demonstrate that the
voting systems could not have contributed to the undervote in the
Florida-13 race. First, as we have cited, we do not have adequate
assurance that all the iVotronic DREs used in Sarasota County used the
firmware certified by the Florida Division of Elections. Without this
assurance, it is difficult for us to apply the results from the other
tests to all 1,499 machines that recorded votes during the election
because we are uncertain that all machines would have behaved in a
similar manner. Further, we believe that expected forms of voter
behavior to select a candidate in the Florida-13 race were not
thoroughly tested. While ES&S asserts that such processes would have no
effect on the iVotronic DRE's ability to capture and record a voter's
selection, we did not identify testing that verified this. Further,
while ES&S states that the testing of a deliberately miscalibrated
iVotronic DRE would result in a clearly visible indication of which
candidate was selected, we could not identify any testing that
demonstrated this.
We acknowledge that the large undervote in Florida's 13th Congressional
District race could have been caused by voters who intentionally
undervoted or voters who did not properly cast their ballots,
potentially because of issues related to the human interaction with the
ballot. However, the focus of our review, as agreed with the task
force, was to review whether the voting systems could have contributed
to the large undervote. ES&S also provided technical comments, which we
incorporated as appropriate.
Mr. Chairman, this completes my prepared statement. I would be happy to
respond to any questions you or other members of the task force may
have at this time.
Contacts and Acknowledgments:
For further information about this statement, please contact Keith
Rhodes, Chief Technologist, at (202) 512-6412 or rhodesk@gao.gov, or
Naba Barkakati at (202) 512-4499 or barkakatin@gao.gov. Contact points
for our Offices of Congressional Relations and Public Affairs may be
found on the last page of this statement. Other key contributors to
this statement include James Ashley, James Fields, Jason Fong, Cynthia
Grant, Geoffrey Hamilton, Richard Hung, John C. Martin, Jan Montgomery,
Jennifer Popovic, Sidney Schwartz, and Daniel Wexler.
[End of section]
Footnotes:
[1] Undervotes occur when the number of choices selected by the voter
is fewer than the maximum allowed for that contest. In this case, it
means ballots that did not record a selection for either candidate in
the congressional contest.
[2] Florida Department of State, Florida Voting System Standards, Form
DS-DE 101 (Jan. 12, 2005).
[3] The certified version of ES&S Voting System Release 4.5, Version 2,
Revision 2, specifies the use of iVotronic hardware version 1.0.
According to Florida Division of Election officials, hardware version
1.1 of the iVotronic DRE has been available since at least 2004 and
should have been included as a part of the certification for ES&S
Voting System Release 4.5, Version 2, Revision 2. According to ES&S
officials, iVotronic firmware version 8.0.1.2 runs in exactly the same
manner on hardware versions 1.0 and 1.1.
[4] Election day voting is the casting of ballots on election day at
polling places. Absentee and early voting are programs that permit
eligible persons to vote prior to election day. Absentee voting is
conducted by mail in advance of election day and early voting is
generally in-person voting in advance of election day at specific
polling locations.
[5] Early and election day ballots include provisional ballots cast
during those respective stages of voting and included in the vote
totals. 160 provisional ballots were included in the vote totals. 37
provisional ballots were excluded.
Because the absentee ballots were not cast using iVotronic voting
systems, we did not verify the absentee ballot counts. When absentee
ballots are included, a total of 142,532 ballots were cast and a total
of 18,412 undervotes were recorded.
[6] Florida Department of State, Audit Report of the Election Systems
and Software, Inc.'s, iVotronic Voting System in the 2006 General
Election for Sarasota County, Florida (Tallahassee, Florida: Feb.
2007), and Security and Assurance in Information Technology Laboratory,
Florida State University, Software Review and Security Analysis of the
ES&S iVotronic 8.0.1.2 Voting Machine Firmware (Tallahassee, Florida:
Feb. 23, 2007).
[7] GAO, Elections: Federal Efforts to Improve Security and Reliability
of Electronic Voting Systems Are Under Way, but Key Activities Need to
Be Completed, GAO-05-956 (Washington, D.C.: Sept. 21, 2005).
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation, and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov). Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office: 441 G Street NW, Room LM:
Washington, DC 20548:
To order by Phone: Voice:
(202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Gloria Jarmon, Managing Director, JarmonG@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, DC 20548:
Public Affairs:
Susan Becker, Acting Manager, BeckerS@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, DC 20548: