Diebold knew about the flaw for years

A report released on Monday by California's secretary of state, Debra Bowen, explains how nearly 200 votes were deleted from the official results for Humboldt County during November's presidential election, and identifies several problems with the e-voting technology from Premier Election Solutions, which was used by the county.

Among the problems identified in the report (PDF document) is one that allows operators of some e-voting machines from Premier, a Diebold subsidiary formerly called Diebold Elections Systems, to delete crucial audit logs that, under federal standards, are supposed to be stored permanently on the systems.

In addition, the report noted the version of Premier's Global Election Management System (GEMS) that was used in Humboldt County's e-voting machines failed to maintain required logs of important system events, and generated inaccurate data and time stamps in several cases.

Later, the company changed its story and said the problem had been caused by a "logic error" in its GEMS source code, not McAfee's software.

Secretary of State Bowen's report, which was submitted to the U.S. Election Assistance Commission, said that Premier had known about the problems for at least four years but had not adequately warned the county about it. "The number of votes erroneously deleted from the election results reported by GEMS in this case greatly exceeds the maximum allowable error rate established by [Help America Vote Act]," the report noted.

The defects in the software version violate voting system standards established in 1990 and would have made the systems ineligible for use in an election had they been detected, the report said.

The secretary's office initiated the investigation after a volunteer group called the Humboldt County Election Transparency Project conducted an independent scan of all the ballots cast in the county during the presidential elections and discovered that 197 more ballots had been cast than in the official count. An inquiry into that discrepancy later led to the issue in the GEMS software.

According to the report, the error resulted from a so-called "Deck 0" flaw in the central counting server in the version of GEMS software used in Humboldt County's e-voting machines. The error "silently deletes" all tallied votes from the first batch or "deck" of ballots that is scanned into the system, the secretary's report said. The deletion results whenever an operator, at any point after the first batch of voted ballots is scanned into the system, deletes any subsequent batch for any reason, the report said.

Premier had known about the vulnerability for years and even had a workaround for it, but it did not communicate it to Humboldt County officials, the report said. Though it knew of the problem since at least October 2004, the company did not notify the Election Assistance Commission nor the National Association of State Election Directors. Instead, the company sent "a vague e-mail to elections officials" in the 11 California counties using the faulty version of GEMS containing the workaround, but without indentifying the problem.

Equally serious was the discovery that the GEMS version used in Humboldt County failed to record or log many important system events, such as the deletion of ballot decks after they had been scanned into the system. Such intentional deletion of "committed decks" happens fairly regularly during an election, such as when an operator thinks the ballots have been improperly scanned, and is supposed to be logged at all times, the report said.

While Premier's central counting server log captured information whenever ballots were rescanned into the system, it did not capture deletions, the report said. The time and date stamps, which are supposed to reflect when the ballots were scanned into the systems, were also wrong. One deck had a date stamp of Nov. 25, though the ballots had been scanned into the system on Nov. 3.

The most "stunning" finding in the report was that the software provided a way for operators to permanently delete audit records from the system, said John Gideon, co-executive director of the VotersUnite.org Web site.

A "Clear" button on the audit screen gave system operators a way to erase information on system activity related to the vote tally, including any system events that required operator intervention, such as deleting and rescanning ballot decks. The "Clear" button is situated between the "Save As" and "Close" buttons and can be accidentally clicked on very easily, the secretary's report noted, pointing to an instance in another county where an official accidentally deleted the logs while trying to print them.

The "Clear" button offers no indication that clicking it would result in the permanent deletion of logs, and it does not require any confirmation from the operator, the report said.

That Premier knew that this capability to delete crucial log information existed despite being a violation of federal standards was surprising, Gideon said. "Audit logs are part of the history of the machine. It is supposed to be there for the life of the system," he said. "They clearly violated federal requirements, and we have been trying since early December to get the [Election Assistance Commission] to refer this to the Department of Justice."
What was surprising was that "this system was federally certified more than once," Gideon said.

Premier Election Solutions did not respond immediately to a request for comment.