A public hand-count of paper ballots in Palm Beach County, FL over the weekend has decisively determined the winners and losers of several disputed elections after paper ballot optical-scan computer tallying systems made by Sequoia Voting Systems (now owned by Dominion Voting) declared the incorrect "winners" of several races in a March 13th election.

But a dispute over who is to blame for the initial failure flared up again over the weekend as Dominion issued a statement that seems to contradict their previous admission that their software was to blame.

"The hand-count was 100%. We weren't missing a ballot," the Palm Beach County Supervisor of Elections Susan Bucher told The BRAD BLOG this afternoon about what happened over the weekend. "Frankly, without paper ballots and without audits, we would have let the wrong winners serve."

What will happen next, however --- for Palm Beach County, one of Florida's largest, as well as the other 285 jurisdictions across the country where the very same voting system is currently in use --- is anything but clear...

Democracy's Gold Standard

John Greene, who was initially announced as the loser of the Wellington City Council race by the Dominion/Sequoia system was pleased with the results of Saturday's public hand-count which was only allowed in the state of Florida --- where it is now illegal to manually examine paper ballots for a hand-count after they've already been tabulated by a computer --- after a judge gave the okay.

"Regardless of where people stood, who they supported, I think they really feel that we've got the right people elected, their vote counted," Greene told WPTV, the local NBC affiliate in Palm Beach County, FL after the manual count. [See video embedded below].

As importantly, if not more so, Shauna Hostetler, who was initially announced as the "winner" of Wellington's Council Seat 1, only to learn she didn't actually win after all, said, "I am very happy that the recount was approved and done and thankful for those who gave their whole day here."

Saturday's public hand-count of paper ballots, which we've long described as "Democracy's Gold Standard" (at least when its done on Election Night, at the precinct, in front of the public) settled all questions about the results. But it hasn't settled all question about what went wrong, and whether voters in the 14 states which currently use the faulty Dominion/Sequoia system --- many of them swing states --- should rely on the results reported by it.

Rather than hand-counting in the first place, and avoiding such problems, Palm Beach County relied on easily-manipulated, oft-malfunctioning optical-scan computer systems to tally the paper ballots. Or, in this case, to mis-tally the ballots. Unfortunately, the state of Florida gives them little choice, as hand-counting ballots is now illegal in the Sunshine State.

Similar, if not identical paper-based computer tallying systems are set to be used across the country for some 70% of the votes cast in this year's Presidential Election, despite the myriad of failures and vulnerabilities in such systems that we've been reporting here at The BRAD BLOG for nearly a decade.

As we explained last week, the computer systems in Palm Beach had mis-reported the results of several races, including the race for Mayor. A post-election audit (really, a spot-check, now mandated by Florida law) tipped off election officials that something had gone terribly wrong. After receiving permission from the court, the hand-count was carried out and the questions were settled. At least the questions about this particular election.

This time officials and candidates and voters were lucky enough that someone bothered to discover the problem. But who knows if that has always been the case in Palm Beach County and elsewhere that the same and similar systems are used.

"I have a feeling that this isn't the first time, but we never noticed," Bucher told us today, referring to elections that took place prior to her tenure as Supervisor of Elections before new state laws mandated a manual 2% post-election spot-check "audit" of paper ballots.

Dominion Changes Its Story

On March 23rd, Waldeep Singh, the Vice President of Dominion Voting, the Canadian-based firm which now oversees and services Sequoia Voting Systems' hardware and software, issued a statement on behalf of the company [PDF] taking responsibility for the failure. He chalked up the failure to "a mismatch between the software which generates the paper ballots and the central tally system."

By Friday, however, Bucher's main contact with the company, Vice President Howard Cramer, was suddenly "on vacation" and unreachable, according to Bucher, and the company was singing a very different tune in a separate statement [PDF] sent directly to the Florida Secretary of State.

"It is clear that the mismatch was not the result of a 'bug' that generated this anomaly," the letter from Dominion President John Poulos to Sec. of State Ken Detzner reads. Poulos says that the software used to create ballot definitions and process results "acted as designed and, as such, it is our opinion that this is not an issue that could have been revealed during state certification testing."

The company concedes, however, "that a shortcoming in the user interface of WinEDS 3.1 makes identifying and recovering from such an event more difficult than it could be."

Poulos went on to recommend that upgrading to WinEDS 4.0, a version which is not currently certified for use by the state of Florida, "provides enhanced features which make it easier for the operator to take corrective action when such a mismatch occurs more readily than in the existing version of software which is in use."

In other words, it wasn't a bug, it was a feature. A bad one. And one that Bucher says her staff could have done little about, even if they had known about the system's shortcoming in advance, since the software is proprietary and her staff is not even allowed to look at most parts of it --- by law. She says she had never even seen the screen the company now recommends using in order to correct the problem, even if they had noticed it before the election somehow. The industry standard "Logic and Accuracy" tests she had performed before the election failed to reveal the problematic 'feature'.

Dominion President Poulos' newer claim to the Secretary of State, that the problem was "not the result of a 'bug'", is in stark contrast to what Dominion Vice President Singh had told Bucher previously, when he sent her his statement on company letterhead explaining: "This synchronization difficulty is a shortcoming of the version of software currently being used by Palm Beach County and that shortcoming has been addressed in a subsequent version of the software."

A "Product Advisory Notice" issued March 30th [PDF] --- almost three weeks after the failed Palm Beach County election and two days after Poulos' letter to the Sec. of State claiming it wasn't the fault of the software --- backs up Bucher's account that the problem is, in fact, in the software:

When a WinEDS database is edited for example to add title blocks to an AVC Edge display layout, the contest order on the ballots in the database can become out of sync with the contest order shown on the corresponding paper ballots. If not identified via the Ballot Management component of WinEDS or through pre-election Logic and Accuracy Testing and corrected, election results will show the correct number of votes, but assigned to the wrong candidate or measure response.

Dominion's "Product Advisory Notice" goes on to explain how to overcome the problem in several different versions of their software and even indicates the problem still exists in WinEDS 4.0 --- the version Poulos told the state to upgrade to --- despite the fact that Singh's statement to Bucher claims the "shortcoming has been addressed in a subsequent version of the software...to prevent such an anomaly from occurring in the future."

The "Product Advisory Notice" is "an admission," Bucher says, that the failure was not due to her or her staff, but that "the software operated just as it should have" --- incorrectly, in this case, with losers being reported as "winners" and vice versa.

She says the designer of the program also admitted to her that he had seen the problem before.

'The Technology Failed, Our Procedures Prevailed'

The inaccurate results were discovered during a routine 2% post-election hand-audit of the March 13th election, Bucher explained.

"We go in to have the post-election audits, 15 cities worked fine, one city didn't. The ballots we counted in the first two precincts matched the results (reported by the poll tapes printed by the precinct-based optical-scanners) for a different candidate."

She knew they had a problem, but what to do about it was another question. Legal complaints started flying, and a judge ultimately allowed for the complete hand-count that occurred over the weekend. Without an order from a judge, it is illegal to hand-count ballots in the state of Florida, even if problems are found like the one that was discovered by the Supervisor of Elections in Palm Beach County during the spot-check of the March 13th election.

Bucher was a Democratic legislator in Florida when the state passed the law allowing for just six days to certify an election. She says she was "ecstatic" when they were able to broker a deal that allowed for the 2% post-election audit, but didn't realize at the time that election officials would run smack-dab into a nightmare if results were certified before the post-election spot-check audit could be satisfactorily completed and resolved in the event that problems were discovered.

In this case, they were able to resolve them with a hand-count and the permission of a judge. In the event of a larger race, such as a Presidential Election, Bucher is not as confident they wouldn't have had a 2000-style nightmare on her hands.

Palm Beach County was the home of the infamous Butterfly Ballot during the 2000 Presidential Election debacle, though Bucher, who says she ran for Supervisor of Elections to fix the long-standing problems in Palm Beach, was only elected in 2008. She says the county has run 23 successful elections under her stewardship, to her knowledge, until this one.

This time, she says, "the technology failed, but our procedures prevailed."

Whether or not that will be the case come this November remains a separate question. Right now, she says, she's uncertain how the county --- the largest one in the state to use the Dominion/Sequoia system --- will proceed. The failure is currently under investigation by the state, though clearly Bucher is furious at the way Dominion has handled the problem and then, more recently, tried to suggest to the media that the fault was hers rather than theirs.

The options for moving to a different voting system and a different vendor, as allowed by the state of Florida, which has yet to even certify the newer version of WinEDS, are few. She says moving to another company's optical-scanners, such as ES&S, whose systems are used across most of the state, is not a good option either, as there have been reported failures with their systems as well.

In February, we reported on ES&S optical scanners which failed in the state of New York during the 2010 election. In one South Bronx precinct, as discovered almost two years after the fact by the New York Daily News, the op-scanners mistallied some 70% of the ballots cast in the September primary election and then 54% of the ballots cast in the November general election.

In Florida, however, Bucher is allowed no such option, as the state legislature has made hand-counting paper ballots illegal without a court order. That has left the Elections Supervisor none too happy. "I don't want you to have to trust your election officials," Bucher told us. "I think your election officials have to prove it."

That's difficult to do when we're all forced to trust in secret vote counting by oft-failed computer systems. And yet, that's what voters will be forced to do once again around the country this year.

According to Verified Voting's database, voting systems made by Sequoia, featuring the "not-bug" which hit Palm Beach in March, are used in some 285 jurisdictions around the country, including a number of swing states.

Sequoia systems will be used again this year in Arizona, California, Colorado, Florida, Idaho, Illinois, Louisiana, Michigan, Missouri, New Jersey, Oregon, Pennsylvania, Virginia and almost all of Wisconsin, where the GOP Presidential Primary will be held tomorrow, and where contentious recall elections for the state's Republican Governor Scott Walker, his Lt. Governor Rebecca Kleefisch and four Republican state Senators are set to begin next month.

Similar paper-based optical-scan systems are used in every other state in the union, and in almost none of them are results fully verified --- or often even verified at all --- before "winners" are announced to the public and official certified as "accurate", whether they actually are or not.

CORRECTION: We had originally reported that Dominion Vice President Waldeep Singh had suddenly gone on vacation in the middle of Palm Beach trying to figure out what had happened. It was, in fact, Palm Beach County's Dominion rep, Vice President Howard Cramer (formerly of Sequoia Voting Systems) who had left for vacation before the county had been able to fully resolve the cause of the problem. We regret the error.

Corporate instead of human/people control anyone? How could something so simple that everyone is waiting to do... be so simple so as to actually work?
People work it with hand marked paper ballots and watch it so chain of command is followed! Posted results on precinct doors!

I just got a comment asking for your help in providing some news coverage. This is from youtube user kearnspalmbay (David Kearns) who has posted several videos titled "Three Minutes for Democracy" in which he confronts Hillsborough County (FL) commissioners, demanding investigation into false voting tallies during the General Election. He claims that the Supervisor of Elections are dirty, that they are stealing elections. 38,000 ballots were re-scanned - not a "glitch". The commissioners just keep forwarding his complaints to the State Dept of Elections. He has also put in complaints with the Justice Department.

@Heidi: with an attitude like that towards the ONLY person who has consistently reported on our compromised elections, Brad Friedman, if I were Mr. Friedman I'd ignore the guy too. He may have his facts straight but that's not an excuse to attack someone with a common cause. His attacks were totally appropriate however to the people who are allowing this to go unchecked.

Heidi & Brad - I am VERY familiar with David Kearns fantastic work in Hillsborough Co.(as is John Washburn, Bill Bucolo, Chris Brudy and several other election trench-buddies of mine in FL). David's a member of our Occupy Rigged Elections Facebook group, and he's got some amazing documentation that Brad really might wanna take a look at...

In the 2010 Sink / Scott race, 38,000 ballots had to be re-scanned late on election night due to a problem with the memory cards. Apparently there had been a "hot removal" - the memory cards had been ejected WHILE they were uploading the data; producing defunct memory cards that were sent out to various districts in Tampa (guess which ones?); memory cards that were unable to record or retain ballot tallies.

They came up blank, if I'm not mistaken. Officials made a lovely show of the re-scanning, done by officials, late at night, behind glass - but there were tons of questions left unanswered.

David has been hammering away at election officials in Hillsborough Co to get documentation - making video after video / blogging / writing a book about it. Finally, several weeks ago we put him in touch w/ the FL officials who helped him get the documentation he needed...

...an admission from Diebold / Dominion that it was all their fault.

None of us can understand how a "hot removal" could occur. Even the most basic computer novice knows you can't remove memory cards while they are uploading / transferring data. (???)

I have been contacted by the SOE' Peg Reese and was provided a document dated March 15, 2012 from Dominion Voting concerning questions I have raised with you.

The document indicates that one machine, as Commissioner Sharpe said in January, was responsible.

However the company noted that it would have been better for the staff to use a number of machines rather than one, in downloading vote counting software to the memory cards.

This solidifies the assertions I made during the March 8, 2012 meeting that it is your duty to demand that the Division of Elections investigate the election of Nov. 2, 2010 as conducted in Hillsborough County.

I urge you to thoughtfully consider the odds against the following series of events all taking place resulting in the "rescanning" of 38,000 ballots.

Only one machine ( the defective one) was selected and used in downloading the software to capture those votes out of as many as 5 available machines.

The 12 failing memory cards were sent out to early voting places in locations I have mentioned repeatedly, and are largely democratic. A resultant repeat of the 2008 "glitch" but for a different technical reasons.

No test of these cards was apparently made to assure they functioned properly either prior to sending them out, or during a test of the OSX machines at those locations.

I remind the commissioners the county SOE has taken considerable time in providing ANY documentation with regard to good faith questions I have posed regarding this election. The SOE's top three officers made false statements to me regarding the times and context of this election malfunction.

While the commission was well aware of problems associated with the machine, as evidenced by Commissioner Sharpe's statement to press on Nov. 4, 2010, and Jan. 5, 2011, the commission is essentially mum thereafter and hasn't concerned itself despite mounting evidence that either million dollar machines used to collect the vote don't work, or that there may be election tampering either from the vendor's end or at the SOE office.

The commissioners refrained from taking part in or signing off on the elections as members of the canvassing board, as evidence by the document I presented to you, dated Nov. 14, 2010. This could indicate the commission didn't want to be associated with the election for reasons mentioned.

You have the ability to adopt a joint resolution on your next regular meeting, to send off to the Division of Elections asking them to investigate this strange election. This would certainly carry more weight than the petition from a member of the public who - it can be argued - continues to do the job of vigilance and oversight for the commissioners themselves.

If you fail to even discuss this issue at your next meeting you are making glaringly evident your willful blindness to a very serious concern which is well documented now.