Mitt Romney's Campaign Meltdown. It Was Technology's Fault!

9:31 pm - 11/10/2012

The Sin of Not Beta Testing. You Fail Romney!

The Romney Campaign’s Ground Game FiascoThey were the gang who couldn’t shoot straight. Romney’s ground-game operation was a disaster—from technology that didn’t work to field operatives who didn’t understand their tasks. The result: Obama won.

When Republican fundraisers solicited the party’s big donors on behalf of Mitt Romney this year, the centerpiece of the pitch was a state-of-the-art campaign to identify the party’s likely voters and make sure they came to the polls on Election Day. Political pros call these county-by-county, block-by-block campaigns the ground game. And while most of the media attention focuses on candidate speeches, debates, and ad buys, it’s the ground game where elections are won and lost.

On Tuesday, Republicans lost the election on the ground. As the Republican Party picks up the pieces from overwhelming defeat, the first fingers are being pointed at a GOP ground game that insiders describe to The Daily Beast as nothing short of a fiasco.

The story starts in 2008. The Romney campaign sought to counter Team Obama’s highly touted, high-tech voter-targeting system, nicknamed Narwhal after the Arctic sea mammal. Narwhal provided the Obama campaign with reams of specific data on voters—finding single women in conservative counties, for instance, or families with children who have disabilities.

In 2012, the Romney campaign unveiled its own killer app and called it Project Orca—the fierce great whale that is the natural predator of the Narwhal. The only problem: Boston’s Orca turned out to be toothless.

The system was different from Narwhal. It was designed to allow Romney poll watchers, in real time, to identify likely Romney supporters who still had not shown up at polling stations on Election Day. By uploading the names of people who had voted, the computers back in Boston could figure out who still needed to be targeted and turned out.

At least that’s the way it was supposed to work. But on Tuesday, it became clear that the deployment of Orca was doing more harm than good. “I think it’s fair to say that pretty much everything about the system that was supposed to work actually failed,” said one campaign official who witnessed the breakdown from the Romney war room on the floor of Boston’s TD Garden.

The Romney high command had cloaked the system in secrecy to maintain what it hoped would be a true competitive turnout advantage. But by limiting the number of people with access to Orca, the campaign was not able to train its field operatives to use it or do the necessary beta-testing to work out the kinks that typically plague new software.

“It did not work perfectly,” said Rich Beeson, the Romney campaign’s political director, in an interview. He acknowledged that Orca crashed in the morning on Election Day. At first the campaign thought the system had been hacked, he said. Passwords and user names for the 34,000 volunteers using the program had to be reset.

But Beeson also said Orca was able to provide voting data on 91 percent of the precincts and accounted for turning out some 14.3 million voters. “At the end of the day I can look any donor in the eye and say we used our resources effectively,” he said. “This is the first time we have attempted to do anything on this scale. By no means was it an abject failure.”

Others who worked on the ground for the campaign disagree, however. In many instances, the voter lists that were loaded on the smart phones of field operatives didn’t match the precincts where they’d been sent, campaign officials said. In addition, there were massive credentialing problems, so Romney poll watchers were not permitted to operate at many precincts. In many rural precincts, poor cellphone coverage made it difficult or impossible for Romney forces to transmit information. Finally, because poll watchers tend to be older, tech-averse volunteers and because there was so little training, many of them simply couldn’t master the technical aspects of the task.

“We were sold on Mitt as this brilliant manager and turnaround artist,” said John Ekdahl, one of those poll watchers in Florida who used Orca. “But it was a snake-oil kind of program. I say this as a Web developer. This was throwing money at a product that just didn’t work.” Ekdahl first published his critique of Orca on the conservative website Ace of Spades. Other poll watchers who asked not to be named had similar complaints.

The mood grew increasingly grim on Tuesday as Romney officials realized that their supposed state-of-the-art answer to the Chicago’s turnout juggernaut was a bust. Walking down the central aisle of the Romney war room you didn’t hear the humming of a well-oiled turnout machine, one campaign official recalled. You heard the panicky tones of operatives flooded with calls from the field about technical snafus and mass confusion.

As campaign officials monitored central computers in Boston, instead of taking in the metrics of a proficient ground game, they saw depressing evidence of a gang that couldn’t shoot straight—anxious messages from operatives who were at the wrong polling place, couldn’t work their smart phones, or were barred from a precinct because they lacked the proper credentials. “It was amateur hour,” lamented one Romney official.

There were other problems for Romney’s ground game in the battleground states. The Obama for America team, for example, had field operations in states like Ohio stay behind after the 2008 election and slowly but surely pick up steam as Election Day 2012 approached. The Republicans closed their field offices after the 2010 midterms.“We were never going to have the same size and staff as the Obama campaign,” Beeson said. He added that the Republican National Committee was several million dollars in debt by the time Michael Steele left the job, putting resource constraints on the party in terms of operating field offices in states like Ohio.

Finally, the Republicans were never able to match the Obama campaign’s ability to use data from purchase histories, voting registration, and campaign contacts to tailor specific messages to specific voters in a process known as micro-targeting. Republicans first pioneered the use of this kind of data in the 2004 election cycle but mainly used the data to target television ads, direct mail, and robocalls, according to Republican strategists. The Democrats were able to use this kind of data in deploying armies of volunteer door-knockers and others who targeted their voters over time.

Beeson acknowledged that his party will be seeking to learn from the Democrats in terms of micro-targeting. “They have taken this to an organic, micro, micro level,” he said. “We will be looking at how they did it.”

At the end of the day, Beeson said he thought Romney ran a good campaign. It’s just that Obama ran a better one. “With the time and resources we had, the campaign was run very effectively,” he said. “We just did not see them being able to turn out the numbers they turned out.”

It was supposed to be a "killer app," but a system deployed to volunteers by Mitt Romney's presidential campaign may have done more harm to Romney's chances on Election Day—largely because of a failure to follow basic best practices for IT projects.

Called "Orca," the effort was supposed to give the Romney campaign its own analytics on what was happening at polling places and to help the campaign direct get-out-the-vote efforts in the key battleground states of Ohio, Florida, Pennsylvania, Iowa, and Colorado.

Instead, volunteers couldn't get the system to work from the field in many states—in some cases because they had been given the wrong login information. The system crashed repeatedly. At one point, the network connection to the Romney campaign's headquarters went down because Internet provider Comcast reportedly thought the traffic was caused by a denial of service attack.

As one Orca user described it to Ars, the entire episode was a "huge clusterfuck." Here's how it happened.

Develop in haste, repent at leisure

The Romney campaign put a lot of stock in Orca, giving PBS NewsHour an advance look at the operation on November 5. But according to volunteers who saw and used the system, it was hardly a model of stability, having been developed in just seven months on a lightning schedule following the Republican primary elections. Orca had been conceived by two men—Romney's Director of Voter Contact Dan Centinello and the campaign's Political Director Rich Beeson. It was named in honor of the killer whale as an allusion to the Obama campaign's own voter identification program, code-named Narwhal; orcas are the top predator of narwhals, Romney campaign staffers explained, and they were preparing to outshine the Democratic voter turnout effort.

As Romney's Communications Director Gail Gitcho put it in the PBS piece, "The Obama campaign likes to brag about their ground operation, but it's nothing compared to this."

To build Orca, the Romney campaign turned to Microsoft and an unnamed application consulting firm. The goal was to put a mobile application in the hands of 37,000 volunteers in swing states, who would station themselves at the polls and track the arrival of known Romney supporters. The information would be monitored by more than 800 volunteers back at Romney's Boston Garden campaign headquarters via a Web-based management console, and it would be used to push out more calls throughout the day to pro-Romney voters who hadn't yet shown up at the polls. A backup voice response system would allow local poll volunteers to call in information from the field if they couldn't access the Web.

But Orca turned out to be toothless, thanks to a series of deployment blunders and network and system failures. While the system was stress-tested using automated testing tools, users received little or no advance training on the system. Crucially, there was no dry run to test how Orca would perform over the public Internet.

Part of the issue was Orca's architecture. While 11 backend database servers had been provisioned for the system—probably running on virtual machines—the "mobile" piece of Orca was a Web application supported by a single Web server and a single application server. Rather than a set of servers in the cloud, "I believe all the servers were in Boston at the Garden or a data center nearby," wrote Hans Dittuobo, a Romney volunteer at Boston Garden, to Ars by e-mail.

Throughout the day, the Orca Web page was repeatedly inaccessible. It remains unclear whether the issue was server load or a lack of available bandwidth, but the result was the same: Orca had not been tested under real-world conditions and repeatedly failed when it was needed the most.

All tell, no show

Before Election Day, volunteer training at Boston headquarters amounted to a series of 90-minute conference calls with Centinello. Users had no hands-on with the Orca application itself, which wasn't turned on until 6:00 AM on Election Day.

"We asked if our laptops needed to be WiFi capable," Dittuobo told Ars. "Dan Centinello went into how the Garden had just finished expansion of its wireless network and that yes, WiFi was required. I was concerned about hacking, jamming the signal, etc...Then we were told that we would not be using WiFi but using Ethernet connections."

Field volunteers also got briefed via conference calls, and they too had no hands-on with the application in advance of Election Day. There was a great deal of confusion among some volunteers in the days leading up to the election as they searched Android and Apple app stores for the Orca application, not knowing it was a Web app.

John Ekdahl, Jr., a Web developer and Romney volunteer, recounted on the Ace of Spades HQ blog that these preparatory calls were "more of the slick marketing speech type than helpful training sessions. I had some serious questions—things like 'Has this been stress tested?', 'Is there redundancy in place?', and 'What steps have been taken to combat a coordinated DDOS attack or the like?', among others. These types of questions were brushed aside (truth be told, they never took one of my questions). They assured us that the system had been relentlessly tested and would be a tremendous success."

In a final training call on November 3, field volunteers were told to expect "packets" shortly containing the information they needed to use Orca. Those packets, which showed up in some volunteers' e-mail inboxes as late as November 5, turned out to be PDF files—huge PDF files which contained instructions on how to use the app and voter rolls for the voting precincts each volunteer would be working. After discovering the PDFs in his e-mail inbox at 10:00 PM on Election Eve, Ekdahl said that "I sat down and cursed, as I would have to print 60+ pages of instructions and voter rolls on my home printer. They expected 75 to 80-year old veteran volunteers to print out 60+ pages on their home computers? The night before election day?"

Invalid passwords, crashing servers

When the Romney campaign finally brought up Orca, the "killer whale" was not ready to perform. Some field volunteers couldn't even report to their posts, because the campaign hadn't told them they first needed to pick up poll watcher credentials from one of Romney's local "victory centers." Others couldn't connect to the Orca site because they entered the URL for the site without the https:// prefix; instead of being redirected to the secure site, they were confronted with a blank page, Ekdahl said.

And for many of those who managed to get to their polling places and who called up the website on their phones, there was another, insurmountable hurdle—their passwords didn't work and attempts to reset passwords through the site also failed. As for the voice-powered backup system, it failed too as many poll watchers received the wrong personal identification numbers needed to access the system. Joel Pollak of Briebart reported that hundreds of volunteers in Colorado and North Carolina couldn't use either the Web-based or the voice-based Orca systems; it wasn't until 6:00 PM on Election Day that the team running Orca admitted they had issued the wrong PIN codes and passwords to everyone in those states, and they reset them. Even then, some volunteers still couldn’t login.

In Boston, things weren't much better. Some of the VoIP phones set up for volunteers were misconfigured. And as volunteers tried to help people in the field get into the system, they ran into similar problems themselves. "I tried to login to the field website," Dittuobo told me, "but none of the user names and passwords worked, though the person next to me could get in. We had zero access to Iowa, Colorado, North Carolina, and Pennsylvania. Seems like the only state that was working was Florida."

As the Web traffic from volunteers attempting to connect to Orca mounted, the system crashed repeatedly because of bandwidth constraints. At one point the network connection to the campaign's data center went down—apparently because the ISP shut it off. "They told us Comcast thought it was a denial of service attack and shut it down," Dittuobu recounted. "(Centinello) was giddy about it," he added—presumably because he thought that so much traffic was sign of heavy system use.

Flying blind

As the day wore on and information still failed to flow in from the field, the Romney campaign was flying blind. Instead of using Orca's vaunted analytics to steer their course, Centinello and the rest of Romney's team had no solid data on how to target late voters, other than what they heard from the media. Meanwhile, volunteers like Ekdahl could do nothing but vote themselves and go home.

This sort of failure is why there's a trend in application testing (particularly in the development of public-facing applications) away from focusing on testing application infrastructure performance and toward focusing on user experience. Automated testing rigs can tell if software components are up to the task of handling expected loads, but they can't show what the system's performance will look like to the end user. And whatever testing environment Romney's campaign team and IT consultants used, it wasn't one that mimicked the conditions of Election Day. As a result, Orca's launch on Election Day was essentially a beta test of the software—not something most IT organizations would do in such a high-stakes environment.

IT projects are easy scapegoats for organizational failures. There's no way to know if Romney could have made up the margins in Ohio if Orca had worked. But the catastrophic failure of the system, purchased at large expense, squandered the campaign's most valuable resource—people—and was symptomatic of a much bigger leadership problem.

"The end result," Ekdahl wrote, "was that 30,000+ of the most active and fired-up volunteers were wandering around confused and frustrated when they could have been doing anything else to help. The bitter irony of this entire endeavor was that a supposedly small government candidate gutted the local structure of [get out the vote] efforts in favor of a centralized, faceless organization in a far off place (in this case, their Boston headquarters). Wrap your head around that."

Republican campaigners will undoubtedly try to wrap their heads around it for some time to come.

As Republicans continue to deconstruct the failure of the Romney campaign, volunteers have revealed that there were serious problems with the candidate's whiz-bang tech solution for getting people to the polls. Prior to election Day, Mitt Romney's team was already bragging about Project Orca, a web app that would connect tens of thousands of volunteer poll watchers in the field with the home office in Boston. It was all part of a massive coordination effort to get Republicans out of the house and into the voting booth.

Poll watching is a time-honored tradition in politics, where volunteers from each campaign literally stand at poll stations with lists of registered supporters and check off the names of people as they come into vote. As the day goes on, they can see who hasn't voted yet and then other volunteers can give them a call or physically go find them and bring them to the polls. Normally, that's done with simple paper and pencils, but Orca was designed to digitize and streamline the process, while also centralizing the effort with headquarters, allowing campaign leaders to re-direct resources on the fly to the areas they were needed most.

On the day after the election, complaints started pouring in from the volunteers themselves indicating that not only did Project Orca not improve the process, it may have actually hindered it. John Ekdahl, a volunteer who writes at Ace Of Spades HQ, outlined the various glitches and breakdowns of the system. Instead of handing out voter lists at local offices, volunteers were emailed 60+ page PDF files and told to print them out at home the night before the election. They weren't given official poll watcher certificates or told that those were required to enter most polling places. The "app" wasn't really an app at all, it was a secure website, creating confusion for volunteers trying to find it in the iTunes store. It also didn't auto-forward users who didn't know to add an S to the http:// protocol in the app's URL (which most browsers don't ask you to type anymore), leaving numerous user lost on a broken webpage.

Volunteers in other parts of the country shared similar complaints. The emailed packets came late or not at all. PINs that were required to login and download the voter lists didn't work and couldn't be reset. Calls and emails to the help desk went unanswered, and the entire system may have just completely crashed in the middle of election day. Frustrated volunteers struggled to get answers that never came, leaving most of them to fend for themselves or simply give up, wasting an entire day without bringing a single new voter to the voting booth.

The whole thing is vaguely reminiscent of the droid army in Star Wars: Episode I (or The Avengers if Marvel is more your thing) in which a fearsome hyper-coordinated fighting force is controlled from afar by a centralized brain trust ... that gets blown up by a little kid in a toy plane. It's also left a lot of GOP volunteers wondering if the Romney campaign, in an effort to micromanage the all-important ground game, may cost itself the election. By pushing Orca into the most important swing states they drew resources away from more battle-tested methods and left local campaign offices flying blind. No wants to admit that the plan built to put the Romney campaign over the top may have actually led to its demise, but it's hard to ignore the damage done by the broken project. Briebart.com can do the math:

"If each of the 37,000 volunteers that had been devoted to Orca had instead brought 20 voters to the polls in those states over the course of the day, Romney would have won the election."