In the wake of Mitt Romney's Election Day defeat, frustration with the Romney campaign's get-out-the-vote effort led to reports of an IT meltdown. Project Orca, the system the Romney campaign used to connect to an army of volunteer poll-watchers tracking voter turnout, quickly became a target for conspiracy theorists, even leading to accusations of sabotage by contract developers. How could the Romney campaign spend so much on technology consultants and yet see its vaunted app fail when it was needed the most?

The truth, according to sources familiar with the matter, is that the applications powering Orca were developed by an internal "skunkworks," one made up of both campaign staffers and volunteers—not by a big-name consulting firm.

Into the skunkworks

According to Federal Election Commission filings that we examined, the Romney campaign spent over $40 million on technology vendors. Targeted Victory, the firm founded by Romney's Director of Digital Zac Moffatt, was the biggest single beneficiary of that spending, receiving more than $14 million as of the end of October. That money was both payment for its own social media consulting and development work, and for "pass-through" IT spending that went to other contractors such as Rockfish Interactive (the developers of Romney's running mate announcement mobile app).

The investment in Orca, however, was a tiny fraction of the campaign's IT spending. Sources tell us the Romney campaign tried systems similar to Orca during the Republican primaries, but on a much smaller scale. To prepare for the general election, the campaign pulled together a makeshift team of IT people and volunteers, and it was this team that built the full version of the Orca Web app and developed the backend systems to power it.

A number of small consulting companies assisted with implementing Orca after its development—specifically, they helped to support more than 800 volunteers who used the system at Romney's headquarters in Boston Garden on Election Day.

Here's how the system was supposed to work—though it probably wasn't a great idea for Romney's communications director, Gail Gitcho, to tell PBS how amazing Orca was going to be before it had been used. "The Obama campaign likes to brag about their ground operation," Gitcho said, "but it's nothing compared to this."

Romney campaign Communications Director Gail Gitcho brags about the power of Orca on PBS NewsHour.

The interface for Orca was a mobile Web application connected to a set of mirrored databases of voter rolls. When a user logged in, the app would load a page with a scrollable list of all the registered voters in the precinct they were assigned to. By swiping a checkbox next to a voter's name, they could record them as having voted; the entry was then transmitted back to Orca's application server. A back-up call response system—similar in nature to the Houdini system used by the Obama campaign in 2008 (a system which also ran into trouble)—provided a way for volunteers to report possible voting irregularities into the system even if their data connectivity failed. Using information gathered through the campaign's digital outreach and pulled from the campaign's voter contact vendor, FLS Connect, the system was supposed to give volunteers in Boston a complete view of Romney supporters in swing states who hadn't yet voted. It would then prompt phone calls asking supporters to vote.

Where were Orca's backend servers located? That remains unclear, though they don't appear to have run in the cloud. Most of the Romney organization's core IT operations, such as desktop support and back-office systems, were outsourced to Best Buy subsidiary MindShift Technologies (a Waltham, Massachusetts managed service provider). But a MindShift spokesperson declined to comment on whether the company's data center was used to support Project Orca, citing a confidentiality agreement with the Romney campaign.

The system had all sorts of problems when it was finally rolled out on the morning of Election Day, but it appears the real issues weren't with the Orca code itself or with its servers. Instead, the problems were in execution. Bandwidth constraints, bad passwords and PINS, and a lag in the operation caused by the surge of initial downloads of data to smartphone browsers all caused widely reported frustrations among volunteers. (Having Comcast suspect the spike in traffic on the lines into the Romney campaign was a denial of service attack didn't help, either.)

Ars was unable to reach Romney campaign Technology Director Kevin Rewkowski or Director of Voter Contact Dan Centinello (Centinello was the political manager of the Orca project) for comment. But there's plenty of reason for them to want to keep a low profile—a website using Centinello's name now accuses him of losing the campaign for Romney with Orca.

106 Reader Comments

It sounds like mostly a scope and scale issue, as opposed to code. IT consulting would have probably identified some of the concerns, however having time to implement solutions would have been challenging.

Who would ha thunk it? Probably only 250K lines of code. No need to call in the Pros yet, or beta test it. Besides which, if you get some non-true believers looking at your code, it negates that (very entrepreneurially efficient) magic fairy dust, and if things do not work after that, you never know which heretic screwed the pooch for you.

These all sound like technical problems under their control:Bandwidth constraints[L]ag in the operation caused by the surge of initial downloads of dataComcast suspect...denial of service attack

Bandwidth constraints were certainly under their control, and any lag in operation caused by an initial data surge is exactly the kind of thing you size ahead of time to ensure you... don't run into bandwidth constraints. Any surge of traffic in a critical period should be discussed with the upstream service provider for exactly this reason - and quite frankly, I don't consider Comcast a tier 1 provider.

So last time someone commented (with a load of upvotes) that "Accenture's Indian Developers" did it and consensus was that it was obviously a code issue. Then it was the contractors sabotaging it all. And now that there is speculation that "internal" volunteers did it - it is scope/scale issue as opposed to code?

Talk about a fractured project! This actually sounds worse than the initial article painted it to be. Money thrown all over the place, including what sounds like a bizarre combination of sweetheart deals and possibly lowest bidder outsourcing, and then trying to do key work in house without creating a seriously funded and properly organized dev shop?

It would be easy (and probably fairly correct in both cases) to claim that neither of the candidates probably had a deep personal understanding of what's necessary to implement a project of this scope. Which is fine and realistic: it's not their jobs. What is telling is how they went about choosing people who would (or wouldn't) have such an understanding and expertise, and how they chose to allocate funds in turn.

While this is hardly all of the campaign by itself, even as one element I still feel it paints a certain stroke in the overall portrait of Romney as a businessman. A certain less than pretty stroke.

Issues with this project aside, I just can't comprehend that a political party believes that, in order to win, they need to actively monitor their 'own' supporters, and push them into doing something.

How is this democracy, etc.

Democracy includes convincing people to do/say/vote in ways you want. Evangelism and project management are in no way contrary to any concept of democracy that I know of, and were around in ancient Greece and the early U.S..

Issues with this project aside, I just can't comprehend that a political party believes that, in order to win, they need to actively monitor their 'own' supporters, and push them into doing something.

How is this democracy, etc.

For the most part, they don't. Most supporters of a given candidate/party make it to the polls under their own power without needing nagging from the campaigns. The key word there however is 'most'. If your nagging campaign gets some people to the polls who otherwise wouldn't have bothered, maybe you can get an extra percentage point or two in the total. Won't turn around a blow-out result, but it can definitely make a difference in a tight race.

It sounds like mostly a scope and scale issue, as opposed to code. IT consulting would have probably identified some of the concerns, however having time to implement solutions would have been challenging.

Love the photo!

I agree that the main issue was scope and scale as well. They could have saved themselves a large amount of headache though. What that needed to do is take something close to what they wanted off the commercial shelf and then modify it to suit their needs. There are more than a few platforms I have in mind personally that are most of the way there.

I don't exactly know what they did here, but it is clear they did not consult any professionals. They sure as hell didn't do any kind of stress testing either. An expert with an existing platform would have been have to size it for them with next to no guess work.

someone commented (with a load of upvotes) that "Accenture's Indian Developers" did it

I heard that Accenture was doing this from someone working IT in the Romney campaign. I haven't talked to the guy since, but I would not be surprised if much of the campaign didn't know about the "roll it yourselves" strategy.

someone commented (with a load of upvotes) that "Accenture's Indian Developers" did it

I heard that Accenture was doing this from someone working IT in the Romney campaign. I haven't talked to the guy since, but I would not be surprised if much of the campaign didn't know about the "roll it yourselves" strategy.

Accenture denied involvement with anything Romney. *That* I have on the record.

Some of the most capable companies in the world - with more than a few inches of skin in the game - have fallen foul of capacity constraints or miscalculations on go-live day. World of Warcraft vanilla was a yoyo for weeks after it launched, and day one of TBC was a nightmare - and all this despite the very significant beta testing that always goes into a Blizzard release. Even Diablo 3's launch - after everything Blizzard has learned of problems with release day capacity issues - fizzled.

If even they can make these mistakes on launch day and take significant amounts of time to fix them, why is it so surprising that this relatively inexperienced IT group somewhat fell on its face on election day? Like so many before them they weren't ready to handle the demand, and had problems as a result. Is a saboteur really the most likely explanation? It's been made clear that Obama's campaign had the same problems in 2008, and - through experience and know how - lifted their game and brought it home in spades this year. Sometimes the better team just wins.

These all sound like technical problems under their control:Bandwidth constraints[L]ag in the operation caused by the surge of initial downloads of dataComcast suspect...denial of service attack

Bandwidth constraints were certainly under their control, and any lag in operation caused by an initial data surge is exactly the kind of thing you size ahead of time to ensure you... don't run into bandwidth constraints. Any surge of traffic in a critical period should be discussed with the upstream service provider for exactly this reason - and quite frankly, I don't consider Comcast a tier 1 provider.

Being European the US election system is looking quite arcane to me anyway, but I do not get the the intended benefit of Orca at all. As I read it, it was supposed to be employed on election day to do what? Keep track on voters? Does this mean if I would be a person that is registered in the Republican voters database as an Republican voter and have not shown up until say, 3 pm on election day, I would have gotten a call from the office to get my ass moving and vote?This really doesn't sound like a way that would work in my country, but then maybe I am getting it all wrong. So my honest question: What was Orca supposed to achieve?

Sorry for veering off topic slightly, but IANA American, so I'm slightly confused by this get-out-the-vote business.There are people from each side who go out and hassle other people to vote? So presumably, they *already know* who the voter is going to vote for?Isn't something wrong with this system?

Sorry for veering off topic slightly, but IANA American, so I'm slightly confused by this get-out-the-vote business.There are people from each side who go out and hassle other people to vote? So presumably, they *already know* who the voter is going to vote for?Isn't something wrong with this system?

Well, they call it "encourage," but otherwise you have it right. They assume who the voter is going to vote for based on demographics, sort of like how people look at me (a white male living in the south) and assume I'm a party Republican.

But, among problems with the US voting system, this is pretty far down the list. At least it's a somewhat more reasonable approach than a friend of mine who went to vote because, if he didn't, he'd have to pay a fine.

Being European the US election system is looking quite arcane to me anyway, but I do not get the the intended benefit of Orca at all. As I read it, it was supposed to be employed on election day to do what? Keep track on voters? Does this mean if I would be a person that is registered in the Republican voters database as an Republican voter and have not shown up until say, 3 pm on election day, I would have gotten a call from the office to get my ass moving and vote?This really doesn't sound like a way that would work in my country, but then maybe I am getting it all wrong. So my honest question: What was Orca supposed to achieve?

Effectively yes, with a couple of caveats. Firstly, you wouldn't explicitly register in the Republican database; you'd be added as a probable Republican voter based on a variety of factors (demographics, whether you had voted in party primaries, things like that). Secondly, because of the way the US Presidential election system is structured, only voters living in a certain set of so-called "swing" states generally end up in databases of this sort. Say maybe about 1/4th of the total US population, roughly speaking. Local candidates might still call you up and harass you into voting, but they generally don't begin to have the budget for anything sophisticated.

Being European the US election system is looking quite arcane to me anyway, but I do not get the the intended benefit of Orca at all. As I read it, it was supposed to be employed on election day to do what? Keep track on voters? Does this mean if I would be a person that is registered in the Republican voters database as an Republican voter and have not shown up until say, 3 pm on election day, I would have gotten a call from the office to get my ass moving and vote?This really doesn't sound like a way that would work in my country, but then maybe I am getting it all wrong. So my honest question: What was Orca supposed to achieve?

Effectively yes, with a couple of caveats. Firstly, you wouldn't explicitly register in the Republican database; you'd be added as a probable Republican voter based on a variety of factors (demographics, whether you had voted in party primaries, things like that). Secondly, because of the way the US Presidential election system is structured, only voters living in a certain set of so-called "swing" states generally end up in databases of this sort. Say maybe about 1/4th of the total US population, roughly speaking. Local candidates might still call you up and harass you into voting, but they generally don't begin to have the budget for anything sophisticated.

Thanks for that clarification. That basically means that Orca is a database with potential voters (based on the factors you mentioned above) where volunteers had an interface to tally the persons that actually voted and other volunteers would take action based on that to persuade the rest to go out and vote.Still wondering how they would tally the voters as in my home the question: "Hello Sir, what is your name and who did you vote for?" would get the reply: "None of your business". Sorry that I keep distracting from the actual topic but I figure this is a good chance for me to get more understanding of the election system in the US.

Being European the US election system is looking quite arcane to me anyway, but I do not get the the intended benefit of Orca at all. As I read it, it was supposed to be employed on election day to do what? Keep track on voters? Does this mean if I would be a person that is registered in the Republican voters database as an Republican voter and have not shown up until say, 3 pm on election day, I would have gotten a call from the office to get my ass moving and vote?This really doesn't sound like a way that would work in my country, but then maybe I am getting it all wrong. So my honest question: What was Orca supposed to achieve?

As a Canadian, we follow the British system fairly closely and don't have voters registered with a party. However, we do have lists of voters and have used a manual system for years.

Each party calls everyone on that list to drum up support and ask if the voter has decided how to vote. If the voter plans to vote for your candidate they get marked as a likely voter. Even without computers, volunteers call their supporters on election day, remind them to vote, and offer a ride to the polls. Paper voting lists would be cross referenced with paper supporter lists to focus the effort.

The Orca/Narwhal projects aren't revolutionary, they're just more efficient ways (in theory) of doing what political parties have done for decades.

Sean Gallagher / Sean is Ars Technica's IT Editor. A former Navy officer, systems administrator, and network systems integrator with 20 years of IT journalism experience, he lives and works in Baltimore, Maryland.