The reelection of Barack Obama was won by people, not by software. But in a contest as close as last week's election, software may have given the Obama for America organization's people a tiny edge—making them by some measures more efficient, better connected, and more engaged than the competition.

That edge was provided by the work of a group of people unique in the history of presidential politics: Team Tech, a dedicated internal team of technology professionals who operated like an Internet startup, leveraging a combination of open source software, Web services, and cloud computing power. The result was the sort of numbers any startup would consider a success. As Scott VanDenPlas, the head of the Obama technology team's DevOps group, put it in a tweet:

But the tech had not always worked so well. Four years ago, the Obama camp was in much the same position as Romney found himself in 2012, coming out of the primaries with only a short amount of time in which to pull together a national campaign. Despite having some people with tech expertise, the 2008 Obama campaign lacked an internal IT team, relying on vendors and field volunteers to pull much of the weight.

"One of the biggest problems in the last campaign was that you had all these people who are out in the field who are volunteering who start building their own versions of these rogue tools to do the same thing over and over again," said Clint Ecker, senior engineer for Obama for America (and an Ars Technica alum). Every field office assembled its own patchwork of tools using spreadsheets or a hacked Web application to track operations. They communicated over Google groups or simple e-mail lists. "It made it hard to keep everyone on the same page," he added.

Then there was the much-vaunted secret weapon, Project Houdini—a get-out-the-vote system that was supposed to revolutionize the Election Day ground game. Each voter in each swing-state voting precinct was assigned a numeric code; when poll watchers recorded the voters arriving, the watchers were supposed to dial in the code to Houdini's automated hotline. But the load on the hotline brought it down, and the campaign had to fail over either to texting or to calling the codes back into local field offices, where the data was re-entered into a webpage manually.

Houdini's database stayed up, and it still played a role in the Obama campaign's efforts on Election Day 2008. But it was clear that the system hadn't been ready for the wave of data it was supposed to handle. "2008 was the 'Jaws' moment," said Obama for America's Chief Technology Officer Harper Reed. "It was, 'Oh my God, we're going to need a bigger boat."

The Narwhal

To pull it off, the Obama team relied almost exclusively on Amazon's cloud computing services for computing and storage power. At its peak, the IT infrastructure for the Obama campaign took up "a significant amount of resources in AWS's Northern Virginia data center," said Ecker. "We actually had to start using beefier servers, because for a period of time we were buying up most of the available smaller Elastic Compute Cloud instance types in the East data center."

"Reed wanted to wire the campaign with its own application programming interface."

Atop Amazon's services, the Obama team built Narwhal—a set of services that acted as an interface to a single shared data store for all of the campaign's applications, making it possible to quickly develop new applications and to integrate existing ones into the campaign's system. Those apps include sophisticated analytics programs like Dreamcatcher, a tool developed to "microtarget" voters based on sentiments within text. And there's Dashboard, the "virtual field office" application that helped volunteers communicate and collaborate.

"Being able to decouple all the apps from each other [by using Narwhal] has such power," Harper Reed, the chief technology officer for the Obama campaign, told Ars. "It allowed us to scale each app individually and to share a lot of data between the apps, and it really saved us a lot of time." The resulting platform gave Obama for America tools that helped "force-multiply" volunteers, giving them organizational and communication tools that made the Obama "ground game" even more effective.

When Reed was brought onboard by Obama campaign Chief Integration and Innovation Officer Michael Slaby in June 2011, the choices Reed made were informed by the 2008 campaign—but also by the mentality of Internet startups.

"We knew we were going to go big," Reed said. "We needed to architect [the campaign's IT infrastructure] in such a way that it actually worked. And we also knew we were going to be resource-constrained—whether it was money or people, we knew we weren't going to have everything we wanted."

They did have the incumbent's advantage, though: time to prepare. "I don't think we would have been able to [build Narwhal] if we had to deal with the primaries," Reed said.

The architecture Reed envisioned had to scale up rapidly. It needed flexibility, allowing developers with any level of experience who joined the campaign to work in and be productive with whatever language they preferred, and it had to integrate with the systems of vendors (such as Blue State Digital and NGP VAN). It needed to handle the needs of volunteers in the field and in the campaign's "Team Digital" (responsible for the campaign's Web presence, social, and other digital media presence), while also feeding the "big data" machine of Team Data (the campaign's analytics department).

In other words, Reed wanted to wire the campaign with its own application programming interface (API). And, like many Internet startups, he looked to the cloud to make it scale.

"I looked at a lot of architectures," Reed said. "Coming out of [previous employer] Threadless, and having watched what Amazon and other large organizations had done—and how powerful these API platforms have become—it was obvious that the best path forward was to follow services oriented architecture (SOA) principles and build a services architecture that allowed for all of our apps to connect together and share one common data store."

But to build Reed's "bigger boat" meant adopting a startup-like strategy in more ways than just architecture. He needed to build a team within the Obama campaign that behaved like an Internet startup, a dedicated staff of engineers willing to work long hours for a rapid ramp-up to the ultimate payout: reelecting the president.

The full team in front of Chicago's "Bean."

The team

Reed went out and recruited people who already knew the territory, snapping up both local talent (such as Ecker) and people from out of town with Internet bona fides—veterans from companies like Google, Facebook, Twitter, and TripIt.

"All these guys have had experience working in startups and experience in scaling apps from nothing to huge in really tight situations like we were in the campaign," Ecker said.

"When you put down the constraints you have, it's pretty easy to figure out who you're hiring," Reed added. "You're looking for engineers who understand APIs—engineers that spend a lot of time on the Internet building platforms. When you talk about SOA, they're not just going to laugh at you and say you're talking about Java."

Rather than focusing on creating something significantly new, Reed said, the team focused on taking what they already knew worked and fitting the pieces together.

"We aggressively stood on the shoulders of giants like Amazon, and used technology that was built by other people," he said. "We had a pretty good culture of using not-invented-here technologies. And we weren't scared about it."

In some cases, he added, it was just the opposite—if something came up as a solution and had never been used at that scale by someone else, it was "super scary" and usually avoided. The Democrats were going to play it conservative.

The cloud

The first recruits started rolling in soon after Reed joined, giving the team 583 days to do their work.

"It really became apparent that there needed to be a centralized place for all this data to flow into from all our vendors and for the new applications," Ecker said. "We needed a standard RESTful API that any project could talk to using any programming language and any Web framework, whatever people are most comfortable in so that they can hit the ground running."

"We aggressively stood on the shoulders of giants and used technology that was built by other people."

That meant Narwhal. Written in Python, the API side of Narwhal exposes data elements through standard HTTP requests. While it was designed to work on top of any data store, the Obama tech team relied on Amazon's MySQL-based Relational Database Service (RDS). The "snapshot" capability of RDS allowed images of databases to be dumped into Simple Storage Service (S3) instances without having to run backups.

Even with the rapidly growing sets of shared data, the Obama tech team was able to stick with RDS for the entire campaign—though it required some finesse.

"We definitely bumped up against some limitations with RDS but they were largely self-inflicted," Ecker said. "We were able to work around those and stretch how far we were able to take RDS. If the campaign had been longer, we would have definitely had to migrate to big EC2 boxes with MySQL on them instead." Not having to switch off RDS meant that the Obama campaign saved "a shitload of money" on hiring additional database administrators, Ecker added.

Ecker says the team also tested Amazon's DynamoDB "NoSQL" database when it was introduced. While it didn't replace the SQL-based RDS service as Narwhal's data store, it was pressed into service for some of the other parts of the campaign's infrastructure. In particular, it was used in conjunction with the campaign's social networking "get-out-the-vote" efforts.

The integration element of Narwhal was built largely using programs that run off Amazon's Simple Queue Service (SQS). It pulled in streams of data from NGP VAN's and Blue State Digital's applications, polling data providers, and many more, and handed them off to worker applications—which in turn stuffed the data into SQS queues for processing and conversion from the vendors' APIs. Another element of Narwhal that used SQS was its e-mail infrastructure for applications, using worker applications to process e-mails, storing them in S3 to pass them in bulk from one stage of handling to another.

Initially, Narwhal development was shared across all the engineers. As the team grew near the beginning of 2012, however, Narwhal development was broken into two groups—an API team that developed the interfaces required for the applications being developed in-house by the campaign, and an integration team that handled connecting the data streams from vendors' applications.

Share this story

Sean Gallagher
Sean is Ars Technica's IT and National Security Editor. A former Navy officer, systems administrator, and network systems integrator with 20 years of IT journalism experience, he lives and works in Baltimore, Maryland. Emailsean.gallagher@arstechnica.com//Twitter@thepacketrat

So the thing I don't understand is... who exactly owns the technology now?

This is possibly the most detailed privately-owned US consumer database in existence, something that would probably be worth a billion dollars to private enterprise. Does someone have the right to license out this technology for personal profit?

Assume for a minute that this technology is only used for political campaign purposes going forward. Who gets to decide how it is used? Obama? David Axelrod?

In 4 years, there will be an open Democratic presidential primary, and all candidates would love to be the beneficiaries of this technology. Is some private consultant entity going to enrich themselves by having the candidates bid for use of their services (and, by proxy, use of this tech)?

Very nice article. It's really great that you guys did an expose of sorts on both sides (after the one about Romney).

And before people get on their high-horses about how this article is *obviously* unfair and more positive towards Obama, just remember that this is Ars Technica and not a political site.

This is about the technology, and I think both articles in the past week have been fair. It's unfortunate the Romney campaign had such trouble, but there is no reason to sugar coat what happened. In addition, it's great to see where some of the technological, and more importantly people-driven, successes are.

Reed doesn't take credit for the success of the campaign. "I truly believe that the Obama campaign could have won without us," he said. "It would have been slower, and would not have been efficient as it was in making voter contact.

I would agree with this sentiment. I have a hard time believing a large number of people change their minds based on frequency of cold-calls or other contact. Living in CO we got bombarded with advertisements and knocks on the door. I never found much substance in either, mostly just re-iterating what the constant tv ads were saying. By the time the election rolled around I kind of hated everyone that had pestered us.

Seems like slightly more efficient apps couldn't account for that much.

Very nice article. It's really great that you guys did an expose of sorts on both sides (after the one about Romney).

And before people get on their high-horses about how this article is *obviously* unfair and more positive towards Obama, just remember that this is Ars Technica and not a political site.

This is about the technology, and I think both articles in the past week have been fair. It's unfortunate the Romney campaign had such trouble, but there is no reason to sugar coat what happened. In addition, it's great to see where some of the technological, and more importantly people-driven, successes are.

We should note that we're working on getting more information from the Romney campaign, too, but it's taking longer.

Concerning the calls. I live in a swing state and we got at least 2 or 3 calls a night for months. I told my wife we should just tell them to go to hell or tell them we would vote for the other guy. She was nicer than I would have been. What I don't understand is that if we told them we were going to vote why didn't they leave us the hell alone?

So the thing I don't understand is... who exactly owns the technology now?

Presumably (or hopefully), it'll go to the DNC, where it could be used to help with 2014; a lot of the information put into it was probably a collaboration of the campaign and DNC sources. As for 2016, it could definitely be ramped up for the general election, but it might be kept from the primaries to force the candidates to build up their own networks. In theory. In practice, though, the big names will already have their own list of names and the Obama team will certainly split up amongst the field, bringing along their own lists and parts.

Interesting. I wonder if the DNC is reading this, trying to get their hands on a copy, or the phone numbers of the engineers involved. If they're not, they should be. Better to build on it in 2014 and 2016 than re-invent the wheel.

Also, kudos to editorial discipline for mentioning Reddit only where relevant in an article littered with "Narwhal". I probably would have buckled.

The secret of Mr. Obama is that he isn't really very good at politics, and he isn't good at politics because he doesn't really get people. The other day a Republican political veteran forwarded me a hiring notice from the Obama 2012 campaign. It read like politics as done by Martians. The "Analytics Department" is looking for "predictive Modeling/Data Mining" specialists to join the campaign's "multi-disciplinary team of statisticians," which will use "predictive modeling" to anticipate the behavior of the electorate. "We will analyze millions of interactions a day, learning from terabytes of historical data, running thousands of experiments, to inform campaign strategy and critical decisions."

--Peggy Noonan

Update: since it wasn't clear in my posting, this is an old column, July 30, 2011, that is discussing hiring notices likely linked to the inception of Narwhal itself:

Quote:

When Reed was brought onboard by Obama campaign Chief Integration and Innovation Officer Michael Slaby in June 2011, the choices Reed made were informed by the 2008 campaign—but also by the mentality of Internet startups.

Amazing to see how much effort and tech went into a politician's campaign and how much it contrasts with UK elections. I can't imagine any volunteers here getting so psyched for a political campaign, never mind developing a software system for managing one.

Then again, most canvassing here consists of clichéd political soundbites on TV and leaflets dropped through the front door that are usually thrown away unread.

Interesting. I wonder if the DNC is reading this, trying to get their hands on a copy, or the phone numbers of the engineers involved. If they're not, they should be. Better to build on it in 2014 and 2016 than re-invent the wheel.

Yeah, I've been thinking about this point too - what happens to all that data, all that code. If the dems don't want to get walloped in 2014 they need this data to GOTV. Hopefully OFA can give/sell it to the DNC and let it be a party-run tool for mid-term elections and then hand it over to whomever wins the nomination in 2016 and have them run with it. I wonder if the DNC would know what to do with it...

This is possibly the most detailed privately-owned US consumer database in existence

How do you think it compares to Google/Amazon/Lexis Nexus? How is the data they collected more detailed?

I might be missing something from the article, but I don't see any reason to believe their database is of extraordinary value. The apps helped streamline cold-calling/canvassing and made it easy for new volunteers to get into the mix, which is what the article suggests could have made a difference. Obviously you need the database, just saying it doesn't sound remotely close to being 'the most detailed privately-owned US consumer database in existence' as you suggest.

I'm just not sure how that happens. This is a system that was developed by private consultants on the Obama team, not the DNC. Are they going to turn over a billion dollar asset to the DNC, free of charge? It's almost like suggesting that Mark Zuckerberg turn over Facebook to an NGO for the "greater public good".

And to be honest, there's no indication that the DNC could even administrate this system effectively. That organization is a revolving door for career bureaucrats and politicians, and looks nothing like the nimble, young group described in the article above.

How do you think it compares to Google/Amazon/Lexis Nexus? How is the data they collected more detailed?

You make an excellent point, and I should correct myself. It is not more detailed, but it is a different level of detail.

I tend to keep a relatively low profile online, but a few months ago a cute college girl came by my house to canvas for a proposition. I was vaguely interested, and got into a casual conversation about my job, politics, and personal info that I wouldn't normally publish about myself to the web. It wasn't until the end of the chat that I realized she was jotting down this minutiae, and it was going to end up in Narwhal by the end of the day.

So I retract my statement that this is the most detailed private database, but I stand by my assertion that there is an enormous financial valuation to it.

Gather information in increasing amounts (they're already eroding the intent of the 4th anyway). Run it through models, figure out exactly how far you can push something without major backlash from the population.

In other words, don't get a big head, Obama political machine folks, you barely won.

A black dude with a foreign name in a shitty economy running on a pro-gay, pro-abortion, pro-taxes platform, beat a handsome "American Success Story" white guy who pushed all the correct conservative social and populist economic buttons (cut taxes, etc.) In what parallel universe is that not a landslide? Or at least a spanking?

I'll grant you: Landslide perhaps not in the statistical sense, but certainly an incredible achievement, thanks largely to an unbelievably effective ground game, of which the IT described in this article was part. I think that's all anyone is trying to present here, along with the article. I agree with the article's assessment toward the end that they also got incredibly lucky by avoiding any extended downtime or killer bugs, otherwise this could've easily read just like the Orca article.

*I* certainly had zero hope for re-election about six months ago, and I'm a "true believer" registered dem (although I do cross party lines occasionally).

Concerning the calls. I live in a swing state and we got at least 2 or 3 calls a night for months. I told my wife we should just tell them to go to hell or tell them we would vote for the other guy. She was nicer than I would have been. What I don't understand is that if we told them we were going to vote why didn't they leave us the hell alone?

I live in Iowa and had the same problem. For all the tech they employed how hard would it have been to flag my number saying that I'm voting stop bothering me. I have to assume that there was a number of voters that either didn't vote or changed simply out of spite for being called so many times.

Pretty desperate stuff from Noonan, calling the President a "loser" at this point.

"It is that nobody loves Obama. This is amazing because every president has people who love him, who feel deep personal affection or connection, who have a stubborn, even beautiful refusal to let what they know are just criticisms affect their feelings of regard. At the height of Bill Clinton's troubles there were always people who'd say, "Look, I love the guy." They'd often be smiling—a wry smile, a shrugging smile. Nobody smiles when they talk about Mr. Obama."

This is ridiculous. People cry when the guy gives speeches. He has incredible affection from the base, or he wouldn't be able to run the ground game that he does.

Noonan actually seems to think that Obama wrote that job notice. All that that shows it that Obama knows how to find the right people to make his hires for him. He was not personally sitting there designing Narwhal. Top engineers from Google and other companies came on board out of a sense of commitment to getting him over the line.

Seems like Noonan did an accidental find-and-replace and put "Obama" in instead of "Romney". He is the guy who couldn't get any love from the base. Obama comes out of the mill of Chicago politics, where if you don't "get" people you could never get things done.

Concerning the calls. I live in a swing state and we got at least 2 or 3 calls a night for months. I told my wife we should just tell them to go to hell or tell them we would vote for the other guy. She was nicer than I would have been. What I don't understand is that if we told them we were going to vote why didn't they leave us the hell alone?

I live in Iowa and had the same problem. For all the tech they employed how hard would it have been to flag my number saying that I'm voting stop bothering me. I have to assume that there was a number of voters that either didn't vote or changed simply out of spite for being called so many times.

While I can see the pestering annoyance factor working against any campaign, I'd assume that they weigh the risk of spite against the benefit of continued contact. Sure you may tell them you're going to vote 3 months out, but will that remain true? So they call and ask again 2 months out, or whenever their metrics say the average voter should be contacted again.

That said, I'm glad I don't live in a swing state where I would have been targeted by the calls. All I got were the every-three-weeks calls for more donations.

And the triumphant tone of this article compared to the one about the failures of Orca MIGHT lead one to think that the election was a landslide. It wasn't, it was one of the top five (I think) closest presidential elections in history.In other words, don't get a big head, Obama political machine folks, you barely won.

The popular vote may have been close and even in the top five, but considering how that doesn't matter at all, your incorrect. While it wasn't as sizable margin as the last election it was still a significant victory for Obama.

How do you think it compares to Google/Amazon/Lexis Nexus? How is the data they collected more detailed?

You make an excellent point, and I should correct myself. It is not more detailed, but it is a different level of detail.

I tend to keep a relatively low profile online, but a few months ago a cute college girl came by my house to canvas for a proposition. I was vaguely interested, and got into a casual conversation about my job, politics, and personal info that I wouldn't normally publish about myself to the web. It wasn't until the end of the chat that I realized she was jotting down this minutiae, and it was going to end up in Narwhal by the end of the day.

So I retract my statement that this is the most detailed private database, but I stand by my assertion that there is an enormous financial valuation to it.

Ah, that context helps a lot. It would be interesting to see what ends up in the database. Of all the people that came to my house, they didn't get any info of mine that wasn't already easy to find, and I'm fairly curt with people coming to my door, so I don't imagine I've contributed to the usefulness of the data. Your experience is probably somewhat atypical, but I'm sure there's a wide spectrum of responses to solicitors between yours and mine, so probably some useful stuff in there.

I did some door to door volunteer stuff for community events, even when you're just telling people about a free event in their community, most people don't like to be bothered. That experience made me skeptical about the effectiveness of any door to door initiative, but really, even if good data only came from 1% of the interactions you could cobble together some useful stats. Kind of an interesting bias, 'those willing to engage strangers at their door in conversation'. The cute girl part probably shifts the bias in many situations.