Despite running a campaign with about twice the money and twice the staff of Governor Mitt Romney's presidential bid, President Barack Obama's campaign under-spent Romney's on IT products and services by $14.5 million, putting the money instead into building an internal tech team. Based on an Ars analysis of Federal Election Commission filings, the Obama campaign, all-inclusive, spent $9.3 million on technology services and consulting and under $2 million on internal technology-related payroll.

The bottom line is that the Obama campaign's emphasis on people over capital and use of open-source tools to develop and operate its sophisticated cloud-based infrastructure ended up actually saving the campaign money. As Scott VanDenPlas, lead DevOps for Obama for America put it in an e-mail interview with Ars, "A lesson which we took to heart from 2008 [was that] operational efficiency is an enormous strategic advantage."

As we revealed in our recent analysis of the Romney team's tech strategy, the Romney campaign spent $23.6 million on outside technology services—most of it on outside "digital media" consulting and data management. It outsourced most of its basic IT operations, while the Obama campaign did the opposite—buying hardware and software licenses, and hiring its own IT department. Just how much emphasis the Obama campaign put on IT is demonstrated by the fact that the campaign's most highly paid staff member was its Chief Integration and Innovation Officer, Michael Slaby, with an annualized salary of about $130,000.

By comparison, Kevin Rekowski, the Romney campaign's Director of Technology, was barely in the top 20 salaries of the Romney campaign, with an annualized salary of $80,000. Zac Moffatt, Romney's Digital Director —a social media planner, not a technology expert—was number five, at $175,000 a year, in addition to whatever he earned from hiring his own firm, Targeted Victory, to handle much of the Romney campaign's digital strategy.

But the advantage of having a personal army of coders wasn't just financial. "Campaigns are serious tests of your creativity and foresight," VanDenPlas explained. "They are unpredictable, agile, and short—an 18 month, $1 billion, essentially disposable organization. Hackers can thrive in an environment like that, to a point where I'm not sure anyone else really can. Everything is over far too quickly to get boring."

Smart, not perfect

The strategy the Obama campaign's DevOps team used to manage the ever-growing number of applications deployed by the campaign was to "choose the lowest cost route to get us the most results—basically, be smart, not perfect," VanDenPlas said. "We did a lot of work to make things simple, and when you have a team that is unfazed by limitations, you get some really amazing and creative solutions, some of which I hope to see come out as open sourced projects here shortly."

Key in maximizing the value of the Obama campaign's IT spending was its use of open source tools and open architectures. Linux—particularly Ubuntu—was used as the server operating system of choice. "We were technology agnostic, and used the right technology for the right purpose," VanDenPlas said. "Someone counted nearly 10 distinct DBMS/NoSQL systems, and we wrote something like 200 apps in Python, Ruby, PHP, Java, and Node.js."

It also helped that the campaign, at least for internally developed applications, relied almost exclusively on Amazon Web Service for its infrastructure, eliminating a lot of the financial burden of infrastructure management. "For the applications built by the OFA [Obama for America] technology team, 99.999 percent were AWS hosted," VanDenPlas said, "purely because it was the best fit for what we were doing. As a whole, if you include privately hosted virtualized environments in the cloud architecture definition, I believe everything was 'cloud,' even down to our development environments running inside of Vagrant on our laptops."

The system configurations for the campaign's Elastic Compute Cloud (EC2) instances were created using the Puppet configuration management tool and were built as Debian packages kept in the campaign's own Advanced Packaging Tool (apt) repository—both for internally developed and third-party applications. As the number of applications and the scale of the campaign's AWS infrastructure use climbed, the DevOps team shifted to using Asgard—an open-source tool developed by Netflix to manage cloud deployments.

To help optimize applications, the OFA technology team used New Relic, a tool also used by the Romney campaign. "It is really a fantastic tool that increases your visibility into where your applications are spending time," VanDenPlas said. "They support the major languages we used (Python, Ruby, PHP) as well as the frameworks (Flask, Rails, Kohana)."

While AWS's tools were used for performance monitoring and to trigger automatic scaling-up of capacity, VanDenPlas said, much of the monitoring was handled by a suite of commercial and open source tools and home-grown code, "consisting of Cacti, Opsview, StatsD, Graphite, and Seyren, and a number of custom applications that continued to evolve right up until Election Day," VanDenPlas said.

To get better aggregated alerting and metric data, the team built a lightweight plugin for Nagios (the open-source basis of Opsview) in Python based on boto (the Python programming interface to AWS's services) and dotCloud's ZeroRPC messaging interface. "Using this," VanDenPlas explained, "we could constantly query thousands of nodes for near real-time statistics and feed them right back into the same alerting and monitoring system (Nagios) we used elsewhere."

Other performance monitoring and user experience data was collected using Chartbeat and Google Analytics. "Akamai also provided very useful statistics and logging," VanDenPlas said, "but these were mostly contextual rather than actionable." But, he added, the most heavily used monitoring system was "our community of internal and external supporters. The human factor in monitoring is huge. There are countless incidents where (OFA User Support Director) Brady Kriss notified us of pending problems derived from community help tickets."

The armor-plated cloud

The OFA engineering team also did a lot of work to ensure that they got the most out of Amazon's cloud architecture in terms of resiliency. As the election approached and the infrastructure demands surged, the engineering team took advantage of Amazon's multiple availability zones within its Virgina data center. "We built out a triply redundant, encrypted, and compressed WAN optimized tunnel between AWS regions," VanDenPlas said, "using a combination of OpenVPN, CloudOptimizer, and some DNS trickery."

The team shifted its domain name service to Amazon's Route 53 service, which uses latency-based routing to direct users to the host running in the AWS availability region with the shortest network trip time. That allowed the Obama team's application deployments to use "regionless" generic configuration settings, making deployments much simpler.

The centerpiece of the whole Obama campaign was its fundraising capabilities, without which all of the other applications may have been moot. The 2012 campaign's online donation system was a complete rebuild from the 2008 effort, VanDenPlas said, "a multi-region, geolocated, three facility processor capable of a per second transaction count sufficiently high enough that we failed to be able to reach it in load testing. It could also operate if every other dependent service had failed, including its own database and every vendor."

The Obama campaign's websites were also hosted on Amazon and hardened. The campaign's engineers built an application that created static HTML snapshots of the sites stored in Amazon's Simple Storage Service (S3); in the event of a Web server failure, requests would be instantly directed to the latest snapshot.

All of that redundancy was given an extra workout in the week before the election as Hurricane Sandy approached the East Coast. VanDenPlas said that a "complete hot replica of our entire infrastructure" was deployed to Amazon's primary West Coast data center in under 24 hours as a precaution.

Build, borrow, or buy

The tech team wasn't the only internal IT operation at Obama for America. The campaign ran its own data analysis shop and had its own army of Web designers and administrators. And with a payroll of over 1,000 people, the IT team had a lot of tech to support for an organization that had essentially a 24-month lifecycle.

That meant buying a lot of hardware and software. CDW, based outside of Chicago, was the go-to supplier for much of the campaign's computer equipment and boxed software purchases. Microsoft also sold $522,210 worth of software licenses to the campaign—which averages out to just under $500 per staffer.

Obama Campaign Hardware and Software purchases (expenses to companies of over $1,000 total)

With its investment in the cloud, the Obama campaign's Web hosting costs were much higher than Romney's. The largest cost, however, was content hosted by Blue State Digital, the social media and interactive advertising agency; the cost for hosting the internally developed applications in the Amazon cloud was a quarter of that:

And then there was the Obama campaign's outside technology help. As mentioned in our previous coverage of the Obama campaign, advertising company Blue State Digital and campaign software provider NGP VAN provided the largest chunks of Obama for America's technology consulting, and are most directly comparable to the over $14 million paid out by the Romney campaign to its digital firm, Targeted Victory. Even taken with the software and Web hosting expenses, the Obama campaign spent a seventh of what the Romney campaign spent on digital and an even smaller fraction of what Romney spent on voter and donor contact.

Technology consulting and services expenditures

Name

Consulting

Services

Total

BLUE STATE DIGITAL, LLC

$1,384,934.46

$1,384,934.46

NGP VAN

$743,455.80

$743,455.80

THE MIS DEPARTMENT, INC.

$730,388.00

$730,388.00

ANALYST INSTITUTE

$222,318.00

$222,318.00

THIRTEEN23 CORP

$125,000.00

$125,000.00

OBAMA VICTORY FUND 2012

$111,500.00

$111,500.00

COMSCORE INC

$90,000.00

$90,000.00

BLUE CITY STUDIOS

$44,512.50

$44,512.50

BRIGHTTAG, INC

$29,420.08

$29,420.08

AVF CONSULTING

$22,365.00

$22,365.00

TERAKEET CORPORATION

$19,754.00

$19,754.00

SYNETECH, LLC

$18,200.00

$18,200.00

AKPD MESSAGE & MEDIA

$17,500.00

$17,500.00

BRIGHT BLUE DATA LLC

$10,000.00

$10,000.00

UDELL, MADELEINE

$6,900.00

$6,900.00

BIG HEAD LABS, INC. (DISQUS)

$6,750.00

$6,750.00

DENTON, GRANT

$5,700.00

$5,700.00

CDW DIRECT

$1,069.63

$4,510.06

$5,579.69

ZENDESK

$3,669.00

$3,669.00

BIG HEAD LABS, INC. (DISQUS) ATTN:ACCOUNTS RECEIVABLES

$2,812.50

$562.50

$3,375.00

TRAMMELL, MARK

$3,000.00

$3,000.00

THE SYNETECH GROUP

$2,750.00

$2,750.00

TARGET SMART COMMUNICATIONS

$1,730.76

$1,730.76

CASTLETON, DAVID

$1,450.00

$1,450.00

ADDIS, DAVID

$1,378.10

$1,378.10

PACIFIC EAST

$853.56

$853.56

WAUGH, KIERAN

$600.00

$600.00

PETERSON, KELLIE

$500.00

$500.00

AMAZON

$401.47

$401.47

DEMOCRACY ENGINE LLC

$400.00

$400.00

GOOGLE WALLET

$360.00

$360.00

APPLE

$317.69

$317.69

PAYPAL

$53.24

$258.00

$311.24

HUMAN RIGHTS CAMPAIGN C/O MIKE MINGS

$245.00

$245.00

ENTRUST TECHNOLOGIES INC

$158.77

$158.77

ILLINOIS INSITITUTE

$150.00

$150.00

Grand Total

$3,553,105.23

$62,822.89

$3,615,928.12

Return on investment

In the end, the deciding factor wasn't what the Obama campaign spent money on, but what it did with all that money. Insourcing gave the campaign a strategic flexibility that the Romney campaign lacked, as well as other intangibles that may have contributed to leading an efficient campaign. And the reduced reliance on outside consultants allowed the Obama campaign to direct capital toward places where it had a bigger impact—such as in advertising, where the Obama campaign outspent Romney by a factor of 5 to 1.

"This is the difference," VanDenPlas said, "between a well run professional machine and a gaggle of amateurs, posing in true Rumsfeldian fashion, who 'don't know what they don't know.' I would be shocked if such a chasm exists next cycle between the parties—these aren’t mistakes to be repeated if you want to do things like win elections."

Given the response from Republican partisans to the failure of Romney's campaign and to the apparent failure of its technology investments, the stakes for the next time—the mid-term elections in 2014, and the next presidential race in 2016—will be that much higher for Republican campaigns. It's doubtful they'll ignore the lessons learned this campaign season.

It would help I suspect if Romney actually understood that technology is a tool to help people, vs people being a tool to help the party...

That's my takeaway of the difference in the two strategies; not that Obama doesn't also see people as tools, but I suspect he doesn't only see them as such. From my understanding of Romney's repeated statements regarding minorities, poor, etc, he sees people as things, objects, and resources to be used, and not as people.

Other outlets have pointed out the irony...the candidate who ran on claims of saving the country through free-market business management acumen and cutting costs, was completely out-maneuvered by a "community organizer" using a true low-cost, high-efficiency management philosophy that got the job done with far less money.

It would help I suspect if Romney actually understood that technology is a tool to help people, vs people being a tool to help the party...

That's my takeaway of the difference in the two strategies; not that Obama doesn't also see people as tools, but I suspect he doesn't only see them as such. From my understanding of Romney's repeated statements regarding minorities, poor, etc, he sees people as things, objects, and resources to be used, and not as people.

I think you're reading a little more into this issue than is warranted.

It would help I suspect if Romney actually understood that technology is a tool to help people, vs people being a tool to help the party...

That's my takeaway of the difference in the two strategies; not that Obama doesn't also see people as tools, but I suspect he doesn't only see them as such. From my understanding of Romney's repeated statements regarding minorities, poor, etc, he sees people as things, objects, and resources to be used, and not as people.

I think you're reading a little more into this issue than is warranted.

And no, I don't support either party.

This is merely my impression. I believe that Romney doesn't care about me, or my family, because I am not a white Christian.

At the very least I can be assured that President Obama has an inkling about racism and white privilege, and he apparently believes that we need to allocate resources, tight as they may be, towards women, the poor, and the non-white.

You can argue he does so to secure our votes, as a means of creating a voter base to equal the more elder, white, Christian voter base that is nominally Republican, but then again Romney also had that option too.

I think it's illuminating that Romney's software contracts went to people that he knew personally...

...and Obama's went to industry-leading experts and top minds in their field.

It says a lot, not only about campaign style, but also about leadership style & thought-process.

Those friends of his made a lot of money off the campaign, even though Romney lost. Makes me think of CEO's who take huge risks, destroy their companies, and still leave millions richer. Guys like... Romney.

I had a very conspiratorial thought while writing this comment: what if Romney was only in the election to make money, not to win?

I think it's illuminating that Romney's software contracts went to people that he knew personally...

...and Obama's went to industry-leading experts and top minds in their field.

It says a lot, not only about campaign style, but also about leadership style & thought-process.

Those friends of his made a lot of money off the campaign, even though Romney lost. Makes me think of CEO's who take huge risks, destroy their companies, and still leave millions richer. Guys like... Romney.

I had a very conspiratorial thought while writing this comment: what if Romney was only in the election to make money, not to win?

I'm rather impressed that we have such a breakdown of expenditures in the Obama campaign, whereas everyone is still trying to figure out where a lot of the money in the Romney campaign went. Or perhaps I just missed a post delineating the line items there?

Also goes to show when you have 4 years to plan you can do it the smart way as you don't need to waste time and resources fighting for the nomination.

The two approaches taken will be studied for the next go around as both candidates will have to go through the primary season and the tools will have matured to the point that mistakes will prove even more costly if you choose wrong.

I'm rather impressed that we have such a breakdown of expenditures in the Obama campaign, whereas everyone is still trying to figure out where a lot of the money in the Romney campaign went. Or perhaps I just missed a post delineating the line items there?

You missed the prior article on Romney's campaign. This stuff is all public knowledge, no hiding the screwups.

I'm rather impressed that we have such a breakdown of expenditures in the Obama campaign, whereas everyone is still trying to figure out where a lot of the money in the Romney campaign went. Or perhaps I just missed a post delineating the line items there?

You missed the prior article on Romney's campaign. This stuff is all public knowledge, no hiding the screwups.

I'm rather impressed that we have such a breakdown of expenditures in the Obama campaign, whereas everyone is still trying to figure out where a lot of the money in the Romney campaign went. Or perhaps I just missed a post delineating the line items there?

Take a look at the stories linked in the series box for this article. Or send me an email and I'll send you the Excel PivotTable.

Other outlets have pointed out the irony...the candidate who ran on claims of saving the country through free-market business management acumen and cutting costs, was completely out-maneuvered by a "community organizer" using a true low-cost, high-efficiency management philosophy that got the job done with far less money.

Not to mention his tech. strategy embodied the worst of crony capitalism rather than free-market philosophy. If that was how he was going to "run the country" I'm even more happy he lost.

Well, it seems obvious to the rest of us. Big man business suit Romney assumes all of humanity are as greedy SOBs as him and his corporate ilk. The fact that some people just love their work and do it for self-satisfaction and/or to uplift the human condition without thought of demanding money or perpetual control over it never occurred to him.

I had a very conspiratorial thought while writing this comment: what if Romney was only in the election to make money, not to win?

I don't think you can say that about Romney; I haven't seen any evidence that he made campaign spending decisions in such a way as to profit personally. And plus he's already super rich, it's not like he needs the money.

A person you could conceivably wonder this about, though, is Zac Moffatt, Romney's digital director. According to Ars' previous reporting, out of $40 million spent on IT stuff by the Romney campaign, $14 million went to Moffatt's own company, Targeted Victory. Some of that is supposed to be "spend-through" money (payments for subcontractors hired by TV to work for Romney), but I haven't seen an accounting yet of how much that is versus how much Targeted Victory pocketed.

(For reference, according to the figures above, the Obama campaign spent a total of $3.6 million for all their services and consulting providers, combined. So $14 million is a lot of money for Team Romney to route to a single vendor.)

At a minimum, having the campaign's digital director decide to hire his own company to handle so much of the campaign's IT spend feels... odd. Did they bid any of that work around to other vendors? How involved was Moffatt in the decision(s) to rely so heavily on his firm? And what deliverables did the Romney campaign get in return for that $14 million, exactly? That seems like an awfully big chunk of change to have flow out into a black box -- especially a black box owned by a person in a position to make or influence the purchasing decisions.

It's possible there's a completely above-board explanation for all this (like 95% of the money was spent through and TV only took enough to cover administrative costs, say), but if I were a Romney donor I'd definitely want to know what that explanation is.

I had a very conspiratorial thought while writing this comment: what if Romney was only in the election to make money, not to win?

I don't think you can say that about Romney; I haven't seen any evidence that he made campaign spending decisions in such a way as to profit personally. And plus he's already super rich, it's not like he needs the money.

I would say the same. To me, the way the Romney campaigned handled all of this is just a natural result of the "business man" mentality. One of the major planks of his campaign was his business experience, but no mention that that type of approach isn't appropriate for certain things, like running a campaign or the government.

Well, it seems obvious to the rest of us. Big man business suit Romney assumes all of humanity are as greedy SOBs as him and his corporate ilk. The fact that some people just love their work and do it for self-satisfaction and/or to uplift the human condition without thought of demanding money or perpetual control over it never occurred to him.

The GPLs were invented to protect humanity from assholes like him.

Yes I would say the GPL was vindicated definitely. (I don't think I have ever seen a picture of R Stallman smiling, much less laughing!)

Thanks for the breakdown! While I don't know about most of the services listed, it sounds to me like OFA used software that was proven, and could be tested to work together with other services. Honestly, the whole schtick with OFA running the campaign like a business is slightly inaccurate. They ran the business exactly how a well-run, big-budget, non-profit organization would run its operations. Some of the tools are the same tools I've dealt with, having been involved in the NGO sector for the last 5 years. And the operation, while being on a completely different level of budget than what I'm used to, is similarly run: Do the tools do what they need to do? Can they be proven to do so? And can the tools serve the people they need to serve in an efficient way? Any well-run non-profit org faces these questions, and needs a yes to all of them. A campaign is more akin to a non-profit than a true business.

Romney's team actually did use traditional business tactics, to his failing. To be fair, though, he managed to still get a vast number of people to vote for him, which says that it's still about people's beliefs. It's just that OFA was slightly better at getting the key voters.

I know right!!! Those dumbass, ridiculous, bible-beating, uneducated, gun toting, toothless, racist, gay hating, Mexican hating, black hating, tech-unsavvy, cheapskate, evil, uninformed Republicans!Let's congratulate ourselves some more for being masters of all wisdom and knowledge, and shining like a beacon straight from the comment section of a tired story-line at Ars!

If you voted this election season, President Obama almost certainly has a file on you. His vast campaign database includes information on voters’ magazine subscriptions, car registrations, housing values and hunting licenses, along with scores estimating how likely they were to cast ballots for his reelection.

And although the election is over, Obama’s database is just getting started.

You want to be the editor, get your own website rather than dictating other people what they should do with theirs.

Quote:

I know there is a lot of joy over the Republican fail, but there are only so many ways to call every republican a piece of shit. So, all you lovers of your freshly defended liberty(?), are you really letting the train pass because self congratulation is more important? Any one of these stories is much more relevant. Unfortunately they don't reflect well on "the party" so maybe they find no friends here.

Or maybe they haven't gotten around to them yet. Your paranoia and partisan worldview says something about you, not them.

I am, however, curious to know if any of their tech achievements could have happened if they would have had to go through a contested primary or had less than 4 years to plan for it.

The short answer is "no."In the first interview I did with the OFA tech team, Harper Reed said that they would not have been able to do all the groundwork for their systems if they had had to contend with the primaries—they would have been much more resource and time constrained. Part of the reason Obama was able to outspend the Republicans overall and invest in things like a tech team early on was that he did not have to fight off a primary challenger, and was the sole recipient of primary-period campaign funding. Romney's fundraising outstripped the President toward the end, but it couldn't make up for lost time.

But it isn't accurate to say they had 4 years to plan for it. Reed and the others on the staff weren't brought on until the summer of 2011. So they effectively had 18 months. They may have had some idea of what *not* to do in advance from 2008, but they didn't have a complete roadmap.