On Heartbleed, Entanglement and the Looming API-calypse

Originally appeared on the Orchestrate Blog

April 16, 2014

Share +

A few months ago, I wrote a blog post outlining what we should all demand from our service providers, Orchestrate included. Many of these services present a straightforward API. Integration of a new service can take as little as a few minutes, leading some to conclude, wrongly, that they haven’t just staked their entire business on a provider about which they know only a little.

Twin concerns are voiced in that post – that we are becoming more entangled as various API-fronted service providers integrate each other into their systems, and that we’re more vulnerable to cascading failures should any one crucial provider fail as a result. With the disclosure last week of the Heartbleed bug, this seems to have come a cropper.

We asked our friend Adam DuVander what his thoughts were about security and these increasingly interdependent API services:

“APIs allow you to focus on the problems you solve best, but that shouldn’t be at the expense of good security practices. Each API you use is certainly another potential attack vector.”

We saw this first hand here at Orchestrate. We watched and waited as Amazon, Stripe, DNSimple, Pager Duty, Fastly and several other services we used to build our service worked diligently to remediate the problem. I began to wonder a couple of things – which of our service providers were also waiting on some of the same service providers as Orchestrate, and what does this interdependency mean for all the service providers and their users? In the API business, we are often even customers of one another. Oh, the webs we weave.

As Tim Prendergrast, CEO of cloud security firm Evident.io and a veteran of Adobe’s big move to the cloud, neatly summarized when I emailed him about this problem:

“As services (not just OSS projects), become more intertwined, we are likely to see failures at the top or middle of the chain impacting downstream providers reliant on these partners for critical service data or infrastructure. The failure is the same, but the tiers at which failure occurs…will change.”

Straightforward enough – Heartbleed impacted everyone, but was felt most acutely when used by popular service providers. However, I also wondered how informed most API users really are. I put the question to Prendergrast:

“Organizations…are unwittingly accepting the liability and risk associated with code that they have never reviewed for security or resiliency.” [Emphasis mine.]

The “unwittingly” part really stands out to me in this quote, but I’ll come back to it because Tim makes a couple of other great points:

“This third-party code can go by many names — Open Source Software (OSS), Free-to-use Services, and Enterprise Products and Services. The differentiating factor [between this code and code written in house] is that we often have complete control over our own codebase, as well as the ability and responsibility to review it periodically for security, reliability, resiliency, and integrity.”

So it isn’t just service providers, but really any code we bring in without proper vetting. Prendergrast ends with a pretty harsh conclusion:

“Heartbleed is a great example of how underfunded OSS development has been pervasively integrated throughout the industry, and has been blindly accepted as “secure” because no glaringly obvious bugs were discovered.”

“Blindly accepted” joins “unwittingly” as the key words for me. How great is the risk? Are we abandoning robustness and resiliency in a quest for the ease and economies of cloud usage? Are all these public API-fronted services at risk of an interlocking collapse that disables huge swaths of the online economy? Is it wise for the CEO of such a company to speculate thusly?

I think it is. I think this is a problem we must address not just to “gain trust” but to continue to grow.

We wrote about transparency as key for public API providers in that list of requirements, but we didn’t think about it broadly enough. This problem begs the very big question: how transparent do we need to be to allow users to make a sound decision? The Orchestrate team took this question seriously. After all, a growing catalogue of developers, startups and enterprise companies are now storing production data with us. Unsurprisingly the answer we came up with was that we need to be “radically transparent.”

What does radically transparent mean? We are actually coming to terms with the implications. We know we are going to disclose as much as we can while keeping our service safe. We are now preparing a list of service providers and OSS we use, carefully balancing full disclosure with the risks associated with disclosure. And we plan to share everything we can so users can make an informed decision, and not blindly or unwittingly incorporate a service they don’t trust. We will share the public APIs we use, all service providers, OSS we use, and ask our providers if they can do the same.

If we alone disclose our interdependencies, what difference will it make? We don’t know. We can only control what we disclose and who we use, including why we chose them and what they choose to disclose.

I will leave off with a reminder from DuVander that going back is not really an option:

“The time spent vetting and tracking the security of APIs you consume is still much less work than implementing them on your own. Even better, if you choose APIs you trust, with excellent security teams, then they in turn become a member of your security team.”

The way is forward. And while I am reluctant to say we got off lucky with Heartbleed, if all service providers don’t do more to inform and educate our users, things will get a whole lot worse.