The Heartbleed Bug Shows How Fragile the Volunteer-Run Internet Can Be

By Tim Fernholz

April 10, 2014

Matthew Prince, CEO of the online security company CloudFlare, watched his company’s top cryptographer turn “white as a ghost” after learning about a bug in the essential infrastructure of the internet last week. That flaw, he says now, is the worst thing to happen to the internet since it became a mass medium in the early 2000s.

The heartbeat read overrun vulnerability—popularly branded as “heartbleed“—is part of the most widely-used encryption systems on the internet, and it exposed practically everyone’s data to both hackers and government spies for the last two years. It’s worth checking to see if you need to change your passwords at many popularly used sites, including Facebook, Yahoo and OkCupid.

The bug was discovered thanks to a new project called HackerOne, a collaboration backed by Microsoft and Facebook, designed to standardize the practice of rewarding hackers who help fix vulnerabilities they find, rather than exploiting them. HackerOne hosts the Internet Bug Bounty, which focuses on software that is essential to internet functionality.

That includes OpenSSL, the open-source software where heartbleed was found by Google researcher Neel Mehta, who donated his $15,000 bounty to the Freedom of the Press Foundation.

Thanks, Neel.

But the bug has existed for two years. Why did it take until last week to discover, and why did the means of the search only exist four months ago? The answer lies in in how the basic infrastructure of the internet is governed by its users—or not.

This software “is as close to a public good that you have,” Prince says. It’s open-source code managed by a foundation. While that has plenty of advantages, it also means the software is comparatively under-invested in by experts in the field and not as efficiently maintained—Prince describes it as a “spaghetti nest of code.” It received less than $1 million in income from donations and consulting work last year.

In other areas of critical infrastructure, Prince noted, the government might be responsible for the management, but NSA surveillance scandals have made many in the tech community (inside and outside the US) loath to trust government agencies. Indeed, the NSA is one of the few actors with the capability to truly exploit the bug, and some have suggested that it may well have done so. The internet has a long and growing tradition of self-governance—see the ongoing evolution of ICANN, which tracks domain names—but there are clearly gaps.

This has left major internet companies to coordinate around the issue, but that creates its own problems, including perceptions of an insiders’ club privy to early warnings of problems. Prince, whose company was informed by OpenSSL soon after the flaw was discovered because it provides security to a significant chunk of the internet, said there is already resentment from those who were not clued-in immediately. Companies including Yahoo were not informed until the public announcement and were left scrambling to protect their users.

Or take some other fallout from the bug. You can learn more about the technical details here, but heartbleed allows an intruder to comb through the most recently-used data on a server. Among the many sensitive things that could be in that data are “SSL certificates”—essentially, keys that create encrypted connections and assure browsers that users’ data (a credit card number, for instance) can be safely entered. If an intruder were to obtain these SSL certificates, the browsers could be fooled into thinking dangerous sites aren’t.

Typically, Prince says, websites don’t use many different certificates. Now, for security reasons, they may need to revoke them all. But because the process to check for revoked certificates takes time, some web browsers don’t check whether certificates they’ve already downloaded are broken. Prince, meanwhile, is worried that revoking and re-issuing the hundreds of thousands of certificates used by his networks will slow connections. He fears that the knock-on effects of resetting the certificates and the way they are processed between web browsers and servers could be an “almost unfixable problem.”

Researchers are working to determine exactly how vulnerable the bug has made online encryption and patch the holes, but the case certainly serves as an eye-opener about the fragility of the internet.