Heartbleed vulnerability may have been exploited months before patch [Updated]

Fewer servers now vulnerable, but the potential damage rises.

Update: Errata Security's Robert Graham has acknowledged that he was mistaken in his assessment, and that private keys could be at risk. The original story below has been marked up accordingly.

There’s good news, bad news, and worse news regarding the “Heartbleed” bug that affected nearly two-thirds of the Internet’s servers dependent on SSL encryption. The good news is that many of those servers (well, about a third) have already been patched. And according to analysis by Robert Graham of Errata Security, the bug won’t expose the private encryption key for servers “in most software"(though others have said several web server distributions are vulnerable to giving up the key under certain circumstances.)

The bad news is that about 600,000 servers are still vulnerable to attacks exploiting the bug. The worse news is that malicious “bot” software may have been attacking servers with the vulnerability for some time—in at least one case, traces of the attack have been found in audit logs dating back to last November. Attacks based on the exploit could date back even further.

Security expert Bruce Schneier calls Heartbleed a catastrophic vulnerability. "On the scale of 1 to 10, this is an 11," he said in a blog post today. The bug affects how OpenSSL, the most widely used cryptographic library for Apache and nginx Web servers, handles a service of Transport Layer Security called Heartbeat—an extension added to TLS in 2012.

Heartbeat allows a connected Web client or application to send messages to keep a connection active during a transfer of data. When a Heartbeat message is received, the server usually simply echoes back what it got to the sender. However, starting with the initial implementation of Heartbeat in OpenSSL 1.01 (and in all subsequent releases up to OpenSSL 1.01f, including the OpenSSL 1.0.2 beta) the extension could be fooled into sending back the contents of its memory buffer by sending a request that advertised itself as 64 kilobytes long but in fact had no content—resulting in “Heartbleed.” Any information still in that buffer from a previous session, such as decrypted usernames and passwords, could be obtained by an attacker in the response message.

Your key is (probably) safe not safe at all?

In a post to the Errata Security blog, Robert Graham explained that it is highly unlikely that private key data would be stored in the memory buffer that could be leaked using the Heartbleed exploit. “What you can eavesdrop on with Heartbleed hacks is dynamic stuff, stuff that was allocated only moments ago,” he wrote. But that assertion has been refuted, and Graham has since rescinded it, as noted above.

A scan performed by Graham Tuesday night found that of the 28 million servers and other devices that responded to an SSL connection request, only about 600,000 were vulnerable to attack using Heartbleed. “We also found 33,531 machines that had Heartbeats enabled, but which did not respond to the Heartbleed attack,” Graham wrote. “Presumably, this means a third of machines had been patched by the time we ran the scan last night.”

Closing the barn door

The reduced likelihood of exposure of private keys—which would allow an attacker to masquerade as the attacked server or client—is cold comfort, however, considering that usernames and passwords on some systems may have been exposed for months or years by the vulnerability, which has been part of every OpenSSL release since March 2012. There are signs that exploits for the vulnerability were in use by someone for some time before the vulnerability was revealed.

Ales Teska, of the mobile security provider SeaCat.mobi, wrote in an email to Ars that his company’s service, while not vulnerable to the Heartbleed exploit, acted as a sort of “honeypot” for attacks based on the exploit because of its use of OpenSSL. “Our OpenSSL-based software was logging Heartbleed attack attempts to its logs by coincidence,” he wrote, starting on March 23. When upgrading the OpenSSL on two test servers, he checked the logs of the servers and found “these two servers are actually logging such an attempt to the log file (as a generic OpenSSL handshake issue).” SeaCat.mobi has detailed the attempted attacks in a blog post.

Update: SeaCat and Teska have now qualified their comments: " EFF correctly pointed out that there are other tools, that can produce the same pattern in the SeaCat server log (see http://blog.erratasec.com/2014/04/no-we-werent-scanning-for-hearbleed.html#.U0Xq4OaSz-l ). I don't have any hard data evidence to support or reject this statement. Since there is a risk that our finding is false positive, I have modified this entry to neutral tone, removing any conclusions. There are real honeypots in the Internet that should provide final evidence when Heartbleed has been broadly exploited for a first time."

Terrence Koeman of MediaMonks told Ars he found signs of attempts to use the exploit dating back to November 2013. He used the packet content of a successful exploit of the Heartbleed vulnerability to check inbound packets logged by his servers and found a number of incoming packets from a network suspected of harboring a number of “bot” servers that were apparently scans for the vulnerability—sending Heartbleed-style requests to two different development servers in requests that were about five minutes apart.

Apache and nginx servers aren’t the only potential victims of the OpenSSL vulnerability. A malicious server can be used to attack client applications that use OpenSSL 1.0.1. Fortunately, few Web browsers do—Chrome on Android 4.1.1 may be vulnerable, for example, since it uses OpenSSL 1.0.1e, but most Chrome and all Firefox browsers use the Mozilla Network Security Services (NSS) libraries. Apple uses its own cryptographic services for OS X and iOS; an earlier, unaffected version of OpenSSL is included in OS X, but Apple discourages its use.

At this point the only sane thing seems to be wait a week or two and change every password you use on any site on the internet and assume they've all be compromised. =(

I'm only worried about ones with sensitive data, e.g. bank, email, sites I make purchases on. I mean, if someone got into my Ars account...oh well. Worst case the account is banned, and I can't post comments anymore (though I suspect with a proper explanation it would be reinstated with a new password). Doesn't really hurt me, and the password is unique to the account.

And I've gone over to randomly generated passwords (and two-factor authentication, where available) for all of the sensitive ones since the Adobe leak (my password was leaked, and I used it most places). Luckily at that point all I had using it that mattered had 2FA anyways.

Can we get an update on Heartbleed and OpenVPN? I've heard anecdotally that OpenVPN would have been vulnerable, but I haven't seen or heard anything about a UDP scan tool that can be used to actually attempt a POC against OpenVPN; much less any information on how likely OpenVPN would be to leak private keys (as compared to Apache, which as it turns out is "not very, and especially not very if you aren't running FreeBSD").

I think it's time for every vendor who ships something with OpenSSL to start looking at what's introduced in new versions and perhaps make some choices about which options need to be enabled by default. It's time-consuming, but if something new is introduced (like heartbeat), and it's not necessary for most use cases, let it be an option that someone can choose to enable on their own if they need to work with it.

My workload was much lighter because my vendor traditionally does not go with bleeding-edge versions of things included in the base OS.

Reading between the lines, the disclosure seems to hint that the attack was active and in the wild before it was disclosed. I had assumed this to be a "State Sponsored" 0-day prior to disclosure, but who knows really? Hence the rash of password resets.

If you're a LastPass user, definitely run the security check and heed the advice.

Nice. I ran the scan this morning and it wasn't doing this, so this is a last seven-hours (or so) sort of thing. I already changed all my passwords, and I'll probably do it again in a week or two or when everything I use is patched (who knows, maybe it is already and I don't know it)

At this point the only sane thing seems to be wait a week or two and change every password you use on any site on the internet and assume they've all be compromised. =(

I'd wait until the sites have an updated security certificate. I just re-logged into Ars and see they have one issued on 4/8/2014, so they have likely taken care of the issue. It's not the most reliable method (if the expiration date just happened to fall around that time for example, the same cert may have simply been re-upped - I'm guessing that can be done, dunno as I'm not an admin), but you should certainly be suspicious of any certs dated before that point.

However, researcher David Litchfield said that the default Web server distribution on FreeBSD will give up the private key if the attack is the first request after the server is reset.

It would be more professional if claims like this came with some substantiation other than a link to the front page of someone's website. In fact FreeBSD does not have a default web server, though many can be installed so this information is of little use.

So... at what point will it become legitimate to break the nose of every individual associated with the NSA or aware of their illegal domestic activities? Alternately, we could use a cattle brand to sear "CONSTITUTIONAL TRAITOR" on their foreheads as a warning to posterity.

One point that doesn't seem to be covered enough is that heartbleed can only leak memory that was used by the running pid. Anything you get from the kernel is effectivly zeroed out before use.

The reason passwords etc are leaking is because of solutions like mod_php that allow you to bundle buisness logic, a web server and an ssl layer inside the same process. A server that seperates these elements out gains free protection from leakage (as well as enjoying far greater granularity in setting permissions since each componant can run as it's own user with appropriate selinux/apparmour protections).

I'm hopeing that this incident might be a wake up call so that sysadmins start seperating these functionalities out.

So... at what point will it become legitimate to break the nose of every individual associated with the NSA or aware of their illegal domestic activities? Alternately, we could use a cattle brand to sear "CONSTITUTIONAL TRAITOR" on their foreheads as a warning to posterity.

One point that doesn't seem to be covered enough is that heartbleed can only leak memory that was used by the running pid. Anything you get from the kernel is effectivly zeroed out before use.

The reason passwords etc are leaking is because of solutions like mod_php that allow you to bundle buisness logic, a web server and an ssl layer inside the same process. A server that seperates these elements out gains free protection from leakage (as well as enjoying far greater granularity in setting permissions since each componant can run as it's own user with appropriate selinux/apparmour protections).

I'm hopeing that this incident might be a wake up call so that sysadmins start seperating these functionalities out.

That's interesting - is it just PHP (again) or are there other commonly used modules that one can look out for?

One point that doesn't seem to be covered enough is that heartbleed can only leak memory that was used by the running pid. Anything you get from the kernel is effectivly zeroed out before use.

The reason passwords etc are leaking is because of solutions like mod_php that allow you to bundle buisness logic, a web server and an ssl layer inside the same process. A server that seperates these elements out gains free protection from leakage (as well as enjoying far greater granularity in setting permissions since each componant can run as it's own user with appropriate selinux/apparmour protections).

I'm hopeing that this incident might be a wake up call so that sysadmins start seperating these functionalities out.

That's interesting - is it just PHP (again) or are there other commonly used modules that one can look out for?

The biggest offender is PHP, even though fpm means that you can now deploy PHP cleanly and easilly (at last). Most other languages already use application servers due to the performance benefits they bring (caching, long running threads, connection pooling). That said, I know that mod_python exists, even if it isn't used much compared to gunicorn, and IIRC mod_ruby exists too.

So... at what point will it become legitimate to break the nose of every individual associated with the NSA or aware of their illegal domestic activities? Alternately, we could use a cattle brand to sear "CONSTITUTIONAL TRAITOR" on their foreheads as a warning to posterity.

Because the NSA is totally the only explanation for this. Totally.

The possibility seems likely enough to be front and center, given their wide constitution-shredding initiatives. They've lost all benefit of the doubt, and they'd stand the most to gain, given the vast access to Internet backbone traffic.

Does OpenSSL use some kind of version control? Who submitted the change? What other projects do they participate in? What other changes have they introduced?

People need to be sharpening their pitchforks and dipping their torches in tar. The world must demonstrate when rights are violated, the penalties and consequences for the individuals responsible are bone-chilling. Anything less enables them to operate in a consequence-free zone.

You aren't debating, you are asserting.

It's a possibility, not a certainty. Your glib comments pointing to the latter just aren't helpful. Find out the facts first, then sharpen pitchforks if necessary. Going after someone for something they may well not have done weakens any attempt to go after them for things they really HAVE done.

So... at what point will it become legitimate to break the nose of every individual associated with the NSA or aware of their illegal domestic activities? Alternately, we could use a cattle brand to sear "CONSTITUTIONAL TRAITOR" on their foreheads as a warning to posterity.

Because the NSA is totally the only explanation for this. Totally.

The possibility seems likely enough to be front and center, given their wide constitution-shredding initiatives. They've lost all benefit of the doubt, and they'd stand the most to gain, given the vast access to Internet backbone traffic.

Does OpenSSL use some kind of version control? Who submitted the change? What other projects do they participate in? What other changes have they introduced?

People need to be sharpening their pitchforks and dipping their torches in tar. The world must demonstrate when rights are violated, the penalties and consequences for the individuals responsible are bone-chilling. Anything less enables them to operate in a consequence-free zone.

You aren't debating, you are asserting.

It's a possibility, not a certainty. Your glib comments pointing to the latter just aren't helpful. Find out the facts first, then sharpen pitchforks if necessary. Going after someone for something they may well not have done weakens any attempt to go after them for things they really HAVE done.

I just want to see more people alarmed by this. This should be stoking the fires in each of our hearts, anyone who cares about freedom of speech, expression, and revisionist attempts to retract citizens' rights from the world. I want a solution to this, not more talk. I want consequences, right up to the very top, even if that means sending Cheney himself to the gallows as a traitor, god damn it.

I fully understand that sentiment, I really do.

To anyone who would know better than I: Has anyone done an analysis of how this happened yet (e.g. how it was introduced, not what the problem was)? Is it possible to do?

So... at what point will it become legitimate to break the nose of every individual associated with the NSA or aware of their illegal domestic activities? Alternately, we could use a cattle brand to sear "CONSTITUTIONAL TRAITOR" on their foreheads as a warning to posterity.

Because the NSA is totally the only explanation for this. Totally.

The possibility seems likely enough to be front and center, given their wide constitution-shredding initiatives. They've lost all benefit of the doubt, and they'd stand the most to gain, given the vast access to Internet backbone traffic.

Does OpenSSL use some kind of version control? Who submitted the change? What other projects do they participate in? What other changes have they introduced?

People need to be sharpening their pitchforks and dipping their torches in tar. The world must demonstrate when rights are violated, the penalties and consequences for the individuals responsible are bone-chilling. Anything less enables them to operate in a consequence-free zone.

You aren't debating, you are asserting.

It's a possibility, not a certainty. Your glib comments pointing to the latter just aren't helpful. Find out the facts first, then sharpen pitchforks if necessary. Going after someone for something they may well not have done weakens any attempt to go after them for things they really HAVE done.

I just want to see more people alarmed by this. This should be stoking the fires in each of our hearts, anyone who cares about freedom of speech, expression, and revisionist attempts to retract citizens' rights from the world. I want a solution to this, not more talk. I want consequences, right up to the very top, even if that means sending Cheney himself to the gallows as a traitor, god damn it.

I fully understand that sentiment, I really do.

To anyone who would know better than I: Has anyone done an analysis of how this happened yet (e.g. how it was introduced, not what the problem was)? Is it possible to do?

The sentiment is understood but it is important not to go off half-cocked because being wrong about something like this, once exposed, leads to apathy. People won't excited the next time because they'll think it's just more screeching.

I would like to see a central list of sites that were vulnerable and those that are no longer vulnerable. If I don't change my passwords I may be hacked. If I change them too early I may still be hacked. If I didn't need to change one or more then I am less annoyed.

So... at what point will it become legitimate to break the nose of every individual associated with the NSA or aware of their illegal domestic activities? Alternately, we could use a cattle brand to sear "CONSTITUTIONAL TRAITOR" on their foreheads as a warning to posterity.

Because the NSA is totally the only explanation for this. Totally.

The possibility seems likely enough to be front and center, given their wide constitution-shredding initiatives. They've lost all benefit of the doubt, and they'd stand the most to gain, given the vast access to Internet backbone traffic.

Does OpenSSL use some kind of version control? Who submitted the change? What other projects do they participate in? What other changes have they introduced?

People need to be sharpening their pitchforks and dipping their torches in tar. The world must demonstrate when rights are violated, the penalties and consequences for the individuals responsible are bone-chilling. Anything less enables them to operate in a consequence-free zone. Certainly the submitted code should be evidence enough for blacklisting them from any security-related project, regardless of their actual intent. But what will the open source community do to exhaust the possibility this was a paid state actor sabotaging Internet privacy?

Yes, OpenSSL has version control.

Yes, we know who wrote the bad code. He was the co-author of the TLS heartbeat RFC. Kinda makes sense for the person who drew up the specs to also write the code. And seeing as how he's a German working at a German university, that doesn't exactly scream "NSA" to me, ya know?

The reason passwords etc are leaking is because of solutions like mod_php that allow you to bundle buisness logic, a web server and an ssl layer inside the same process. A server that seperates these elements out gains free protection from leakage (as well as enjoying far greater granularity in setting permissions since each componant can run as it's own user with appropriate selinux/apparmour protections).

That's interesting to know. So is there any possibility that a server with http user auth (htpasswd) would be leaking that login info? Where does that live in the apache process?

The one box I'm concerned about is behind http auth, and knowing that apache was not giving up this info would make me quite happy.

So... at what point will it become legitimate to break the nose of every individual associated with the NSA or aware of their illegal domestic activities? Alternately, we could use a cattle brand to sear "CONSTITUTIONAL TRAITOR" on their foreheads as a warning to posterity.

Because the NSA is totally the only explanation for this. Totally.

The possibility seems likely enough to be front and center, given their wide constitution-shredding initiatives. They've lost all benefit of the doubt, and they'd stand the most to gain, given the vast access to Internet backbone traffic.

Does OpenSSL use some kind of version control? Who submitted the change? What other projects do they participate in? What other changes have they introduced?

People need to be sharpening their pitchforks and dipping their torches in tar. The world must demonstrate when rights are violated, the penalties and consequences for the individuals responsible are bone-chilling. Anything less enables them to operate in a consequence-free zone.

You aren't debating, you are asserting.

It's a possibility, not a certainty. Your glib comments pointing to the latter just aren't helpful. Find out the facts first, then sharpen pitchforks if necessary. Going after someone for something they may well not have done weakens any attempt to go after them for things they really HAVE done.

I just want to see more people alarmed by this. This should be stoking the fires in each of our hearts, anyone who cares about freedom of speech, expression, and revisionist attempts to retract citizens' rights from the world. I want a solution to this, not more talk. I want consequences, right up to the very top, even if that means sending Cheney himself to the gallows as a traitor, god damn it.

No, we don't know what is going, but a pattern of shit is emerging that should keep each and every one of us awake at night. Do we have a problem? We have a problem? Is the US a failed state possessed by an executive branch that is above the reach of rule of law?

At the risk of feeding the over-the-top paranoia troll: Don't you think the NSA would insert something more reliable than something that just *might* -- if executed at exactly the right moment, on exactly the right server (since most sites of any note have to load-balance among *many* servers) -- manage to grab a particular target's credentials? Because, unless there's proof that a server's cert can be hijacked using this flaw (which apparently is still undetermined), that's ALL this is. HARDLY conducive to effective and timely investigations of potential (NSA-concerned) threats.

Take off the tinfoil hat long enough to actually try to comprehend what's going on here, wouldya?!?

I would like to see a central list of sites that were vulnerable and those that are no longer vulnerable. If I don't change my passwords I may be hacked. If I change them too early I may still be hacked. If I didn't need to change one or more then I am less annoyed.

Sean Gallagher / Sean is Ars Technica's IT Editor. A former Navy officer, systems administrator, and network systems integrator with 20 years of IT journalism experience, he lives and works in Baltimore, Maryland.