Accidental Heartbleed Vulnerabilities Undercut Recovery Effort

Scans find 300,000 affected servers, but a surprising number of newly vulnerable servers have surfaced since Heartbleed warning was first sounded.

Update (5/13/2014): After working with F5 to track down the apparent surge in machines sporting the Heartbleed vulnerability, researcher Yngve Petterson has revised his findings: "Due to an issue with the network connection of the prober the test used to detect F5 BigIP [servers, it] showed higher numbers than it should have, and the numbers of such servers therefore got very inflated for the scans that were run in the past month. This means that the BigIP-related information and conclusions are not correct. ... My apologies to F5 and their customers for this mistake."

10 Ways To Fight Digital Theft & Fraud

(Click image for larger view and slideshow.)

Heartbleed recovery efforts are being partially undermined by new servers coming online that use a vulnerable version of the popular OpenSSL cryptographic software library.

That warning was sounded by software developer Yngve Pettersen at Vivaldi Technologies, based on ongoing scans he's conducted -- focusing on sites listed on the Alexa list of the world's million most popular websites that sport the OpenSSL vulnerability. He began regularly scanning about 500,000 different servers, 20% of which include Heartbeat TLS Extension support, after the vulnerability was discovered in early April.

The good news is that overall rates have sharply declined. "In the six scans I have made since April 11, the number of vulnerable servers [has] trended sharply downward, from 5.36% of all servers, to 2.33% this week," Pettersen said Wednesday in a blog post. In fact, he said 75% of all vulnerable servers appeared to have been patched within four days of the bug being discovered.

Other scanning efforts have also charted a steep decline in the number of vulnerable servers. For example, port 443 scans conducted by Errata CEO Robert Graham one month ago found that more than half a million servers appeared to have the vulnerability. Since then, however, the number of vulnerable systems appears to have halved.

"Whereas my previous scan a month ago found 600,000 vulnerable systems, today's scan found roughly 300,000 thousand systems -- 318,239 to be precise," Graham said Thursday in a blog post. That was out of a total of 1.5 million systems he detected that support the "heartbeat" feature. "Note that only OpenSSL supports heartbeats, meaning that the vast majority of SSL-supporting servers are based on software other than OpenSSL," he said. Graham also emphasized that he scanned IPv4 addresses, which won't produce like-for-like comparisons with other types of scans, for example of the DNS domain names Pettersen scanned.

While the overall drop in vulnerable sites -- by either measure -- is good news, based on his ongoing scans, Pettersen has unearthed two problems. First, many patched servers are still using their old digital certificates. "Given that any server that was patched after April 7 has to be assumed to have had its certificate private key compromised -- because criminals may have used Heartbleed to compromise their server -- this indicates a serious problem for the users of those sites," Pettersen said.

Second, there's been an alarming rise in the number of new servers that sport Heartbleed, including a sizeable number of F5's BigIP crypto accelerator servers. "In my most recent scan, 20% of the currently vulnerable servers -- as distinguished by IP addresses -- and 32% of the vulnerable BigIP servers were NOT vulnerable when they were scanned previously," he said. "This means that thousands of sites have gone from not having a Heartbleed problem to having a Heartbleed problem."

Pettersen sees two likely explanations for the increase in new servers harboring Heartbleed. First, the number of BigIP servers he's spotting has doubled in the past month, likely because F5 customers have brought new BigIP servers online. Evidently, many of those servers are using older, vulnerable firmware and haven't yet been patched to eliminate Heartbleed.

Another potential explanation for the surge in newly vulnerable machines is that some IT administrators may have thought that their OpenSSL-using servers were vulnerable to Heartbleed when they weren't. "This, perhaps combined with administrative pressure and a need to 'do something,' led them to upgrade an unaffected server to a newer but still buggy version of the system, perhaps because the system variant had not yet been officially patched," Pettersen said.

Going forward, patch managers will need to keep a close eye on version control to avoid accidentally upgrading to a Heartbleed-vulnerable version of their enterprise software before a patched version becomes available. "We expect to see administrators update to a fixed version, revoking and re-issuing certificates, and take various other steps that are applicable for their setup, as soon as possible," said Kasper Lindgaard, director of research and security at Secunia, via email.

But patches are not yet available for all vulnerable products. Accordingly, Lindgaard said, "We are not yet at a stage where everybody can simply patch. Some vendors are still in the process of identifying vulnerable products and developing patches. It will definitely take some time before we reach the milestone where it is only the end users that need to patch."

In my experience many system administrators do not pay very close attention to security issues. The commons sysadmin is more concerned with getting a working server up and online. In the same way that programmers are generally more concerned with producing usable code over secure coding, sysadmins have a similar mindset when it comes to servers.

Therefore, it is the responsibility of the information security office to constantly remind IT operations of the security vulnerabilities in a particular product. Also, it is of the upmost importance that prior to allowing a server to be put in production or placed in the DMZ, it must be evaluated for vulnerabilities.

Some more bad news about Heartbleed - A study from Netcraft (http://news.netcraft.com/archives/2014/05/09/keys-left-unchanged-in-many-heartbleed-replacement-certificates.html) found that of all the scanned sites impacted by Heartbleed, only 43 percent had reissued their SSL certificates. On top of that, 7 percent had done so with the same private key.

After patching our Apache web servers, our IT group conducted an audit to see if bad guys had used the Heartbleed exploit against us previously. How did they do that? Well, our company makes a network appliance that parses all SSL transactions going over the wire in our environment, among other things. We could look back to see if there were any heartbeat messages (the attack vector in this case) to our devices since March 2012 when the vulnerability was first introduced. Our dog food tastes great! You can read the details here: http://www.extrahop.com/post/blog/how-extrahops-it-team-performed-a-heartbleed-audit-going-back-years/

Heartbleed is a vulnerability arising from the hearbeat functionality of OpenSSL. I'm not sure where you got the idea that this is malware or an "infection." Hearbleed is an exploit technique against vulnerable version of OpenSSL. You're giving out incorrect information and you should correct this in your article.