The Bleeding Hearts Club: Heartbleed Recovery for System Administrators

The Heartbleed SSL vulnerability presents significant concerns for users and major challenges for site operators. This article presents a series of steps server and site owners should carry out as soon as possible to help protect the public. We acknowledge that some steps might not be feasible, important, or even relevant for every site, so the steps are given in order both of their importance and the order they should be carried out.

1. Update Your Servers

If you haven't yet, update any and all of your systems that use OpenSSL for TLS encrypted communications. This includes most web servers, load balancers, cache servers, mail servers, messaging and chat servers, VPN servers, and file servers, especially those running on Linux, Unix, BSD, Mac OS X, or Cygwin.

The vulnerable OpenSSL version numbers are 1.0.1 through 1.0.1f and 1.0.2-beta1. The flaw is fixed in OpenSSL 1.0.1g. However, some operating systems have introduced the fix to earlier branches of OpenSSL, and may instruct you to install packages with minimum versions such as 1.0.1e-2+deb7u5 (in the case of Debian GNU/Linux).

After installing a fixed version of OpenSSL, be sure to restart all services that depend on it. On your sysytem this might include web and proxy servers such as apache, nginx, pound, and squid, caches such as memcached and redis, databases like mysql and postgres, and mail services like postfix, exim, and dovecot. When in doubt, reboot the entire server if possible.

If you manage systems with custom operating systems like switches and routers, you may need to ask your vendor for a patch directly.

If you haven't updated your systems yet, stop reading and do it now. If this is the only step you can carry out in your environment, you will still have done the most important thing by far.

2. Test Your Servers

It's important to verify that the hole has been closed, especially if you have multiple servers and services to stitch up. The bad news is that this vulnerability is relatively easy to exploit. The good news is that means there are a few tools available to see if you're safe.

The SSL Server Test from Qualys SSL Labs will let you know if your web server remains vulnerable. If you have servers running on other ports to test, or STARTTLS mail servers, you can try the hb-test.py script. The hbcheck script can help you test an internal network using nmap. Finally, if you have a large number of hostnames to test, my hb-batch.py script might be helpful.

Please note these tests might not be completely reliable, and running them against servers you do not own might not always be considered polite.

3. Be Safer Next Time

This is the worst and biggest security flaw we've seen recently, but it won't be the last. Putting good practices into play for Heartbleed can help you prepare for anything else that might come down the pike next.

One of the strongest protections you can have against TLS vulnerabilities is Perfect Forward Secrecy. This is not simple to configure, and does not yet have global browser support. However, it is the encryption technology that provides the best defense against attacks with the potential to steal your private key and use it to decrypt your traffic.

You should also make sure you're practicing good password discipline. Use a password vault, use strong passwords, change them regularly, and don't reuse them.

Practice least authority for certificates, too. If you don't need to give everyone root access to every server, you probably don't need to give every server a certificate for *.example.com.

Finally, make sure you have reliable (if not automated) process for providing all of your servers with security updates quickly. After all, the only thing worse than getting pwned by a zero-day vulnerability is getting pwned by a one-day.

4. Consider Rekeying Your Servers

One of the worst things about the Heartbleed vulnerability is that it makes it theoretically possible for an attacker to recover your server's private key. Fortunately, the probability of this being possible on a given server appears quite low. Unfortunately, we can't yet be completely sure if that's true.

Key theft is a terrible attack because it tends to be undetectable by you, the server operator. Worse still is the harsh truth that, unless all your connections are served with Perfect Forward Secrecy, this would allow such an attacker not only to decrypt any newly intercepted traffic but to decode records of past traffic. If you run a server that intelligence agencies are likely to attack, this is a seriousproblem.

[Update: Cloudflare's Heartbleed Challenge has shown private key recovery to be achievable in a realistic scenario. This was done with a more work-intensive and discoverable attack than things like Heartbleed password and session recovery, but it might not be the only real-world attack targeting private key material.]

That means you may wish to consider revoking and regenerating your existing SSL certificates using new keys. Doing so will protect against the possibility of passive traffic decryption (if you don't use PFS) and man-in-the-middle attacks with a stolen key.

Because private key compromise via Heartbleed currently appears to be quite rare, this may not need to be a priority except for high value services (large or sensitive email and messaging systems, software distribution points, banks). Other services may not need to panic and rush to rekey quite so urgently. For most threat scenarios, adopting PFS provides greater overall protections than rekeying so we will remind you to make PFS a priority.

The details of the rekeying process will vary depending on the Certificate Authority you use to generate certificates and/or manage domain names. Some will allow you to regenerate in one step. Some will require you to revoke the old certificate before requesting a new one.1 Most will have a prominent link in their control panels, and many will waive their normal fees right now.

If you are given the option during the certificate regeneration process, it's a good idea to create a .csr file (Certificate Signing Request) and private key locally on your server using the openssl command. It might seem strange to prefer trusting OpenSSL at the moment, but it's still a safer bet than trusting a third party with your private key right off the bat.

5. Consider Changing Passwords

Unlike private key compromises, Heartbleed leakage of recently-used passwords from server processes linked to OpenSSL appears to have been quite common. Unfortunately, this could affect not just your operators and staff but your users.

This means you should perform risk assessment and determine which categories of passwords on your servers and services may need immediate resets, user-reset-on-next-login, or advisories suggesting resets. Variables in the risk assessment include how quickly you were able to patch your servers after the vulnerability was publicly disclosed at around 17:30UTC on 2014-04-07,2 the sensitivity and value of potentially accessible accounts, whether accounts had been used recently (meaning their passwords were in RAM), and the probability that random or specific people on the Internet might have found your servers to be interesting targets.

You should determine which passwords are of sufficient value to deserve precautionary resets, and perform these after the steps above, in order to offer the new passwords proper protection. (If you've decided to rekey because of a concern about private key exposure, that is another reason to change users' passwords.)

You should also consider changing CSRF and OAUTH authentication tokens, invalidating session cookies, and rotating authentication cookies. These steps can be performed independently of passwords changes and may be far less disruptive.

6. Update Your Users

Your users have already heard of this scary Internet password thing, and chances are they're concerned about how it affects your site. Let them know what you've done, what you will do, and what the remaining risks are. Don't try to give them a false sense of security. Knowing that you're working on it and reaching out to them at all will work wonders.

7. Turn on Perfect Forward Secrecy

Because you skipped it in step three, didn't you? That's okay. There's still time. We'll wait.

1. The infrastructure for revocation of certificates is quite broken, but that's a story for another post.

Related Updates

"The laws of mathematics are very commendable but the only law that applies in Australia is the law of Australia", said Australian Prime Minister Malcolm Turnbull today. He has been rightly mocked for this nonsense claim, that foreshadows moves to require online messaging providers to provide law enforcement with...

Call your Congressmember now to save online privacy!Back in October of 2016, the Federal Communications Commission passed some pretty awesome rules that would bar your Internet provider from invading your privacy. The rules would keep Internet providers like Comcast and Time Warner Cable from doing things like...

Poisonous political divisions have spawned an encryption arms race across the Trump administration, as both the president’s advisers and career civil servants scramble to cover their digital tracks in a capital nervous about leaks. The surge in the use of scrambled-communication technology — enabled by free smartphone apps such as...

It’s worth noting, though, that unlike other secure messaging apps, like standard-bearer Signal, Confide’s encryption is closed source and proprietary, meaning no one outside the company knows what’s going on under the hood of the app. “One key is always, do you make code publicly available that’s been...

The last year has seen enormous progress in encrypting the web. Two categories in particular have made extraordinary strides: news sites and US government sites. The progress in those fields is due to months of hard work from many technologists; it can also be attributed in part to ...

As the presidential campaign was in full swing early last year, now-President Trump made his feelings on encryption clear. Commenting on the Apple-FBI fight in San Bernardino, Trump threatened to boycott Apple if they didn’t cooperate: “to think that Apple won't allow us to get into [the] cell phone,” Trump...

Peter Eckersley of the Electronic Frontier Foundation (EFF) built upon former senator Ted Stevens’ analogy that said the internet is like a bunch of tubes, saying, “If you use HTTP, those tubes are totally transparent. Anyone along the way can look inside and see exactly what you’re doing.” Use HTTPS...

This year was one of the busiest in recent memory when it comes to cryptography law in the United States and around the world. But for all the Sturm und Drang, surprisingly little actually changed in the U.S. In this post, we’ll run down the list of things that happened...

This was a great year for adoption of HTTPS encryption for secure connections to websites. HTTPS is an essential technology for security and privacy on the Web, and we've long been asking sites to turn it on to protect their users from spying (and from censorship and tampering with site...

Many have contacted us with concerns about yesterday’s election results. At this critical moment, we want digital civil liberties supporters worldwide to feel confident that EFF remains steadfast in its mission and method: to use law and technology to champion civil liberties and provide a potent check against overreach. EFF...