As recently as yesterday, Cloudflare published preliminary findings that seemed to indicate that it would be difficult, if not impossible, to use Heartbleed to get the vital key that essentially unlocks the secure sockets layer padlock in millions of browsers. To be extra-sure, Cloudflare launched “The Heartbleed Challenge” to see how other people exploiting Heartbleed might fare. The company set up an nginx server running a Heartbleed-vulnerable version of OpenSSL and invited the Internet at large to steal its private key.

Just nine hours later, software engineer Fedor Indutny and Ilkka Mattila at NCSC-FI had obtained the server's private keys using nothing but the Heartbleed vulnerability. As of this writing, CloudFlare had confirmed a total of four winners: Rubin Xu, a PhD student in the Security group of Cambridge University, as well as security researcher Ben Murphy.

The results are a strong indication that merely updating servers to a version of OpenSSL that's not vulnerable to Heartbleed isn't enough. Because Heartbleed exploits don't by default show up in server logs, there's no way for sites that were vulnerable to rule out the possibility the private certificate key was plucked out of memory by hackers. Anyone possessing the private key can use it to host an impostor site that is virtually impossible for most end users to detect. Anyone visiting the bogus site would see the same https prefix and padlock icon accompanying the site's authentic server.

The demonstration that it's possible to extract private SSL certificates means that out of an abundance of caution, administrators of sites that used vulnerable versions of OpenSSL should revoke and replace old certificates with new ones as soon as possible. Given the huge number of sites affected, the revelation could create problems.

"The bad news is that [discovery] changes our recommendation from: reissue and revoke as a medium priority to reissue and revoke as a high priority," Matthew Prince, CEO of CloudFlare wrote in an e-mail to Ars. "We've accelerated our own reissuance and revocation process."

Cloudflare had originally reasoned that, at least on the Linux-based platform it uses, a server's certificate and private keys are usually stored in the server's memory early on after booting up, and because servers are not booted up frequently, it would be difficult to find a situation in which the block of memory that Heartbleed can be used to access (which can be up to 64Kb of information) would contain a server's private keys.

The broad-scale theft of private SSL keys is a worst-case scenario, as it leaves websites open to impersonation and it makes ostensibly-secure communication available to eavesdroppers.

Ultimately, Indutny sent over 2.5 million requests to Cloudflare's exposed server, and Mattila sent around 100,000 in the same amount of time, and both were able to steal the server's private keys. Cloudflare said it rebooted the server about six hours into the challenge, and the company theorized that the reboot, “may have caused the key to be available in uninitiallized heap memory.”

The availability of such sensitive information is very troubling, and the fix is now not as easy as getting users to change their passwords. “Our recommendation based on this finding is that everyone reissue and revoke their private keys,” Cloudflare wrote in an update today. “CloudFlare has accelerated this effort on behalf of the customers whose SSL keys we manage.”

The first winner of the Heartbleed Challenge, Fedor Indutny, wrote on Twitter, "I have approximated total number of alive TLS/SSL certs to be 6.5 millions." Obviously, not every site with such a certificate was dependent on OpenSSL, but it does put into startling perspective the massive cleanup project that Heartbleed is shaping up to be.

The process of revoking and reissuing certificates is unwieldy and slow even without half of the Internet trying to do the same process at the same time. “If every site revoked its certificates, it would impose a significant burden and performance penalty on the Internet,” wrote CloudFlare in a Friday blog post. “At CloudFlare-scale the reissuance and revocation process could break the CA [Certificate Authority] infrastructure.”

The company said that for its customers running on CloudFlare infrastructure, it has already begun the process of reissuing and revoking SSL certificates in stages, and expects to be done with the process sometime next week.