23,000 HTTPS certs will be axed in next 24 hours after private keys leak

Customers of HTTPS certificate reseller Trustico are reeling after being told their website security certs – as many as 23,000 – will be rendered useless within the next 24 hours.

This is allegedly due to a security blunder in which the private keys for said certificates ended up in an email sent by Trustico. Those keys are supposed to be secret, and only held by the cert owners, and certainly not to be disclosed in messages. In the wrong hands, they can be used by malicious websites to masquerade as legit operations.

Unless the affected certificates are replaced in time, visitors to websites using Trustico-sold HTTPS certs will be turned away by their browsers, due to the digital certificates being revoked.

The whole situation is a mess, and possibly the result of a turf war. Here's what we've managed to ascertain.

What is Trustico?

Trustico, based in Croydon, UK, touted SSL/TLS certificates, which are used by websites to encrypt and secure their connections. It resold certs from the Symantec brand umbrella: Symantec, GeoTrust, Thawte, and RapidSSL. This umbrella is now owned and operated by DigiCert.

If you wanted to buy, say, a RapidSSL-issued certificate, you could do so via Trustico. The HTTPS cert ultimately leads back, along a chain of trust, to DigiCert, a root certificate authority trusted by web browsers and other software. In turn, a website presenting the Trustico-sold cert is trusted, its traffic secured using encryption, and the reassuring green padlock is displayed in visitors' browsers.

Why are the certificates being revoked?

According to DigiCert's chief product officer Jeremy Rowley earlier today, Trustico told DigiCert in early February that its resold certificates had been in some way "compromised," and that the certs needed to be mass revoked as a result.

DigiCert staff, we're told, asked Trustico for more information on this security mishap. The reseller replied it had a copy of the private keys, which is usually grounds for revocation, and thus insisted that DigiCert revoke the certificates.

When pressed for evidence, Trustico on Tuesday simply emailed DigiCert 23,000 certificates' private keys as proof it held this information, it is claimed. This forced DigiCert's hand: under the rulebook of standards set by the elders of the certificate security and browser worlds, the Trustico-sold certificates had to be revoked as a precaution within 24 hours. Specifically, the ones with their private keys in the email will be canceled.

"Trustico has not provided any information about how these certificates were compromised or how they acquired the private keys," explained Rowley.

"As is standard practice for a Certificate Authority, DigiCert never had possession of these private keys. Currently, we are only revoking the certificates if we received the private keys. There are additional certificates the reseller requested to have revoked, but DigiCert has decided to disregard that request until we receive proof of compromise or more information about the cause of this incident."

On Twitter, Rowley continued: "I'll likely be posting the private keys later once people have a fair chance to replace their certificates ... The allegation of compromise, keys compromised, and request for revocation all came from Trustico."

Before you raise an eyebrow too high, by posting the private keys, Rowley plans to disclose self-signed certificates, produced using the private keys, to prove the secret information was sent to DigiCert without revealing the actual information in public. Some have already popped online as proof DigiCert received the secret keys from Trustico.

Alarm bells

To warn netizens to the upcoming mass revocation, DigiCert's RapidSSL business sent out email alerts to Trustico customers urging them to get new HTTPS certificates or watch their sites go dark. Here's a copy of the memo, passed to El Reg:

Trustico requested revocation of their Symantec, GeoTrust, Thawte and RapidSSL certificates, claiming the certificates were compromised. When we asked for proof of the “compromise,” Trustico did not provide details on why they were requesting the immediate revocation. Trustico’s CEO indicated that Trustico held the private keys for those certificates, and then emailed us approximately 20,000 certificate private keys.

When he sent us those keys, his action gave us no choice but to act in accordance with the CA/Browser Forum Baseline Requirements, which mandate that we revoke a compromised certificate within 24 hours. As a CA, we had no choice but to follow the Baseline Requirements.

Following our standard revocation process, we gave notice via email to each certificate holder whose private keys had been exposed to us by Trustico, so they could have time to get a replacement certificate.

Now, over to Trustico.

Upset and denials

We asked the Brit biz for comment, and had yet to hear back at time of writing. However, posting on Mozilla's security policy newsgroup, Trustico product manager Zane Lucas was clearly upset that DigiCert sent out the above alert.

"We didn't authorise DigiCert to contact our customers and we didn't approve the content of their email," wrote Lucas.

"At no time had any private keys been compromised, nor had we ever informed to you that any private keys had been compromised. During our many discussions over the past week we put it to you that we believe Symantec to have operated our account in a manner whereby it had been compromised. Your usage of the word compromise has been twisted by you to your benefit and is absolutely defamatory."

To put this in context: Trustico was fed up with using Symantec certs, and on February 13, it formally abandoned the umbrella of brands – ahead of Google Chrome and Mozilla Firefox officially distrusting the certificates due to past security fumbles by Symantec. Trustico said it had complained privately to Symantec of long-running concerns over the security safeguards on Symantec-branded of certificates, hence Lucas' reference to its Symantec account.

Although Lucas stressed the private keys for Trustico's resold certificates were not compromised, it did, according to DigiCert, email a copy of 23,000 of them to the root authority seemingly to trigger their revocation. At that point, DigiCert considered the certificates at risk, and started the countdown clock to cancel them.

Trustico and DigiCert have clearly majorly fallen out, with the pair going their separate ways this month amid the behind-the-scenes drama. It even appears Trustico tried to stop DigiCert from using its online portal to send out today's emailed warning.

In future, Trustico will flog Comodo HTTPS certificates rather than peddle Symantec-branded certs. Cynics have suggested the Brit reseller ordered the revocation of its Symantec-umbrella certs so it could drive its customers onto Comodo certificates, and thus avoid the looming Google Chrome HTTPS certificate apocalypse without losing many, if any, punters. In effect, website owners have been caught up in a turf war between Trustico and DigiCert.

How did Trustico get the private keys to certificates it resold? We don't know for sure – but it did, and still does, offer an online private key generator for certificates. Just saying.

In an email sent to customers a few hours ago, and seen by The Register, Trustico said it will provide free certificates to replace the soon-to-be-nuked SSL/TLS certs:

Recently we wrote to you to let you know that we are no longer offering Symantec, GeoTrust, RapidSSL and Thawte branded SSL Certificates. Unfortunately, Google Chrome has decided to distrust these SSL Certificates. It's important to us that you SSL Certificate continues to function as normal, and not be compromised by the distrust of the Symantec brands. It is now required that you replace any existing distrusted SSL Certificate with one that is trusted by all web browsers.

Rest assured, there hasn't been any type of compromise of our systems. However, Symantec brands will cease to function correctly due to Google Chrome's decision to distrust them.

Recently DigiCert acquired the Symantec SSL Certificate division and subsequently an e-mail was sent by DigiCert to some of our SSL Certificate customers advising of the revocation of their distrusted SSL Certificate. We didn't authorise this e-mail to be sent and had specifically disabled it within the DigiCert system. We understand that the e-mail sent about your distrusted SSL Certificates may be confusing. It's important that you take the opportunity to replace your SSL Certificate as soon as possible.

We're providing free replacement of affected SSL Certificates. To enable a free replacement, you'll receive an e-mail report today if you have affected SSL Certificates. Your report will contain a unique coupon code for each affected SSL Certificate. When you replace your distrusted SSL Certificates using your unique coupon codes you'll receive extra validity free of charge. If you have any questions please feel free to reply to this e-mail.

Meanwhile, DigiCert said it, too, will offer free replacement certs to folks using Symantec-branded HTTPS certificates, which will be ignored by web browsers later this year. And, of course, don't forget you can grab free HTTPS certificates from Let's Encrypt that all major browsers trust.

Today has been marred with confusion. Trustico's customer support lines have been jammed with complaints and queries, following DigiCert's email alerts. Reg readers told us they felt left in the dark. Perhaps it'll all be clearer in a few hours, when the dust has settled – and the certs have been nuked. ®

Updated to add

Trustico kept the private keys to its customers' certificates in cold storage, and provided them to DigiCert to start the revocation process.