Knowledge of a CA private key would allow MitM attackers to transparently supplant any certificates signed by that private key. It would also allow cyber criminals to start forging their own trusted certificates and selling them on the black market.

Given the huge profits that could be made with such knowledge, and the fact that a highly trusted and ubiquitous certificate (such as any of the main Verisign keys) would be a very difficult thing to revoke quickly, it stands to reason that there would be highly motivated and well funded criminal elements attempting to get their hands on such keys on a regular basis.

How do certification authorities deal with this threat? It sounds like a real nightmare, having to ring-fence the keys away from all human eyes, even the sysadmins. All the while the keys have to be used on a daily basis, often by internet-connected signing services.

7 Answers
7

Serious certification authorities use heavy procedures. At the core, the CA key will be stored in a Hardware Security Module; but that's only part of the thing. The CA itself must be physically protected, which includes proactive and retrospective measures.

Proactive measures are about preventing attacks from succeeding. For instance, the CA will be stored in a vault, with steel doors and guards. The machines themselves are locked, with several padlocks, and nobody holds more than one padlock key. Physical security is of paramount importance; the HSM is only the deepest layer.

Retrospective measures are about recovering after an incident. The HSM will log all signatures. The machine is under 24/7 video surveillance, with off-site recording. These measures are about knowing what has happened (if you prefer, knowing a priori that, should a problem occur, we will be able to analyze it a posteriori). For instance, if "illegal" certificates have been emitted but the complete list of such certificates can be rebuilt, then recovery is as "easy" as revoking the offending certificates.

For extra recovery, the CA is often split into a long-lived root CA which is kept offline, and a short-lived intermediate CA. Both machines are in the cage and bunker; the root CA is never connected to a network. The root CA is physically accessed, with dual control (at least two people together, and video recording) on a regular basis, to emit the certificate for the intermediate CA, and the CRL. This allows revoking an intermediate CA if it got thoroughly hacked (to the point that its private key was stolen, or the list of fraudulently emitted certificates cannot be rebuilt).

Initial setup of a serious root CA involves a Key Ceremony with herds of auditors with prying eyes, and a formalism which would not have been scorned by a Chinese Emperor from the Song dynasty. No amount of auditing can guarantee the absence of vulnerabilities; however, that kind of ceremony can be used to know what was done, to show that security issues have been thought about, and, come what may, to identify the culprit if trouble arises. I have been involved in several such ceremonies; they really are a big "security theater" but have merits beyond the mere display of activities: they force people to have written procedures for everything.

The question is now: are existing CA really serious, in the way described above ? In my experience, they mostly are. If the CA has anything to do with VISA or MasterCard, then you can be sure that HSM, steel and ill-tempered pitbulls are part of the installation; VISA and MasterCard are about money and take it very seriously.

For the CA which are included in Web browsers and operating systems, the browser or OS vendor tends to require a lot of insurance. There again, this is about money; but the insurance company will then require all the physical measures and accounting and auditing. Formally, this will use certifications like WebTrust.

This is true even for infamous CA like DigiNotar or Comodo: note that while they got hacked and fake certificates were issued, the said certificates are known and were revoked (and Microsoft added them to a list of "forbidden certificates" which can be seen as a kind of "revoked, and we really mean it" -- software must go out of his way to accept them nonetheless).

The "weak" CA are mostly the State-controlled root CA. Microsoft can refuse to include a root key from a private venture, if Microsoft feels that enough insurance has not been provided (they want to be sure that the CA is financially robust, so that operations can be maintained); I know of a bank with millions of customers who tried to get their root key included in Windows, and were dismissed on the basis that they were "too small". However, Microsoft is much weaker against official CA from sovereign states; if they want to do business in country X, they cannot really afford to reject the root CA key from government X. Unfortunately, not all governments are "serious" when it comes to protecting their CA keys...

Thanks for the great wealth of information. I have images of hooded robes and candles. But seriously, it all sounds very impressive.
–
lynksDec 3 '12 at 16:08

Namely MSFTs audit requirements for Government-managed are different than commercial CAs (source) , though I don't have enough information to say if the requirements & audit are "weaker" or "stronger" for state-based CAs
–
makerofthings7-C.LamontDec 3 '12 at 16:09

The information on Diginotar is incorrect; you see, Diginotar didn't even know what was issued, and what wasn't issued in terms of credentials. The reason why handling the Diginotar compromise was so difficult was this lack of knowledge. Later on, the certificates were seen in the wild, but even now one cannot be sure that is the comprehensive list of certificates created. This information can be read in the FoxIT report, or alternatively understanding the mechanism and poor security practices of Diginotar.
–
BrennanDec 3 '12 at 17:33

@Brennan Is that difference the main reason why Comodo is still in the certificate business while Diginotar was bankrupted and is in liquidation; or were there other extenuating circumstances in Comodo's case that allowed them to keep operating?
–
Dan NeelyDec 3 '12 at 21:35

So, then, how much does that usually all cost to do? Of course, its very expensive to maintain the security stuff, but knowing how much each part costs would be nice.
–
AJMansfieldDec 4 '12 at 1:40

The simple (and rather unfortunate) answer is that they don't, really. The sysadmins are trusted to be safe, and they secure their signing boxes in the same way that you'd secure any sensitive corporate server. In cases of performing CSRs there may be a human step required, but there's absolutely nothing to stop a determined attacker from compromising a CA network if they put enough time and effort in. At most, they'd be using a HSM to store the private keys, though I'd bet good money that it's not commonplace.

Use of HSMs had crossed my mind. I suppose in that case it is possible for no human to ever know the key. It's just in the box. But it still seems to be that a target so valuable would be under serious (read :foreign government) attack on a daily basis. And given that every network is broken eventually, I wondered why I had never heard of such a breach. Am I misoverestimating (thanks george) the value?
–
lynksDec 3 '12 at 14:21

The problem with a foreign government attacking it is that if it could be traced to the foreign government, they'd be in major political hot water. There isn't enough to gain from it.
–
AJ HendersonDec 3 '12 at 14:34

@lynks I think AJ is largely right here. If attribution is gained, it puts them in a nasty political situation. It's also cheaper and safer for them to just get one of their own domestic CAs to issue a fake certificate which their citizens' browsers will happily accept.
–
PolynomialDec 3 '12 at 14:37

3

Any downvoters care to actually explain their vote? Not much point in giving something a -1 if nobody knows why.
–
PolynomialDec 4 '12 at 6:57

3

I'd be guessing that it's because your bet, and what appears to be guess work, is contradicted in other answers and comments...
–
naught101Dec 14 '12 at 12:50

One thing that many do is use sub-certificates and revocation lists. This doesn't mean problems don't happen and private keys have been stolen of trusted CAs, but if the CA doesn't make their private key accessible ever, they have a certain degree of protection. They can instead use their private key to sign a derived CA cert and then use that CA cert (with a much more limited validity period) to sign certificates for distribution. Then the master private key never has to be exposed to the network.

If a breach does happen, since they know that the sub-cert will have to specify the revocation list they control to be valid, they can also then revoke a compromised sub-cert.

An easy way to check if a particular CA does this is to check the chain of trust on the cert. Often times, there will be one or even several CA certs in between the given certificate and the root trusted CA cert.

On the physical side they first keep the root CA completely offline. Typically what happens is that they set up the root CA, make subordinates, then take the root CA completely offline and take the hard drives and HSMs (sometimes even the whole server) and essentially lock them in a safe.

Next, they segment the network to keep those subordinate/issuing CAs secure.

Finally, on those subordinates they use HSMs to store the certificates.

OCSP and CRLs are used to revoke certs, but it's not just that simple.

On the procedural side, they have a crazy amount of layers including locked rooms that require multiple people at the same time to revoke/sign subordinates or roots (A Key Signing Ceremony). These are 24/7 recorded (audio and video) and require a person of trust from the CA, in addition to the admins and executives. It's a huge deal. Here's the video.

Edit: Contrary to what Polynomial has said, from what I have seen, HSMs are commonplace for CAs, and even for large implementations of internal CAs.

Yeah, it's not like diginotar was hacked, and comodo has lost control of one of it's subsidiary certificates before now. They are supposedly audited before they are trusted by major browsers, but the system is not flawless and it only takes one bad CA to ruin the whole system. And I expect even at the more reputable CA's the executives like to cut corners and costs at every turn they can.
–
ewanm89Dec 3 '12 at 15:24

Great video, great high-level view of what's going on. One note; that key signing ceremony was for the Root DNS Certificate Authority; "one certificate to rule them all". If someone ever got that private key, they could take down the entire Internet by basically masquerading as the DNS root and replacing all the trusted DNS servers on the Net with its own. That is the foremost example of a key that needs serious protection as per Thomas Pornin's answer.
–
KeithSDec 4 '12 at 2:33

@KeithS I noticed they all seemed to be wearing DNS - marked clothing, I didn't realise the DNS used PKI, although it makes sense now I think about it.
–
lynksDec 5 '12 at 12:02

Not every CA (government, commercial, or private) stores private keys the same way. Most legitimate operators use a HSM. It's possible that the vendor publishes CRL revocation lists using a one way link from the root to the SubCa. (Transmit-only serial cables, audio cables, QR codes, Ethernet with only a few pins connected.... etc.)

To get specific then it really depends on what product you're talking about.

Each of those vendors has different standards as to which CAs are qualified to be "Root CAs" in that given product. Specifically, each vendor has different certifications, processes and procedures (such a cold storage requirements) each CA must adhere to before being included in that root trust list.

Microsoft has the Root Certificate Program and that requires every CA to follow the following audit standards (which should address all your technical, and procedural questions)

ETSI 102 042

ETSI 101 456

WebTrust for CAs

WebTrust EV Readiness audits

ISO 21188 ( Note they do not accept ISO 27001 )

Specifically, the CA must complete an audit and submit audit results to Microsoft every twelve (12) months. The audit must cover the full PKI hierarchy that will be enabled by Microsoft through the assignment of Extended Key Usages (EKUs). All certificate usages that we enable must be audited periodically. The audit report must document the full scope of the PKI hierarchy including any sub-CA that issues a specific type of certificate covered by an audit. Eligible audits include:

ISO 21188:2006, “Public key infrastructure for financial services -- Practices and policy framework,” completed by either a licensed WebTrust for CAs auditor, or an audit authority operating according to the laws and policies for assessors in the same jurisdiction as the CA.

(Note these audit requirements don't apply to government CAs and may be different)

It's definitely possible to store a private key in a HSM that never exposes the private key (offloads the requests to that key) and permanently track every usage of that private key. I don't remember if any of those standards require a HSM, but given the relatively minor cost of a HSM I can't imagine why it's not being done.

Thanks for all the links and things to google, thats a great help. I assume by 'vendor' you are referring to 'browser/device vendors'? In other words: people who would be integrating certificates into their products.
–
lynksDec 3 '12 at 16:01

Every phone (Android, iPhone, Blackberry), OS (Linux, Windows, etc), and even some web browsers (Firefox) maintain independent trust stores. I didn't cover them all, but there are many more. I have no idea what the requirements for inclusion are.
–
makerofthings7-C.LamontDec 3 '12 at 16:03

Yep thanks, was just clarifying that I understood what you meant by vendor.
–
lynksDec 3 '12 at 16:06

Rather than going to all the effort of breaking into a bunker and stealing the private key mission impossible style, one could simply crack the private key. This can all be done from the comfort of your own home. All you need is the CA Public Key, some data that was signed, sample signature, and a quantum computer. I have 75% of the stuff already, if someone has the last bit I'm happy to split the spoils 50/50.

whilst I agree with @Thomas's very thorough response above I'd like to add that I installed the root CA system for a UK banks own debit card system (each wee chip on the card is actually a certificate).

We used Thales HSM devices (6 figure £ costs) that had proper tamper proof systems for tilt, temperature, physical attack and so on.

in terms of the key ceremony, which as Thomas describes has herd of auditors standing around cold machine rooms - it's more complicated than that. You need to ensure that the key generation activity and the key transport to the HSM are separated activities and that by collusion the key cannot be compromised. The Thales HSM tools allow the key to be split into segments, each encrypted with a transport key so that individual key holders can make their way to the key ceremony separately, by different transport (for heavens sake, no car sharing) to the location usually at the HSM in the production server room.

Please don't consider being a root CA is simple - yes, I know that you can create a Root CA using free-to-use tools - You need to work out your own risk appetite and whether you can build a root ca to match the threat model. e.g. for debit card certificates (where the consequential loss could be "the entire bank"), the total amount spent was enormous and had lots of people running around to ensure that no single person could collude or have control of the key.

If you are just creating a root ca as a local test system then clearly the risk model is different.

My advice would be that unless you are the size of a large organisation (Fortune 500) or it is your actual business to be a Root CA - then outsource.

I forgot to add that one of our key ceremonies got quite stressful when the transport key would not enter properly. because we couldn't look at what was being typed in by the key holder - we got them to keep on trying in case of input error. Then, lightbulb moment "has the key got an I or the letter O in it?" "yes", which explains the problem because itit was a Hex code!
–
Callum WilsonDec 4 '12 at 11:15

1

Lol at people involved at that level of internet security not recognizing a hex code when they see it :)
–
Stijn de WittApr 4 '14 at 19:39