Comodo Certificate Issue – Follow Up

On 15th March 2011, a RA partner of the Comodo CA suffered an internal security breach (Comodo incident report). The attacker used the RA’s account with Comodo to cause 9 fraudulent certificates to be issued.

The domain names of the certificates were as follows:

addons.mozilla.org

login.live.com

mail.google.com

www.google.com

login.yahoo.com (x3)

login.skype.com

global trustee

These certificates were issued between 6 and 8pm on 15th March from the UTN-UserFirst-Hardware root, without the use of an intermediate certificate. The attack was discovered almost immediately by an internal Comodo check, and the certificates were revoked (via both CRL and OCSP). The compromised RA account has been suspended.

So far, according to Comodo, the only OCSP activity seen relating to these certificates has been two hits, one from the attacker’s IP and one from another very close by, plus hits from testing by Comodo and the notified companies. This suggests that the certificates have not been deployed in an attack, though it is possible that the attackers would block OCSP requests as well.

The IP address 212.95.136.18 appeared to be associated with the attack. According to Comodo, this IP address is an ADSL connection in Iran. From the OCSP traffic mentioned above, we think the attacker is aware that the certificates have been revoked.

Risk Analysis

Given the above, we do not believe that there has been a root key compromise. Nevertheless, an attacker armed with these fraudulent certificates and an ability to control their victim’s network could impersonate the sites in a way that would be undetectable to most users.

With particular regard to Mozilla, the certificate for ‘addons.mozilla.org’ would allow the attacker to imitate the addons.mozilla.org website, and potentially deceive users into downloading malicious software. However, they would not be able to interfere with automatic updates, either to Firefox or to addons. These mechanisms use different domain names, and updates to Firefox also do additional checks to match the certificate issuer to the expected value (which is not UTN-UserFirst-Hardware).

Mitigating Risk

On being informed of this issue by Comodo at 9.47pm GMT on 16th March, Mozilla considered a number of technical avenues. Although Comodo’s revocation is a significant mitigating step, we thought that additional measures made sense and eventually decided to hard-code a blacklist of the certificate serial numbers into Firefox. We therefore produced RC2 of Firefox 4 (released as Firefox 4 final on 22nd March), with two additional code patches (1, 2). These patches disable these specific certificates, plus one additional certificate issued to us by Comodo for testing, making a total of 10. These fixes were also included in updates to Firefox 3.5 and 3.6, also released on 22nd March. As soon as all the patched versions were released, we made a release announcement with some details of the problem.

Mozilla did not publish the information we received prior to shipping a patch. In early discussions, we were concerned that any indication that we knew about the attack would lead to attackers blocking our security updates as well. We also recognized that the obvious mitigation advice we might offer (to change Firefox’s security preferences to require a valid OCSP response in all cases, or to remove trust from Comodo’s certificates, or both) risked causing a significant portion of the legitimate web to break as well.

Additionally, neither we nor Comodo have found any evidence of access to their OCSP responder being blocked, either in Iran or anywhere else. We have also found no evidence of any other sort of attack.

In hindsight, while it was made in good faith, this was the wrong decision. We should have informed web users more quickly about the threat and the potential mitigations as well as their side-effects.

Monitor their OCSP logs for evidence of use of these certificates, or evidence that access to their OCSP responders is being blocked from any geographical locations. (So far: no sign of use or blocking.)

Cancel all relationship with the RA concerned. (So far: the RA is suspended.)

Change their practices to use intermediate certs rather than issuing directly off the root, and use a different one for each RA.

Ongoing Discussion

Unfortunately, the practice of issuing certs directly from the root eliminated some possible steps we could have taken to mitigate the problem. We are concerned about the amount of trust Comodo seems to have placed in RAs whose network security they did not oversee.

Comodo appropriately revoked the certificates immediately and notified the major browser vendors so that more proactive actions could be taken. We are still working with Comodo to find more information on how the breach occurred and what measures they plan to put in place to prevent future recurrences.

This issue raises many questions about the systems surrounding authentication and security on the web. We intend to have a vigorous discussion about what technical and policy changes we can make to significantly improve the situation. You can join the discussion in the mozilla.dev.security.policy forum.

30 responses

I am not sure they would not be able to interfere with automatic updates, if they can impersonate a server they can block Mozilla update servers entirely without needed to impersonate them (and other browser vendors), leaving a lot of browsers without the update with the embedded CRLs

Since this is the second incident with a Comodo agent being caught issuing valid certificates in error, the first being in 2008, I am curious to know what it would take for a CA to get blacklisted from Mozilla entirely? Here we had a certificate issued to someone who didn’t own the domains in question, and one of the domains for which a fraudulent certificate was issued was owned by Mozilla. I figured issuing a fraudulent certificate to domain owned by a major browser vendor and then sending said fraudulent certificate to an IP address in Iran would be enough to lose all trust in a CA, apparently not.

@Ross: Blacklisting an entire CA, especially a fairly large one like Comodo, is not really practical. All of the legitimate websites whose certificates are signed by Comodo (and there are a lot of them) would then be untrusted by Mozilla users. This would result in major problems for users of those sites, as well as for the sites themselves, which would then have to purchase certificates from some other CA in order to maintain trust with anyone who uses Firefox.

I have seen many (legitimate) certificates issued by Comodo RAs. All of them, however, were signed by intermediate CA certificates (e.g. UTN-UserFirst-Hardware -> XXX CA -> http://www.site.com). From what we read here, it appears that in this case the attacker was able to get stuff signed by the root CA.

What exactly is the role of “UTN-UserFirst-Hardware” in the Comodo PKI infrastructure and who is in possession of the private key?

Which strikes me as meaning, essentially, that once you trust a given CA, you can never un-trust them again, no matter what mistakes they make.

If you’re not willing to revoke trust because it’s painful, then you need to be way, WAY more careful about extending trust in the first place. I’d suggest exploring the separation of CAs into bundles for specific regions. You have us all trusting a bunch of tiny CAs in countries that seem quite corrupt and authoritarian. I, for one, find that slightly terrifying, and I would rather see useful bundles of CA certificates for specific zones, rather than defaulting to granting global trust for everyone you deal with.

Mozilla’s internal update code (including that used in Firefox) performs a number of additional checks beyond just establishing an HTTPS connection to the update server. These checks include verifying the cert is one that we expect, and not one issued by some trusted but unexpected party.

Quoting:
“In hindsight, while it was made in good faith, this was the wrong decision. We should have informed web users more quickly about the threat and the potential mitigations as well as their side-effects.”

I think the details were only pre-shared with browser manufacturers on the condition that disclosure be limited to cert blacklists, no? You have to respect that otherwise you come off as a tool, and you don’t get the heads-up (nor any ability to protect your users) next time!

Comodo did issue a cert to someone who was not entrusted to do so.
I expect a Trust Company to actually verify the requester (in this case to check if Yahoo, Mozilla and others had issues a new CSR and requested the to be signed certificate in question.

This wasn’t the case…

I do think that dropping Comodo or any other trust companies which are not doing a human check (with all the papers and time involved) would be a good sign that cheap is not always good.

Is it sensible to set up something comparable to a fire drill? I mean: make it possible for e.g. (major) browsers to create fake security attacks, for testing, after which the CA’s should report e.g. to those browsers, in a well-defined way, exactly as if it were a real attack?

So, what you’re saying is that Comodo is in a position to keep everyone silent about a big screw-up like this and that’s OK?!

I’m sorry, but I’m siding with ioerror on this one. Public disclosure in this case would not have benefited any other attackers. The risks would not have increased, but a lot of people could have protected themselves by being aware of the issue.

In the worst case scenario this could have cost some Iranian activists their live and you’re saying: “Watch it Mozilla or next time you won’t get the heads-up”?

I say that’s BS and I highly doubt Comodo would be able to keep everyone else informed about something like this, but keep Mozilla out of the loop because “last time they spilled the beans.” Imagine the scandal and the hit to their reputation if they would even attempt to do something like that— endanger millions of users.

I’m from Iran and has recently (about a week) faced problems reaching HTTPS sites mentioned above including addons.mozilla.org, login.yahoo.com and most of the time, mail.google.com which are unreachable. The error which is shown is “The requested URL could not be retrieved” “The DNS server returned Time out” which is shown in both Firefox and Opera.
I wondered if this might have any relation to the forementioned certificates issue?
I hope that both Comodo and Mozilla look for any activity from hackers more precisely.

The whole concept of PKI is utterly useless, just have a look at the long list of root authorities that mozilla trusts indifferently. There are several agencies from countries with governments well known to be capable of torture and you can take it for granted that these agencies will issue faked certificates whenever it fits their government. Most likely it is happening routinely.

At the very least the issuing Root-CA must be shown prominently for every SSL page.

I wonder if I sign a piece of code with a code-signing cert, and have it time-stamped by a 3rd party server, and later the CA issuer gets its authority revoked altogether, will my code still provide “Sig is OK” by the OS?

@Taylor Hedberg
I think this would not be practical for any kind of product but if there is a safty problem with your car you would be happy if there was a product recall. Just like with any faulty commodity.

COMODO customers must be offered refunds for any certificates purchased through this CA and Mozilla / Microsoft should force a product recall!

All affected parties (Google, Mozilla, Microsoft, Skype, Yahoo etc) should seek the full warranty to be paid on this mis-issued COMODO certificates.

The only forseeable way of fixing PKI, and this type of attack is to use DNSSEC and which can be used to verify the certificate is the correct one (for how this works, see Dan Kaminsky’s blog on the subject – http://dankaminsky.com/2010/12/19/dnssec-ch4/ ).

justmoi: there’s a proposed specification for that, but it’s not going to help if a CA gets hacked. Similarly Comodo should have had processes in place to automatically check domain ownership but those got bypassed in this attack.

There are similar proposal that would have _browsers_ check that the CA in the presented certificate matches the expected CA. These show more promise because it’s a check independent of the potentially-compromised certificate authority. One proposal requires DNSSEC support first, the other is to remember which CA you saw last time.

mhn: that addons certificate is the one fraudulently issued in the Comodo hack. It has been revoked.