Share this story

Content delivery network Cloudflare is introducing a free service designed to make it harder for browser-trusted HTTPS certificates to fall into the hands of bad guys who exploit Internet weaknesses at the time the certificates are issued.

Browser-trusted certificate authorities are required to use a process known as domain control validation to verify that a person requesting a certificate for a given domain is the legitimate owner. It requires the requesting party to do one of three things:

create a domain name system resource record with a specific text string;

upload a document with a specific text string to a Web server using the domain;

prove receipt of the email address containing a text string sent to the administrative contact for the domain

The Princeton researchers demonstrated that this validation process can be bypassed by BGP attacks. Before applying for a certificate to a targeted domain, an adversary can update the Internet’s BGP routing tables to hijack traffic destined for the domain. Then, when a CA checks the DNS record or visits a URL, the CA's query goes to an attacker-controlled server rather than the legitimate server of the domain operator. When the attacker is able to produce the text string designated by the CA, that is considered proof of domain ownership and the CA issues a certificate to the wrong party.

Reining it in

But these attacks come with limitations. BGP attacks usually hijack only a portion of a domain’s incoming traffic, rather than all of it. As a result, computers in one part of the world will be directed to the attacker’s imposter server, while computers elsewhere will still reach the legitimate server.

Cloudflare, with more than 175 datacenters worldwide, is unveiling a new service called multipath domain control validation that’s designed to exploit this limitation of BGP hijacking. As its name suggests, it performs the validation process from multiple origins that follow different Internet paths to the domain. Unless the results from multiple queries are identical, the validation will fail.

“We’re going to be leveraging Cloudflare’s global network to perform this domain check, whether it’s DNS or HTTP, from various vantage points that are connected through various networks,” Nick Sullivan, head of cryptography at Cloudflare, told Ars. “If you’re hijacked, [the fraudulent data] only applies to a subset of the requests.”

Agents and orchestrators

Cloudflare will be making a programming interface available for free to all certificate authorities. The multipath check for domain control validation consists of two services: agents that perform domain validation out of a specific datacenter, and a domain validation “orchestrator” that handles multipath requests from CAs and dispatches them to a subset of agents.

When a CA wants to ensure a domain validation hasn’t been intercepted, it can send a request to the Cloudflare API that specifies the type of check it wants. The orchestrator then forwards a request to more than 20 randomly selected agents in different datacenters. Each agent performs the domain validation request and forwards the result to the orchestrator, which aggregates what each agent observed and returns the results to the CA.

Sullivan said Cloudflare has designed the new service to be an effective measure against another potential domain validation attack that spoofs IP addresses in DNS requests that use the user datagram protocol (UDP). Because the IP address of the computer making the request can be spoofed, an attacker can make a request to a targeted domain appear to come from a CA. Then, by manipulating a maximum fragment size setting, the attacker can receive a second identical response.

The new Cloudflare API prevents these DNS spoofing attacks because it sends queries from multiple locations that can’t be predicted by the attacker, Sullivan said. In a message, he wrote:

Multipath DCV was designed for and is primarily effective against on-path attacks. An additional feature that we built into the service that helps protect against off-path attackers is DNS query source IP randomization. By making the source IP unpredictable to the attacker, it becomes more challenging to spoof the second fragment of the forged DNS response to the DCV validation agent.

Sullivan said Cloudflare is offering the service for free because the company believes that attacks on the certificate authority system harms the security of the entire Internet. He said he expects the use of multipath domain validation to become standard practice, particularly if it’s offered by other large networks. Eventually, he said, it may be mandated by the CA/browser forum, which sets industry guidelines for the issuance of TLS certificates.

“I’m a little surprised this hasn’t happened yet,” Sullivan said. “We’re hoping that this announcement and this product helps spur the CA/Browser forum to adopt and require this more robust multiperspective validation for certificate authorities. It truly is a risk that hasn’t been exploited yet, and it’s just a matter of time.”

Perhaps I need more coffee, but the impression I get from my first read-through is that if a BGP attack has been executed on a domain, both the original server and the attacker's server will be rejected because the multi-pathing won't line up. This sounds like it could be a good way for hackers to make legitimate servers inaccessible to CloudFlare users by setting a BGP to their server that then redirects to the legitimate server, just to make the pathing not line up.

Can someone do a BGP attack on the new Cloudflare API and just intercept a little earlier in the process?

The point here is visibility. Malicious actors would have to BGP hijack a large percentage of Cloudflare's network ranges/Autonomous Systems (AS). That kind of move requires huge bandwidth at the destination and it's highly visible.

Contrast that to hijacking the network range of a smallish DNS server.

Perhaps I need more coffee, but the impression I get from my first read-through is that if a BGP attack has been executed on a domain, both the original server and the attacker's server will be rejected because the multi-pathing won't line up. This sounds like it could be a good way for hackers to make legitimate servers inaccessible to CloudFlare users by setting a BGP to their server that then redirects to the legitimate server, just to make the pathing not line up.

This doesn't take the server(s) offline (it's not a real-time check during server access), it's to prevent the issuance of certificates. So the denial of service would only be limited to the timeframe in which a server was attempting to renew or issue a certificate.

In addition, if you read the Cloudflare blog, the request is sent to a random subset of the given domain validation servers; it's not the entire world that has to agree. That means the attacker must control the BGP routes for either a lot of servers to guarantee that the random subset will exist in his or her control, or to be very lucky.

Since this is a new process, I would also expect some fine tuning. You could introduce knobs for tuning the number of servers that have to agree, or a minimum geographic distribution, or other parameters.

Perhaps I need more coffee, but the impression I get from my first read-through is that if a BGP attack has been executed on a domain, both the original server and the attacker's server will be rejected because the multi-pathing won't line up. This sounds like it could be a good way for hackers to make legitimate servers inaccessible to CloudFlare users by setting a BGP to their server that then redirects to the legitimate server, just to make the pathing not line up.

This particular validation only applies when registering a new certificate. If the original server has an already issued and valid certificate, the BGP attack and validation failure won’t impact the existing certificate already issued for the original server. It would just result in a failure to issue a new certificate to the attacker’s server.

It would prevent new certificates from being registered to the original server, but only if (by bad luck or deliberate timing) they’re trying to renew their certificates at that exact time.

I am both very pleased that CloudFlare is being very gracious and allowing CAs to use it for free, considering what a threat this is, and dismayed at how such a critically important Layer 4 protocol can be used to such effect and how the likelihood of it being updated anytime soon is small.

I'm also wondering how this will be integrated into SSL Labs' test. This definitely seems like a "Grade capped at C" or "Automatic fail" after it has had a chance to be adopted.

Perhaps I need more coffee, but the impression I get from my first read-through is that if a BGP attack has been executed on a domain, both the original server and the attacker's server will be rejected because the multi-pathing won't line up. This sounds like it could be a good way for hackers to make legitimate servers inaccessible to CloudFlare users by setting a BGP to their server that then redirects to the legitimate server, just to make the pathing not line up.

This particular validation only applies when registering a new certificate. If the original server has an already issued and valid certificate, the BGP attack and validation failure won’t impact the existing certificate already issued for the original server. It would just result in a failure to issue a new certificate to the attacker’s server.

It would prevent new certificates from being registered to the original server, but only if (by bad luck or deliberate timing) they’re trying to renew their certificates at that exact time.

That was what I missed, that it was for registration only, not for checking validation of existing certificates. More coffee has been acquired.

Nice intention, but unless/until every single trusted CA adopt it, it is pointless, as a malicious agent would still be able to create a cert using that CA and clients don't make more verification than checking if the cert was released by a trusted CA.

Nice intention, but unless/until every single trusted CA adopt it, it is pointless, as a malicious agent would still be able to create a cert using that CA and clients don't make more verification than checking if the cert was released by a trusted CA.

Sure, but the same applies to any new security feature.

For example the seatbelt being made available for free to manufacturers other than its inventor Volvo definitely aided in its uptake, so that's all we can hope for.

Cloudflare has been in the news fairly frequently lately, all with a positive note towards the company. Either they are doing things right or have a really effective marketing department.

It's a mix of both. Cloudflare are trying to position themselves as an indispensable backbone of the internet. It's in their interest to be (and be seen as) benevolent. Besides, securing the internet is in their interest too. So people working at Cloudflare have the freedom to be altruistic up to a point.

Of course for anyone who isn't Cloudflare it's a double-edged sword. On the one hand you get some great free services but on the other hand you might wake up one day to find they're so entrenched that competition is near impossible.

I am both very pleased that CloudFlare is being very gracious and allowing CAs to use it for free, considering what a threat this is, and dismayed at how such a critically important Layer 4 protocol can be used to such effect and how the likelihood of it being updated anytime soon is small.

I'm also wondering how this will be integrated into SSL Labs' test. This definitely seems like a "Grade capped at C" or "Automatic fail" after it has had a chance to be adopted.

Are you saying SSL Lab should conduct some DCV check on their own when doing an automated testing of a 443 site? That seems like it would be very hard since DCV by nature requires some shared secret.

“I’m a little surprised this hasn’t happened yet,” Sullivan said. “We’re hoping that this announcement and this product helps spur the CA/Browser forum to adopt and require this more robust multiperspective validation for certificate authorities. It truly is a risk that hasn’t been exploited yet, and it’s just a matter of time.”

I'm in full agreement with this. Network-hijacking attacks on issuance infrastructure are one of the few remaining weaknesses in TLS, and I'm really surprised it hasn't gotten more attention in the CAB, especially given the number of successful BGP hijacks that have occurred in recent years. (Recent example: https://dyn.com/blog/bgp-hijack-of-amazon-dns-to-steal-crypto-currency/)

To my knowledge no one has yet used a BGP attack to issue fraudulent TLS certs, which might be why this hasn't gotten as much attention as maybe it should have, but I see no reason why it couldn't happen. I suspect the only reason it hasn't yet is that nobody with sufficient resources and motivation has bothered to try. And with the removal of certificate pinning from popular browsers, most sites don't even have a way to proactively protect themselves against this kind of attack anymore. It's a pretty bad situation.

For the http attack against something like the ACME protocol that Letsencrypt uses, am I correct in assuming that the BGP attack would need to hijack both my DNS provided and my own server?

Consequence would depend on what service a hijacked IP is known to provide. If say the IP for your domain’s nameserver is hijacked then it would be trivial to pass ACME DNS-01 challenge. DNSSEC or similar signed DNS records could protect against that perhaps.

Similarly, if your server’s IP is hijacked they can easily pass HTTP-01 challenge.

For the http attack against something like the ACME protocol that Letsencrypt uses, am I correct in assuming that the BGP attack would need to hijack both my DNS provided and my own server?

Consequence would depend on what service a hijacked IP is known to provide. If say the IP for your domain’s nameserver is hijacked then it would be trivial to pass ACME DNS-01 challenge. DNSSEC or similar signed DNS records could protect against that perhaps.

Similarly, if your server’s IP is hijacked they can easily pass HTTP-01 challenge.

Of course both require knowing the challenge secret to publish.

Knowing the challenge token is trivial if the attacker is the one who's requesting certs to be issued. The CA will send them that information by design. It's also not a secret. The challenge token gets published in the clear, either on public DNS (for DNS-01 validation) or on your server (for HTTP-01).

Other than that, you're correct. Hijacking either the DNS _or_ your server's IP would be sufficient for an attacker to get a cert.

DNS hijacking through BGP can be prevented by DNSSEC, but there's AFAIK no similar way of securing the server IP against a similar attack.

Frequently email is the most convenient DCV method. Cloudflare doesn’t say how their service will approach that, but it’s going to suck to respond to multiple emails. Imagine logging in to your bank and getting 3 SMS each with a different code…

This seems more like a solution looking for a problem to me. If a nation state can mount a BGP misdirect wouldn’t it be easier just to pull an inside job on a CA under their jurisdiction?

I'm actually not sure if Cloudflare's system here works with email validation, but if it does then having to deal with multiple emails still seems like a reasonable price to pay for securing the entire TLS PKI against BGP attacks. Especially since you only have to do it once every few months. If that's still too much for you, there are fully automated methods available which are much less hassle to use than email.

Inside jobs are much less viable now thanks to certificate transparency. A BGP-based attack can be carried out _without_ the CA immediately getting blamed for the misissuance and distrusted by all major browsers.

Perhaps I need more coffee, but the impression I get from my first read-through is that if a BGP attack has been executed on a domain, both the original server and the attacker's server will be rejected because the multi-pathing won't line up. This sounds like it could be a good way for hackers to make legitimate servers inaccessible to CloudFlare users by setting a BGP to their server that then redirects to the legitimate server, just to make the pathing not line up.

This particular validation only applies when registering a new certificate. If the original server has an already issued and valid certificate, the BGP attack and validation failure won’t impact the existing certificate already issued for the original server. It would just result in a failure to issue a new certificate to the attacker’s server.

It would prevent new certificates from being registered to the original server, but only if (by bad luck or deliberate timing) they’re trying to renew their certificates at that exact time.

That was what I missed, that it was for registration only, not for checking validation of existing certificates. More coffee has been acquired.

Frequently email is the most convenient DCV method. Cloudflare doesn’t say how their service will approach that, but it’s going to suck to respond to multiple emails. Imagine logging in to your bank and getting 3 SMS each with a different code…

This seems more like a solution looking for a problem to me. If a nation state can mount a BGP misdirect wouldn’t it be easier just to pull an inside job on a CA under their jurisdiction?

It checks if there is a unique directory (name :the string provided by the CA) exists on the target server then tries to read a file containing a unique string in said directory.And that is what the multiple verification does, X different checks on that file, if someone hijacks the website through BGP one or more of those checks will bounce. It does not do something as moronic as sending X e-mails, one from X different data centers.

Also it is not a solution trying to solve the problem of a compromised CA, there are other measures for that. This is solving the problem that someone can use the BGP to redirect traffic.

[edit]Should have written that the multiple checks should have one or more bounces most of the time since it is always possible, though not likely, that all the paths go through the hijacked routes.So this option would make it infeasible to get a cert through BGP hijacking if one has to rely on luck to get it.[/edit]

Frequently email is the most convenient DCV method. Cloudflare doesn’t say how their service will approach that, but it’s going to suck to respond to multiple emails. Imagine logging in to your bank and getting 3 SMS each with a different code…

This seems more like a solution looking for a problem to me. If a nation state can mount a BGP misdirect wouldn’t it be easier just to pull an inside job on a CA under their jurisdiction?

I'm actually not sure if Cloudflare's system here works with email validation, but if it does then having to deal with multiple emails still seems like a reasonable price to pay for securing the entire TLS PKI against BGP attacks. Especially since you only have to do it once every few months. If that's still too much for you, there are fully automated methods available which are much less hassle to use than email.

Inside jobs are much less viable now thanks to certificate transparency. A BGP-based attack can be carried out _without_ the CA immediately getting blamed for the misissuance and distrusted by all major browsers.

I wasn’t thinking right, yeah attacker would already have the challenge secret.

CT is just another layer of defense. Depending on how many valid SCTs a client needs at minimal to pass a certificate I suppose a nation state could mess with a CT log publisher just like they can have inroads with CAs.

Frequently email is the most convenient DCV method. Cloudflare doesn’t say how their service will approach that, but it’s going to suck to respond to multiple emails. Imagine logging in to your bank and getting 3 SMS each with a different code…

This seems more like a solution looking for a problem to me. If a nation state can mount a BGP misdirect wouldn’t it be easier just to pull an inside job on a CA under their jurisdiction?

I'm actually not sure if Cloudflare's system here works with email validation, but if it does then having to deal with multiple emails still seems like a reasonable price to pay for securing the entire TLS PKI against BGP attacks. Especially since you only have to do it once every few months. If that's still too much for you, there are fully automated methods available which are much less hassle to use than email.

Inside jobs are much less viable now thanks to certificate transparency. A BGP-based attack can be carried out _without_ the CA immediately getting blamed for the misissuance and distrusted by all major browsers.

I wasn’t thinking right, yeah attacker would already have the challenge secret.

CT is just another layer of defense. Depending on how many valid SCTs a client needs at minimal to pass a certificate I suppose a nation state could mess with a CT log publisher just like they can have inroads with CAs.

The difference is that CT logs can be audited automatically by browsers and third party monitors. If they sign a SCT and then don't actually include that certificate in the log, it's trivial for a browser to find out about that. So not only would the nation state have to compromise a CA and multiple CT logs (the number of logs required is only going to grow larger in the future, but I think right now Google requires 3); they would _also_ have to ensure that the fraudulent certificate never gets used in an attack against a browser that is capable of noticing the log discrepancy and reporting it.

Nice intention, but unless/until every single trusted CA adopt it, it is pointless, as a malicious agent would still be able to create a cert using that CA and clients don't make more verification than checking if the cert was released by a trusted CA.

Sure, but the same applies to any new security feature.

For example the seatbelt being made available for free to manufacturers other than its inventor Volvo definitely aided in its uptake, so that's all we can hope for.

No, I think Macleone was right. It's not like getting seat belts into more cars, because unlike crashes which are distributed randomly across all drivers, malicious actors will target a certificate authority that's vulnerable to BGP hijacking. So long as one authority remains vulnerable, the bad guys will target that one. They don't need to target the CA of your domain's certificate. They can target any vulnerable CA, because the browser doesn't care if the CA of your certificate suddenly changes, right?

But like the article says, the CA/Browser Forum could mandate multipath validation for all CAs, and then this attack becomes much harder.

Can someone do a BGP attack on the new Cloudflare API and just intercept a little earlier in the process?

The difference is that domain control validation is typically done using unencrypted protocols (HTTP, SMTP) as the domain being verified often doesn't have certificates yet to use for HTTPS.

Cloudflare API however, is provided over HTTPS, so in addition to performing a BGP attack to redirect the traffic, they would also need to obtain invalidly issued certificates for Cloudflare's API domain. Furthermore, I would hope that CAs using this API would use certificate pinning and the whole nine-yards to avoid those sorts of attacks.

Can someone do a BGP attack on the new Cloudflare API and just intercept a little earlier in the process?

The point here is visibility. Malicious actors would have to BGP hijack a large percentage of Cloudflare's network ranges/Autonomous Systems (AS). That kind of move requires huge bandwidth at the destination and it's highly visible.

Contrast that to hijacking the network range of a smallish DNS server.

I think he's looking at a slightly different attack point than you are:

In this situation, all the attacker has to do is BGP poison the zone the API calls are going to. So it doesn't matter what the results of CloudFlare's service are... the attacker fakes the API and returns a valid result to the certificate authority.

However, I can see ways this can be avoided: if the API requires a pre-registered key to use it, any MITM attack on the API will cause the API handshake to fail, alerting both sides that a third party is mucking about. It will be interesting to see how CloudFlare actually implements this.

Nice intention, but unless/until every single trusted CA adopt it, it is pointless, as a malicious agent would still be able to create a cert using that CA and clients don't make more verification than checking if the cert was released by a trusted CA.

Sure, but the same applies to any new security feature.

For example the seatbelt being made available for free to manufacturers other than its inventor Volvo definitely aided in its uptake, so that's all we can hope for.

No, I think Macleone was right. It's not like getting seat belts into more cars, because unlike crashes which are distributed randomly across all drivers, malicious actors will target a certificate authority that's vulnerable to BGP hijacking. So long as one authority remains vulnerable, the bad guys will target that one. They don't need to target the CA of your domain's certificate. They can target any vulnerable CA, because the browser doesn't care if the CA of your certificate suddenly changes, right?

But like the article says, the CA/Browser Forum could mandate multipath validation for all CAs, and then this attack becomes much harder.

This can be used in conjunction with CAA records which CAs are supposed to check before issuing certificates. So if I use CAA to restrict certificates from an issuer that uses multipath verification and non multipath issuers honor CAA records I should still be safe. And if they are compromised, certificate transparency should help detect that.

Our CA system is leaking all over. No single fix is going to remedy that, but every improvement reduces the window for abuse.

Are you saying SSL Lab should conduct some DCV check on their own when doing an automated testing of a 443 site? That seems like it would be very hard since DCV by nature requires some shared secret.

Well, this is a new service, so there will inevitably be some improvements over time, but there are plenty of ways that you could check for a distributed validation. For instance, it could be an additional field in the X.509 certificate itself to indicate the cert was obtained after a distributed validation check. Or the certificate transparency log could be extended to include that data.

SSL Labs wouldn't do the domain validation itself; it doesn't do that today, after all. But if this becomes a mainstream way to issue certificates, it would definitely be a useful piece of data when scoring the security of a website's TLS connection.

Can someone do a BGP attack on the new Cloudflare API and just intercept a little earlier in the process?

The API itself would run under https, so harder to MITM through BGP. Though Cloudflare should use some form of certificate pinning to prevent the attacker to use BPG to obtain a fake Cloudfare certificate.

I commend this and I use their services, but let’s remember that this is the same group that allows “flexible” ssl that shows an ssl cert to end users and then passes this “secure” data as unencrypted data from their servers to the origin servers.

That is very very bad.

Criticism of this to their CEO Matthew Prince has been met with the response that it is better than nothing which is very scary - either he doesn’t understand that a false sense of security is worse than no security or he is complacent in creating a false sense of security.

I commend this and I use their services, but let’s remember that this is the same group that allows “flexible” ssl that shows an ssl cert to end users and then passes this “secure” data as unencrypted data from their servers to the origin servers.

That is very very bad.

Criticism of this to their CEO Matthew Prince has been met with the response that it is better than nothing which is very scary - either he doesn’t understand that a false sense of security is worse than no security or he is complacent in creating a false sense of security.

They do some amazing things but "Timeo Danaos et dona ferentes.”

This is utterly ridiculous.

First off, encrypting the connection from the client machine into the Cloudflare datacenters is a meaningful, real improvement on security. For the vast majority of the people on the planet, the biggest threats are right at the edges of the network; malicious hotspots, firewalls, local MiTM.

It is absolutely more secure to encrypt all the way to the origin server, but that doesn't mean there's no value in encrypting up to the CDN.

In addition, their flexible SSL has multiple, easy to implement security models that allow end to end encryption, including using their own origin certificates which allow the implementation of a fully encrypted connection without having to manage certificate rotation or incurring cost.

Maybe this all seems trivial because, hey, Let's Encrypt is free and has management tools, but the reality is that some people can't/won't do that. Giving someone a tool to encrypt to the CDN is still better than not doing it, and the people who are leaving the backside of that connection unsecured are the same people who would be otherwise leaving the frontside unsecured.

I commend this and I use their services, but let’s remember that this is the same group that allows “flexible” ssl that shows an ssl cert to end users and then passes this “secure” data as unencrypted data from their servers to the origin servers.

That is very very bad.

Criticism of this to their CEO Matthew Prince has been met with the response that it is better than nothing which is very scary - either he doesn’t understand that a false sense of security is worse than no security or he is complacent in creating a false sense of security.

They do some amazing things but "Timeo Danaos et dona ferentes.”

This is utterly ridiculous.

First off, encrypting the connection from the client machine into the Cloudflare datacenters is a meaningful, real improvement on security. For the vast majority of the people on the planet, the biggest threats are right at the edges of the network; malicious hotspots, firewalls, local MiTM.

It is absolutely more secure to encrypt all the way to the origin server, but that doesn't mean there's no value in encrypting up to the CDN.

In addition, their flexible SSL has multiple, easy to implement security models that allow end to end encryption, including using their own origin certificates which allow the implementation of a fully encrypted connection without having to manage certificate rotation or incurring cost.

Maybe this all seems trivial because, hey, Let's Encrypt is free and has management tools, but the reality is that some people can't/won't do that. Giving someone a tool to encrypt to the CDN is still better than not doing it, and the people who are leaving the backside of that connection unsecured are the same people who would be otherwise leaving the frontside unsecured.

No. That is absolutely not true. If I get a valid ssl certificate I expect if I enter a credit card number it is sent via ssl all the way to the origin, not to some data center then converted to plain text and sent through a bunch of random servers for people to intercept. The idea that it is somehow more secure is ludicrous.

No. That is absolutely not true. If I get a valid ssl certificate I expect if I enter a credit card number it is sent via ssl all the way to the origin, not to some data center then converted to plain text and sent through a bunch of random servers for people to intercept. The idea that it is somehow more secure is ludicrous.

You already have no idea how this data is being handled. You have no idea where any TLS socket truncates. Reverse proxies and CDNs exist everywhere.