cryptostorm's community forum

Ξ welcome to cryptostorm's member forums ~ you don't have to be a cryptostorm member to post here Ξ
Ξ any OpenVPN configs found on the forum are likely outdated. For the latest, visit here or GitHub Ξ
Ξ If you're looking for tutorials/guides, check out the new https://cryptostorm.is/#section6 Ξ

I glanced at these last nite, when they were posted. Something was, right away, not right... but I was trying not to really look at this until later this week. However, it kept nagging at me and finally I came back to look more closely.

Tragic.

The zip file included in the original post includes the following files (plus some other stuff):

That nomenclature is surprising. To me anyhow, having fiddled with these three lines of openvpn parameters waaaaaay more hours than I'd like to admit. Here's the exact lines as quoted from an early (beta) version of cryptostorm's Windows conf's:

There's several stories embedded in this little params snippet. First, the way we chose to do our PKI for cryptostorm is not the default suggested by the OpenVPN documentation; hence the reference to the forum thread (pki.cryptostorm.ch) to explain what we're doing, and why. Because in fact, the "clientgeneric.crt" and "clientgeneric.key" are not used in our PKI model. More, they aren't even distributed with the widget, or with any of our other client-side tools. I did generate them, long ago, server-side but they never got used. They're purely vestigial.

Second, the decision to rename those two (purely vestigial) files from the default generated by the easyRSA scripts was debated, heatedly, amoungst our staff. Viciously is really a better description: it got ugly. I wanted them re-named, as it made it clearer that we were not generating individualised client-side private keys; hence "generic." However, renaming them breaks all sorts of stuff in the standard deployments of openvpn, and requires some fiddling with the install to get it settled. Which can be a pain in the ass, across a large deployment. So my decision to start re-naming PKI files that nobody other than us would likely ever notice was.... not popular. This whole kerfuffle was resolved when we realised we could completely remove those "generic" elements of client-side identity validation from our PKI setup. They remained only as vestigial entries in our beta conf's, referring to vestigial files that we never circulated nor used in production.

So that's why the Mullvad naming caught my eye, I think. There's the two client-side files with customised names, but the "ca,crt" file - which is supposed to be the certificate the client uses to validate the server's identity via RSA handshake during the TLS session setup of the control channel of OpenVPN - is left with the default name. My choice, in cryptostorm's pre-launch days, to make a similar eccentric naming selection was unusual... so it was odd to see someone else who had chosen a similar procedure.

But, and here's where tragedy strikes, what happens when we look at those files. Ca.crt is supposed to be a simple ASCII-armoured --certificate textbrick. However, here's what is in the Mullvad ca.crt file:

Yikes. That's the entire server-side certificate as well as the PEM-encoded client-side version of it (the first --CERTIFICATE textblock) used to validate the server. The good news is there's no server key to correspond to that server-side certificate. I checked. By the way, you can verify that the first --CERTIFICATE block is derived from the cert materials above it using this online tool, or from local terminal if you prefer. CN is listed as master.mullvad.net.

So if that's the server-side certificate, plus client-side cert validation block... so what's that second --CERTIFICATE textblock? It's a PEM-encoded version of something..,. but what? It's not related to the server certificate included (incorrectly, btw) in ca.crt.

Here's where things get tragic.

That second --CERTIFICATE textblock is actually the PEM-encoded version of mullvad.crt. That's easy enough to confirm yourself. So... oh no. What does that make mullvad.key? Gold star for those who have already guessed...

It's the server-side private key corresponding to mullvad.crt. Confirm for yourself using this, or via local terminal.

Here's some text that explains what that key is:

Your private key is intended to remain on the server. While we try to make this process as secure as possible by using SSL to encrypt the key when it is sent to the server, for complete security, we recommend that you manually check the modulus of the private key on your server...

That private key, in other words, is NEVER supposed to leave the server that generated it. For it to be disclosed is a massive - tragic - breach of the entire security of the VPN itself, top to bottom.

With that server-side key, a passive attacker has the easiest MiTM attack ever. Using one of several freely-available, well-developed MiTM toolsets, you simply tell the client that you are the "server" it is trying to connect to. You have the cryptographic materials to do so, right in these files. Then you have all of your target's plaintext traffic, running through your cloned "Mullvad" server.

To put this in perspective, the entire attack method of the "heartbleed" bug, this summer, on openvpn-based systems involved the possibility of these server-side key materials getting leaked via heartbleed's memory sloppiness. That was the worst-case risk of heartbleed, for VPN services. Indeed, Mullvad has a blog entry talking about hearbleed, and the risk of disclosing private keys:

Mullvad is distributing those key materials with every installer they ship out.

The best I can guess, whoever did these Mullvad materials just didn't understand - at all - how PKI works. They tried with the default materials generated by easyRSA, and got ca.crt... but something wasn't working. So they re-generated (using a new CN), and renamed the new materials mullvad.crt and mullvad.key. Perhaps they looked at cryptostorm's publicly-published conf's which include my quirky - but never used in production - vestigial naming convention and figured somehow this was worth cloning. Sadly, not only did they not notice we don't use those files, they didn't even clone properly (if that's what happened). Because, somehow, they ended up pasting the PEM-versioned ca.crt of the second set of PKI materials into the ca.crt file from the first ones. What a mess.

This leaves a bit of a challenge for the openvpn daemon to parse, at runtime. You've got a mishmash of PKI materials referenced from within the conf, and included in the installer files... my initial read is that the master.mullvad.net cert will be ignored - it doesn't match the directives in the conf - and the second PEM-block --CERTIFICATE is parsed as matching the mullvad.crt server-side PEM-block in mullvad.crt. What openvpn makes of the server-side key being included in mullvad.key I honestly don't know without further wiresharking...

- - -

Look, everyone makes mistakes. I get that. I've made them, our team at cryptostorm has made them... that's life. One corrects them quickly, works to avoid them in the first place, and generally acknowledges that being in the security industry means mistakes are not a laughing matter. But this stuff, publishing private keys... these aren't just "mistakes." Nor are they naive crypto choices that a more experienced practitioner might rightfully question (and speaking of that, Mullvad fails to specify any cipher suites in their config files, which means rollback version attacks to useless cipher suites is easy to do... although basically unnecessary, since they publish the server key anyway).

These "mistakes" are horrible, sloppy, painful examples of amateurs at work. I'm sorry, but there's no nice way to say that. This isn't some small thing, nor is it a one-off issue (bad day at the office, etc.). These materials are being pumped out to the public day in and day out, even as paying customers of Mullvad trust this "privacy service" to keep them safe from surveillance. This "encryption" is utterly worthless because of the key publication; that Mullvad also supports PPTP - a worthless VPN protocol - is doubly tragic.

Once again, this is a "VPN service" that gets high marks from all sorts of review and tech websites... apparently nobody bothered to actually look at the files being included with the installer. That's also tragic. And indefensible.

We use OpenVPN. We also provide PPTP because some people want it but we strongly recommend against it. Encryption algorithms and key lengths are important but often get way too much attention at the expense of other important but harder to measure things such as leaks and computer security.

I know I sound like I'm being a dick to Mullvad, and I regret that. I've nothing against the guys running this service - hell they are obviously good marketers as they've convinced lots of folks to list them as a solid, reliable, professional privacy service. That, in practice, they can't even get a default openvpn installation running without turning the PKI framework into a dog's breakfast is... tragic.

Cheers,

~ pj

edited to add: a look at the certificate revocation materials included in the installer (crl.pem), confirms that the cert revoked as a result of heartbleed (and mentioned in the Mullvad blog post, above) is master.mullvad.net - the cert that does not have it's private key published in this installer. That's confirmation that the private key & cert materials included are, in fact, in production and not left over from pre-heartbleed. Here's the PEM textblock:

No server key is distributed with the configuration package. The only key present (mullvad.key) is the private key for the client certificate (mullvad.crt). The client certificate is not included in the certificate authority file (ca.crt), as you claim it is.

Judging by the screenshot you simply copied-and-pasted the wrong certificate into the online tool. The client certificate was used instead of the one from the certificate authority file. Of course the client key matches its own certificate.

Even though no actual vulnerability was published, it is still unfortunate that you chose not to practice responsible disclosure. Had your claims been true you would have subjected our users to unnecessary risk by not contacting us. We still haven't received any notice from you.

We are competitors but we are also colleagues fighting against a common threat. When we find potential vulnerabilities (such as OpenVPN's vulnerability to Shellshock or Heartbleed) that affect other VPN services we warn them first if appropriate.

Mullvad wrote:No server key is distributed with the configuration package. The only key present (mullvad.key) is the private key for the client certificate (mullvad.crt). The client certificate is not included in the certificate authority file (ca.crt), as you claim it is.

There are actually three "certificates" in this file. Two are different versions of the revoked pre-heartbleed cert; the third is the PEM-encoded version of the certificate in human-readable form that is included in the file "mullvad.crt."

Let's go through these three "certs" one by one. Here's the PEM text from mullvad.crt:

So there's three separate blocks of PEM-encoded certificate materials included in the installer: two in the file ca.crt (as you can confirm simply by looking at the file), and one in the file mullvad.crt. Two of those PEM-encodings have their equivalent human-readable materials also included in the installer, although it's not clear there's any benefit to having such duplicate versions of the same "certificate" propagated (nor is there a security risk in doing so, to be clear).

The third block of PEM-encoded materials does not have a corresponding non-PEM cert included in the installer.

Those are the raw materials, as distributed in the installer.

Judging by the screenshot you simply copied-and-pasted the wrong certificate into the online tool. The client certificate was used instead of the one from the certificate authority file. Of course the client key matches its own certificate.

Now we have only one PEM-encoded "private key," which is in mullvad.key and is the following:

One of these certificates is, apparently, intended to be cancelled via inclusion in crl.pem. Because the encoding of that PEM-block is nonstandard, it's difficult to confirm which of the three certificates is actually intended to be cancelled as pre-heartbleed.

You state that "the client certificate was used instead of the one from the certificate authority file" - but there's actually two certificates in the ca.crt file (well, three... two are commutative transforms of each other). The private key matches neither of those two certificates, agreed.

Given the mis-mash of cert materials included in this installer, perhaps it's best I simply ask you to explain which cert is supposed to be which: which is cancelled as pre-heartbleed, which is a client-side cert, and which is the public hash of the server-side certs you are using? I've done my best, per the post above, to impute intentionality as to which is supposed to be which... but in the end that is (as I said in the post) a bit of a toss-up because of the surfeit of cert materials - and the improperly PEM'd crl entry - included in the installer.

Even though no actual vulnerability was published, it is still unfortunate that you chose not to practice responsible disclosure. Had your claims been true you would have subjected our users to unnecessary risk by not contacting us. We still haven't received any notice from you.

It is not our experience in regards to disclosure that a full and well-documented publication of a vulnerability is "irresponsible." Indeed, the days of attempts to negotiate corrections of vulns via private appeals to vendors proved to be largely ineffective. Hence the near-universal move to public disclosure of vulns.

An irresponsible approach would have been for us to share these findings only with select parties, or to sell this vuln to a third party. We did neither - we published it, and this enables you to correct it quickly. It also allows us to discuss it publicly, as we're doing here, which is healthy for all.

We are competitors but we are also colleagues fighting against a common threat. When we find potential vulnerabilities (such as OpenVPN's vulnerability to Shellshock or Heartbleed) that affect other VPN services we warn them first if appropriate.

We agree that we have common interests in protecting people from dragnet surveillance, and it is in that spirit that we publish this vuln in the first place. As we've said before - as I've said, personally, before - the only thing we earn when we disclose such things is enemies. It benefits us as a company not at all.

It does benefit the larger community, and this is why we do it.

So, in closing, I'd encourage you to clean up the cert materials in your installer. Remove the chuff, as it has confused the process to the point where materials not intended for disclosure have been disclosed. It also likely resulted in your attempted revocation of pre-hb certificate materials ineffective, unfortunately.

Thank you for taking the time and care to reply, and we hope this has been a constructive dialogue.

I believe the heart of the problem here is the inclusion of excess cert materials in ca.crt; I'm just not clear on why there's two PEM-blocks and one human-readable cert in "ca.crt" - and I can't think of a good reason for it. As I said in the original post, without running this through openvpn for test connects, I am not even sure what the client-side daemon is going to do with these duplicate ASCII-armoured PEMs... does it just try the first one and, if that finds a suitable match with materials provided by the server via TLS handshake, assume that's correct? If the first isn't validated server-side, does it then try the second, stepwise?

This shouldn't be such a chore to untangle. Simply put, ca.crt is supposed to be the public side of the server's identify verification. For example, our ca.crt materials (which we inline in conf's, as is preferred in current builds of openvpn), are as follows:

That a 1:1 mapping between ca.crt and the certificate server-side whose private key us used for TLS validation exists seems utterly fundamental to this process; introducing one-to-many ambiguity in that mapping is the root of what's gone wrong here, as far as I can tell.

...both of the "certificates" included in ca.crt show "valid from" dates of 2009; the first (PEM-encoded) one shows 24 March 2009 as valid-from; the second one shows 23 March 2009.

Despite that, it seems that the first is intended to be a post-hb server certificate... with the second actually being the pre-heartbleed server certificate? Neither is a CSR and both ASCII-identify as proper certificates,..

Perhaps there's a rational structure here that I simply am not grasping; as I said in my original post in this thread, the surfeit of cert materials makes parsing this less clearcut than would be optimal.

Mullvad wrote:Unlike what you claim in your original post. Thus your entire conclusion is false. No private server keys have been leaked.

I'm not as confident in echoing this conclusion as you are, given that it's not clear which of these certificates is being used by openvpn as representing the server!

My intention is not to be tendentious, and as I've said repeatedly perhaps there's some structural explanation here that I have simply failed to impute. But when faced with ca.crt including the following materials, it's not obvious what happens in production:

I'd like to propose that this question be settled by doing some test connects to the network, and dumping verb-9 connection logs as well as pcaps of the session. This will tell us what the sessions are actually doing, what keys are actually being used, and what cipher suites are instantiated.

I do not believe this is going to come clear at a level appropriate for final resolution by only looking at the conflicting cert materials in the installer.

Yes, that's what I've pointed out - a broken server validation process doesn't actually need "PoC" anything; one simply sets up a server-side instance, points the client-side files at it, and confirms that a connection is possible. This is, as I said, less than a 15 minute job... as I am sure you will agree

This process is functionally commutative with an analysis of as-observed TLS mechanics during connections to default server instances. In fact, one could well argue that looking at actual sessions to actual servers is more instructive - or instructive in a different way - than is a PoC that causes sessions to initiate with a spoofed server-side instance.

Or, indeed, one can simply do both... and see what falls out of the tree, as it were. I vote for that one, as it's most informative for all. Agreed?

Which is, in fact, one of the PEM-blocks included in the currently-distributed version of ca.crt. It's the second block, actually. This is, most likely, pre-heartbleed certificate material... but if that's the case, and as you've said in twitter it's not the certificate that is intended to be revoked in the CRL file (leaving the question of which cert is being revoked, or intended to be revoked?), then why is a pre-heartbleed cert being circulated in cr.crt?

Again, I do not know what openvpn will do when presented with multiple cert blocks client-side and asked to validate server identity with them. I do not recall ever reading this use-case in the official documentation, and I've never set up a system in this way to test firsthand. So this reflects ignorance on my part - rather than guess, test.

It is a well-known characteristic of X-509-based frameworks that they can be brittle in production. Indeed, smart people have noted how common these issues are in the wild... and that's with CA-issued certs, not self-generated! Throwing known-erroneous materials at the x509 cert process is a recipe for unexpected, and security-unfriendly, outcomes.

You make the point that my original assertion that the "private key" published in your installer is a confirmed match for the server certificate is not the only way to read the materials in the installer. That point is valid, and I am in agreement with that point. I did, in my original post, acknowledge that the excess volume of cert material in the installer bundle makes interpretation a challenge... perhaps I should have been more forceful in making that point, and acknowledging the ambiguity it inevitably brings forth. In that respect, I apologise for a lack of forcefulness in balancing that ambiguity against my conclusion that the private key is a match for the server-side materials actually being used by network sessions given the connection parameters included in the installer.

However, I've also asked for an explanation of how the multiple-cert reality of the installer framework you are distributing is intended to be parsed by openvpn... and so far I don't think that's been provided. Without that, we are left to guess as to intent - or to test and see what actually happens.

What does seem clear is that pre-heartbleed cert material is being distributed in your installers. Further, the conflicting/contradictory cert materials in ca.crt, as well as the overlapping naming conventions used in both the ca.crt certificates and the (intended) "client" cert in mullvad.crt call into question exactly how this jumble is going to get parsed in actual network sessions. This, too, seems an unambiguous conclusion given the data available.

I suspect that testing, and log analysis, will show much more than further argument about what the installer files are intended to do, or "should" do, or are correctly being read as doing. But, as always, I could certainly be wrong.

It is this kind of heated discussion that helps shake out the thin spots in our understanding of PKI and related concepts. I speak personally, here, in terms of benefitting from a debate/discussion such as this with a knowledgeable person who is willing to make strong arguments. And for this, I am genuinely thankful... I also suspect some folks reading along are benefitting as this dialogue unfolds, even during the rough spots.

Actually good to see stuff getting flushed out - helps everyone be better, as both PJ and Mullvad mentioned up above. Good works, all!

*runs to store for more popcorn*

------------------------My avatar is pretty much what I look like. <-- ...actually true, says pj
WebMonkey, Foilhat, cstorm evangelnomitron.
Twitter: @grazestorm.
For any time sensitive help requests, best to email the fine bots in support@cryptostorm.is or via Bitmessage at BM-NBjJaLNBwWiwZeQF5BMLYqarawbgycwJ

Pattern_Juggled wrote:You make the point that my original assertion that the "private key" published in your installer is a confirmed match for the server certificate is not the only way to read the materials in the installer. That point is valid, and I am in agreement with that point.

We make the point that both your assertion and conclusion were dead wrong and that you spent your day gloating about it here and all over Twitter without even notifying us. Instead you could have taken a few minutes to double check your results, think about their implications or even, you know, test that it actually worked. Any one of those three actions would have prevented this spectacle.

All other threads of the discussion - how to read CRLs, PKI design, why some certificates generated pre-Heartbleed don't need to be revoked - just diverts attention away from your original claims.

Pattern_Juggled wrote:I did, in my original post, acknowledge that the excess volume of cert material in the installer bundle makes interpretation a challenge... perhaps I should have been more forceful in making that point, and acknowledging the ambiguity it inevitably brings forth. In that respect, I apologise for a lack of forcefulness in balancing that ambiguity against my conclusion that the private key is a match for the server-side materials actually being used by network sessions given the connection parameters included in the installer.

This is woefully inadequate. The honourable thing is to make the apology at least as strongly and as publicly as the error.

Mullvad wrote:We make the point that both your assertion and conclusion were dead wrong and that you spent your day gloating about it here and all over Twitter without even notifying us. Instead you could have taken a few minutes to double check your results, think about their implications or even, you know, test that it actually worked. Any one of those three actions would have prevented this spectacle.

We are not in agreement in this regard, and I have invested considerable time in this thread documenting why that is the case.

All other threads of the discussion - how to read CRLs, PKI design, why some certificates generated pre-Heartbleed don't need to be revoked - just diverts attention away from your original claims.

That you fail to see the connection amoungst these inter-related problems with your setup is, in a word, disappointing. That you have refused to provide even a modicum of competent explanation when asked basic questions about these issues is, in a word, instructive.

This is woefully inadequate. The honourable thing is to make the apology at least as strongly and as publicly as the error.

Whilst more than willing to apologises for errors, I am equally unwilling to apologise simply because someone is made uncomfortable due to a lack of available cogent explanation for obvious flaws in security systems.

Finally, that you feel it appropriate to veer towards needlessly sharp words rather than responding to direct questions regarding cryptographic fundamentals is unfortunate.