Security devices have been updated after warnings they could expose TOR traffic.

Cyberoam, a maker of appliances designed to secure sensitive networks, said it has issued an update to fix a flaw that was reported by members of TOR anonymity network.

Cyberoam issued the hotfix on Monday to a variety of its unified threat management tools. The devices, which are used to inspect individual packets entering or exiting an organization's network, previously used the same cryptographic certificate. Researchers with the TOR network recently reported the flaw and said it caused a user to seek a fake certificate for thetorproject.org when one of the DPI (or deep packet inspection) devices was being used to monitor his connection.

"Examination of a certificate chain generated by a Cyberoam DPI device shows that all such devices share the same CA certificate and hence the same private key," TOR researcher Runa A. Sandvik wrote in a blog post published last Tuesday. "It is therefore possible to intercept traffic from any victim of a Cyberoam device with any other Cyberoam device—or to extract the key from the device and import it into other DPI devices, and use those for interception." Someone commenting on the post went on to publish the purported private key used by the Cyberoam certificate.

Monday's post announcing the Cyberoam hotfix suggested competing DPI devices may contain the same vulnerability. "We, at Cyberoam, do understand the critical nature of this issue though we have been singled out and have been put into a situation that requires us to react urgently, keeping our customers' best interest in mind," the post stated. "We think that the industry needs to react to this on an urgent basis so that a deeper crisis is averted." The post didn't name specific devices or manufacturers who may also be vulnerable.

In a post published last week, Cyberoam officials said: "Cyberoam UTM either accepts or rejects, but does not store HTTPS Deep Scan Inspection data, as processing is done in real-time. The possibility of data interception between any two Cyberoam appliances is hence nullified."

Monday's over-the-air fix forcefully generates a unique certificate on each UTM appliance. Customers will know the device has been successfully updated when it displays a "positive alert."

Article updated on July 17, 2012, to correct details about the vulnerability.

Promoted Comments

I don't understand, I thought the idea of deep packet inspection was to catch/stop stuff like TOR.

This. I'm rather confused about this is well; it seems more like this is a weakness of TOR? Some clarification on this would be helpful.

The only way a DPI appliance is able to inspect the cleartext of SSL traffic is by terminating the SSL session (and thus decrypting the data) from the server to the appliance and then re-encrypting the data in a brand-new SSL session between the appliance and the end-user. Due to the nature of SSL, the new SSL session that is created between the appliance and the end-user cannot use the same SSL certificate that the original server used, it must instead use its own "faked" certificate with its own private key. Normally this is going to throw up an error on the client side, because this is literally a "man-in-the-middle attack", unless the client has been pre-configured to trust the DPI appliance's certificate, which is usually the case in an enterprise environment where most of these DPI appliances are getting used.

The problem here is that all Cyberroam appliances were using the SAME "FAKE" CERTIFICATE shipped from the factory. This means anybody who gets their hands on a Cyberroam appliance (or even just a firmware image) has access to the same private key that is encrypting every other Cyberoam customer's data that has been inspected and re-encrypted by the DPI appliance. It would be a very simple matter to sniff all of that re-encrypted data and decrypt it because you've got the private key. What a stupid, bone-headed mistake by Cyberoam. I'm curious now what other vendors have made the same mistake.

Rule #1 for Security Appliance Self-Security -- Don't ship your shit with the same private keys or passwords as ANY OTHER SYSTEM YOU ARE SHIPPING! Ship it with unique keys and passwords across the board. Unfortunately, VERY few vendors get this right. One of the few examples of "doing it right" that I see every day is with HP iLO, where every server is shipped with a unique default iLO password that is printed on a removable tag attached to the server, and unique SSL and SSH keys are generated on first-use. I wish every appliance took this approach.

Edit: Also, you DO NOT need to have another Cyberoam appliance to take advantage of this vulnerability. The simple fact of the matter is you've got a whole mess of SSL encrypted data leaving the DPI appliance encrypted with ONE private key. If you have your hands on that private key then you can decrypt all of that SSL data with your tool of choice.

The major problem with what you describe is that the easily decrypted data will only be found on the LAN side of the device. Sniffing would not be possible outside of the LAN as that uses the original cert of the site / service visited (ie on the internet side) and would require either physical access or some kind of malware infecting one of the PCs on the LAN.

The major problem with what you describe is that the easily decrypted data will only be found on the LAN side of the device. Sniffing would not be possible outside of the LAN as that uses the original cert of the site / service visited (ie on the internet side) and would require either physical access or some kind of malware infecting one of the PCs on the LAN.

Correct. And that IS a major problem for some people. Any malicious user on the LAN side could decrypt every other LAN user's SSL traffic which was being terminated by the DPI appliance. Hey, let's read the boss's emails! Corporate espionage, anyone? Also, as you mentioned, malware... the malicious user wouldn't even have to be on the LAN as long as he could infect a machine on the LAN with some packet-sniffing malware.

This is a really amateur mistake to have made for what's supposedly a seasoned security company and I'm surprised it has taken this long for anyone to blow the whistle on the SSL private key issue. People with the know-how and influence in purchasing decisions for the types of appliances Cyberoam sells should likely consider what other poor design choices were made.

How can the Cyberoam device decrypt the SSL contents if the client initiated the session? Wouldn't the device have to be snooping on the outbound connection as well to figure out how to decrypt it?

I'm not much of a SSL guy but it seems that if cryptography is working properly the only two devices with the key to decrypt the traffic would be the endpoints.

Web servers (and other publicly accessible applications) are not immune from man in the middle attacks, which, as mentioned previously is exactly what these and many similar appliances do.

Very simply put, SSL currently (with a man in the middle) works like this:

1) The client (your web browser or other application) asks the server for a certificate2) The server responds with its certificate and the client establishes an SSL session with it - in this case, the "server" is actually the "man in the middle" appliance in question. Since the client has no choice but to either accept the certificate, or close the connection, at this point the appliance has the ability to decrypt communications intended for the public web server.

Normally, this would present a browser warning about an untrusted certificate. However, in corporate environments, the man in the middle certificate is usually installed as trusted on all employee workstations.

3) The client encrypts data sent to the server with the man in the middle's certificate4) The man in the middle establishes a separate SSL connection to the intended web server5) The man in the middle decrypts the data from the web server, and re-encrypts in the separate SSL session it has with the client

The "servers" can decrypt the messages sent to them by using their private keys which as their name implies, should be kept private. If either of the "server" private keys are compromised, then the data can be decrypted for that particular SSL session.

SSL does provide for client authentication on the server side. However, this would involve every person that wants to use any site or application having to send in their personal public key to the site/application in question. I wouldn't hold my breath for that, just by the sheer logistics and horrible privacy concerns involved in positively identifying every web user.

Hacker infects PC behind Cyberoam device that causes encrypted communications to be sent to him as well. (or at a later date, when he sends command etc). He purchases cheap Cyberoam device from ebay and because cert is the same key he can decrypt said communications. This does not just affect TOR, but could compromise any https communication including bank security, credit card security etc.

I don't understand, I thought the idea of deep packet inspection was to catch/stop stuff like TOR.

This. I'm rather confused about this is well; it seems more like this is a weakness of TOR? Some clarification on this would be helpful.

The only way a DPI appliance is able to inspect the cleartext of SSL traffic is by terminating the SSL session (and thus decrypting the data) from the server to the appliance and then re-encrypting the data in a brand-new SSL session between the appliance and the end-user. Due to the nature of SSL, the new SSL session that is created between the appliance and the end-user cannot use the same SSL certificate that the original server used, it must instead use its own "faked" certificate with its own private key. Normally this is going to throw up an error on the client side, because this is literally a "man-in-the-middle attack", unless the client has been pre-configured to trust the DPI appliance's certificate, which is usually the case in an enterprise environment where most of these DPI appliances are getting used.

The problem here is that all Cyberroam appliances were using the SAME "FAKE" CERTIFICATE shipped from the factory. This means anybody who gets their hands on a Cyberroam appliance (or even just a firmware image) has access to the same private key that is encrypting every other Cyberoam customer's data that has been inspected and re-encrypted by the DPI appliance. It would be a very simple matter to sniff all of that re-encrypted data and decrypt it because you've got the private key. What a stupid, bone-headed mistake by Cyberoam. I'm curious now what other vendors have made the same mistake.

Rule #1 for Security Appliance Self-Security -- Don't ship your shit with the same private keys or passwords as ANY OTHER SYSTEM YOU ARE SHIPPING! Ship it with unique keys and passwords across the board. Unfortunately, VERY few vendors get this right. One of the few examples of "doing it right" that I see every day is with HP iLO, where every server is shipped with a unique default iLO password that is printed on a removable tag attached to the server, and unique SSL and SSH keys are generated on first-use. I wish every appliance took this approach.

Edit: Also, you DO NOT need to have another Cyberoam appliance to take advantage of this vulnerability. The simple fact of the matter is you've got a whole mess of SSL encrypted data leaving the DPI appliance encrypted with ONE private key. If you have your hands on that private key then you can decrypt all of that SSL data with your tool of choice.

I don't understand, I thought the idea of deep packet inspection was to catch/stop stuff like TOR.

This. I'm rather confused about this is well; it seems more like this is a weakness of TOR? Some clarification on this would be helpful.

The only way a DPI appliance is able to inspect the cleartext of SSL traffic is by terminating the SSL session (and thus decrypting the data) from the server to the appliance and then re-encrypting the data in a brand-new SSL session between the appliance and the end-user. Due to the nature of SSL, the new SSL session that is created between the appliance and the end-user cannot use the same SSL certificate that the original server used, it must instead use its own "faked" certificate with its own private key. Normally this is going to throw up an error on the client side, because this is literally a "man-in-the-middle attack", unless the client has been pre-configured to trust the DPI appliance's certificate, which is usually the case in an enterprise environment where most of these DPI appliances are getting used.

The problem here is that all Cyberroam appliances were using the SAME "FAKE" CERTIFICATE shipped from the factory. This means anybody who gets their hands on a Cyberroam appliance (or even just a firmware image) has access to the same private key that is encrypting every other Cyberoam customer's data that has been inspected and re-encrypted by the DPI appliance. It would be a very simple matter to sniff all of that re-encrypted data and decrypt it because you've got the private key. What a stupid, bone-headed mistake by Cyberoam. I'm curious now what other vendors have made the same mistake.

Rule #1 for Security Appliance Self-Security -- Don't ship your shit with the same private keys or passwords as ANY OTHER SYSTEM YOU ARE SHIPPING! Ship it with unique keys and passwords across the board. Unfortunately, VERY few vendors get this right. One of the few examples of "doing it right" that I see every day is with HP iLO, where every server is shipped with a unique default iLO password that is printed on a removable tag attached to the server, and unique SSL and SSH keys are generated on first-use. I wish every appliance took this approach.

Edit: Also, you DO NOT need to have another Cyberoam appliance to take advantage of this vulnerability. The simple fact of the matter is you've got a whole mess of SSL encrypted data leaving the DPI appliance encrypted with ONE private key. If you have your hands on that private key then you can decrypt all of that SSL data with your tool of choice.

The major problem with what you describe is that the easily decrypted data will only be found on the LAN side of the device. Sniffing would not be possible outside of the LAN as that uses the original cert of the site / service visited (ie on the internet side) and would require either physical access or some kind of malware infecting one of the PCs on the LAN.

The major problem with what you describe is that the easily decrypted data will only be found on the LAN side of the device. Sniffing would not be possible outside of the LAN as that uses the original cert of the site / service visited (ie on the internet side) and would require either physical access or some kind of malware infecting one of the PCs on the LAN.

Correct. And that IS a major problem for some people. Any malicious user on the LAN side could decrypt every other LAN user's SSL traffic which was being terminated by the DPI appliance. Hey, let's read the boss's emails! Corporate espionage, anyone? Also, as you mentioned, malware... the malicious user wouldn't even have to be on the LAN as long as he could infect a machine on the LAN with some packet-sniffing malware.

2. In order to perform that attack a rogue CA cert has to be loaded into the browser / email client or whatever. This cert is given to the organisation by Cyberoam. The organisation then goes and installs it onto every PC they own.

So far so good. This is just your standard SSL MITM attack performed by many organisations. If they are following the rules this is presumably it is in their employment contracts or whatever so everybody knows what is going on. The problem is:

3. Cyberoam uses the same cert for all their devices.

This means that not only the organisation can see it's employees decrypted SSL connections, every noob on the planet that owns a Cyberoam DPI device can silently inspect data flowing through the organisations SSL encrypted streams. Not only is this something the employees might not want, the organisation that bought the Cyberoam is probably going to take a fairly dim view of it as well.

Tor uses TLS to establish its encrypted tunnels. I don't know the protocol, but I assume they have some arrangement that precludes MITM attacks - maybe they get the certs for the nodes they are going to connect to via a secondary channel. In any case because they have to be secure against nation states attempting to perform MITM attacks they would be the first to notice someone using a rogue CA.

This means that not only the organisation can see it's employees decrypted SSL connections, every noob on the planet that owns a Cyberoam DPI device can silently inspect data flowing through the organisations SSL encrypted streams.

Not quite - the only people that can inspect the data would either i) have to be behind the DPI device or ii) be in control of a computer (via malware etc) behind the device.

The internet facing side of the DPI device is no less secure than ordinary https traffic.

The major problem with what you describe is that the easily decrypted data will only be found on the LAN side of the device. Sniffing would not be possible outside of the LAN as that uses the original cert of the site / service visited (ie on the internet side) and would require either physical access or some kind of malware infecting one of the PCs on the LAN.

Correct. And that IS a major problem for some people. Any malicious user on the LAN side could decrypt every other LAN user's SSL traffic which was being terminated by the DPI appliance. Hey, let's read the boss's emails! Corporate espionage, anyone? Also, as you mentioned, malware... the malicious user wouldn't even have to be on the LAN as long as he could infect a machine on the LAN with some packet-sniffing malware.

Yes and No. IF, and this is probably a big if now a days, you are seeing the traffic you can decrypt it but you won't see the traffic unless

A) You are on the same HUB as the actual recipient as a switch will only send traffic destined for something on that port.

or

B) You are on port on a managed hub that has been configured to sniff all traffic.

In any decent modern corporate network both of these situations should be pretty rare.

Hacker infects PC behind Cyberoam device that causes encrypted communications to be sent to him as well. (or at a later date, when he sends command etc). He purchases cheap Cyberoam device from ebay and because cert is the same key he can decrypt said communications. This does not just affect TOR, but could compromise any https communication including bank security, credit card security etc.

Not really a good scenario - if the hacker had control of the end user's PC, he wouldn't have to worry about the encryption - he could capture keystrokes for example. I think the vulnerability would only really be relevant if the hacker somehow gained control of a router in between the target user and the cyberroam device?

I don't understand, I thought the idea of deep packet inspection was to catch/stop stuff like TOR.

This. I'm rather confused about this is well; it seems more like this is a weakness of TOR? Some clarification on this would be helpful.

The only way a DPI appliance is able to inspect the cleartext of SSL traffic is by terminating the SSL session (and thus decrypting the data) from the server to the appliance and then re-encrypting the data in a brand-new SSL session between the appliance and the end-user. Due to the nature of SSL, the new SSL session that is created between the appliance and the end-user cannot use the same SSL certificate that the original server used, it must instead use its own "faked" certificate with its own private key. Normally this is going to throw up an error on the client side, because this is literally a "man-in-the-middle attack", unless the client has been pre-configured to trust the DPI appliance's certificate, which is usually the case in an enterprise environment where most of these DPI appliances are getting used.

The problem here is that all Cyberroam appliances were using the SAME "FAKE" CERTIFICATE shipped from the factory. This means anybody who gets their hands on a Cyberroam appliance (or even just a firmware image) has access to the same private key that is encrypting every other Cyberoam customer's data that has been inspected and re-encrypted by the DPI appliance. It would be a very simple matter to sniff all of that re-encrypted data and decrypt it because you've got the private key. What a stupid, bone-headed mistake by Cyberoam. I'm curious now what other vendors have made the same mistake.

Rule #1 for Security Appliance Self-Security -- Don't ship your shit with the same private keys or passwords as ANY OTHER SYSTEM YOU ARE SHIPPING! Ship it with unique keys and passwords across the board. Unfortunately, VERY few vendors get this right. One of the few examples of "doing it right" that I see every day is with HP iLO, where every server is shipped with a unique default iLO password that is printed on a removable tag attached to the server, and unique SSL and SSH keys are generated on first-use. I wish every appliance took this approach.

Edit: Also, you DO NOT need to have another Cyberoam appliance to take advantage of this vulnerability. The simple fact of the matter is you've got a whole mess of SSL encrypted data leaving the DPI appliance encrypted with ONE private key. If you have your hands on that private key then you can decrypt all of that SSL data with your tool of choice.

The major problem with what you describe is that the easily decrypted data will only be found on the LAN side of the device. Sniffing would not be possible outside of the LAN as that uses the original cert of the site / service visited (ie on the internet side) and would require either physical access or some kind of malware infecting one of the PCs on the LAN.

Governments and/or Telcos (and in many cases, they're the same group) can intercept just as easily anywhere along the route that they control. Many tor users live in places where this is a real concern.

This has nothing to do with TOR: The Cyberoam devices proxy SSL traffic. Someone visting https://www.torproject.org noticed the invalid proxy certificate and reported it. Furthermore, the devices shipped with the same CA keypair on each device, meaning that someone who already knew the private key could decrypt traffic on the LAN side.

This is a really amateur mistake to have made for what's supposedly a seasoned security company and I'm surprised it has taken this long for anyone to blow the whistle on the SSL private key issue. People with the know-how and influence in purchasing decisions for the types of appliances Cyberoam sells should likely consider what other poor design choices were made.

How can the Cyberoam device decrypt the SSL contents if the client initiated the session? Wouldn't the device have to be snooping on the outbound connection as well to figure out how to decrypt it?

I'm not much of a SSL guy but it seems that if cryptography is working properly the only two devices with the key to decrypt the traffic would be the endpoints.

Web servers (and other publicly accessible applications) are not immune from man in the middle attacks, which, as mentioned previously is exactly what these and many similar appliances do.

Very simply put, SSL currently (with a man in the middle) works like this:

1) The client (your web browser or other application) asks the server for a certificate2) The server responds with its certificate and the client establishes an SSL session with it - in this case, the "server" is actually the "man in the middle" appliance in question. Since the client has no choice but to either accept the certificate, or close the connection, at this point the appliance has the ability to decrypt communications intended for the public web server.

Normally, this would present a browser warning about an untrusted certificate. However, in corporate environments, the man in the middle certificate is usually installed as trusted on all employee workstations.

3) The client encrypts data sent to the server with the man in the middle's certificate4) The man in the middle establishes a separate SSL connection to the intended web server5) The man in the middle decrypts the data from the web server, and re-encrypts in the separate SSL session it has with the client

The "servers" can decrypt the messages sent to them by using their private keys which as their name implies, should be kept private. If either of the "server" private keys are compromised, then the data can be decrypted for that particular SSL session.

SSL does provide for client authentication on the server side. However, this would involve every person that wants to use any site or application having to send in their personal public key to the site/application in question. I wouldn't hold my breath for that, just by the sheer logistics and horrible privacy concerns involved in positively identifying every web user.

The major problem with what you describe is that the easily decrypted data will only be found on the LAN side of the device. Sniffing would not be possible outside of the LAN as that uses the original cert of the site / service visited (ie on the internet side) and would require either physical access or some kind of malware infecting one of the PCs on the LAN.

Correct. And that IS a major problem for some people. Any malicious user on the LAN side could decrypt every other LAN user's SSL traffic which was being terminated by the DPI appliance. Hey, let's read the boss's emails! Corporate espionage, anyone? Also, as you mentioned, malware... the malicious user wouldn't even have to be on the LAN as long as he could infect a machine on the LAN with some packet-sniffing malware.

Yes and No. IF, and this is probably a big if now a days, you are seeing the traffic you can decrypt it but you won't see the traffic unless

A) You are on the same HUB as the actual recipient as a switch will only send traffic destined for something on that port.

or

B) You are on port on a managed hub that has been configured to sniff all traffic.

In any decent modern corporate network both of these situations should be pretty rare.

No need. ARP spoofing attacks can disrupt the port routing logic and allow packets to be sent to all the other ports.

This means that not only the organisation can see it's employees decrypted SSL connections, every noob on the planet that owns a Cyberoam DPI device can silently inspect data flowing through the organisations SSL encrypted streams.

Not quite - the only people that can inspect the data would either i) have to be behind the DPI device or ii) be in control of a computer (via malware etc) behind the device.

You are forgeting about laptops. Once they leave the network (e.g. reps connecting to public WiFi) they will no longer be protected by the corperate firewall but will still trust the Root cert of the Cyberoam DPI device which everybody now knows. So anyone could MITM your company laptops anywhere they go....

This is a somewhat low quality article which should have gotten more research or explanation.

The problem is not specific to TOR, only reported by the TOR site operators. It would have been good for the article to explain a bit more why having the same certificate for all devices was bad in general SSL terms.

TIL: The only thing you can trust about technology is that somewhere in the stack somebody will have messed up and wrecked the purpose of the whole thing by some stupid oversight or untested bug. Secure communication setup? Maybe in theory, thanks for building that, sweet spec and all, but in reality probably not so much.

I sometimes wonder how people with security responsibilities feel from this kind of stuff, I'd expect something a bit like a rabbit in a forest full of foxes. You just sit tight, dig your nest deeper and hope as hell they don't smell your blood because if they do you're dead: you house is made of sand and all they have to do is dig harder then you.

Yes and No. IF, and this is probably a big if now a days, you are seeing the traffic you can decrypt it but you won't see the traffic unless

A) You are on the same HUB as the actual recipient as a switch will only send traffic destined for something on that port.

or

B) You are on port on a managed hub that has been configured to sniff all traffic.

In any decent modern corporate network both of these situations should be pretty rare.

A decent modern corporate network is the aggregate product of people who takes threats like this seriously. People who dismiss issues like this are the ones who make networks less secure, because they're probably in the habit of dismissing trivial issue. Combine two or three of those trivial issues and some intruder will build a major issue out of it.

I'll admit, before reading the comments, I was confused. Very confused. And couldn't figure out why DPI being able to read TOR traffic was a BAD thing, from the perspective of the guys making the DPI hardware. Much appreciated for the clarifications guys

Next question is, why are you running through your corporate firewall with TOR in the first place? Is TOR seeing use for securing corporate traffic?

Ummm... if a DPI program can expose TOR traffic, that sounds like a flaw in TOR, not the DPI program. It seems like the DPI program is working as designed.

How much did you actually bother to read before skipping down here to comment? This flaw has nothing at all to do with TOR, it was just a TOR user that noticed the problem and brought it to attention, probably because knowledgeable TOR users are going to be picky about the security of their TLS sessions and will notice when something is awry, as they should be. The flaw in question is most definitely in the DPI appliance.

Adding that the comments are what explained the article for me. This is why I love Ars.

heartburnkid wrote:

Ummm... if a DPI program can expose TOR traffic, that sounds like a flaw in TOR, not the DPI program. It seems like the DPI program is working as designed.

Awesome. Remember this post of yours next time you think your opinion and interpretation is just soooo right, but in reality is probably just ignorant and stupid.

Wow. So I didn't quite understand the article and didn't read all the comments. Sorry about that. No need for you to be such a giant dick about it, especially when you admit you didn't understand it at first either.

Woooooooooow.... this is such a obvious, bone-headed mistake: it just goes to show that most companies don't give a flying fig about their customer's security.

.... the same certificate on every device.... it's like a construction company using the same key/lock in every home they build. It makes me ill just thinking about it.

I can think of a couple of ways this happened. 1) Pointy-haired-Boss (PHB) decided to save some money and cut out a step. Doesn't apply for certificate for each device or doesn't establish a CA . Probably saves 10% on each device. Firmware gets programmed into gate-array without incorporating unique certificate step of the process.

2) Project has been "out-sourced" to a programmer/contract firm that tells them they need to setup CA and PHB decides to ignore contractor and jams in the same certificate.

Adding that the comments are what explained the article for me. This is why I love Ars.

heartburnkid wrote:

Ummm... if a DPI program can expose TOR traffic, that sounds like a flaw in TOR, not the DPI program. It seems like the DPI program is working as designed.

Awesome. Remember this post of yours next time you think your opinion and interpretation is just soooo right, but in reality is probably just ignorant and stupid.

Wow. So I didn't quite understand the article and didn't read all the comments. Sorry about that. No need for you to be such a giant dick about it, especially when you admit you didn't understand it at first either.

The difference is I actually read the article and, recognizing I didn't understand the topic fully, read the whole thread for more information before posting, unlike you, who ignored the thread and rushed to post your ignorance, written in the voice of a teenage girl. Ummm... wow, indeed.