Debugging SSL transaction level Issues

Recently I was interacting with our support team and SSL based cases still remain the tough ones to crack. It gets tougher if the issue is at transaction level and you cannot see anything in the payload. Nothing comes for free and that holds true when we talk about all the security technologies around us. Getting SSL to encrypt your communication is the best way to stop bad guys from eavesdropping on the communication channel. In today’s world with all the business happening online there is no better way than SSL to protect your communication channel.

So it is given that SSL is what we need and we have to live with the debugging challenges which come with it. SSL communication requires the whole TCP payload to be encrypted beyond the initial handshake messages. Once the handshake is complete with FINISHED message, everything ahead is going to be encrypted on that session. Here are some of the common issues where you need to get inside the wire level details.

SSL Handshake is failing

You got lucky if the issue is as simple as SSL Handshake failure. In most cases the handshake failures will be seen in initial SSL handshake messages. This part of the communication is not encrypted and you can debug it by taking a trace and looking at where the failure has occurred. Most likely you have received one of the following alert messages:

handshake_failure(40)

export_restriction(60)

protocol_version(70)

no_renegotiation(100)

Simple reasons are around not having matching ciphers or one of the parties not sending expected handshake message etc. You can check for the respective counters on NetScaler as well.

Certificate Errors

Certificate errors are technically part of the SSL handshake messages but they would not be seen if you are not doing client authentication. While doing client authentication you may hit different errors like:

bad_certificate(42)

unsupported_certificate(43)

certificate_revoked(44)

certificate_expired(45)

certificate_unknown(46)

unknown_ca(48)

As you can see there can be multiple issues around the certificates and all of these are fatal error which results in termination of the handshake. Again these can be seen in clear text and for all these alerts NetScaler raises internal counters which also tell you the reason to match with what you see in trace.

Bulk processing failure

Here comes the interesting stuff… when we say the failure is in Bulk processing… you are now looking at the actual encrypted payload beyond the initial handshake. There are some error types which are very specific to SSL record layer processing and you would see those failures in the record layer processing while encrypting or decrypting bulk data. For example when the SSL record is being decrypted we find the MAC of the record is not properly computed. Or the whole decryption routine failed because of some issue during the encryption process on other side. At times there can be errors which cannot be put under a category and record layer protocol says that it is an Internal Error. Such errors are even more difficult to debug and analyze. Some of the common alert messages would be:

unexpected_message(10)

bad_record_mac(20)

decryption_failed(21)

record_overflow(22)

illegal_parameter(47)

access_denied(49)

decode_error(50)

decrypt_error(51)

internal_error(80)

The issue gets tougher when you have them cropping up in upper layer protocol like HTTP. We would notice such issue more often where the failure is in HTTP processing or one of the policy modules on NetScaler. In such cases you just cannot understand the reason without decrypting the trace and getting the cleartext data to understand what is going wrong.

Quick tips on decrypting the trace file:

You need to collect the trace on client or NetScaler side

In most cases taking trace on NetScaler is better

Ensure that trace collected captures complete session

Ensure the session is not reused otherwise decryption would fail

You need the server private key file for decryption

Once you have the key file, here is the process (assuming you use wireshark)

Go to Edit -> Preferences

Go to Protocol-> SSL

Do reassemble SSL records across TCP

Do reassemble SSL App Data across records

Specify RSA key as: ServerIP, Port, Protocol, Key-File-Name

“Key-File-Name” is the actual key file which is on the local system

You need to specify complete path of the file on local system

A simple mistake on wireshark would give you real tough time decrypting the trace file. Thus every step is mandatory and needs to be taken very carefully. The issue gets more interesting when you have customer saying that I cannot provide you the private key file. In many cases you will run into such accounts where sharing the key is out of question then how do you get the plaintext data??

Here we get back to our favorite Linux tools and have the trace file decrypted at customer end. On SSL layer the most useful one is “ssldump”. This is command line tool which takes trace file and Key file as an input and generates a decrypted output. This is an excellent tool which provides you very granular packet and protocol level details along with decrypted payload. Now you can get this file without getting the key file and still look through the decrypted data.

🙂 Happy Debugging 🙂

Share

Tagged under:

Director of Product Management for Cloud Networking Group. Looking after NetScaler and ecosystem products and working with global customers. With Citrix from over 9 years in NetScaler group and spent time in various engineering and externally focused roles.