In my recent projects I've had to do a lot with certificates, java and HTTPS with client-side authentication. In most of these projects, either during testing, or setting up a new environment, I've run into various SSL configuration errors that often resulted in a rather uncomprehensive error such as:

javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:352)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:397)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:150)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:575)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)

In most of the cases it was misconfiguration where keystores didn't containt the correct certificates, the certificate chain was incomplete or the client didn't supply a valid certificate. So in the last project I decided to document what was happening and what caused specific errors during the SSL handshake.

In this article I'll show you why specific SSL errors occur, how you can detect them by analyzing the handshake information, and how to solve them. For this I use the following scenario:

Server uses a certificate issued by a CA and requires client authentication. The server uses a simple truststore that lists this CA as trusted.

Client connects using a certificate issued by this single trusted CA and has it's own trustore that also contains this certificate from the server.

Not a very complicated situation, but one you often see. Note that the following information can also be used to identify problems when you don't work with client certificates or use self-signed certificates. The way to determine the problem in those cases, is pretty much the same.

Happy Flow

First we'll look at the happy flow, what happens in the handshake when we use client certificates. We won't look at the complete negotiation phase, but only until both the client and the server have exchanged their certificates and have validated the received certificate. If everything goes well until that point, the rest should work. The following is what you see when you run the client and the server using the java VM parameter: -Djavax.net.debug=ssl:handshake.

The first thing that happens is that the client sends a ClientHello message using the TLS protocol version he supports, a random number and a list of suggested cipher suites and compression methods. From our client this looks like this:

So in this case we're going to use SSL_RSA_WITH_RC4_128_MD5 as Cipher Suite. The next step is also done by the server. The server next sends a Certificate message that contains its complete certificate chain:

In this message you can see that the issuer of this certificate is our example CA. Our client checks to see if this certificate is trusted, which it is in this case. Since we require the client to authenticate itself the server requests a certificate from the client and after that sends a helloDone.

In this message you can see that the server provides a list of Cert Authorities it trusts. The client will use this information to determine if it has a keypair that matches this CA. In our happy flow, it has one and responds with a Certificate message.

This certificate is checked on the server side and if all is well, the final steps in the handshake are executed to setup the secured connection. Note that there is a CertificateVerify step. In this step the client signs a message with its private key. This is done so the server can verify the client has access to its private key. This might seem a step where things can go wrong in an incorrectly configured environment. In the default java implementation this won't happen. In the phase where the client has to determine which certificate to present to the server, the java implementation already checks if the privatekey is available.

What could possibly go wrong

So what could possibly go wrong in this handshake? In the next couple of sections we'll look at some scenarios, and how to detect them.

Passwords

Now that we've seen what happens when things go right, lets look at a couple of scenarios where things go wrong. We'll start simple with the following exception, that we get at the moment we start up the client application:

Exception in thread "main" java.security.UnrecoverableKeyException: Cannot recover key at sun.security.provider.KeyProtector.recover(KeyProtector.java:311) at sun.security.provider.JavaKeyStore.engineGetKey(JavaKeyStore.java:121) at sun.security.provider.JavaKeyStore$JKS.engineGetKey(JavaKeyStore.java:38) at java.security.KeyStore.getKey(KeyStore.java:763) at com.sun.net.ssl.internal.ssl.SunX509KeyManagerImpl.(SunX509KeyManagerImpl.java:113) at com.sun.net.ssl.internal.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:48) at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:239) at org.apache.http.conn.ssl.SSLSocketFactory.createSSLContext(SSLSocketFactory.java:186) at org.apache.http.conn.ssl.SSLSocketFactory.(SSLSocketFactory.java:260)

This very helpful message is thrown when (from the javadoc) " .. a key in the keystore cannot be recovered". There are a couple of reasons this can happen, but normally this occurs when the key in the keystore is accessed with the wrong password. Usually when you use the keytool to create and manage your keys, the keystore password is usually the same as the key password. However, if you import keys from a PKCS#12 type keystore, the password of the keystore can be easily set to a different value. Not all the SSL client allow you to specify a different password for the key and the keystore. If that is the case you can use the following command, to change the password of the key:

keytool -keypasswd -alias <keyalias> -keystore <keystore>

It is also possible to set an incorrect password for the keystore. Luckily in that case the error message that is thrown is much more helpful:

If this occurs at the server side, we can see the same message when the SSL listener is being set up.

Incomplete CA Chains

Now lets look at the first of the "peer not authenticated" exceptions. In the logging we see this exception at the client side:

javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:352)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:397)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:150)

So enable SSL logging, run again, and we'll start with analyzing the handshake. We'll start by looking from the client side. If we look through the logging we find the following CertificateRequest message from the server and the ServerHelloDone.

So thus far, everything went ok. The server has already sent its certificate, and since our client doesn't throw an error on that part, we can assume it is trusted by the client. So something seems to be wrong with the steps that come after this message from the server. If you look closer at this message, you can see that the server doesn't specify a set of Cert Authorities it trusts. This could be a misconfiguration at the server side, or it could just be that the server expects one of the trusted Root CAs. In any case, the client is free to send any certificate he wants. So the client sends the following certificate:

You can see that we received the certificate from the client, and directly after that we get this error. This error however doesn't really tell us anything. We do however have enough information to at least limit the possible errors. We know that the server didn't sent a list of CAs, we can see that the client sent a valid certificate, and that server somehow isn't able to process it. It looks like a problem with the server truststore. In this case the best approach is to look at the certificates the server trusts. Either in the cacerts file or in it's own truststore. Validate whether the CA certificate our client sends is in the server's truststore, and the server actually loads the stores we expect.

It's of course also possible that the client has an incomplete chain of trust for the certificate received from the server. In that case we once again get the "peer not authenticated" error at the client side. If we look at the SSL debug logging, we see the following exception occuring at the client side:

Following the same reasoning as for the server we can conclude that there is something wrong with the client side truststore. For completeness sake, the server receives this error message when this situation occurs at the client:

javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:352)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:397)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:150)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:575)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)

When you receive an internal error, there is usually something wrong at the server side. So looking at the serverside, lets see what caused this error.

Hmm.. somewhat more useful. It seems that there is something wrong with the algorithm we used, the client seems to have provided an incorrect certificate. But what is wrong? If you look back at the happy flow, you can send that at a certain time the server asks the client for a certificate using a "Certificate" message. Lets look a bit closer at this message and the response:

What you can see here is that the server specifies the cert types it accepts, and the authorities it accepts. The client responses in this case however with a DSA public key. Depending on the server implementation this can cause this strange message. Another possible scenario I've seen (especially with self-signed certificates) is that with a "CertificateRequest" message like this:

This client won't respond with a certificate at all, if you only have DSA based keys in your keystore. It won't throw an error on the client side, but will cause a "null certificate chain" message as the server side. I haven't seen this scenario, though, when you don't use self-signed certificates.

Certificate expiration

So far we've seen how you can analyze the SSL handshake to determine where to look for configuration errors. In this last example we'll look at what happens when a certificate expires. In this case we once again see the very cryptic message at the client side:

If we look at the phase of the SSL handshake we're in, we can see that we've already sent our client certificate and finishing up the handshake when we receive this error. The error on the serverside is actually pretty helpful. After receiving the invalid certificate, in the debug logging, it shows us the following:

It tells us that during the validation of the certificate, a timestamp check failed. This tells us that we should look at the validity of the certificates in our certificate chain to see what is happening.

Summary

In this article you've seen a couple of common causes for SSL exceptions and ways to identify the exception. Their can be many causes for these kind of exceptions, the most common though are the following:

Incorrect certificate chains in the client truststore

Incorrect certificate chains in the server truststore

Invalid key algorithm used for private keys

Expired certificate or expired CA certificate

Incorrect passwords used to access the keys

Multiple private keys to choose from

If you're presented with a such an exception a good general approach is this. You first check the keystores that are involved. Use the java keytool for this:

keytool -list -v -keystore <location_of_keystore>

This will print out all the certificates and keys in the keystore. Check whether the keys are of a supported type, the required CA certificates are stored and that your application is using the correct one (spent hours figuring out an issue because I was looking into a truststore for my private key). If everything seems to be OK at first glance it's time to enable ssl debugging (-Djavax.net.debug=ssl:handshake) and check the handshake messages that are sent. Wikipedia has a nice overview of which message is sent at a specific time. For more information on the content of the messages look at the RFC 5246 (or the one of the SSL/TLS version you're using, but the handshake changes are minimal between versions). Using the messages and the handshake, determine at what place in the handshake things go wrong, taking into account that the client will continue with the handshake, while the server is processing it's certificate.