Debug SSH Connection issue in key exchange

Securing a server means hardening the SSH server settings, but doing so can also cause issues with ssh clients. Finding the cipher or algorithm causing a failled connection can be tricky. Depending on the client used, the error message might be very generic like “Failed to start SSH session”.

Using a openssh client will allow increasing the log level (-v) until the cause of the problem shows up in the output. Experience has shown that most issues are not related to openssh client configuration but closed source or third-party applications that provide less details via their user interface. In the case explained here it was an iPad/iPhone app, which had updated its ssh library. The app did not provide full details about the error as space is limited.

Advertisements

From the first error message “Failed to start SSH session: Unable to exchange encryption keys.” it was clear that there was a fundamental issue with the connection setup, but without any details from the client debugging had to be performed on the server side.

To debug the connection issue from the ssh daemon, the following log needs to be monitored on CentOS (other distributions might log to a different file).

$ tail /var/log/secure

In debian based distributions like Ubuntu, the log file for the ssh daemon is the following.

$ tail -f /var/log/auth.log

No matching MAC algorithem

The following line should be under the last lines of the log, indicating what the initial error message already suggested. Client and server could not setup a session as they could not agree on a MAC (message authentication code) algorithm.

But this error message already shows us more detail. It lists the client and server offered MACs. The client offered 6 MACs and the server offered 2, but no MAC is in both lists.

To continue debugging the connection issues the following MAC algorithm was enabled in the ssh daemon configuration, chosen at random from the client offered MAC algorithms. The resulting ssh daemon configuration after applying the settings from hardening the SSH server settings and with the added MAC algorithm looks like this.

After changing the configuration of the ssh daemon, sshd needs to be instructed to reload its configuration.

$ service sshd reload

The next connection attempt shows the agreed MAC algorithm in “hmac-ripemd160” the log clearly.

... sshd[634]: debug2: mac_setup: found hmac-ripemd160

No matching KEX algorithm

Even with the MAC algorithm agreed, the next problem might arise when the KEX (Key EXchange) algorithm can not be negotiated. The situation about the KEX negotiation is indicated very clearly.

... sshd[6260]: fatal: Unable to negotiate a key exchange method

Sadly this message does not provide any details about the proposed KEX algorithms from either side of the connection. To debug the connection on CentOS 6 running the OpenSSH 5.3 daemon, the debug level DEBUG2 is needed. To enable the logging on the ssh server, the following line of configuration has to be added to the /etc/ssh/sshd_config. It will not alter the behaviour of the ssh daemon itself, except to add additional logging to understand the decissions made during the connection setup. This configuration should be used with care on a busy ssh daemon as it increases the amount of logging significantly.

LogLevel DEBUG2

After changing the configuration of the ssh daemon, sshd needs to be instructed to reload its configuration again. The second command will open the log file to follow the login attempt.

$ service sshd reload
$ tail /var/log/secure

With the increased log level, a connection attempt will look similar to this example (here I have removed the beginning of each line where there would normally be a timestamp and the host name).

There is no clear indication on the offered KEX algorithms in the log. To read this information out of the log, a deeper look into where the log lines come from is necessary. What’s interesting at this point are the log lines containing the “debug2: kex_parse_kexinit:”.

Digging into the logs

A quick peek into the function kex_buf2prop() of the file kex.c on branch V_5_3 of OpenSSH reads (shortened slightly) like this.

Without digging deeper, the name of the parameters suggest, first the function is called with the locally configured algorithms and second with the offered algorithms from the remote. In this case, that would indicate that first the proposal from the ssh deamon is logged and the second block contains the proposal from the (remote) connecting client.

As such, the “kex_parse_kexinit” lines can be seperated into the following two blocks. The first one is the ssh daemon side.

For some reason, which would probably require a deeper understanding of the source, the cipher and MAC list is shown twice. The block of “kex_parse_kexinit” lines ends with the line “kex_parse_kexinit: reserved 0”.

The second block is the proposals from the remote side, which is the ssh client trying to connect.

Knowing this, we can see that the KEX algorithms from the ssh daemon and the client’s proposed KEX algorithms do not have any algorithms in common, which causes the error message we saw earlier.

Of course, enabling one of the client’s proposed KEX algorithms would allow the client to login, but enabling one of these MAC or KEX algorithms needs to be handled with care. Many of the disabled ciphers and algorithms might be disabled for security reasons as described in Harden the SSH server settings.

As a proof, one of the client’s proposed KEX algorithms (I randomly chose “diffie-hellman-group-exchange-sha1”) can be enabled in the shh daemon. The configuration might look like this with the additional KEX algorithm.

After changing the configuration of the ssh daemon, sshd needs to be instructed to reload its configuration again.

$ service sshd reload

The next login attempt from the client should succeed as the MAC and KEX algorithms the client proposes are enabled for the ssh daemon.

Modern distributions, different log

The above explanation about interpreting the log information to find the KEX that were offered applies to conservative distributions like CentOS 6 / RHEL 6 (Red Hat Enterprise Linux) and possibly Debian. But what about modern distributions? Modern distributions like Ubuntu and possibly more recent versions of CentOS and RHEL use a more up-to-date version of OpenSSH. At the time of writing Ubuntu uses OpenSSH 7.2 where the logging is more self explanatory.

The above example shows that instead of the “kex_parse_kexinit” log lines, the log identifies the offered algorithms. This makes it much easier to identify the MAC, KEX and ciphers offered from both sides. There is even a line between the blocks identifying the origin of the offer.

This way, guessing and digging in the source to understand the log is not needed any more. Sady not all distributions use the newer version of OpenSSH. With CentOS 6, OpenSSH 5.3 is still the latest, patched version available in the default repositories.