Host Key Verification for SSH Agents

I upgraded the SSH Slaves plugin to the latest and I see the following warning in Manage Jenkins: SSH Host Key Verifiers are not configured for all SSH slaves on this Jenkins instance. This could leave these slaves open to man-in-the-middle attacks. Update your slave configuration to resolve this

My agents fail to connect to my Jenkins master with one of the following stacktrace:

[04/03/17 10:03:51] [SSH] Opening SSH connection to [AGENT_HOSTNAME]:22.
[04/03/17 10:03:51] [SSH] WARNING: No entry currently exists in the Known Hosts file for this host. Connections will be denied until this new host and its associated key is added to the Known Hosts file.
Key exchange was not finished, connection is closed.
java.io.IOException: There was a problem while connecting to [AGENT_HOSTNAME]:22

[04/04/17 08:45:39] [SSH] Opening SSH connection to [AGENT_HOSTNAME]:22.
[04/04/17 08:45:39] [SSH] WARNING: The SSH key for this host does not match the key required in the connection configuration. Connections will be denied until until the host key matches the configuration key.
Key exchange was not finished, connection is closed.
java.io.IOException: There was a problem while connecting to [AGENT_HOSTNAME]:22

Until now, SSH Agent were launched without using any Host Key verification which was a security concern. The release of SSH Slaves plugin1.15 fixes this by introducing a Host Key Verification strategy to SSH Agents. This new feature is designed to prevent man-in-the-middle attack as explained in the Jenkins Security Advisory 2017-03-20.

Note: The Man-in-the-middle attacks happens when a server pretend to be the remote Host, between you and the server you intend to connect to. In that case you are connecting to the “man-in-the-middle” that can intercept the information you transmit when you try to authenticate and use them to establish connection with the remote host.

Host Keys are stored on the SSH Server under /etc/ssh/ and are used to identify the server (Jenkins agents acts as SSH Servers)

The SSH Client keeps a list of Host keys that it trusts under ~/.ssh/known_hosts(Jenkins master acts as the SSH Client)

The purpose of Host key verification is to ensure that you are connecting to the right remote host - the host you intend to connect to. Host keys are stored in the Known Hosts file (usually under ~/.ssh/known_hosts). When the SSH Client initiates a connection to a remote host, the remote host (SSH Server) sends its Host Key. The SSH Client then goes through this Known Hosts file and looks for existing host keys for this host. If one is found, it checks if the host key send by the remote host matches the Known host key.

The verification can fail for different reason. For more details, have a look at the Troubleshooting section below.

It is recommended to use an Host Key Verification strategy when configuring an SSH Agent. If none is selected - because you just upgraded - you will see the warning SSH Host Key Verifiers are not configured for all SSH slaves on this Jenkins instance. This could leave these slaves open to man-in-the-middle attacks. Update your slave configuration to resolve this.

With this configuration, an authorised Jenkins user with Computer.CONFIGURE need to manually approve/disapprove the remote host key in Jenkins if it has changed. Once approved, the host key will be marked as trusted.

The host key received on the initial connection will be automatically trusted. You have the option to enable Require manual verification of initial connection so that even the first connection requires approval from an authorised Jenkins user.

Example: Approval on initial connection:

Example: Approval after change of remote Host Key:

Note: You need to relaunch the agent manually after approving the host key

A Shared Agent is meant to be provisioned to Client Masters and therefore only connects when provisioned. The shared agent item on the CJOC is simply a definition/template of an agent used to create nodes on the Client Master master that requests it. The connection - and therefore the Host Key verification - only happens between Client Master and Provisioned agent.

In this regards, some strategy might be more appealing and suitable than others:

Know Hosts file Verification Strategy: Requires to add the Host Key of the shared agent manually on each client master’s Known Host File

Manually provided Verification Strategy: No configuration on the master. Provide the Host Key once in the Shared Agent configuration.

Manually trusted Verification Strategy: No configuration on the master. Note that if Require manual verification of initial connection is ticked, every time the node is provisioned an authorised user of the Client Master will have to manually hit Trust SSH Host Key. So this option is not really suitable for Shared Agent.

If the agent logs shows the IOException The server hostkey was not accepted by the verifier callback, then the Host Key verification fails when connecting the agent, and you need to fix it. You can troubleshoot this from master’s machine logged in as the jenkins user with the command:

The SSH Client does not know the host and cannot verify its identity. The entry needs to be added manually to the Known Hosts file.

With strict Host Key verification but wrong/unexpected host key, the connection would fail with the following scary message:

jenkins@ubuntu-1 jenkins:~$ ssh -o UserKnownHostsFile=~/.ssh/known_hosts -o StrictHostKeyChecking=yes -o HostKeyAlgorithms='ssh-rsa,ssh-dss' -i ~/.ssh/jenkins_node_rsa jenkins@10.1.2.5
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:ajWjatHT78IPzOcR9pMG3bMRHVhprv39jCgQMypcMNY.
Please contact your system administrator.
Add correct host key in /Users/jenkins/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/jenkins/.ssh/known_hosts:39
RSA host key for 10.1.2.5 has changed and you have requested strict checking.
Host key verification failed.

This means that the host key sent by the remote host does not match the one expected - i.e. the one stored in the Known Hosts file. There could be different reasons for receiving that message:

It could be a sign of an attack

This could be due to the host key entry being stored using the wrong Key Algorithm

1 Comments

The only SSH slaves we have are Docker containers. But there's no option in the container template settings to select a host key verification strategy. So please provide a method to silence this annoying message.