Topics

How to record SSH sessions with OpenSSH servers

Jan 18, 2018
by
Ev Kontsevoy

Recording SSH sessions can be useful for compliance as well as for educational
reasons. Replaying a session recording can be the ultimate answer to the “how
did they do that?” question. This blog post is about implementing SSH session
recording using open source tools.

People who cannot run the Teleport daemon on every server have been asking
us how to record SSH sessions using good old OpenSSH. Even organizations who
would like to be full Teleport users usually have legacy pockets where sshd
cannot be replaced.

TLDR; Starting with version 2.4, Teleport can now be used in a “recording proxy” mode, so you can keep your fleet of OpenSSH servers and have your SSH sessions stored on the Teleport audit server.

Before we dive into how to configure it, let’s take a look at how session
recording usually works in a pure-Teleport installation.

Architecture

In the default Teleport cluster configuration, destination nodes submit SSH
session traffic to the auth server for storage. These recorded sessions can be
replayed later via the tsh play command or in a web browser.

Some Teleport users believe that audit and session recording happen on the
Teleport proxy server. This is not the case because a proxy cannot see the
encrypted traffic, it is encrypted end-to-end, i.e. from an SSH client to an
SSH server/node, see the diagram below:

Starting with Teleport 2.4, it is now possible to configure the Teleport proxy
to enable the “recording proxy mode”. In this mode, the proxy terminates
(decrypts) the SSH connection using the certificate supplied by the client via
SSH agent forwarding and then establishes its own SSH connection to the final
destination server, effectively becoming an authorized “man in the middle”.

This allows the proxy server to forward SSH session data to the auth server to
be recorded, as shown below:

The recording proxy mode, although less secure, was added to allow Teleport
users to enable session recording for OpenSSH servers running sshd, which is
helpful when gradually transitioning large server fleets to Teleport.

We consider the “recording proxy mode” to be less secure for two reasons:

It grants additional privileges to the Teleport proxy. In the default mode,
the proxy stores no secrets and cannot “see” the decrypted data. This makes
a proxy less critical to the security of the overall cluster. If an attacker
gains physical access to a proxy node running in the “recording proxy” mode,
they will be able to see the decrypted traffic and client keys stored in
proxy’s process memory.

Recording proxy mode requires the SSH agent forwarding. Agent forwarding is
required because without it a proxy will not be able to establish the 2nd
connection to a destination node.

How to set it up

To add session recording to your fleet of OpenSSH servers you will have to:

Install a Teleport auth+proxy server. For simplicity of this post we can
assume that a single Teleport process is running both.

Installing Teleport

Teleport is a single binary which can be downloaded from the
Teleport’s download page
or you can build your own from the source. Before starting Teleport, you have
to create /etc/teleport.yaml first as shown below:

auth_service:
# IMPORTANT: this line enables the proxy recording mode:
session_recording: "proxy" # can also be "off" and "on" (default)
# For better security it’s recommended to enable host checking as well,
# this is when the Teleport proxy will verify the identity of the
# nodes. Teleport documentation covers how to issue host certificates,
# but for simplicity of this tutorial we are disabling strict host
# checking here
proxy_checks_host_keys: no
# turn 2FA off to make the tutorial easier to follow
authentication:
second_factor: off

Now you can start Teleport proxy + auth servers. They can be executed by the same binary:

$ teleport start --roles=auth,proxy

With the server running, let’s add a user:

# this line says that a user “joe” can request a certificate for himself but also for ‘root’
$ tctl users add joe joe,root

This will print a URL where Joe can finish creating his Teleport user record.
Teleport will now be able to issue self-expiring SSH certificates for Joe.

Configuring OpenSSH servers

Now we must configure OpenSSH servers to trust users who will be connecting via
a Teleport proxy. We need to provide them with the Teleport public CA key,
which we need to export first:

$ tctl auth export --type=user > teleport-ca.pub

Remove “cert-authority” prefix from the file (so the first line starts with
“ssh-”) and save it to /etc/ssh/teleport-ca.pub on every OpenSSH machine.

Add the following line to /etc/ssh/sshd_config:

TrustedUserCAKeys /etc/ssh/teleport-ca.pub

IMPORTANT: make sure to restart sshd daemon after updating its configuration.

Trying it out

Now, Joe can request a certificate (assuming that Teleport is running on proxy.example.com):

$ tsh login --proxy=proxy.example.com

Remember, the SSH agent forwarding must be enabled for this to work. TSH will
try to load the certificate into the active agent. You can verify if the agent
is active and contains the certificate by executing ssh-add -L.