Trisul Network Analytics Blog

Tech tips and tricks from the world of network traffic and security monitoring

Secure Shell (SSH) is a ubiquitous protocol used everywhere for logins, file transfers, and to execute remote commands. In this article, we are looking to use passive traffic analysis to detect various SSH events like login, keypress, and presence of SSH tunnels. Lets start with a question.

What do you see on the wire when you press a key in a SSH terminal?

You could observe anywhere between 28 and 96 bytes of SSH payload depending on the type of secure channel negotiated. Lets dive a little deeper to see if we can get a accurate answer.

TLDR Use traffic analysis to detect successful login , keystrokes, and Tunnels – reverse and forward. The approach is to use knowledge of the ciphers and MAC used in SSH and calculate the SSH message lengths on the wire. For login detection, we use the Terminal Capabilties Exchange , there are only a handful of terminal types so the message is predictable.

The basics : MACs and Ciphers

The SSH protocol offers both encryption and message integrity. Each packet is encrypted using a Cipher and authenticated using a MAC. If you capture packets using a tool like Wireshark, this is what a SSH record would look like. (without the TCP/IP headers)

The general flow of the SSH protocol is

The client and the server first exchange packets and agree on the MAC and Cipher algorithms. This is followed by a key exchange usually Diffie Hellman. This negotiation is unencrypted. In the real world, both directions use the same ciphers and MACs even though the SSH protocol itself does not mandate it.

Ciphers

Ciphers are used to encrypt your payload. Type ssh -Q cipher to get a list of supported ciphers by your client. You would see something like below. The comments are added by me

AEAD vs Plain Ciphers — the important classes of ciphers today are the plain ciphers vs AEAD ciphers. The AES128-CTR is a plain cipher and the AES128-GCM is an example of an AEAD cipher. The AEAD ciphers generate an authentication tag in addition to encrypting. The cipher families used widely today in ssh are aes* , the *gcm, and chacha20-poly1305. ChaCha20-Poly1305 is an AEAD cipher like AES128-GCM but it is handled differently in SSH as we will see later. When SSH negotiates an AEAD cipher ( -gcm or chacha20-poly1305) the negotiated MAC is completely ignored.

MACs

The MAC algorithms authenticate the SSH message. Type ssh -Q mac to get a list of supported MACs

ETM vs non-ETM MACs — ETM stands for “Encrypt then MAC”. This represents a break from the older SSH which used “MAC then Encrypt”. Whether SSH negotiates an ETM or non-ETMMAC has a rather big implication for our traffic analysis.

The cryptographic doom principle and the SSH -etm MACs

The older non-ETM MACs like hmac-md5 first computed the MAC on the unencrypted SSH payload and then encrypted the message. This was MAC then Encrypt. I am not an expert on this and I can only guess why the initial SSH developers did this. The reason might have been to encrypt the packet length to thwart traffic analysis. Even today when you use hmac-sha2-256 you have to first decrypt the packet and then get the packet length. This means that you have to run a decryption operation on an unverified message.

Touching an unverified message is frowned upon by crypto experts

“if you have to perform any cryptographic operation before verifying the MAC on a message you’ve received, it will somehow inevitably lead to doom.” — Moxie Marlinspike

OpenSSH now defaults to the *-etm ciphers. These first encrypt the payload then apply the MAC. The flip side is that now you need to transmit the packet length in the clear. Why? Because the receiver needs to check the MAC first before decryption. To do this you need to know the total length of the packet so you can find the MAC ! This exposes the protocol to traffic analysis but avoids the ‘moxie crypto doom principle’.

A single keystroke on the wire

Getting back to the traffic analysis of a single keystroke. We have the following so far

the handshake is unencrypted, so we can synchronize the Cipher/MAC state with the SSH session

the AEAD ciphers dont use a separate MAC

the *etm MACs transmit unencrypted packet lengths

the ciphers negotiated arent very important from a traffic analysis perspective as most of them are stream ciphers with a 16-byte block length except ChaCha20 which is 8 bytes

So lets jump in and see how this looks on the wire with a concrete example.

Say you hit a Space Character ASCII 32 (Hex 20) in a SSH session that has negotiated an aes128-ctr cipher. Per the SSH protocol RFC 4253 the encapsulation for carrying this single byte is the structure SSH_MSG_CHANNEL_DATA

byte SSH_MSG_CHANNEL_DATA
uint32 recipient channel
string data

and the string datatype is a 4-byte length followed by the raw string data. The full encapsulation would then look like this

To get it to a 16byte boundary you need a padding of just 1 byte. But the "SSH Transport Layer RFC 4253 ":https://www.ietf.org/rfc/rfc4253.txt states

must pad to a multiple of cipher block size or 8 whichever is larger. Here the AES128-CTR block size is 16 bytes

must have a minimum pad of 4 bytes

So, we cant just pad 1 byte, we must now add 17 bytes of padding to bring it up to 32 bytes which is the next multiple of the cipher block size. The full packet looks like below.

Again note that the packet length itself is encrypted. This thwarts passive reconstruction of SSHPDU (Messages) from a TCP byte stream. This is more resistant to traffic analysis but it breaks the Moxie Cryptographic Doom Principle by forcing the receiver to do a decrypt operation on unverified message. Thats where the ETM MACs come in.

sha1 ETM – 40 bytes per keystroke

By using an ETMMAC you not only get better security but the packet length drops from 52 to 40 bytes. Carrying on from the previous example. When you use an ETMMAC the packet length isnt encrypted. The message to be encrypted is

Now you need to add 5 bytes of padding to bring the total message up to the AES128-CTR cipher block length of 16. The good news here is that 5 bytes of padding is more than the mandated minimum of 4 bytes. The total length on the wire is 40 bytes as shown below

Here the packet length is unencypted. This allows traffic analysis tools like Trisul to reconstruct the SSH PDUs from a TCP byte stream.

chacha20-poly1305 – 36 with encrypted length

This particular cipher is currently zooming in popularity due to Chrome’s adoption. It is an AEAD cipher that is apparently computationally very efficient and it is as secure as AES. It is also the default cipher for SSH in latest versions of Ubuntu when I checked. ChaCha20-Poly1305 is worth a closer look because OpenSSH treats this cipher differently.

On the wire ChaCha20 is exactly like the ETM case, the packet length is not part of the encrypted message. The message is :

Now the block size of ChaCha20-Poly1305 is 8 bytes – so you need 5 bytes of padding to bring it up to 16 bytes. To that, add the 16 byte AEAD token created by Poly1305. You end up with total message size of 36 bytes. At this point the OpenSSH implementation adds a twist.

The chacha20 packet length field is also encrypted. They use a separate instance of ChaCha20 for this. I suppose this doesnt break the aforementioned “Moxie Doom” principle because the length isnt authenticated. It seems to me like the only reason OpenSSH does this is to thwart traffic analysis. The other AEAD ciphers AES-128-GCM do not encrypt the packet length. This how a single keystroke on a ChaCha20-Poly1305 SSH session looks like on the wire.

Detect a successful login

A naive way to detect a successful login is to alert when a total traffic exchanged on a SSH flow is greater than a threshold value. We currently use that with Trisul and the results are hit and miss. Few factors that can confuse us:

Use of login banners with variable length strings

Pub key authentication involve exchanging more data

Turns out there is a much better way to detect login. After a successful password or key authentication, the client does two things in succession

sends a SSH_MSG_CHANNEL_REQUEST —> requests a pseudo terminal

sends a SSH_MSG_CHANNEL_REQUEST —> requests a shell

It turns out there arent that many different types of terminals so these two messages are mostly of fixed length.

The only variable part of this message is the “encoded terminal modes”. There arent that many types of terminals so this part is mostly fixed. This message is 312 bytes for all linux based terminals we tested and 288 bytes for PUTTY versions. The difference is due to missing terminal modes in putty.

In a similar manner you can construct a table for each of the ciphers and MACs. See ssh_dissect.lua on our Github page.

Detecting keystroke after login

As we saw a single keystroke results in 10 byte payload, which can be easily translated to on-wire lengths

for *etm MACs and GCM ciphers – the packet length is directly sent.

for chacha20 you look for TCP window updates of 36 bytes

you also need to add in a timing check. Since Pseudo-TTYs echo the characters you type. If you detect a client keypress and immediately within (N seconds) a server echo, you can declare a keypress event.

Detecting SSH Tunnels

This was the real motivation for me looking at this protocol in detail. SSH Tunnels are completely opaque to systems like Trisul/Bro/Suricata which look at packets. If a hostile element manages to establish an autossh reverse tunnel out of your network, you are really pWnd. He can come and go deep into your network at will, no firewall or IDS, IPS will detect that.

We had an earlier implementation mentioned in this blog Detecting Reverse SSH Tunnels that used a range of packet lengths to detect a tunnel. It was a reasonable approach but a lot would have slipped through because we didnt explictly track the MACs and Ciphers.

With SSH Tunnels there are two sessions.

Parent session – this has its own MAC/Cipher negotiated

Tunnel sessions – has a separate MAC/Cipher negotiated – ALLSSH messages on this tunnel are carried inside a “Channel” on the parent tunnel.

Channels — The parent SSH tunnel itself can be a live interactive session. SSH distinguishes tunneled traffic from through traffic using the Channel Number The Parent session is Channel 0 and the tunnels are Channel 1, 2, 3 etc.

Example of a single keystroke inside a SSH Tunnel. We have seen that a single keystroke in a SSH session using chacha20-poly1305 is 36 bytes. Now imagine that this is now carried inside a SSH tunnel that is also using chacha20-poly1305. How does one calculate the packet length? A single keystroke on a chacha20 SSH session that is run over a chacha20 tunnel is 76 bytes. The computation is quite simple if you followed the previous examples. Here is how that looks. Notice that ALL the traffic from the tunnel is carried inside a Channel 1 wrapped by the MSG_CHANNEL_DATA.

Once we got this far, we apply the same techniques to calculate a lookup table. See ssh_dissect.lua

Programming detection into your tool

Given the unmistakable move towards total encryption, traffic analysis techniques are going to become more important for “network security monitoring” platforms. Recently Damien Miller , an OpenSSH committer blogged about how thwarting traffic analysis was a high priority in his article ChaCha20 and Poly1305 in OpenSSH

Programming this in your tool of choice would include :

in the clear part of the SSH handshake process : synchonize the cipher/mac used

Successful login : When the packet lengths for Terminal Capability and the Shell request are seen

Keystroke after login : When the packet length for single key press and response are seen

SSH Tunnels : Packet lengths for tunnel key press are seen

The reassembly offered by the platform must be real time and unbuffered. This means that as soon as the TCP Chunks arrive and are reassembled they must be made available to the traffic analysis script.

You can turn off reassembly after the initial detection. To detect tunnels you only need the TCP Chunk sizes not a full blown reassembly.

How can SSH developers really thwart traffic analysis? Here are a couple of ideas from the top of my head

Implement the random padding requirement in RFC 4253 “Secure Shell Transport Protocol”. Today the OpenSSH suite pads up to the cipher block boundary they can extend it to pad to a random boundary . In my view, this has the drawback that the random padding has to be a multiple of block size so there are only so many discrete choices. Adding too much padding – say 512 bytes – for a single keystroke is just way too much overhead. On the other hand if you only pad to a few known values, we can easily account for those in the detection.

Use a technique like the TLSGREASE extensions to throw off traffic analysis by inserting bogus Terminal Capabilities or NOOP bogus keystrokes. This doesnt make much sense because unlike the TLS handshakes that GREASE focuses on, the SSH messages are encrypted. Bogus keystrokes could be hard to handle though.

Throw off TCP by buffering. This is just wicked. When TCP increments do not corresponding to the message lengths pushed it would be impossible to know when one message ends and another starts. They dont impact the *-etm MACs but for ChaCha20-Poly1305 and non-ETM this would be a dead end. We will see if the SSH developers are willing to go this far.

You can find the detection scripts we created for Trisul on our Github Repo trisul-scripts