by developers, for developers

We often have to work with remote servers in our jobs as developers. For simple projects you might chose FTP (or SFTP), which is fine for copying files, but for more complex applications which require more complex deployment processes, you will need something more powerful. There are many ways to access remote systems but SSH is probably one of the most widely used; deeper knowledge of it may simplify a developer’s tasks and allow them direct access to a remote operating system in a fast and secure way.

This post shows how to use SSH: not only to connect to a remote server, it shows how to use SSH in secure way and how to use it to automate tasks, and includes tips for all kinds of users. Developers may learn how to access remote databases behind a firewall, administrators will be interested in mosh (mobile shell) or the parallel execution of tasks, and even project managers may want to use an SSH proxy to bypass limitations of guest wifi during meetings. We’ll start out with some tips for tackling the unexpected.

Troubleshooting SSH Connections

If still something is wrong you can enable verbose output by adding the “-v” option to the SSH command and examining the output. In verbose mode SSH generates a lot of output, but there are some key things to look out for:

1

2

3

4

5

6

7

8

9

10

11

12

13

# ssh -v mprzytulski@example.inviqa.com 'uname -a'

OpenSSH_6.1p1-hpn13v11,OpenSSL1.0.1c10May2012

...

debug1:identity file/home/mprzytulski/.ssh/id_rsa type-1

debug1:identity file/home/mprzytulski/.ssh/id_rsa-cert type-1

debug1:identity file/home/mprzytulski/.ssh/id_dsa type-1

debug1:identity file/home/mprzytulski/.ssh/id_dsa-cert type-1

debug1:identity file/home/mprzytulski/.ssh/id_ecdsa type3

debug1:identity file/home/mprzytulski/.ssh/id_ecdsa-cert type-1

...

debug1:Authentications that can continue:publickey,keyboard-interactive

debug1:Next authentication method:publickey

...

The sections show tell you useful information like where your system looked for private keys, and whether your remote server supports public key authentication. The verbose output flag is a very handy switch to remember for the future, as it can help you solve many common SSH problems.

Multiple Servers and SSH-Agent

Now you know how to use SSH keys directly, we can simplify this process. If you use more than one key, or you require access to one server through another server, you might like to use a simple daemon called ssh-agent. This daemon is installed and running by default on Mac and *nix operating systems.

Most applications and tools which can authenticate on remote servers can use agent key authentication (for example this method is supported by: ssh, scp, git and rsync commands). Every application which supports agent authentication will try to connect with it using a unix socket file specified in the SSH_AUTH_SOCK environment variable; you also need to add your key to the agent to allow authentication with this key.

To work with ssh-agent, first of all we need to start the daemon, using this nasty-looking command:

1

2

# eval $(ssh-agent)

Agent pid12337

This starts the ssh-agent and exports all the necessary environment variables for the current user. We can check that everything is running correctly:

1

2

3

4

# echo $SSH_AUTH_SOCK

/tmp/ssh-d0oO5phFZovl/agent.12336

# ssh-add -l

The agent has no identities.

This shows that the agent is started and is listening on the socket, but when you try to list the registered identities, there aren’t any. We can add a key to the agent using the ssh-add command (if you leave the key name blank, it uses the default key):

SSH Agent Forwarding

Key-based authentication is very powerful but it does have limitations. For example consider the scenario where you need direct access to the server on which you want to authenticate, but in everyday work you may require authentication between the two servers – for example to copy files between them. You can copy your private key to the other remote server but this is insecure and is not a recommended solution. To solve this problem you may use the “agent forwarding” feature.

To enable SSH agent forwarding you need to add two lines to the SSH client configuration file located at ~/.ssh/config.
An example entry enabling ssh agent forwarding for inviqa.com would look something like:

1

2

Host example.inviqa.com

AgentForwarding yes

This simple configuration directive will enable the agent forwarding feature for every connection established with host example.inviqa.com. To check that everything works correctly, connect to the first server and check the value of the SSH_AUTH_SOCK environment variable.

1

2

# ssh mprzytulski@example.inviqa.com 'echo $SSH_AUTH_SOCK'

/tmp/ssh-d0oO5phFZovl/agent.12336

From now on, every connection established from the remote server example.inviqa.com will be authenticated through your local agent.

SSH Tunnelling

Now you know how to authenticate users on remote systems using keys, let’s move on and look at some more tricks you can do with SSH.

In general, the tools ssh, scp and rsync provide secure channels for remote command execution and file system manipulation, but using SSH we are also able to access other remote system resources like TCP/IP sockets or even use a remote system as a proxy to other resources.

SSH provides a very useful feature called “port forwarding” which allows you to forward ports on your local machine to ports on a remote system and vice versa.

In a nutshell, when using SSH tunnelling your SSH client is listening on a local TCP/IP port. When a connection is made to this port the data will be transported over a secure connection to the other end of the tunnel, which is on the remote server. Additionally, the SSH server will do same thing with data from the remote location, which will be securely transported back to your local machine.

Let’s consider an example: we want to connect a local application to a remote MySQL running on the standard port of 3306 on the server example.inviqa.com. We want to test our application with this remote database, but as it is inaccessible directly we use SSH tunnelling to allow the connection. To open new connection and establish secure tunnel we need to use this command:

1

ssh-L13306:127.0.0.1:3306example.inviqa.com

This command will instruct the SSH client to listen on the local port 13306, and tunnel all traffic on that port to 127.0.0.1:3306 port on the remote end of the tunnel (on the remote server). After that you can connect any tool to the remote MySQL server by connecting to your local port 13306. To check if everything works, try this simple telnet command:

1

2

3

4

5

6

# telnet 127.0.0.1 13306

Trying127.0.0.1...

Connected to127.0.0.1.

Escape character is'^]'.

N

5.5.30-log�)Cmz=aQQ#!�Z{|!'BUdxPf/mysql_native_password

The response there is MySQL’s binary protocol – so you know you are connected.

When working with SSH tunnelling there are a few things that you will need to know:

On Linux- and Unix-based operating systems, all ports bellow 1024 are privileged ports, which can only be opened for listening by users with root permissions. Any user can work with higher numbered ports however, so consider using these.

You must specify an unused local port, so you may choose a random high-numbered port when setting up a tunnel.

Reverse SSH Tunnels

Now you know how to open tunnel to a remote resource, you may use the same technique to open a tunnel which will allow your local resources to be accessed from a remote server, or even use remote server as a proxy to your local resource. This can be very useful if you work behind NAT and want to show results of your work to your client without uploading your code to a server.

To do this, you can use the following command to create the tunnel again, but this time going in the opposite direction:

1

ssh-R *:8080:127.0.0.1:80mprzytulski@example.inviqa.com

This command simply forwards all traffic from port 8080 on the remote host to your local port 80. As SSH by default binds to the loopback interface you need to specify interface to which you want to bind; this is done by specifying the first part of the “-R” parameter as ‘*’, which means that this can be changed to any available address on remote end.

To check that everything works as expected, you can compare the response from your localhost server to the response returned by tunnelled port using curl:

1

2

3

4

5

6

7

8

9

10

11

12

# curl -i 127.0.0.1

HTTP/1.1200OK

Server:nginx/1.2.6

Date:Tue,09Apr201305:48:29GMT

Content-Type:text/html

Transfer-Encoding:chunked

Connection:keep-alive

Keep-Alive:timeout=20

X-Powered-By:PHP/5.4.10--pl0-gentoo

SERVER_ADDR:127.0.0.1

REMOTE_ADDR:127.0.0.1

1

2

3

4

5

6

7

8

9

10

11

12

# curl -i http://example.inviqa.com:8080

HTTP/1.1200OK

Server:nginx/1.2.6

Date:Tue,09Apr201305:56:16GMT

Content-Type:text/html

Transfer-Encoding:chunked

Connection:keep-alive

Keep-Alive:timeout=20

X-Powered-By:PHP/5.4.10--pl0-gentoo

SERVER_ADDR:127.0.0.1

REMOTE_ADDR:127.0.0.1

The remote IP address is the same for both requests because all the traffic from the remote end of the tunnel is securely transported to your local computer, from which the IP is resolved. Using remote forwarding you are not limited to your local resources; you can forward traffic to any resource which is accessible from your computer.

Working With Multiple Servers

We know how to work with a single server but how about if we had five or even twenty servers, on which we need to do the same work? Perhaps you could write a simple bash loop to run the commands on each server, which may be fine for commands with a short execution time but if it creates backups of the databases and takes 20 minutes on each server, it will take a really long time. You could try to optimize your script by executing commands in the background or in separate screen sessions, or you may redirect stderr and stdout to files to handle output more efficiently, but you will need to solve more and more problems as you proceed with this approach. There is a simple, alternative way to be able to run the same commands on multiple servers; you can use package called PSSH, which contains simple parallel wrappers for standard SSH tools.

The PSSH package contains parallel versions of ssh, scp, rsync and two additional commands pnuke and pslurp which simplify operations on remote hosts.

For example, imagine we want to synchronise the system time on all of our servers, which can be done by executing ntpdate ntp2c.mcc.ac.uk on each one. To do this we can use the parallel version of SSH provided by the PSSH package. For testing purposes, we will provide a list of hosts as a part of the command:

By default PSSH will return only a status status indicator based on the return code from the executed command, but you may also capture output generated by adding the “-i” parameter to the call. This will return inline output and errors during execution of the command:

14Apr16:38:38ntpdate[30270]:adjust time server130.88.200.6offset0.341139sec

[2]15:38:38[SUCCESS]10.7.1.12

14Apr16:38:38ntpdate[31717]:adjust time server130.88.200.6offset-0.000000sec

[3]15:38:38[SUCCESS]10.7.1.11

14Apr16:38:38ntpdate[28134]:adjust time server130.88.200.6offset0.110710sec

You may notice that every time you execute PSSH commands, the results may be displayed in a different order, this is because the calls are executed in parallel, so the order of output depends which of the remote servers returns output first.

You may also store your list of servers in a separate file and use that with your PSSH commands. To do this, create a text file with a list of hosts (each host on a new line), and add the location of this file as a “-h” option to the PSSH tool rather than using lots of -H parameters. Here’s the same example but using this configuration file technique:

1

2

3

4

# pssh -l root -h ./hosts ‘ntpdate ntp2c.mcc.ac.uk’

[1]15:36:39[SUCCESS]10.7.1.10

[2]15:36:39[SUCCESS]10.7.1.11

[3]15:36:39[SUCCESS]10.7.1.12

From time to time you may need to add new servers to the list, and this may be problematic if you use a PSSH tool as a part of an automation script because when connecting to new host for the first time, you need to accept the host key fingerprint. One option, if you connect to trusted internal servers, is to temporarily disable host key verification by setting the following SSH options:

Piping Data Over SSH

Unix stream piping is one of the most powerful and widely-used techniques in everyday work with *nix systems. It allows you to transform data received on stdin and output the result to stdout, chaining a series of commands together. This simple approach allows for very powerful transformations and the speedup of many everyday tasks.

One common task is the copying of large file sets between servers. This can be done in many ways, perhaps using scp with the recursive flag, rsync or netcat – but I want to show you how to do this using data pipes.

Consider this example command:

1

# tar -cz /var/www/ | ssh root@10.7.1.10 'cat -&gt; ./www.tar.gz'

This simple command will create a compressed archive of the /var/www directory on your local machine, and put it directly into a www.tar.gz file on the remote server.

Doing it this way has several advantages:

Reduced network traffic as all data is compressed before any network transmission occurs

Reduced I/O operations as data is read from disk at source, any compression is performed in memory, and storage is performed on the remote server

Reduced free space requirements – as every operation excluding read is performed in memory or on remote server this technique is very useful when you need to create backup of data on server with limited disk capabilities

You can also extract data directly on remote server instead of creating archive. The command could look something like this:

1

# tar -cz /var/www/ | ssh root@10.7.1.10 'tar xzvf - -C /root/www'

With some small modifications, you can use data pipes for copying data from a remote server to a local archive. You can also use the dd command instead of cat to achieve this.

1

# ssh root@10.7.1.10 'tar czf - /var/www' | dd of=’/root/www.tar.gz’

As you can see, it is possible to use dd as a useful part of data pipes, so you can transfer even very large data sets over the network. For example you could try a command like this:

Running X11 applications

You already know how to authenticate to a remote system using different methods, to execute commands, and to transfer data between two servers, now its time to show you how to execute applications with a graphical user interface using a feature called “X11 forwarding”.

All Linux graphical applications rely on software called X Window System; applications connect with this server to render graphical output. Using X11 forwarding, it is possible to redirect a server application connection to a remote X Server in a similar way to normal port forwarding.

If you use Linux a with graphical user interface you already have X11 Server installed, users of Mac OS X 10.5 and below can use the built-in X11.app. On newer versions of Mac OS X you will need to install the Quartz application. Windows users may also get X11 Server from the Xming project.

After installing the X11 Server, we will be able to open a remote session with X11 forwarding, allowing us to see the graphical interface of a program running on one machine, visible on another. We can do this using the “-Y” switch for SSH.

1

# ssh -Y root@10.8.0.14

You can also start a graphical application immediately after the connection has been established:

1

# ssh -Y root@10.8.0.14 ‘xclock’

Mosh: the Mobile Shell

One last thing worthy of mention is mosh, an application designed as a replacement for SSH, especially for use on mobile networks. It is fast, secure and very fault tolerant, in fact you can start your session in one location and then reconnect to the same session over different connections or in other locations.

To use mosh it needs to be installed on both server and client, but the popular linux distributions all provide packages for mosh. Starting mosh session is much the same as starting an SSH session; the command looks like this:

1

mosh user@host.example.com

Once you’re connected with mosh you can enjoy a fault-tolerant connection to the server.

Making the Most of SSH

We’ve seen how we can use SSH connections not only to access a remote system via a shell, but also how to effectively interact between remote locations, copy data between servers, or even how to create your own ‘VPN’ using only SSH. SSH can do many more things besides – for example you may share the same connection between more than one session, edit files directly on server with your local editor via sshfs. If you would like to know a bit more about options for SSH, take a look at the man pages, and also the man page for the client side configuration file~/.ssh/.config which contains a lot of useful settings.
If you’ve got an SSH tip or trick to share, we’d love to hear it – please leave us a comment.