Lightweight, (Almost) Crypto-Free Remote System Operation

"There are two ways of constructing a software design. One
way is to make it so simple that there are obviously no deficiencies,
and the other is to make it so complicated that there are no obvious
deficiencies." - C.A.R. Hoare

"Sure I'm paranoid, but am I paranoid
ENOUGH?" - Unknown

Introduction

System administrators frequently want to be able to work on the machines
they run even when they are far away from them. There are secure tools
that allow full remote shell access, like ssh and lsh, but due to their
complexity they have suffered critical exploits from time to time.
In addition, their overhead can be excessive for some purposes. Fortunately,
other options are available that can be used alone or can be combined with
remote shells to create a more secure overall system.

Overview

Maybe the pager has just gone off when you're home in bed, and the boss
wants you to fix the broken database now. Or perhaps you're out
for lunch and someone calls to tell you the mailserver has been cracked
and is currently spamming the world, and you need to bring it down fast.
Possibly you've checked and your Web server has wedged itself and needs to
be restarted. Or suppose you're just on vacation and find you want to
update your home Web site with some new photos. In all these cases, you'd
like to do something to the machine over the Internet without having to
actually sit in front of it - things you don't want just anybody
to be able to do.

The Problem

Tools like ssh and lsh are great for allowing
secure remote access to your system. They offer essentially full, flexible
remote control of a machine, in an encrypted and authenticated manner. But
they are complex pieces of software; there's no way to do what they do
without being complex. And with complexity comes bugs. SSH and lsh,
and related tools like Webmin, have
all had serious flaws that would allow an attacker to get full control over
your system. Leaving them available all the time is a risk - sometimes it's
necessary, but it's still a risk. And in some cases, you'd like to be able
to tell the machine to do something, but it's not even attached to the
network on a regular basis.

Some Solutions

It would be nice to enable remote shell access only when necessary. And
perhaps (for something like shutting down a mail server) you don't even
need a full shell, just a way to fire off a script remotely. Of course, the
problem then becomes, how do you know that the alternative software is any
more secure than ssh itself? Various people have worked on this problem in
the past, and several potential solutions are available, ranging from the
simple and venerable to the new and exotic.

Xringd uses a modem to control a machine remotely.
Mail filters can be used to trigger actions based on
special messages. Some solutions (like 'port knocking'
and 'Net::Pcap') use the network, but without requiring
even a single open port. Lando runs commands over a
network, using username and password. Most recently, a program specifically
for secure remote execution called Ostiary has been
developed.

The Options

The eXtended Ring
Daemon, or "Xringd", uses a modem to monitor rings on a phone line. It
counts the number of rings, and the time between them. If a 'sequence'
matches one of the ones that it has been set up to detect, Xringd will run
an associated command.

This is very nice from a security perspective. Since it uses no network
connection at all, it's entirely immune to network attacks like buffer
overflows. It can be used even when a network connection is unavailable
(it's often used to cause a computer to initiate a dialup connection).
The only 'client' you need is a phone. If you use it to start up ssh on
demand, then the attacker needs to know the right phone number and the
right ring pattern - it's quite hard to sniff that kind of thing remotely.
It's also highly resistant to a man in the
middle attack. (If you have to worry about someone rerouting your
phone calls, you're in more trouble than Xringd can save you from.)

There are some practical issues that may make this unattractive in
some circumstances. You need a modem and a telephone line to the server.
(Fortunately, you don't need a fast modem at all; even a 1200 baud
one will do nicely, but some servers are not placed close to a telephone
jack.) Also, things like answering machines or voicemail (or even other
people answering the telephone) can interfere with Xringd. If you give the
server a dedicated line, you can avoid these problems, but that can be
costly.

Finally, note that the rings you hear when making a call are not
necessarily synchronized with the ring signals actually sent to the
telephone. In most circumstances, they are close enough, but reliability can
be an issue at times.

Most of the mail filtering programs have a way to invoke scripts when
mail matching a pattern is received (in the simplest case, mail to a
particular address). Assuming the server is running an SMTP daemon,
this can be a nice way of triggering
actions remotely. Technically, one could even send a shell script
to be run, and have it e-mail the results back to you, giving you the
equivalent of a very slow remote shell. The only client needed
is an e-mail program, or even a webmail account.

The first problem is that if the box you want to talk to doesn't accept
e-mail, this obviously won't work. (Adding an entire mail server, with the
attendant risks of bugs, spam load, etc., just for remote execution doesn't
make a lot of sense.) Some machines only periodically collect e-mail from
a primary server, so there can be a substantial delay between when a
command is sent and when it is acted upon.

Furthermore, if you don't encrypt the traffic in some way (or at least
sign it with PGP), then anyone sniffing traffic between you and your server
may be able to take advantage of the same channel to do mischief, or
perform a man-in-the-middle-attack. (E-mail traffic is notoriously easy to
falsify; hence the avalanche of spam these days.)

CVTSA, or "ClairVoyanT
SysAdmin", is a system designed specifically for running commands
through e-mail. It has some support for using passwords, but does not
(currently) encrypt them in transit, so a sniffer could capture them
and use them again.

Of course, if the only things you want to do with this type of system
are emergency shutdowns and other such (hopefully rare) crisis management,
then even an unencrypted channel might work. However, you'll need to
change the 'magic trigger pattern' each time after you use it, or you
take the risk that an attacker might capture it and 'replay' it at an
inconvenient time.

With port
knocking, a daemon monitors firewall logs, looking for particular
sequences of connection attempts to particular (closed) ports. When it
sees a sequence it recognizes, it runs the associated command. This
isn't terribly bandwidth efficient, but it has some nice properties.
First, it's hard to tell if a server is listening for port knocks.
Second (and most important), it's awfully hard to crack a closed
port. (Linksys routers have had a simple version of this for a while,
BTW, that they call port
triggering.)

However, a clever attacker with a sniffer could notice this traffic,
and duplicate it for their own use. More complicated encodings could
express something like a PGP signature (indeed, in theory one could
create an entire network protocol based on port knocks), but things
rapidly become difficult to work with. As with 'mail filtering'
solutions, one can either use it sparingly in emergencies, or move to
real cryptography.

It's also important to realize that this system is critically dependent
on the probe packets actually being delivered, and delivered in the
order that they were sent. This is not guaranteed on the Internet.
What's more, depending on where you're at (e.g., an Internet cafe or
behind a business firewall), you might not be allowed to connect out to
arbitrary ports. The more complex you make the 'knocks', the less
reliable the system will be.

Also, notice that at least one entire IP packet (28 bytes or so minimum)
is used to transmit roughly one bit of information. In terms of network
efficiency, it's almost hideous. For a simple 'open up ssh' message, it's
not a consideration, but actually adding cryptographic security to this
system could use up a decent chunk of the available bandwidth.

Finally, this increases the CPU load for each entry in the firewall
log. Depending on how detailed the logs are, and how fast and busy
the network is, this can be a significant drain on resources.

Another interesting approach is to use Net::Pcap
or other network capturing software to look for specific packets on the
network (e.g., DNS requests) and examine them for particular data (e.g.,
a particular address). If found, it can enable ssh temporarily, or
perform other actions.

One potential benefit of this approach is that a computer doesn't have
to have an address on a network in order to monitor traffic on
that network. You can set the card to 'promiscuous mode' and examine
all the traffic on the wire. (It's very hard to hack a machine
you don't even know is there.) Once the 'trigger' is spotted, the
sniffer can use other means (a separate network, a serial link, even
Xringd) to open up SSH on a target machine. Of course, you can also
simply run the sniffer directly on the target.

Again, a clever attacker with their own sniffer may be able to detect
the unusual activity and correlate it. To make this system truly
secure, you would need more complex encoding/encryption of the 'trigger'
traffic.

Additionally, the CPU load for this solution can be even worse than for
'port knocking' systems. A 'port knocking' daemon monitors firewall
logs, which can have variable levels of detail. By necessity, a
'sniffer' solution must examine every packet on the network
segment, which can be a substantial task for a busy gigabit line.

Lando allows a user to
run a preconfigured set of commands remotely, using passwords, and even
allowing the user to supply arguments to them. While it currently has
only a Windows client, and passwords are sent in the clear (making it
suitable only for use on a trusted local network, or perhaps on a VPN),
it can be very useful for, e.g. operating a local firewall box without
going to the trouble of logging in.

All of the above solutions have their advantages, but each has some
practical issues that can make them unsuitable for particular
applications. Ostiary was designed
to be a secure alternative that uses minimal resources. It tackles this
problem with what might be termed "aggressive simplicity". It does
require an active connection to the network (unlike Xringd and sniffing),
but allows for much better default security with very low CPU, RAM, disk,
and network bandwidth requirements.

An Ostiary server has one open port that it listens on. When someone
connects, the server sends a random fixed length 'salt' message 16
bytes in size - the size of an MD5 hash. It
then waits (with a timeout) for a reply from the client. It reads (at
most) 16 bytes of reply, and closes the connection.

Ostiary has a list of commands to run, with associated passwords. It
runs through the list, and hashes these passwords with
the 'salt' it sent to the client. If one of these hashes matches the
reply from the client, the associated command is run. (One final touch
is that a record is kept of connections, and clients with too many
failed attempts are 'locked out', and all subsequent communication from
them is ignored.)

A detailed security
analysis is available, but a few things about this system should
be clear. With a protocol this simple, the chances for dangerous
bugs are drastically reduced. Using fixed-length messages essentially
eliminates the chances of a buffer
overflow or other memory error. (Indeed, Ostiary does no dynamic
memory allocation of any kind - everything is stored in static,
fixed-size data structures.) Replay and man-in-the-middle attacks are
also effectively useless. Ostiary limits how fast it accepts connections,
enforcing low CPU and network usage. (The first production Ostiary server
was a 16MHz 68030 machine.) Client requirements are even lower: Clients
are available for Palm Pilots and even Windows.

Unlike a procmail-based solution, where you can put arbitrary commands (with
arguments) in the message, Ostiary can only run the fixed set of commands
you have preconfigured. The only argument it supplies to the commands is the
IP address of the client that initiated the command. It requires an active
network connection (unlike Xringd) and an open port (unlike port knocking or
sniffing), which may entail configuring a firewall to open a new port.
(Although one could run Ostiary on, say, port 22, and upon receipt of
the correct command, it could terminate itself and spawn sshd...)

Since Ostiary uses TCP, it is as reliable as the network it uses to
communicate. Problems with miscounted phone rings (a la Xringd) or randomly
dropped packets (a la port knocking) are not a concern.

Summary

The following table summarizes the pros and cons of the various systems
outlined above. "Replay" and "Man-in-the-middle" indicate if the default
system is vulnerable to the corresponding attacks. "Command arguments"
indicates if the system can run arbitrary commands with arguments. "CPU
load" indicates that CPU time can be a significant consideration. "Special
client" indicates that a specific client program is needed to work with
that system.

System

Xringd

Mail filter

Port knocking

Sniffers

Lando

Ostiary

Network Required?

Yes

Yes

Yes

Yes

Yes

Port Required?

Yes

Yes

Yes

Modem Required?

Yes

Replay?

Yes

Yes

Yes

Yes

Man-in-the-middle?

Yes

Yes

Yes

Yes

Command arguments?

Yes

Yes

CPU load?

Sometimes

Yes

Yes

Special client?

Sometimes

Sometimes

Yes

Yes

None of these approaches is right for everyone. But all of them can be
used to make attacks at least more inconvenient, and in many cases far
more difficult. Remember, though, to analyze their pros and cons
relative to your specific situation. Also remember that true security
is a process, not a goal - you can never just install some software
and be done thinking about it.

Ray Ingles has been involved with Linux since 1995. In
addition to being an active member of the
Metro Detroit Linux User's Group,
he has made minor contributions to the UPS HOWTO and the Linux
Joystick Driver.