Linux System Administration Best Practices

This guide will describe the decisions and tasks involved in running a
secure Linux or unix-based system. It does not contain step by step
instructions, because these vary between systems. You can use an
Internet search engine to find detailed instructions. If you are
affiliated with SEAS, you can contact CETS for advice, but we are not
familiar with every operating system, distribution, and version.

It is best if each machine has one person that is responsible for
managing and maintaining the machine. Other people can help out, but
if no one is in charge of making sure that the machine is patched and
backed up, then often everyone assumes that someone else will do it.

You have to worry about backups and security for every OS
installation, which includes dual boot and virtual installations. It
is very difficult to keep a machine patched if it is the "unbooted"
side of a dual-boot machine, so when it is booted it is often
immediately compromised. We don't support dual boot systems for this
reason - we simply can't keep them secure.

Backups

If you are responsible for a machine, you are responsible for the
files stored on that machine. You must decide how to back up the
files. If you are taking responsibility for a system that someone else
set up, you should review the backup decisions to make sure that you
understand them, and that they are still appropriate. A yearly test
restore and review of your backup model is a good idea.

Backups allow you to recover if your machine is compromised, or if
there is a disk failure or other hardware failure. Disks often fail
suddenly and without warning, and events such as fires, electrical
surges, and theft can cause data loss with no warning. Mirrors or
RAIDs do not protect data from fire, electrical surges, physical
theft, or compromised security (cyber break-ins).

In the past, people in SEAS have lost many months of work because they
did not have a reliable backup. This sort of disaster can result in
losing valuable research data, papers ready to submit for publication,
grant proposals in preparation, etc. Many grants require that the
computers involved in the research be administered according to
currently accepted "best practices" which include regular backups and
secure configuration.

There are five basic approaches to backing up your machine:

Store important files on the SEAS file server, and only store
temporary files, the OS and applications on your machine. If
necessary, the OS and applications can be reinstalled from original
media or from a trusted source.

Store important files on your local disks, and contract with CETS to
perform automatic network backups (see the "Help from CETS" section below).

Store non-volatile data (eg, input data) away from
the machine on DVD or other media, and have a working copy on the
machine's disks.

Have two machines, preferably in different rooms or buildings, and
automatically copy data from one to the other. This doesn't help if
the deleted or corrupted data is automatically copied before you
notice the problem. Ideally you would have more than one remote copy,
perhaps a daily copy and a weekly copy.

Copy your data periodically to a DVD or other media and store the
copies away from the machine. We don't recommend this method. In our
experience, when something goes wrong and you need the backups, you
discover that the person who was supposed to do the backups graduated
and your most recent backup is six months old.

It's not always possible to back up a database directly. The database
files may not be in an internally consistent state or data may be
stored in memory. Dump a snapshot of the database to flat files and
then back them up, or halt the database server, perform backups, and
restart the database server. See your database documentation for how to
do backups.

Security

We are seeing constant, aggressive attacks on SEAS machines. The
attackers are methodically scanning the network, so no machine is
safe. They have already gotten root on four SEAS machines that we
know of.

We have seen machines where every disk was wiped clean by someone who
had broken into the machine. Taking back control of a compromised
machine is time consuming, which can be disastrous if there is a
deadline approaching. The machine will be unavailable for a time, and
you and the other users of the system will need to spend time securing
your passwords and accounts.

Most of the compromises that we have detected have come in via SSH.
We have also seen successful attacks come against web applications. There
is no single remedy for these attacks, short of disconnecting your
machine from the network.

SSH

Consider turning off incoming SSH connections completely. SSH is used for remote shell access and remote execution of commands, and also for tunneling other protocols such as FTP and X Windows. SSH, like all network services, should be disabled if not needed.

If you need SSH, we recommend disabling root logins via SSH. Root is the account that they try the most and that you want to protect the most, so block all SSH logins to root.
To work remotely as root, login as a normal user and su to root. In most cases, you can just add

PermitRootLogin no

to /etc/ssh/sshd_config .

Lock out connections from IPs that have been trying bad passwords. Leave the IPs blocked forever.
The SSH parameter MaxAuthTries limits the number of failed passwords per connection, but does not permanently block the IP address.
The attacker can just connect back. Intrusion detection systems like OSSEC and fail2ban can monitor your system for failed SSH connections and block attackers automatically.

Running the SSH daemon on a different port can reduce the number of attacks that you will receive.

Firewall

Make sure that your system has a firewall. Different OS's have
different ways of implementing firewalls. For most Linux distros,
iptables is a good choice.

If possible, do not allow connections from unknown networks. Use
iptables or similar firewall software to allow only hosts or networks
that you and the other users use. When traveling, people can SSH to
eniac.seas.upenn.edu, and SSH from there to your machine.

The more tightly you can restrict network access, the better. It's
better to allow only machines at Penn, Stanford, and CMU, than to
leave it open to the entire Internet. Until we have a better sense of how IPv6 impacts system security, we recommend blocking all IPv6 incoming connections unless you have a specific need for IPv6.

Block all unnecessary services on your machine. You should be aware
of all network services offered by your computer, and you should know
how each is used and why it is necessary. FTP, X Windows, rsync, etc
should always be tunneled through SSH. You should not have ports open
for those services. Ask CETS to scan your machine and tell you what
ports are open on your machine.

Patches

If possible, configure your software to automatically install security
patches. Security patches are often released in response to attacks
that are actively being used throughout the Internet.

It is a good idea to install security patches the day that they are
announced. Unless you have a team of sysadmins or you want to install
patches when on vacation or over break, it's best to install them
automatically.

Attackers know that they have a small window of time before machines
are patched, so they try to attack as many machines as possible as
quickly as possible. Often these attacks are made on a holiday when
the attackers hope that most sysadmins will not be able to patch their
machines. These attacks are automated, with each newly compromised
machine joining in the attacking force.

Any application can have a security hole. For example, Oracle,
Adobe Reader, Adobe Flash, and Java have had serious security holes
that needed patching. Install any application patches the day that
they come out, same as OS patches.

Accounts

Set a root password that is at least ten characters and unusual. Only
use the root account for systems administration. Login as yourself
and su to root when you are doing systems administration.

At least once a year, you should remove or lock accounts that are no
longer needed, and make sure that the passwords on active accounts are
changed. If you can't get in touch with someone to discuss their
account and password, then we recommend removing or locking the
account. If you use email, make sure that you are not exchanging
email with someone who has cracked the email account. It's much
better to discuss account security by phone or in person.

You should have different passwords on machines managed by different
people or organizations. Confidential data should be protected by
different passwords or additional passwords. For example, use
different passwords for your PennKey password, a machine you run
yourself, online services, etc.

Other Suggestions

Email

Don't run a mail server; if you run a mail server you not only have to
worry about compromises to your machine through that service, but you
need to worry about preventing spammers and others from using your
mail server to send spam and phishing email. Running a mail server
securely is a huge pain, so just use the SEAS mail server, gmail,
hotmail, etc.

Mail sent to every account on your machine should be forwarded to
wherever that person reads mail, and mail sent to any system account
(eg "root") should be forwarded to wherever you read mail. Many
software services or systems send mail to "root" if something unusual
happens. Often break-ins are detected because of some small
side-effect which triggers a warning. For example, someone breaks in
and fills up a partition with large amounts of copyrighted material.
That causes some routine accounting task to fail because it can't
create a temporary file. You (root) get mail that a routine
accounting task failed, investigate, and discover that the machine has
been compromised.

Servers

If you are running a server, contact CETS. We can give you a static IP address and hostname. Web access is blocked to all SEASnet machines by default. If you are running a web server, we can enable web access to your machine.

Keep informed

Subscribe to the security announcement mailing list for your operating system version and any applications that you have installed. These lists will announce when new security patches are available. It will also tell you about security problems that do not yet have a patch, but which have a "work-around".

Go to https://forms.us-cert.gov/maillists/ and sign up for at least
alerts - better to sign up for everything and then unsubscribe if you
feel that you don't need a particular category.

Monitor your machine's logs and processes regularly, so that you can
recognize what's normal and unusual on your machine. Make sure NTP
is configured correctly and running (and will run at boot time).
This will ensure that the logs have accurate timestamps.

Prepare for the worst

Read this CERT article, and
make sure that you have everything that you'd need to recover from a
root compromise.

Make an archival backup of the operating system and applications in
order to speed up restores in the future to a known clean state.

Help from CETS

Contact your computer support organization if you see evidence that a
machine or an account has been compromised (this is required by Penn
policy). If you are affiliated with SEAS, contact CETS. We can help
you through the recovery process. We do not charge for providing
advice to SEAS sysadmins.

CETS can provide some services for a fee, such as providing weekly,
automated backups over our network. The cost of this service varies
depending on the amount of data that you have and other factors. For
current pricing please contact CETS. CETS can also do systems
administration work on machines, such as configuring and securing the
machine, setting up services, etc. There is an hourly rate charged for
this sort of work. The rate depends on the amount of experience of the
individual doing the work.

If you have any further questions, please feel free to contact cets@seas.