Temporary exile

Server Breakage and Rebirth

Apr 13th, 2009

Now that I’ve caught up on the work I missed yesterday, now seems
like a good time to write about the massively bad server blowout I
had yesterday. Since my laptop’s upgrade from Debian Lenny to
Squeeze went so well, I got a little cavalier and was sloppy when
doing a dist-upgrade on a server on which I had done some
heavy-configuration. The summarized story is that I borked a kernel
dependency and actually suceeded in breaking the packaging system
on that machine, a first for me in at least the most recent
four-five years. Notice, still, that the breakage was caused by me
and not by the most-excellent Debian package management tools. It
worked out for the better as I had been sort-of-almost-definately
been meaning to rebuild that particular machine anyway. Since the
hardware is fine (though the disk might die, soonish), I
reinstalled Debian Squeeze from a nightly-build installer and took
the opportunity to change a few things in my network’s setup. The
first major change I made was to switch all the machines on my
network to use debtorrent
instead of apt-proxy which had
been behaving unreliably. In particular, apt-proxy had been
randomly hanging after a few transfers, thus causing the machines
using it to be unable to upgrade their packages or install new
ones. So, in an attempt to fix it and give a little back to the
community, I have installed debtorrent on my main server and
configured my other machines to use it. So far, it’s working quite
well, the download speed is rather fast and it caches packages so
that other machines may download them. The next thing I changed was
to take a few more security measures than I normally do. I have
been using various known strategies for some time now but a fresh
start seemed like a good opportunity to tighten things up with a
fresh install. First, the
Securing Debian
manual is required reading for any sysadmin and I re-read over it
while waiting for lengthy processes to finish. Since my machines
are already behind a firewall that only lets SSH traffic in and
then only through to the server in question, my revised security
checklist goes something like this:
1. Remove all RPC services:

`sudo aptitude --purge remove portmap nfs-common`

Remove root login option (especially since I disable the root
account) from /etc/ssh/sshd_config by making sure that the
relevant line reads PermitRootLogin no and then restarting ssh

Installing some helpful security packages:
sudo aptitude install debsums logcheck denyhosts chkrootkit and
then doing a dpkg-reconfigure on debsums to make sure it does a
daily integrity check and altering the
denyhosts config file to make
it sync with the global denyhosts database (this helped cut down on
automated ssh attacks tremendously)

One of the most important things to do is also to make sure
that you get your local mail delivered so that you can see status
reports. I do a sudo dpkg-reconfigure exim4-config to make
certain everything is as I like it and that no holes are left open
but I still get my system mail.

The last thing that I’ll do is to install nmap and scan myself
to see what’s showing. For this particular box, I saw nothing but
SSH and SMTP from the box itself and nothing but SSH from the
outside. Good.

There might be a few other things which I do but I can’t recall
them now. I would install SELinux but my understanding
(according to the Debian Wiki) is
that it’s still in the experimental stage so I won’t move on that
just yet. Is there something huge and obvious that I’m forgetting
security wise? Is a file-integrity checker going to be useful if I
have constantly-changing and files and I am continually-updating
packages? The other major change is that I moved from the XFS
filesystem back to ext3 with
the intention of soon trying the upgrade-in-place features found in
ext4 now that it’s got so many
things which I liked about XFS.
Since the Debian installer didn’t present me with an option to use
ext4, this seemed like the best idea. Was I very wrong? As a side
note, I tried out weechat on the
console for about an hour before immediately going back to
ERC
on Emacs because it integrates so well with my alltime favorite
editor.